%0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e63754 %T Ensuring General Data Protection Regulation Compliance and Security in a Clinical Data Warehouse From a University Hospital: Implementation Study %A Riou,Christine %A El Azzouzi,Mohamed %A Hespel,Anne %A Guillou,Emeric %A Coatrieux,Gouenou %A Cuggia,Marc %K clinical data warehouse %K privacy %K personal data protection %K legislation %K security %K compliance %K personal data %K applicability %K experiential analysis %K university hospitals %K French %K France %K data hub %K operational challenge %D 2025 %7 17.4.2025 %9 %J JMIR Med Inform %G English %X Background: The European Union’s General Data Protection Regulation (GDPR) has profoundly influenced health data management, with significant implications for clinical data warehouses (CDWs). In 2021, France pioneered a national framework for GDPR-compliant CDW implementation, established by its data protection authority (Commission Nationale de l’Informatique et des Libertés). This framework provides detailed guidelines for health care institutions, offering a unique opportunity to assess practical GDPR implementation in health data management. Objective: This study evaluates the real-world applicability of France’s CDW framework through its implementation at a major university hospital. It identifies practical challenges for its implementation by health institutions and proposes adaptations relevant to regulatory authorities in order to facilitate research in secondary use data domains. Methods: A systematic assessment was conducted in May 2023 at the University Hospital of Rennes, which manages data for over 2 million patients through the eHOP CDW system. The evaluation examined 116 criteria across 13 categories using a dual-assessment approach validated by information security and data protection officers. Compliance was rated as met, unmet, or not applicable, with criteria classified as software-related (n=25) or institution-related (n=91). Results: Software-related criteria showed 60% (n=15) compliance, with 28% (n=7) noncompliant or partially compliant and 12% (n=3) not applicable. Institution-related criteria achieved 72% (n=28) compliance for security requirements. Key challenges included managing genetic data, implementing automated archiving, and controlling data exports. The findings revealed effective privacy protection measures but also highlighted areas requiring regulatory adjustments to better support research. Conclusions: This first empirical assessment of a national CDW compliance framework offers valuable insights for health care institutions implementing GDPR requirements. While the framework establishes robust privacy protections, certain provisions may overly constrain research activities. The study identifies opportunities for framework evolution, balancing data protection with research imperatives. %R 10.2196/63754 %U https://medinform.jmir.org/2025/1/e63754 %U https://doi.org/10.2196/63754 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e64113 %T Identification of Patients With Congestive Heart Failure From the Electronic Health Records of Two Hospitals: Retrospective Study %A Sumsion,Daniel %A Davis,Elijah %A Fernandes,Marta %A Wei,Ruoqi %A Milde,Rebecca %A Veltink,Jet Malou %A Kong,Wan-Yee %A Xiong,Yiwen %A Rao,Samvrit %A Westover,Tara %A Petersen,Lydia %A Turley,Niels %A Singh,Arjun %A Buss,Stephanie %A Mukerji,Shibani %A Zafar,Sahar %A Das,Sudeshna %A Junior,Valdery Moura %A Ghanta,Manohar %A Gupta,Aditya %A Kim,Jennifer %A Stone,Katie %A Mignot,Emmanuel %A Hwang,Dennis %A Trotti,Lynn Marie %A Clifford,Gari D %A Katwa,Umakanth %A Thomas,Robert %A Westover,M Brandon %A Sun,Haoqi %+ Department of Neurology, Beth Israel Deaconess Medical Center, DA-0815, 330 Brookline Avenue, Boston, MA, 02215, United States, 1 781 539 9402, hsun3@bidmc.harvard.edu %K electronic health record %K machine learning %K artificial intelligence %K phenotype %K congestive heart failure %K medication %K claims database %K International Classification of Diseases %K effectiveness %K natural language processing %K model performance %K logistic regression %K validity %D 2025 %7 10.4.2025 %9 Original Paper %J JMIR Med Inform %G English %X Background: Congestive heart failure (CHF) is a common cause of hospital admissions. Medical records contain valuable information about CHF, but manual chart review is time-consuming. Claims databases (using International Classification of Diseases [ICD] codes) provide a scalable alternative but are less accurate. Automated analysis of medical records through natural language processing (NLP) enables more efficient adjudication but has not yet been validated across multiple sites. Objective: We seek to accurately classify the diagnosis of CHF based on structured and unstructured data from each patient, including medications, ICD codes, and information extracted through NLP of notes left by providers, by comparing the effectiveness of several machine learning models. Methods: We developed an NLP model to identify CHF from medical records using electronic health records (EHRs) from two hospitals (Mass General Hospital and Beth Israel Deaconess Medical Center; from 2010 to 2023), with 2800 clinical visit notes from 1821 patients. We trained and compared the performance of logistic regression, random forests, and RoBERTa models. We measured model performance using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). These models were also externally validated by training the data on one hospital sample and testing on the other, and an overall estimated error was calculated using a completely random sample from both hospitals. Results: The average age of the patients was 66.7 (SD 17.2) years; 978 (54.3%) out of 1821 patients were female. The logistic regression model achieved the best performance using a combination of ICD codes, medications, and notes, with an AUROC of 0.968 (95% CI 0.940-0.982) and an AUPRC of 0.921 (95% CI 0.835-0.969). The models that only used ICD codes or medications had lower performance. The estimated overall error rate in a random EHR sample was 1.6%. The model also showed high external validity from training on Mass General Hospital data and testing on Beth Israel Deaconess Medical Center data (AUROC 0.927, 95% CI 0.908-0.944) and vice versa (AUROC 0.968, 95% CI 0.957-0.976). Conclusions: The proposed EHR-based phenotyping model for CHF achieved excellent performance, external validity, and generalization across two institutions. The model enables multiple downstream uses, paving the way for large-scale studies of CHF treatment effectiveness, comorbidities, outcomes, and mechanisms. %M 40208662 %R 10.2196/64113 %U https://medinform.jmir.org/2025/1/e64113 %U https://doi.org/10.2196/64113 %U http://www.ncbi.nlm.nih.gov/pubmed/40208662 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e68704 %T Improving Phenotyping of Patients With Immune-Mediated Inflammatory Diseases Through Automated Processing of Discharge Summaries: Multicenter Cohort Study %A Remaki,Adam %A Ung,Jacques %A Pages,Pierre %A Wajsburt,Perceval %A Liu,Elise %A Faure,Guillaume %A Petit-Jean,Thomas %A Tannier,Xavier %A Gérardin,Christel %+ Limics, Université Sorbonne Paris-Nord, Inserm, Sorbonne Université, 15 Rue de l'École de Médecine, Paris, 75006, France, 33 783049678, ad.remaki@gmail.com %K secondary use of clinical data for research and surveillance %K clinical informatics %K clinical data warehouse %K electronic health record %K data science %K artificial intelligence %K AI %K natural language processing %K ontologies %K classifications %K coding %K tools %K programs and algorithms %K immune-mediated inflammatory diseases %D 2025 %7 9.4.2025 %9 Original Paper %J JMIR Med Inform %G English %X Background: Valuable insights gathered by clinicians during their inquiries and documented in textual reports are often unavailable in the structured data recorded in electronic health records (EHRs). Objective: This study aimed to highlight that mining unstructured textual data with natural language processing techniques complements the available structured data and enables more comprehensive patient phenotyping. A proof-of-concept for patients diagnosed with specific autoimmune diseases is presented, in which the extraction of information on laboratory tests and drug treatments is performed. Methods: We collected EHRs available in the clinical data warehouse of the Greater Paris University Hospitals from 2012 to 2021 for patients hospitalized and diagnosed with 1 of 4 immune-mediated inflammatory diseases: systemic lupus erythematosus, systemic sclerosis, antiphospholipid syndrome, and Takayasu arteritis. Then, we built, trained, and validated natural language processing algorithms on 103 discharge summaries selected from the cohort and annotated by a clinician. Finally, all discharge summaries in the cohort were processed with the algorithms, and the extracted data on laboratory tests and drug treatments were compared with the structured data. Results: Named entity recognition followed by normalization yielded F1-scores of 71.1 (95% CI 63.6-77.8) for the laboratory tests and 89.3 (95% CI 85.9-91.6) for the drugs. Application of the algorithms to 18,604 EHRs increased the detection of antibody results and drug treatments. For instance, among patients in the systemic lupus erythematosus cohort with positive antinuclear antibodies, the rate increased from 18.34% (752/4102) to 71.87% (2949/4102), making the results more consistent with the literature. Conclusions: While challenges remain in standardizing laboratory tests, particularly with abbreviations, this work, based on secondary use of clinical data, demonstrates that automated processing of discharge summaries enriched the information available in structured data and facilitated more comprehensive patient profiling. %M 40203304 %R 10.2196/68704 %U https://medinform.jmir.org/2025/1/e68704 %U https://doi.org/10.2196/68704 %U http://www.ncbi.nlm.nih.gov/pubmed/40203304 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 14 %N %P e60917 %T Conversion of Sensitive Data to the Observational Medical Outcomes Partnership Common Data Model: Protocol for the Development and Use of Carrot %A Cox,Samuel %A Masood,Erum %A Panagi,Vasiliki %A Macdonald,Calum %A Milligan,Gordon %A Horban,Scott %A Santos,Roberto %A Hall,Chris %A Lea,Daniel %A Tarr,Simon %A Mumtaz,Shahzad %A Akashili,Emeka %A Rae,Andy %A Urwin,Esmond %A Cole,Christian %A Sheikh,Aziz %A Jefferson,Emily %A Quinlan,Philip Roy %+ NIHR Nottingham Biomedical Research Centre, School of Medicine, University of Nottingham, Queens Medical Centre, Nottingham, NG7 2UH, United Kingdom, 44 0115 951 5151, philip.quinlan@nottingham.ac.uk %K data standardization %K OMOP %K Observational Medical Outcomes Partnership %K ETL %K extract, transform, and load %K data discovery %K transparency %K Carrot tool %K common data model %K data standard %K health care %K data model %K data protection %K data privacy %K open-source %D 2025 %7 2.4.2025 %9 Protocol %J JMIR Res Protoc %G English %X Background: The use of data standards is low across the health care system, and converting data to a common data model (CDM) is usually required to undertake international research. One such model is the Observational Medical Outcomes Partnership (OMOP) CDM. It has gained substantial traction across researchers and those who have developed data platforms. The Observational Health Care Data Sciences and Informatics (OHDSI) partnership manages OMOP and provides many open-source tools to assist in converting data to the OMOP CDM. The challenge, however, is in the skills, knowledge, know-how, and capacity within teams to convert their data to OMOP. The European Health Care Data Evidence Network provided funds to allow data owners to bring in external resources to do the required conversions. The Carrot software (University of Nottingham) is a new set of open-source tools designed to help address these challenges while not requiring data access by external resources. Objective: The use of data protection rules is increasing, and privacy by design is a core principle under the European and UK legislations related to data protection. Our aims for the Carrot software were to have a standardized mechanism for managing the data curation process, capturing the rules used to convert the data, and creating a platform that can reuse rules across projects to drive standardization of process and improve the speed without compromising on quality. Most importantly, we aimed to deliver this design-by-privacy approach without requiring data access to those creating the rules. Methods: The software was developed using Agile approaches by both software engineers and data engineers, who would ultimately use the system. Experts in OMOP were used to ensure the approaches were correct. An incremental release program was initiated to ensure we delivered continuous progress. Results: Carrot has been delivered and used on a project called COVID-Curated and Open Analysis and Research Platform (CO-CONNECT) to assist in the process of allowing datasets to be discovered via a federated platform. It has been used to create over 45,000 rules, and over 5 million patient records have been converted. This has been achieved while maintaining our principle of not allowing access to the underlying data by the team creating the rules. It has also facilitated the reuse of existing rules, with most rules being reused rather than manually curated. Conclusions: Carrot has demonstrated how it can be used alongside existing OHDSI tools with a focus on the mapping stage. The COVID-Curated and Open Analysis and Research Platform project successfully managed to reuse rules across datasets. The approach is valid and brings the benefits expected, with future work continuing to optimize the generation of rules. International Registered Report Identifier (IRRID): RR1-10.2196/60917 %M 40173432 %R 10.2196/60917 %U https://www.researchprotocols.org/2025/1/e60917 %U https://doi.org/10.2196/60917 %U http://www.ncbi.nlm.nih.gov/pubmed/40173432 %0 Journal Article %@ 2561-7605 %I JMIR Publications %V 8 %N %P e65178 %T Unsupervised Deep Learning of Electronic Health Records to Characterize Heterogeneity Across Alzheimer Disease and Related Dementias: Cross-Sectional Study %A West,Matthew %A Cheng,You %A He,Yingnan %A Leng,Yu %A Magdamo,Colin %A Hyman,Bradley T %A Dickson,John R %A Serrano-Pozo,Alberto %A Blacker,Deborah %A Das,Sudeshna %+ Massachusetts General Hospital, 65 Landsdowne Street, Cambridge, MA, 02139, United States, 1 617 768 8254, sdas5@mgh.harvard.edu %K Alzheimer disease and related dementias %K electronic health records %K large language models %K clustering %K unsupervised learning %D 2025 %7 31.3.2025 %9 Original Paper %J JMIR Aging %G English %X Background: Alzheimer disease and related dementias (ADRD) exhibit prominent heterogeneity. Identifying clinically meaningful ADRD subtypes is essential for tailoring treatments to specific patient phenotypes. Objective: We aimed to use unsupervised learning techniques on electronic health records (EHRs) from memory clinic patients to identify ADRD subtypes. Methods: We used pretrained embeddings of non-ADRD diagnosis codes (International Classification of Diseases, Ninth Revision) and large language model (LLM)–derived embeddings of clinical notes from patient EHRs. Hierarchical clustering of these embeddings was used to identify ADRD subtypes. Clusters were characterized regarding their demographic and clinical features. Results: We analyzed a cohort of 3454 patients with ADRD from a memory clinic at Massachusetts General Hospital, each with a specialist diagnosis. Clustering pretrained embeddings of the non-ADRD diagnosis codes in patient EHRs revealed the following 3 patient subtypes: one with skin conditions, another with psychiatric disorders and an earlier age of onset, and a third with diabetes complications. Similarly, using LLM-derived embeddings of clinical notes, we identified 3 subtypes of patients as follows: one with psychiatric manifestations and higher prevalence of female participants (prevalence ratio: 1.59), another with cardiovascular and motor problems and higher prevalence of male participants (prevalence ratio: 1.75), and a third one with geriatric health disorders. Notably, we observed significant overlap between clusters from both data modalities (χ24=89.4; P<.001). Conclusions: By integrating International Classification of Diseases, Ninth Revision codes and LLM-derived embeddings, our analysis delineated 2 distinct ADRD subtypes with sex-specific comorbid and clinical presentations, offering insights for potential precision medicine approaches. %M 40163031 %R 10.2196/65178 %U https://aging.jmir.org/2025/1/e65178 %U https://doi.org/10.2196/65178 %U http://www.ncbi.nlm.nih.gov/pubmed/40163031 %0 Journal Article %@ 2817-1705 %I JMIR Publications %V 4 %N %P e62985 %T Limitations of Binary Classification for Long-Horizon Diagnosis Prediction and Advantages of a Discrete-Time Time-to-Event Approach: Empirical Analysis %A Loh,De Rong %A Hill,Elliot D %A Liu,Nan %A Dawson,Geraldine %A Engelhard,Matthew M %K machine learning %K artificial intelligence %K deep learning %K predictive models %K practical models %K early detection %K electronic health records %K right-censoring %K survival analysis %K distributional shifts %D 2025 %7 27.3.2025 %9 %J JMIR AI %G English %X Background: A major challenge in using electronic health records (EHR) is the inconsistency of patient follow-up, resulting in right-censored outcomes. This becomes particularly problematic in long-horizon event predictions, such as autism and attention-deficit/hyperactivity disorder (ADHD) diagnoses, where a significant number of patients are lost to follow-up before the outcome can be observed. Consequently, fully supervised methods such as binary classification (BC), which are trained to predict observed diagnoses, are substantially affected by the probability of sufficient follow-up, leading to biased results. Objective: This empirical analysis aims to characterize BC’s inherent limitations for long-horizon diagnosis prediction from EHR; and quantify the benefits of a specific time-to-event (TTE) approach, the discrete-time neural network (DTNN). Methods: Records within the Duke University Health System EHR were analyzed, extracting features such as ICD-10 (International Classification of Diseases, Tenth Revision) diagnosis codes, medications, laboratories, and procedures. We compared a DTNN to 3 BC approaches and a deep Cox proportional hazards model across 4 clinical conditions to examine distributional patterns across various subgroups. Time-varying area under the receiving operating characteristic curve (AUCt) and time-varying average precision (APt) were our primary evaluation metrics. Results: TTE models consistently had comparable or higher AUCt and APt than BC for all conditions. At clinically relevant operating time points, the area under the receiving operating characteristic curve (AUC) values for DTNNYOB≤2020 (year-of-birth) and DCPHYOB≤2020 (deep Cox proportional hazard) were 0.70 (95% CI 0.66‐0.77) and 0.72 (95% CI 0.66‐0.78) at t=5 for autism, 0.72 (95% CI 0.65‐0.76) and 0.68 (95% CI 0.62‐0.74) at t=7 for ADHD, 0.72 (95% CI 0.70‐0.75) and 0.71 (95% CI 0.69‐0.74) at t=1 for recurrent otitis media, and 0.74 (95% CI 0.68‐0.82) and 0.71 (95% CI 0.63‐0.77) at t=1 for food allergy, compared to 0.6 (95% CI 0.55‐0.66), 0.47 (95% CI 0.40‐0.54), 0.73 (95% CI 0.70‐0.75), and 0.77 (95% CI 0.71‐0.82) for BCYOB≤2020, respectively. The probabilities predicted by BC models were positively correlated with censoring times, particularly for autism and ADHD prediction. Filtering strategies based on YOB or length of follow-up only partially corrected these biases. In subgroup analyses, only DTNN predicted diagnosis probabilities that accurately reflect actual clinical prevalence and temporal trends. Conclusions: BC models substantially underpredicted diagnosis likelihood and inappropriately assigned lower probability scores to individuals with earlier censoring. Common filtering strategies did not adequately address this limitation. TTE approaches, particularly DTNN, effectively mitigated bias from the censoring distribution, resulting in superior discrimination and calibration performance and more accurate prediction of clinical prevalence. Machine learning practitioners should recognize the limitations of BC for long-horizon diagnosis prediction and adopt TTE approaches. The DTNN in particular is well-suited to mitigate the effects of right-censoring and maximize prediction performance in this setting. %R 10.2196/62985 %U https://ai.jmir.org/2025/1/e62985 %U https://doi.org/10.2196/62985 %0 Journal Article %@ 2817-1705 %I JMIR Publications %V 4 %N %P e67363 %T Generative Large Language Model—Powered Conversational AI App for Personalized Risk Assessment: Case Study in COVID-19 %A Roshani,Mohammad Amin %A Zhou,Xiangyu %A Qiang,Yao %A Suresh,Srinivasan %A Hicks,Steven %A Sethuraman,Usha %A Zhu,Dongxiao %+ Department of Computer Science, Wayne State University, 5057 Woodward Ave, Suite 14101.3, Detroit, MI, 48202, United States, 1 3135773104, dzhu@wayne.edu %K personalized risk assessment %K large language model %K conversational AI %K artificial intelligence %K COVID-19 %D 2025 %7 27.3.2025 %9 Original Paper %J JMIR AI %G English %X Background: Large language models (LLMs) have demonstrated powerful capabilities in natural language tasks and are increasingly being integrated into health care for tasks like disease risk assessment. Traditional machine learning methods rely on structured data and coding, limiting their flexibility in dynamic clinical environments. This study presents a novel approach to disease risk assessment using generative LLMs through conversational artificial intelligence (AI), eliminating the need for programming. Objective: This study evaluates the use of pretrained generative LLMs, including LLaMA2-7b and Flan-T5-xl, for COVID-19 severity prediction with the goal of enabling a real-time, no-code, risk assessment solution through chatbot-based, question-answering interactions. To contextualize their performance, we compare LLMs with traditional machine learning classifiers, such as logistic regression, extreme gradient boosting (XGBoost), and random forest, which rely on tabular data. Methods: We fine-tuned LLMs using few-shot natural language examples from a dataset of 393 pediatric patients, developing a mobile app that integrates these models to provide real-time, no-code, COVID-19 severity risk assessment through clinician-patient interaction. The LLMs were compared with traditional classifiers across different experimental settings, using the area under the curve (AUC) as the primary evaluation metric. Feature importance derived from LLM attention layers was also analyzed to enhance interpretability. Results: Generative LLMs demonstrated strong performance in low-data settings. In zero-shot scenarios, the T0-3b-T model achieved an AUC of 0.75, while other LLMs, such as T0pp(8bit)-T and Flan-T5-xl-T, reached 0.67 and 0.69, respectively. At 2-shot settings, logistic regression and random forest achieved an AUC of 0.57, while Flan-T5-xl-T and T0-3b-T obtained 0.69 and 0.65, respectively. By 32-shot settings, Flan-T5-xl-T reached 0.70, similar to logistic regression (0.69) and random forest (0.68), while XGBoost improved to 0.65. These results illustrate the differences in how generative LLMs and traditional models handle the increasing data availability. LLMs perform well in low-data scenarios, whereas traditional models rely more on structured tabular data and labeled training examples. Furthermore, the mobile app provides real-time, COVID-19 severity assessments and personalized insights through attention-based feature importance, adding value to the clinical interpretation of the results. Conclusions: Generative LLMs provide a robust alternative to traditional classifiers, particularly in scenarios with limited labeled data. Their ability to handle unstructured inputs and deliver personalized, real-time assessments without coding makes them highly adaptable to clinical settings. This study underscores the potential of LLM-powered conversational artificial intelligence (AI) in health care and encourages further exploration of its use for real-time, disease risk assessment and decision-making support. %M 40146990 %R 10.2196/67363 %U https://ai.jmir.org/2025/1/e67363 %U https://doi.org/10.2196/67363 %U http://www.ncbi.nlm.nih.gov/pubmed/40146990 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 9 %N %P e54803 %T Synthetic Data-Driven Approaches for Chinese Medical Abstract Sentence Classification: Computational Study %A Li,Jiajia %A Wang,Zikai %A Yu,Longxuan %A Liu,Hui %A Song,Haitao %K medical abstract sentence classification %K large language models %K synthetic datasets %K deep learning %K Chinese medical %K dataset %K traditional Chinese medicine %K global medical research %K algorithm %K robustness %K efficiency %K accuracy %D 2025 %7 19.3.2025 %9 %J JMIR Form Res %G English %X Background: Medical abstract sentence classification is crucial for enhancing medical database searches, literature reviews, and generating new abstracts. However, Chinese medical abstract classification research is hindered by a lack of suitable datasets. Given the vastness of Chinese medical literature and the unique value of traditional Chinese medicine, precise classification of these abstracts is vital for advancing global medical research. Objective: This study aims to address the data scarcity issue by generating a large volume of labeled Chinese abstract sentences without manual annotation, thereby creating new training datasets. Additionally, we seek to develop more accurate text classification algorithms to improve the precision of Chinese medical abstract classification. Methods: We developed 3 training datasets (dataset #1, dataset #2, and dataset #3) and a test dataset to evaluate our model. Dataset #1 contains 15,000 abstract sentences translated from the PubMed dataset into Chinese. Datasets #2 and #3, each with 15,000 sentences, were generated using GPT-3.5 from 40,000 Chinese medical abstracts in the CSL database. Dataset #2 used titles and keywords for pseudolabeling, while dataset #3 aligned abstracts with category labels. The test dataset includes 87,000 sentences from 20,000 abstracts. We used SBERT embeddings for deeper semantic analysis and evaluated our model using clustering (SBERT-DocSCAN) and supervised methods (SBERT-MEC). Extensive ablation studies and feature analyses were conducted to validate the model’s effectiveness and robustness. Results: Our experiments involved training both clustering and supervised models on the 3 datasets, followed by comprehensive evaluation using the test dataset. The outcomes demonstrated that our models outperformed the baseline metrics. Specifically, when trained on dataset #1, the SBERT-DocSCAN model registered an impressive accuracy and F1-score of 89.85% on the test dataset. Concurrently, the SBERT-MEC algorithm exhibited comparable performance with an accuracy of 89.38% and an identical F1-score. Training on dataset #2 yielded similarly positive results for the SBERT-DocSCAN model, achieving an accuracy and F1-score of 89.83%, while the SBERT-MEC algorithm recorded an accuracy of 86.73% and an F1-score of 86.51%. Notably, training with dataset #3 allowed the SBERT-DocSCAN model to attain the best with an accuracy and F1-score of 91.30%, whereas the SBERT-MEC algorithm also showed robust performance, obtaining an accuracy of 90.39% and an F1-score of 90.35%. Ablation analysis highlighted the critical role of integrated features and methodologies in improving classification efficiency. Conclusions: Our approach addresses the challenge of limited datasets for Chinese medical abstract classification by generating novel datasets. The deployment of SBERT-DocSCAN and SBERT-MEC models significantly enhances the precision of classifying Chinese medical abstracts, even when using synthetic datasets with pseudolabels. %R 10.2196/54803 %U https://formative.jmir.org/2025/1/e54803 %U https://doi.org/10.2196/54803 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e65651 %T Assessment of the Efficiency of a ChatGPT-Based Tool, MyGenAssist, in an Industry Pharmacovigilance Department for Case Documentation: Cross-Over Study %A Benaïche,Alexandre %A Billaut-Laden,Ingrid %A Randriamihaja,Herivelo %A Bertocchio,Jean-Philippe %+ Bayer Healthcare SAS France, 1 Rue Claude Bernard, Lille, 59000, France, 33 320445962, benaichealexandre@gmail.com %K MyGenAssist %K large language model %K artificial intelligence %K ChatGPT %K pharmacovigilance %K efficiency %D 2025 %7 10.3.2025 %9 Original Paper %J J Med Internet Res %G English %X Background: At the end of 2023, Bayer AG launched its own internal large language model (LLM), MyGenAssist, based on ChatGPT technology to overcome data privacy concerns. It may offer the possibility to decrease their harshness and save time spent on repetitive and recurrent tasks that could then be dedicated to activities with higher added value. Although there is a current worldwide reflection on whether artificial intelligence should be integrated into pharmacovigilance, medical literature does not provide enough data concerning LLMs and their daily applications in such a setting. Here, we studied how this tool could improve the case documentation process, which is a duty for authorization holders as per European and French good vigilance practices. Objective: The aim of the study is to test whether the use of an LLM could improve the pharmacovigilance documentation process. Methods: MyGenAssist was trained to draft templates for case documentation letters meant to be sent to the reporters. Information provided within the template changes depending on the case: such data come from a table sent to the LLM. We then measured the time spent on each case for a period of 4 months (2 months before using the tool and 2 months after its implementation). A multiple linear regression model was created with the time spent on each case as the explained variable, and all parameters that could influence this time were included as explanatory variables (use of MyGenAssist, type of recipient, number of questions, and user). To test if the use of this tool impacts the process, we compared the recipients’ response rates with and without the use of MyGenAssist. Results: An average of 23.3% (95% CI 13.8%-32.8%) of time saving was made thanks to MyGenAssist (P<.001; adjusted R2=0.286) on each case, which could represent an average of 10.7 (SD 3.6) working days saved each year. The answer rate was not modified by the use of MyGenAssist (20/48, 42% vs 27/74, 36%; P=.57) whether the recipient was a physician or a patient. No significant difference was found regarding the time spent by the recipient to answer (mean 2.20, SD 3.27 days vs mean 2.65, SD 3.30 days after the last attempt of contact; P=.64). The implementation of MyGenAssist for this activity only required a 2-hour training session for the pharmacovigilance team. Conclusions: Our study is the first to show that a ChatGPT-based tool can improve the efficiency of a good practice activity without needing a long training session for the affected workforce. These first encouraging results could be an incentive for the implementation of LLMs in other processes. %M 40063946 %R 10.2196/65651 %U https://www.jmir.org/2025/1/e65651 %U https://doi.org/10.2196/65651 %U http://www.ncbi.nlm.nih.gov/pubmed/40063946 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e65590 %T Data Interoperability in COVID-19 Vaccine Trials: Methodological Approach in the VACCELERATE Project %A Malik,Salma %A Dorothea,Zoi Pana %A Argyropoulos,Christos D %A Themistocleous,Sophia %A Macken,Alan J %A Valdenmaiier,Olena %A Scheckenbach,Frank %A Bardach,Elena %A Pfeiffer,Andrea %A Loens,Katherine %A Ochando,Jordi Cano %A Cornely,Oliver A %A Demotes-Mainard,Jacques %A Contrino,Sergio %A Felder,Gerd %+ European Clinical Research Infrastructure Network, 30 Bd Saint-Jacques, Paris, 75014, France, 33 01 80 05 86 29, salma.malik@ecrin.org %K interoperability %K metadata %K data management %K clinical trials %K protocol %K harmonization %K adult %K pediatric %K systems %K standards %D 2025 %7 7.3.2025 %9 Original Paper %J JMIR Med Inform %G English %X Background: Data standards are not only key to making data processing efficient but also fundamental to ensuring data interoperability. When clinical trial data are structured according to international standards, they become significantly easier to analyze, reducing the efforts required for data cleaning, preprocessing, and secondary use. A common language and a shared set of expectations facilitate interoperability between systems and devices. Objective: The main objectives of this study were to identify commonalities and differences in clinical trial metadata, protocols, and data collection systems/items within the VACCELERATE project. Methods: To assess the degree of interoperability achieved in the project and suggest methodological improvements, interoperable points were identified based on the core outcome areas—immunogenicity, safety, and efficacy (clinical/physiological). These points were emphasized in the development of the master protocol template and were manually compared in the following ways: (1) summaries, objectives, and end points in the protocols of 3 VACCELERATE clinical trials (EU-COVAT-1_AGED, EU-COVAT-2_BOOSTAVAC, and EU-COVPT-1_CoVacc) against the master protocol template; (2) metadata of all 3 clinical trials; and (3) evaluations from a questionnaire survey regarding differences in data management systems and structures that enabled data exchange within the VACCELERATE network. Results: The noncommonalities identified in the protocols and metadata were attributed to differences in populations, variations in protocol design, and vaccination patterns. The detailed metadata released for all 3 vaccine trials were clearly structured using internal standards, terminology, and the general approach of Clinical Data Acquisition Standards Harmonisation (CDASH) for data collection (eg, on electronic case report forms). VACCELERATE benefited significantly from the selection of the Clinical Trials Centre Cologne as the sole data management provider. With system database development coordinated by a single individual and no need for coordination among different trial units, a high degree of uniformity was achieved automatically. The harmonized transfer of data to all sites, using well-established methods, enabled quick exchanges and provided a relatively secure means of data transfer. Conclusions: This study demonstrated that using master protocols can significantly enhance trial operational efficiency and data interoperability, provided that similar infrastructure and data management procedures are adopted across multiple trials. To further improve data interoperability and facilitate interpretation and analysis, shared data should be structured, described, formatted, and stored using widely recognized data and metadata standards. Trial Registration: EudraCT 2021-004526-29; https://www.clinicaltrialsregister.eu/ctr-search/trial/2021-004526-29/DE/; 2021-004889-35; https://www.clinicaltrialsregister.eu/ctr-search/search?query=eudract_number:2021-004889-35; and 2021-004526-29; https://www.clinicaltrialsregister.eu/ctr-search/search?query=eudract_number:2021-004526-29 %M 40056469 %R 10.2196/65590 %U https://medinform.jmir.org/2025/1/e65590 %U https://doi.org/10.2196/65590 %U http://www.ncbi.nlm.nih.gov/pubmed/40056469 %0 Journal Article %@ 2817-1705 %I JMIR Publications %V 4 %N %P e64279 %T Deep Learning Models to Predict Diagnostic and Billing Codes Following Visits to a Family Medicine Practice: Development and Validation Study %A Rajaram,Akshay %A Judd,Michael %A Barber,David %K machine learning %K ML %K artificial intelligence %K algorithm %K predictive model %K predictive analytics %K predictive system %K family medicine %K primary care %K family doctor %K family physician %K income %K billing code %K electronic notes %K electronic health record %K electronic medical record %K EMR %K patient record %K health record %K personal health record %D 2025 %7 7.3.2025 %9 %J JMIR AI %G English %X Background: Despite significant time spent on billing, family physicians routinely make errors and miss billing opportunities. In other disciplines, machine learning models have predicted Current Procedural Terminology codes with high accuracy. Objective: Our objective was to derive machine learning models capable of predicting diagnostic and billing codes from notes recorded in the electronic medical record. Methods: We conducted a retrospective algorithm development and validation study involving an academic family medicine practice. Visits between July 1, 2015, and June 30, 2020, containing a physician-authored note and an invoice in the electronic medical record were eligible for inclusion. We trained 2 deep learning models and compared their predictions to codes submitted for reimbursement. We calculated accuracy, recall, precision, F1-score, and area under the receiver operating characteristic curve. Results: Of the 245,045 visits eligible for inclusion, 198,802 (81%) were included in model development. Accuracy was 99.8% and 99.5% for the diagnostic and billing code models, respectively. Recall was 49.4% and 70.3% for the diagnostic and billing code models, respectively. Precision was 55.3% and 76.7% for the diagnostic and billing code models, respectively. The area under the receiver operating characteristic curve was 0.983 for the diagnostic code model and 0.993 for the billing code model. Conclusions: We developed models capable of predicting diagnostic and billing codes from electronic notes following visits to a family medicine practice. The billing code model outperformed the diagnostic code model in terms of recall and precision, likely due to fewer codes being predicted. Work is underway to further enhance model performance and assess the generalizability of these models to other family medicine practices. %R 10.2196/64279 %U https://ai.jmir.org/2025/1/e64279 %U https://doi.org/10.2196/64279 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e68863 %T Performance Improvement of a Natural Language Processing Tool for Extracting Patient Narratives Related to Medical States From Japanese Pharmaceutical Care Records by Increasing the Amount of Training Data: Natural Language Processing Analysis and Validation Study %A Ohno,Yukiko %A Aomori,Tohru %A Nishiyama,Tomohiro %A Kato,Riri %A Fujiki,Reina %A Ishikawa,Haruki %A Kiyomiya,Keisuke %A Isawa,Minae %A Mochizuki,Mayumi %A Aramaki,Eiji %A Ohtani,Hisakazu %+ Graduate School of Pharmaceutical Sciences, Keio University, 1-5-30, Shibakouen, Minato, Tokyo, 105-8512, Japan, 81 3 5400 2486, ohtani@keio.jp %K natural language processing %K NLP %K named entity recognition %K NER %K deep learning %K pharmaceutical care record %K electronic medical record %K EMR %K Japanese %D 2025 %7 4.3.2025 %9 Original Paper %J JMIR Med Inform %G English %X Background: Patients’ oral expressions serve as valuable sources of clinical information to improve pharmacotherapy. Natural language processing (NLP) is a useful approach for analyzing unstructured text data, such as patient narratives. However, few studies have focused on using NLP for narratives in the Japanese language. Objective: We aimed to develop a high-performance NLP system for extracting clinical information from patient narratives by examining the performance progression with a gradual increase in the amount of training data. Methods: We used subjective texts from the pharmaceutical care records of Keio University Hospital from April 1, 2018, to March 31, 2019, comprising 12,004 records from 6559 cases. After preprocessing, we annotated diseases and symptoms within the texts. We then trained and evaluated a deep learning model (bidirectional encoder representations from transformers combined with a conditional random field [BERT-CRF]) through 10-fold cross-validation. The annotated data were divided into 10 subsets, and the amount of training data was progressively increased over 10 steps. We also analyzed the causes of errors. Finally, we applied the developed system to the analysis of case report texts to evaluate its usability for texts from other sources. Results: The F1-score of the system improved from 0.67 to 0.82 as the amount of training data increased from 1200 to 12,004 records. The F1-score reached 0.78 with 3600 records and was largely similar thereafter. As performance improved, errors from incorrect extractions decreased significantly, which resulted in an increase in precision. For case reports, the F1-score also increased from 0.34 to 0.41 as the training dataset expanded from 1200 to 12,004 records. Performance was lower for extracting symptoms from case report texts compared with pharmaceutical care records, suggesting that this system is more specialized for analyzing subjective data from pharmaceutical care records. Conclusions: We successfully developed a high-performance system specialized in analyzing subjective data from pharmaceutical care records by training a large dataset, with near-complete saturation of system performance with about 3600 training records. This system will be useful for monitoring symptoms, offering benefits for both clinical practice and research. %M 40053805 %R 10.2196/68863 %U https://medinform.jmir.org/2025/1/e68863 %U https://doi.org/10.2196/68863 %U http://www.ncbi.nlm.nih.gov/pubmed/40053805 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e67984 %T A Validation Tool (VaPCE) for Postcoordinated SNOMED CT Expressions: Development and Usability Study %A Ohlsen,Tessa %A Hofer,Viola %A Ingenerf,Josef %K SNOMED CT %K PCE %K postcoordination %K FHIR %K validation %K postcoordinated expression %K Fast Healthcare Interoperability Resource %D 2025 %7 28.2.2025 %9 %J JMIR Med Inform %G English %X Background: The digitalization of health care has increased the demand for efficient data exchange, emphasizing semantic interoperability. SNOMED Clinical Terms (SNOMED CT), a comprehensive terminology with over 360,000 medical concepts, supports this need. However, it cannot cover all medical scenarios, particularly in complex cases. To address this, SNOMED CT allows postcoordination, where users combine precoordinated concepts with new expressions. Despite SNOMED CT’s potential, the creation and validation of postcoordinated expressions (PCEs) remain challenging due to complex syntactic and semantic rules. Objective: This work aims to develop a tool that validates postcoordinated SNOMED CT expressions, focusing on providing users with detailed, automated correction instructions for syntactic and semantic errors. The goal is not just validation, but also offering user-friendly, actionable suggestions for improving PCEs. Methods: A tool was created using the Fast Healthcare Interoperability Resource (FHIR) service $validate-code and the terminology server Ontoserver to check the correctness of PCEs. When errors are detected, the tool processes the SNOMED CT Concept Model in JSON format and applies predefined error categories. For each error type, specific correction suggestions are generated and displayed to users. The key added value of the tool is in generating specific correction suggestions for each identified error, which are displayed to the users. The tool was integrated into a web application, where users can validate individual PCEs or bulk-upload files. The tool was tested with real existing PCEs, which were used as input and validated. In the event of errors, appropriate error messages were generated as output. Results: In the validation of 136 PCEs from 304 FHIR Questionnaires, 18 (13.2%) PCEs were invalid, with the most common errors being invalid attribute values. Additionally, 868 OncoTree codes were evaluated, resulting in 161 (20.9%) PCEs containing inactive concepts, which were successfully replaced with valid alternatives. A user survey reflects a favorable evaluation of the tool’s functionality. Participants found the error categorization and correction suggestions to be precise, offering clear guidance for addressing issues. However, there is potential for enhancement, particularly regarding the level of detail in the error messages. Conclusions: The validation tool significantly improves the accuracy of postcoordinated SNOMED CT expressions by not only identifying errors but also offering detailed correction instructions. This approach supports health care professionals in ensuring that their PCEs are syntactically and semantically valid, enhancing data quality and interoperability across systems. %R 10.2196/67984 %U https://medinform.jmir.org/2025/1/e67984 %U https://doi.org/10.2196/67984 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 9 %N %P e63396 %T Exploring Metadata Catalogs in Health Care Data Ecosystems: Taxonomy Development Study %A Scheider,Simon %A Mallick,Mostafa Kamal %+ Fraunhofer Institute for Software and Systems Engineering, Speicherstraße 6, Dortmund, 44147, Germany, 49 231976774, simon.scheider@isst.fraunhofer.de %K data catalogs %K data ecosystems %K findability, accessibility, interoperability, and reusability %K FAIR %K health care %K metadata %K taxonomy %D 2025 %7 18.2.2025 %9 Original Paper %J JMIR Form Res %G English %X Background: In the European health care industry, recent years have seen increasing investments in data ecosystems to “FAIRify” and capitalize the ever-rising amount of health data. Within such networks, health metadata catalogs (HMDCs) assume a key function as they enable data allocation, sharing, and use practices. By design, HMDCs orchestrate health information for the purpose of findability, accessibility, interoperability, and reusability (FAIR). However, despite various European initiatives pushing health care data ecosystems forward, actionable design knowledge about HMDCs is scarce. This impedes both their effective development in practice and their scientific exploration, causing huge unused innovation potential of health data. Objective: This study aims to explore the structural design elements of HMDCs, classifying them alongside empirically reasonable dimensions and characteristics. In doing so, the development of HMDCs in practice is facilitated while also closing a crucial gap in theory (ie, the literature about actionable HMDC design knowledge). Methods: We applied a rigorous methodology for taxonomy building following well-known and established guidelines from the domain of information systems. Within this methodological framework, inductive and deductive research methods were applied to iteratively design and evaluate the evolving set of HMDC dimensions and characteristics. Specifically, a systematic literature review was conducted to identify and analyze 38 articles, while a multicase study was conducted to examine 17 HMDCs from practice. These findings were evaluated and refined in 2 extensive focus group sessions by 7 interdisciplinary experts with deep knowledge about HMDCs. Results: The artifact generated by the study is an iteratively conceptualized and empirically grounded taxonomy with elaborate explanations. It proposes 20 dimensions encompassing 101 characteristics alongside which FAIR HMDCs can be structured and classified. The taxonomy describes basic design characteristics that need to be considered to implement FAIR HMDCs effectively. A major finding was that a particular focus in developing HMDCs is on the design of their published dataset offerings (ie, their metadata assets) as well as on data security and governance. The taxonomy is evaluated against the background of 4 use cases, which were cocreated with experts. These illustrative scenarios add depth and context to the taxonomy as they underline its relevance and applicability in real-world settings. Conclusions: The findings contribute fundamental, yet actionable, design knowledge for building HMDCs in European health care data ecosystems. They provide guidance for health care practitioners, while allowing both scientists and policy makers to navigate through this evolving research field and anchor their work. Therefore, this study closes the research gap outlined earlier, which has prevailed in theory and practice. %M 39964739 %R 10.2196/63396 %U https://formative.jmir.org/2025/1/e63396 %U https://doi.org/10.2196/63396 %U http://www.ncbi.nlm.nih.gov/pubmed/39964739 %0 Journal Article %@ 2292-9495 %I JMIR Publications %V 12 %N %P e59961 %T Application of Clinical Department–Specific AI-Assisted Coding Using Taiwan Diagnosis-Related Groups: Retrospective Validation Study %A Lu,An-Tai %A Liou,Chong-Sin %A Lai,Chia-Hsin %A Shian,Bo-Tsz %A Li,Ming-Ta %A Sun,Chih-Yen %A Kao,Hao-Yun %A Dai,Hong-Jie %A Tsai,Ming-Ju %K diagnosis-related group %K artificial intelligence coding %K International Classification of Diseases, Tenth Revision, Clinical Modification %K ICD-10-CM %K coding professionals %D 2025 %7 12.2.2025 %9 %J JMIR Hum Factors %G English %X Background: The accuracy of the ICD-10-CM (International Classification of Diseases, Tenth Revision, Clinical Modification) procedure coding system (PCS) is crucial for generating correct Taiwan diagnosis-related groups (DRGs), as coding errors can lead to financial losses for hospitals. Objective: The study aimed to determine the consistency between an artificial intelligence (AI)-assisted coding module and manual coding, as well as to identify clinical specialties suitable for implementing the developed AI-assisted coding module. Methods: This study examined the AI-assisted coding module from the perspective of health care professionals. The research period started in February 2023. The study excluded cases outside of Taiwan DRGs, those with incomplete medical records, and cases with Taiwan DRG disposals ICD-10 (International Statistical Classification of Diseases, Tenth Revision) PCS. Data collection was conducted through retrospective medical record review. The AI-assisted module was constructed using a hierarchical attention network. The verification of the Taiwan DRGs results from the AI-assisted coding model focused on the major diagnostic categories (MDCs). Statistical computations were conducted using SPSS version 19. Research variables consisted of categorical variables represented by MDC, and continuous variables were represented by the relative weight of Taiwan DRGs. Results: A total of 2632 discharge records meeting the research criteria were collected from February to April 2023. In terms of inferential statistics, κ statistics were used for MDC analysis. The infectious and parasitic diseases MDC, as well as the respiratory diseases MDC had κ values exceeding 0.8. Clinical inpatient specialties were statistically analyzed using the Wilcoxon signed rank test. There was not a difference in coding results between the 23 clinical departments, such as the Division of Cardiology, the Division of Nephrology, and the Department of Urology. Conclusions: For human coders, with the assistance of the ICD-10-CM AI-assisted coding system, work time is reduced. Additionally, strengthening knowledge in clinical documentation enables human coders to maximize their role. This positions them to become clinical documentation experts, preparing them for further career development. Future research will apply the same method to validate the ICD-10 AI-assisted coding module. %R 10.2196/59961 %U https://humanfactors.jmir.org/2025/1/e59961 %U https://doi.org/10.2196/59961 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 9 %N %P e60095 %T Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection %A Puts,Sander %A Zegers,Catharina M L %A Dekker,Andre %A Bermejo,Iñigo %K International Classification of Diseases %K ICD-10 %K computer-assisted-coding %K GPT-4 %K coding %K term extraction %K code analysis %K computer assisted coding %K transformer model %K artificial intelligence %K AI automation %K retrieval-augmented generation %K RAG %K large language model %K LLM %K Bidirectional Encoder Representations from Transformers %K Robustly Optimized BERT Pretraining Approach %K RoBERTa %K named entity recognition %K NER %D 2025 %7 11.2.2025 %9 %J JMIR Form Res %G English %X Background: The International Classification of Diseases (ICD), developed by the World Health Organization, standardizes health condition coding to support health care policy, research, and billing, but artificial intelligence automation, while promising, still underperforms compared with human accuracy and lacks the explainability needed for adoption in medical settings. Objective: The potential of large language models for assisting medical coders in the ICD-10 coding was explored through the development of a computer-assisted coding system. This study aimed to augment human coding by initially identifying lead terms and using retrieval-augmented generation (RAG)–based methods for computer-assisted coding enhancement. Methods: The explainability dataset from the CodiEsp challenge (CodiEsp-X) was used, featuring 1000 Spanish clinical cases annotated with ICD-10 codes. A new dataset, CodiEsp-X-lead, was generated using GPT-4 to replace full-textual evidence annotations with lead term annotations. A Robustly Optimized BERT (Bidirectional Encoder Representations from Transformers) Pretraining Approach transformer model was fine-tuned for named entity recognition to extract lead terms. GPT-4 was subsequently employed to generate code descriptions from the extracted textual evidence. Using a RAG approach, ICD codes were assigned to the lead terms by querying a vector database of ICD code descriptions with OpenAI’s text-embedding-ada-002 model. Results: The fine-tuned Robustly Optimized BERT Pretraining Approach achieved an overall F1-score of 0.80 for ICD lead term extraction on the new CodiEsp-X-lead dataset. GPT-4-generated code descriptions reduced retrieval failures in the RAG approach by approximately 5% for both diagnoses and procedures. However, the overall explainability F1-score for the CodiEsp-X task was limited to 0.305, significantly lower than the state-of-the-art F1-score of 0.633. The diminished performance was partly due to the reliance on code descriptions, as some ICD codes lacked descriptions, and the approach did not fully align with the medical coder’s workflow. Conclusions: While lead term extraction showed promising results, the subsequent RAG-based code assignment using GPT-4 and code descriptions was less effective. Future research should focus on refining the approach to more closely mimic the medical coder’s workflow, potentially integrating the alphabetic index and official coding guidelines, rather than relying solely on code descriptions. This alignment may enhance system accuracy and better support medical coders in practice. %R 10.2196/60095 %U https://formative.jmir.org/2025/1/e60095 %U https://doi.org/10.2196/60095 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e67589 %T An Ontology for Digital Medicine Outcomes: Development of the Digital Medicine Outcomes Value Set (DOVeS) %A Rosner,Benjamin %A Horridge,Matthew %A Austria,Guillen %A Lee,Tiffany %A Auerbach,Andrew %+ , Division of Clinical Informatics and Digital Transformation, University of California San Francisco, 10 Koret Way, Room K-301, UCSF DOM CLIIR Box 0737, San Francisco, CA, 94143, United States, 1 415 502 1371, benjamin.rosner@ucsf.edu %K digital health %K digital medicine %K digital therapeutics %K ontology %K medical informatics %K value set %K ontology %K development %K digital health tool %K DHT %K health systems %K digital medicine outcomes value set %K prototype %K users %D 2025 %7 6.2.2025 %9 Original Paper %J JMIR Med Inform %G English %X Background: Over the last 10-15 years, US health care and the practice of medicine itself have been transformed by a proliferation of digital medicine and digital therapeutic products (collectively, digital health tools [DHTs]). While a number of DHT classifications have been proposed to help organize these tools for discovery, retrieval, and comparison by health care organizations seeking to potentially implement them, none have specifically addressed that organizations considering their implementation approach the DHT discovery process with one or more specific outcomes in mind. An outcomes-based DHT ontology could therefore be valuable not only for health systems seeking to evaluate tools that influence certain outcomes, but also for regulators and vendors seeking to ascertain potential substantial equivalence to predicate devices. Objective: This study aimed to develop, with inputs from industry, health care providers, payers, regulatory bodies, and patients through the Accelerated Digital Clinical Ecosystem (ADviCE) consortium, an ontology specific to DHT outcomes, the Digital medicine Outcomes Value Set (DOVeS), and to make this ontology publicly available and free to use. Methods: From a starting point of a 4-generation–deep hierarchical taxonomy developed by ADviCE, we developed DOVeS using the Web Ontology Language through the open-source ontology editor Protégé, and data from 185 vendors who had submitted structured product information to ADviCE. We used a custom, decentralized, collaborative ontology engineering methodology, and were guided by Open Biological and Biomedical Ontologies (OBO) Foundry principles. We incorporated the Mondo Disease Ontology (MONDO) and the Ontology of Adverse Events. After development, DOVeS was field-tested between December 2022 and May 2023 with 40 additional independent vendors previously unfamiliar with ADviCE or DOVeS. As a proof of concept, we subsequently developed a prototype DHT Application Finder leveraging DOVeS to enable a user to query for DHT products based on specific outcomes of interest. Results: In its current state, DOVeS contains 42,320 and 9481 native axioms and distinct classes, respectively. These numbers are enhanced when taking into account the axioms and classes contributed by MONDO and the Ontology of Adverse Events. Conclusions: DOVeS is publicly available on BioPortal and GitHub, and has a Creative Commons license CC-BY-SA that is intended to encourage stakeholders to modify, adapt, build upon, and distribute it. While no ontology is complete, DOVeS will benefit from a strong and engaged user base to help it grow and evolve in a way that best serves DHT stakeholders and the patients they serve. %M 39914801 %R 10.2196/67589 %U https://medinform.jmir.org/2025/1/e67589 %U https://doi.org/10.2196/67589 %U http://www.ncbi.nlm.nih.gov/pubmed/39914801 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e54133 %T Robust Automated Harmonization of Heterogeneous Data Through Ensemble Machine Learning: Algorithm Development and Validation Study %A Yang,Doris %A Zhou,Doudou %A Cai,Steven %A Gan,Ziming %A Pencina,Michael %A Avillach,Paul %A Cai,Tianxi %A Hong,Chuan %K ensemble learning %K semantic learning %K distribution learning %K variable harmonization %K machine learning %K cardiovascular health study %K intracohort comparison %K intercohort comparison %K gold standard labels %D 2025 %7 22.1.2025 %9 %J JMIR Med Inform %G English %X Background: Cohort studies contain rich clinical data across large and diverse patient populations and are a common source of observational data for clinical research. Because large scale cohort studies are both time and resource intensive, one alternative is to harmonize data from existing cohorts through multicohort studies. However, given differences in variable encoding, accurate variable harmonization is difficult. Objective: We propose SONAR (Semantic and Distribution-Based Harmonization) as a method for harmonizing variables across cohort studies to facilitate multicohort studies. Methods: SONAR used semantic learning from variable descriptions and distribution learning from study participant data. Our method learned an embedding vector for each variable and used pairwise cosine similarity to score the similarity between variables. This approach was built off 3 National Institutes of Health cohorts, including the Cardiovascular Health Study, the Multi-Ethnic Study of Atherosclerosis, and the Women’s Health Initiative. We also used gold standard labels to further refine the embeddings in a supervised manner. Results: The method was evaluated using manually curated gold standard labels from the 3 National Institutes of Health cohorts. We evaluated both the intracohort and intercohort variable harmonization performance. The supervised SONAR method outperformed existing benchmark methods for almost all intracohort and intercohort comparisons using area under the curve and top-k accuracy metrics. Notably, SONAR was able to significantly improve harmonization of concepts that were difficult for existing semantic methods to harmonize. Conclusions: SONAR achieves accurate variable harmonization within and between cohort studies by harnessing the complementary strengths of semantic learning and variable distribution learning. %R 10.2196/54133 %U https://medinform.jmir.org/2025/1/e54133 %U https://doi.org/10.2196/54133 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e55721 %T An Intelligent System for Classifying Patient Complaints Using Machine Learning and Natural Language Processing: Development and Validation Study %A Li,Xiadong %A Shu,Qiang %A Kong,Canhong %A Wang,Jinhu %A Li,Gang %A Fang,Xin %A Lou,Xiaomin %A Yu,Gang %+ Children's Hospital, Zhejiang University School of Medicine, National Clinical Research Center For Child Health, No. 3333, Binsheng Road, Binjiang District, Hangzhou, Hang Zhou, 310020, China, 86 13588773370, yugbme@zju.edu.cn %K complaint analysis %K text classification %K natural language processing %K NLP %K machine learning %K ML %K patient complaints %D 2025 %7 8.1.2025 %9 Original Paper %J J Med Internet Res %G English %X Background: Accurate classification of patient complaints is crucial for enhancing patient satisfaction management in health care settings. Traditional manual methods for categorizing complaints often lack efficiency and precision. Thus, there is a growing demand for advanced and automated approaches to streamline the classification process. Objective: This study aimed to develop and validate an intelligent system for automatically classifying patient complaints using machine learning (ML) and natural language processing (NLP) techniques. Methods: An ML-based NLP technology was proposed to extract frequently occurring dissatisfactory words related to departments, staff, and key treatment procedures. A dataset containing 1465 complaint records from 2019 to 2023 was used for training and validation, with an additional 376 complaints from Hangzhou Cancer Hospital serving as an external test set. Complaints were categorized into 4 types—communication problems, diagnosis and treatment issues, management problems, and sense of responsibility concerns. The imbalanced data were balanced using the Synthetic Minority Oversampling Technique (SMOTE) algorithm to ensure equal representation across all categories. A total of 3 ML algorithms (Multifactor Logistic Regression, Multinomial Naive Bayes, and Support Vector Machines [SVM]) were used for model training and validation. The best-performing model was tested using a 5-fold cross-validation on external data. Results: The original dataset consisted of 719, 376, 260, and 86 records for communication problems, diagnosis and treatment issues, management problems, and sense of responsibility concerns, respectively. The Multifactor Logistic Regression and SVM models achieved weighted average accuracies of 0.89 and 0.93 in the training set, and 0.83 and 0.87 in the internal test set, respectively. Ngram-level term frequency–inverse document frequency did not significantly improve classification performance, with only a marginal 1% increase in precision, recall, and F1-score when implementing Ngram-level term frequency–inverse document frequency (n=2) from 0.91 to 0.92. The SVM algorithm performed best in prediction, achieving an average accuracy of 0.91 on the external test set with a 95% CI of 0.87-0.97. Conclusions: The NLP-driven SVM algorithm demonstrates effective classification performance in automatically categorizing patient complaint texts. It showed superior performance in both internal and external test sets for communication and management problems. However, caution is advised when using it for classifying sense of responsibility complaints. This approach holds promises for implementation in medical institutions with high complaint volumes and limited resources for addressing patient feedback. %M 39778195 %R 10.2196/55721 %U https://www.jmir.org/2025/1/e55721 %U https://doi.org/10.2196/55721 %U http://www.ncbi.nlm.nih.gov/pubmed/39778195 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e57824 %T Applying AI to Structured Real-World Data for Pharmacovigilance Purposes: Scoping Review %A Dimitsaki,Stella %A Natsiavas,Pantelis %A Jaulent,Marie-Christine %+ Laboratoire d'Informatique Médicale et d'Ingénierie des Connaissances en e-Santé - LIMICS, Inserm, Université Sorbonne Paris-Nord, Sorbonne Université, 15 Rue de l'École de Médecine, Paris, 75006, France, 33 767968072, Stella.Dimitsaki@etu.sorbonne-universite.fr %K pharmacovigilance %K drug safety %K artificial intelligence %K machine learning %K real-world data %K scoping review %D 2024 %7 30.12.2024 %9 Review %J J Med Internet Res %G English %X Background: Artificial intelligence (AI) applied to real-world data (RWD; eg, electronic health care records) has been identified as a potentially promising technical paradigm for the pharmacovigilance field. There are several instances of AI approaches applied to RWD; however, most studies focus on unstructured RWD (conducting natural language processing on various data sources, eg, clinical notes, social media, and blogs). Hence, it is essential to investigate how AI is currently applied to structured RWD in pharmacovigilance and how new approaches could enrich the existing methodology. Objective: This scoping review depicts the emerging use of AI on structured RWD for pharmacovigilance purposes to identify relevant trends and potential research gaps. Methods: The scoping review methodology is based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We queried the MEDLINE database through the PubMed search engine. Relevant scientific manuscripts published from January 2010 to January 2024 were retrieved. The included studies were “mapped” against a set of evaluation criteria, including applied AI approaches, code availability, description of the data preprocessing pipeline, clinical validation of AI models, and implementation of trustworthy AI criteria following the guidelines of the FUTURE (Fairness, Universality, Traceability, Usability, Robustness, and Explainability)-AI initiative. Results: The scoping review ultimately yielded 36 studies. There has been a significant increase in relevant studies after 2019. Most of the articles focused on adverse drug reaction detection procedures (23/36, 64%) for specific adverse effects. Furthermore, a substantial number of studies (34/36, 94%) used nonsymbolic AI approaches, emphasizing classification tasks. Random forest was the most popular machine learning approach identified in this review (17/36, 47%). The most common RWD sources used were electronic health care records (28/36, 78%). Typically, these data were not available in a widely acknowledged data model to facilitate interoperability, and they came from proprietary databases, limiting their availability for reproducing results. On the basis of the evaluation criteria classification, 10% (4/36) of the studies published their code in public registries, 16% (6/36) tested their AI models in clinical environments, and 36% (13/36) provided information about the data preprocessing pipeline. In addition, in terms of trustworthy AI, 89% (32/36) of the studies followed at least half of the trustworthy AI initiative guidelines. Finally, selection and confounding biases were the most common biases in the included studies. Conclusions: AI, along with structured RWD, constitutes a promising line of work for drug safety and pharmacovigilance. However, in terms of AI, some approaches have not been examined extensively in this field (such as explainable AI and causal AI). Moreover, it would be helpful to have a data preprocessing protocol for RWD to support pharmacovigilance processes. Finally, because of personal data sensitivity, evaluation procedures have to be investigated further. %M 39753222 %R 10.2196/57824 %U https://www.jmir.org/2024/1/e57824 %U https://doi.org/10.2196/57824 %U http://www.ncbi.nlm.nih.gov/pubmed/39753222 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e60017 %T Visualizing Patient Pathways and Identifying Data Repositories in a UK Neurosciences Center: Exploratory Study %A Knight,Jo %A Chandrabalan,Vishnu Vardhan %A Emsley,Hedley C A %K health data %K business process monitoring notation %K neurology %K process monitoring %K patient pathway %K clinical pathway %K patient care %K EHR %K electronic health record %K dataset %K questionnaire %K patient data %K NHS %K National Health Service %D 2024 %7 24.12.2024 %9 %J JMIR Med Inform %G English %X Background: Health and clinical activity data are a vital resource for research, improving patient care and service efficiency. Health care data are inherently complex, and their acquisition, storage, retrieval, and subsequent analysis require a thorough understanding of the clinical pathways underpinning such data. Better use of health care data could lead to improvements in patient care and service delivery. However, this depends on the identification of relevant datasets. Objective: We aimed to demonstrate the application of business process modeling notation (BPMN) to represent clinical pathways at a UK neurosciences center and map the clinical activity to corresponding data flows into electronic health records and other nonstandard data repositories. Methods: We used BPMN to map and visualize a patient journey and the subsequent movement and storage of patient data. After identifying several datasets that were being held outside of the standard applications, we collected information about these datasets using a questionnaire. Results: We identified 13 standard applications where neurology clinical activity was captured as part of the patient’s electronic health record including applications and databases for managing referrals, outpatient activity, laboratory data, imaging data, and clinic letters. We also identified 22 distinct datasets not within standard applications that were created and managed within the neurosciences department, either by individuals or teams. These were being used to deliver direct patient care and included datasets for tracking patient blood results, recording home visits, and tracking triage status. Conclusions: Mapping patient data flows and repositories allowed us to identify areas wherein the current electronic health record does not fulfill the needs of day-to-day patient care. Data that are being stored outside of standard applications represent a potential duplication in the effort and risks being overlooked. Future work should identify unmet data needs to inform correct data capture and centralization within appropriate data architectures. %R 10.2196/60017 %U https://medinform.jmir.org/2024/1/e60017 %U https://doi.org/10.2196/60017 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e67056 %T Automated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study %A Kim,Sanghwan %A Jang,Sowon %A Kim,Borham %A Sunwoo,Leonard %A Kim,Seok %A Chung,Jin-Haeng %A Nam,Sejin %A Cho,Hyeongmin %A Lee,Donghyoung %A Lee,Keehyuck %A Yoo,Sooyoung %+ Office of eHealth Research and Business, Seoul National University Bundang Hospital, Healthcare Innovation Park, Seongnam, 13605, Republic of Korea, 82 317878980, yoosoo0@snubh.org %K AJCC Cancer Staging Manual 8th edition %K American Joint Committee on Cancer %K large language model %K chain-of-thought %K rationale %K lung cancer %K report analysis %K AI %K surgery %K pathology reports %K tertiary hospital %K generative language models %K efficiency %K accuracy %K automated %D 2024 %7 20.12.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. Objective: This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. Methods: We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. Results: Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. Conclusions: The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types. %M 39705675 %R 10.2196/67056 %U https://medinform.jmir.org/2024/1/e67056 %U https://doi.org/10.2196/67056 %U http://www.ncbi.nlm.nih.gov/pubmed/39705675 %0 Journal Article %@ 2368-7959 %I JMIR Publications %V 11 %N %P e59113 %T Implementing Findable, Accessible, Interoperable, Reusable (FAIR) Principles in Child and Adolescent Mental Health Research: Mixed Methods Approach %A de Groot,Rowdy %A van der Graaff,Frank %A van der Doelen,Daniël %A Luijten,Michiel %A De Meyer,Ronald %A Alrouh,Hekmat %A van Oers,Hedy %A Tieskens,Jacintha %A Zijlmans,Josjan %A Bartels,Meike %A Popma,Arne %A de Keizer,Nicolette %A Cornet,Ronald %A Polderman,Tinca J C %K FAIR data %K research data management %K data interoperability %K data standardization %K OMOP CDM %K implementation %K health data %K data quality %K FAIR principles %D 2024 %7 19.12.2024 %9 %J JMIR Ment Health %G English %X Background: The FAIR (Findable, Accessible, Interoperable, Reusable) data principles are a guideline to improve the reusability of data. However, properly implementing these principles is challenging due to a wide range of barriers. Objectives: To further the field of FAIR data, this study aimed to systematically identify barriers regarding implementing the FAIR principles in the area of child and adolescent mental health research, define the most challenging barriers, and provide recommendations for these barriers. Methods: Three sources were used as input to identify barriers: (1) evaluation of the implementation process of the Observational Medical Outcomes Partnership Common Data Model by 3 data managers; (2) interviews with experts on mental health research, reusable health data, and data quality; and (3) a rapid literature review. All barriers were categorized according to type as described previously, the affected FAIR principle, a category to add detail about the origin of the barrier, and whether a barrier was mental health specific. The barriers were assessed and ranked on impact with the data managers using the Delphi method. Results: Thirteen barriers were identified by the data managers, 7 were identified by the experts, and 30 barriers were extracted from the literature. This resulted in 45 unique barriers. The characteristics that were most assigned to the barriers were, respectively, external type (n=32/45; eg, organizational policy preventing the use of required software), tooling category (n=19/45; ie, software and databases), all FAIR principles (n=15/45), and not mental health specific (n=43/45). Consensus on ranking the scores of the barriers was reached after 2 rounds of the Delphi method. The most important recommendations to overcome the barriers are adding a FAIR data steward to the research team, accessible step-by-step guides, and ensuring sustainable funding for the implementation and long-term use of FAIR data. Conclusions: By systematically listing these barriers and providing recommendations, we intend to enhance the awareness of researchers and grant providers that making data FAIR demands specific expertise, available tooling, and proper investments. %R 10.2196/59113 %U https://mental.jmir.org/2024/1/e59113 %U https://doi.org/10.2196/59113 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e60665 %T An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontology-Enhanced Large Language Models: Development Study %A Cao,Lang %A Sun,Jimeng %A Cross,Adam %K rare disease %K clinical informatics %K LLM %K natural language processing %K machine learning %K artificial intelligence %K large language models %K data extraction %K ontologies %K knowledge graphs %K text mining %D 2024 %7 18.12.2024 %9 %J JMIR Med Inform %G English %X Background: Rare diseases affect millions worldwide but sometimes face limited research focus individually due to low prevalence. Many rare diseases do not have specific International Classification of Diseases, Ninth Edition (ICD-9) and Tenth Edition (ICD-10), codes and therefore cannot be reliably extracted from granular fields like “Diagnosis” and “Problem List” entries, which complicates tasks that require identification of patients with these conditions, including clinical trial recruitment and research efforts. Recent advancements in large language models (LLMs) have shown promise in automating the extraction of medical information, offering the potential to improve medical research, diagnosis, and management. However, most LLMs lack professional medical knowledge, especially concerning specific rare diseases, and cannot effectively manage rare disease data in its various ontological forms, making it unsuitable for these tasks. Objective: Our aim is to create an end-to-end system called automated rare disease mining (AutoRD), which automates the extraction of rare disease–related information from medical text, focusing on entities and their relations to other medical concepts, such as signs and symptoms. AutoRD integrates up-to-date ontologies with other structured knowledge and demonstrates superior performance in rare disease extraction tasks. We conducted various experiments to evaluate AutoRD’s performance, aiming to surpass common LLMs and traditional methods. Methods: AutoRD is a pipeline system that involves data preprocessing, entity extraction, relation extraction, entity calibration, and knowledge graph construction. We implemented this system using GPT-4 and medical knowledge graphs developed from the open-source Human Phenotype and Orphanet ontologies, using techniques such as chain-of-thought reasoning and prompt engineering. We quantitatively evaluated our system’s performance in entity extraction, relation extraction, and knowledge graph construction. The experiment used the well-curated dataset RareDis2023, which contains medical literature focused on rare disease entities and their relations, making it an ideal dataset for training and testing our methodology. Results: On the RareDis2023 dataset, AutoRD achieved an overall entity extraction F1-score of 56.1% and a relation extraction F1-score of 38.6%, marking a 14.4% improvement over the baseline LLM. Notably, the F1-score for rare disease entity extraction reached 83.5%, indicating high precision and recall in identifying rare disease mentions. These results demonstrate the effectiveness of integrating LLMs with medical ontologies in extracting complex rare disease information. Conclusions: AutoRD is an automated end-to-end system for extracting rare disease information from text to build knowledge graphs, addressing critical limitations of existing LLMs by improving identification of these diseases and connecting them to related clinical features. This work underscores the significant potential of LLMs in transforming health care, particularly in the rare disease domain. By leveraging ontology-enhanced LLMs, AutoRD constructs a robust medical knowledge base that incorporates up-to-date rare disease information, facilitating improved identification of patients and resulting in more inclusive research and trial candidacy efforts. %R 10.2196/60665 %U https://medinform.jmir.org/2024/1/e60665 %U https://doi.org/10.2196/60665 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e63710 %T Enhancing Standardized and Structured Recording by Elderly Care Physicians for Reusing Electronic Health Record Data: Interview Study %A Albers,Charlotte A W %A Wieland-Jorna,Yvonne %A de Bruijne,Martine C %A Smalbrugge,Martin %A Joling,Karlijn J %A de Boer,Marike E %+ Department of Medicine for Older People, Location Vrije Universiteit Amsterdam, Amsterdam UMC, De Boelelaan 1117, Amsterdam, 1081 HV, Netherlands, 31 615867548, c.a.w.albers@amsterdamumc.nl %K electronic health records %K health information interoperability %K health information exchange %K reference standards %K long-term care %K nursing homes %K medical records %K attitude of health personnel %K qualitative research %K digital health %D 2024 %7 13.12.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Elderly care physicians (ECPs) in nursing homes document patients’ health, medical conditions, and the care provided in electronic health records (EHRs). However, much of these health data currently lack structure and standardization, limiting their potential for health information exchange across care providers and reuse for quality improvement, policy development, and scientific research. Enhancing this potential requires insight into the attitudes and behaviors of ECPs toward standardized and structured recording in EHRs. Objective: This study aims to answer why and how ECPs record their findings in EHRs and what factors influence them to record in a standardized and structured manner. The findings will be used to formulate recommendations aimed at enhancing standardized and structured data recording for the reuse of EHR data. Methods: Semistructured interviews were conducted with 13 ECPs working in Dutch nursing homes. We recruited participants through purposive sampling, aiming for diversity in age, gender, health care organization, and use of EHR systems. Interviews continued until we reached data saturation. Analysis was performed using inductive thematic analysis. Results: ECPs primarily use EHRs to document daily patient care, ensure continuity of care, and fulfill their obligation to record specific information for accountability purposes. The EHR serves as a record to justify their actions in the event of a complaint. In addition, some respondents also mentioned recording information for secondary purposes, such as research and quality improvement. Several factors were found to influence standardized and structured recording. At a personal level, it is crucial to experience the added value of standardized and structured recording. At the organizational level, clear internal guidelines and a focus on their implementation can have a substantial impact. At the level of the EHR system, user-friendliness, interoperability, and guidance were most frequently mentioned as being important. At a national level, the alignment of internal guidelines with overarching standards plays a pivotal role in encouraging standardized and structured recording. Conclusions: The results of our study are similar to the findings of previous research in hospital care and general practice. Therefore, long-term care can learn from solutions regarding standardized and structured recording in other health care sectors. The main motives for ECPs to record in EHRs are the daily patient care and ensuring continuity of care. Standardized and structured recording can be improved by aligning the recording method in EHRs with the primary care process. In addition, there are incentives for motivating ECPs to record in a standardized and structured way, mainly at the personal, organizational, EHR system, and national levels. %M 39671580 %R 10.2196/63710 %U https://medinform.jmir.org/2024/1/e63710 %U https://doi.org/10.2196/63710 %U http://www.ncbi.nlm.nih.gov/pubmed/39671580 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e63445 %T Using Large Language Models to Abstract Complex Social Determinants of Health From Original and Deidentified Medical Notes: Development and Validation Study %A Ralevski,Alexandra %A Taiyab,Nadaa %A Nossal,Michael %A Mico,Lindsay %A Piekos,Samantha %A Hadlock,Jennifer %+ Institute for Systems Biology, 401 Terry Ave N, Seattle, WA, 98121, United States, 1 732 1359, jhadlock@isbscience.org %K housing instability %K housing insecurity %K housing %K machine learning %K artificial intelligence %K AI %K large language model %K LLM %K natural language processing %K NLP %K electronic health record %K EHR %K electronic medical record %K EMR %K social determinants of health %K exposome %K pregnancy %K obstetric %K deidentification %D 2024 %7 19.11.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Social determinants of health (SDoH) such as housing insecurity are known to be intricately linked to patients’ health status. More efficient methods for abstracting structured data on SDoH can help accelerate the inclusion of exposome variables in biomedical research and support health care systems in identifying patients who could benefit from proactive outreach. Large language models (LLMs) developed from Generative Pre-trained Transformers (GPTs) have shown potential for performing complex abstraction tasks on unstructured clinical notes. Objective: Here, we assess the performance of GPTs on identifying temporal aspects of housing insecurity and compare results between both original and deidentified notes. Methods: We compared the ability of GPT-3.5 and GPT-4 to identify instances of both current and past housing instability, as well as general housing status, from 25,217 notes from 795 pregnant women. Results were compared with manual abstraction, a named entity recognition model, and regular expressions. Results: Compared with GPT-3.5 and the named entity recognition model, GPT-4 had the highest performance and had a much higher recall (0.924) than human abstractors (0.702) in identifying patients experiencing current or past housing instability, although precision was lower (0.850) compared with human abstractors (0.971). GPT-4’s precision improved slightly (0.936 original, 0.939 deidentified) on deidentified versions of the same notes, while recall dropped (0.781 original, 0.704 deidentified). Conclusions: This work demonstrates that while manual abstraction is likely to yield slightly more accurate results overall, LLMs can provide a scalable, cost-effective solution with the advantage of greater recall. This could support semiautomated abstraction, but given the potential risk for harm, human review would be essential before using results for any patient engagement or care decisions. Furthermore, recall was lower when notes were deidentified prior to LLM abstraction. %M 39561354 %R 10.2196/63445 %U https://www.jmir.org/2024/1/e63445 %U https://doi.org/10.2196/63445 %U http://www.ncbi.nlm.nih.gov/pubmed/39561354 %0 Journal Article %@ 1929-073X %I JMIR Publications %V 13 %N %P e55865 %T Visual Modeling Languages in Patient Pathways: Scoping Review %A Bogale,Binyam %A Vesinurm,Märt %A Lillrank,Paul %A Celius,Elisabeth Gulowsen %A Halvorsrud,Ragnhild %+ Department of Neurology, Institute of Clinical Medicine, University of Oslo, Bldg 20, Ludvig Kastensvei 15, Oslo, 1064, Norway, 47 96711061, binyadad@gmail.com %K patient pathways %K visual modeling languages %K business process model and notation %K BPMN %K unified modeling language %K UML %K domain-specific modeling languages %K scoping review %D 2024 %7 15.11.2024 %9 Review %J Interact J Med Res %G English %X Background: Patient pathways (PPs) are presented as a panacea solution to enhance health system functions. It is a complex concept that needs to be described and communicated well. Modeling plays a crucial role in promoting communication, fostering a shared understanding, and streamlining processes. Only a few existing systematic reviews have focused on modeling methods and standardized modeling languages. There remains a gap in consolidated knowledge regarding the use of diverse visual modeling languages. Objective: This scoping review aimed to compile visual modeling languages used to represent PPs, including the justifications and the context in which a modeling language was adopted, adapted, combined, or developed. Methods: After initial experimentation with the keywords used to describe the concepts of PPs and visual modeling languages, we developed a search strategy that was further refined and customized to the major databases identified as topically relevant. In addition, we consulted gray literature and conducted hand searches of the referenced articles. Two reviewers independently screened the articles in 2 stages using preset inclusion criteria, and a third reviewer voted on the discordance. Data charting was done using an iteratively developed form in the Covidence software. Descriptive and thematic summaries were presented following rounds of discussion to produce the final report. Results: Of 1838 articles retrieved after deduplication, 22 satisfied our inclusion criteria. Clinical pathway is the most used phrase to represent the PP concept, and most papers discussed the concept without providing their operational definition. We categorized the visual modeling languages into five categories: (1) general purpose–modeling language (GPML) adopted without major extension or modification, (2) GPML used with formal extension recommendations, (3) combination of 2 or more modeling languages, (4) a developed domain-specific modeling language (DSML), and (5) ontological modeling languages. The justifications for adopting, adapting, combining, and developing visual modeling languages varied accordingly and ranged from versatility, expressiveness, tool support, and extensibility of a language to domain needs, integration, and simplification. Conclusions: Various visual modeling languages were used in PP modeling, each with varying levels of abstraction and granularity. The categorization we made could aid in a better understanding of the complex combination of PP and modeling languages. Standardized GPMLs were used with or without any modifications. The rationale to propose any modification to GPMLs evolved as more evidence was presented following requirement analyses to support domain constructs. DSMLs are infrequently used due to their resource-intensive development, often initiated at a project level. The justifications provided and the context where DSMLs were created are paramount. Future studies should assess the merits and demerits of using a visual modeling language to facilitate PP communications among stakeholders and use evaluation frameworks to identify, modify, or develop them, depending on the scope and goal of the modeling need. %M 39546800 %R 10.2196/55865 %U https://www.i-jmr.org/2024/1/e55865 %U https://doi.org/10.2196/55865 %U http://www.ncbi.nlm.nih.gov/pubmed/39546800 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e56681 %T The Impact of International Classification of Disease–Triggered Prescription Support on Telemedicine: Observational Analysis of Efficiency and Guideline Adherence %A Accorsi,Tarso Augusto Duenhas %A Eduardo,Anderson Aires %A Baptista,Carlos Guilherme %A Moreira,Flavio Tocci %A Morbeck,Renata Albaladejo %A Köhler,Karen Francine %A Lima,Karine de Amicis %A Pedrotti,Carlos Henrique Sartorato %+ Telemedicine Department, Hospital Israelita Albert Einstein, Avenue Albert Einstein, 627 - Bloco B, 2º andar, Secretaria da Unidade de Telemedicina, São Paulo, 05652-900, Brazil, 55 1121515420, taccorsi@einstein.br %K telemedicine %K clinical decision support systems %K electronic prescriptions %K guideline adherence %K consultation efficiency %K International Classification of Disease–coded prescriptions %K teleheath %K eHealth %D 2024 %7 25.10.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Integrating decision support systems into telemedicine may optimize consultation efficiency and adherence to clinical guidelines; however, the extent of such effects remains underexplored. Objective: This study aims to evaluate the use of ICD (International Classification of Disease)-coded prescription decision support systems (PDSSs) and the effects of these systems on consultation duration and guideline adherence during telemedicine encounters. Methods: In this retrospective, single-center, observational study conducted from October 2021 to March 2022, adult patients who sought urgent digital care via direct-to-consumer video consultations were included. Physicians had access to current guidelines and could use an ICD-triggered PDSS (which was introduced in January 2022 after a preliminary test in the preceding month) for 26 guideline-based conditions. This study analyzed the impact of implementing automated prescription systems and compared these systems to manual prescription processes in terms of consultation duration and guideline adherence. Results: This study included 10,485 telemedicine encounters involving 9644 patients, with 12,346 prescriptions issued by 290 physicians. Automated prescriptions were used in 5022 (40.67%) of the consultations following system integration. Before introducing decision support, 4497 (36.42%) prescriptions were issued, which increased to 7849 (63.57%) postimplementation. The physician’s average consultation time decreased significantly to 9.5 (SD 5.5) minutes from 11.2 (SD 5.9) minutes after PDSS implementation (P<.001). Of the 12,346 prescriptions, 8683 (70.34%) were aligned with disease-specific international guidelines tailored for telemedicine encounters. Primary medication adherence in accordance with existing guidelines was significantly greater in the decision support group than in the manual group (n=4697, 93.53% vs n=1389, 49.14%; P<.001). Conclusions: Most of the physicians adopted the PDSS, and the results demonstrated the use of the ICD-code system in reducing consultation times and increasing guideline adherence. These systems appear to be valuable for enhancing the efficiency and quality of telemedicine consultations by supporting evidence-based clinical decision-making. %M 39453703 %R 10.2196/56681 %U https://medinform.jmir.org/2024/1/e56681 %U https://doi.org/10.2196/56681 %U http://www.ncbi.nlm.nih.gov/pubmed/39453703 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e58541 %T Bridging Data Models in Health Care With a Novel Intermediate Query Format for Feasibility Queries: Mixed Methods Study %A Rosenau,Lorenz %A Gruendner,Julian %A Kiel,Alexander %A Köhler,Thomas %A Schaffer,Bastian %A Majeed,Raphael W %K feasibility %K FHIR %K CQL %K eligibility criteria %K clinical research %K intermediate query format %K healthcare interoperability %K cohort definition %K query %K queries %K interoperability %K interoperable %K informatics %K portal %K portals %K implementation %K develop %K development %K ontology %K ontologies %K JSON %D 2024 %7 14.10.2024 %9 %J JMIR Med Inform %G English %X Background: To advance research with clinical data, it is essential to make access to the available data as fast and easy as possible for researchers, which is especially challenging for data from different source systems within and across institutions. Over the years, many research repositories and data standards have been created. One of these is the Fast Healthcare Interoperability Resources (FHIR) standard, used by the German Medical Informatics Initiative (MII) to harmonize and standardize data across university hospitals in Germany. One of the first steps to make these data available is to allow researchers to create feasibility queries to determine the data availability for a specific research question. Given the heterogeneity of different query languages to access different data across and even within standards such as FHIR (eg, CQL and FHIR Search), creating an intermediate query syntax for feasibility queries reduces the complexity of query translation and improves interoperability across different research repositories and query languages. Objective: This study describes the creation and implementation of an intermediate query syntax for feasibility queries and how it integrates into the federated German health research portal (Forschungsdatenportal Gesundheit) and the MII. Methods: We analyzed the requirements for feasibility queries and the feasibility tools that are currently available in research repositories. Based on this analysis, we developed an intermediate query syntax that can be easily translated into different research repository–specific query languages. Results: The resulting Clinical Cohort Definition Language (CCDL) for feasibility queries combines inclusion criteria in a conjunctive normal form and exclusion criteria in a disjunctive normal form, allowing for additional filters like time or numerical restrictions. The inclusion and exclusion results are combined via an expression to specify feasibility queries. We defined a JSON schema for the CCDL, generated an ontology, and demonstrated the use and translatability of the CCDL across multiple studies and real-world use cases. Conclusions: We developed and evaluated a structured query syntax for feasibility queries and demonstrated its use in a real-world example as part of a research platform across 39 German university hospitals. %R 10.2196/58541 %U https://medinform.jmir.org/2024/1/e58541 %U https://doi.org/10.2196/58541 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e56735 %T Enhancing the Functionalities of Personal Health Record Systems: Empirical Study Based on the HL7 Personal Health Record System Functional Model Release 1 %A Cao,Teng %A Chen,Zhi %A Nakayama,Masaharu %K fast healthcare interoperability resources %K logical observation identifiers names and codes %K personal health record system functional model %K personal health records %D 2024 %7 9.10.2024 %9 %J JMIR Med Inform %G English %X Background: The increasing demand for personal health record (PHR) systems is driven by individuals’ desire to actively manage their health care. However, the limited functionality of current PHR systems has affected users’ willingness to adopt them, leading to lower-than-expected usage rates. The HL7 (Health Level Seven) PHR System Functional Model (PHR-S FM) was proposed to address this issue, outlining all possible functionalities in PHR systems. Although the PHR-S FM provides a comprehensive theoretical framework, its practical effectiveness and applicability have not been fully explored. Objective: This study aimed to design and develop a tethered PHR prototype in accordance with the guidelines of the PHR-S FM. It sought to explore the feasibility of applying the PHR-S FM in PHR systems by comparing the prototype with the results of previous research. Methods: The PHR-S FM profile was defined to meet broad clinical data management requirements based on previous research. We designed and developed a PHR prototype as a web application using the Fast Healthcare Interoperability Resources R4 (FHIR) and Logical Observation Identifiers Names and Codes (LOINC) coding system for interoperability and data consistency. We validated the prototype using the Synthea dataset, which provided realistic synthetic medical records. In addition, we compared the results produced by the prototype with those of previous studies to evaluate the feasibility and implementation of the PHR-S FM framework. Results: The PHR prototype was developed based on the PHR-S FM profile. We verified its functionality by demonstrating its ability to synchronize data with the FHIR server, effectively managing and displaying various health data types. Validation using the Synthea dataset confirmed the prototype’s accuracy, achieving 100% coverage across 1157 data items. A comparison with the findings of previous studies indicated the feasibility of implementing the PHR-S FM and highlighted areas for future research and improvements. Conclusions: The results of this study offer valuable insights into the potential for practical application and broad adoption of the PHR-S FM in real-world health care settings. %R 10.2196/56735 %U https://medinform.jmir.org/2024/1/e56735 %U https://doi.org/10.2196/56735 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e62924 %T Use of SNOMED CT in Large Language Models: Scoping Review %A Chang,Eunsuk %A Sung,Sumi %+ Department of Nursing Science, Research Institute of Nursing Science, Chungbuk National University, 1 Chungdae-ro, Seowon-gu, Cheongju, 28644, Republic of Korea, 82 43 249 1731, sumisung@cbnu.ac.kr %K SNOMED CT %K ontology %K knowledge graph %K large language models %K natural language processing %K language models %D 2024 %7 7.10.2024 %9 Review %J JMIR Med Inform %G English %X Background: Large language models (LLMs) have substantially advanced natural language processing (NLP) capabilities but often struggle with knowledge-driven tasks in specialized domains such as biomedicine. Integrating biomedical knowledge sources such as SNOMED CT into LLMs may enhance their performance on biomedical tasks. However, the methodologies and effectiveness of incorporating SNOMED CT into LLMs have not been systematically reviewed. Objective: This scoping review aims to examine how SNOMED CT is integrated into LLMs, focusing on (1) the types and components of LLMs being integrated with SNOMED CT, (2) which contents of SNOMED CT are being integrated, and (3) whether this integration improves LLM performance on NLP tasks. Methods: Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, we searched ACM Digital Library, ACL Anthology, IEEE Xplore, PubMed, and Embase for relevant studies published from 2018 to 2023. Studies were included if they incorporated SNOMED CT into LLM pipelines for natural language understanding or generation tasks. Data on LLM types, SNOMED CT integration methods, end tasks, and performance metrics were extracted and synthesized. Results: The review included 37 studies. Bidirectional Encoder Representations from Transformers and its biomedical variants were the most commonly used LLMs. Three main approaches for integrating SNOMED CT were identified: (1) incorporating SNOMED CT into LLM inputs (28/37, 76%), primarily using concept descriptions to expand training corpora; (2) integrating SNOMED CT into additional fusion modules (5/37, 14%); and (3) using SNOMED CT as an external knowledge retriever during inference (5/37, 14%). The most frequent end task was medical concept normalization (15/37, 41%), followed by entity extraction or typing and classification. While most studies (17/19, 89%) reported performance improvements after SNOMED CT integration, only a small fraction (19/37, 51%) provided direct comparisons. The reported gains varied widely across different metrics and tasks, ranging from 0.87% to 131.66%. However, some studies showed either no improvement or a decline in certain performance metrics. Conclusions: This review demonstrates diverse approaches for integrating SNOMED CT into LLMs, with a focus on using concept descriptions to enhance biomedical language understanding and generation. While the results suggest potential benefits of SNOMED CT integration, the lack of standardized evaluation methods and comprehensive performance reporting hinders definitive conclusions about its effectiveness. Future research should prioritize consistent reporting of performance comparisons and explore more sophisticated methods for incorporating SNOMED CT’s relational structure into LLMs. In addition, the biomedical NLP community should develop standardized evaluation frameworks to better assess the impact of ontology integration on LLM performance. %M 39374057 %R 10.2196/62924 %U https://medinform.jmir.org/2024/1/e62924 %U https://doi.org/10.2196/62924 %U http://www.ncbi.nlm.nih.gov/pubmed/39374057 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e60293 %T Toward Better Semantic Interoperability of Data Element Repositories in Medicine: Analysis Study %A Hu,Zhengyong %A Wang,Anran %A Duan,Yifan %A Zhou,Jiayin %A Hu,Wanfei %A Wu,Sizhu %+ Institute of Medical Information/Medical Library, Chinese Academy of Medical Sciences & Peking Union Medical College, No. 3 Yabao Road, Chaoyang District, Beijing, 100020, China, 86 52328760, Wu.sizhu@imicams.ac.cn %K data element repository %K FAIR %K ISO/IEC 11179 %K metadata %K semantic interoperability %D 2024 %7 30.9.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Data element repositories facilitate high-quality medical data sharing by standardizing data and enhancing semantic interoperability. However, the application of repositories is confined to specific projects and institutions. Objective: This study aims to explore potential issues and promote broader application of data element repositories within the medical field by evaluating and analyzing typical repositories. Methods: Following the inclusion of 5 data element repositories through a literature review, a novel analysis framework consisting of 7 dimensions and 36 secondary indicators was constructed and used for evaluation and analysis. Results: The study’s results delineate the unique characteristics of different repositories and uncover specific issues in their construction. These issues include the absence of data reuse protocols and insufficient information regarding the application scenarios and efficacy of data elements. The repositories fully comply with only 45% (9/20) of the subprinciples for Findable and Reusable in the FAIR principle, while achieving a 90% (19/20 subprinciples) compliance rate for Accessible and 67% (10/15 subprinciples) for Interoperable. Conclusions: The recommendations proposed in this study address the issues to improve the construction and application of repositories, offering valuable insights to data managers, computer experts, and other pertinent stakeholders. %M 39348178 %R 10.2196/60293 %U https://medinform.jmir.org/2024/1/e60293 %U https://doi.org/10.2196/60293 %U http://www.ncbi.nlm.nih.gov/pubmed/39348178 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 8 %N %P e53711 %T An Ontology to Bridge the Clinical Management of Patients and Public Health Responses for Strengthening Infectious Disease Surveillance: Design Science Study %A Lim,Sachiko %A Johannesson,Paul %+ Department of Computer and Systems Sciences, Stockholm University, Nodhuset, Borgarfjordsgatan 12, Kista, SE-164 07, Sweden, 46 0760968462, sachiko@dsv.su.se %K infectious disease %K ontology %K IoT %K infectious disease surveillance %K patient monitoring %K infectious disease management %K risk analysis %K early warning %K data integration %K semantic interoperability %K public health %D 2024 %7 26.9.2024 %9 Original Paper %J JMIR Form Res %G English %X Background: Novel surveillance approaches using digital technologies, including the Internet of Things (IoT), have evolved, enhancing traditional infectious disease surveillance systems by enabling real-time detection of outbreaks and reaching a wider population. However, disparate, heterogenous infectious disease surveillance systems often operate in silos due to a lack of interoperability. As a life-changing clinical use case, the COVID-19 pandemic has manifested that a lack of interoperability can severely inhibit public health responses to emerging infectious diseases. Interoperability is thus critical for building a robust ecosystem of infectious disease surveillance and enhancing preparedness for future outbreaks. The primary enabler for semantic interoperability is ontology. Objective: This study aims to design the IoT-based management of infectious disease ontology (IoT-MIDO) to enhance data sharing and integration of data collected from IoT-driven patient health monitoring, clinical management of individual patients, and disparate heterogeneous infectious disease surveillance. Methods: The ontology modeling approach was chosen for its semantic richness in knowledge representation, flexibility, ease of extensibility, and capability for knowledge inference and reasoning. The IoT-MIDO was developed using the basic formal ontology (BFO) as the top-level ontology. We reused the classes from existing BFO-based ontologies as much as possible to maximize the interoperability with other BFO-based ontologies and databases that rely on them. We formulated the competency questions as requirements for the ontology to achieve the intended goals. Results: We designed an ontology to integrate data from heterogeneous sources, including IoT-driven patient monitoring, clinical management of individual patients, and infectious disease surveillance systems. This integration aims to facilitate the collaboration between clinical care and public health domains. We also demonstrate five use cases using the simplified ontological models to show the potential applications of IoT-MIDO: (1) IoT-driven patient monitoring, risk assessment, early warning, and risk management; (2) clinical management of patients with infectious diseases; (3) epidemic risk analysis for timely response at the public health level; (4) infectious disease surveillance; and (5) transforming patient information into surveillance information. Conclusions: The development of the IoT-MIDO was driven by competency questions. Being able to answer all the formulated competency questions, we successfully demonstrated that our ontology has the potential to facilitate data sharing and integration for orchestrating IoT-driven patient health monitoring in the context of an infectious disease epidemic, clinical patient management, infectious disease surveillance, and epidemic risk analysis. The novelty and uniqueness of the ontology lie in building a bridge to link IoT-based individual patient monitoring and early warning based on patient risk assessment to infectious disease epidemic surveillance at the public health level. The ontology can also serve as a starting point to enable potential decision support systems, providing actionable insights to support public health organizations and practitioners in making informed decisions in a timely manner. %M 39325530 %R 10.2196/53711 %U https://formative.jmir.org/2024/1/e53711 %U https://doi.org/10.2196/53711 %U http://www.ncbi.nlm.nih.gov/pubmed/39325530 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e57853 %T PCEtoFHIR: Decomposition of Postcoordinated SNOMED CT Expressions for Storage as HL7 FHIR Resources %A Ohlsen,Tessa %A Ingenerf,Josef %A Essenwanger,Andrea %A Drenkhahn,Cora %+ IT Center for Clinical Research, University of Luebeck, Ratzeburger Allee 160, Luebeck, 23562, Germany, 49 45131015623, t.ohlsen@uni-luebeck.de %K SNOMED CT %K HL7 FHIR %K TermInfo %K postcoordination %K semantic interoperability %K terminology %K OWL %K semantic similarity %D 2024 %7 17.9.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: To ensure interoperability, both structural and semantic standards must be followed. For exchanging medical data between information systems, the structural standard FHIR (Fast Healthcare Interoperability Resources) has recently gained popularity. Regarding semantic interoperability, the reference terminology SNOMED Clinical Terms (SNOMED CT), as a semantic standard, allows for postcoordination, offering advantages over many other vocabularies. These postcoordinated expressions (PCEs) make SNOMED CT an expressive and flexible interlingua, allowing for precise coding of medical facts. However, this comes at the cost of increased complexity, as well as challenges in storage and processing. Additionally, the boundary between semantic (terminology) and structural (information model) standards becomes blurred, leading to what is known as the TermInfo problem. Although often viewed critically, the TermInfo overlap can also be explored for its potential benefits, such as enabling flexible transformation of parts of PCEs. Objective: In this paper, an alternative solution for storing PCEs is presented, which involves combining them with the FHIR data model. Ultimately, all components of a PCE should be expressible solely through precoordinated concepts that are linked to the appropriate elements of the information model. Methods: The approach involves storing PCEs decomposed into their components in alignment with FHIR resources. By utilizing the Web Ontology Language (OWL) to generate an OWL ClassExpression, and combining it with an external reasoner and semantic similarity measures, a precoordinated SNOMED CT concept that most accurately describes the PCE is identified as a Superconcept. In addition, the nonmatching attribute relationships between the Superconcept and the PCE are identified as the “Delta.” Once SNOMED CT attributes are manually mapped to FHIR elements, FHIRPath expressions can be defined for both the Superconcept and the Delta, allowing the identified precoordinated codes to be stored within FHIR resources. Results: A web application called PCEtoFHIR was developed to implement this approach. In a validation process with 600 randomly selected precoordinated concepts, the formal correctness of the generated OWL ClassExpressions was verified. Additionally, 33 PCEs were used for two separate validation tests. Based on these validations, it was demonstrated that a previously proposed semantic similarity calculation is suitable for determining the Superconcept. Additionally, the 33 PCEs were used to confirm the correct functioning of the entire approach. Furthermore, the FHIR StructureMaps were reviewed and deemed meaningful by FHIR experts. Conclusions: PCEtoFHIR offers services to decompose PCEs for storage within FHIR resources. When creating structure mappings for specific subdomains of SNOMED CT concepts (eg, allergies) to desired FHIR profiles, the use of SNOMED CT Expression Templates has proven highly effective. Domain experts can create templates with appropriate mappings, which can then be easily reused in a constrained manner by end users. %M 39287966 %R 10.2196/57853 %U https://medinform.jmir.org/2024/1/e57853 %U https://doi.org/10.2196/57853 %U http://www.ncbi.nlm.nih.gov/pubmed/39287966 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e57949 %T Natural Language Processing Versus Diagnosis Code–Based Methods for Postherpetic Neuralgia Identification: Algorithm Development and Validation %A Zheng,Chengyi %A Ackerson,Bradley %A Qiu,Sijia %A Sy,Lina S %A Daily,Leticia I Vega %A Song,Jeannie %A Qian,Lei %A Luo,Yi %A Ku,Jennifer H %A Cheng,Yanjun %A Wu,Jun %A Tseng,Hung Fu %K postherpetic neuralgia %K herpes zoster %K natural language processing %K electronic health record %K real-world data %K artificial intelligence %K development %K validation %K diagnosis %K EHR %K algorithm %K EHR data %K sensitivity %K specificity %K validation data %K neuralgia %K recombinant zoster vaccine %D 2024 %7 10.9.2024 %9 %J JMIR Med Inform %G English %X Background: Diagnosis codes and prescription data are used in algorithms to identify postherpetic neuralgia (PHN), a debilitating complication of herpes zoster (HZ). Because of the questionable accuracy of codes and prescription data, manual chart review is sometimes used to identify PHN in electronic health records (EHRs), which can be costly and time-consuming. Objective: This study aims to develop and validate a natural language processing (NLP) algorithm for automatically identifying PHN from unstructured EHR data and to compare its performance with that of code-based methods. Methods: This retrospective study used EHR data from Kaiser Permanente Southern California, a large integrated health care system that serves over 4.8 million members. The source population included members aged ≥50 years who received an incident HZ diagnosis and accompanying antiviral prescription between 2018 and 2020 and had ≥1 encounter within 90‐180 days of the incident HZ diagnosis. The study team manually reviewed the EHR and identified PHN cases. For NLP development and validation, 500 and 800 random samples from the source population were selected, respectively. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F-score, and Matthews correlation coefficient (MCC) of NLP and the code-based methods were evaluated using chart-reviewed results as the reference standard. Results: The NLP algorithm identified PHN cases with a 90.9% sensitivity, 98.5% specificity, 82% PPV, and 99.3% NPV. The composite scores of the NLP algorithm were 0.89 (F-score) and 0.85 (MCC). The prevalences of PHN in the validation data were 6.9% (reference standard), 7.6% (NLP), and 5.4%‐13.1% (code-based). The code-based methods achieved a 52.7%‐61.8% sensitivity, 89.8%‐98.4% specificity, 27.6%‐72.1% PPV, and 96.3%‐97.1% NPV. The F-scores and MCCs ranged between 0.45 and 0.59 and between 0.32 and 0.61, respectively. Conclusions: The automated NLP-based approach identified PHN cases from the EHR with good accuracy. This method could be useful in population-based PHN research. %R 10.2196/57949 %U https://medinform.jmir.org/2024/1/e57949 %U https://doi.org/10.2196/57949 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e52506 %T Impact of Gold-Standard Label Errors on Evaluating Performance of Deep Learning Models in Diabetic Retinopathy Screening: Nationwide Real-World Validation Study %A Wang,Yueye %A Han,Xiaotong %A Li,Cong %A Luo,Lixia %A Yin,Qiuxia %A Zhang,Jian %A Peng,Guankai %A Shi,Danli %A He,Mingguang %+ School of Optometry, The Hong Kong Polytechnic University, Hung Hom, Kowloon, China (Hong Kong), 852 34002795, mingguang.he@polyu.edu.hk %K artificial intelligence %K diabetic retinopathy %K diabetes %K real world %K deep learning %D 2024 %7 14.8.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: For medical artificial intelligence (AI) training and validation, human expert labels are considered the gold standard that represents the correct answers or desired outputs for a given data set. These labels serve as a reference or benchmark against which the model’s predictions are compared. Objective: This study aimed to assess the accuracy of a custom deep learning (DL) algorithm on classifying diabetic retinopathy (DR) and further demonstrate how label errors may contribute to this assessment in a nationwide DR-screening program. Methods: Fundus photographs from the Lifeline Express, a nationwide DR-screening program, were analyzed to identify the presence of referable DR using both (1) manual grading by National Health Service England–certificated graders and (2) a DL-based DR-screening algorithm with validated good lab performance. To assess the accuracy of labels, a random sample of images with disagreement between the DL algorithm and the labels was adjudicated by ophthalmologists who were masked to the previous grading results. The error rates of labels in this sample were then used to correct the number of negative and positive cases in the entire data set, serving as postcorrection labels. The DL algorithm’s performance was evaluated against both pre- and postcorrection labels. Results: The analysis included 736,083 images from 237,824 participants. The DL algorithm exhibited a gap between the real-world performance and the lab-reported performance in this nationwide data set, with a sensitivity increase of 12.5% (from 79.6% to 92.5%, P<.001) and a specificity increase of 6.9% (from 91.6% to 98.5%, P<.001). In the random sample, 63.6% (560/880) of negative images and 5.2% (140/2710) of positive images were misclassified in the precorrection human labels. High myopia was the primary reason for misclassifying non-DR images as referable DR images, while laser spots were predominantly responsible for misclassified referable cases. The estimated label error rate for the entire data set was 1.2%. The label correction was estimated to bring about a 12.5% enhancement in the estimated sensitivity of the DL algorithm (P<.001). Conclusions: Label errors based on human image grading, although in a small percentage, can significantly affect the performance evaluation of DL algorithms in real-world DR screening. %M 39141915 %R 10.2196/52506 %U https://www.jmir.org/2024/1/e52506 %U https://doi.org/10.2196/52506 %U http://www.ncbi.nlm.nih.gov/pubmed/39141915 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e55090 %T Research on Traditional Chinese Medicine: Domain Knowledge Graph Completion and Quality Evaluation %A Liu,Chang %A Li,Zhan %A Li,Jianmin %A Qu,Yiqian %A Chang,Ying %A Han,Qing %A Cao,Lingyong %A Lin,Shuyuan %+ School of Basic Medical Sciences, Zhejiang Chinese Medical University, 548 Binwen Road, Binjiang District, Hangzhou, 310053, China, 86 057186633015, lin_shuyuan@foxmail.com %K graph completion %K traditional Chinese medicine %K graph quality evaluation %K graph representation %K knowledge graph %D 2024 %7 2.8.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Knowledge graphs (KGs) can integrate domain knowledge into a traditional Chinese medicine (TCM) intelligent syndrome differentiation model. However, the quality of current KGs in the TCM domain varies greatly, related to the lack of knowledge graph completion (KGC) and evaluation methods. Objective: This study aims to investigate KGC and evaluation methods tailored for TCM domain knowledge. Methods: In the KGC phase, according to the characteristics of TCM domain knowledge, we proposed a 3-step “entity-ontology-path” completion approach. This approach uses path reasoning, ontology rule reasoning, and association rules. In the KGC quality evaluation phase, we proposed a 3-dimensional evaluation framework that encompasses completeness, accuracy, and usability, using quantitative metrics such as complex network analysis, ontology reasoning, and graph representation. Furthermore, we compared the impact of different graph representation models on KG usability. Results: In the KGC phase, 52, 107, 27, and 479 triples were added by outlier analysis, rule-based reasoning, association rules, and path-based reasoning, respectively. In addition, rule-based reasoning identified 14 contradictory triples. In the KGC quality evaluation phase, in terms of completeness, KG had higher density and lower sparsity after completion, and there were no contradictory rules within the KG. In terms of accuracy, KG after completion was more consistent with prior knowledge. In terms of usability, the mean reciprocal ranking, mean rank, and hit rate of the first N tail entities predicted by the model (Hits@N) of the TransE, RotatE, DistMult, and ComplEx graph representation models all showed improvement after KGC. Among them, the RotatE model achieved the best representation. Conclusions: The 3-step completion approach can effectively improve the completeness, accuracy, and availability of KGs, and the 3-dimensional evaluation framework can be used for comprehensive KGC evaluation. In the TCM field, the RotatE model performed better at KG representation. %M 39094109 %R 10.2196/55090 %U https://medinform.jmir.org/2024/1/e55090 %U https://doi.org/10.2196/55090 %U http://www.ncbi.nlm.nih.gov/pubmed/39094109 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e52896 %T Unsupervised Feature Selection to Identify Important ICD-10 and ATC Codes for Machine Learning on a Cohort of Patients With Coronary Heart Disease: Retrospective Study %A Ghasemi,Peyman %A Lee,Joon %K unsupervised feature selection %K ICD-10 %K International Classification of Diseases %K ATC %K Anatomical Therapeutic Chemical %K concrete autoencoder %K Laplacian score %K unsupervised feature selection for multicluster data %K autoencoder-inspired unsupervised feature selection %K principal feature analysis %K machine learning %K artificial intelligence %K case study %K coronary artery disease %K artery disease %K patient cohort %K artery %K mortality prediction %K mortality %K data set %K interpretability %K International Classification of Diseases, Tenth Revision %D 2024 %7 26.7.2024 %9 %J JMIR Med Inform %G English %X Background: The application of machine learning in health care often necessitates the use of hierarchical codes such as the International Classification of Diseases (ICD) and Anatomical Therapeutic Chemical (ATC) systems. These codes classify diseases and medications, respectively, thereby forming extensive data dimensions. Unsupervised feature selection tackles the “curse of dimensionality” and helps to improve the accuracy and performance of supervised learning models by reducing the number of irrelevant or redundant features and avoiding overfitting. Techniques for unsupervised feature selection, such as filter, wrapper, and embedded methods, are implemented to select the most important features with the most intrinsic information. However, they face challenges due to the sheer volume of ICD and ATC codes and the hierarchical structures of these systems. Objective: The objective of this study was to compare several unsupervised feature selection methods for ICD and ATC code databases of patients with coronary artery disease in different aspects of performance and complexity and select the best set of features representing these patients. Methods: We compared several unsupervised feature selection methods for 2 ICD and 1 ATC code databases of 51,506 patients with coronary artery disease in Alberta, Canada. Specifically, we used the Laplacian score, unsupervised feature selection for multicluster data, autoencoder-inspired unsupervised feature selection, principal feature analysis, and concrete autoencoders with and without ICD or ATC tree weight adjustment to select the 100 best features from over 9000 ICD and 2000 ATC codes. We assessed the selected features based on their ability to reconstruct the initial feature space and predict 90-day mortality following discharge. We also compared the complexity of the selected features by mean code level in the ICD or ATC tree and the interpretability of the features in the mortality prediction task using Shapley analysis. Results: In feature space reconstruction and mortality prediction, the concrete autoencoder–based methods outperformed other techniques. Particularly, a weight-adjusted concrete autoencoder variant demonstrated improved reconstruction accuracy and significant predictive performance enhancement, confirmed by DeLong and McNemar tests (P<.05). Concrete autoencoders preferred more general codes, and they consistently reconstructed all features accurately. Additionally, features selected by weight-adjusted concrete autoencoders yielded higher Shapley values in mortality prediction than most alternatives. Conclusions: This study scrutinized 5 feature selection methods in ICD and ATC code data sets in an unsupervised context. Our findings underscore the superiority of the concrete autoencoder method in selecting salient features that represent the entire data set, offering a potential asset for subsequent machine learning research. We also present a novel weight adjustment approach for the concrete autoencoders specifically tailored for ICD and ATC code data sets to enhance the generalizability and interpretability of the selected features. %R 10.2196/52896 %U https://medinform.jmir.org/2024/1/e52896 %U https://doi.org/10.2196/52896 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e49865 %T Introducing Attribute Association Graphs to Facilitate Medical Data Exploration: Development and Evaluation Using Epidemiological Study Data %A Bellmann,Louis %A Wiederhold,Alexander Johannes %A Trübe,Leona %A Twerenbold,Raphael %A Ückert,Frank %A Gottfried,Karl %+ Institute for Applied Medical Informatics, University Medical Center Hamburg-Eppendorf, Martinistr. 52, Hamburg, 20246, Germany, 49 15228842404, l.bellmann@uke.de %K data exploration %K cohort studies %K data visualization %K big data %K statistical models %K medical knowledge %K data analysis %K cardiovascular diseases %K usability %D 2024 %7 24.7.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Interpretability and intuitive visualization facilitate medical knowledge generation through big data. In addition, robustness to high-dimensional and missing data is a requirement for statistical approaches in the medical domain. A method tailored to the needs of physicians must meet all the abovementioned criteria. Objective: This study aims to develop an accessible tool for visual data exploration without the need for programming knowledge, adjusting complex parameterizations, or handling missing data. We sought to use statistical analysis using the setting of disease and control cohorts familiar to clinical researchers. We aimed to guide the user by identifying and highlighting data patterns associated with disease and reveal relations between attributes within the data set. Methods: We introduce the attribute association graph, a novel graph structure designed for visual data exploration using robust statistical metrics. The nodes capture frequencies of participant attributes in disease and control cohorts as well as deviations between groups. The edges represent conditional relations between attributes. The graph is visualized using the Neo4j (Neo4j, Inc) data platform and can be interactively explored without the need for technical knowledge. Nodes with high deviations between cohorts and edges of noticeable conditional relationship are highlighted to guide the user during the exploration. The graph is accompanied by a dashboard visualizing variable distributions. For evaluation, we applied the graph and dashboard to the Hamburg City Health Study data set, a large cohort study conducted in the city of Hamburg, Germany. All data structures can be accessed freely by researchers, physicians, and patients. In addition, we developed a user test conducted with physicians incorporating the System Usability Scale, individual questions, and user tasks. Results: We evaluated the attribute association graph and dashboard through an exemplary data analysis of participants with a general cardiovascular disease in the Hamburg City Health Study data set. All results extracted from the graph structure and dashboard are in accordance with findings from the literature, except for unusually low cholesterol levels in participants with cardiovascular disease, which could be induced by medication. In addition, 95% CIs of Pearson correlation coefficients were calculated for all associations identified during the data analysis, confirming the results. In addition, a user test with 10 physicians assessing the usability of the proposed methods was conducted. A System Usability Scale score of 70.5% and average successful task completion of 81.4% were reported. Conclusions: The proposed attribute association graph and dashboard enable intuitive visual data exploration. They are robust to high-dimensional as well as missing data and require no parameterization. The usability for clinicians was confirmed via a user test, and the validity of the statistical results was confirmed by associations known from literature and standard statistical inference. %M 39046780 %R 10.2196/49865 %U https://medinform.jmir.org/2024/1/e49865 %U https://doi.org/10.2196/49865 %U http://www.ncbi.nlm.nih.gov/pubmed/39046780 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e59187 %T Integrating Clinical Data and Medical Imaging in Lung Cancer: Feasibility Study Using the Observational Medical Outcomes Partnership Common Data Model Extension %A Ji,Hyerim %A Kim,Seok %A Sunwoo,Leonard %A Jang,Sowon %A Lee,Ho-Young %A Yoo,Sooyoung %+ Office of eHealth Research and Business, Seoul National University Bundang Hospital, 172, Dolma-ro, Bundang-gu, Seongnam-si, 13605, Republic of Korea, 82 317878980, yoosoo0@snubh.org %K DICOM %K OMOP %K CDM %K lung cancer %K medical imaging %K data integration %K data quality %K Common Data Model %K Digital Imaging and Communications in Medicine %K Observational Medical Outcomes Partnership %D 2024 %7 12.7.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Digital transformation, particularly the integration of medical imaging with clinical data, is vital in personalized medicine. The Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) standardizes health data. However, integrating medical imaging remains a challenge. Objective: This study proposes a method for combining medical imaging data with the OMOP CDM to improve multimodal research. Methods: Our approach included the analysis and selection of digital imaging and communications in medicine header tags, validation of data formats, and alignment according to the OMOP CDM framework. The Fast Healthcare Interoperability Resources ImagingStudy profile guided our consistency in column naming and definitions. Imaging Common Data Model (I-CDM), constructed using the entity-attribute-value model, facilitates scalable and efficient medical imaging data management. For patients with lung cancer diagnosed between 2010 and 2017, we introduced 4 new tables—IMAGING_STUDY, IMAGING_SERIES, IMAGING_ANNOTATION, and FILEPATH—to standardize various imaging-related data and link to clinical data. Results: This framework underscores the effectiveness of I-CDM in enhancing our understanding of lung cancer diagnostics and treatment strategies. The implementation of the I-CDM tables enabled the structured organization of a comprehensive data set, including 282,098 IMAGING_STUDY, 5,674,425 IMAGING_SERIES, and 48,536 IMAGING_ANNOTATION records, illustrating the extensive scope and depth of the approach. A scenario-based analysis using actual data from patients with lung cancer underscored the feasibility of our approach. A data quality check applying 44 specific rules confirmed the high integrity of the constructed data set, with all checks successfully passed, underscoring the reliability of our findings. Conclusions: These findings indicate that I-CDM can improve the integration and analysis of medical imaging and clinical data. By addressing the challenges in data standardization and management, our approach contributes toward enhancing diagnostics and treatment strategies. Future research should expand the application of I-CDM to diverse disease populations and explore its wide-ranging utility for medical conditions. %M 38996330 %R 10.2196/59187 %U https://medinform.jmir.org/2024/1/e59187 %U https://doi.org/10.2196/59187 %U http://www.ncbi.nlm.nih.gov/pubmed/38996330 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e59680 %T Is Boundary Annotation Necessary? Evaluating Boundary-Free Approaches to Improve Clinical Named Entity Annotation Efficiency: Case Study %A Herman Bernardim Andrade,Gabriel %A Yada,Shuntaro %A Aramaki,Eiji %+ Graduate School of Science and Technology, Nara Institute of Science and Technology, 8916-5, Takayama-cho, Ikoma, 630-0192, Japan, 81 743 72 5250, aramaki@is.naist.jp %K natural language processing %K named entity recognition %K information extraction %K text annotation %K entity boundaries %K lenient annotation %K case reports %K annotation %K case study %K medical case report %K efficiency %K model %K model performance %K dataset %K Japan %K Japanese %K entity %K clinical domain %K clinical %D 2024 %7 2.7.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Named entity recognition (NER) is a fundamental task in natural language processing. However, it is typically preceded by named entity annotation, which poses several challenges, especially in the clinical domain. For instance, determining entity boundaries is one of the most common sources of disagreements between annotators due to questions such as whether modifiers or peripheral words should be annotated. If unresolved, these can induce inconsistency in the produced corpora, yet, on the other hand, strict guidelines or adjudication sessions can further prolong an already slow and convoluted process. Objective: The aim of this study is to address these challenges by evaluating 2 novel annotation methodologies, lenient span and point annotation, aiming to mitigate the difficulty of precisely determining entity boundaries. Methods: We evaluate their effects through an annotation case study on a Japanese medical case report data set. We compare annotation time, annotator agreement, and the quality of the produced labeling and assess the impact on the performance of an NER system trained on the annotated corpus. Results: We saw significant improvements in the labeling process efficiency, with up to a 25% reduction in overall annotation time and even a 10% improvement in annotator agreement compared to the traditional boundary-strict approach. However, even the best-achieved NER model presented some drop in performance compared to the traditional annotation methodology. Conclusions: Our findings demonstrate a balance between annotation speed and model performance. Although disregarding boundary information affects model performance to some extent, this is counterbalanced by significant reductions in the annotator’s workload and notable improvements in the speed of the annotation process. These benefits may prove valuable in various applications, offering an attractive compromise for developers and researchers. %M 38954456 %R 10.2196/59680 %U https://medinform.jmir.org/2024/1/e59680 %U https://doi.org/10.2196/59680 %U http://www.ncbi.nlm.nih.gov/pubmed/38954456 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e50980 %T An Ontology-Based Decision Support System for Tailored Clinical Nutrition Recommendations for Patients With Chronic Obstructive Pulmonary Disease: Development and Acceptability Study %A Spoladore,Daniele %A Colombo,Vera %A Fumagalli,Alessia %A Tosi,Martina %A Lorenzini,Erna Cecilia %A Sacco,Marco %+ Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council of Italy, Via G. Previati 1E, Lecco, 23900, Italy, 39 03412350202, daniele.spoladore@stiima.cnr.it %K ontology-based decision support system %K nutritional recommendation %K chronic obstructive pulmonary disease %K clinical decision support system %K pulmonary rehabilitation %D 2024 %7 26.6.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Chronic obstructive pulmonary disease (COPD) is a chronic condition among the main causes of morbidity and mortality worldwide, representing a burden on health care systems. Scientific literature highlights that nutrition is pivotal in respiratory inflammatory processes connected to COPD, including exacerbations. Patients with COPD have an increased risk of developing nutrition-related comorbidities, such as diabetes, cardiovascular diseases, and malnutrition. Moreover, these patients often manifest sarcopenia and cachexia. Therefore, an adequate nutritional assessment and therapy are essential to help individuals with COPD in managing the progress of the disease. However, the role of nutrition in pulmonary rehabilitation (PR) programs is often underestimated due to a lack of resources and dedicated services, mostly because pneumologists may lack the specialized training for such a discipline. Objective: This work proposes a novel knowledge-based decision support system to support pneumologists in considering nutritional aspects in PR. The system provides clinicians with patient-tailored dietary recommendations leveraging expert knowledge. Methods: The expert knowledge—acquired from experts and clinical literature—was formalized in domain ontologies and rules, which were developed leveraging the support of Italian clinicians with expertise in the rehabilitation of patients with COPD. Thus, by following an agile ontology engineering methodology, the relevant formal ontologies were developed to act as a backbone for an application targeted at pneumologists. The recommendations provided by the decision support system were validated by a group of nutrition experts, whereas the acceptability of such an application in the context of PR was evaluated by pneumologists. Results: A total of 7 dieticians (mean age 46.60, SD 13.35 years) were interviewed to assess their level of agreement with the decision support system’s recommendations by evaluating 5 patients’ health conditions. The preliminary results indicate that the system performed more than adequately (with an overall average score of 4.23, SD 0.52 out of 5 points), providing meaningful and safe recommendations in compliance with clinical practice. With regard to the acceptability of the system by lung specialists (mean age 44.71, SD 11.94 years), the usefulness and relevance of the proposed solution were extremely positive—the scores on each of the perceived usefulness subscales of the technology acceptance model 3 were 4.86 (SD 0.38) out of 5 points, whereas the score on the intention to use subscale was 4.14 (SD 0.38) out of 5 points. Conclusions: Although designed for the Italian clinical context, the proposed system can be adapted for any other national clinical context by modifying the domain ontologies, thus providing a multidisciplinary approach to the management of patients with COPD. %M 38922666 %R 10.2196/50980 %U https://medinform.jmir.org/2024/1/e50980 %U https://doi.org/10.2196/50980 %U http://www.ncbi.nlm.nih.gov/pubmed/38922666 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e56780 %T Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses %A Luo,Xufei %A Chen,Fengxian %A Zhu,Di %A Wang,Ling %A Wang,Zijun %A Liu,Hui %A Lyu,Meng %A Wang,Ye %A Wang,Qi %A Chen,Yaolong %+ Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, No 199 Donggang West Road, Chengguan District, Lanzhou, 730000, China, 86 13893104140, chevidence@lzu.edu.cn %K large language model %K ChatGPT %K systematic review %K chatbot %K meta-analysis %D 2024 %7 25.6.2024 %9 Viewpoint %J J Med Internet Res %G English %X Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses. %M 38819655 %R 10.2196/56780 %U https://www.jmir.org/2024/1/e56780 %U https://doi.org/10.2196/56780 %U http://www.ncbi.nlm.nih.gov/pubmed/38819655 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e49613 %T Dermoscopy Differential Diagnosis Explorer (D3X) Ontology to Aggregate and Link Dermoscopic Patterns to Differential Diagnoses: Development and Usability Study %A Lin,Rebecca Z %A Amith,Muhammad Tuan %A Wang,Cynthia X %A Strickley,John %A Tao,Cui %+ Department of Artificial Intelligence and Informatics, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL, 32224, United States, 1 9049530255, Tao.Cui@mayo.edu %K medical informatics %K biomedical ontology %K ontology %K ontologies %K vocabulary %K OWL %K web ontology language %K skin %K semiotic %K web app %K web application %K visual %K visualization %K dermoscopic %K diagnosis %K diagnoses %K diagnostic %K information storage %K information retrieval %K skin lesion %K skin diseases %K dermoscopy differential diagnosis explorer %K dermatology %K dermoscopy %K differential diagnosis %K information storage and retrieval %D 2024 %7 21.6.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Dermoscopy is a growing field that uses microscopy to allow dermatologists and primary care physicians to identify skin lesions. For a given skin lesion, a wide variety of differential diagnoses exist, which may be challenging for inexperienced users to name and understand. Objective: In this study, we describe the creation of the dermoscopy differential diagnosis explorer (D3X), an ontology linking dermoscopic patterns to differential diagnoses. Methods: Existing ontologies that were incorporated into D3X include the elements of visuals ontology and dermoscopy elements of visuals ontology, which connect visual features to dermoscopic patterns. A list of differential diagnoses for each pattern was generated from the literature and in consultation with domain experts. Open-source images were incorporated from DermNet, Dermoscopedia, and open-access research papers. Results: D3X was encoded in the OWL 2 web ontology language and includes 3041 logical axioms, 1519 classes, 103 object properties, and 20 data properties. We compared D3X with publicly available ontologies in the dermatology domain using a semiotic theory–driven metric to measure the innate qualities of D3X with others. The results indicate that D3X is adequately comparable with other ontologies of the dermatology domain. Conclusions: The D3X ontology is a resource that can link and integrate dermoscopic differential diagnoses and supplementary information with existing ontology-based resources. Future directions include developing a web application based on D3X for dermoscopy education and clinical practice. %M 38904996 %R 10.2196/49613 %U https://medinform.jmir.org/2024/1/e49613 %U https://doi.org/10.2196/49613 %U http://www.ncbi.nlm.nih.gov/pubmed/38904996 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 13 %N %P e54910 %T Communication and Contextual Factors in Robotic-Assisted Surgical Teams: Protocol for Developing a Taxonomy %A Nyein,Kyi Phyu %A Condron,Claire %+ SIM Centre for Simulation Education and Research, Royal College of Surgeons in Ireland, University of Medicine and Health Sciences, 123 St Stephen’s Green, Dublin 2, Dublin, D02 YN77, Ireland, 353 (01) 402 2100, kyiphyunyein@rcsi.ie %K communication %K teams %K robotic surgery %K robotic-assisted %K simulation %D 2024 %7 17.6.2024 %9 Protocol %J JMIR Res Protoc %G English %X Background: Robotic-assisted surgery (RAS) has been rapidly integrated into surgical practice in the past few decades. The setup in the operating theater for RAS differs from that for open or laparoscopic surgery such that the operating surgeon sits at a console separate from the rest of the surgical team and the patient. Communication and team dynamics are altered due to this physical separation and visual barriers imposed by the robotic equipment. However, the factors that might comprise effective communication among members of RAS teams and the contextual factors that facilitate or inhibit effective communication in RAS remain unknown. Objective: We aim to develop a taxonomy of communication behaviors and contextual factors that influence communication in RAS teams. We also aim to examine the patterns of communication behaviors based on gender. Methods: We will first perform a scoping review on communication in RAS to develop a preliminary taxonomy of communication based on the existing literature. We will then conduct semistructured interviews with RAS team members, including the surgeon, assisting surgeon or trainee, bedside or first assistant, nurses, and anesthetists. Participants will represent different disciplines, including urology, general surgery, and gynecology, and have a range of experiences in RAS. We will use a reflexive thematic analysis to analyze the data and further refine the taxonomy. We will also observe live robotic surgeries at Royal College of Surgeons in Ireland (RCSI)–affiliated hospitals. We will observe varying lengths and conditions of RAS procedures to a capture a wide range of communication behaviors and contextual factors to help finalize the taxonomy. Although we anticipate conducting 30 interviews and 30 observations, we will collect data until we achieve data sufficiency. We will conduct data collection in parallel with data analysis such that if we identify a new behavior in an interview, we will follow up with questions related to that behavior in additional interviews and/or observations. Results: The taxonomy from this project will include a list of actionable communication behaviors, contextual factors, their descriptions, and examples. As of May 2024, this project has been approved by the RCSI Research and Ethics Committee. Data collection started in June 2024 and will continue throughout the year. We plan to publish the findings as meaningful results emerge in our data analysis in 2024 and 2025. Conclusions: The results from this project will be used to observe and train surgical teams in a simulated environment to effectively communicate with each other and prevent communication breakdowns. The developed taxonomy will also add to the knowledge base on the role of gender in communication in RAS and produce recommendations that can be incorporated into training. Overall, this project will contribute to the improvement of communication skills of surgical teams and the quality and safety of patient care. International Registered Report Identifier (IRRID): PRR1-10.2196/54910 %M 38885018 %R 10.2196/54910 %U https://www.researchprotocols.org/2024/1/e54910 %U https://doi.org/10.2196/54910 %U http://www.ncbi.nlm.nih.gov/pubmed/38885018 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e50049 %T Creation of Standardized Common Data Elements for Diagnostic Tests in Infectious Disease Studies: Semantic and Syntactic Mapping %A Stellmach,Caroline %A Hopff,Sina Marie %A Jaenisch,Thomas %A Nunes de Miranda,Susana Marina %A Rinaldi,Eugenia %A , %+ Berlin Institute of Health, Charité - Universitätsmedizin Berlin, Anna-Louisa-Karsch-Str 2, Berlin, 10178, Germany, 49 15752614677, caroline.stellmach@charite.de %K core data element %K CDE %K case report form %K CRF %K interoperability %K semantic standards %K infectious disease %K diagnostic test %K covid19 %K COVID-19 %K mpox %K ZIKV %K patient data %K data model %K syntactic interoperability %K clinical data %K FHIR %K SNOMED CT %K LOINC %K virus infection %K common element %D 2024 %7 10.6.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: It is necessary to harmonize and standardize data variables used in case report forms (CRFs) of clinical studies to facilitate the merging and sharing of the collected patient data across several clinical studies. This is particularly true for clinical studies that focus on infectious diseases. Public health may be highly dependent on the findings of such studies. Hence, there is an elevated urgency to generate meaningful, reliable insights, ideally based on a high sample number and quality data. The implementation of core data elements and the incorporation of interoperability standards can facilitate the creation of harmonized clinical data sets. Objective: This study’s objective was to compare, harmonize, and standardize variables focused on diagnostic tests used as part of CRFs in 6 international clinical studies of infectious diseases in order to, ultimately, then make available the panstudy common data elements (CDEs) for ongoing and future studies to foster interoperability and comparability of collected data across trials. Methods: We reviewed and compared the metadata that comprised the CRFs used for data collection in and across all 6 infectious disease studies under consideration in order to identify CDEs. We examined the availability of international semantic standard codes within the Systemized Nomenclature of Medicine - Clinical Terms, the National Cancer Institute Thesaurus, and the Logical Observation Identifiers Names and Codes system for the unambiguous representation of diagnostic testing information that makes up the CDEs. We then proposed 2 data models that incorporate semantic and syntactic standards for the identified CDEs. Results: Of 216 variables that were considered in the scope of the analysis, we identified 11 CDEs to describe diagnostic tests (in particular, serology and sequencing) for infectious diseases: viral lineage/clade; test date, type, performer, and manufacturer; target gene; quantitative and qualitative results; and specimen identifier, type, and collection date. Conclusions: The identification of CDEs for infectious diseases is the first step in facilitating the exchange and possible merging of a subset of data across clinical studies (and with that, large research projects) for possible shared analysis to increase the power of findings. The path to harmonization and standardization of clinical study data in the interest of interoperability can be paved in 2 ways. First, a map to standard terminologies ensures that each data element’s (variable’s) definition is unambiguous and that it has a single, unique interpretation across studies. Second, the exchange of these data is assisted by “wrapping” them in a standard exchange format, such as Fast Health care Interoperability Resources or the Clinical Data Interchange Standards Consortium’s Clinical Data Acquisition Standards Harmonization Model. %M 38857066 %R 10.2196/50049 %U https://www.jmir.org/2024/1/e50049 %U https://doi.org/10.2196/50049 %U http://www.ncbi.nlm.nih.gov/pubmed/38857066 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 8 %N %P e55798 %T Using the Natural Language Processing System Medical Named Entity Recognition-Japanese to Analyze Pharmaceutical Care Records: Natural Language Processing Analysis %A Ohno,Yukiko %A Kato,Riri %A Ishikawa,Haruki %A Nishiyama,Tomohiro %A Isawa,Minae %A Mochizuki,Mayumi %A Aramaki,Eiji %A Aomori,Tohru %+ Faculty of Pharmacy, Takasaki University of Health and Welfare, 37-1 Nakaorui-machi, Takasaki-shi, Gunma, 370-0033, Japan, 81 273521290, aomori-t@takasaki-u.ac.jp %K natural language processing %K NLP %K named entity recognition %K pharmaceutical care records %K machine learning %K cefazolin sodium %K electronic medical record %K EMR %K extraction %K Japanese %D 2024 %7 4.6.2024 %9 Original Paper %J JMIR Form Res %G English %X Background: Large language models have propelled recent advances in artificial intelligence technology, facilitating the extraction of medical information from unstructured data such as medical records. Although named entity recognition (NER) is used to extract data from physicians’ records, it has yet to be widely applied to pharmaceutical care records. Objective: In this study, we aimed to investigate the feasibility of automatic extraction of the information regarding patients’ diseases and symptoms from pharmaceutical care records. The verification was performed using Medical Named Entity Recognition-Japanese (MedNER-J), a Japanese disease-extraction system designed for physicians’ records. Methods: MedNER-J was applied to subjective, objective, assessment, and plan data from the care records of 49 patients who received cefazolin sodium injection at Keio University Hospital between April 2018 and March 2019. The performance of MedNER-J was evaluated in terms of precision, recall, and F1-score. Results: The F1-scores of NER for subjective, objective, assessment, and plan data were 0.46, 0.70, 0.76, and 0.35, respectively. In NER and positive-negative classification, the F1-scores were 0.28, 0.39, 0.64, and 0.077, respectively. The F1-scores of NER for objective (0.70) and assessment data (0.76) were higher than those for subjective and plan data, which supported the superiority of NER performance for objective and assessment data. This might be because objective and assessment data contained many technical terms, similar to the training data for MedNER-J. Meanwhile, the F1-score of NER and positive-negative classification was high for assessment data alone (F1-score=0.64), which was attributed to the similarity of its description format and contents to those of the training data. Conclusions: MedNER-J successfully read pharmaceutical care records and showed the best performance for assessment data. However, challenges remain in analyzing records other than assessment data. Therefore, it will be necessary to reinforce the training data for subjective data in order to apply the system to pharmaceutical care records. %M 38833694 %R 10.2196/55798 %U https://formative.jmir.org/2024/1/e55798 %U https://doi.org/10.2196/55798 %U http://www.ncbi.nlm.nih.gov/pubmed/38833694 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e47682 %T A Taxonomy for Health Information Systems %A Janssen,Anna %A Donnelly,Candice %A Shaw,Tim %+ Faculty of Medicine and Health, The University of Sydney, Level 2, Charles Perkins Centre D17, Sydney, 2066, Australia, 61 02 9036 9406, anna.janssen@sydney.edu.au %K eHealth %K digital health %K electronic health data %K data revolution %K actionable data %K mobile phone %D 2024 %7 31.5.2024 %9 Viewpoint %J J Med Internet Res %G English %X The health sector is highly digitized, which is enabling the collection of vast quantities of electronic data about health and well-being. These data are collected by a diverse array of information and communication technologies, including systems used by health care organizations, consumer and community sources such as information collected on the web, and passively collected data from technologies such as wearables and devices. Understanding the breadth of IT that collect these data and how it can be actioned is a challenge for the significant portion of the digital health workforce that interact with health data as part of their duties but are not for informatics experts. This viewpoint aims to present a taxonomy categorizing common information and communication technologies that collect electronic data. An initial classification of key information systems collecting electronic health data was undertaken via a rapid review of the literature. Subsequently, a purposeful search of the scholarly and gray literature was undertaken to extract key information about the systems within each category to generate definitions of the systems and describe the strengths and limitations of these systems. %M 38820575 %R 10.2196/47682 %U https://www.jmir.org/2024/1/e47682 %U https://doi.org/10.2196/47682 %U http://www.ncbi.nlm.nih.gov/pubmed/38820575 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e54974 %T Harnessing ChatGPT for Thematic Analysis: Are We Ready? %A Lee,V Vien %A van der Lubbe,Stephanie C C %A Goh,Lay Hoon %A Valderas,Jose Maria %+ Division of Family Medicine, Yong Loo Lin School of Medicine, National University of Singapore, NUHS Tower Block, Level 9, 1E Kent Ridge Road, Singapore, 119228, Singapore, 65 67723874, jmvalderas@nus.edu.sg %K ChatGPT %K thematic analysis %K natural language processing %K NLP %K medical research %K qualitative research %K qualitative data %K technology %K viewpoint %K efficiency %D 2024 %7 31.5.2024 %9 Viewpoint %J J Med Internet Res %G English %X ChatGPT (OpenAI) is an advanced natural language processing tool with growing applications across various disciplines in medical research. Thematic analysis, a qualitative research method to identify and interpret patterns in data, is one application that stands to benefit from this technology. This viewpoint explores the use of ChatGPT in three core phases of thematic analysis within a medical context: (1) direct coding of transcripts, (2) generating themes from a predefined list of codes, and (3) preprocessing quotes for manuscript inclusion. Additionally, we explore the potential of ChatGPT to generate interview transcripts, which may be used for training purposes. We assess the strengths and limitations of using ChatGPT in these roles, highlighting areas where human intervention remains necessary. Overall, we argue that ChatGPT can function as a valuable tool during analysis, enhancing the efficiency of the thematic analysis and offering additional insights into the qualitative data. While ChatGPT may not adequately capture the full context of each participant, it can serve as an additional member of the analysis team, contributing to researcher triangulation through knowledge building and sensemaking. %M 38819896 %R 10.2196/54974 %U https://www.jmir.org/2024/1/e54974 %U https://doi.org/10.2196/54974 %U http://www.ncbi.nlm.nih.gov/pubmed/38819896 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e56614 %T Redefining Health Care Data Interoperability: Empirical Exploration of Large Language Models in Information Exchange %A Yoon,Dukyong %A Han,Changho %A Kim,Dong Won %A Kim,Songsoo %A Bae,SungA %A Ryu,Jee An %A Choi,Yujin %+ Department of Biomedical Systems Informatics, Yonsei University College of Medicine, 50-1 Yonsei-ro Seodaemun-gu, Seoul, 03722, Republic of Korea, 82 31 5189 8450, dukyong.yoon@yonsei.ac.kr %K health care interoperability %K large language models %K medical data transformation %K data standardization %K text-based %D 2024 %7 31.5.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Efficient data exchange and health care interoperability are impeded by medical records often being in nonstandardized or unstructured natural language format. Advanced language models, such as large language models (LLMs), may help overcome current challenges in information exchange. Objective: This study aims to evaluate the capability of LLMs in transforming and transferring health care data to support interoperability. Methods: Using data from the Medical Information Mart for Intensive Care III and UK Biobank, the study conducted 3 experiments. Experiment 1 assessed the accuracy of transforming structured laboratory results into unstructured format. Experiment 2 explored the conversion of diagnostic codes between the coding frameworks of the ICD-9-CM (International Classification of Diseases, Ninth Revision, Clinical Modification), and Systematized Nomenclature of Medicine Clinical Terms (SNOMED-CT) using a traditional mapping table and a text-based approach facilitated by the LLM ChatGPT. Experiment 3 focused on extracting targeted information from unstructured records that included comprehensive clinical information (discharge notes). Results: The text-based approach showed a high conversion accuracy in transforming laboratory results (experiment 1) and an enhanced consistency in diagnostic code conversion, particularly for frequently used diagnostic names, compared with the traditional mapping approach (experiment 2). In experiment 3, the LLM showed a positive predictive value of 87.2% in extracting generic drug names. Conclusions: This study highlighted the potential role of LLMs in significantly improving health care data interoperability, demonstrated by their high accuracy and efficiency in data transformation and exchange. The LLMs hold vast potential for enhancing medical data exchange without complex standardization for medical terms and data structure. %M 38819879 %R 10.2196/56614 %U https://www.jmir.org/2024/1/e56614 %U https://doi.org/10.2196/56614 %U http://www.ncbi.nlm.nih.gov/pubmed/38819879 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e52655 %T Searching COVID-19 Clinical Research Using Graph Queries: Algorithm Development and Validation %A Invernici,Francesco %A Bernasconi,Anna %A Ceri,Stefano %+ Department of Electronics, Information, and Bioengineering, Politecnico di Milano, Via Ponzio 34/5, Milan, 20133, Italy, 39 23993494, anna.bernasconi@polimi.it %K big data corpus %K clinical research %K co-occurrence network %K COVID-19 Open Research Dataset %K CORD-19 %K graph search %K Named Entity Recognition %K Neo4j %K text mining %D 2024 %7 30.5.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Since the beginning of the COVID-19 pandemic, >1 million studies have been collected within the COVID-19 Open Research Dataset, a corpus of manuscripts created to accelerate research against the disease. Their related abstracts hold a wealth of information that remains largely unexplored and difficult to search due to its unstructured nature. Keyword-based search is the standard approach, which allows users to retrieve the documents of a corpus that contain (all or some of) the words in a target list. This type of search, however, does not provide visual support to the task and is not suited to expressing complex queries or compensating for missing specifications. Objective: This study aims to consider small graphs of concepts and exploit them for expressing graph searches over existing COVID-19–related literature, leveraging the increasing use of graphs to represent and query scientific knowledge and providing a user-friendly search and exploration experience. Methods: We considered the COVID-19 Open Research Dataset corpus and summarized its content by annotating the publications’ abstracts using terms selected from the Unified Medical Language System and the Ontology of Coronavirus Infectious Disease. Then, we built a co-occurrence network that includes all relevant concepts mentioned in the corpus, establishing connections when their mutual information is relevant. A sophisticated graph query engine was built to allow the identification of the best matches of graph queries on the network. It also supports partial matches and suggests potential query completions using shortest paths. Results: We built a large co-occurrence network, consisting of 128,249 entities and 47,198,965 relationships; the GRAPH-SEARCH interface allows users to explore the network by formulating or adapting graph queries; it produces a bibliography of publications, which are globally ranked; and each publication is further associated with the specific parts of the query that it explains, thereby allowing the user to understand each aspect of the matching. Conclusions: Our approach supports the process of query formulation and evidence search upon a large text corpus; it can be reapplied to any scientific domain where documents corpora and curated ontologies are made available. %M 38814687 %R 10.2196/52655 %U https://www.jmir.org/2024/1/e52655 %U https://doi.org/10.2196/52655 %U http://www.ncbi.nlm.nih.gov/pubmed/38814687 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e53724 %T Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases %A Chiu,Wan Hang Keith %A Ko,Wei Sum Koel %A Cho,William Chi Shing %A Hui,Sin Yu Joanne %A Chan,Wing Chi Lawrence %A Kuo,Michael D %+ Ensemble Group, 10541 E Firewheel Drive, Scottsdale, AZ, 85259, United States, 1 4084512341, mikedkuo@gmail.com %K large language model %K hospital %K health center %K Massachusetts %K statistical analysis %K chi-square %K ANOVA %K clinician %K physician %K performance %K proficiency %K disease etiology %D 2024 %7 13.5.2024 %9 Research Letter %J J Med Internet Res %G English %X Large language models showed interpretative reasoning in solving diagnostically challenging medical cases. %M 38739441 %R 10.2196/53724 %U https://www.jmir.org/2024/1/e53724 %U https://doi.org/10.2196/53724 %U http://www.ncbi.nlm.nih.gov/pubmed/38739441 %0 Journal Article %@ 2291-9694 %I %V 12 %N %P e51274 %T Evaluation of SNOMED CT Grouper Accuracy and Coverage in Organizing the Electronic Health Record Problem List by Clinical System: Observational Study %A Senior,Rashaud %A Tsai,Timothy %A Ratliff,William %A Nadler,Lisa %A Balu,Suresh %A Malcolm,Elizabeth %A McPeek Hinz,Eugenia %K electronic health record %K problem List %K problem list organization %K problem list management %K SNOMED CT %K SNOMED CT Groupers %K Systematized Nomenclature of Medicine %K clinical term %K ICD-10 %K International Classification of Diseases %D 2024 %7 9.5.2024 %9 %J JMIR Med Inform %G English %X Background: The problem list (PL) is a repository of diagnoses for patients’ medical conditions and health-related issues. Unfortunately, over time, our PLs have become overloaded with duplications, conflicting entries, and no-longer-valid diagnoses. The lack of a standardized structure for review adds to the challenges of clinical use. Previously, our default electronic health record (EHR) organized the PL primarily via alphabetization, with other options available, for example, organization by clinical systems or priority settings. The system’s PL was built with limited groupers, resulting in many diagnoses that were inconsistent with the expected clinical systems or not associated with any clinical systems at all. As a consequence of these limited EHR configuration options, our PL organization has poorly supported clinical use over time, particularly as the number of diagnoses on the PL has increased. Objective: We aimed to measure the accuracy of sorting PL diagnoses into PL system groupers based on Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) concept groupers implemented in our EHR. Methods: We transformed and developed 21 system- or condition-based groupers, using 1211 SNOMED CT hierarchal concepts refined with Boolean logic, to reorganize the PL in our EHR. To evaluate the clinical utility of our new groupers, we extracted all diagnoses on the PLs from a convenience sample of 50 patients with 3 or more encounters in the previous year. To provide a spectrum of clinical diagnoses, we included patients from all ages and divided them by sex in a deidentified format. Two physicians independently determined whether each diagnosis was correctly attributed to the expected clinical system grouper. Discrepancies were discussed, and if no consensus was reached, they were adjudicated by a third physician. Descriptive statistics and Cohen κ statistics for interrater reliability were calculated. Results: Our 50-patient sample had a total of 869 diagnoses (range 4-59; median 12, IQR 9-24). The reviewers initially agreed on 821 system attributions. Of the remaining 48 items, 16 required adjudication with the tie-breaking third physician. The calculated κ statistic was 0.7. The PL groupers appropriately associated diagnoses to the expected clinical system with a sensitivity of 97.6%, a specificity of 58.7%, a positive predictive value of 96.8%, and an F1-score of 0.972. Conclusions: We found that PL organization by clinical specialty or condition using SNOMED CT concept groupers accurately reflects clinical systems. Our system groupers were subsequently adopted by our vendor EHR in their foundation system for PL organization. %R 10.2196/51274 %U https://medinform.jmir.org/2024/1/e51274 %U https://doi.org/10.2196/51274 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 16 %N %P e53445 %T Characterization of Post–COVID-19 Definitions and Clinical Coding Practices: Longitudinal Study %A Maripuri,Monika %A Dey,Andrew %A Honerlaw,Jacqueline %A Hong,Chuan %A Ho,Yuk-Lam %A Tanukonda,Vidisha %A Chen,Alicia W %A Panickan,Vidul Ayakulangara %A Wang,Xuan %A Zhang,Harrison G %A Yang,Doris %A Samayamuthu,Malarkodi Jebathilagam %A Morris,Michele %A Visweswaran,Shyam %A Beaulieu-Jones,Brendin %A Ramoni,Rachel %A Muralidhar,Sumitra %A Gaziano,J Michael %A Liao,Katherine %A Xia,Zongqi %A Brat,Gabriel A %A Cai,Tianxi %A Cho,Kelly %+ Veterans Affairs Boston Healthcare System, 2 Avenue de Lafayette, Boston, MA, 02111, United States, 1 617 390 6976, monika.maripuri@va.gov %K veterans %K long COVID-19 %K postacute sequelae of SARS-CoV-2 %K PASC %K International Classification of Diseases %K U09.9 ICD-10 code %K algorithm validation %K chart review %K electronic health records %K COVID-19 %D 2024 %7 3.5.2024 %9 Original Paper %J Online J Public Health Inform %G English %X Background: Post–COVID-19 condition (colloquially known as “long COVID-19”) characterized as postacute sequelae of SARS-CoV-2 has no universal clinical case definition. Recent efforts have focused on understanding long COVID-19 symptoms, and electronic health record (EHR) data provide a unique resource for understanding this condition. The introduction of the International Classification of Diseases, Tenth Revision (ICD-10) code U09.9 for “Post COVID-19 condition, unspecified” to identify patients with long COVID-19 has provided a method of evaluating this condition in EHRs; however, the accuracy of this code is unclear. Objective: This study aimed to characterize the utility and accuracy of the U09.9 code across 3 health care systems—the Veterans Health Administration, the Beth Israel Deaconess Medical Center, and the University of Pittsburgh Medical Center—against patients identified with long COVID-19 via a chart review by operationalizing the World Health Organization (WHO) and Centers for Disease Control and Prevention (CDC) definitions. Methods: Patients who were COVID-19 positive with either a U07.1 ICD-10 code or positive polymerase chain reaction test within these health care systems were identified for chart review. Among this cohort, we sampled patients based on two approaches: (1) with a U09.9 code and (2) without a U09.9 code but with a new onset long COVID-19–related ICD-10 code, which allows us to assess the sensitivity of the U09.9 code. To operationalize the long COVID-19 definition based on health agency guidelines, symptoms were grouped into a “core” cluster of 11 commonly reported symptoms among patients with long COVID-19 and an extended cluster that captured all other symptoms by disease domain. Patients having ≥2 symptoms persisting for ≥60 days that were new onset after their COVID-19 infection, with ≥1 symptom in the core cluster, were labeled as having long COVID-19 per chart review. The code’s performance was compared across 3 health care systems and across different time periods of the pandemic. Results: Overall, 900 patient charts were reviewed across 3 health care systems. The prevalence of long COVID-19 among the cohort with the U09.9 ICD-10 code based on the operationalized WHO definition was between 23.2% and 62.4% across these health care systems. We also evaluated a less stringent version of the WHO definition and the CDC definition and observed an increase in the prevalence of long COVID-19 at all 3 health care systems. Conclusions: This is one of the first studies to evaluate the U09.9 code against a clinical case definition for long COVID-19, as well as the first to apply this definition to EHR data using a chart review approach on a nationwide cohort across multiple health care systems. This chart review approach can be implemented at other EHR systems to further evaluate the utility and performance of the U09.9 code. %M 38700929 %R 10.2196/53445 %U https://ojphi.jmir.org/2024/1/e53445 %U https://doi.org/10.2196/53445 %U http://www.ncbi.nlm.nih.gov/pubmed/38700929 %0 Journal Article %@ 2291-9694 %I %V 12 %N %P e53535 %T Semantic Interoperability of Electronic Health Records: Systematic Review of Alternative Approaches for Enhancing Patient Information Availability %A Palojoki,Sari %A Lehtonen,Lasse %A Vuokko,Riikka %K electronic health record %K health records %K EHR %K EHRs %K semantic %K health care data %K semantic interoperability %K interoperability %K standardize %K standardized %K standardization %K cross-border data exchange %K systematic review %K synthesis %K syntheses %K review methods %K review methodology %K search %K searches %K searching %K systematic %K data exchange %K information sharing %K ontology %K ontologies %K terminology %K terminologies %K standard %K standards %K classification %K PRISMA %K data sharing %K Preferred Reporting Items for Systematic Reviews and Meta-Analyses %D 2024 %7 25.4.2024 %9 %J JMIR Med Inform %G English %X Background: Semantic interoperability facilitates the exchange of and access to health data that are being documented in electronic health records (EHRs) with various semantic features. The main goals of semantic interoperability development entail patient data availability and use in diverse EHRs without a loss of meaning. Internationally, current initiatives aim to enhance semantic development of EHR data and, consequently, the availability of patient data. Interoperability between health information systems is among the core goals of the European Health Data Space regulation proposal and the World Health Organization’s Global Strategy on Digital Health 2020-2025. Objective: To achieve integrated health data ecosystems, stakeholders need to overcome challenges of implementing semantic interoperability elements. To research the available scientific evidence on semantic interoperability development, we defined the following research questions: What are the key elements of and approaches for building semantic interoperability integrated in EHRs? What kinds of goals are driving the development? and What kinds of clinical benefits are perceived following this development? Methods: Our research questions focused on key aspects and approaches for semantic interoperability and on possible clinical and semantic benefits of these choices in the context of EHRs. Therefore, we performed a systematic literature review in PubMed by defining our study framework based on previous research. Results: Our analysis consisted of 14 studies where data models, ontologies, terminologies, classifications, and standards were applied for building interoperability. All articles reported clinical benefits of the selected approach to enhancing semantic interoperability. We identified 3 main categories: increasing the availability of data for clinicians (n=6, 43%), increasing the quality of care (n=4, 29%), and enhancing clinical data use and reuse for varied purposes (n=4, 29%). Regarding semantic development goals, data harmonization and developing semantic interoperability between different EHRs was the largest category (n=8, 57%). Enhancing health data quality through standardization (n=5, 36%) and developing EHR-integrated tools based on interoperable data (n=1, 7%) were the other identified categories. The results were closely coupled with the need to build usable and computable data out of heterogeneous medical information that is accessible through various EHRs and databases (eg, registers). Conclusions: When heading toward semantic harmonization of clinical data, more experiences and analyses are needed to assess how applicable the chosen solutions are for semantic interoperability of health care data. Instead of promoting a single approach, semantic interoperability should be assessed through several levels of semantic requirements A dual model or multimodel approach is possibly usable to address different semantic interoperability issues during development. The objectives of semantic interoperability are to be achieved in diffuse and disconnected clinical care environments. Therefore, approaches for enhancing clinical data availability should be well prepared, thought out, and justified to meet economically sustainable and long-term outcomes. %R 10.2196/53535 %U https://medinform.jmir.org/2024/1/e53535 %U https://doi.org/10.2196/53535 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e54419 %T Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study %A Kernberg,Annessa %A Gold,Jeffrey A %A Mohan,Vishnu %+ Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Sciences University, 3181 SW Sam Jackson Park Road, Portland, OR, 97239, United States, 1 5034944469, mohanV@ohsu.edu %K generative AI %K generative artificial intelligence %K ChatGPT %K simulation %K large language model %K clinical documentation %K quality %K accuracy %K reproducibility %K publicly available %K medical note %K medical notes %K generation %K medical documentation %K documentation %K documentations %K AI %K artificial intelligence %K transcript %K transcripts %K ChatGPT-4 %D 2024 %7 22.4.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Medical documentation plays a crucial role in clinical practice, facilitating accurate patient management and communication among health care professionals. However, inaccuracies in medical notes can lead to miscommunication and diagnostic errors. Additionally, the demands of documentation contribute to physician burnout. Although intermediaries like medical scribes and speech recognition software have been used to ease this burden, they have limitations in terms of accuracy and addressing provider-specific metrics. The integration of ambient artificial intelligence (AI)–powered solutions offers a promising way to improve documentation while fitting seamlessly into existing workflows. Objective: This study aims to assess the accuracy and quality of Subjective, Objective, Assessment, and Plan (SOAP) notes generated by ChatGPT-4, an AI model, using established transcripts of History and Physical Examination as the gold standard. We seek to identify potential errors and evaluate the model’s performance across different categories. Methods: We conducted simulated patient-provider encounters representing various ambulatory specialties and transcribed the audio files. Key reportable elements were identified, and ChatGPT-4 was used to generate SOAP notes based on these transcripts. Three versions of each note were created and compared to the gold standard via chart review; errors generated from the comparison were categorized as omissions, incorrect information, or additions. We compared the accuracy of data elements across versions, transcript length, and data categories. Additionally, we assessed note quality using the Physician Documentation Quality Instrument (PDQI) scoring system. Results: Although ChatGPT-4 consistently generated SOAP-style notes, there were, on average, 23.6 errors per clinical case, with errors of omission (86%) being the most common, followed by addition errors (10.5%) and inclusion of incorrect facts (3.2%). There was significant variance between replicates of the same case, with only 52.9% of data elements reported correctly across all 3 replicates. The accuracy of data elements varied across cases, with the highest accuracy observed in the “Objective” section. Consequently, the measure of note quality, assessed by PDQI, demonstrated intra- and intercase variance. Finally, the accuracy of ChatGPT-4 was inversely correlated to both the transcript length (P=.05) and the number of scorable data elements (P=.05). Conclusions: Our study reveals substantial variability in errors, accuracy, and note quality generated by ChatGPT-4. Errors were not limited to specific sections, and the inconsistency in error types across replicates complicated predictability. Transcript length and data complexity were inversely correlated with note accuracy, raising concerns about the model’s effectiveness in handling complex medical cases. The quality and reliability of clinical notes produced by ChatGPT-4 do not meet the standards required for clinical use. Although AI holds promise in health care, caution should be exercised before widespread adoption. Further research is needed to address accuracy, variability, and potential errors. ChatGPT-4, while valuable in various applications, should not be considered a safe alternative to human-generated clinical documentation at this time. %M 38648636 %R 10.2196/54419 %U https://www.jmir.org/2024/1/e54419 %U https://doi.org/10.2196/54419 %U http://www.ncbi.nlm.nih.gov/pubmed/38648636 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e46777 %T The Alzheimer’s Knowledge Base: A Knowledge Graph for Alzheimer Disease Research %A Romano,Joseph D %A Truong,Van %A Kumar,Rachit %A Venkatesan,Mythreye %A Graham,Britney E %A Hao,Yun %A Matsumoto,Nick %A Li,Xi %A Wang,Zhiping %A Ritchie,Marylyn D %A Shen,Li %A Moore,Jason H %+ Institute for Biomedical Informatics, Perelman School of Medicine, University of Pennsylvania, 403 Blockley Hall, 423 Guardian Drive, Philadelphia, PA, 19104, United States, 1 2155735571, joseph.romano@pennmedicine.upenn.edu %K Alzheimer disease %K knowledge graph %K knowledge base %K artificial intelligence %K drug repurposing %K drug discovery %K open source %K Alzheimer %K etiology %K heterogeneous graph %K therapeutic targets %K machine learning %K therapeutic discovery %D 2024 %7 18.4.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: As global populations age and become susceptible to neurodegenerative illnesses, new therapies for Alzheimer disease (AD) are urgently needed. Existing data resources for drug discovery and repurposing fail to capture relationships central to the disease’s etiology and response to drugs. Objective: We designed the Alzheimer’s Knowledge Base (AlzKB) to alleviate this need by providing a comprehensive knowledge representation of AD etiology and candidate therapeutics. Methods: We designed the AlzKB as a large, heterogeneous graph knowledge base assembled using 22 diverse external data sources describing biological and pharmaceutical entities at different levels of organization (eg, chemicals, genes, anatomy, and diseases). AlzKB uses a Web Ontology Language 2 ontology to enforce semantic consistency and allow for ontological inference. We provide a public version of AlzKB and allow users to run and modify local versions of the knowledge base. Results: AlzKB is freely available on the web and currently contains 118,902 entities with 1,309,527 relationships between those entities. To demonstrate its value, we used graph data science and machine learning to (1) propose new therapeutic targets based on similarities of AD to Parkinson disease and (2) repurpose existing drugs that may treat AD. For each use case, AlzKB recovers known therapeutic associations while proposing biologically plausible new ones. Conclusions: AlzKB is a new, publicly available knowledge resource that enables researchers to discover complex translational associations for AD drug discovery. Through 2 use cases, we show that it is a valuable tool for proposing novel therapeutic hypotheses based on public biomedical knowledge. %M 38635981 %R 10.2196/46777 %U https://www.jmir.org/2024/1/e46777 %U https://doi.org/10.2196/46777 %U http://www.ncbi.nlm.nih.gov/pubmed/38635981 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 13 %N %P e54838 %T Global Trends of Medical Misadventures Using International Classification of Diseases, Tenth Revision Cluster Y62-Y69 Comparing Pre–, Intra–, and Post–COVID-19 Pandemic Phases: Protocol for a Retrospective Analysis Using the TriNetX Platform %A Caruso,Rosario %A Di Muzio,Marco %A Di Simone,Emanuele %A Dionisi,Sara %A Magon,Arianna %A Conte,Gianluca %A Stievano,Alessandro %A Girani,Emanuele %A Boveri,Sara %A Menicanti,Lorenzo %A Dolansky,Mary A %+ Health Professions Research and Development Unit, IRCCS Policlinico San Donato, via morandi 30, San Donato Milanese, 20097, Italy, 39 025277 ext 4940, rosario.caruso@grupposandonato.it %K COVID-19 %K curve-fitting analyses %K health care quality %K health care safety %K International Classification of Diseases, Tenth Revision %K ICD-10 %K incidence rates %K safety %K TriNetX %D 2024 %7 17.4.2024 %9 Protocol %J JMIR Res Protoc %G English %X Background: The COVID-19 pandemic has sharpened the focus on health care safety and quality, underscoring the importance of using standardized metrics such as the International Classification of Diseases, Tenth Revision (ICD-10). In this regard, the ICD-10 cluster Y62-Y69 serves as a proxy assessment of safety and quality in health care systems, allowing researchers to evaluate medical misadventures. Thus far, extensive research and reports support the need for more attention to safety and quality in health care. The study aims to leverage the pandemic’s unique challenges to explore health care safety and quality trends during prepandemic, intrapandemic, and postpandemic phases, using the ICD-10 cluster Y62-Y69 as a key tool for their evaluation. Objective: This research aims to perform a comprehensive retrospective analysis of incidence rates associated with ICD-10 cluster Y62-Y69, capturing both linear and nonlinear trends across prepandemic, intrapandemic, and postpandemic phases over an 8-year span. Therefore, it seeks to understand how these trends inform health care safety and quality improvements, policy, and future research. Methods: This study uses the extensive data available through the TriNetX platform, using an observational, retrospective design and applying curve-fitting analyses and quadratic models to comprehend the relationships between incidence rates over an 8-year span (from 2015 to 2023). These techniques will enable the identification of nuanced trends in the data, facilitating a deeper understanding of the impacts of the COVID-19 pandemic on medical misadventures. The anticipated results aim to outline complex patterns in health care safety and quality during the COVID-19 pandemic, using global real-world data for robust and generalizable conclusions. This study will explore significant shifts in health care practices and outcomes, with a special focus on geographical variations and key clinical conditions in cardiovascular and oncological care, ensuring a comprehensive analysis of the pandemic’s impact across different regions and medical fields. Results: This study is currently in the data collection phase, with funding secured in November 2023 through the Ricerca Corrente scheme of the Italian Ministry of Health. Data collection via the TriNetX platform is anticipated to be completed in May 2024, covering an 8-year period from January 2015 to December 2023. This dataset spans pre-pandemic, intra-pandemic, and early post-pandemic phases, enabling a comprehensive analysis of trends in medical misadventures using the ICD-10 cluster Y62-Y69. The final analytics are anticipated to be completed by June 2024. The study's findings aim to provide actionable insights for enhancing healthcare safety and quality, reflecting on the pandemic's transformative impact on global healthcare systems. Conclusions: This study is anticipated to contribute significantly to health care safety and quality literature. It will provide actionable insights for health care professionals, policy makers, and researchers. It will highlight critical areas for intervention and funding to enhance health care safety and quality globally by examining the incidence rates of medical misadventures before, during, and after the pandemic. In addition, the use of global real-world data enhances the study’s strength by providing a practical view of health care safety and quality, paving the way for initiatives that are informed by data and tailored to specific contexts worldwide. This approach ensures the findings are applicable and actionable across different health care settings, contributing significantly to the global understanding and improvement of health care safety and quality. International Registered Report Identifier (IRRID): PRR1-10.2196/54838 %M 38630516 %R 10.2196/54838 %U https://www.researchprotocols.org/2024/1/e54838 %U https://doi.org/10.2196/54838 %U http://www.ncbi.nlm.nih.gov/pubmed/38630516 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e55318 %T An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study %A Sivarajkumar,Sonish %A Kelley,Mark %A Samolyk-Mazzanti,Alyssa %A Visweswaran,Shyam %A Wang,Yanshan %+ Department of Health Information Management, University of Pittsburgh, 6026 Forbes Tower, Pittsburgh, PA, 15260, United States, 1 4123832712, yanshan.wang@pitt.edu %K large language model %K LLM %K LLMs %K natural language processing %K NLP %K in-context learning %K prompt engineering %K evaluation %K zero-shot %K few shot %K prompting %K GPT %K language model %K language %K models %K machine learning %K clinical data %K clinical information %K extraction %K BARD %K Gemini %K LLaMA-2 %K heuristic %K prompt %K prompts %K ensemble %D 2024 %7 8.4.2024 %9 Original Paper %J JMIR Med Inform %G English %X Background: Large language models (LLMs) have shown remarkable capabilities in natural language processing (NLP), especially in domains where labeled data are scarce or expensive, such as the clinical domain. However, to unlock the clinical knowledge hidden in these LLMs, we need to design effective prompts that can guide them to perform specific clinical NLP tasks without any task-specific training data. This is known as in-context learning, which is an art and science that requires understanding the strengths and weaknesses of different LLMs and prompt engineering approaches. Objective: The objective of this study is to assess the effectiveness of various prompt engineering techniques, including 2 newly introduced types—heuristic and ensemble prompts, for zero-shot and few-shot clinical information extraction using pretrained language models. Methods: This comprehensive experimental study evaluated different prompt types (simple prefix, simple cloze, chain of thought, anticipatory, heuristic, and ensemble) across 5 clinical NLP tasks: clinical sense disambiguation, biomedical evidence extraction, coreference resolution, medication status extraction, and medication attribute extraction. The performance of these prompts was assessed using 3 state-of-the-art language models: GPT-3.5 (OpenAI), Gemini (Google), and LLaMA-2 (Meta). The study contrasted zero-shot with few-shot prompting and explored the effectiveness of ensemble approaches. Results: The study revealed that task-specific prompt tailoring is vital for the high performance of LLMs for zero-shot clinical NLP. In clinical sense disambiguation, GPT-3.5 achieved an accuracy of 0.96 with heuristic prompts and 0.94 in biomedical evidence extraction. Heuristic prompts, alongside chain of thought prompts, were highly effective across tasks. Few-shot prompting improved performance in complex scenarios, and ensemble approaches capitalized on multiple prompt strengths. GPT-3.5 consistently outperformed Gemini and LLaMA-2 across tasks and prompt types. Conclusions: This study provides a rigorous evaluation of prompt engineering methodologies and introduces innovative techniques for clinical information extraction, demonstrating the potential of in-context learning in the clinical domain. These findings offer clear guidelines for future prompt-based clinical NLP research, facilitating engagement by non-NLP experts in clinical NLP advancements. To the best of our knowledge, this is one of the first works on the empirical evaluation of different prompt engineering approaches for clinical NLP in this era of generative artificial intelligence, and we hope that it will inspire and inform future research in this area. %M 38587879 %R 10.2196/55318 %U https://medinform.jmir.org/2024/1/e55318 %U https://doi.org/10.2196/55318 %U http://www.ncbi.nlm.nih.gov/pubmed/38587879 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e52935 %T Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study %A Mugaanyi,Joseph %A Cai,Liuying %A Cheng,Sumei %A Lu,Caide %A Huang,Jing %+ Department of Hepato-Pancreato-Biliary Surgery, Ningbo Medical Center Lihuili Hospital, Health Science Center, Ningbo University, No 1111 Jiangnan Road, Ningbo, 315000, China, 86 13819803591, huangjingonline@163.com %K large language models %K accuracy %K academic writing %K AI %K cross-disciplinary evaluation %K scholarly writing %K ChatGPT %K GPT-3.5 %K writing tool %K scholarly %K academic discourse %K LLMs %K machine learning algorithms %K NLP %K natural language processing %K citations %K references %K natural science %K humanities %K chatbot %K artificial intelligence %D 2024 %7 5.4.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022. Objective: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities. Methods: Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines. Results: Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy. Conclusions: ChatGPT’s performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy. %M 38578685 %R 10.2196/52935 %U https://www.jmir.org/2024/1/e52935 %U https://doi.org/10.2196/52935 %U http://www.ncbi.nlm.nih.gov/pubmed/38578685 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e53367 %T Moving Biosurveillance Beyond Coded Data Using AI for Symptom Detection From Physician Notes: Retrospective Cohort Study %A McMurry,Andrew J %A Zipursky,Amy R %A Geva,Alon %A Olson,Karen L %A Jones,James R %A Ignatov,Vladimir %A Miller,Timothy A %A Mandl,Kenneth D %+ Computational Health Informatics Program, Boston Children's Hospital, Landmark 5506 Mail Stop BCH3187, 401 Park Drive, Boston, MA, 02215, United States, 1 6173554145, kenneth_mandl@harvard.edu %K natural language processing %K COVID-19 %K artificial intelligence %K AI %K public health, biosurveillance %K surveillance %K respiratory %K infectious %K pulmonary %K SARS-CoV-2 %K symptom %K symptoms %K detect %K detection %K pipeline %K pipelines %K clinical note %K clinical notes %K documentation %K emergency %K urgent %K pediatric %K pediatrics %K paediatric %K paediatrics %K child %K children %K youth %K adolescent %K adolescents %K teen %K teens %K teenager %K teenagers %K diagnose %K diagnosis %K diagnostic %K diagnostics %D 2024 %7 4.4.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Real-time surveillance of emerging infectious diseases necessitates a dynamically evolving, computable case definition, which frequently incorporates symptom-related criteria. For symptom detection, both population health monitoring platforms and research initiatives primarily depend on structured data extracted from electronic health records. Objective: This study sought to validate and test an artificial intelligence (AI)–based natural language processing (NLP) pipeline for detecting COVID-19 symptoms from physician notes in pediatric patients. We specifically study patients presenting to the emergency department (ED) who can be sentinel cases in an outbreak. Methods: Subjects in this retrospective cohort study are patients who are 21 years of age and younger, who presented to a pediatric ED at a large academic children’s hospital between March 1, 2020, and May 31, 2022. The ED notes for all patients were processed with an NLP pipeline tuned to detect the mention of 11 COVID-19 symptoms based on Centers for Disease Control and Prevention (CDC) criteria. For a gold standard, 3 subject matter experts labeled 226 ED notes and had strong agreement (F1-score=0.986; positive predictive value [PPV]=0.972; and sensitivity=1.0). F1-score, PPV, and sensitivity were used to compare the performance of both NLP and the International Classification of Diseases, 10th Revision (ICD-10) coding to the gold standard chart review. As a formative use case, variations in symptom patterns were measured across SARS-CoV-2 variant eras. Results: There were 85,678 ED encounters during the study period, including 4% (n=3420) with patients with COVID-19. NLP was more accurate at identifying encounters with patients that had any of the COVID-19 symptoms (F1-score=0.796) than ICD-10 codes (F1-score =0.451). NLP accuracy was higher for positive symptoms (sensitivity=0.930) than ICD-10 (sensitivity=0.300). However, ICD-10 accuracy was higher for negative symptoms (specificity=0.994) than NLP (specificity=0.917). Congestion or runny nose showed the highest accuracy difference (NLP: F1-score=0.828 and ICD-10: F1-score=0.042). For encounters with patients with COVID-19, prevalence estimates of each NLP symptom differed across variant eras. Patients with COVID-19 were more likely to have each NLP symptom detected than patients without this disease. Effect sizes (odds ratios) varied across pandemic eras. Conclusions: This study establishes the value of AI-based NLP as a highly effective tool for real-time COVID-19 symptom detection in pediatric patients, outperforming traditional ICD-10 methods. It also reveals the evolving nature of symptom prevalence across different virus variants, underscoring the need for dynamic, technology-driven approaches in infectious disease surveillance. %M 38573752 %R 10.2196/53367 %U https://www.jmir.org/2024/1/e53367 %U https://doi.org/10.2196/53367 %U http://www.ncbi.nlm.nih.gov/pubmed/38573752 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 8 %N %P e52462 %T Automated Category and Trend Analysis of Scientific Articles on Ophthalmology Using Large Language Models: Development and Usability Study %A Raja,Hina %A Munawar,Asim %A Mylonas,Nikolaos %A Delsoz,Mohammad %A Madadi,Yeganeh %A Elahi,Muhammad %A Hassan,Amr %A Abu Serhan,Hashem %A Inam,Onur %A Hernandez,Luis %A Chen,Hao %A Tran,Sang %A Munir,Wuqaas %A Abd-Alrazaq,Alaa %A Yousefi,Siamak %+ Department of Ophthalmology, University of Tennessee Health Science Center, 930 Madison Avenue, Ste. 468, Memphis, TN, 38111, United States, 1 9016595035, hinaraja65@gmail.com %K Bidirectional and Auto-Regressive Transformers %K BART %K bidirectional encoder representations from transformers %K BERT %K ophthalmology %K text classification %K large language model %K LLM %K trend analysis %D 2024 %7 22.3.2024 %9 Original Paper %J JMIR Form Res %G English %X Background: In this paper, we present an automated method for article classification, leveraging the power of large language models (LLMs). Objective: The aim of this study is to evaluate the applicability of various LLMs based on textual content of scientific ophthalmology papers. Methods: We developed a model based on natural language processing techniques, including advanced LLMs, to process and analyze the textual content of scientific papers. Specifically, we used zero-shot learning LLMs and compared Bidirectional and Auto-Regressive Transformers (BART) and its variants with Bidirectional Encoder Representations from Transformers (BERT) and its variants, such as distilBERT, SciBERT, PubmedBERT, and BioBERT. To evaluate the LLMs, we compiled a data set (retinal diseases [RenD] ) of 1000 ocular disease–related articles, which were expertly annotated by a panel of 6 specialists into 19 distinct categories. In addition to the classification of articles, we also performed analysis on different classified groups to find the patterns and trends in the field. Results: The classification results demonstrate the effectiveness of LLMs in categorizing a large number of ophthalmology papers without human intervention. The model achieved a mean accuracy of 0.86 and a mean F1-score of 0.85 based on the RenD data set. Conclusions: The proposed framework achieves notable improvements in both accuracy and efficiency. Its application in the domain of ophthalmology showcases its potential for knowledge organization and retrieval. We performed a trend analysis that enables researchers and clinicians to easily categorize and retrieve relevant papers, saving time and effort in literature review and information gathering as well as identification of emerging scientific trends within different disciplines. Moreover, the extendibility of the model to other scientific fields broadens its impact in facilitating research and trend analysis across diverse disciplines. %M 38517457 %R 10.2196/52462 %U https://formative.jmir.org/2024/1/e52462 %U https://doi.org/10.2196/52462 %U http://www.ncbi.nlm.nih.gov/pubmed/38517457 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 16 %N %P e52845 %T A Semantic Approach to Describe Social and Economic Characteristics That Impact Health Outcomes (Social Determinants of Health): Ontology Development Study %A Dally,Daniela %A Amith,Muhammad %A Mauldin,Rebecca L %A Thomas,Latisha %A Dang,Yifang %A Tao,Cui %+ Department of Artificial Intelligence and Informatics, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL, 32224, United States, 1 904 953 0255, Tao.Cui@mayo.edu %K social determinants of health %K ontology %K semantics %K knowledge representation %D 2024 %7 13.3.2024 %9 Original Paper %J Online J Public Health Inform %G English %X Background: Social determinants of health (SDoH) have been described by the World Health Organization as the conditions in which individuals are born, live, work, and age. These conditions can be grouped into 3 interrelated levels known as macrolevel (societal), mesolevel (community), and microlevel (individual) determinants. The scope of SDoH expands beyond the biomedical level, and there remains a need to connect other areas such as economics, public policy, and social factors. Objective: Providing a computable artifact that can link health data to concepts involving the different levels of determinants may improve our understanding of the impact SDoH have on human populations. Modeling SDoH may help to reduce existing gaps in the literature through explicit links between the determinants and biological factors. This in turn can allow researchers and clinicians to make better sense of data and discover new knowledge through the use of semantic links. Methods: An experimental ontology was developed to represent knowledge of the social and economic characteristics of SDoH. Information from 27 literature sources was analyzed to gather concepts and encoded using Web Ontology Language, version 2 (OWL2) and Protégé. Four evaluators independently reviewed the ontology axioms using natural language translation. The analyses from the evaluations and selected terminologies from the Basic Formal Ontology were used to create a revised ontology with a broad spectrum of knowledge concepts ranging from the macrolevel to the microlevel determinants. Results: The literature search identified several topics of discussion for each determinant level. Publications for the macrolevel determinants centered around health policy, income inequality, welfare, and the environment. Articles relating to the mesolevel determinants discussed work, work conditions, psychosocial factors, socioeconomic position, outcomes, food, poverty, housing, and crime. Finally, sources found for the microlevel determinants examined gender, ethnicity, race, and behavior. Concepts were gathered from the literature and used to produce an ontology consisting of 383 classes, 109 object properties, and 748 logical axioms. A reasoning test revealed no inconsistent axioms. Conclusions: This ontology models heterogeneous social and economic concepts to represent aspects of SDoH. The scope of SDoH is expansive, and although the ontology is broad, it is still in its early stages. To our current understanding, this ontology represents the first attempt to concentrate on knowledge concepts that are currently not covered by existing ontologies. Future direction will include further expanding the ontology to link with other biomedical ontologies, including alignment for granular semantics. %M 38477963 %R 10.2196/52845 %U https://ojphi.jmir.org/2024/1/e52845 %U https://doi.org/10.2196/52845 %U http://www.ncbi.nlm.nih.gov/pubmed/38477963 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 13 %N %P e54593 %T Improving Quality of ICD-10 (International Statistical Classification of Diseases, Tenth Revision) Coding Using AI: Protocol for a Crossover Randomized Controlled Trial %A Chomutare,Taridzo %A Lamproudis,Anastasios %A Budrionis,Andrius %A Svenning,Therese Olsen %A Hind,Lill Irene %A Ngo,Phuong Dinh %A Mikalsen,Karl Øyvind %A Dalianis,Hercules %+ Department of Computer Science, UiT The Arctic University of Norway, Realfagbygget, Hansine Hansens vei 54, Tromsø, 9019, Norway, 47 47680032, taridzo.chomutare@uit.no %K International Classification of Diseases, Tenth Revision %K ICD-10 %K International Classification of Diseases, Eleventh Revision %K ICD-11 %K Easy-ICD %K clinical coding %K artificial intelligence %K machine learning %K deep learning %D 2024 %7 12.3.2024 %9 Protocol %J JMIR Res Protoc %G English %X Background: Computer-assisted clinical coding (CAC) tools are designed to help clinical coders assign standardized codes, such as the ICD-10 (International Statistical Classification of Diseases, Tenth Revision), to clinical texts, such as discharge summaries. Maintaining the integrity of these standardized codes is important both for the functioning of health systems and for ensuring data used for secondary purposes are of high quality. Clinical coding is an error-prone cumbersome task, and the complexity of modern classification systems such as the ICD-11 (International Classification of Diseases, Eleventh Revision) presents significant barriers to implementation. To date, there have only been a few user studies; therefore, our understanding is still limited regarding the role CAC systems can play in reducing the burden of coding and improving the overall quality of coding. Objective: The objective of the user study is to generate both qualitative and quantitative data for measuring the usefulness of a CAC system, Easy-ICD, that was developed for recommending ICD-10 codes. Specifically, our goal is to assess whether our tool can reduce the burden on clinical coders and also improve coding quality. Methods: The user study is based on a crossover randomized controlled trial study design, where we measure the performance of clinical coders when they use our CAC tool versus when they do not. Performance is measured by the time it takes them to assign codes to both simple and complex clinical texts as well as the coding quality, that is, the accuracy of code assignment. Results: We expect the study to provide us with a measurement of the effectiveness of the CAC system compared to manual coding processes, both in terms of time use and coding quality. Positive outcomes from this study will imply that CAC tools hold the potential to reduce the burden on health care staff and will have major implications for the adoption of artificial intelligence–based CAC innovations to improve coding practice. Expected results to be published summer 2024. Conclusions: The planned user study promises a greater understanding of the impact CAC systems might have on clinical coding in real-life settings, especially with regard to coding time and quality. Further, the study may add new insights on how to meaningfully exploit current clinical text mining capabilities, with a view to reducing the burden on clinical coders, thus lowering the barriers and paving a more sustainable path to the adoption of modern coding systems, such as the new ICD-11. Trial Registration: clinicaltrials.gov NCT06286865; https://clinicaltrials.gov/study/NCT06286865 International Registered Report Identifier (IRRID): DERR1-10.2196/54593 %M 38470476 %R 10.2196/54593 %U https://www.researchprotocols.org/2024/1/e54593 %U https://doi.org/10.2196/54593 %U http://www.ncbi.nlm.nih.gov/pubmed/38470476 %0 Journal Article %@ 1929-073X %I JMIR Publications %V 13 %N %P e52296 %T Improvements in Neoplasm Classification in the International Classification of Diseases, Eleventh Revision: Systematic Comparative Study With the Chinese Clinical Modification of the International Classification of Diseases, Tenth Revision %A Xu,Yicong %A Zhou,Jingya %A Li,Hongxia %A Cai,Dong %A Zhu,Huanbing %A Pan,Shengdong %+ Department of Medical Administration, The Second Affiliated Hospital, Zhejiang University School of Medicine, 88 Jiefang Road, Hangzhou, 310009, China, 86 1 366 666 9123, psd618@zju.edu.cn %K Chinese Clinical Modification of the International Classification of Diseases, Tenth Revision %K ICD-10 %K ICD-10-CCM %K ICD-11 %K improvement %K International Classification of Diseases, Eleventh Revision %K International Classification of Diseases, Tenth Revision %K International Classification of Diseases %K neoplasm %K transition %D 2024 %7 8.3.2024 %9 Original Paper %J Interact J Med Res %G English %X Background: The International Classification of Diseases, Eleventh Revision (ICD-11) improved neoplasm classification. Objective: We aimed to study the alterations in the ICD-11 compared to the Chinese Clinical Modification of the International Classification of Diseases, Tenth Revision (ICD-10-CCM) for neoplasm classification and to provide evidence supporting the transition to the ICD-11. Methods: We downloaded public data files from the World Health Organization and the National Health Commission of the People’s Republic of China. The ICD-10-CCM neoplasm codes were manually recoded with the ICD-11 coding tool, and an ICD-10-CCM/ICD-11 mapping table was generated. The existing files and the ICD-10-CCM/ICD-11 mapping table were used to compare the coding, classification, and expression features of neoplasms between the ICD-10-CCM and ICD-11. Results: The ICD-11 coding structure for neoplasms has dramatically changed. It provides advantages in coding granularity, coding capacity, and expression flexibility. In total, 27.4% (207/755) of ICD-10 codes and 38% (1359/3576) of ICD-10-CCM codes underwent grouping changes, which was a significantly different change (χ21=30.3; P<.001). Notably, 67.8% (2424/3576) of ICD-10-CCM codes could be fully represented by ICD-11 codes. Another 7% (252/3576) could be fully described by uniform resource identifiers. The ICD-11 had a significant difference in expression ability among the 4 ICD-10-CCM groups (χ23=93.7; P<.001), as well as a considerable difference between the changed and unchanged groups (χ21=74.7; P<.001). Expression ability negatively correlated with grouping changes (r=–.144; P<.001). In the ICD-10-CCM/ICD-11 mapping table, 60.5% (2164/3576) of codes were postcoordinated. The top 3 postcoordinated results were specific anatomy (1907/3576, 53.3%), histopathology (201/3576, 5.6%), and alternative severity 2 (70/3576, 2%). The expression ability of postcoordination was not fully reflected. Conclusions: The ICD-11 includes many improvements in neoplasm classification, especially the new coding system, improved expression ability, and good semantic interoperability. The transition to the ICD-11 will inevitably bring challenges for clinicians, coders, policy makers and IT technicians, and many preparations will be necessary. %M 38457228 %R 10.2196/52296 %U https://www.i-jmr.org/2024/1/e52296 %U https://doi.org/10.2196/52296 %U http://www.ncbi.nlm.nih.gov/pubmed/38457228 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e51560 %T Frameworks, Dimensions, Definitions of Aspects, and Assessment Methods for the Appraisal of Quality of Health Data for Secondary Use: Comprehensive Overview of Reviews %A Declerck,Jens %A Kalra,Dipak %A Vander Stichele,Robert %A Coorevits,Pascal %+ Department of Public Health and Primary Care, Unit of Medical Informatics and Statistics, Ghent University, Campus UZ-Ghent, Entrance 42, 6th Floor, Corneel Heymanslaan 10, Ghent, 9000, Belgium, 32 93323628, jens.declerck@ugent.be %K data quality %K data quality dimensions %K data quality assessment %K secondary use %K data quality framework %K fit for purpose %D 2024 %7 6.3.2024 %9 Review %J JMIR Med Inform %G English %X Background: Health care has not reached the full potential of the secondary use of health data because of—among other issues—concerns about the quality of the data being used. The shift toward digital health has led to an increase in the volume of health data. However, this increase in quantity has not been matched by a proportional improvement in the quality of health data. Objective: This review aims to offer a comprehensive overview of the existing frameworks for data quality dimensions and assessment methods for the secondary use of health data. In addition, it aims to consolidate the results into a unified framework. Methods: A review of reviews was conducted including reviews describing frameworks of data quality dimensions and their assessment methods, specifically from a secondary use perspective. Reviews were excluded if they were not related to the health care ecosystem, lacked relevant information related to our research objective, and were published in languages other than English. Results: A total of 22 reviews were included, comprising 22 frameworks, with 23 different terms for dimensions, and 62 definitions of dimensions. All dimensions were mapped toward the data quality framework of the European Institute for Innovation through Health Data. In total, 8 reviews mentioned 38 different assessment methods, pertaining to 31 definitions of the dimensions. Conclusions: The findings in this review revealed a lack of consensus in the literature regarding the terminology, definitions, and assessment methods for data quality dimensions. This creates ambiguity and difficulties in developing specific assessment methods. This study goes a step further by assigning all observed definitions to a consolidated framework of 9 data quality dimensions. %M 38446534 %R 10.2196/51560 %U https://medinform.jmir.org/2024/1/e51560 %U https://doi.org/10.2196/51560 %U http://www.ncbi.nlm.nih.gov/pubmed/38446534 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 8 %N %P e49411 %T Machine Learning–Based Approach for Identifying Research Gaps: COVID-19 as a Case Study %A Abd-alrazaq,Alaa %A Nashwan,Abdulqadir J %A Shah,Zubair %A Abujaber,Ahmad %A Alhuwail,Dari %A Schneider,Jens %A AlSaad,Rawan %A Ali,Hazrat %A Alomoush,Waleed %A Ahmed,Arfan %A Aziz,Sarah %+ AI Center for Precision Health, Weill Cornell Medicine-Qatar, A031, Weill Cornell Medicine-Qatar, Education City, Al Luqta St, Doha, 23435, Qatar, 974 55708599, aaa4027@qatar-med.cornell.edu %K research gaps %K research gap %K research topic %K research topics %K scientific literature %K literature review %K machine learning %K COVID-19 %K BERTopic %K topic clustering %K text analysis %K BERT %K NLP %K natural language processing %K review methods %K review methodology %K SARS-CoV-2 %K coronavirus %K COVID %D 2024 %7 5.3.2024 %9 Original Paper %J JMIR Form Res %G English %X Background: Research gaps refer to unanswered questions in the existing body of knowledge, either due to a lack of studies or inconclusive results. Research gaps are essential starting points and motivation in scientific research. Traditional methods for identifying research gaps, such as literature reviews and expert opinions, can be time consuming, labor intensive, and prone to bias. They may also fall short when dealing with rapidly evolving or time-sensitive subjects. Thus, innovative scalable approaches are needed to identify research gaps, systematically assess the literature, and prioritize areas for further study in the topic of interest. Objective: In this paper, we propose a machine learning–based approach for identifying research gaps through the analysis of scientific literature. We used the COVID-19 pandemic as a case study. Methods: We conducted an analysis to identify research gaps in COVID-19 literature using the COVID-19 Open Research (CORD-19) data set, which comprises 1,121,433 papers related to the COVID-19 pandemic. Our approach is based on the BERTopic topic modeling technique, which leverages transformers and class-based term frequency-inverse document frequency to create dense clusters allowing for easily interpretable topics. Our BERTopic-based approach involves 3 stages: embedding documents, clustering documents (dimension reduction and clustering), and representing topics (generating candidates and maximizing candidate relevance). Results: After applying the study selection criteria, we included 33,206 abstracts in the analysis of this study. The final list of research gaps identified 21 different areas, which were grouped into 6 principal topics. These topics were: “virus of COVID-19,” “risk factors of COVID-19,” “prevention of COVID-19,” “treatment of COVID-19,” “health care delivery during COVID-19,” “and impact of COVID-19.” The most prominent topic, observed in over half of the analyzed studies, was “the impact of COVID-19.” Conclusions: The proposed machine learning–based approach has the potential to identify research gaps in scientific literature. This study is not intended to replace individual literature research within a selected topic. Instead, it can serve as a guide to formulate precise literature search queries in specific areas associated with research questions that previous publications have earmarked for future exploration. Future research should leverage an up-to-date list of studies that are retrieved from the most common databases in the target area. When feasible, full texts or, at minimum, discussion sections should be analyzed rather than limiting their analysis to abstracts. Furthermore, future studies could evaluate more efficient modeling algorithms, especially those combining topic modeling with statistical uncertainty quantification, such as conformal prediction. %M 38441952 %R 10.2196/49411 %U https://formative.jmir.org/2024/1/e49411 %U https://doi.org/10.2196/49411 %U http://www.ncbi.nlm.nih.gov/pubmed/38441952 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e49139 %T Use of Large Language Models to Assess the Likelihood of Epidemics From the Content of Tweets: Infodemiology Study %A Deiner,Michael S %A Deiner,Natalie A %A Hristidis,Vagelis %A McLeod,Stephen D %A Doan,Thuy %A Lietman,Thomas M %A Porco,Travis C %+ Francis I. Proctor Foundation for Research in Ophthalmology, University of California, San Francisco, 490 Illinois St., Box 0944, San Francisco, CA, 94143-0944, United States, 1 415 476 0527, travis.porco@ucsf.edu %K conjunctivitis %K microblog %K social media %K generative large language model %K Generative Pre-trained Transformers %K GPT-3.5 %K GPT-4 %K epidemic detection %K Twitter %K X formerly known as Twitter %K infectious eye disease %D 2024 %7 1.3.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Previous work suggests that Google searches could be useful in identifying conjunctivitis epidemics. Content-based assessment of social media content may provide additional value in serving as early indicators of conjunctivitis and other systemic infectious diseases. Objective: We investigated whether large language models, specifically GPT-3.5 and GPT-4 (OpenAI), can provide probabilistic assessments of whether social media posts about conjunctivitis could indicate a regional outbreak. Methods: A total of 12,194 conjunctivitis-related tweets were obtained using a targeted Boolean search in multiple languages from India, Guam (United States), Martinique (France), the Philippines, American Samoa (United States), Fiji, Costa Rica, Haiti, and the Bahamas, covering the time frame from January 1, 2012, to March 13, 2023. By providing these tweets via prompts to GPT-3.5 and GPT-4, we obtained probabilistic assessments that were validated by 2 human raters. We then calculated Pearson correlations of these time series with tweet volume and the occurrence of known outbreaks in these 9 locations, with time series bootstrap used to compute CIs. Results: Probabilistic assessments derived from GPT-3.5 showed correlations of 0.60 (95% CI 0.47-0.70) and 0.53 (95% CI 0.40-0.65) with the 2 human raters, with higher results for GPT-4. The weekly averages of GPT-3.5 probabilities showed substantial correlations with weekly tweet volume for 44% (4/9) of the countries, with correlations ranging from 0.10 (95% CI 0.0-0.29) to 0.53 (95% CI 0.39-0.89), with larger correlations for GPT-4. More modest correlations were found for correlation with known epidemics, with substantial correlation only in American Samoa (0.40, 95% CI 0.16-0.81). Conclusions: These findings suggest that GPT prompting can efficiently assess the content of social media posts and indicate possible disease outbreaks to a degree of accuracy comparable to that of humans. Furthermore, we found that automated content analysis of tweets is related to tweet volume for conjunctivitis-related posts in some locations and to the occurrence of actual epidemics. Future work may improve the sensitivity and specificity of these methods for disease outbreak detection. %M 38427404 %R 10.2196/49139 %U https://www.jmir.org/2024/1/e49139 %U https://doi.org/10.2196/49139 %U http://www.ncbi.nlm.nih.gov/pubmed/38427404 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e47739 %T Identifying Functional Status Impairment in People Living With Dementia Through Natural Language Processing of Clinical Documents: Cross-Sectional Study %A Laurentiev,John %A Kim,Dae Hyun %A Mahesri,Mufaddal %A Wang,Kuan-Yuan %A Bessette,Lily G %A York,Cassandra %A Zakoul,Heidi %A Lee,Su Been %A Zhou,Li %A Lin,Kueiyu Joshua %+ Department of Medicine, Brigham and Women's Hospital, 1620 Tremont St. Suite 3030, Boston, MA, 02120, United States, 1 617 278 0930, jklin@bwh.harvard.edu %K activities of daily living %K ADLs %K clinical note %K dementia %K electronic health record %K EHR %K functional impairment %K instrumental activities of daily living %K iADLs %K machine learning %K natural language processing %K NLP %D 2024 %7 13.2.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Assessment of activities of daily living (ADLs) and instrumental ADLs (iADLs) is key to determining the severity of dementia and care needs among older adults. However, such information is often only documented in free-text clinical notes within the electronic health record and can be challenging to find. Objective: This study aims to develop and validate machine learning models to determine the status of ADL and iADL impairments based on clinical notes. Methods: This cross-sectional study leveraged electronic health record clinical notes from Mass General Brigham’s Research Patient Data Repository linked with Medicare fee-for-service claims data from 2007 to 2017 to identify individuals aged 65 years or older with at least 1 diagnosis of dementia. Notes for encounters both 180 days before and after the first date of dementia diagnosis were randomly sampled. Models were trained and validated using note sentences filtered by expert-curated keywords (filtered cohort) and further evaluated using unfiltered sentences (unfiltered cohort). The model’s performance was compared using area under the receiver operating characteristic curve and area under the precision-recall curve (AUPRC). Results: The study included 10,000 key-term–filtered sentences representing 441 people (n=283, 64.2% women; mean age 82.7, SD 7.9 years) and 1000 unfiltered sentences representing 80 people (n=56, 70% women; mean age 82.8, SD 7.5 years). Area under the receiver operating characteristic curve was high for the best-performing ADL and iADL models on both cohorts (>0.97). For ADL impairment identification, the random forest model achieved the best AUPRC (0.89, 95% CI 0.86-0.91) on the filtered cohort; the support vector machine model achieved the highest AUPRC (0.82, 95% CI 0.75-0.89) for the unfiltered cohort. For iADL impairment, the Bio+Clinical bidirectional encoder representations from transformers (BERT) model had the highest AUPRC (filtered: 0.76, 95% CI 0.68-0.82; unfiltered: 0.58, 95% CI 0.001-1.0). Compared with a keyword-search approach on the unfiltered cohort, machine learning reduced false-positive rates from 4.5% to 0.2% for ADL and 1.8% to 0.1% for iADL. Conclusions: In this study, we demonstrated the ability of machine learning models to accurately identify ADL and iADL impairment based on free-text clinical notes, which could be useful in determining the severity of dementia. %M 38349732 %R 10.2196/47739 %U https://www.jmir.org/2024/1/e47739 %U https://doi.org/10.2196/47739 %U http://www.ncbi.nlm.nih.gov/pubmed/38349732 %0 Journal Article %@ 2369-3762 %I JMIR Publications %V 10 %N %P e48514 %T Performance of ChatGPT on the Chinese Postgraduate Examination for Clinical Medicine: Survey Study %A Yu,Peng %A Fang,Changchang %A Liu,Xiaolin %A Fu,Wanying %A Ling,Jitao %A Yan,Zhiwei %A Jiang,Yuan %A Cao,Zhengyu %A Wu,Maoxiong %A Chen,Zhiteng %A Zhu,Wengen %A Zhang,Yuling %A Abudukeremu,Ayiguli %A Wang,Yue %A Liu,Xiao %A Wang,Jingfeng %+ Department of Cardiology, Sun Yat-sen Memorial Hospital of Sun Yat-sen University, 107 Yanjiang West Road, Guangzhou, China, 86 15083827378, liux587@mail.sysu.edu.cn %K ChatGPT %K Chinese Postgraduate Examination for Clinical Medicine %K medical student %K performance %K artificial intelligence %K medical care %K qualitative feedback %K medical education %K clinical decision-making %D 2024 %7 9.2.2024 %9 Original Paper %J JMIR Med Educ %G English %X Background: ChatGPT, an artificial intelligence (AI) based on large-scale language models, has sparked interest in the field of health care. Nonetheless, the capabilities of AI in text comprehension and generation are constrained by the quality and volume of available training data for a specific language, and the performance of AI across different languages requires further investigation. While AI harbors substantial potential in medicine, it is imperative to tackle challenges such as the formulation of clinical care standards; facilitating cultural transitions in medical education and practice; and managing ethical issues including data privacy, consent, and bias. Objective: The study aimed to evaluate ChatGPT’s performance in processing Chinese Postgraduate Examination for Clinical Medicine questions, assess its clinical reasoning ability, investigate potential limitations with the Chinese language, and explore its potential as a valuable tool for medical professionals in the Chinese context. Methods: A data set of Chinese Postgraduate Examination for Clinical Medicine questions was used to assess the effectiveness of ChatGPT’s (version 3.5) medical knowledge in the Chinese language, which has a data set of 165 medical questions that were divided into three categories: (1) common questions (n=90) assessing basic medical knowledge, (2) case analysis questions (n=45) focusing on clinical decision-making through patient case evaluations, and (3) multichoice questions (n=30) requiring the selection of multiple correct answers. First of all, we assessed whether ChatGPT could meet the stringent cutoff score defined by the government agency, which requires a performance within the top 20% of candidates. Additionally, in our evaluation of ChatGPT’s performance on both original and encoded medical questions, 3 primary indicators were used: accuracy, concordance (which validates the answer), and the frequency of insights. Results: Our evaluation revealed that ChatGPT scored 153.5 out of 300 for original questions in Chinese, which signifies the minimum score set to ensure that at least 20% more candidates pass than the enrollment quota. However, ChatGPT had low accuracy in answering open-ended medical questions, with only 31.5% total accuracy. The accuracy for common questions, multichoice questions, and case analysis questions was 42%, 37%, and 17%, respectively. ChatGPT achieved a 90% concordance across all questions. Among correct responses, the concordance was 100%, significantly exceeding that of incorrect responses (n=57, 50%; P<.001). ChatGPT provided innovative insights for 80% (n=132) of all questions, with an average of 2.95 insights per accurate response. Conclusions: Although ChatGPT surpassed the passing threshold for the Chinese Postgraduate Examination for Clinical Medicine, its performance in answering open-ended medical questions was suboptimal. Nonetheless, ChatGPT exhibited high internal concordance and the ability to generate multiple insights in the Chinese language. Future research should investigate the language-based discrepancies in ChatGPT’s performance within the health care context. %M 38335017 %R 10.2196/48514 %U https://mededu.jmir.org/2024/1/e48514 %U https://doi.org/10.2196/48514 %U http://www.ncbi.nlm.nih.gov/pubmed/38335017 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 8 %N %P e53216 %T Investigating the Impact of Prompt Engineering on the Performance of Large Language Models for Standardizing Obstetric Diagnosis Text: Comparative Study %A Wang,Lei %A Bi,Wenshuai %A Zhao,Suling %A Ma,Yinyao %A Lv,Longting %A Meng,Chenwei %A Fu,Jingru %A Lv,Hanlin %+ BGI Research, Building 11, Beishan Industrial Zone, Yantian District, Shenzhen, 518083, China, 86 18707190886, lvhanlin@genomics.cn %K obstetric data %K similarity embedding %K term standardization %K large language models %K LLMs %D 2024 %7 8.2.2024 %9 Original Paper %J JMIR Form Res %G English %X Background: The accumulation of vast electronic medical records (EMRs) through medical informatization creates significant research value, particularly in obstetrics. Diagnostic standardization across different health care institutions and regions is vital for medical data analysis. Large language models (LLMs) have been extensively used for various medical tasks. Prompt engineering is key to use LLMs effectively. Objective: This study aims to evaluate and compare the performance of LLMs with various prompt engineering techniques on the task of standardizing obstetric diagnostic terminology using real-world obstetric data. Methods: The paper describes a 4-step approach used for mapping diagnoses in electronic medical records to the International Classification of Diseases, 10th revision, observation domain. First, similarity measures were used for mapping the diagnoses. Second, candidate mapping terms were collected based on similarity scores above a threshold, to be used as the training data set. For generating optimal mapping terms, we used two LLMs (ChatGLM2 and Qwen-14B-Chat [QWEN]) for zero-shot learning in step 3. Finally, a performance comparison was conducted by using 3 pretrained bidirectional encoder representations from transformers (BERTs), including BERT, whole word masking BERT, and momentum contrastive learning with BERT (MC-BERT), for unsupervised optimal mapping term generation in the fourth step. Results: LLMs and BERT demonstrated comparable performance at their respective optimal levels. LLMs showed clear advantages in terms of performance and efficiency in unsupervised settings. Interestingly, the performance of the LLMs varied significantly across different prompt engineering setups. For instance, when applying the self-consistency approach in QWEN, the F1-score improved by 5%, with precision increasing by 7.9%, outperforming the zero-shot method. Likewise, ChatGLM2 delivered similar rates of accurately generated responses. During the analysis, the BERT series served as a comparative model with comparable results. Among the 3 models, MC-BERT demonstrated the highest level of performance. However, the differences among the versions of BERT in this study were relatively insignificant. Conclusions: After applying LLMs to standardize diagnoses and designing 4 different prompts, we compared the results to those generated by the BERT model. Our findings indicate that QWEN prompts largely outperformed the other prompts, with precision comparable to that of the BERT model. These results demonstrate the potential of unsupervised approaches in improving the efficiency of aligning diagnostic terms in daily research and uncovering hidden information values in patient data. %M 38329787 %R 10.2196/53216 %U https://formative.jmir.org/2024/1/e53216 %U https://doi.org/10.2196/53216 %U http://www.ncbi.nlm.nih.gov/pubmed/38329787 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 8 %N %P e32690 %T Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study %A Ji,Jia %A Hou,Yongshuai %A Chen,Xinyu %A Pan,Youcheng %A Xiang,Yang %+ Peng Cheng Laboratory, No. 2 Xingke 1st Street, Shenzhen, 518000, China, 86 18566668732, panyoucheng4@gmail.com %K clinical image %K radiology report generation %K vision-language model %K multistage fine-tuning %K prior knowledge %D 2024 %7 8.2.2024 %9 Original Paper %J JMIR Form Res %G English %X Background: The automatic generation of radiology reports, which seeks to create a free-text description from a clinical radiograph, is emerging as a pivotal intersection between clinical medicine and artificial intelligence. Leveraging natural language processing technologies can accelerate report creation, enhancing health care quality and standardization. However, most existing studies have not yet fully tapped into the combined potential of advanced language and vision models. Objective: The purpose of this study was to explore the integration of pretrained vision-language models into radiology report generation. This would enable the vision-language model to automatically convert clinical images into high-quality textual reports. Methods: In our research, we introduced a radiology report generation model named ClinicalBLIP, building upon the foundational InstructBLIP model and refining it using clinical image-to-text data sets. A multistage fine-tuning approach via low-rank adaptation was proposed to deepen the semantic comprehension of the visual encoder and the large language model for clinical imagery. Furthermore, prior knowledge was integrated through prompt learning to enhance the precision of the reports generated. Experiments were conducted on both the IU X-RAY and MIMIC-CXR data sets, with ClinicalBLIP compared to several leading methods. Results: Experimental results revealed that ClinicalBLIP obtained superior scores of 0.570/0.365 and 0.534/0.313 on the IU X-RAY/MIMIC-CXR test sets for the Metric for Evaluation of Translation with Explicit Ordering (METEOR) and the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) evaluations, respectively. This performance notably surpasses that of existing state-of-the-art methods. Further evaluations confirmed the effectiveness of the multistage fine-tuning and the integration of prior information, leading to substantial improvements. Conclusions: The proposed ClinicalBLIP model demonstrated robustness and effectiveness in enhancing clinical radiology report generation, suggesting significant promise for real-world clinical applications. %M 38329788 %R 10.2196/32690 %U https://formative.jmir.org/2024/1/e32690 %U https://doi.org/10.2196/32690 %U http://www.ncbi.nlm.nih.gov/pubmed/38329788 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e53516 %T Using a Natural Language Processing Approach to Support Rapid Knowledge Acquisition %A Koonce,Taneya Y %A Giuse,Dario A %A Williams,Annette M %A Blasingame,Mallory N %A Krump,Poppy A %A Su,Jing %A Giuse,Nunzia B %+ Center for Knowledge Management, Vanderbilt University Medical Center, 3401 West End, Suite 304, Nashville, TN, 37203, United States, 1 6159365790, taneya.koonce@vumc.org %K natural language processing %K electronic health records %K machine learning %K data mining %K knowledge management %K NLP %D 2024 %7 30.1.2024 %9 Viewpoint %J JMIR Med Inform %G English %X Implementing artificial intelligence to extract insights from large, real-world clinical data sets can supplement and enhance knowledge management efforts for health sciences research and clinical care. At Vanderbilt University Medical Center (VUMC), the in-house developed Word Cloud natural language processing system extracts coded concepts from patient records in VUMC’s electronic health record repository using the Unified Medical Language System terminology. Through this process, the Word Cloud extracts the most prominent concepts found in the clinical documentation of a specific patient or population. The Word Cloud provides added value for clinical care decision-making and research. This viewpoint paper describes a use case for how the VUMC Center for Knowledge Management leverages the condition-disease associations represented by the Word Cloud to aid in the knowledge generation needed to inform the interpretation of phenome-wide association studies. %M 38289670 %R 10.2196/53516 %U https://medinform.jmir.org/2024/1/e53516 %U https://doi.org/10.2196/53516 %U http://www.ncbi.nlm.nih.gov/pubmed/38289670 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e48443 %T Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study %A Lee,You-Qian %A Chen,Ching-Tai %A Chen,Chien-Chang %A Lee,Chung-Hong %A Chen,Peitsz %A Wu,Chi-Shin %A Dai,Hong-Jie %+ Intelligent System Laboratory, Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Kaohsiung University of Science and Technology, No. 415, Jiangong Road, Sanmin District, Kaohsiung, 80778, Taiwan, 886 73814526 ext 15510, hjdai@nkust.edu.tw %K code mixing %K electronic health record %K deidentification %K pretrained language model %K large language model %K ChatGPT %D 2024 %7 25.1.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: The widespread use of electronic health records in the clinical and biomedical fields makes the removal of protected health information (PHI) essential to maintain privacy. However, a significant portion of information is recorded in unstructured textual forms, posing a challenge for deidentification. In multilingual countries, medical records could be written in a mixture of more than one language, referred to as code mixing. Most current clinical natural language processing techniques are designed for monolingual text, and there is a need to address the deidentification of code-mixed text. Objective: The aim of this study was to investigate the effectiveness and underlying mechanism of fine-tuned pretrained language models (PLMs) in identifying PHI in the code-mixed context. Additionally, we aimed to evaluate the potential of prompting large language models (LLMs) for recognizing PHI in a zero-shot manner. Methods: We compiled the first clinical code-mixed deidentification data set consisting of text written in Chinese and English. We explored the effectiveness of fine-tuned PLMs for recognizing PHI in code-mixed content, with a focus on whether PLMs exploit naming regularity and mention coverage to achieve superior performance, by probing the developed models’ outputs to examine their decision-making process. Furthermore, we investigated the potential of prompt-based in-context learning of LLMs for recognizing PHI in code-mixed text. Results: The developed methods were evaluated on a code-mixed deidentification corpus of 1700 discharge summaries. We observed that different PHI types had preferences in their occurrences within the different types of language-mixed sentences, and PLMs could effectively recognize PHI by exploiting the learned name regularity. However, the models may exhibit suboptimal results when regularity is weak or mentions contain unknown words that the representations cannot generate well. We also found that the availability of code-mixed training instances is essential for the model’s performance. Furthermore, the LLM-based deidentification method was a feasible and appealing approach that can be controlled and enhanced through natural language prompts. Conclusions: The study contributes to understanding the underlying mechanism of PLMs in addressing the deidentification process in the code-mixed context and highlights the significance of incorporating code-mixed training instances into the model training phase. To support the advancement of research, we created a manipulated subset of the resynthesized data set available for research purposes. Based on the compiled data set, we found that the LLM-based deidentification method is a feasible approach, but carefully crafted prompts are essential to avoid unwanted output. However, the use of such methods in the hospital setting requires careful consideration of data security and privacy concerns. Further research could explore the augmentation of PLMs and LLMs with external knowledge to improve their strength in recognizing rare PHI. %M 38271060 %R 10.2196/48443 %U https://www.jmir.org/2024/1/e48443 %U https://doi.org/10.2196/48443 %U http://www.ncbi.nlm.nih.gov/pubmed/38271060 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e51501 %T Text Dialogue Analysis for Primary Screening of Mild Cognitive Impairment: Development and Validation Study %A Wang,Changyu %A Liu,Siru %A Li,Aiqing %A Liu,Jialin %+ Information Center, West China Hospital, Sichuan University, No. 37 Guo Xue Xiang28 85422306, Chengdu, 610041, China, 86 28 85422306, DLJL8@163.com %K artificial intelligence %K AI %K AI models %K ChatGPT %K primary screening %K mild cognitive impairment %K standardization %K prompt design %K design %K artificial intelligence %K cognitive impairment %K screening %K model %K clinician %K diagnosis %D 2023 %7 29.12.2023 %9 Original Paper %J J Med Internet Res %G English %X Background: Artificial intelligence models tailored to diagnose cognitive impairment have shown excellent results. However, it is unclear whether large linguistic models can rival specialized models by text alone. Objective: In this study, we explored the performance of ChatGPT for primary screening of mild cognitive impairment (MCI) and standardized the design steps and components of the prompts. Methods: We gathered a total of 174 participants from the DementiaBank screening and classified 70% of them into the training set and 30% of them into the test set. Only text dialogues were kept. Sentences were cleaned using a macro code, followed by a manual check. The prompt consisted of 5 main parts, including character setting, scoring system setting, indicator setting, output setting, and explanatory information setting. Three dimensions of variables from published studies were included: vocabulary (ie, word frequency and word ratio, phrase frequency and phrase ratio, and lexical complexity), syntax and grammar (ie, syntactic complexity and grammatical components), and semantics (ie, semantic density and semantic coherence). We used R 4.3.0. for the analysis of variables and diagnostic indicators. Results: Three additional indicators related to the severity of MCI were incorporated into the final prompt for the model. These indicators were effective in discriminating between MCI and cognitively normal participants: tip-of-the-tongue phenomenon (P<.001), difficulty with complex ideas (P<.001), and memory issues (P<.001). The final GPT-4 model achieved a sensitivity of 0.8636, a specificity of 0.9487, and an area under the curve of 0.9062 on the training set; on the test set, the sensitivity, specificity, and area under the curve reached 0.7727, 0.8333, and 0.8030, respectively. Conclusions: ChatGPT was effective in the primary screening of participants with possible MCI. Improved standardization of prompts by clinicians would also improve the performance of the model. It is important to note that ChatGPT is not a substitute for a clinician making a diagnosis. %M 38157230 %R 10.2196/51501 %U https://www.jmir.org/2023/1/e51501 %U https://doi.org/10.2196/51501 %U http://www.ncbi.nlm.nih.gov/pubmed/38157230 %0 Journal Article %@ 2369-3762 %I JMIR Publications %V 9 %N %P e48904 %T Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study %A Liao,Wenxiong %A Liu,Zhengliang %A Dai,Haixing %A Xu,Shaochen %A Wu,Zihao %A Zhang,Yiyang %A Huang,Xiaoke %A Zhu,Dajiang %A Cai,Hongmin %A Li,Quanzheng %A Liu,Tianming %A Li,Xiang %+ Department of Radiology, Massachusetts General Hospital, 55 Fruit St, Boston, MA, 02114, United States, 1 7062480264, xli60@mgh.harvard.edu %K ChatGPT %K medical ethics %K linguistic analysis %K text classification %K artificial intelligence %K medical texts %K machine learning %D 2023 %7 28.12.2023 %9 Original Paper %J JMIR Med Educ %G English %X Background: Large language models, such as ChatGPT, are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the internet. However, medical texts, such as clinical notes and diagnoses, require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to health care and the general public. Objective: This study is among the first on responsible artificial intelligence–generated content in medicine. We focus on analyzing the differences between medical texts written by human experts and those generated by ChatGPT and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods: We first constructed a suite of data sets containing medical texts written by human experts and generated by ChatGPT. We analyzed the linguistic features of these 2 types of content and uncovered differences in vocabulary, parts-of-speech, dependency, sentiment, perplexity, and other aspects. Finally, we designed and implemented machine learning methods to detect medical text generated by ChatGPT. The data and code used in this paper are published on GitHub. Results: Medical texts written by humans were more concrete, more diverse, and typically contained more useful information, while medical texts generated by ChatGPT paid more attention to fluency and logic and usually expressed general terminologies rather than effective information specific to the context of the problem. A bidirectional encoder representations from transformers–based model effectively detected medical texts generated by ChatGPT, and the F1 score exceeded 95%. Conclusions: Although text generated by ChatGPT is grammatically perfect and human-like, the linguistic characteristics of generated medical texts were different from those written by human experts. Medical text generated by ChatGPT could be effectively detected by the proposed machine learning algorithms. This study provides a pathway toward trustworthy and accountable use of large language models in medicine. %M 38153785 %R 10.2196/48904 %U https://mededu.jmir.org/2023/1/e48904 %U https://doi.org/10.2196/48904 %U http://www.ncbi.nlm.nih.gov/pubmed/38153785 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e50017 %T Extending cBioPortal for Therapy Recommendation Documentation in Molecular Tumor Boards: Development and Usability Study %A Renner,Christopher %A Reimer,Niklas %A Christoph,Jan %A Busch,Hauke %A Metzger,Patrick %A Boerries,Melanie %A Ustjanzew,Arsenij %A Boehm,Dominik %A Unberath,Philipp %+ Chair of Medical Informatics, Friedrich-Alexander University Erlangen-Nuremberg, Wetterkreuz 15, Erlangen, 91058, Germany, 49 1733735424, philipp.unberath@fau.de %K molecular tumor board %K documentation platform %K usability evaluation %K cBioPortal %K precision medicine %K genomics %K health information interoperability %K tumor %K implementation %K cancer %K tool %K platform %K development %K precision %K use %K user-centered %D 2023 %7 11.12.2023 %9 Original Paper %J JMIR Med Inform %G English %X Background: In molecular tumor boards (MTBs), patients with rare or advanced cancers are discussed by a multidisciplinary team of health care professionals. Software support for MTBs is lacking; in particular, tools for preparing and documenting MTB therapy recommendations need to be developed. Objective: We aimed to implement an extension to cBioPortal to provide a tool for the documentation of therapy recommendations from MTB sessions in a secure and standardized manner. The developed extension should be embedded in the patient view of cBioPortal to enable easy documentation during MTB sessions. The resulting architecture for storing therapy recommendations should be integrable into various hospital information systems. Methods: On the basis of a requirements analysis and technology analysis for authentication techniques, a prototype was developed and iteratively refined through a user-centered development process. In conclusion, the tool was evaluated via a usability evaluation, including interviews, structured questionnaires, and the System Usability Scale. Results: The patient view of cBioPortal was extended with a new tab that enables users to document MTB sessions and therapy recommendations. The role-based access control was expanded to allow for a finer distinction among the rights to view, edit, and delete data. The usability evaluation showed overall good usability and a System Usability Scale score of 83.57. Conclusions: This study demonstrates how cBioPortal can be extended to not only visualize MTB patient data but also be used as a documentation platform for therapy recommendations. %M 38079196 %R 10.2196/50017 %U https://medinform.jmir.org/2023/1/e50017 %U https://doi.org/10.2196/50017 %U http://www.ncbi.nlm.nih.gov/pubmed/38079196 %0 Journal Article %@ 2369-3762 %I JMIR Publications %V 9 %N %P e49183 %T ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions %A Buhr,Christoph Raphael %A Smith,Harry %A Huppertz,Tilman %A Bahr-Hamm,Katharina %A Matthias,Christoph %A Blaikie,Andrew %A Kelsey,Tom %A Kuhn,Sebastian %A Eckrich,Jonas %+ Department of Otorhinolaryngology, University Medical Center of the Johannes Gutenberg-University Mainz, Langenbeckstraße 1, Mainz, 55131, Germany, 49 6131 17 7361, buhrchri@uni-mainz.de %K large language models %K LLMs %K LLM %K artificial intelligence %K AI %K ChatGPT %K otorhinolaryngology %K ORL %K digital health %K chatbots %K global health %K low- and middle-income countries %K telemedicine %K telehealth %K language model %K chatbot %D 2023 %7 5.12.2023 %9 Original Paper %J JMIR Med Educ %G English %X Background: Large language models (LLMs), such as ChatGPT (Open AI), are increasingly used in medicine and supplement standard search engines as information sources. This leads to more “consultations” of LLMs about personal medical symptoms. Objective: This study aims to evaluate ChatGPT’s performance in answering clinical case–based questions in otorhinolaryngology (ORL) in comparison to ORL consultants’ answers. Methods: We used 41 case-based questions from established ORL study books and past German state examinations for doctors. The questions were answered by both ORL consultants and ChatGPT 3. ORL consultants rated all responses, except their own, on medical adequacy, conciseness, coherence, and comprehensibility using a 6-point Likert scale. They also identified (in a blinded setting) if the answer was created by an ORL consultant or ChatGPT. Additionally, the character count was compared. Due to the rapidly evolving pace of technology, a comparison between responses generated by ChatGPT 3 and ChatGPT 4 was included to give an insight into the evolving potential of LLMs. Results: Ratings in all categories were significantly higher for ORL consultants (P<.001). Although inferior to the scores of the ORL consultants, ChatGPT’s scores were relatively higher in semantic categories (conciseness, coherence, and comprehensibility) compared to medical adequacy. ORL consultants identified ChatGPT as the source correctly in 98.4% (121/123) of cases. ChatGPT’s answers had a significantly higher character count compared to ORL consultants (P<.001). Comparison between responses generated by ChatGPT 3 and ChatGPT 4 showed a slight improvement in medical accuracy as well as a better coherence of the answers provided. Contrarily, neither the conciseness (P=.06) nor the comprehensibility (P=.08) improved significantly despite the significant increase in the mean amount of characters by 52.5% (n= (1470-964)/964; P<.001). Conclusions: While ChatGPT provided longer answers to medical problems, medical adequacy and conciseness were significantly lower compared to ORL consultants’ answers. LLMs have potential as augmentative tools for medical care, but their “consultation” for medical problems carries a high risk of misinformation as their high semantic quality may mask contextual deficits. %M 38051578 %R 10.2196/49183 %U https://mededu.jmir.org/2023/1/e49183 %U https://doi.org/10.2196/49183 %U http://www.ncbi.nlm.nih.gov/pubmed/38051578 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e48030 %T The Journey of Data Within a Global Data Sharing Initiative: A Federated 3-Layer Data Analysis Pipeline to Scale Up Multiple Sclerosis Research %A Pirmani,Ashkan %A De Brouwer,Edward %A Geys,Lotte %A Parciak,Tina %A Moreau,Yves %A Peeters,Liesbet M %+ Biomedical Research Institute, Hasselt University, Agoralaan, Building C, Diepenbeek, 3590, Belgium, 32 11 26 92 05, liesbet.peeters@uhasselt.be %K data analysis pipeline %K federated model sharing %K real-world data %K evidence-based decision-making %K end-to-end pipeline %K multiple sclerosis %K data analysis %K pipeline %K data science %K federated %K neurology %K brain %K spine %K spinal nervous system %K neuroscience %K data sharing %K rare %K low prevalence %D 2023 %7 9.11.2023 %9 Original Paper %J JMIR Med Inform %G English %X Background: Investigating low-prevalence diseases such as multiple sclerosis is challenging because of the rather small number of individuals affected by this disease and the scattering of real-world data across numerous data sources. These obstacles impair data integration, standardization, and analysis, which negatively impact the generation of significant meaningful clinical evidence. Objective: This study aims to present a comprehensive, research question–agnostic, multistakeholder-driven end-to-end data analysis pipeline that accommodates 3 prevalent data-sharing streams: individual data sharing, core data set sharing, and federated model sharing. Methods: A demand-driven methodology is employed for standardization, followed by 3 streams of data acquisition, a data quality enhancement process, a data integration procedure, and a concluding analysis stage to fulfill real-world data-sharing requirements. This pipeline’s effectiveness was demonstrated through its successful implementation in the COVID-19 and multiple sclerosis global data sharing initiative. Results: The global data sharing initiative yielded multiple scientific publications and provided extensive worldwide guidance for the community with multiple sclerosis. The pipeline facilitated gathering pertinent data from various sources, accommodating distinct sharing streams and assimilating them into a unified data set for subsequent statistical analysis or secure data examination. This pipeline contributed to the assembly of the largest data set of people with multiple sclerosis infected with COVID-19. Conclusions: The proposed data analysis pipeline exemplifies the potential of global stakeholder collaboration and underlines the significance of evidence-based decision-making. It serves as a paradigm for how data sharing initiatives can propel advancements in health care, emphasizing its adaptability and capacity to address diverse research inquiries. %M 37943585 %R 10.2196/48030 %U https://medinform.jmir.org/2023/1/e48030 %U https://doi.org/10.2196/48030 %U http://www.ncbi.nlm.nih.gov/pubmed/37943585 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e48808 %T ChatGPT-Generated Differential Diagnosis Lists for Complex Case–Derived Clinical Vignettes: Diagnostic Accuracy Evaluation %A Hirosawa,Takanobu %A Kawamura,Ren %A Harada,Yukinori %A Mizuta,Kazuya %A Tokumasu,Kazuki %A Kaji,Yuki %A Suzuki,Tomoharu %A Shimizu,Taro %+ Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, 880 Kitakobayashi, Mibu-cho, Shimotsuga, Tochigi, 321-0293, Japan, 81 282861111, hirosawa@dokkyomed.ac.jp %K artificial intelligence %K AI chatbot %K ChatGPT %K large language models %K clinical decision support %K natural language processing %K diagnostic excellence %K language model %K vignette %K case study %K diagnostic %K accuracy %K decision support %K diagnosis %D 2023 %7 9.10.2023 %9 Original Paper %J JMIR Med Inform %G English %X Background: The diagnostic accuracy of differential diagnoses generated by artificial intelligence chatbots, including ChatGPT models, for complex clinical vignettes derived from general internal medicine (GIM) department case reports is unknown. Objective: This study aims to evaluate the accuracy of the differential diagnosis lists generated by both third-generation ChatGPT (ChatGPT-3.5) and fourth-generation ChatGPT (ChatGPT-4) by using case vignettes from case reports published by the Department of GIM of Dokkyo Medical University Hospital, Japan. Methods: We searched PubMed for case reports. Upon identification, physicians selected diagnostic cases, determined the final diagnosis, and displayed them into clinical vignettes. Physicians typed the determined text with the clinical vignettes in the ChatGPT-3.5 and ChatGPT-4 prompts to generate the top 10 differential diagnoses. The ChatGPT models were not specially trained or further reinforced for this task. Three GIM physicians from other medical institutions created differential diagnosis lists by reading the same clinical vignettes. We measured the rate of correct diagnosis within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and the top diagnosis. Results: In total, 52 case reports were analyzed. The rates of correct diagnosis by ChatGPT-4 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 83% (43/52), 81% (42/52), and 60% (31/52), respectively. The rates of correct diagnosis by ChatGPT-3.5 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 73% (38/52), 65% (34/52), and 42% (22/52), respectively. The rates of correct diagnosis by ChatGPT-4 were comparable to those by physicians within the top 10 (43/52, 83% vs 39/52, 75%, respectively; P=.47) and within the top 5 (42/52, 81% vs 35/52, 67%, respectively; P=.18) differential diagnosis lists and top diagnosis (31/52, 60% vs 26/52, 50%, respectively; P=.43) although the difference was not significant. The ChatGPT models’ diagnostic accuracy did not significantly vary based on open access status or the publication date (before 2011 vs 2022). Conclusions: This study demonstrates the potential diagnostic accuracy of differential diagnosis lists generated using ChatGPT-3.5 and ChatGPT-4 for complex clinical vignettes from case reports published by the GIM department. The rate of correct diagnoses within the top 10 and top 5 differential diagnosis lists generated by ChatGPT-4 exceeds 80%. Although derived from a limited data set of case reports from a single department, our findings highlight the potential utility of ChatGPT-4 as a supplementary tool for physicians, particularly for those affiliated with the GIM department. Further investigations should explore the diagnostic accuracy of ChatGPT by using distinct case materials beyond its training data. Such efforts will provide a comprehensive insight into the role of artificial intelligence in enhancing clinical decision-making. %M 37812468 %R 10.2196/48808 %U https://medinform.jmir.org/2023/1/e48808 %U https://doi.org/10.2196/48808 %U http://www.ncbi.nlm.nih.gov/pubmed/37812468 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e44892 %T A Multilabel Text Classifier of Cancer Literature at the Publication Level: Methods Study of Medical Text Classification %A Zhang,Ying %A Li,Xiaoying %A Liu,Yi %A Li,Aihua %A Yang,Xuemei %A Tang,Xiaoli %+ Institute of Medical Information, Chinese Academy of Medical Sciences, No 69, Dongdan North Street, Beijing, 100020, China, 86 10 52328902, tang.xiaoli@imicams.ac.cn %K text classification %K publication-level classifier %K cancer literature %K deep learning %D 2023 %7 5.10.2023 %9 Original Paper %J JMIR Med Inform %G English %X Background: Given the threat posed by cancer to human health, there is a rapid growth in the volume of data in the cancer field and interdisciplinary and collaborative research is becoming increasingly important for fine-grained classification. The low-resolution classifier of reported studies at the journal level fails to satisfy advanced searching demands, and a single label does not adequately characterize the literature originated from interdisciplinary research results. There is thus a need to establish a multilabel classifier with higher resolution to support literature retrieval for cancer research and reduce the burden of screening papers for clinical relevance. Objective: The primary objective of this research was to address the low-resolution issue of cancer literature classification due to the ambiguity of the existing journal-level classifier in order to support gaining high-relevance evidence for clinical consideration and all-sided results for literature retrieval. Methods: We trained a multilabel classifier with scalability for classifying the literature on cancer research directly at the publication level to assign proper content-derived labels based on the “Bidirectional Encoder Representation from Transformers (BERT) + X” model and obtain the best option for X. First, a corpus of 70,599 cancer publications retrieved from the Dimensions database was divided into a training and a testing set in a ratio of 7:3. Second, using the classification terminology of International Cancer Research Partnership cancer types, we compared the performance of classifiers developed using BERT and 5 classical deep learning models, such as the text recurrent neural network (TextRNN) and FastText, followed by metrics analysis. Results: After comparing various combined deep learning models, we obtained a classifier based on the optimal combination “BERT + TextRNN,” with a precision of 93.09%, a recall of 87.75%, and an F1-score of 90.34%. Moreover, we quantified the distinctive characteristics in the text structure and multilabel distribution in order to generalize the model to other fields with similar characteristics. Conclusions: The “BERT + TextRNN” model was trained for high-resolution classification of cancer literature at the publication level to support accurate retrieval and academic statistics. The model automatically assigns 1 or more labels to each cancer paper, as required. Quantitative comparison verified that the “BERT + TextRNN” model is the best fit for multilabel classification of cancer literature compared to other models. More data from diverse fields will be collected to testify the scalability and extensibility of the proposed model in the future. %M 37796584 %R 10.2196/44892 %U https://medinform.jmir.org/2023/1/e44892 %U https://doi.org/10.2196/44892 %U http://www.ncbi.nlm.nih.gov/pubmed/37796584 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e49944 %T A Natural Language Processing Model for COVID-19 Detection Based on Dutch General Practice Electronic Health Records by Using Bidirectional Encoder Representations From Transformers: Development and Validation Study %A Homburg,Maarten %A Meijer,Eline %A Berends,Matthijs %A Kupers,Thijmen %A Olde Hartman,Tim %A Muris,Jean %A de Schepper,Evelien %A Velek,Premysl %A Kuiper,Jeroen %A Berger,Marjolein %A Peters,Lilian %+ Department of Primary- and Long-Term Care, University Medical Center Groningen, Home Post Code FA21, PO Box 196, Groningen, 9700 RB, Netherlands, 31 050 3616161, t.m.homburg@umcg.nl %K natural language processing %K primary care %K COVID-19 %K EHR %K electronic health records %K public health %K multidisciplinary %K NLP %K disease identification %K BERT model %K model development %K prediction %D 2023 %7 4.10.2023 %9 Original Paper %J J Med Internet Res %G English %X Background: Natural language processing (NLP) models such as bidirectional encoder representations from transformers (BERT) hold promise in revolutionizing disease identification from electronic health records (EHRs) by potentially enhancing efficiency and accuracy. However, their practical application in practice settings demands a comprehensive and multidisciplinary approach to development and validation. The COVID-19 pandemic highlighted challenges in disease identification due to limited testing availability and challenges in handling unstructured data. In the Netherlands, where general practitioners (GPs) serve as the first point of contact for health care, EHRs generated by these primary care providers contain a wealth of potentially valuable information. Nonetheless, the unstructured nature of free-text entries in EHRs poses challenges in identifying trends, detecting disease outbreaks, or accurately pinpointing COVID-19 cases. Objective: This study aims to develop and validate a BERT model for detecting COVID-19 consultations in general practice EHRs in the Netherlands. Methods: The BERT model was initially pretrained on Dutch language data and fine-tuned using a comprehensive EHR data set comprising confirmed COVID-19 GP consultations and non–COVID-19–related consultations. The data set was partitioned into a training and development set, and the model’s performance was evaluated on an independent test set that served as the primary measure of its effectiveness in COVID-19 detection. To validate the final model, its performance was assessed through 3 approaches. First, external validation was applied on an EHR data set from a different geographic region in the Netherlands. Second, validation was conducted using results of polymerase chain reaction (PCR) test data obtained from municipal health services. Lastly, correlation between predicted outcomes and COVID-19–related hospitalizations in the Netherlands was assessed, encompassing the period around the outbreak of the pandemic in the Netherlands, that is, the period before widespread testing. Results: The model development used 300,359 GP consultations. We developed a highly accurate model for COVID-19 consultations (accuracy 0.97, F1-score 0.90, precision 0.85, recall 0.85, specificity 0.99). External validations showed comparable high performance. Validation on PCR test data showed high recall but low precision and specificity. Validation using hospital data showed significant correlation between COVID-19 predictions of the model and COVID-19–related hospitalizations (F1-score 96.8; P<.001; R2=0.69). Most importantly, the model was able to predict COVID-19 cases weeks before the first confirmed case in the Netherlands. Conclusions: The developed BERT model was able to accurately identify COVID-19 cases among GP consultations even preceding confirmed cases. The validated efficacy of our BERT model highlights the potential of NLP models to identify disease outbreaks early, exemplifying the power of multidisciplinary efforts in harnessing technology for disease identification. Moreover, the implications of this study extend beyond COVID-19 and offer a blueprint for the early recognition of various illnesses, revealing that such models could revolutionize disease surveillance. %M 37792444 %R 10.2196/49944 %U https://www.jmir.org/2023/1/e49944 %U https://doi.org/10.2196/49944 %U http://www.ncbi.nlm.nih.gov/pubmed/37792444 %0 Journal Article %@ 1929-073X %I JMIR Publications %V 12 %N %P e46322 %T Web-Based Application Based on Human-in-the-Loop Deep Learning for Deidentifying Free-Text Data in Electronic Medical Records: Development and Usability Study %A Liu,Leibo %A Perez-Concha,Oscar %A Nguyen,Anthony %A Bennett,Vicki %A Blake,Victoria %A Gallego,Blanca %A Jorm,Louisa %+ Centre for Big Data Research in Health, University of New South Wales, Level 2, AGSM Building, G27, Botany St, Sydney, 2052, Australia, 61 290657847, z5250377@ad.unsw.edu.au %K web-based system %K deidentification %K electronic medical records %K deep learning %K narrative free text %K human in the loop %K free text %K unstructured data %K electronic health records %K machine learning %D 2023 %7 25.8.2023 %9 Original Paper %J Interact J Med Res %G English %X Background: The narrative free-text data in electronic medical records (EMRs) contain valuable clinical information for analysis and research to inform better patient care. However, the release of free text for secondary use is hindered by concerns surrounding personally identifiable information (PII), as protecting individuals' privacy is paramount. Therefore, it is necessary to deidentify free text to remove PII. Manual deidentification is a time-consuming and labor-intensive process. Numerous automated deidentification approaches and systems have been attempted to overcome this challenge over the past decade. Objective: We sought to develop an accurate, web-based system deidentifying free text (DEFT), which can be readily and easily adopted in real-world settings for deidentification of free text in EMRs. The system has several key features including a simple and task-focused web user interface, customized PII types, use of a state-of-the-art deep learning model for tagging PII from free text, preannotation by an interactive learning loop, rapid manual annotation with autosave, support for project management and team collaboration, user access control, and central data storage. Methods: DEFT comprises frontend and backend modules and communicates with central data storage through a filesystem path access. The frontend web user interface provides end users with a user-friendly workspace for managing and annotating free text. The backend module processes the requests from the frontend and performs relevant persistence operations. DEFT manages the deidentification workflow as a project, which can contain one or more data sets. Customized PII types and user access control can also be configured. The deep learning model is based on a Bidirectional Long Short-Term Memory-Conditional Random Field (BiLSTM-CRF) with RoBERTa as the word embedding layer. The interactive learning loop is further integrated into DEFT to speed up the deidentification process and increase its performance over time. Results: DEFT has many advantages over existing deidentification systems in terms of its support for project management, user access control, data management, and an interactive learning process. Experimental results from DEFT on the 2014 i2b2 data set obtained the highest performance compared to 5 benchmark models in terms of microaverage strict entity–level recall and F1-scores of 0.9563 and 0.9627, respectively. In a real-world use case of deidentifying clinical notes, extracted from 1 referral hospital in Sydney, New South Wales, Australia, DEFT achieved a high microaverage strict entity–level F1-score of 0.9507 on a corpus of 600 annotated clinical notes. Moreover, the manual annotation process with preannotation demonstrated a 43% increase in work efficiency compared to the process without preannotation. Conclusions: DEFT is designed for health domain researchers and data custodians to easily deidentify free text in EMRs. DEFT supports an interactive learning loop and end users with minimal technical knowledge can perform the deidentification work with only a shallow learning curve. %M 37624624 %R 10.2196/46322 %U https://www.i-jmr.org/2023/1/e46322 %U https://doi.org/10.2196/46322 %U http://www.ncbi.nlm.nih.gov/pubmed/37624624 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 12 %N %P e48645 %T Comprehensive Ontology of Fibroproliferative Diseases: Protocol for a Semantic Technology Study %A Golec,Marcin %A Kamdar,Maulik %A Barteit,Sandra %+ Heidelberg Institute of Global Health, Faculty of Medicine and University Hospital, Heidelberg University, Im Neuenheimer Feld 130.3, Heidelberg, 69120, Germany, 352 0691800035, marcin.golec@uni-heidelberg.de %K fibroproliferative disease %K fibrosis %K fibrotic disease %K ontology %K OWL %K semantic technology %K Web Ontology Language %D 2023 %7 11.8.2023 %9 Protocol %J JMIR Res Protoc %G English %X Background: Fibroproliferative or fibrotic diseases (FDs), which represent a significant proportion of age-related pathologies and account for over 40% of mortality in developed nations, are often underrepresented in focused research. Typically, these conditions are studied individually, such as chronic obstructive pulmonary disease or idiopathic pulmonary fibrosis (IPF), rather than as a collective entity, thereby limiting the holistic understanding and development of effective treatments. To address this, we propose creating and publicizing a comprehensive fibroproliferative disease ontology (FDO) to unify the understanding of FDs. Objective: This paper aims to delineate the study protocol for the creation of the FDO, foster transparency and high quality standards during its development, and subsequently promote its use once it becomes publicly available. Methods: We aim to establish an ontology encapsulating the broad spectrum of FDs, constructed in the Web Ontology Language format using the Protégé ontology editor, adhering to ontology development life cycle principles. The modeling process will leverage Protégé in accordance with a methodologically defined process, involving targeted scoping reviews of MEDLINE and PubMed information, expert knowledge, and an ontology development process. A hybrid top-down and bottom-up strategy will guide the identification of core concepts and relations, conducted by a team of domain experts based on systematic iterations of scientific literature reviews. Results: The result will be an exhaustive FDO accommodating a wide variety of crucial biomedical concepts, augmented with synonyms, definitions, and references. The FDO aims to encapsulate diverse perspectives on the FD domain, including those of clinicians, health informaticians, medical researchers, and public health experts. Conclusions: The FDO is expected to stimulate broader and more in-depth FD research by enabling reasoning, inference, and the identification of relationships between concepts for application in multiple contexts, such as developing specialized software, fostering research communities, and enhancing domain comprehension. A common vocabulary and understanding of relationships among medical professionals could potentially expedite scientific progress and the discovery of innovative solutions. The publicly available FDO will form the foundation for future research, technological advancements, and public health initiatives. International Registered Report Identifier (IRRID): PRR1-10.2196/48645 %M 37566458 %R 10.2196/48645 %U https://www.researchprotocols.org/2023/1/e48645 %U https://doi.org/10.2196/48645 %U http://www.ncbi.nlm.nih.gov/pubmed/37566458 %0 Journal Article %@ 2291-9694 %I %V 11 %N %P e45105 %T Identifying Risk Factors Associated With Lower Back Pain in Electronic Medical Record Free Text: Deep Learning Approach Using Clinical Note Annotations %A Jaiswal,Aman %A Katz,Alan %A Nesca,Marcello %A Milios,Evangelos %K machine learning %K lower back pain %K natural language processing %K semantic textual similarity %K electronic medical records %K risk factors %K deep learning %D 2023 %7 9.8.2023 %9 %J JMIR Med Inform %G English %X Background: Lower back pain is a common weakening condition that affects a large population. It is a leading cause of disability and lost productivity, and the associated medical costs and lost wages place a substantial burden on individuals and society. Recent advances in artificial intelligence and natural language processing have opened new opportunities for the identification and management of risk factors for lower back pain. In this paper, we propose and train a deep learning model on a data set of clinical notes that have been annotated with relevant risk factors, and we evaluate the model’s performance in identifying risk factors in new clinical notes. Objective: The primary objective is to develop a novel deep learning approach to detect risk factors for underlying disease in patients presenting with lower back pain in clinical encounter notes. The secondary objective is to propose solutions to potential challenges of using deep learning and natural language processing techniques for identifying risk factors in electronic medical record free text and make practical recommendations for future research in this area. Methods: We manually annotated clinical notes for the presence of six risk factors for severe underlying disease in patients presenting with lower back pain. Data were highly imbalanced, with only 12% (n=296) of the annotated notes having at least one risk factor. To address imbalanced data, a combination of semantic textual similarity and regular expressions was used to further capture notes for annotation. Further analysis was conducted to study the impact of downsampling, binary formulation of multi-label classification, and unsupervised pretraining on classification performance. Results: Of 2749 labeled clinical notes, 347 exhibited at least one risk factor, while 2402 exhibited none. The initial analysis shows that downsampling the training set to equalize the ratio of clinical notes with and without risk factors improved the macro–area under the receiver operating characteristic curve (AUROC) by 2%. The Bidirectional Encoder Representations from Transformers (BERT) model improved the macro-AUROC by 15% over the traditional machine learning baseline. In experiment 2, the proposed BERT–convolutional neural network (CNN) model for longer texts improved (4% macro-AUROC) over the BERT baseline, and the multitask models are more stable for minority classes. In experiment 3, domain adaptation of BERTCNN using masked language modeling improved the macro-AUROC by 2%. Conclusions: Primary care clinical notes are likely to require manipulation to perform meaningful free-text analysis. The application of BERT models for multi-label classification on downsampled annotated clinical notes is useful in detecting risk factors suggesting an indication for imaging for patients with lower back pain. %R 10.2196/45105 %U https://medinform.jmir.org/2023/1/e45105 %U https://doi.org/10.2196/45105 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e45116 %T Methods Used in the Development of Common Data Models for Health Data: Scoping Review %A Ahmadi,Najia %A Zoch,Michele %A Kelbert,Patricia %A Noll,Richard %A Schaaf,Jannik %A Wolfien,Markus %A Sedlmayr,Martin %+ Institute for Medical Informatics and Biometry, Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Fetscherstr 74, Dresden, 01307, Germany, 49 351458 87 7704, najia.ahmadi@tu-dresden.de %K common data model %K common data elements %K health data %K electronic health record %K Observational Medical Outcomes Partnership %K stakeholder involvement %K Data harmonisation %K Interoperability %K Standardized Data Repositories %K Suggestive Development Process %K Healthcare %K Medical Informatics %K %D 2023 %7 3.8.2023 %9 Review %J JMIR Med Inform %G English %X Background: Common data models (CDMs) are essential tools for data harmonization, which can lead to significant improvements in the health domain. CDMs unite data from disparate sources and ease collaborations across institutions, resulting in the generation of large standardized data repositories across different entities. An overview of existing CDMs and methods used to develop these data sets may assist in the development process of future models for the health domain, such as for decision support systems. Objective: This scoping review investigates methods used in the development of CDMs for health data. We aim to provide a broad overview of approaches and guidelines that are used in the development of CDMs (ie, common data elements or common data sets) for different health domains on an international level. Methods: This scoping review followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist. We conducted the literature search in prominent databases, namely, PubMed, Web of Science, Science Direct, and Scopus, starting from January 2000 until March 2022. We identified and screened 1309 articles. The included articles were evaluated based on the type of adopted method, which was used in the conception, users’ needs collection, implementation, and evaluation phases of CDMs, and whether stakeholders (such as medical experts, patients’ representatives, and IT staff) were involved during the process. Moreover, the models were grouped into iterative or linear types based on the imperativeness of the stages during development. Results: We finally identified 59 articles that fit our eligibility criteria. Of these articles, 45 specifically focused on common medical conditions, 10 focused on rare medical conditions, and the remaining 4 focused on both conditions. The development process usually involved stakeholders but in different ways (eg, working group meetings, Delphi approaches, interviews, and questionnaires). Twenty-two models followed an iterative process. Conclusions: The included articles showed the diversity of methods used to develop a CDM in different domains of health. We highlight the need for more specialized CDM development methods in the health domain and propose a suggestive development process that might ease the development of CDMs in the health domain in the future. %M 37535410 %R 10.2196/45116 %U https://medinform.jmir.org/2023/1/e45116 %U https://doi.org/10.2196/45116 %U http://www.ncbi.nlm.nih.gov/pubmed/37535410 %0 Journal Article %@ 2369-3762 %I JMIR Publications %V 9 %N %P e41953 %T Evaluating the Applicability of Existing Lexicon-Based Sentiment Analysis Techniques on Family Medicine Resident Feedback Field Notes: Retrospective Cohort Study %A Lu,Kevin Jia Qi %A Meaney,Christopher %A Guo,Elaine %A Leung,Fok-Han %+ Department of Family and Community Medicine, University of Toronto, 500 University Avenue, 5th Floor, Toronto, ON, M5G 1V7, Canada, 1 6133020132, klu1@shn.ca %K medical education %K medical resident %K feedback %K field note %K text mining %K data mining %K sentiment analysis %K lexicon %K lexical %K dictionary %K dictionaries %K vocabulary %K resident %K medical student %K medical trainee %K residency %K utility %K feasibility %D 2023 %7 27.7.2023 %9 Original Paper %J JMIR Med Educ %G English %X Background: Field notes, a form for resident-preceptor clinical encounter feedback, are widely adopted across Canadian medical residency training programs for documenting residents’ performance. This process generates a sizeable cumulative collection of feedback text, which is difficult for medical education faculty to navigate. As sentiment analysis is a subfield of text mining that can efficiently synthesize the polarity of a text collection, sentiment analysis may serve as an innovative solution. Objective: This study aimed to examine the feasibility and utility of sentiment analysis using 3 popular sentiment lexicons on medical resident field notes. Methods: We used a retrospective cohort design, curating text data from University of Toronto medical resident field notes gathered over 2 years (from July 2019 to June 2021). Lexicon-based sentiment analysis was applied using 3 standardized dictionaries, modified by removing ambiguous words as determined by a medical subject matter expert. Our modified lexicons assigned words from the text data a sentiment score, and we aggregated the word-level scores to a document-level polarity score. Agreement between dictionaries was assessed, and the document-level polarity was correlated with the overall preceptor rating of the clinical encounter under assessment. Results: Across the 3 original dictionaries, approximately a third of labeled words in our field note corpus were deemed ambiguous and were removed to create modified dictionaries. Across the 3 modified dictionaries, the mean sentiment for the “Strengths” section of the field notes was mildly positive, while it was slightly less positive in the “Areas of Improvement” section. We observed reasonable agreement between dictionaries for sentiment scores in both field note sections. Overall, the proportion of positively labeled documents increased with the overall preceptor rating, and the proportion of negatively labeled documents decreased with the overall preceptor rating. Conclusions: Applying sentiment analysis to systematically analyze field notes is feasible. However, the applicability of existing lexicons is limited in the medical setting, even after the removal of ambiguous words. Limited applicability warrants the need to generate new dictionaries specific to the medical education context. Additionally, aspect-based sentiment analysis may be applied to navigate the more nuanced structure of texts when identifying sentiments. Ultimately, this will allow for more robust inferences to discover opportunities for improving resident teaching curriculums. %M 37498660 %R 10.2196/41953 %U https://mededu.jmir.org/2023/1/e41953 %U https://doi.org/10.2196/41953 %U http://www.ncbi.nlm.nih.gov/pubmed/37498660 %0 Journal Article %@ 2291-9694 %I %V 11 %N %P e45850 %T An Ontology-Based Approach to Improving Medication Appropriateness in Older Patients: Algorithm Development and Validation Study %A Calvo-Cidoncha,Elena %A Verdinelli,Julián %A González-Bueno,Javier %A López-Soto,Alfonso %A Camacho Hernando,Concepción %A Pastor-Duran,Xavier %A Codina-Jané,Carles %A Lozano-Rubí,Raimundo %K biological ontologies %K decision support systems %K inappropriate prescribing %K elderly %K medication regimen complexity %K anticholinergic drug burden %K trigger tool %K clinical %K ontologies %K pharmacy %K medication %K decision support %K pharmaceutic %K pharmacology %K chronic condition %K chronic disease %K domain %K adverse event %K ontology-based %K alert %D 2023 %7 10.7.2023 %9 %J JMIR Med Inform %G English %X Background: Inappropriate medication in older patients with multimorbidity results in a greater risk of adverse drug events. Clinical decision support systems (CDSSs) are intended to improve medication appropriateness. One approach to improving CDSSs is to use ontologies instead of relational databases. Previously, we developed OntoPharma—an ontology-based CDSS for reducing medication prescribing errors. Objective: The primary aim was to model a domain for improving medication appropriateness in older patients (chronic patient domain). The secondary aim was to implement the version of OntoPharma containing the chronic patient domain in a hospital setting. Methods: A 4-step process was proposed. The first step was defining the domain scope. The chronic patient domain focused on improving medication appropriateness in older patients. A group of experts selected the following three use cases: medication regimen complexity, anticholinergic and sedative drug burden, and the presence of triggers for identifying possible adverse events. The second step was domain model representation. The implementation was conducted by medical informatics specialists and clinical pharmacists using Protégé-OWL (Stanford Center for Biomedical Informatics Research). The third step was OntoPharma-driven alert module adaptation. We reused the existing framework based on SPARQL to query ontologies. The fourth step was implementing the version of OntoPharma containing the chronic patient domain in a hospital setting. Alerts generated from July to September 2022 were analyzed. Results: We proposed 6 new classes and 5 new properties, introducing the necessary changes in the ontologies previously created. An alert is shown if the Medication Regimen Complexity Index is ≥40, if the Drug Burden Index is ≥1, or if there is a trigger based on an abnormal laboratory value. A total of 364 alerts were generated for 107 patients; 154 (42.3%) alerts were accepted. Conclusions: We proposed an ontology-based approach to provide support for improving medication appropriateness in older patients with multimorbidity in a scalable, sustainable, and reusable way. The chronic patient domain was built based on our previous research, reusing the existing framework. OntoPharma has been implemented in clinical practice and generates alerts, considering the following use cases: medication regimen complexity, anticholinergic and sedative drug burden, and the presence of triggers for identifying possible adverse events. %R 10.2196/45850 %U https://medinform.jmir.org/2023/1/e45850 %U https://doi.org/10.2196/45850 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e44870 %T Transferability Based on Drug Structure Similarity in the Automatic Classification of Noncompliant Drug Use on Social Media: Natural Language Processing Approach %A Nishiyama,Tomohiro %A Yada,Shuntaro %A Wakamiya,Shoko %A Hori,Satoko %A Aramaki,Eiji %+ Department of Information Science, Nara Institute of Science and Technology, 8916-5, Takayama-cho, Ikoma, 630-0192, Japan, 81 743 72 5250, aramaki@is.naist.jp %K data mining %K machine learning %K medication noncompliance %K natural language processing %K pharmacovigilance %K transfer learning %K text classification %D 2023 %7 3.5.2023 %9 Original Paper %J J Med Internet Res %G English %X Background: Medication noncompliance is a critical issue because of the increased number of drugs sold on the web. Web-based drug distribution is difficult to control, causing problems such as drug noncompliance and abuse. The existing medication compliance surveys lack completeness because it is impossible to cover patients who do not go to the hospital or provide accurate information to their doctors, so a social media–based approach is being explored to collect information about drug use. Social media data, which includes information on drug usage by users, can be used to detect drug abuse and medication compliance in patients. Objective: This study aimed to assess how the structural similarity of drugs affects the efficiency of machine learning models for text classification of drug noncompliance. Methods: This study analyzed 22,022 tweets about 20 different drugs. The tweets were labeled as either noncompliant use or mention, noncompliant sales, general use, or general mention. The study compares 2 methods for training machine learning models for text classification: single-sub-corpus transfer learning, in which a model is trained on tweets about a single drug and then tested on tweets about other drugs, and multi-sub-corpus incremental learning, in which models are trained on tweets about drugs in order of their structural similarity. The performance of a machine learning model trained on a single subcorpus (a data set of tweets about a specific category of drugs) was compared to the performance of a model trained on multiple subcorpora (data sets of tweets about multiple categories of drugs). Results: The results showed that the performance of the model trained on a single subcorpus varied depending on the specific drug used for training. The Tanimoto similarity (a measure of the structural similarity between compounds) was weakly correlated with the classification results. The model trained by transfer learning a corpus of drugs with close structural similarity performed better than the model trained by randomly adding a subcorpus when the number of subcorpora was small. Conclusions: The results suggest that structural similarity improves the classification performance of messages about unknown drugs if the drugs in the training corpus are few. On the other hand, this indicates that there is little need to consider the influence of the Tanimoto structural similarity if a sufficient variety of drugs are ensured. %M 37133915 %R 10.2196/44870 %U https://www.jmir.org/2023/1/e44870 %U https://doi.org/10.2196/44870 %U http://www.ncbi.nlm.nih.gov/pubmed/37133915 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e41100 %T Supervised Relation Extraction Between Suicide-Related Entities and Drugs: Development and Usability Study of an Annotated PubMed Corpus %A Karapetian,Karina %A Jeon,Soo Min %A Kwon,Jin-Won %A Suh,Young-Kyoon %+ School of Computer Science and Engineering, Kyungpook National University, Rm. 520, IT-5, 80 Daehak-ro, Bukgu, Daegu, 41566, Republic of Korea, 82 53 950 6372, yksuh@knu.ac.kr %K suicide %K adverse drug events %K information extraction %K relation classification %K bidirectional encoder representations from transformers %K pharmacovigilance %K natural language processing %K PubMed %K corpus %K language model %D 2023 %7 8.3.2023 %9 Original Paper %J J Med Internet Res %G English %X Background: Drug-induced suicide has been debated as a crucial issue in both clinical and public health research. Published research articles contain valuable data on the drugs associated with suicidal adverse events. An automated process that extracts such information and rapidly detects drugs related to suicide risk is essential but has not been well established. Moreover, few data sets are available for training and validating classification models on drug-induced suicide. Objective: This study aimed to build a corpus of drug-suicide relations containing annotated entities for drugs, suicidal adverse events, and their relations. To confirm the effectiveness of the drug-suicide relation corpus, we evaluated the performance of a relation classification model using the corpus in conjunction with various embeddings. Methods: We collected the abstracts and titles of research articles associated with drugs and suicide from PubMed and manually annotated them along with their relations at the sentence level (adverse drug events, treatment, suicide means, or miscellaneous). To reduce the manual annotation effort, we preliminarily selected sentences with a pretrained zero-shot classifier or sentences containing only drug and suicide keywords. We trained a relation classification model using various Bidirectional Encoder Representations from Transformer embeddings with the proposed corpus. We then compared the performances of the model with different Bidirectional Encoder Representations from Transformer–based embeddings and selected the most suitable embedding for our corpus. Results: Our corpus comprised 11,894 sentences extracted from the titles and abstracts of the PubMed research articles. Each sentence was annotated with drug and suicide entities and the relationship between these 2 entities (adverse drug events, treatment, means, and miscellaneous). All of the tested relation classification models that were fine-tuned on the corpus accurately detected sentences of suicidal adverse events regardless of their pretrained type and data set properties. Conclusions: To our knowledge, this is the first and most extensive corpus of drug-suicide relations. %M 36884281 %R 10.2196/41100 %U https://www.jmir.org/2023/1/e41100 %U https://doi.org/10.2196/41100 %U http://www.ncbi.nlm.nih.gov/pubmed/36884281 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 7 %N %P e43014 %T Extracting Medical Information From Free-Text and Unstructured Patient-Generated Health Data Using Natural Language Processing Methods: Feasibility Study With Real-world Data %A Sezgin,Emre %A Hussain,Syed-Amad %A Rust,Steve %A Huang,Yungui %+ The Abigail Wexner Research Institute at Nationwide Children's Hospital, 700 Children's Drive, Columbus, OH, 43205, United States, 1 614 722 3179, emre.sezgin@nationwidechildrens.org %K patient-generated health data %K natural language processing %K named entity recognition %K patient health records %K text notes %K voice %K audio real-world data %D 2023 %7 7.3.2023 %9 Original Paper %J JMIR Form Res %G English %X Background: Patient-generated health data (PGHD) captured via smart devices or digital health technologies can reflect an individual health journey. PGHD enables tracking and monitoring of personal health conditions, symptoms, and medications out of the clinic, which is crucial for self-care and shared clinical decisions. In addition to self-reported measures and structured PGHD (eg, self-screening, sensor-based biometric data), free-text and unstructured PGHD (eg, patient care note, medical diary) can provide a broader view of a patient’s journey and health condition. Natural language processing (NLP) is used to process and analyze unstructured data to create meaningful summaries and insights, showing promise to improve the utilization of PGHD. Objective: Our aim is to understand and demonstrate the feasibility of an NLP pipeline to extract medication and symptom information from real-world patient and caregiver data. Methods: We report a secondary data analysis, using a data set collected from 24 parents of children with special health care needs (CSHCN) who were recruited via a nonrandom sampling approach. Participants used a voice-interactive app for 2 weeks, generating free-text patient notes (audio transcription or text entry). We built an NLP pipeline using a zero-shot approach (adaptive to low-resource settings). We used named entity recognition (NER) and medical ontologies (RXNorm and SNOMED CT [Systematized Nomenclature of Medicine Clinical Terms]) to identify medication and symptoms. Sentence-level dependency parse trees and part-of-speech tags were used to extract additional entity information using the syntactic properties of a note. We assessed the data; evaluated the pipeline with the patient notes; and reported the precision, recall, and F1 scores. Results: In total, 87 patient notes are included (audio transcriptions n=78 and text entries n=9) from 24 parents who have at least one CSHCN. The participants were between the ages of 26 and 59 years. The majority were White (n=22, 92%), had more than one child (n=16, 67%), lived in Ohio (n=22, 92%), had mid- or upper-mid household income (n=15, 62.5%), and had higher level education (n=24, 58%). Out of 87 notes, 30 were drug and medication related, and 46 were symptom related. We captured medication instances (medication, unit, quantity, and date) and symptoms satisfactorily (precision >0.65, recall >0.77, F1>0.72). These results indicate the potential when using NER and dependency parsing through an NLP pipeline on information extraction from unstructured PGHD. Conclusions: The proposed NLP pipeline was found to be feasible for use with real-world unstructured PGHD to accomplish medication and symptom extraction. Unstructured PGHD can be leveraged to inform clinical decision-making, remote monitoring, and self-care including medical adherence and chronic disease management. With customizable information extraction methods using NER and medical ontologies, NLP models can feasibly extract a broad range of clinical information from unstructured PGHD in low-resource settings (eg, a limited number of patient notes or training data). %M 36881467 %R 10.2196/43014 %U https://formative.jmir.org/2023/1/e43014 %U https://doi.org/10.2196/43014 %U http://www.ncbi.nlm.nih.gov/pubmed/36881467 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 7 %N %P e39077 %T German Medical Named Entity Recognition Model and Data Set Creation Using Machine Translation and Word Alignment: Algorithm Development and Validation %A Frei,Johann %A Kramer,Frank %+ IT Infrastructure for Translational Medical Research, University of Augsburg, Alter Postweg 101, Augsburg, 86159, Germany, 49 17691464136, johann.frei@informatik.uni-augsburg.de %K natural language processing %K named entity recognition %K information extraction %D 2023 %7 28.2.2023 %9 Original Paper %J JMIR Form Res %G English %X Background: Data mining in the field of medical data analysis often needs to rely solely on the processing of unstructured data to retrieve relevant data. For German natural language processing, few open medical neural named entity recognition (NER) models have been published before this work. A major issue can be attributed to the lack of German training data. Objective: We developed a synthetic data set and a novel German medical NER model for public access to demonstrate the feasibility of our approach. In order to bypass legal restrictions due to potential data leaks through model analysis, we did not make use of internal, proprietary data sets, which is a frequent veto factor for data set publication. Methods: The underlying German data set was retrieved by translation and word alignment of a public English data set. The data set served as a foundation for model training and evaluation. For demonstration purposes, our NER model follows a simple network architecture that is designed for low computational requirements. Results: The obtained data set consisted of 8599 sentences including 30,233 annotations. The model achieved a class frequency–averaged F1 score of 0.82 on the test set after training across 7 different NER types. Artifacts in the synthesized data set with regard to translation and alignment induced by the proposed method were exposed. The annotation performance was evaluated on an external data set and measured in comparison with an existing baseline model that has been trained on a dedicated German data set in a traditional fashion. We discussed the drop in annotation performance on an external data set for our simple NER model. Our model is publicly available. Conclusions: We demonstrated the feasibility of obtaining a data set and training a German medical NER model by the exclusive use of public training data through our suggested method. The discussion on the limitations of our approach includes ways to further mitigate remaining problems in future work. %M 36853741 %R 10.2196/39077 %U https://formative.jmir.org/2023/1/e39077 %U https://doi.org/10.2196/39077 %U http://www.ncbi.nlm.nih.gov/pubmed/36853741 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e40672 %T A Comprehensive and Improved Definition for Hospital-Acquired Pressure Injury Classification Based on Electronic Health Records: Comparative Study %A Sotoodeh,Mani %A Zhang,Wenhui %A Simpson,Roy L %A Hertzberg,Vicki Stover %A Ho,Joyce C %+ School of Nursing, Emory University, Room 224, 1520 Clifton Rd, Atlanta, GA, 30322, United States, 1 512 968 8985, wenhui.zhang@emory.edu %K pressure ulcer %K decubitus ulcer %K electronic medical records %K bedsore %K nursing %K data mining %K electronic health record %K EHR %K nursing assessment %K pressure ulcer care %K pressure ulcer prevention %K EHR data %K EHR systems %K nursing quality %D 2023 %7 23.2.2023 %9 Original Paper %J JMIR Med Inform %G English %X Background: Patients develop pressure injuries (PIs) in the hospital owing to low mobility, exposure to localized pressure, circulatory conditions, and other predisposing factors. Over 2.5 million Americans develop PIs annually. The Center for Medicare and Medicaid considers hospital-acquired PIs (HAPIs) as the most frequent preventable event, and they are the second most common claim in lawsuits. With the growing use of electronic health records (EHRs) in hospitals, an opportunity exists to build machine learning models to identify and predict HAPI rather than relying on occasional manual assessments by human experts. However, accurate computational models rely on high-quality HAPI data labels. Unfortunately, the different data sources within EHRs can provide conflicting information on HAPI occurrence in the same patient. Furthermore, the existing definitions of HAPI disagree with each other, even within the same patient population. The inconsistent criteria make it impossible to benchmark machine learning methods to predict HAPI. Objective: The objective of this project was threefold. We aimed to identify discrepancies in HAPI sources within EHRs, to develop a comprehensive definition for HAPI classification using data from all EHR sources, and to illustrate the importance of an improved HAPI definition. Methods: We assessed the congruence among HAPI occurrences documented in clinical notes, diagnosis codes, procedure codes, and chart events from the Medical Information Mart for Intensive Care III database. We analyzed the criteria used for the 3 existing HAPI definitions and their adherence to the regulatory guidelines. We proposed the Emory HAPI (EHAPI), which is an improved and more comprehensive HAPI definition. We then evaluated the importance of the labels in training a HAPI classification model using tree-based and sequential neural network classifiers. Results: We illustrate the complexity of defining HAPI, with <13% of hospital stays having at least 3 PI indications documented across 4 data sources. Although chart events were the most common indicator, it was the only PI documentation for >49% of the stays. We demonstrate a lack of congruence across existing HAPI definitions and EHAPI, with only 219 stays having a consensus positive label. Our analysis highlights the importance of our improved HAPI definition, with classifiers trained using our labels outperforming others on a small manually labeled set from nurse annotators and a consensus set in which all definitions agreed on the label. Conclusions: Standardized HAPI definitions are important for accurately assessing HAPI nursing quality metric and determining HAPI incidence for preventive measures. We demonstrate the complexity of defining an occurrence of HAPI, given the conflicting and incomplete EHR data. Our EHAPI definition has favorable properties, making it a suitable candidate for HAPI classification tasks. %M 36649481 %R 10.2196/40672 %U https://medinform.jmir.org/2023/1/e40672 %U https://doi.org/10.2196/40672 %U http://www.ncbi.nlm.nih.gov/pubmed/36649481 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e40569 %T Pregex: Rule-Based Detection and Extraction of Twitter Data in Pregnancy %A Klein,Ari Z %A Kunatharaju,Shriya %A O'Connor,Karen %A Gonzalez-Hernandez,Graciela %+ Department of Computational Biomedicine, Cedars-Sinai Medical Center, Pacific Design Center, Suite G549F, 700 North San Vicente Boulevard, West Hollywood, CA, 90069, United States, 1 310 423 3521, Graciela.GonzalezHernandez@csmc.edu %K natural language processing %K data mining %K social media %K pregnancy %D 2023 %7 9.2.2023 %9 Research Letter %J J Med Internet Res %G English %X %M 36757756 %R 10.2196/40569 %U https://www.jmir.org/2023/1/e40569 %U https://doi.org/10.2196/40569 %U http://www.ncbi.nlm.nih.gov/pubmed/36757756 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e43750 %T Systematized Nomenclature of Medicine–Clinical Terminology (SNOMED CT) Clinical Use Cases in the Context of Electronic Health Record Systems: Systematic Literature Review %A Vuokko,Riikka %A Vakkuri,Anne %A Palojoki,Sari %+ Unit for Digitalization and Management, Ministry of Social Affairs and Health, PO Box 33, Helsinki, FI-00023 Government, Finland, 358 50 453 6377, riikka.vuokko@gov.fi %K clinical %K electronic health record %K EHR %K review method %K literature review %K SNOMED CT %K Systematized Nomenclature for Medicine %K use case %K terminology %K terminologies %K SNOMED %D 2023 %7 6.2.2023 %9 Review %J JMIR Med Inform %G English %X Background: The Systematized Medical Nomenclature for Medicine–Clinical Terminology (SNOMED CT) is a clinical terminology system that provides a standardized and scientifically validated way of representing clinical information captured by clinicians. It can be integrated into electronic health records (EHRs) to increase the possibilities for effective data use and ensure a better quality of documentation that supports continuity of care, thus enabling better quality in the care process. Even though SNOMED CT consists of extensively studied clinical terminology, previous research has repeatedly documented a lack of scientific evidence for SNOMED CT in the form of reported clinical use cases in electronic health record systems. Objective: The aim of this study was to explore evidence in previous literature reviews of clinical use cases of SNOMED CT integrated into EHR systems or other clinical applications during the last 5 years of continued development. The study sought to identify the main clinical use purposes, use phases, and key clinical benefits documented in SNOMED CT use cases. Methods: The Cochrane review protocol was applied for the study design. The application of the protocol was modified step-by-step to fit the research problem by first defining the search strategy, identifying the articles for the review by isolating the exclusion and inclusion criteria for assessing the search results, and lastly, evaluating and summarizing the review results. Results: In total, 17 research articles illustrating SNOMED CT clinical use cases were reviewed. The use purpose of SNOMED CT was documented in all the articles, with the terminology as a standard in EHR being the most common (8/17). The clinical use phase was documented in all the articles. The most common category of use phases was SNOMED CT in development (6/17). Core benefits achieved by applying SNOMED CT in a clinical context were identified by the researchers. These were related to terminology use outcomes, that is, to data quality in general or to enabling a consistent way of indexing, storing, retrieving, and aggregating clinical data (8/17). Additional benefits were linked to the productivity of coding or to advances in the quality and continuity of care. Conclusions: While the SNOMED CT use categories were well supported by previous research, this review demonstrates that further systematic research on clinical use cases is needed to promote the scalability of the review results. To achieve the best out-of-use case reports, more emphasis is suggested on describing the contextual factors, such as the electronic health care system and the use of previous frameworks to enable comparability of results. A lesson to be drawn from our study is that SNOMED CT is essential for structuring clinical data; however, research is needed to gather more evidence of how SNOMED CT benefits clinical care and patient safety. %M 36745498 %R 10.2196/43750 %U https://medinform.jmir.org/2023/1/e43750 %U https://doi.org/10.2196/43750 %U http://www.ncbi.nlm.nih.gov/pubmed/36745498 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e38861 %T A Linked Open Data–Based Terminology to Describe Libre/Free and Open-source Software: Incremental Development Study %A Jahn,Franziska %A Ammenwerth,Elske %A Dornauer,Verena %A Höffner,Konrad %A Bindel,Michelle %A Karopka,Thomas %A Winter,Alfred %+ Institute of Medical Informatics, Statistics and Epidemiology, Faculty of Medicine, Leipzig University, Härtelstr. 16-18, Leipzig, 04107, Germany, 49 341 97 16194, franziska.jahn@imise.uni-leipzig.de %K health informatics %K ontology %K free/libre open-source software %K software applications %K health IT %K terminology %D 2023 %7 20.1.2023 %9 Original Paper %J JMIR Med Inform %G English %X Background: There is a variety of libre/free and open-source software (LIFOSS) products for medicine and health care. To support health care and IT professionals select an appropriate software product for given tasks, several comparison studies and web platforms, such as Medfloss.org, are available. However, due to the lack of a uniform terminology for health informatics, ambiguous or imprecise terms are used to describe the functionalities of LIFOSS. This makes comparisons of LIFOSS difficult and may lead to inappropriate software selection decisions. Using Linked Open Data (LOD) promises to address these challenges. Objective: We describe LIFOSS systematically with the help of the underlying Health Information Technology Ontology (HITO). We publish HITO and HITO-based software product descriptions using LOD to obtain the following benefits: (1) linking and reusing existing terminologies and (2) using Semantic Web tools for viewing and querying the LIFOSS data on the World Wide Web. Methods: HITO was incrementally developed and implemented. First, classes for the description of software products in health IT evaluation studies were identified. Second, requirements for describing LIFOSS were elicited by interviewing domain experts. Third, to describe domain-specific functionalities of software products, existing catalogues of features and enterprise functions were analyzed and integrated into the HITO knowledge base. As a proof of concept, HITO was used to describe 25 LIFOSS products. Results: HITO provides a defined set of classes and their relationships to describe LIFOSS in medicine and health care. With the help of linked or integrated catalogues for languages, programming languages, licenses, features, and enterprise functions, the functionalities of LIFOSS can be precisely described and compared. We publish HITO and the LIFOSS descriptions as LOD; they can be queried and viewed using different Semantic Web tools, such as Resource Description Framework (RDF) browsers, SPARQL Protocol and RDF Query Language (SPARQL) queries, and faceted searches. The advantages of providing HITO as LOD are demonstrated by practical examples. Conclusions: HITO is a building block to achieving unambiguous communication among health IT professionals and researchers. Providing LIFOSS product information as LOD enables barrier-free and easy access to data that are often hidden in user manuals of software products or are not available at all. Efforts to establish a unique terminology of medical and health informatics should be further supported and continued. %M 36662569 %R 10.2196/38861 %U https://medinform.jmir.org/2023/1/e38861 %U https://doi.org/10.2196/38861 %U http://www.ncbi.nlm.nih.gov/pubmed/36662569 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 11 %N %P e43053 %T Ontologies Applied in Clinical Decision Support System Rules: Systematic Review %A Jing,Xia %A Min,Hua %A Gong,Yang %A Biondich,Paul %A Robinson,David %A Law,Timothy %A Nohr,Christian %A Faxvaag,Arild %A Rennert,Lior %A Hubig,Nina %A Gimbel,Ronald %+ Department of Public Health Sciences, Clemson University, 519 Edwards Hall, Clemson, SC, 29634, United States, 1 8646563347, xjing@clemson.edu %K clinical decision support system rules %K clinical decision support systems %K interoperability %K ontology %K Semantic Web technology %D 2023 %7 19.1.2023 %9 Review %J JMIR Med Inform %G English %X Background: Clinical decision support systems (CDSSs) are important for the quality and safety of health care delivery. Although CDSS rules guide CDSS behavior, they are not routinely shared and reused. Objective: Ontologies have the potential to promote the reuse of CDSS rules. Therefore, we systematically screened the literature to elaborate on the current status of ontologies applied in CDSS rules, such as rule management, which uses captured CDSS rule usage data and user feedback data to tailor CDSS services to be more accurate, and maintenance, which updates CDSS rules. Through this systematic literature review, we aim to identify the frontiers of ontologies used in CDSS rules. Methods: The literature search was focused on the intersection of ontologies; clinical decision support; and rules in PubMed, the Association for Computing Machinery (ACM) Digital Library, and the Nursing & Allied Health Database. Grounded theory and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines were followed. One author initiated the screening and literature review, while 2 authors validated the processes and results independently. The inclusion and exclusion criteria were developed and refined iteratively. Results: CDSSs were primarily used to manage chronic conditions, alerts for medication prescriptions, reminders for immunizations and preventive services, diagnoses, and treatment recommendations among 81 included publications. The CDSS rules were presented in Semantic Web Rule Language, Jess, or Jena formats. Despite the fact that ontologies have been used to provide medical knowledge, CDSS rules, and terminologies, they have not been used in CDSS rule management or to facilitate the reuse of CDSS rules. Conclusions: Ontologies have been used to organize and represent medical knowledge, controlled vocabularies, and the content of CDSS rules. So far, there has been little reuse of CDSS rules. More work is needed to improve the reusability and interoperability of CDSS rules. This review identified and described the ontologies that, despite their limitations, enable Semantic Web technologies and their applications in CDSS rules. %M 36534739 %R 10.2196/43053 %U https://medinform.jmir.org/2023/1/e43053 %U https://doi.org/10.2196/43053 %U http://www.ncbi.nlm.nih.gov/pubmed/36534739 %0 Journal Article %@ 2369-2960 %I JMIR Publications %V 8 %N 12 %P e24938 %T Drug Abuse Ontology to Harness Web-Based Data for Substance Use Epidemiology Research: Ontology Development Study %A Lokala,Usha %A Lamy,Francois %A Daniulaityte,Raminta %A Gaur,Manas %A Gyrard,Amelie %A Thirunarayan,Krishnaprasad %A Kursuncu,Ugur %A Sheth,Amit %+ AI Institute, University of South Carolina, 1112 Greene St, Columbia, SC, 29208, United States, 1 803 777 9707, nlokala@email.sc.edu %K ontology %K knowledge graph %K semantic web %K illicit drugs %K cryptomarket %K social media %D 2022 %7 23.12.2022 %9 Original Paper %J JMIR Public Health Surveill %G English %X Background: Web-based resources and social media platforms play an increasingly important role in health-related knowledge and experience sharing. There is a growing interest in the use of these novel data sources for epidemiological surveillance of substance use behaviors and trends. Objective: The key aims were to describe the development and application of the drug abuse ontology (DAO) as a framework for analyzing web-based and social media data to inform public health and substance use research in the following areas: determining user knowledge, attitudes, and behaviors related to nonmedical use of buprenorphine and illicitly manufactured opioids through the analysis of web forum data Prescription Drug Abuse Online Surveillance; analyzing patterns and trends of cannabis product use in the context of evolving cannabis legalization policies in the United States through analysis of Twitter and web forum data (eDrugTrends); assessing trends in the availability of novel synthetic opioids through the analysis of cryptomarket data (eDarkTrends); and analyzing COVID-19 pandemic trends in social media data related to 13 states in the United States as per Mental Health America reports. Methods: The domain and scope of the DAO were defined using competency questions from popular ontology methodology (101 ontology development). The 101 method includes determining the domain and scope of ontology, reusing existing knowledge, enumerating important terms in ontology, defining the classes, their properties and creating instances of the classes. The quality of the ontology was evaluated using a set of tools and best practices recognized by the semantic web community and the artificial intelligence community that engage in natural language processing. Results: The current version of the DAO comprises 315 classes, 31 relationships, and 814 instances among the classes. The ontology is flexible and can easily accommodate new concepts. The integration of the ontology with machine learning algorithms dramatically decreased the false alarm rate by adding external knowledge to the machine learning process. The ontology is recurrently updated to capture evolving concepts in different contexts and applied to analyze data related to social media and dark web marketplaces. Conclusions: The DAO provides a powerful framework and a useful resource that can be expanded and adapted to a wide range of substance use and mental health domains to help advance big data analytics of web-based data for substance use epidemiology research. %M 36563032 %R 10.2196/24938 %U https://publichealth.jmir.org/2022/12/e24938 %U https://doi.org/10.2196/24938 %U http://www.ncbi.nlm.nih.gov/pubmed/36563032 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 12 %P e40534 %T Natural Language Processing and Graph Theory: Making Sense of Imaging Records in a Novel Representation Frame %A Binsfeld Gonçalves,Laurent %A Nesic,Ivan %A Obradovic,Marko %A Stieltjes,Bram %A Weikert,Thomas %A Bremerich,Jens %+ Clinic of Radiology & Nuclear Medicine, University Hospital Basel, University of Basel, Petersgraben, 4, Basel, 4031, Switzerland, 352 621517916, laurent.binsfeld@gmail.com %K radiology %K deep learning %K NLP %K radiology reports %K imaging record %K temporal referrals %K date extraction %K graph theory %K health care information system %K resource planning. %D 2022 %7 21.12.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: A concise visualization framework of related reports would increase readability and improve patient management. To this end, temporal referrals to prior comparative exams are an essential connection to previous exams in written reports. Due to unstructured narrative texts' variable structure and content, their extraction is hampered by poor computer readability. Natural language processing (NLP) permits the extraction of structured information from unstructured texts automatically and can serve as an essential input for such a novel visualization framework. Objective: This study proposes and evaluates an NLP-based algorithm capable of extracting the temporal referrals in written radiology reports, applies it to all the radiology reports generated for 10 years, introduces a graphical representation of imaging reports, and investigates its benefits for clinical and research purposes. Methods: In this single-center, university hospital, retrospective study, we developed a convolutional neural network capable of extracting the date of referrals from imaging reports. The model's performance was assessed by calculating precision, recall, and F1-score using an independent test set of 149 reports. Next, the algorithm was applied to our department's radiology reports generated from 2011 to 2021. Finally, the reports and their metadata were represented in a modulable graph. Results: For extracting the date of referrals, the named-entity recognition (NER) model had a high precision of 0.93, a recall of 0.95, and an F1-score of 0.94. A total of 1,684,635 reports were included in the analysis. Temporal reference was mentioned in 53.3% (656,852/1,684,635), explicitly stated as not available in 21.0% (258,386/1,684,635), and omitted in 25.7% (317,059/1,684,635) of the reports. Imaging records can be visualized in a directed and modulable graph, in which the referring links represent the connecting arrows. Conclusions: Automatically extracting the date of referrals from unstructured radiology reports using deep learning NLP algorithms is feasible. Graphs refined the selection of distinct pathology pathways, facilitated the revelation of missing comparisons, and enabled the query of specific referring exam sequences. Further work is needed to evaluate its benefits in clinics, research, and resource planning. %M 36542426 %R 10.2196/40534 %U https://medinform.jmir.org/2022/12/e40534 %U https://doi.org/10.2196/40534 %U http://www.ncbi.nlm.nih.gov/pubmed/36542426 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 12 %P e42664 %T Telehealth System Based on the Ontology Design of a Diabetes Management Pathway Model in China: Development and Usability Study %A Fan,ZhiYuan %A Cui,LiYuan %A Ye,Ying %A Li,ShouCheng %A Deng,Ning %+ College of Biomedical Engineering and Instrument Science, Ministry of Education Key Laboratory of Biomedical Engineering, Zhejiang University, Zhouyiqing Bldg 512, 38 Zheda Rd, Hangzhou, 310000, China, 86 571 2295 2693, zju.dengning@gmail.com %K diabetes %K chronic disease management %K Chronic Disease Management Pathway %K ontology %K Semantic Web Rule Language rules %K SWRL rules %D 2022 %7 19.12.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Diabetes needs to be under control through management and intervention. Management of diabetes through mobile health is a practical approach; however, most diabetes mobile health management systems do not meet expectations, which may be because of the lack of standardized management processes in the systems and the lack of intervention implementation recommendations in the management knowledge base. Objective: In this study, we aimed to construct a diabetes management care pathway suitable for the actual situation in China to express the diabetes management care pathway using ontology and develop a diabetes closed-loop system based on the construction results of the diabetes management pathway and apply it practically. Methods: This study proposes a diabetes management care pathway model in which the management process of diabetes is divided into 9 management tasks, and the Diabetes Care Pathway Ontology (DCPO) is constructed to represent the knowledge contained in this pathway model. A telehealth system, which can support the comprehensive management of patients with diabetes while providing active intervention by physicians, was designed and developed based on the DCPO. A retrospective study was performed based on the data records extracted from the system to analyze the usability and treatment effects of the DCPO. Results: The diabetes management pathway ontology constructed in this study contains 119 newly added classes, 28 object properties, 58 data properties, 81 individuals, 426 axioms, and 192 Semantic Web Rule Language rules. The developed mobile medical system was applied to 272 patients with diabetes. Within 3 months, the average fasting blood glucose of the patients decreased by 1.34 mmol/L (P=.003), and the average 2-hour postprandial blood glucose decreased by 2.63 mmol/L (P=.003); the average systolic and diastolic blood pressures decreased by 11.84 mmHg (P=.02) and 8.8 mmHg (P=.02), respectively. In patients who received physician interventions owing to abnormal attention or low-compliance warnings, the average fasting blood glucose decreased by 2.45 mmol/L (P=.003), and the average 2-hour postprandial blood glucose decreased by 2.89 mmol/L (P=.003) in all patients with diabetes; the average systolic and diastolic blood pressure decreased by 20.06 mmHg (P=.02) and 17.37 mmHg (P=.02), respectively, in patients with both hypertension and diabetes during the 3-month management period. Conclusions: This study helps guide the timing and content of interactive interventions between physicians and patients and regulates physicians’ medical service behavior. Different management plans are formulated for physicians and patients according to different characteristics to comprehensively manage various cardiovascular risk factors. The application of the DCPO in the diabetes management system can provide effective and adequate management support for patients with diabetes and those with both diabetes and hypertension. %M 36534448 %R 10.2196/42664 %U https://medinform.jmir.org/2022/12/e42664 %U https://doi.org/10.2196/42664 %U http://www.ncbi.nlm.nih.gov/pubmed/36534448 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 12 %P e42379 %T Construction of Cohorts of Similar Patients From Automatic Extraction of Medical Concepts: Phenotype Extraction Study %A Gérardin,Christel %A Mageau,Arthur %A Mékinian,Arsène %A Tannier,Xavier %A Carrat,Fabrice %+ Institute Pierre Louis Epidemiology and Public Health, Institut National de la Santé et de la Recherche Médicale, Sorbonne Université, 27 rue de Chaligny, Paris, 75012, France, 33 678148466, christel.ducroz-gerardin@iplesp.upmc.fr %K natural language processing %K similar patient cohort %K phenotype %K systemic disease %K NLP %K algorithm %K automatic extraction %K automated extraction %K named entity %K MeSH %K medical subject heading %K data extraction %K text extraction %D 2022 %7 19.12.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Reliable and interpretable automatic extraction of clinical phenotypes from large electronic medical record databases remains a challenge, especially in a language other than English. Objective: We aimed to provide an automated end-to-end extraction of cohorts of similar patients from electronic health records for systemic diseases. Methods: Our multistep algorithm includes a named-entity recognition step, a multilabel classification using medical subject headings ontology, and the computation of patient similarity. A selection of cohorts of similar patients on a priori annotated phenotypes was performed. Six phenotypes were selected for their clinical significance: P1, osteoporosis; P2, nephritis in systemic erythematosus lupus; P3, interstitial lung disease in systemic sclerosis; P4, lung infection; P5, obstetric antiphospholipid syndrome; and P6, Takayasu arteritis. We used a training set of 151 clinical notes and an independent validation set of 256 clinical notes, with annotated phenotypes, both extracted from the Assistance Publique-Hôpitaux de Paris data warehouse. We evaluated the precision of the 3 patients closest to the index patient for each phenotype with precision-at-3 and recall and average precision. Results: For P1-P4, the precision-at-3 ranged from 0.85 (95% CI 0.75-0.95) to 0.99 (95% CI 0.98-1), the recall ranged from 0.53 (95% CI 0.50-0.55) to 0.83 (95% CI 0.81-0.84), and the average precision ranged from 0.58 (95% CI 0.54-0.62) to 0.88 (95% CI 0.85-0.90). P5-P6 phenotypes could not be analyzed due to the limited number of phenotypes. Conclusions: Using a method close to clinical reasoning, we built a scalable and interpretable end-to-end algorithm for extracting cohorts of similar patients. %M 36534446 %R 10.2196/42379 %U https://medinform.jmir.org/2022/12/e42379 %U https://doi.org/10.2196/42379 %U http://www.ncbi.nlm.nih.gov/pubmed/36534446 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 12 %P e41312 %T Pan-Canadian Electronic Medical Record Diagnostic and Unstructured Text Data for Capturing PTSD: Retrospective Observational Study %A Kosowan,Leanne %A Singer,Alexander %A Zulkernine,Farhana %A Zafari,Hasan %A Nesca,Marcello %A Muthumuni,Dhasni %+ Department of Family Medicine, Rady Faculty of Health Sciences, University of Manitoba, D009-780 Bannatyne Ave., Winnipeg, MB, R3E0W2, Canada, 1 204 789 3314, alexander.singer@umanitoba.ca %K electronic health records %K EHR %K natural language processing %K NLP %K medical informatics %K primary health care %K stress disorders, posttraumatic %K posttraumatic stress disorder %K PTSD %D 2022 %7 13.12.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: The availability of electronic medical record (EMR) free-text data for research varies. However, access to short diagnostic text fields is more widely available. Objective: This study assesses agreement between free-text and short diagnostic text data from primary care EMR for identification of posttraumatic stress disorder (PTSD). Methods: This retrospective cross-sectional study used EMR data from a pan-Canadian repository representing 1574 primary care providers at 265 clinics using 11 EMR vendors. Medical record review using free text and short diagnostic text fields of the EMR produced reference standards for PTSD. Agreement was assessed with sensitivity, specificity, positive predictive value, negative predictive value, and accuracy. Results: Our reference set contained 327 patients with free text and short diagnostic text. Among these patients, agreement between free text and short diagnostic text had an accuracy of 93.6% (CI 90.4%-96.0%). In a single Canadian province, case definitions 1 and 4 had a sensitivity of 82.6% (CI 74.4%-89.0%) and specificity of 99.5% (CI 97.4%-100%). However, when the reference set was expanded to a pan-Canada reference (n=12,104 patients), case definition 4 had the strongest agreement (sensitivity: 91.1%, CI 90.1%-91.9%; specificity: 99.1%, CI 98.9%-99.3%). Conclusions: Inclusion of free-text encounter notes during medical record review did not lead to improved capture of PTSD cases, nor did it lead to significant changes in case definition agreement. Within this pan-Canadian database, jurisdictional differences in diagnostic codes and EMR structure suggested the need to supplement diagnostic codes with natural language processing to capture PTSD. When unavailable, short diagnostic text can supplement free-text data for reference set creation and case validation. Application of the PTSD case definition can inform PTSD prevalence and characteristics. %M 36512389 %R 10.2196/41312 %U https://medinform.jmir.org/2022/12/e41312 %U https://doi.org/10.2196/41312 %U http://www.ncbi.nlm.nih.gov/pubmed/36512389 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 6 %N 11 %P e38677 %T Extraction and Quantification of Words Representing Degrees of Diseases: Combining the Fuzzy C-Means Method and Gaussian Membership %A Han,Feng %A Zhang,ZiHeng %A Zhang,Hongjian %A Nakaya,Jun %A Kudo,Kohsuke %A Ogasawara,Katsuhiko %+ Graduate School of Health Sciences, Medical Management and Informatics, Hokkaido University, N12 W5, Sapporo, 060-0812, Japan, 81 011 706 3409, oga@hs.hokudai.ac.jp %K medical text %K fuzzy c-means %K cluster %K algorithm %K machine learning %K word quantification %K fuzzification %K Gauss %K radiology %K medical report %K documentation %K text mining %K data mining %K extraction %K unstructured %K free text %K quantification %K fuzzy %K diagnosis %K diagnostic %K EHR %K support system %D 2022 %7 18.11.2022 %9 Original Paper %J JMIR Form Res %G English %X Background: Due to the development of medical data, a large amount of clinical data has been generated. These unstructured data contain substantial information. Extracting useful knowledge from this data and making scientific decisions for diagnosing and treating diseases have become increasingly necessary. Unstructured data, such as in the Marketplace for Medical Information in Intensive Care III (MIMIC-III) data set, contain several ambiguous words that demonstrate the subjectivity of doctors, such as descriptions of patient symptoms. These data could be used to further improve the accuracy of medical diagnostic system assessments. To the best of our knowledge, there is currently no method for extracting subjective words that express the extent of these symptoms (hereinafter, “degree words”). Objective: Therefore, we propose using the fuzzy c-means (FCM) method and Gaussian membership to quantify the degree words in the clinical medical data set MIMIC-III. Methods: First, we preprocessed the 381,091 radiology reports collected in MIMIC-III, and then we used the FCM method to extract degree words from unstructured text. Thereafter, we used the Gaussian membership method to quantify the extracted degree words, which transform the fuzzy words extracted from the medical text into computer-recognizable numbers. Results: The results showed that the digitization of ambiguous words in medical texts is feasible. The words representing each degree of each disease had a range of corresponding values. Examples of membership medians were 2.971 (atelectasis), 3.121 (pneumonia), 2.899 (pneumothorax), 3.051 (pulmonary edema), and 2.435 (pulmonary embolus). Additionally, all extracted words contained the same subjective words (low, high, etc), which allows for an objective evaluation method. Furthermore, we will verify the specific impact of the quantification results of ambiguous words such as symptom words and degree words on the use of medical texts in subsequent studies. These same ambiguous words may be used as a new set of feature values to represent the disorders. Conclusions: This study proposes an innovative method for handling subjective words. We used the FCM method to extract the subjective degree words in the English-interpreted report of the MIMIC-III and then used the Gaussian functions to quantify the subjective degree words. In this method, words containing subjectivity in unstructured texts can be automatically processed and transformed into numerical ranges by digital processing. It was concluded that the digitization of ambiguous words in medical texts is feasible. %M 36399376 %R 10.2196/38677 %U https://formative.jmir.org/2022/11/e38677 %U https://doi.org/10.2196/38677 %U http://www.ncbi.nlm.nih.gov/pubmed/36399376 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 11 %P e41342 %T Training a Deep Contextualized Language Model for International Classification of Diseases, 10th Revision Classification via Federated Learning: Model Development and Validation Study %A Chen,Pei-Fu %A He,Tai-Liang %A Lin,Sheng-Che %A Chu,Yuan-Chia %A Kuo,Chen-Tsung %A Lai,Feipei %A Wang,Ssu-Ming %A Zhu,Wan-Xuan %A Chen,Kuan-Chih %A Kuo,Lu-Cheng %A Hung,Fang-Ming %A Lin,Yu-Cheng %A Tsai,I-Chang %A Chiu,Chi-Hao %A Chang,Shu-Chih %A Yang,Chi-Yu %+ Department of Information Technology, Far Eastern Memorial Hospital, No 21, Section 2, Nanya S Rd, Banciao District, New Taipei City, 220216, Taiwan, 886 2 8966 7000, chiyuyang1959@gmail.com %K federated learning %K International Classification of Diseases %K machine learning %K natural language processing %K multilabel text classification %D 2022 %7 10.11.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: The automatic coding of clinical text documents by using the International Classification of Diseases, 10th Revision (ICD-10) can be performed for statistical analyses and reimbursements. With the development of natural language processing models, new transformer architectures with attention mechanisms have outperformed previous models. Although multicenter training may increase a model’s performance and external validity, the privacy of clinical documents should be protected. We used federated learning to train a model with multicenter data, without sharing data per se. Objective: This study aims to train a classification model via federated learning for ICD-10 multilabel classification. Methods: Text data from discharge notes in electronic medical records were collected from the following three medical centers: Far Eastern Memorial Hospital, National Taiwan University Hospital, and Taipei Veterans General Hospital. After comparing the performance of different variants of bidirectional encoder representations from transformers (BERT), PubMedBERT was chosen for the word embeddings. With regard to preprocessing, the nonalphanumeric characters were retained because the model’s performance decreased after the removal of these characters. To explain the outputs of our model, we added a label attention mechanism to the model architecture. The model was trained with data from each of the three hospitals separately and via federated learning. The models trained via federated learning and the models trained with local data were compared on a testing set that was composed of data from the three hospitals. The micro F1 score was used to evaluate model performance across all 3 centers. Results: The F1 scores of PubMedBERT, RoBERTa (Robustly Optimized BERT Pretraining Approach), ClinicalBERT, and BioBERT (BERT for Biomedical Text Mining) were 0.735, 0.692, 0.711, and 0.721, respectively. The F1 score of the model that retained nonalphanumeric characters was 0.8120, whereas the F1 score after removing these characters was 0.7875—a decrease of 0.0245 (3.11%). The F1 scores on the testing set were 0.6142, 0.4472, 0.5353, and 0.2522 for the federated learning, Far Eastern Memorial Hospital, National Taiwan University Hospital, and Taipei Veterans General Hospital models, respectively. The explainable predictions were displayed with highlighted input words via the label attention architecture. Conclusions: Federated learning was used to train the ICD-10 classification model on multicenter clinical text while protecting data privacy. The model’s performance was better than that of models that were trained locally. %M 36355417 %R 10.2196/41342 %U https://medinform.jmir.org/2022/11/e41342 %U https://doi.org/10.2196/41342 %U http://www.ncbi.nlm.nih.gov/pubmed/36355417 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 11 %P e36711 %T Linking Biomedical Data Warehouse Records With the National Mortality Database in France: Large-scale Matching Algorithm %A Guardiolle,Vianney %A Bazoge,Adrien %A Morin,Emmanuel %A Daille,Béatrice %A Toublant,Delphine %A Bouzillé,Guillaume %A Merel,Youenn %A Pierre-Jean,Morgane %A Filiot,Alexandre %A Cuggia,Marc %A Wargny,Matthieu %A Lamer,Antoine %A Gourraud,Pierre-Antoine %+ Univ. Lille, CHU Lille, ULR 2694, METRICS: Évaluation des Technologies de santé et des Pratiques médicales, F-59000, 1 place de Verdun, Lille, 59000, France, 33 320626969, antoine.lamer@univ-lille.fr %K data warehousing %K clinical data warehouse %K medical informatics applications %K medical record linkage %K French National Mortality Database %K data reuse %K open data, R %K clinical informatics %D 2022 %7 1.11.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Often missing from or uncertain in a biomedical data warehouse (BDW), vital status after discharge is central to the value of a BDW in medical research. The French National Mortality Database (FNMD) offers open-source nominative records of every death. Matching large-scale BDWs records with the FNMD combines multiple challenges: absence of unique common identifiers between the 2 databases, names changing over life, clerical errors, and the exponential growth of the number of comparisons to compute. Objective: We aimed to develop a new algorithm for matching BDW records to the FNMD and evaluated its performance. Methods: We developed a deterministic algorithm based on advanced data cleaning and knowledge of the naming system and the Damerau-Levenshtein distance (DLD). The algorithm’s performance was independently assessed using BDW data of 3 university hospitals: Lille, Nantes, and Rennes. Specificity was evaluated with living patients on January 1, 2016 (ie, patients with at least 1 hospital encounter before and after this date). Sensitivity was evaluated with patients recorded as deceased between January 1, 2001, and December 31, 2020. The DLD-based algorithm was compared to a direct matching algorithm with minimal data cleaning as a reference. Results: All centers combined, sensitivity was 11% higher for the DLD-based algorithm (93.3%, 95% CI 92.8-93.9) than for the direct algorithm (82.7%, 95% CI 81.8-83.6; P<.001). Sensitivity was superior for men at 2 centers (Nantes: 87%, 95% CI 85.1-89 vs 83.6%, 95% CI 81.4-85.8; P=.006; Rennes: 98.6%, 95% CI 98.1-99.2 vs 96%, 95% CI 94.9-97.1; P<.001) and for patients born in France at all centers (Nantes: 85.8%, 95% CI 84.3-87.3 vs 74.9%, 95% CI 72.8-77.0; P<.001). The DLD-based algorithm revealed significant differences in sensitivity among centers (Nantes, 85.3% vs Lille and Rennes, 97.3%, P<.001). Specificity was >98% in all subgroups. Our algorithm matched tens of millions of death records from BDWs, with parallel computing capabilities and low RAM requirements. We used the Inseehop open-source R script for this measurement. Conclusions: Overall, sensitivity/recall was 11% higher using the DLD-based algorithm than that using the direct algorithm. This shows the importance of advanced data cleaning and knowledge of a naming system through DLD use. Statistically significant differences in sensitivity between groups could be found and must be considered when performing an analysis to avoid differential biases. Our algorithm, originally conceived for linking a BDW with the FNMD, can be used to match any large-scale databases. While matching operations using names are considered sensitive computational operations, the Inseehop package released here is easy to run on premises, thereby facilitating compliance with cybersecurity local framework. The use of an advanced deterministic matching algorithm such as the DLD-based algorithm is an insightful example of combining open-source external data to improve the usage value of BDWs. %M 36318244 %R 10.2196/36711 %U https://medinform.jmir.org/2022/11/e36711 %U https://doi.org/10.2196/36711 %U http://www.ncbi.nlm.nih.gov/pubmed/36318244 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 10 %P e39616 %T Evaluating the Impact on Clinical Task Efficiency of a Natural Language Processing Algorithm for Searching Medical Documents: Prospective Crossover Study %A Park,Eunsoo H %A Watson,Hannah I %A Mehendale,Felicity V %A O'Neil,Alison Q %A , %+ Edinburgh Medical School, College of Medicine and Veterinary Medicine, University of Edinburgh, The Chancellor's Building, 49 Little France Crescent, Edinburgh, EH16 4SB, United Kingdom, 44 1312426792, e.park-7@sms.ed.ac.uk %K clinical decision support %K electronic health records %K natural language processing %K semantic search %K clinical informatics %D 2022 %7 26.10.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Information retrieval (IR) from the free text within electronic health records (EHRs) is time consuming and complex. We hypothesize that natural language processing (NLP)–enhanced search functionality for EHRs can make clinical workflows more efficient and reduce cognitive load for clinicians. Objective: This study aimed to evaluate the efficacy of 3 levels of search functionality (no search, string search, and NLP-enhanced search) in supporting IR for clinical users from the free text of EHR documents in a simulated clinical environment. Methods: A clinical environment was simulated by uploading 3 sets of patient notes into an EHR research software application and presenting these alongside 3 corresponding IR tasks. Tasks contained a mixture of multiple-choice and free-text questions. A prospective crossover study design was used, for which 3 groups of evaluators were recruited, which comprised doctors (n=19) and medical students (n=16). Evaluators performed the 3 tasks using each of the search functionalities in an order in accordance with their randomly assigned group. The speed and accuracy of task completion were measured and analyzed, and user perceptions of NLP-enhanced search were reviewed in a feedback survey. Results: NLP-enhanced search facilitated more accurate task completion than both string search (5.14%; P=.02) and no search (5.13%; P=.08). NLP-enhanced search and string search facilitated similar task speeds, both showing an increase in speed compared to the no search function, by 11.5% (P=.008) and 16.0% (P=.007) respectively. Overall, 93% of evaluators agreed that NLP-enhanced search would make clinical workflows more efficient than string search, with qualitative feedback reporting that NLP-enhanced search reduced cognitive load. Conclusions: To the best of our knowledge, this study is the largest evaluation to date of different search functionalities for supporting target clinical users in realistic clinical workflows, with a 3-way prospective crossover study design. NLP-enhanced search improved both accuracy and speed of clinical EHR IR tasks compared to browsing clinical notes without search. NLP-enhanced search improved accuracy and reduced the number of searches required for clinical EHR IR tasks compared to direct search term matching. %M 36287591 %R 10.2196/39616 %U https://medinform.jmir.org/2022/10/e39616 %U https://doi.org/10.2196/39616 %U http://www.ncbi.nlm.nih.gov/pubmed/36287591 %0 Journal Article %@ 2561-7605 %I JMIR Publications %V 5 %N 4 %P e38464 %T Evolving Hybrid Partial Genetic Algorithm Classification Model for Cost-effective Frailty Screening: Investigative Study %A Oates,John %A Shafiabady,Niusha %A Ambagtsheer,Rachel %A Beilby,Justin %A Seiboth,Chris %A Dent,Elsa %+ College of Engineering, Information Technology and Environment, Charles Darwin University, 815 George Street, Haymarket, NSW, 2000, Australia, 61 80474147, niusha.shafiabady@cdu.edu.au %K machine learning %K frailty screening %K partial genetic algorithms %K SVM %K KNN %K decision trees %K frailty %K algorithm %K cost %K model %K index %K database %K ai %K ageing %K adults %K older people %K screening %K tool %D 2022 %7 7.10.2022 %9 Original Paper %J JMIR Aging %G English %X Background: A commonly used method for measuring frailty is the accumulation of deficits expressed as a frailty index (FI). FIs can be readily adapted to many databases, as the parameters to use are not prescribed but rather reflect a subset of extracted features (variables). Unfortunately, the structure of many databases does not permit the direct extraction of a suitable subset, requiring additional effort to determine and verify the value of features for each record and thus significantly increasing cost. Objective: Our objective is to describe how an artificial intelligence (AI) optimization technique called partial genetic algorithms can be used to refine the subset of features used to calculate an FI and favor features that have the least cost of acquisition. Methods: This is a secondary analysis of a residential care database compiled from 10 facilities in Queensland, Australia. The database is comprised of routinely collected administrative data and unstructured patient notes for 592 residents aged 75 years and over. The primary study derived an electronic frailty index (eFI) calculated from 36 suitable features. We then structurally modified a genetic algorithm to find an optimal predictor of the calculated eFI (0.21 threshold) from 2 sets of features. Partial genetic algorithms were used to optimize 4 underlying classification models: logistic regression, decision trees, random forest, and support vector machines. Results: Among the underlying models, logistic regression was found to produce the best models in almost all scenarios and feature set sizes. The best models were built using all the low-cost features and as few as 10 high-cost features, and they performed well enough (sensitivity 89%, specificity 87%) to be considered candidates for a low-cost frailty screening test. Conclusions: In this study, a systematic approach for selecting an optimal set of features with a low cost of acquisition and performance comparable to the eFI for detecting frailty was demonstrated on an aged care database. Partial genetic algorithms have proven useful in offering a trade-off between cost and accuracy to systematically identify frailty. %M 36206042 %R 10.2196/38464 %U https://aging.jmir.org/2022/4/e38464 %U https://doi.org/10.2196/38464 %U http://www.ncbi.nlm.nih.gov/pubmed/36206042 %0 Journal Article %@ 2561-7605 %I JMIR Publications %V 5 %N 3 %P e40241 %T Identifying Caregiver Availability Using Medical Notes With Rule-Based Natural Language Processing: Retrospective Cohort Study %A Mahmoudi,Elham %A Wu,Wenbo %A Najarian,Cyrus %A Aikens,James %A Bynum,Julie %A Vydiswaran,V G Vinod %+ Department of Family Medicine, Medical School, University of Michigan, NCRC Building 14, Room G234, 2800 Plymouth Rd, Ann Arbor, MI, 48109, United States, 1 248 765 3927, mahmoudi@med.umich.edu %K natural language processing %K caregiver %K medical notes %K Alzheimer %K dementia %K pragmatic %K aging %K care planning %K health care %K elderly care %K elderly population %K algorithm %D 2022 %7 22.9.2022 %9 Original Paper %J JMIR Aging %G English %X Background: Identifying caregiver availability, particularly for patients with dementia or those with a disability, is critical to informing the appropriate care planning by the health systems, hospitals, and providers. This information is not readily available, and there is a paucity of pragmatic approaches to automatically identifying caregiver availability and type. Objective: Our main objective was to use medical notes to assess caregiver availability and type for hospitalized patients with dementia. Our second objective was to identify whether the patient lived at home or resided at an institution. Methods: In this retrospective cohort study, we used 2016-2019 telephone-encounter medical notes from a single institution to develop a rule-based natural language processing (NLP) algorithm to identify the patient’s caregiver availability and place of residence. Using note-level data, we compared the results of the NLP algorithm with human-conducted chart abstraction for both training (749/976, 77%) and test sets (227/976, 23%) for a total of 223 adults aged 65 years and older diagnosed with dementia. Our outcomes included determining whether the patients (1) reside at home or in an institution, (2) have a formal caregiver, and (3) have an informal caregiver. Results: Test set results indicated that our NLP algorithm had high level of accuracy and reliability for identifying whether patients had an informal caregiver (F1=0.94, accuracy=0.95, sensitivity=0.97, and specificity=0.93), but was relatively less able to identify whether the patient lived at an institution (F1=0.64, accuracy=0.90, sensitivity=0.51, and specificity=0.98). The most common explanations for NLP misclassifications across all categories were (1) incomplete or misspelled facility names; (2) past, uncertain, or undecided status; (3) uncommon abbreviations; and (4) irregular use of templates. Conclusions: This innovative work was the first to use medical notes to pragmatically determine caregiver availability. Our NLP algorithm identified whether hospitalized patients with dementia have a formal or informal caregiver and, to a lesser extent, whether they lived at home or in an institutional setting. There is merit in using NLP to identify caregivers. This study serves as a proof of concept. Future work can use other approaches and further identify caregivers and the extent of their availability. %M 35998328 %R 10.2196/40241 %U https://aging.jmir.org/2022/3/e40241 %U https://doi.org/10.2196/40241 %U http://www.ncbi.nlm.nih.gov/pubmed/35998328 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 6 %N 9 %P e32460 %T Attention-Based Models for Classifying Small Data Sets Using Community-Engaged Research Protocols: Classification System Development and Validation Pilot Study %A Ferrell,Brian J %A Raskin,Sarah E %A Zimmerman,Emily B %A Timberline,David H %A McInnes,Bridget T %A Krist,Alex H %+ Center for Community Engagement and Impact, Virginia Commonwealth University, 907 Floyd Ave, Richmond, VA, 23284, United States, 1 8043505426, ferrellbj@vcu.edu %K data augmentation %K BERT %K transformer-based models %K text classification %K community engagement %K prototype %K IRB research %K community-engaged research %K participatory research %K deep learning %D 2022 %7 6.9.2022 %9 Original Paper %J JMIR Form Res %G English %X Background: Community-engaged research (CEnR) is a research approach in which scholars partner with community organizations or individuals with whom they share an interest in the study topic, typically with the goal of supporting that community’s well-being. CEnR is well-established in numerous disciplines including the clinical and social sciences. However, universities experience challenges reporting comprehensive CEnR metrics, limiting the development of appropriate CEnR infrastructure and the advancement of relationships with communities, funders, and stakeholders. Objective: We propose a novel approach to identifying and categorizing community-engaged studies by applying attention-based deep learning models to human participants protocols that have been submitted to the university’s institutional review board (IRB). Methods: We manually classified a sample of 280 protocols submitted to the IRB using a 3- and 6-level CEnR heuristic. We then trained an attention-based bidirectional long short-term memory unit (Bi-LSTM) on the classified protocols and compared it to transformer models such as Bidirectional Encoder Representations From Transformers (BERT), Bio + Clinical BERT, and Cross-lingual Language Model–Robustly Optimized BERT Pre-training Approach (XLM-RoBERTa). We applied the best-performing models to the full sample of unlabeled IRB protocols submitted in the years 2013-2019 (n>6000). Results: Although transfer learning is superior, receiving a 0.9952 evaluation F1 score for all transformer models implemented compared to the attention-based Bi-LSTM (between 48%-80%), there were key issues with overfitting. This finding is consistent across several methodological adjustments: an augmented data set with and without cross-validation, an unaugmented data set with and without cross-validation, a 6-class CEnR spectrum, and a 3-class one. Conclusions: Transfer learning is a more viable method than the attention-based bidirectional-LSTM for differentiating small data sets characterized by the idiosyncrasies and variability of CEnR descriptions used by principal investigators in research protocols. Despite these issues involving overfitting, BERT and the other transformer models remarkably showed an understanding of our data unlike the attention-based Bi-LSTM model, promising a more realistic path toward solving this real-world application. %M 36066925 %R 10.2196/32460 %U https://formative.jmir.org/2022/9/e32460 %U https://doi.org/10.2196/32460 %U http://www.ncbi.nlm.nih.gov/pubmed/36066925 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 8 %P e39057 %T Standard Vocabularies to Improve Machine Learning Model Transferability With Electronic Health Record Data: Retrospective Cohort Study Using Health Care–Associated Infection %A Kiser,Amber C %A Eilbeck,Karen %A Ferraro,Jeffrey P %A Skarda,David E %A Samore,Matthew H %A Bucher,Brian %+ Department of Biomedical Informatics, School of Medicine, University of Utah, 421 Wakara Way, Suite 140, Salt Lake City, UT, 84108, United States, 1 801 581 4080, amber.kiser@utah.edu %K standard vocabularies %K machine learning %K electronic health records %K model transferability %K data heterogeneity %K machine learning %D 2022 %7 30.8.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: With the widespread adoption of electronic healthcare records (EHRs) by US hospitals, there is an opportunity to leverage this data for the development of predictive algorithms to improve clinical care. A key barrier in model development and implementation includes the external validation of model discrimination, which is rare and often results in worse performance. One reason why machine learning models are not externally generalizable is data heterogeneity. A potential solution to address the substantial data heterogeneity between health care systems is to use standard vocabularies to map EHR data elements. The advantage of these vocabularies is a hierarchical relationship between elements, which allows the aggregation of specific clinical features to more general grouped concepts. Objective: This study aimed to evaluate grouping EHR data using standard vocabularies to improve the transferability of machine learning models for the detection of postoperative health care–associated infections across institutions with different EHR systems. Methods: Patients who underwent surgery from the University of Utah Health and Intermountain Healthcare from July 2014 to August 2017 with complete follow-up data were included. The primary outcome was a health care–associated infection within 30 days of the procedure. EHR data from 0-30 days after the operation were mapped to standard vocabularies and grouped using the hierarchical relationships of the vocabularies. Model performance was measured using the area under the receiver operating characteristic curve (AUC) and F1-score in internal and external validations. To evaluate model transferability, a difference-in-difference metric was defined as the difference in performance drop between internal and external validations for the baseline and grouped models. Results: A total of 5775 patients from the University of Utah and 15,434 patients from Intermountain Healthcare were included. The prevalence of selected outcomes was from 4.9% (761/15,434) to 5% (291/5775) for surgical site infections, from 0.8% (44/5775) to 1.1% (171/15,434) for pneumonia, from 2.6% (400/15,434) to 3% (175/5775) for sepsis, and from 0.8% (125/15,434) to 0.9% (50/5775) for urinary tract infections. In all outcomes, the grouping of data using standard vocabularies resulted in a reduced drop in AUC and F1-score in external validation compared to baseline features (all P<.001, except urinary tract infection AUC: P=.002). The difference-in-difference metrics ranged from 0.005 to 0.248 for AUC and from 0.075 to 0.216 for F1-score. Conclusions: We demonstrated that grouping machine learning model features based on standard vocabularies improved model transferability between data sets across 2 institutions. Improving model transferability using standard vocabularies has the potential to improve the generalization of clinical prediction models across the health care system. %M 36040784 %R 10.2196/39057 %U https://medinform.jmir.org/2022/8/e39057 %U https://doi.org/10.2196/39057 %U http://www.ncbi.nlm.nih.gov/pubmed/36040784 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 6 %N 8 %P e37821 %T Methodological Issues in Using a Common Data Model of COVID-19 Vaccine Uptake and Important Adverse Events of Interest: Feasibility Study of Data and Connectivity COVID-19 Vaccines Pharmacovigilance in the United Kingdom %A Delanerolle,Gayathri %A Williams,Robert %A Stipancic,Ana %A Byford,Rachel %A Forbes,Anna %A Tsang,Ruby S M %A Anand,Sneha N %A Bradley,Declan %A Murphy,Siobhán %A Akbari,Ashley %A Bedston,Stuart %A Lyons,Ronan A %A Owen,Rhiannon %A Torabi,Fatemeh %A Beggs,Jillian %A Chuter,Antony %A Balharry,Dominique %A Joy,Mark %A Sheikh,Aziz %A Hobbs,F D Richard %A de Lusignan,Simon %+ Nuffield Department of Primary Care Health Sciences, University of Oxford, Eagle House, Walton Road, Oxford, United Kingdom, 44 01865 617283, simon.delusignan@phc.ox.ac.uk %K Systematized Nomenclature of Medicine %K COVID-19 vaccines %K COVID-19 %K sinus thrombosis %K anaphylaxis %K pharmacovigilance %K vaccine uptake %K medical outcome %K clinical coding system %K health database %K health information %K clinical outcome %K vaccine effect %K data model %D 2022 %7 22.8.2022 %9 Original Paper %J JMIR Form Res %G English %X Background: The Data and Connectivity COVID-19 Vaccines Pharmacovigilance (DaC-VaP) UK-wide collaboration was created to monitor vaccine uptake and effectiveness and provide pharmacovigilance using routine clinical and administrative data. To monitor these, pooled analyses may be needed. However, variation in terminologies present a barrier as England uses the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT), while the rest of the United Kingdom uses the Read v2 terminology in primary care. The availability of data sources is not uniform across the United Kingdom. Objective: This study aims to use the concept mappings in the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to identify common concepts recorded and to report these in a repeated cross-sectional study. We planned to do this for vaccine coverage and 2 adverse events of interest (AEIs), cerebral venous sinus thrombosis (CVST) and anaphylaxis. We identified concept mappings to SNOMED CT, Read v2, the World Health Organization’s International Classification of Disease Tenth Revision (ICD-10) terminology, and the UK Dictionary of Medicines and Devices (dm+d). Methods: Exposures and outcomes of interest to DaC-VaP for pharmacovigilance studies were selected. Mappings of these variables to different terminologies used across the United Kingdom’s devolved nations’ health services were identified from the Observational Health Data Sciences and Informatics (OHDSI) Automated Terminology Harmonization, Extraction, and Normalization for Analytics (ATHENA) online browser. Lead analysts from each nation then confirmed or added to the mappings identified. These mappings were then used to report AEIs in a common format. We reported rates for windows of 0-2 and 3-28 days postvaccine every 28 days. Results: We listed the mappings between Read v2, SNOMED CT, ICD-10, and dm+d. For vaccine exposure, we found clear mapping from OMOP to our clinical terminologies, though dm+d had codes not listed by OMOP at the time of searching. We found a list of CVST and anaphylaxis codes. For CVST, we had to use a broader cerebral venous thrombosis conceptual approach to include Read v2. We identified 56 SNOMED CT codes, of which we selected 47 (84%), and 15 Read v2 codes. For anaphylaxis, our refined search identified 60 SNOMED CT codes and 9 Read v2 codes, of which we selected 10 (17%) and 4 (44%), respectively, to include in our repeated cross-sectional studies. Conclusions: This approach enables the use of mappings to different terminologies within the OMOP CDM without the need to catalogue an entire database. However, Read v2 has less granular concepts than some terminologies, such as SNOMED CT. Additionally, the OMOP CDM cannot compensate for limitations in the clinical coding system. Neither Read v2 nor ICD-10 is sufficiently granular to enable CVST to be specifically flagged. Hence, any pooled analysis will have to be at the less specific level of cerebrovascular venous thrombosis. Overall, the mappings within this CDM are useful, and our method could be used for rapid collaborations where there are only a limited number of concepts to pool. %M 35786634 %R 10.2196/37821 %U https://formative.jmir.org/2022/8/e37821 %U https://doi.org/10.2196/37821 %U http://www.ncbi.nlm.nih.gov/pubmed/35786634 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 24 %N 8 %P e39888 %T Deciphering the Diversity of Mental Models in Neurodevelopmental Disorders: Knowledge Graph Representation of Public Data Using Natural Language Processing %A Kaur,Manpreet %A Costello,Jeremy %A Willis,Elyse %A Kelm,Karen %A Reformat,Marek Z %A Bolduc,Francois V %+ Department of Pediatrics, University of Alberta, 3-020 Katz Building, 11315 87 Avenue, Edmonton, AB, T6G 2E1, Canada, 1 780 492 9713, fbolduc@ualberta.ca %K concept map %K neurodevelopmental disorder %K knowledge graph %K text analysis %K semantic relatedness %K PubMed %K forums %K mental model %D 2022 %7 5.8.2022 %9 Original Paper %J J Med Internet Res %G English %X Background: Understanding how individuals think about a topic, known as the mental model, can significantly improve communication, especially in the medical domain where emotions and implications are high. Neurodevelopmental disorders (NDDs) represent a group of diagnoses, affecting up to 18% of the global population, involving differences in the development of cognitive or social functions. In this study, we focus on 2 NDDs, attention deficit hyperactivity disorder (ADHD) and autism spectrum disorder (ASD), which involve multiple symptoms and interventions requiring interactions between 2 important stakeholders: parents and health professionals. There is a gap in our understanding of differences between mental models for each stakeholder, making communication between stakeholders more difficult than it could be. Objective: We aim to build knowledge graphs (KGs) from web-based information relevant to each stakeholder as proxies of mental models. These KGs will accelerate the identification of shared and divergent concerns between stakeholders. The developed KGs can help improve knowledge mobilization, communication, and care for individuals with ADHD and ASD. Methods: We created 2 data sets by collecting the posts from web-based forums and PubMed abstracts related to ADHD and ASD. We utilized the Unified Medical Language System (UMLS) to detect biomedical concepts and applied Positive Pointwise Mutual Information followed by truncated Singular Value Decomposition to obtain corpus-based concept embeddings for each data set. Each data set is represented as a KG using a property graph model. Semantic relatedness between concepts is calculated to rank the relation strength of concepts and stored in the KG as relation weights. UMLS disorder-relevant semantic types are used to provide additional categorical information about each concept’s domain. Results: The developed KGs contain concepts from both data sets, with node sizes representing the co-occurrence frequency of concepts and edge sizes representing relevance between concepts. ADHD- and ASD-related concepts from different semantic types shows diverse areas of concerns and complex needs of the conditions. KG identifies converging and diverging concepts between health professionals literature (PubMed) and parental concerns (web-based forums), which may correspond to the differences between mental models for each stakeholder. Conclusions: We show for the first time that generating KGs from web-based data can capture the complex needs of families dealing with ADHD or ASD. Moreover, we showed points of convergence between families and health professionals’ KGs. Natural language processing–based KG provides access to a large sample size, which is often a limiting factor for traditional in-person mental model mapping. Our work offers a high throughput access to mental model maps, which could be used for further in-person validation, knowledge mobilization projects, and basis for communication about potential blind spots from stakeholders in interactions about NDDs. Future research will be needed to identify how concepts could interact together differently for each stakeholder. %M 35930346 %R 10.2196/39888 %U https://www.jmir.org/2022/8/e39888 %U https://doi.org/10.2196/39888 %U http://www.ncbi.nlm.nih.gov/pubmed/35930346 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 8 %P e37817 %T A Syntactic Information–Based Classification Model for Medical Literature: Algorithm Development and Validation Study %A Tang,Wentai %A Wang,Jian %A Lin,Hongfei %A Zhao,Di %A Xu,Bo %A Zhang,Yijia %A Yang,Zhihao %+ College of Computer Science and Technology, Dalian University of Technology, No 2 Linggong Road, Ganjingzi District, Dalian, 116023, China, 86 13604119266, wangjian@dlut.edu.cn %K medical relation extraction %K syntactic features %K pruning method %K neural networks %K medical literature %K medical text %K extraction %K syntactic %K classification %K interaction %K text %K literature %K semantic %D 2022 %7 2.8.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: The ever-increasing volume of medical literature necessitates the classification of medical literature. Medical relation extraction is a typical method of classifying a large volume of medical literature. With the development of arithmetic power, medical relation extraction models have evolved from rule-based models to neural network models. The single neural network model discards the shallow syntactic information while discarding the traditional rules. Therefore, we propose a syntactic information–based classification model that complements and equalizes syntactic information to enhance the model. Objective: We aim to complete a syntactic information–based relation extraction model for more efficient medical literature classification. Methods: We devised 2 methods for enhancing syntactic information in the model. First, we introduced shallow syntactic information into the convolutional neural network to enhance nonlocal syntactic interactions. Second, we devise a cross-domain pruning method to equalize local and nonlocal syntactic interactions. Results: We experimented with 3 data sets related to the classification of medical literature. The F1 values were 65.5% and 91.5% on the BioCreative ViCPR (CPR) and Phenotype-Gene Relationship data sets, respectively, and the accuracy was 88.7% on the PubMed data set. Our model outperforms the current state-of-the-art baseline model in the experiments. Conclusions: Our model based on syntactic information effectively enhances medical relation extraction. Furthermore, the results of the experiments show that shallow syntactic information helps obtain nonlocal interaction in sentences and effectively reinforces syntactic features. It also provides new ideas for future research directions. %M 35917162 %R 10.2196/37817 %U https://medinform.jmir.org/2022/8/e37817 %U https://doi.org/10.2196/37817 %U http://www.ncbi.nlm.nih.gov/pubmed/35917162 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 6 %N 8 %P e27990 %T A Personalized Ontology-Based Decision Support System for Complex Chronic Patients: Retrospective Observational Study %A Román-Villarán,Esther %A Alvarez-Romero,Celia %A Martínez-García,Alicia %A Escobar-Rodríguez,German Antonio %A García-Lozano,María José %A Barón-Franco,Bosco %A Moreno-Gaviño,Lourdes %A Moreno-Conde,Jesús %A Rivas-González,José Antonio %A Parra-Calderón,Carlos Luis %+ Computational Health Informatics Group, Institute of Biomedicine of Seville, Virgen del Rocío University Hospital, Consejo Superior de Investigaciones Científicas, University of Seville, Avenida Manuel Siurot s/n, Seville, 41013, Spain, 34 955 013 662, celia.alvarez@juntadeandalucia.es %K adherence %K ontology %K clinical decision support system %K CDSS %K complex chronic patients %K functional validation %K multimorbidity %K polypharmacy %K atrial fibrillation %K anticoagulants %D 2022 %7 2.8.2022 %9 Original Paper %J JMIR Form Res %G English %X Background: Due to an increase in life expectancy, the prevalence of chronic diseases is also on the rise. Clinical practice guidelines (CPGs) provide recommendations for suitable interventions regarding different chronic diseases, but a deficiency in the implementation of these CPGs has been identified. The PITeS-TiiSS (Telemedicine and eHealth Innovation Platform: Information Communications Technology for Research and Information Challenges in Health Services) tool, a personalized ontology-based clinical decision support system (CDSS), aims to reduce variability, prevent errors, and consider interactions between different CPG recommendations, among other benefits. Objective: The aim of this study is to design, develop, and validate an ontology-based CDSS that provides personalized recommendations related to drug prescription. The target population is older adult patients with chronic diseases and polypharmacy, and the goal is to reduce complications related to these types of conditions while offering integrated care. Methods: A study scenario about atrial fibrillation and treatment with anticoagulants was selected to validate the tool. After this, a series of knowledge sources were identified, including CPGs, PROFUND index, LESS/CHRON criteria, and STOPP/START criteria, to extract the information. Modeling was carried out using an ontology, and mapping was done with Health Level 7 Fast Healthcare Interoperability Resources (HL7 FHIR) and Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT; International Health Terminology Standards Development Organisation). Once the CDSS was developed, validation was carried out by using a retrospective case study. Results: This project was funded in January 2015 and approved by the Virgen del Rocio University Hospital ethics committee on November 24, 2015. Two different tasks were carried out to test the functioning of the tool. First, retrospective data from a real patient who met the inclusion criteria were used. Second, the analysis of an adoption model was performed through the study of the requirements and characteristics that a CDSS must meet in order to be well accepted and used by health professionals. The results are favorable and allow the proposed research to continue to the next phase. Conclusions: An ontology-based CDSS was successfully designed, developed, and validated. However, in future work, validation in a real environment should be performed to ensure the tool is usable and reliable. %M 35916719 %R 10.2196/27990 %U https://formative.jmir.org/2022/8/e27990 %U https://doi.org/10.2196/27990 %U http://www.ncbi.nlm.nih.gov/pubmed/35916719 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 8 %P e35726 %T Harnessing the Electronic Health Care Record to Optimize Patient Safety in Primary Care: Framework for Evaluating e–Safety-Netting Tools %A Black,Georgia Bell %A Bhuiya,Afsana %A Friedemann Smith,Claire %A Hirst,Yasemin %A Nicholson,Brian David %+ Department of Applied Health Research, University College London, 1-19 Torrington Place, London, WC1E 7HB, United Kingdom, 44 2031083157, g.black@ucl.ac.uk %K primary care %K patient safety %K electronic health record %K safety %K optimize %K framework %K evaluation %K tool %K diagnostic %K uncertainty %K management %K netting %K software %K criteria %D 2022 %7 1.8.2022 %9 Viewpoint %J JMIR Med Inform %G English %X The management of diagnostic uncertainty is part of every primary care physician’s role. e–Safety-netting tools help health care professionals to manage diagnostic uncertainty. Using software in addition to verbal or paper based safety-netting methods could make diagnostic delays and errors less likely. There are an increasing number of software products that have been identified as e–safety-netting tools, particularly since the start of the COVID-19 pandemic. e–Safety-netting tools can have a variety of functions, such as sending clinician alerts, facilitating administrative tasking, providing decision support, and sending reminder text messages to patients. However, these tools have not been evaluated by using robust research designs for patient safety interventions. We present an emergent framework of criteria for effective e–safety-netting tools that can be used to support the development of software. The framework is based on validated frameworks for electronic health record development and patient safety. There are currently no tools available that meet all of the criteria in the framework. We hope that the framework will stimulate clinical and public conversations about e–safety-netting tools. In the future, a validated framework would drive audits and improvements. We outline key areas for future research both in primary care and within integrated care systems. %M 35916722 %R 10.2196/35726 %U https://medinform.jmir.org/2022/8/e35726 %U https://doi.org/10.2196/35726 %U http://www.ncbi.nlm.nih.gov/pubmed/35916722 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 6 %P e37557 %T Automatic International Classification of Diseases Coding System: Deep Contextualized Language Model With Rule-Based Approaches %A Chen,Pei-Fu %A Chen,Kuan-Chih %A Liao,Wei-Chih %A Lai,Feipei %A He,Tai-Liang %A Lin,Sheng-Che %A Chen,Wei-Jen %A Yang,Chi-Yu %A Lin,Yu-Cheng %A Tsai,I-Chang %A Chiu,Chi-Hao %A Chang,Shu-Chih %A Hung,Fang-Ming %+ Department of Medical Affairs, Far Eastern Memorial Hospital, No. 21, Sec. 2, Nanya S. Rd., Banciao Dist., New Taipei City, 220216, Taiwan, 886 2 8966 7000, drphilip101@gmail.com %K deep learning %K International Classification of Diseases %K medical records %K multilabel text classification %K natural language processing %K coding system %K algorithm %K electronic health record %K data mining %D 2022 %7 29.6.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: The tenth revision of the International Classification of Diseases (ICD-10) is widely used for epidemiological research and health management. The clinical modification (CM) and procedure coding system (PCS) of ICD-10 were developed to describe more clinical details with increasing diagnosis and procedure codes and applied in disease-related groups for reimbursement. The expansion of codes made the coding time-consuming and less accurate. The state-of-the-art model using deep contextual word embeddings was used for automatic multilabel text classification of ICD-10. In addition to input discharge diagnoses (DD), the performance can be improved by appropriate preprocessing methods for the text from other document types, such as medical history, comorbidity and complication, surgical method, and special examination. Objective: This study aims to establish a contextual language model with rule-based preprocessing methods to develop the model for ICD-10 multilabel classification. Methods: We retrieved electronic health records from a medical center. We first compared different word embedding methods. Second, we compared the preprocessing methods using the best-performing embeddings. We compared biomedical bidirectional encoder representations from transformers (BioBERT), clinical generalized autoregressive pretraining for language understanding (Clinical XLNet), label tree-based attention-aware deep model for high-performance extreme multilabel text classification (AttentionXLM), and word-to-vector (Word2Vec) to predict ICD-10-CM. To compare different preprocessing methods for ICD-10-CM, we included DD, medical history, and comorbidity and complication as inputs. We compared the performance of ICD-10-CM prediction using different preprocesses, including definition training, external cause code removal, number conversion, and combination code filtering. For the ICD-10 PCS, the model was trained using different combinations of DD, surgical method, and key words of special examination. The micro F1 score and the micro area under the receiver operating characteristic curve were used to compare the model’s performance with that of different preprocessing methods. Results: BioBERT had an F1 score of 0.701 and outperformed other models such as Clinical XLNet, AttentionXLM, and Word2Vec. For the ICD-10-CM, the model had an F1 score that significantly increased from 0.749 (95% CI 0.744-0.753) to 0.769 (95% CI 0.764-0.773) with the ICD-10 definition training, external cause code removal, number conversion, and combination code filter. For the ICD-10-PCS, the model had an F1 score that significantly increased from 0.670 (95% CI 0.663-0.678) to 0.726 (95% CI 0.719-0.732) with a combination of discharge diagnoses, surgical methods, and key words of special examination. With our preprocessing methods, the model had the highest area under the receiver operating characteristic curve of 0.853 (95% CI 0.849-0.855) and 0.831 (95% CI 0.827-0.834) for ICD-10-CM and ICD-10-PCS, respectively. Conclusions: The performance of our model with the pretrained contextualized language model and rule-based preprocessing method is better than that of the state-of-the-art model for ICD-10-CM or ICD-10-PCS. This study highlights the importance of rule-based preprocessing methods based on coder coding rules. %M 35767353 %R 10.2196/37557 %U https://medinform.jmir.org/2022/6/e37557 %U https://doi.org/10.2196/37557 %U http://www.ncbi.nlm.nih.gov/pubmed/35767353 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 6 %P e33847 %T Personalized Recommendations for Physical Activity e-Coaching (OntoRecoModel): Ontological Modeling %A Chatterjee,Ayan %A Prinz,Andreas %+ Department of Information and Communication Technology, Center for eHealth, University of Agder, Jon Lilletuns Vei 9, Grimstad, 4879, Norway, 47 94719372, ayan.chatterjee@uia.no %K descriptive logic %K ontology %K e-coach %K reasoning %K recommendation generation %D 2022 %7 23.6.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Automatic e-coaching may motivate individuals to lead a healthy lifestyle with early health risk prediction, personalized recommendation generation, and goal evaluation. Multiple studies have reported on uninterrupted and automatic monitoring of behavioral aspects (such as sedentary time, amount, and type of physical activity); however, e-coaching and personalized feedback techniques are still in a nascent stage. Current intelligent coaching strategies are mostly based on the handcrafted string messages that rarely individualize to each user’s needs, context, and preferences. Therefore, more realistic, flexible, practical, sophisticated, and engaging strategies are needed to model personalized recommendations. Objective: This study aims to design and develop an ontology to model personalized recommendation message intent, components (such as suggestion, feedback, argument, and follow-ups), and contents (such as spatial and temporal content and objects relevant to perform the recommended activities). A reasoning technique will help to discover implied knowledge from the proposed ontology. Furthermore, recommendation messages can be classified into different categories in the proposed ontology. Methods: The ontology was created using Protégé (version 5.5.0) open-source software. We used the Java-based Jena Framework (version 3.16) to build a semantic web application as a proof of concept, which included Resource Description Framework application programming interface, World Wide Web Consortium Web Ontology Language application programming interface, native tuple database, and SPARQL Protocol and Resource Description Framework Query Language query engine. The HermiT (version 1.4.3.x) ontology reasoner available in Protégé 5.x implemented the logical and structural consistency of the proposed ontology. To verify the proposed ontology model, we simulated data for 8 test cases. The personalized recommendation messages were generated based on the processing of personal activity data in combination with contextual weather data and personal preference data. The developed ontology was processed using a query engine against a rule base to generate personalized recommendations. Results: The proposed ontology was implemented in automatic activity coaching to generate and deliver meaningful, personalized lifestyle recommendations. The ontology can be visualized using OWLViz and OntoGraf. In addition, we developed an ontology verification module that behaves similar to a rule-based decision support system to analyze the generation and delivery of personalized recommendation messages following a logical structure. Conclusions: This study led to the creation of a meaningful ontology to generate and model personalized recommendation messages for physical activity coaching. %M 35737439 %R 10.2196/33847 %U https://medinform.jmir.org/2022/6/e33847 %U https://doi.org/10.2196/33847 %U http://www.ncbi.nlm.nih.gov/pubmed/35737439 %0 Journal Article %@ 2563-3570 %I JMIR Publications %V 3 %N 1 %P e33186 %T Differential Expression of Long Noncoding RNAs in Murine Myoblasts After Short Hairpin RNA-Mediated Dysferlin Silencing In Vitro: Microarray Profiling %A Singhal,Richa %A Lukose,Rachel %A Carr,Gwenyth %A Moktar,Afsoon %A Gonzales-Urday,Ana Lucia %A Rouchka,Eric C %A Vajravelu,Bathri N %+ Department of Physician Assistant Studies, Massachusetts College of Pharmacy and Health Sciences, 179 Longwood Avenue, Boston, MA, 02115, United States, 1 617 732 2961, bathri.vajravelu@mcphs.edu %K dysferlinopathy %K long noncoding RNAs %K lncRNA %K abnormal expression %K muscular dystrophy %K limb-girdle muscular dystrophy 2B %K LGMD-2B %K messenger RNA %K mRNA %K quantitative real-time polymerase chain reaction %K qRT-PCR %K gene ontology %K bioinformatics %K transcription %K noncoding RNA %K protein expression %D 2022 %7 17.6.2022 %9 Original Paper %J JMIR Bioinform Biotech %G English %X Background: Long noncoding RNAs (lncRNAs) are noncoding RNA transcripts greater than 200 nucleotides in length and are known to play a role in regulating the transcription of genes involved in vital cellular functions. We hypothesized the disease process in dysferlinopathy is linked to an aberrant expression of lncRNAs and messenger RNAs (mRNAs). Objective: In this study, we compared the lncRNA and mRNA expression profiles between wild-type and dysferlin-deficient murine myoblasts (C2C12 cells). Methods: LncRNA and mRNA expression profiling were performed using a microarray. Several lncRNAs with differential expression were validated using quantitative real-time polymerase chain reaction. Gene Ontology (GO) analysis was performed to understand the functional role of the differentially expressed mRNAs. Further bioinformatics analysis was used to explore the potential function, lncRNA-mRNA correlation, and potential targets of the differentially expressed lncRNAs. Results: We found 3195 lncRNAs and 1966 mRNAs that were differentially expressed. The chromosomal distribution of the differentially expressed lncRNAs and mRNAs was unequal, with chromosome 2 having the highest number of lncRNAs and chromosome 7 having the highest number of mRNAs that were differentially expressed. Pathway analysis of the differentially expressed genes indicated the involvement of several signaling pathways including PI3K-Akt, Hippo, and pathways regulating the pluripotency of stem cells. The differentially expressed genes were also enriched for the GO terms, developmental process and muscle system process. Network analysis identified 8 statistically significant (P<.05) network objects from the upregulated lncRNAs and 3 statistically significant network objects from the downregulated lncRNAs. Conclusions: Our results thus far imply that dysferlinopathy is associated with an aberrant expression of multiple lncRNAs, many of which may have a specific function in the disease process. GO terms and network analysis suggest a muscle-specific role for these lncRNAs. To elucidate the specific roles of these abnormally expressed noncoding RNAs, further studies engineering their expression are required. %R 10.2196/33186 %U https://bioinform.jmir.org/2022/1/e33186 %U https://doi.org/10.2196/33186 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 24 %N 6 %P e37213 %T Deep Phenotyping of Chinese Electronic Health Records by Recognizing Linguistic Patterns of Phenotypic Narratives With a Sequence Motif Discovery Tool: Algorithm Development and Validation %A Li,Shicheng %A Deng,Lizong %A Zhang,Xu %A Chen,Luming %A Yang,Tao %A Qi,Yifan %A Jiang,Taijiao %+ Institute of Systems Medicine, Chinese Academy of Medical Sciences & Peking Union Medical College, No.100 Chongwen Road, Suzhou, Jiangsu Province, China, Beijing, 215000, China, 86 13366191184, taijiaobioinfor@ism.cams.cn %K deep phenotyping %K Chinese EHRs %K linguistic pattern %K motif discovery %K pattern recognition %D 2022 %7 3.6.2022 %9 Original Paper %J J Med Internet Res %G English %X Background: Phenotype information in electronic health records (EHRs) is mainly recorded in unstructured free text, which cannot be directly used for clinical research. EHR-based deep-phenotyping methods can structure phenotype information in EHRs with high fidelity, making it the focus of medical informatics. However, developing a deep-phenotyping method for non-English EHRs (ie, Chinese EHRs) is challenging. Although numerous EHR resources exist in China, fine-grained annotation data that are suitable for developing deep-phenotyping methods are limited. It is challenging to develop a deep-phenotyping method for Chinese EHRs in such a low-resource scenario. Objective: In this study, we aimed to develop a deep-phenotyping method with good generalization ability for Chinese EHRs based on limited fine-grained annotation data. Methods: The core of the methodology was to identify linguistic patterns of phenotype descriptions in Chinese EHRs with a sequence motif discovery tool and perform deep phenotyping of Chinese EHRs by recognizing linguistic patterns in free text. Specifically, 1000 Chinese EHRs were manually annotated based on a fine-grained information model, PhenoSSU (Semantic Structured Unit of Phenotypes). The annotation data set was randomly divided into a training set (n=700, 70%) and a testing set (n=300, 30%). The process for mining linguistic patterns was divided into three steps. First, free text in the training set was encoded as single-letter sequences (P: phenotype, A: attribute). Second, a biological sequence analysis tool—MEME (Multiple Expectation Maximums for Motif Elicitation)—was used to identify motifs in the single-letter sequences. Finally, the identified motifs were reduced to a series of regular expressions representing linguistic patterns of PhenoSSU instances in Chinese EHRs. Based on the discovered linguistic patterns, we developed a deep-phenotyping method for Chinese EHRs, including a deep learning–based method for named entity recognition and a pattern recognition–based method for attribute prediction. Results: In total, 51 sequence motifs with statistical significance were mined from 700 Chinese EHRs in the training set and were combined into six regular expressions. It was found that these six regular expressions could be learned from a mean of 134 (SD 9.7) annotated EHRs in the training set. The deep-phenotyping algorithm for Chinese EHRs could recognize PhenoSSU instances with an overall accuracy of 0.844 on the test set. For the subtask of entity recognition, the algorithm achieved an F1 score of 0.898 with the Bidirectional Encoder Representations from Transformers–bidirectional long short-term memory and conditional random field model; for the subtask of attribute prediction, the algorithm achieved a weighted accuracy of 0.940 with the linguistic pattern–based method. Conclusions: We developed a simple but effective strategy to perform deep phenotyping of Chinese EHRs with limited fine-grained annotation data. Our work will promote the second use of Chinese EHRs and give inspiration to other non–English-speaking countries. %M 35657661 %R 10.2196/37213 %U https://www.jmir.org/2022/6/e37213 %U https://doi.org/10.2196/37213 %U http://www.ncbi.nlm.nih.gov/pubmed/35657661 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 5 %P e36709 %T The Architecture of a Feasibility Query Portal for Distributed COVID-19 Fast Healthcare Interoperability Resources (FHIR) Patient Data Repositories: Design and Implementation Study %A Gruendner,Julian %A Deppenwiese,Noemi %A Folz,Michael %A Köhler,Thomas %A Kroll,Björn %A Prokosch,Hans-Ulrich %A Rosenau,Lorenz %A Rühle,Mathias %A Scheidl,Marc-Anton %A Schüttler,Christina %A Sedlmayr,Brita %A Twrdik,Alexander %A Kiel,Alexander %A Majeed,Raphael W %+ Chair of Medical Informatics, Friedrich-Alexander University Erlangen-Nürnberg, Wetterkreuz 15, Erlangen, 91058, Germany, 49 9131 8567787, julian.gruendner@fau.de %K federated feasibility queries %K FHIR %K distributed analysis %K feasibility study %K HL7 FHIR %K FHIR Search %K CQL %K COVID-19 %K pandemic %K health data %K query %K patient data %K consensus data set %K medical informatics %K Fast Healthcare Interoperability Resources %D 2022 %7 25.5.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: An essential step in any medical research project after identifying the research question is to determine if there are sufficient patients available for a study and where to find them. Pursuing digital feasibility queries on available patient data registries has proven to be an excellent way of reusing existing real-world data sources. To support multicentric research, these feasibility queries should be designed and implemented to run across multiple sites and securely access local data. Working across hospitals usually involves working with different data formats and vocabularies. Recently, the Fast Healthcare Interoperability Resources (FHIR) standard was developed by Health Level Seven to address this concern and describe patient data in a standardized format. The Medical Informatics Initiative in Germany has committed to this standard and created data integration centers, which convert existing data into the FHIR format at each hospital. This partially solves the interoperability problem; however, a distributed feasibility query platform for the FHIR standard is still missing. Objective: This study described the design and implementation of the components involved in creating a cross-hospital feasibility query platform for researchers based on FHIR resources. This effort was part of a large COVID-19 data exchange platform and was designed to be scalable for a broad range of patient data. Methods: We analyzed and designed the abstract components necessary for a distributed feasibility query. This included a user interface for creating the query, backend with an ontology and terminology service, middleware for query distribution, and FHIR feasibility query execution service. Results: We implemented the components described in the Methods section. The resulting solution was distributed to 33 German university hospitals. The functionality of the comprehensive network infrastructure was demonstrated using a test data set based on the German Corona Consensus Data Set. A performance test using specifically created synthetic data revealed the applicability of our solution to data sets containing millions of FHIR resources. The solution can be easily deployed across hospitals and supports feasibility queries, combining multiple inclusion and exclusion criteria using standard Health Level Seven query languages such as Clinical Quality Language and FHIR Search. Developing a platform based on multiple microservices allowed us to create an extendable platform and support multiple Health Level Seven query languages and middleware components to allow integration with future directions of the Medical Informatics Initiative. Conclusions: We designed and implemented a feasibility platform for distributed feasibility queries, which works directly on FHIR-formatted data and distributed it across 33 university hospitals in Germany. We showed that developing a feasibility platform directly on the FHIR standard is feasible. %M 35486893 %R 10.2196/36709 %U https://medinform.jmir.org/2022/5/e36709 %U https://doi.org/10.2196/36709 %U http://www.ncbi.nlm.nih.gov/pubmed/35486893 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 5 %P e32158 %T Domain-Specific Common Data Elements for Rare Disease Registration: Conceptual Approach of a European Joint Initiative Toward Semantic Interoperability in Rare Disease Research %A Abaza,Haitham %A Kadioglu,Dennis %A Martin,Simona %A Papadopoulou,Andri %A dos Santos Vieira,Bruna %A Schaefer,Franz %A Storf,Holger %+ Institute of Medical Informatics, Goethe University Frankfurt, University Hospital Frankfurt, Theodor-Stern-Kai 7, Frankfurt am Main, 60590, Germany, 49 696301 ext 80692, dennis.kadioglu@kgu.de %K semantic interoperability %K common data elements %K standardization %K data collection %K data discoverability %K rare diseases %K EJP RD %K EU RD Platform %K ERNs %K FAIRification %K health infrastructure %K industry %K medical informatics %K health platforms %K health registries %K health and research platforms %K health domains %D 2022 %7 20.5.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: With hundreds of registries across Europe, rare diseases (RDs) suffer from fragmented knowledge, expertise, and research. A joint initiative of the European Commission Joint Research Center and its European Platform on Rare Disease Registration (EU RD Platform), the European Reference Networks (ERNs), and the European Joint Programme on Rare Diseases (EJP RD) was launched in 2020. The purpose was to extend the set of common data elements (CDEs) for RD registration by defining domain-specific CDEs (DCDEs). Objective: This study aims to introduce and assess the feasibility of the concept of a joint initiative that unites the efforts of the European Platform on Rare Disease Registration Platform, ERNs, and European Joint Programme on Rare Diseases toward extending RD CDEs, aiming to improve the semantic interoperability of RD registries and enhance the quality of RD research. Methods: A joint conference was conducted in December 2020. All 24 ERNs were invited. Before the conference, a survey was communicated to all ERNs, proposing 18 medical domains and requesting them to identify highly relevant choices. After the conference, a 3-phase plan for defining and modeling DCDEs was drafted. Expected outcomes included harmonized lists of DCDEs. Results: All ERNs attended the conference. The survey results indicated that genetic, congenital, pediatric, and cancer were the most overlapping domains. Accordingly, the proposed list was reorganized into 10 domain groups and recommunicated to all ERNs, aiming at a smaller number of domains. Conclusions: The approach described for defining DCDEs appears to be feasible. However, it remains dynamic and should be repeated regularly based on arising research needs. %M 35594066 %R 10.2196/32158 %U https://medinform.jmir.org/2022/5/e32158 %U https://doi.org/10.2196/32158 %U http://www.ncbi.nlm.nih.gov/pubmed/35594066 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 4 %P e35789 %T Generation of a Fast Healthcare Interoperability Resources (FHIR)-based Ontology for Federated Feasibility Queries in the Context of COVID-19: Feasibility Study %A Rosenau,Lorenz %A Majeed,Raphael W %A Ingenerf,Josef %A Kiel,Alexander %A Kroll,Björn %A Köhler,Thomas %A Prokosch,Hans-Ulrich %A Gruendner,Julian %+ IT Center for Clinical Research, Gebäude 64, 2.OG, Raum 05, Ratzeburger Allee 160, Lübeck, 23562, Germany, 49 451 3101 5636, lorenz.rosenau@uni-luebeck.de %K federated queries %K feasibility study %K Fast Healthcare Interoperability Resource %K FHIR Search %K CQL %K ontology %K terminology server %K query %K feasibility %K FHIR %K terminology %K development %K COVID-19 %K automation %K user interface %K map %K input %K hospital %K data %K Germany %K accessibility %K harmonized %D 2022 %7 27.4.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: The COVID-19 pandemic highlighted the importance of making research data from all German hospitals available to scientists to respond to current and future pandemics promptly. The heterogeneous data originating from proprietary systems at hospitals' sites must be harmonized and accessible. The German Corona Consensus Dataset (GECCO) specifies how data for COVID-19 patients will be standardized in Fast Healthcare Interoperability Resources (FHIR) profiles across German hospitals. However, given the complexity of the FHIR standard, the data harmonization is not sufficient to make the data accessible. A simplified visual representation is needed to reduce the technical burden, while allowing feasibility queries. Objective: This study investigates how a search ontology can be automatically generated using FHIR profiles and a terminology server. Furthermore, it describes how this ontology can be used in a user interface (UI) and how a mapping and a terminology tree created together with the ontology can translate user input into FHIR queries. Methods: We used the FHIR profiles from the GECCO data set combined with a terminology server to generate an ontology and the required mapping files for the translation. We analyzed the profiles and identified search criteria for the visual representation. In this process, we reduced the complex profiles to code value pairs for improved usability. We enriched our ontology with the necessary information to display it in a UI. We also developed an intermediate query language to transform the queries from the UI to federated FHIR requests. Separation of concerns resulted in discrepancies between the criteria used in the intermediate query format and the target query language. Therefore, a mapping was created to reintroduce all information relevant for creating the query in its target language. Further, we generated a tree representation of the ontology hierarchy, which allows resolving child concepts in the process. Results: In the scope of this project, 82 (99%) of 83 elements defined in the GECCO profile were successfully implemented. We verified our solution based on an independently developed test patient. A discrepancy between the test data and the criteria was found in 6 cases due to different versions used to generate the test data and the UI profiles, the support for specific code systems, and the evaluation of postcoordinated Systematized Nomenclature of Medicine (SNOMED) codes. Our results highlight the need for governance mechanisms for version changes, concept mapping between values from different code systems encoding the same concept, and support for different unit dimensions. Conclusions: We developed an automatic process to generate ontology and mapping files for FHIR-formatted data. Our tests found that this process works for most of our chosen FHIR profile criteria. The process established here works directly with FHIR profiles and a terminology server, making it extendable to other FHIR profiles and demonstrating that automatic ontology generation on FHIR profiles is feasible. %M 35380548 %R 10.2196/35789 %U https://medinform.jmir.org/2022/4/e35789 %U https://doi.org/10.2196/35789 %U http://www.ncbi.nlm.nih.gov/pubmed/35380548 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 4 %P e35606 %T Multi-Label Classification in Patient-Doctor Dialogues With the RoBERTa-WWM-ext + CNN (Robustly Optimized Bidirectional Encoder Representations From Transformers Pretraining Approach With Whole Word Masking Extended Combining a Convolutional Neural Network) Model: Named Entity Study %A Sun,Yuanyuan %A Gao,Dongping %A Shen,Xifeng %A Li,Meiting %A Nan,Jiale %A Zhang,Weining %+ Institute of Medical Information, Chinese Academy of Medical Sciences, Peking Union Medical College, No 3 Yabao Road, Chaoyang District, Beijing, 100020, China, 86 10 5232 8720, gaodp_gaodp@126.com %K online consultation %K named entity %K automatic classification %K ERNIE %K Enhanced Representation through Knowledge Integration %K BERT %K Bidirectional Encoder Representations from Transformers %K machine learning %K neural network %K model %K China %K Chinese %K classification %K patient-physician dialogue %K patient doctor dialogue %K semantics %K natural language processing %D 2022 %7 21.4.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: With the prevalence of online consultation, many patient-doctor dialogues have accumulated, which, in an authentic language environment, are of significant value to the research and development of intelligent question answering and automated triage in recent natural language processing studies. Objective: The purpose of this study was to design a front-end task module for the network inquiry of intelligent medical services. Through the study of automatic labeling of real doctor-patient dialogue text on the internet, a method of identifying the negative and positive entities of dialogues with higher accuracy has been explored. Methods: The data set used for this study was from the Spring Rain Doctor internet online consultation, which was downloaded from the official data set of Alibaba Tianchi Lab. We proposed a composite abutting joint model, which was able to automatically classify the types of clinical finding entities into the following 4 attributes: positive, negative, other, and empty. We adapted a downstream architecture in Chinese Robustly Optimized Bidirectional Encoder Representations from Transformers Pretraining Approach (RoBERTa) with whole word masking (WWM) extended (RoBERTa-WWM-ext) combining a text convolutional neural network (CNN). We used RoBERTa-WWM-ext to express sentence semantics as a text vector and then extracted the local features of the sentence through the CNN, which was our new fusion model. To verify its knowledge learning ability, we chose Enhanced Representation through Knowledge Integration (ERNIE), original Bidirectional Encoder Representations from Transformers (BERT), and Chinese BERT with WWM to perform the same task, and then compared the results. Precision, recall, and macro-F1 were used to evaluate the performance of the methods. Results: We found that the ERNIE model, which was trained with a large Chinese corpus, had a total score (macro-F1) of 65.78290014, while BERT and BERT-WWM had scores of 53.18247117 and 69.2795315, respectively. Our composite abutting joint model (RoBERTa-WWM-ext + CNN) had a macro-F1 value of 70.55936311, showing that our model outperformed the other models in the task. Conclusions: The accuracy of the original model can be greatly improved by giving priority to WWM and replacing the word-based mask with unit to classify and label medical entities. Better results can be obtained by effectively optimizing the downstream tasks of the model and the integration of multiple models later on. The study findings contribute to the translation of online consultation information into machine-readable information. %M 35451969 %R 10.2196/35606 %U https://medinform.jmir.org/2022/4/e35606 %U https://doi.org/10.2196/35606 %U http://www.ncbi.nlm.nih.gov/pubmed/35451969 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 4 %P e35257 %T Natural Language Processing for Assessing Quality Indicators in Free-Text Colonoscopy and Pathology Reports: Development and Usability Study %A Bae,Jung Ho %A Han,Hyun Wook %A Yang,Sun Young %A Song,Gyuseon %A Sa,Soonok %A Chung,Goh Eun %A Seo,Ji Yeon %A Jin,Eun Hyo %A Kim,Heecheon %A An,DongUk %+ Department of Biomedical Informatics, CHA University School of Medicine, CHA University, 335, Pangyo-ro, Bundang-gu, Seongnam, 13488, Republic of Korea, 82 31 881 7109, stepano7@gmail.com %K natural language processing %K colonoscopy %K adenoma %K endoscopy %D 2022 %7 15.4.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Manual data extraction of colonoscopy quality indicators is time and labor intensive. Natural language processing (NLP), a computer-based linguistics technique, can automate the extraction of important clinical information, such as adverse events, from unstructured free-text reports. NLP information extraction can facilitate the optimization of clinical work by helping to improve quality control and patient management. Objective: We developed an NLP pipeline to analyze free-text colonoscopy and pathology reports and evaluated its ability to automatically assess adenoma detection rate (ADR), sessile serrated lesion detection rate (SDR), and postcolonoscopy surveillance intervals. Methods: The NLP tool for extracting colonoscopy quality indicators was developed using a data set of 2000 screening colonoscopy reports from a single health care system, with an associated 1425 pathology reports. The NLP system was then tested on a data set of 1000 colonoscopy reports and its performance was compared with that of 5 human annotators. Additionally, data from 54,562 colonoscopies performed between 2010 and 2019 were analyzed using the NLP pipeline. Results: The NLP pipeline achieved an overall accuracy of 0.99-1.00 for identifying polyp subtypes, 0.99-1.00 for identifying the anatomical location of polyps, and 0.98 for counting the number of neoplastic polyps. The NLP pipeline achieved performance similar to clinical experts for assessing ADR, SDR, and surveillance intervals. NLP analysis of a 10-year colonoscopy data set identified great individual variance in colonoscopy quality indicators among 25 endoscopists. Conclusions: The NLP pipeline could accurately extract information from colonoscopy and pathology reports and demonstrated clinical efficacy for assessing ADR, SDR, and surveillance intervals in these reports. Implementation of the system enabled automated analysis and feedback on quality indicators, which could motivate endoscopists to improve the quality of their performance and improve clinical decision-making in colorectal cancer screening programs. %M 35436226 %R 10.2196/35257 %U https://medinform.jmir.org/2022/4/e35257 %U https://doi.org/10.2196/35257 %U http://www.ncbi.nlm.nih.gov/pubmed/35436226 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 4 %P e26353 %T Neural Translation and Automated Recognition of ICD-10 Medical Entities From Natural Language: Model Development and Performance Assessment %A Falissard,Louis %A Morgand,Claire %A Ghosn,Walid %A Imbaud,Claire %A Bounebache,Karim %A Rey,Grégoire %+ Centre for Epidemiology on Medical Causes of Death, Inserm, 80 Rue du Général Leclerc, Le Kremlin Bicêtre, 94270, France, 33 679649178, louis.falissard@gmail.com %K machine learning %K deep learning %K machine translation %K mortality statistics %K automated medical entity recognition %K ICD-10 coding %D 2022 %7 11.4.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: The recognition of medical entities from natural language is a ubiquitous problem in the medical field, with applications ranging from medical coding to the analysis of electronic health data for public health. It is, however, a complex task usually requiring human expert intervention, thus making it expansive and time-consuming. Recent advances in artificial intelligence, specifically the rise of deep learning methods, have enabled computers to make efficient decisions on a number of complex problems, with the notable example of neural sequence models and their powerful applications in natural language processing. However, they require a considerable amount of data to learn from, which is typically their main limiting factor. The Centre for Epidemiology on Medical Causes of Death (CépiDc) stores an exhaustive database of death certificates at the French national scale, amounting to several millions of natural language examples provided with their associated human-coded medical entities available to the machine learning practitioner. Objective: The aim of this paper was to investigate the application of deep neural sequence models to the problem of medical entity recognition from natural language. Methods: The investigated data set included every French death certificate from 2011 to 2016. These certificates contain information such as the subject’s age, the subject’s gender, and the chain of events leading to his or her death, both in French and encoded as International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) medical entities, for a total of around 3 million observations in the data set. The task of automatically recognizing ICD-10 medical entities from the French natural language–based chain of events leading to death was then formulated as a type of predictive modeling problem known as a sequence-to-sequence modeling problem. A deep neural network–based model, known as the Transformer, was then slightly adapted and fit to the data set. Its performance was then assessed on an external data set and compared to the current state-of-the-art approach. CIs for derived measurements were estimated via bootstrapping. Results: The proposed approach resulted in an F-measure value of 0.952 (95% CI 0.946-0.957), which constitutes a significant improvement over the current state-of-the-art approach and its previously reported F-measure value of 0.825 as assessed on a comparable data set. Such an improvement makes possible a whole field of new applications, from nosologist-level automated coding to temporal harmonization of death statistics. Conclusions: This paper shows that a deep artificial neural network can directly learn from voluminous data sets in order to identify complex relationships between natural language and medical entities, without any explicit prior knowledge. Although not entirely free from mistakes, the derived model constitutes a powerful tool for automated coding of medical entities from medical language with promising potential applications. %M 35404262 %R 10.2196/26353 %U https://medinform.jmir.org/2022/4/e26353 %U https://doi.org/10.2196/26353 %U http://www.ncbi.nlm.nih.gov/pubmed/35404262 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 4 %P e29290 %T A Traditional Chinese Medicine Syndrome Classification Model Based on Cross-Feature Generation by Convolution Neural Network: Model Development and Validation %A Huang,Zonghai %A Miao,Jiaqing %A Chen,Ju %A Zhong,Yanmei %A Yang,Simin %A Ma,Yiyi %A Wen,Chuanbiao %+ College of Medical Information Engineering, Chengdu University of Traditional Chinese Medicine, 37 Shierqiao Road, Chengdu, 610075, China, 86 18980069966, 228237222@qq.com %K intelligent syndrome differentiation %K cross-FGCNN %K TCM %D 2022 %7 6.4.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Nowadays, intelligent medicine is gaining widespread attention, and great progress has been made in Western medicine with the help of artificial intelligence to assist in decision making. Compared with Western medicine, traditional Chinese medicine (TCM) involves selecting the specific treatment method, prescription, and medication based on the dialectical results of each patient’s symptoms. For this reason, the development of a TCM-assisted decision-making system has lagged. Treatment based on syndrome differentiation is the core of TCM treatment; TCM doctors can dialectically classify diseases according to patients’ symptoms and optimize treatment in time. Therefore, the essence of a TCM-assisted decision-making system is a TCM intelligent, dialectical algorithm. Symptoms stored in electronic medical records are mostly associated with patients’ diseases; however, symptoms of TCM are mostly subjectively identified. In general electronic medical records, there are many missing values. TCM medical records, in which symptoms tend to cause high-dimensional sparse data, reduce algorithm accuracy. Objective: This study aims to construct an algorithm model compatible for the multidimensional, highly sparse, and multiclassification task of TCM syndrome differentiation, so that it can be effectively applied to the intelligent dialectic of different diseases. Methods: The relevant terms in electronic medical records were standardized with respect to symptoms and evidence-based criteria of TCM. We structuralized case data based on the classification of different symptoms and physical signs according to the 4 diagnostic examinations in TCM diagnosis. A novel cross-feature generation by convolution neural network model performed evidence-based recommendations based on the input embedded, structured medical record data. Results: The data set included 5273 real dysmenorrhea cases from the Sichuan TCM big data management platform and the Chinese literature database, which were embedded into 60 fields after being structured and standardized. The training set and test set were randomly constructed in a ratio of 3:1. For the classification of different syndrome types, compared with 6 traditional, intelligent dialectical models and 3 click-through-rate models, the new model showed a good generalization ability and good classification effect. The comprehensive accuracy rate reached 96.21%. Conclusions: The main contribution of this study is the construction of a new intelligent dialectical model combining the characteristics of TCM by treating intelligent dialectics as a high-dimensional sparse vector classification task. Owing to the standardization of the input symptoms, all the common symptoms of TCM are covered, and the model can differentiate the symptoms with a variety of missing values. Therefore, with the continuous improvement of disease data sets, this model has the potential to be applied to the dialectical classification of different diseases in TCM. %M 35384854 %R 10.2196/29290 %U https://medinform.jmir.org/2022/4/e29290 %U https://doi.org/10.2196/29290 %U http://www.ncbi.nlm.nih.gov/pubmed/35384854 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 24 %N 3 %P e27210 %T A Question-and-Answer System to Extract Data From Free-Text Oncological Pathology Reports (CancerBERT Network): Development Study %A Mitchell,Joseph Ross %A Szepietowski,Phillip %A Howard,Rachel %A Reisman,Phillip %A Jones,Jennie D %A Lewis,Patricia %A Fridley,Brooke L %A Rollison,Dana E %+ Department of Health Data Services, H Lee Moffitt Cancer Center and Research Institute, 12902 Magnolia Drive, Tampa, FL, 33612, United States, 1 813 745 6530, Dana.Rollison@moffitt.org %K natural language processing %K NLP %K BERT %K transformer %K pathology %K ICD-O-3 %K deep learning %K cancer %D 2022 %7 23.3.2022 %9 Original Paper %J J Med Internet Res %G English %X Background: Information in pathology reports is critical for cancer care. Natural language processing (NLP) systems used to extract information from pathology reports are often narrow in scope or require extensive tuning. Consequently, there is growing interest in automated deep learning approaches. A powerful new NLP algorithm, bidirectional encoder representations from transformers (BERT), was published in late 2018. BERT set new performance standards on tasks as diverse as question answering, named entity recognition, speech recognition, and more. Objective: The aim of this study is to develop a BERT-based system to automatically extract detailed tumor site and histology information from free-text oncological pathology reports. Methods: We pursued three specific aims: extract accurate tumor site and histology descriptions from free-text pathology reports, accommodate the diverse terminology used to indicate the same pathology, and provide accurate standardized tumor site and histology codes for use by downstream applications. We first trained a base language model to comprehend the technical language in pathology reports. This involved unsupervised learning on a training corpus of 275,605 electronic pathology reports from 164,531 unique patients that included 121 million words. Next, we trained a question-and-answer (Q&A) model that connects a Q&A layer to the base pathology language model to answer pathology questions. Our Q&A system was designed to search for the answers to two predefined questions in each pathology report: What organ contains the tumor? and What is the kind of tumor or carcinoma? This involved supervised training on 8197 pathology reports, each with ground truth answers to these 2 questions determined by certified tumor registrars. The data set included 214 tumor sites and 193 histologies. The tumor site and histology phrases extracted by the Q&A model were used to predict International Classification of Diseases for Oncology, Third Edition (ICD-O-3), site and histology codes. This involved fine-tuning two additional BERT models: one to predict site codes and another to predict histology codes. Our final system includes a network of 3 BERT-based models. We call this CancerBERT network (caBERTnet). We evaluated caBERTnet using a sequestered test data set of 2050 pathology reports with ground truth answers determined by certified tumor registrars. Results: caBERTnet’s accuracies for predicting group-level site and histology codes were 93.53% (1895/2026) and 97.6% (1993/2042), respectively. The top 5 accuracies for predicting fine-grained ICD-O-3 site and histology codes with 5 or more samples each in the training data set were 92.95% (1794/1930) and 96.01% (1853/1930), respectively. Conclusions: We have developed an NLP system that outperforms existing algorithms at predicting ICD-O-3 codes across an extensive range of tumor sites and histologies. Our new system could help reduce treatment delays, increase enrollment in clinical trials of new therapies, and improve patient outcomes. %M 35319481 %R 10.2196/27210 %U https://www.jmir.org/2022/3/e27210 %U https://doi.org/10.2196/27210 %U http://www.ncbi.nlm.nih.gov/pubmed/35319481 %0 Journal Article %@ 2292-9495 %I JMIR Publications %V 9 %N 1 %P e31021 %T Concept Libraries for Repeatable and Reusable Research: Qualitative Study Exploring the Needs of Users %A Almowil,Zahra %A Zhou,Shang-Ming %A Brophy,Sinead %A Croxall,Jodie %+ Data Science Building, Medical School, Swansea University, Sketty, Swansea, Wales, SA2 8PP, United Kingdom, 44 07552894384, 934467@swansea.ac.uk %K electronic health records %K record linkage %K reproducible research %K clinical codes %K concept libraries %D 2022 %7 15.3.2022 %9 Original Paper %J JMIR Hum Factors %G English %X Background: Big data research in the field of health sciences is hindered by a lack of agreement on how to identify and define different conditions and their medications. This means that researchers and health professionals often have different phenotype definitions for the same condition. This lack of agreement makes it difficult to compare different study findings and hinders the ability to conduct repeatable and reusable research. Objective: This study aims to examine the requirements of various users, such as researchers, clinicians, machine learning experts, and managers, in the development of a data portal for phenotypes (a concept library). Methods: This was a qualitative study using interviews and focus group discussion. One-to-one interviews were conducted with researchers, clinicians, machine learning experts, and senior research managers in health data science (N=6) to explore their specific needs in the development of a concept library. In addition, a focus group discussion with researchers (N=14) working with the Secured Anonymized Information Linkage databank, a national eHealth data linkage infrastructure, was held to perform a SWOT (strengths, weaknesses, opportunities, and threats) analysis for the phenotyping system and the proposed concept library. The interviews and focus group discussion were transcribed verbatim, and 2 thematic analyses were performed. Results: Most of the participants thought that the prototype concept library would be a very helpful resource for conducting repeatable research, but they specified that many requirements are needed before its development. Although all the participants stated that they were aware of some existing concept libraries, most of them expressed negative perceptions about them. The participants mentioned several facilitators that would stimulate them to share their work and reuse the work of others, and they pointed out several barriers that could inhibit them from sharing their work and reusing the work of others. The participants suggested some developments that they would like to see to improve reproducible research output using routine data. Conclusions: The study indicated that most interviewees valued a concept library for phenotypes. However, only half of the participants felt that they would contribute by providing definitions for the concept library, and they reported many barriers regarding sharing their work on a publicly accessible platform. Analysis of interviews and the focus group discussion revealed that different stakeholders have different requirements, facilitators, barriers, and concerns about a prototype concept library. %M 35289755 %R 10.2196/31021 %U https://humanfactors.jmir.org/2022/1/e31021 %U https://doi.org/10.2196/31021 %U http://www.ncbi.nlm.nih.gov/pubmed/35289755 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 3 %P e35104 %T Patient-Level Fall Risk Prediction Using the Observational Medical Outcomes Partnership’s Common Data Model: Pilot Feasibility Study %A Jung,Hyesil %A Yoo,Sooyoung %A Kim,Seok %A Heo,Eunjeong %A Kim,Borham %A Lee,Ho-Young %A Hwang,Hee %+ Office of eHealth Research and Business, Seoul National University Bundang Hospital, 172 Dolma-ro, Bundang-gu, Seongnam-si, 13620, Republic of Korea, 82 31 787 8980, yoosoo0@snubh.org %K common data model %K accidental falls %K Observational Medical Outcomes Partnership %K nursing records %K medical informatics %K health data %K electronic health record %K data model %K prediction model %K risk prediction %K fall risk %D 2022 %7 11.3.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Falls in acute care settings threaten patients’ safety. Researchers have been developing fall risk prediction models and exploring risk factors to provide evidence-based fall prevention practices; however, such efforts are hindered by insufficient samples, limited covariates, and a lack of standardized methodologies that aid study replication. Objective: The objectives of this study were to (1) convert fall-related electronic health record data into the standardized Observational Medical Outcome Partnership's (OMOP) common data model format and (2) develop models that predict fall risk during 2 time periods. Methods: As a pilot feasibility test, we converted fall-related electronic health record data (nursing notes, fall risk assessment sheet, patient acuity assessment sheet, and clinical observation sheet) into standardized OMOP common data model format using an extraction, transformation, and load process. We developed fall risk prediction models for 2 time periods (within 7 days of admission and during the entire hospital stay) using 2 algorithms (least absolute shrinkage and selection operator logistic regression and random forest). Results: In total, 6277 nursing statements, 747,049,486 clinical observation sheet records, 1,554,775 fall risk scores, and 5,685,011 patient acuity scores were converted into OMOP common data model format. All our models (area under the receiver operating characteristic curve 0.692-0.726) performed better than the Hendrich II Fall Risk Model. Patient acuity score, fall history, age ≥60 years, movement disorder, and central nervous system agents were the most important predictors in the logistic regression models. Conclusions: To enhance model performance further, we are currently converting all nursing records into the OMOP common data model data format, which will then be included in the models. Thus, in the near future, the performance of fall risk prediction models could be improved through the application of abundant nursing records and external validation. %M 35275076 %R 10.2196/35104 %U https://medinform.jmir.org/2022/3/e35104 %U https://doi.org/10.2196/35104 %U http://www.ncbi.nlm.nih.gov/pubmed/35275076 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 24 %N 2 %P e27146 %T Age- and Sex-Specific Differences in Multimorbidity Patterns and Temporal Trends on Assessing Hospital Discharge Records in Southwest China: Network-Based Study %A Wang,Liya %A Qiu,Hang %A Luo,Li %A Zhou,Li %+ School of Computer Science and Engineering, University of Electronic Science and Technology of China, No.2006, Xiyuan Ave, West Hi-Tech Zone, Chengdu, 611731, China, 86 28 61830278, qiuhang@uestc.edu.cn %K multimorbidity pattern %K temporal trend %K network analysis %K multimorbidity prevalence %K administrative data %K longitudinal study %K regional research %D 2022 %7 25.2.2022 %9 Original Paper %J J Med Internet Res %G English %X Background: Multimorbidity represents a global health challenge, which requires a more global understanding of multimorbidity patterns and trends. However, the majority of studies completed to date have often relied on self-reported conditions, and a simultaneous assessment of the entire spectrum of chronic disease co-occurrence, especially in developing regions, has not yet been performed. Objective: We attempted to provide a multidimensional approach to understand the full spectrum of chronic disease co-occurrence among general inpatients in southwest China, in order to investigate multimorbidity patterns and temporal trends, and assess their age and sex differences. Methods: We conducted a retrospective cohort analysis based on 8.8 million hospital discharge records of about 5.0 million individuals of all ages from 2015 to 2019 in a megacity in southwest China. We examined all chronic diagnoses using the ICD-10 (International Classification of Diseases, 10th revision) codes at 3 digits and focused on chronic diseases with ≥1% prevalence for each of the age and sex strata, which resulted in a total of 149 and 145 chronic diseases in males and females, respectively. We constructed multimorbidity networks in the general population based on sex and age, and used the cosine index to measure the co-occurrence of chronic diseases. Then, we divided the networks into communities and assessed their temporal trends. Results: The results showed complex interactions among chronic diseases, with more intensive connections among males and inpatients ≥40 years old. A total of 9 chronic diseases were simultaneously classified as central diseases, hubs, and bursts in the multimorbidity networks. Among them, 5 diseases were common to both males and females, including hypertension, chronic ischemic heart disease, cerebral infarction, other cerebrovascular diseases, and atherosclerosis. The earliest leaps (degree leaps ≥6) appeared at a disorder of glycoprotein metabolism that happened at 25-29 years in males, about 15 years earlier than in females. The number of chronic diseases in the community increased over time, but the new entrants did not replace the root of the community. Conclusions: Our multimorbidity network analysis identified specific differences in the co-occurrence of chronic diagnoses by sex and age, which could help in the design of clinical interventions for inpatient multimorbidity. %M 35212632 %R 10.2196/27146 %U https://www.jmir.org/2022/2/e27146 %U https://doi.org/10.2196/27146 %U http://www.ncbi.nlm.nih.gov/pubmed/35212632 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 2 %P e29803 %T Identification of Prediabetes Discussions in Unstructured Clinical Documentation: Validation of a Natural Language Processing Algorithm %A Schwartz,Jessica L %A Tseng,Eva %A Maruthur,Nisa M %A Rouhizadeh,Masoud %+ Division of General Internal Medicine, Johns Hopkins School of Medicine, 2024 E Monument St., Ste 2-604D, Baltimore, MD, 21205, United States, 1 973 722 8552, jschwa64@jhmi.edu %K prediabetes %K prediabetes discussions %K prediabetes management %K chronic disease management %K physician-patient communication %K natural language processing %K machine learning %D 2022 %7 24.2.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Prediabetes affects 1 in 3 US adults. Most are not receiving evidence-based interventions, so understanding how providers discuss prediabetes with patients will inform how to improve their care. Objective: This study aimed to develop a natural language processing (NLP) algorithm using machine learning techniques to identify discussions of prediabetes in narrative documentation. Methods: We developed and applied a keyword search strategy to identify discussions of prediabetes in clinical documentation for patients with prediabetes. We manually reviewed matching notes to determine which represented actual prediabetes discussions. We applied 7 machine learning models against our manual annotation. Results: Machine learning classifiers were able to achieve classification results that were close to human performance with up to 98% precision and recall to identify prediabetes discussions in clinical documentation. Conclusions: We demonstrated that prediabetes discussions can be accurately identified using an NLP algorithm. This approach can be used to understand and identify prediabetes management practices in primary care, thereby informing interventions to improve guideline-concordant care. %M 35200154 %R 10.2196/29803 %U https://medinform.jmir.org/2022/2/e29803 %U https://doi.org/10.2196/29803 %U http://www.ncbi.nlm.nih.gov/pubmed/35200154 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 10 %N 1 %P e31063 %T Development of a Pipeline for Adverse Drug Reaction Identification in Clinical Notes: Word Embedding Models and String Matching %A Siegersma,Klaske R %A Evers,Maxime %A Bots,Sophie H %A Groepenhoff,Floor %A Appelman,Yolande %A Hofstra,Leonard %A Tulevski,Igor I %A Somsen,G Aernout %A den Ruijter,Hester M %A Spruit,Marco %A Onland-Moret,N Charlotte %+ Department of Epidemiology, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG, Netherlands, 31 887569610, N.C.Onland@umcutrecht.nl %K adverse drug reactions %K word embeddings %K clinical notes %D 2022 %7 25.1.2022 %9 Original Paper %J JMIR Med Inform %G English %X Background: Knowledge about adverse drug reactions (ADRs) in the population is limited because of underreporting, which hampers surveillance and assessment of drug safety. Therefore, gathering accurate information that can be retrieved from clinical notes about the incidence of ADRs is of great relevance. However, manual labeling of these notes is time-consuming, and automatization can improve the use of free-text clinical notes for the identification of ADRs. Furthermore, tools for language processing in languages other than English are not widely available. Objective: The aim of this study is to design and evaluate a method for automatic extraction of medication and Adverse Drug Reaction Identification in Clinical Notes (ADRIN). Methods: Dutch free-text clinical notes (N=277,398) and medication registrations (N=499,435) from the Cardiology Centers of the Netherlands database were used. All clinical notes were used to develop word embedding models. Vector representations of word embedding models and string matching with a medical dictionary (Medical Dictionary for Regulatory Activities [MedDRA]) were used for identification of ADRs and medication in a test set of clinical notes that were manually labeled. Several settings, including search area and punctuation, could be adjusted in the prototype to evaluate the optimal version of the prototype. Results: The ADRIN method was evaluated using a test set of 988 clinical notes written on the stop date of a drug. Multiple versions of the prototype were evaluated for a variety of tasks. Binary classification of ADR presence achieved the highest accuracy of 0.84. Reduced search area and inclusion of punctuation improved performance, whereas incorporation of the MedDRA did not improve the performance of the pipeline. Conclusions: The ADRIN method and prototype are effective in recognizing ADRs in Dutch clinical notes from cardiac diagnostic screening centers. Surprisingly, incorporation of the MedDRA did not result in improved identification on top of word embedding models. The implementation of the ADRIN tool may help increase the identification of ADRs, resulting in better care and saving substantial health care costs. %M 35076407 %R 10.2196/31063 %U https://medinform.jmir.org/2022/1/e31063 %U https://doi.org/10.2196/31063 %U http://www.ncbi.nlm.nih.gov/pubmed/35076407 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 24 %N 1 %P e27434 %T Improving Diabetes-Related Biomedical Literature Exploration in the Clinical Decision-making Process via Interactive Classification and Topic Discovery: Methodology Development Study %A Ahne,Adrian %A Fagherazzi,Guy %A Tannier,Xavier %A Czernichow,Thomas %A Orchard,Francisco %+ Exposome and Heredity team, Center of Epidemiology and Population Health, Hospital Gustave Roussy, Inserm, Paris-Saclay University, 20 Rue du Dr Pinel, Villejuif, 94800, France, 33 142115386, adrian.ahne@protonmail.com %K evidence-based medicine %K clinical decision making %K clinical decision support %K digital health %K medical informatics %K transparency %K hierarchical clustering %K active learning %K classification %K memory consumption %K natural language processing %D 2022 %7 18.1.2022 %9 Original Paper %J J Med Internet Res %G English %X Background: The amount of available textual health data such as scientific and biomedical literature is constantly growing and becoming more and more challenging for health professionals to properly summarize those data and practice evidence-based clinical decision making. Moreover, the exploration of unstructured health text data is challenging for professionals without computer science knowledge due to limited time, resources, and skills. Current tools to explore text data lack ease of use, require high computational efforts, and incorporate domain knowledge and focus on topics of interest with difficulty. Objective: We developed a methodology able to explore and target topics of interest via an interactive user interface for health professionals with limited computer science knowledge. We aim to reach near state-of-the-art performance while reducing memory consumption, increasing scalability, and minimizing user interaction effort to improve the clinical decision-making process. The performance was evaluated on diabetes-related abstracts from PubMed. Methods: The methodology consists of 4 parts: (1) a novel interpretable hierarchical clustering of documents where each node is defined by headwords (words that best represent the documents in the node), (2) an efficient classification system to target topics, (3) minimized user interaction effort through active learning, and (4) a visual user interface. We evaluated our approach on 50,911 diabetes-related abstracts providing a hierarchical Medical Subject Headings (MeSH) structure, a unique identifier for a topic. Hierarchical clustering performance was compared against the implementation in the machine learning library scikit-learn. On a subset of 2000 randomly chosen diabetes abstracts, our active learning strategy was compared against 3 other strategies: random selection of training instances, uncertainty sampling that chooses instances about which the model is most uncertain, and an expected gradient length strategy based on convolutional neural networks (CNNs). Results: For the hierarchical clustering performance, we achieved an F1 score of 0.73 compared to 0.76 achieved by scikit-learn. Concerning active learning performance, after 200 chosen training samples based on these strategies, the weighted F1 score of all MeSH codes resulted in a satisfying 0.62 F1 score using our approach, 0.61 using the uncertainty strategy, 0.63 using the CNN, and 0.45 using the random strategy. Moreover, our methodology showed a constant low memory use with increased number of documents. Conclusions: We proposed an easy-to-use tool for health professionals with limited computer science knowledge who combine their domain knowledge with topic exploration and target specific topics of interest while improving transparency. Furthermore, our approach is memory efficient and highly parallelizable, making it interesting for large Big Data sets. This approach can be used by health professionals to gain deep insights into biomedical literature to ultimately improve the evidence-based clinical decision making process. %M 35040795 %R 10.2196/27434 %U https://www.jmir.org/2022/1/e27434 %U https://doi.org/10.2196/27434 %U http://www.ncbi.nlm.nih.gov/pubmed/35040795 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 24 %N 1 %P e25440 %T Understanding the Nature of Metadata: Systematic Review %A Ulrich,Hannes %A Kock-Schoppenhauer,Ann-Kristin %A Deppenwiese,Noemi %A Gött,Robert %A Kern,Jori %A Lablans,Martin %A Majeed,Raphael W %A Stöhr,Mark R %A Stausberg,Jürgen %A Varghese,Julian %A Dugas,Martin %A Ingenerf,Josef %+ IT Center for Clinical Research, University of Lübeck, Ratzeburger Allee 160, Lübeck, 23564, Germany, 49 45131015607, h.ulrich@uni-luebeck.de %K metadata %K metadata definition %K systematic review %K data integration %K data identification %K data classification %D 2022 %7 11.1.2022 %9 Review %J J Med Internet Res %G English %X Background: Metadata are created to describe the corresponding data in a detailed and unambiguous way and is used for various applications in different research areas, for example, data identification and classification. However, a clear definition of metadata is crucial for further use. Unfortunately, extensive experience with the processing and management of metadata has shown that the term “metadata” and its use is not always unambiguous. Objective: This study aimed to understand the definition of metadata and the challenges resulting from metadata reuse. Methods: A systematic literature search was performed in this study following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for reporting on systematic reviews. Five research questions were identified to streamline the review process, addressing metadata characteristics, metadata standards, use cases, and problems encountered. This review was preceded by a harmonization process to achieve a general understanding of the terms used. Results: The harmonization process resulted in a clear set of definitions for metadata processing focusing on data integration. The following literature review was conducted by 10 reviewers with different backgrounds and using the harmonized definitions. This study included 81 peer-reviewed papers from the last decade after applying various filtering steps to identify the most relevant papers. The 5 research questions could be answered, resulting in a broad overview of the standards, use cases, problems, and corresponding solutions for the application of metadata in different research areas. Conclusions: Metadata can be a powerful tool for identifying, describing, and processing information, but its meaningful creation is costly and challenging. This review process uncovered many standards, use cases, problems, and solutions for dealing with metadata. The presented harmonized definitions and the new schema have the potential to improve the classification and generation of metadata by creating a shared understanding of metadata and its context. %M 35014967 %R 10.2196/25440 %U https://www.jmir.org/2022/1/e25440 %U https://doi.org/10.2196/25440 %U http://www.ncbi.nlm.nih.gov/pubmed/35014967 %0 Journal Article %@ 2291-5222 %I JMIR Publications %V 10 %N 1 %P e30557 %T Enabling Research and Clinical Use of Patient-Generated Health Data (the mindLAMP Platform): Digital Phenotyping Study %A Vaidyam,Aditya %A Halamka,John %A Torous,John %+ Beth Israel Deaconess Medical Center, 330 Brrokline Avenue, Boston, MA, 02215, United States, 1 6176676700, jtorous@bidmc.harvard.edu %K digital phenotyping %K mHealth %K apps %K FHIR %K digital health %K health data %K patient-generated health data %K mobile health %K smartphones %K wearables %K mobile apps %K mental health, mobile phone %D 2022 %7 7.1.2022 %9 Original Paper %J JMIR Mhealth Uhealth %G English %X Background: There is a growing need for the integration of patient-generated health data (PGHD) into research and clinical care to enable personalized, preventive, and interactive care, but technical and organizational challenges, such as the lack of standards and easy-to-use tools, preclude the effective use of PGHD generated from consumer devices, such as smartphones and wearables. Objective: This study outlines how we used mobile apps and semantic web standards such as HTTP 2.0, Representational State Transfer, JSON (JavaScript Object Notation), JSON Schema, Transport Layer Security (version 1.3), Advanced Encryption Standard-256, OpenAPI, HTML5, and Vega, in conjunction with patient and provider feedback to completely update a previous version of mindLAMP. Methods: The Learn, Assess, Manage, and Prevent (LAMP) platform addresses the abovementioned challenges in enhancing clinical insight by supporting research, data analysis, and implementation efforts around PGHD as an open-source solution with freely accessible and shared code. Results: With a simplified programming interface and novel data representation that captures additional metadata, the LAMP platform enables interoperability with existing Fast Healthcare Interoperability Resources–based health care systems as well as consumer wearables and services such as Apple HealthKit and Google Fit. The companion Cortex data analysis and machine learning toolkit offer robust support for artificial intelligence, behavioral feature extraction, interactive visualizations, and high-performance data processing through parallelization and vectorization techniques. Conclusions: The LAMP platform incorporates feedback from patients and clinicians alongside a standards-based approach to address these needs and functions across a wide range of use cases through its customizable and flexible components. These range from simple survey-based research to international consortiums capturing multimodal data to simple delivery of mindfulness exercises through personalized, just-in-time adaptive interventions. %M 34994710 %R 10.2196/30557 %U https://mhealth.jmir.org/2022/1/e30557 %U https://doi.org/10.2196/30557 %U http://www.ncbi.nlm.nih.gov/pubmed/34994710 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 12 %P e27386 %T Benchmarking Effectiveness and Efficiency of Deep Learning Models for Semantic Textual Similarity in the Clinical Domain: Validation Study %A Chen,Qingyu %A Rankine,Alex %A Peng,Yifan %A Aghaarabi,Elaheh %A Lu,Zhiyong %+ National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, MD, 20894, United States, 1 301 594 7089, luzh@ncbi.nlm.nih.gov %K semantic textual similarity %K deep learning %K biomedical and clinical text mining %K word embeddings %K sentence embeddings %K transformers %D 2021 %7 30.12.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Semantic textual similarity (STS) measures the degree of relatedness between sentence pairs. The Open Health Natural Language Processing (OHNLP) Consortium released an expertly annotated STS data set and called for the National Natural Language Processing Clinical Challenges. This work describes our entry, an ensemble model that leverages a range of deep learning (DL) models. Our team from the National Library of Medicine obtained a Pearson correlation of 0.8967 in an official test set during 2019 National Natural Language Processing Clinical Challenges/Open Health Natural Language Processing shared task and achieved a second rank. Objective: Although our models strongly correlate with manual annotations, annotator-level correlation was only moderate (weighted Cohen κ=0.60). We are cautious of the potential use of DL models in production systems and argue that it is more critical to evaluate the models in-depth, especially those with extremely high correlations. In this study, we benchmark the effectiveness and efficiency of top-ranked DL models. We quantify their robustness and inference times to validate their usefulness in real-time applications. Methods: We benchmarked five DL models, which are the top-ranked systems for STS tasks: Convolutional Neural Network, BioSentVec, BioBERT, BlueBERT, and ClinicalBERT. We evaluated a random forest model as an additional baseline. For each model, we repeated the experiment 10 times, using the official training and testing sets. We reported 95% CI of the Wilcoxon rank-sum test on the average Pearson correlation (official evaluation metric) and running time. We further evaluated Spearman correlation, R², and mean squared error as additional measures. Results: Using only the official training set, all models obtained highly effective results. BioSentVec and BioBERT achieved the highest average Pearson correlations (0.8497 and 0.8481, respectively). BioSentVec also had the highest results in 3 of 4 effectiveness measures, followed by BioBERT. However, their robustness to sentence pairs of different similarity levels varies significantly. A particular observation is that BERT models made the most errors (a mean squared error of over 2.5) on highly similar sentence pairs. They cannot capture highly similar sentence pairs effectively when they have different negation terms or word orders. In addition, time efficiency is dramatically different from the effectiveness results. On average, the BERT models were approximately 20 times and 50 times slower than the Convolutional Neural Network and BioSentVec models, respectively. This results in challenges for real-time applications. Conclusions: Despite the excitement of further improving Pearson correlations in this data set, our results highlight that evaluations of the effectiveness and efficiency of STS models are critical. In future, we suggest more evaluations on the generalization capability and user-level testing of the models. We call for community efforts to create more biomedical and clinical STS data sets from different perspectives to reflect the multifaceted notion of sentence-relatedness. %M 34967748 %R 10.2196/27386 %U https://medinform.jmir.org/2021/12/e27386 %U https://doi.org/10.2196/27386 %U http://www.ncbi.nlm.nih.gov/pubmed/34967748 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 12 %P e28632 %T Text Mining of Adverse Events in Clinical Trials: Deep Learning Approach %A Chopard,Daphne %A Treder,Matthias S %A Corcoran,Padraig %A Ahmed,Nagheen %A Johnson,Claire %A Busse,Monica %A Spasic,Irena %+ School of Computer Science & Informatics, Cardiff University, Abacws, Senghennydd Road, Cardiff, CF24 4AG, United Kingdom, 44 2920870032, spasici@cardiff.ac.uk %K natural language processing %K deep learning %K machine learning %K classification %D 2021 %7 24.12.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Pharmacovigilance and safety reporting, which involve processes for monitoring the use of medicines in clinical trials, play a critical role in the identification of previously unrecognized adverse events or changes in the patterns of adverse events. Objective: This study aims to demonstrate the feasibility of automating the coding of adverse events described in the narrative section of the serious adverse event report forms to enable statistical analysis of the aforementioned patterns. Methods: We used the Unified Medical Language System (UMLS) as the coding scheme, which integrates 217 source vocabularies, thus enabling coding against other relevant terminologies such as the International Classification of Diseases–10th Revision, Medical Dictionary for Regulatory Activities, and Systematized Nomenclature of Medicine). We used MetaMap, a highly configurable dictionary lookup software, to identify the mentions of the UMLS concepts. We trained a binary classifier using Bidirectional Encoder Representations from Transformers (BERT), a transformer-based language model that captures contextual relationships, to differentiate between mentions of the UMLS concepts that represented adverse events and those that did not. Results: The model achieved a high F1 score of 0.8080, despite the class imbalance. This is 10.15 percent points lower than human-like performance but also 17.45 percent points higher than that of the baseline approach. Conclusions: These results confirmed that automated coding of adverse events described in the narrative section of serious adverse event reports is feasible. Once coded, adverse events can be statistically analyzed so that any correlations with the trialed medicines can be estimated in a timely fashion. %M 34951601 %R 10.2196/28632 %U https://medinform.jmir.org/2021/12/e28632 %U https://doi.org/10.2196/28632 %U http://www.ncbi.nlm.nih.gov/pubmed/34951601 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 12 %P e30970 %T Transformation and Evaluation of the MIMIC Database in the OMOP Common Data Model: Development and Usability Study %A Paris,Nicolas %A Lamer,Antoine %A Parrot,Adrien %+ InterHop, 30 avenue du Maine, Paris, 75015, France, 33 3 20 62 69 69, nicolas.paris@riseup.net %K data reuse %K open data %K OMOP %K common data model %K critical care %K machine learning %K big data %K health informatics %K health data %K health database %K electronic health records %K open access database %K digital health %K intensive care %K health care %D 2021 %7 14.12.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: In the era of big data, the intensive care unit (ICU) is likely to benefit from real-time computer analysis and modeling based on close patient monitoring and electronic health record data. The Medical Information Mart for Intensive Care (MIMIC) is the first open access database in the ICU domain. Many studies have shown that common data models (CDMs) improve database searching by allowing code, tools, and experience to be shared. The Observational Medical Outcomes Partnership (OMOP) CDM is spreading all over the world. Objective: The objective was to transform MIMIC into an OMOP database and to evaluate the benefits of this transformation for analysts. Methods: We transformed MIMIC (version 1.4.21) into OMOP format (version 5.3.3.1) through semantic and structural mapping. The structural mapping aimed at moving the MIMIC data into the right place in OMOP, with some data transformations. The mapping was divided into 3 phases: conception, implementation, and evaluation. The conceptual mapping aimed at aligning the MIMIC local terminologies to OMOP's standard ones. It consisted of 3 phases: integration, alignment, and evaluation. A documented, tested, versioned, exemplified, and open repository was set up to support the transformation and improvement of the MIMIC community's source code. The resulting data set was evaluated over a 48-hour datathon. Results: With an investment of 2 people for 500 hours, 64% of the data items of the 26 MIMIC tables were standardized into the OMOP CDM and 78% of the source concepts mapped to reference terminologies. The model proved its ability to support community contributions and was well received during the datathon, with 160 participants and 15,000 requests executed with a maximum duration of 1 minute. Conclusions: The resulting MIMIC-OMOP data set is the first MIMIC-OMOP data set available free of charge with real disidentified data ready for replicable intensive care research. This approach can be generalized to any medical field. %M 34904958 %R 10.2196/30970 %U https://medinform.jmir.org/2021/12/e30970 %U https://doi.org/10.2196/30970 %U http://www.ncbi.nlm.nih.gov/pubmed/34904958 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 12 %P e29286 %T Leveraging National Claims and Hospital Big Data: Cohort Study on a Statin-Drug Interaction Use Case %A Bannay,Aurélie %A Bories,Mathilde %A Le Corre,Pascal %A Riou,Christine %A Lemordant,Pierre %A Van Hille,Pascal %A Chazard,Emmanuel %A Dode,Xavier %A Cuggia,Marc %A Bouzillé,Guillaume %+ Inserm, Laboratoire Traitement du Signal et de l'Image - UMR 1099, Centre Hospitalier Universitaire de Rennes, Université de Rennes 1, UFR Santé, laboratoire d'informatique médicale, 2 avenue du Professeur Léon Bernard, Rennes, 35000, France, 33 615711230, guillaume.bouzille@gmail.com %K drug interactions %K statins %K administrative claims %K health care %K big data %K data linking %K data warehousing %D 2021 %7 13.12.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Linking different sources of medical data is a promising approach to analyze care trajectories. The aim of the INSHARE (Integrating and Sharing Health Big Data for Research) project was to provide the blueprint for a technological platform that facilitates integration, sharing, and reuse of data from 2 sources: the clinical data warehouse (CDW) of the Rennes academic hospital, called eHOP (entrepôt Hôpital), and a data set extracted from the French national claim data warehouse (Système National des Données de Santé [SNDS]). Objective: This study aims to demonstrate how the INSHARE platform can support big data analytic tasks in the health field using a pharmacovigilance use case based on statin consumption and statin-drug interactions. Methods: A Spark distributed cluster-computing framework was used for the record linkage procedure and all analyses. A semideterministic record linkage method based on the common variables between the chosen data sources was developed to identify all patients discharged after at least one hospital stay at the Rennes academic hospital between 2015 and 2017. The use-case study focused on a cohort of patients treated with statins prescribed by their general practitioner or during their hospital stay. Results: The whole process (record linkage procedure and use-case analyses) required 88 minutes. Of the 161,532 and 164,316 patients from the SNDS and eHOP CDW data sets, respectively, 159,495 patients were successfully linked (98.74% and 97.07% of patients from SNDS and eHOP CDW, respectively). Of the 16,806 patients with at least one statin delivery, 8293 patients started the consumption before and continued during the hospital stay, 6382 patients stopped statin consumption at hospital admission, and 2131 patients initiated statins in hospital. Statin-drug interactions occurred more frequently during hospitalization than in the community (3800/10,424, 36.45% and 3253/14,675, 22.17%, respectively; P<.001). Only 121 patients had the most severe level of statin-drug interaction. Hospital stay burden (length of stay and in-hospital mortality) was more severe in patients with statin-drug interactions during hospitalization. Conclusions: This study demonstrates the added value of combining and reusing clinical and claim data to provide large-scale measures of drug-drug interaction prevalence and care pathways outside hospitals. It builds a path to move the current health care system toward a Learning Health System using knowledge generated from research on real-world health data. %M 34898457 %R 10.2196/29286 %U https://medinform.jmir.org/2021/12/e29286 %U https://doi.org/10.2196/29286 %U http://www.ncbi.nlm.nih.gov/pubmed/34898457 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 12 %P e27188 %T The Utility of Different Data Standards to Document Adverse Drug Event Symptoms and Diagnoses: Mixed Methods Study %A Chan,Erina %A Small,Serena S %A Wickham,Maeve E %A Cheng,Vicki %A Balka,Ellen %A Hohl,Corinne M %+ Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute, 828 West 10th Avenue, Vancouver, BC, V5Z 1M9, Canada, 1 604 875 4111 ext 55219, Serena.Small@ubc.ca %K adverse drug events %K health information technology %K data standards %D 2021 %7 10.12.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Existing systems to document adverse drug events often use free text data entry, which produces nonstandardized and unstructured data that are prone to misinterpretation. Standardized terminology may improve data quality; however, it is unclear which data standard is most appropriate for documenting adverse drug event symptoms and diagnoses. Objective: This study aims to compare the utility, strengths, and weaknesses of different data standards for documenting adverse drug event symptoms and diagnoses. Methods: We performed a mixed methods substudy of a multicenter retrospective chart review. We reviewed the research records of prospectively diagnosed adverse drug events at 5 Canadian hospitals. A total of 2 pharmacy research assistants independently entered the symptoms and diagnoses for the adverse drug events using four standards: Medical Dictionary for Regulatory Activities (MedDRA), Systematized Nomenclature of Medicine (SNOMED) Clinical Terms, SNOMED Adverse Reaction (SNOMED ADR), and International Classification of Diseases (ICD) 11th Revision. Disagreements between research assistants regarding the case-specific utility of data standards were discussed until a consensus was reached. We used consensus ratings to determine the proportion of adverse drug events covered by a data standard and coded and analyzed field notes from the consensus sessions. Results: We reviewed 573 adverse drug events and found that MedDRA and ICD-11 had excellent coverage of adverse drug event symptoms and diagnoses. MedDRA had the highest number of matches between the research assistants, whereas ICD-11 had the fewest. SNOMED ADR had the lowest proportion of adverse drug event coverage. The research assistants were most likely to encounter terminological challenges with SNOMED ADR and usability challenges with ICD-11, whereas least likely to encounter challenges with MedDRA. Conclusions: Usability, comprehensiveness, and accuracy are important features of data standards for documenting adverse drug event symptoms and diagnoses. On the basis of our results, we recommend the use of MedDRA. %M 34890351 %R 10.2196/27188 %U https://www.jmir.org/2021/12/e27188 %U https://doi.org/10.2196/27188 %U http://www.ncbi.nlm.nih.gov/pubmed/34890351 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 12 %P e32698 %T A BERT-Based Generation Model to Transform Medical Texts to SQL Queries for Electronic Medical Records: Model Development and Validation %A Pan,Youcheng %A Wang,Chenghao %A Hu,Baotian %A Xiang,Yang %A Wang,Xiaolong %A Chen,Qingcai %A Chen,Junjie %A Du,Jingcheng %+ Intelligent Computing Research Center, Harbin Institute of Technology, No. 6, Pingshan 1st Road, Shenzhen, 518055, China, 86 136 9164 0856, hubaotian@hit.edu.cn %K electronic medical record %K text-to-SQL generation %K BERT %K grammar-based decoding %K tree-structured intermediate representation %D 2021 %7 8.12.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Electronic medical records (EMRs) are usually stored in relational databases that require SQL queries to retrieve information of interest. Effectively completing such queries can be a challenging task for medical experts due to the barriers in expertise. Existing text-to-SQL generation studies have not been fully embraced in the medical domain. Objective: The objective of this study was to propose a neural generation model that can jointly consider the characteristics of medical text and the SQL structure to automatically transform medical texts to SQL queries for EMRs. Methods: We proposed a medical text–to-SQL model (MedTS), which employed a pretrained Bidirectional Encoder Representations From Transformers model as the encoder and leveraged a grammar-based long short-term memory network as the decoder to predict the intermediate representation that can easily be transformed into the final SQL query. We adopted the syntax tree as the intermediate representation rather than directly regarding the SQL query as an ordinary word sequence, which is more in line with the tree-structure nature of SQL and can also effectively reduce the search space during generation. Experiments were conducted on the MIMICSQL dataset, and 5 competitor methods were compared. Results: Experimental results demonstrated that MedTS achieved the accuracy of 0.784 and 0.899 on the test set in terms of logic form and execution, respectively, which significantly outperformed the existing state-of-the-art methods. Further analyses proved that the performance on each component of the generated SQL was relatively balanced and offered substantial improvements. Conclusions: The proposed MedTS was effective and robust for improving the performance of medical text–to-SQL generation, indicating strong potential to be applied in the real medical scenario. %M 34889749 %R 10.2196/32698 %U https://medinform.jmir.org/2021/12/e32698 %U https://doi.org/10.2196/32698 %U http://www.ncbi.nlm.nih.gov/pubmed/34889749 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 11 %P e30308 %T The Collaborative Metadata Repository (CoMetaR) Web App: Quantitative and Qualitative Usability Evaluation %A Stöhr,Mark R %A Günther,Andreas %A Majeed,Raphael W %+ Justus-Liebig-University Giessen, Universities of Giessen and Marburg Lung Center (UGMLC), German Center for Lung Research (DZL), Klinikstraße 36, Gießen, 35392, Germany, 49 641 985 42117, mark.stoehr@innere.med.uni-giessen.de %K usability %K metadata %K data visualization %K semantic web %K data management %K data warehousing %K communication barriers %K quality improvement %K biological ontologies %K data curation %D 2021 %7 29.11.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: In the field of medicine and medical informatics, the importance of comprehensive metadata has long been recognized, and the composition of metadata has become its own field of profession and research. To ensure sustainable and meaningful metadata are maintained, standards and guidelines such as the FAIR (Findability, Accessibility, Interoperability, Reusability) principles have been published. The compilation and maintenance of metadata is performed by field experts supported by metadata management apps. The usability of these apps, for example, in terms of ease of use, efficiency, and error tolerance, crucially determines their benefit to those interested in the data. Objective: This study aims to provide a metadata management app with high usability that assists scientists in compiling and using rich metadata. We aim to evaluate our recently developed interactive web app for our collaborative metadata repository (CoMetaR). This study reflects how real users perceive the app by assessing usability scores and explicit usability issues. Methods: We evaluated the CoMetaR web app by measuring the usability of 3 modules: core module, provenance module, and data integration module. We defined 10 tasks in which users must acquire information specific to their user role. The participants were asked to complete the tasks in a live web meeting. We used the System Usability Scale questionnaire to measure the usability of the app. For qualitative analysis, we applied a modified think aloud method with the following thematic analysis and categorization into the ISO 9241-110 usability categories. Results: A total of 12 individuals participated in the study. We found that over 97% (85/88) of all the tasks were completed successfully. We measured usability scores of 81, 81, and 72 for the 3 evaluated modules. The qualitative analysis resulted in 24 issues with the app. Conclusions: A usability score of 81 implies very good usability for the 2 modules, whereas a usability score of 72 still indicates acceptable usability for the third module. We identified 24 issues that serve as starting points for further development. Our method proved to be effective and efficient in terms of effort and outcome. It can be adapted to evaluate apps within the medical informatics field and potentially beyond. %M 34847059 %R 10.2196/30308 %U https://medinform.jmir.org/2021/11/e30308 %U https://doi.org/10.2196/30308 %U http://www.ncbi.nlm.nih.gov/pubmed/34847059 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 11 %P e29768 %T A Pipeline to Understand Emerging Illness Via Social Media Data Analysis: Case Study on Breast Implant Illness %A Dey,Vishal %A Krasniak,Peter %A Nguyen,Minh %A Lee,Clara %A Ning,Xia %+ Department of Biomedical Informatics, The Ohio State University, 1800 Cannon Drive, Columbus, OH, 43210, United States, 1 6143662287, ning.104@osu.edu %K breast implant illness %K social media %K natural language processing %K topic modeling %D 2021 %7 29.11.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: A new illness can come to public attention through social media before it is medically defined, formally documented, or systematically studied. One example is a condition known as breast implant illness (BII), which has been extensively discussed on social media, although it is vaguely defined in the medical literature. Objective: The objective of this study is to construct a data analysis pipeline to understand emerging illnesses using social media data and to apply the pipeline to understand the key attributes of BII. Methods: We constructed a pipeline of social media data analysis using natural language processing and topic modeling. Mentions related to signs, symptoms, diseases, disorders, and medical procedures were extracted from social media data using the clinical Text Analysis and Knowledge Extraction System. We mapped the mentions to standard medical concepts and then summarized these mapped concepts as topics using latent Dirichlet allocation. Finally, we applied this pipeline to understand BII from several BII-dedicated social media sites. Results: Our pipeline identified topics related to toxicity, cancer, and mental health issues that were highly associated with BII. Our pipeline also showed that cancers, autoimmune disorders, and mental health problems were emerging concerns associated with breast implants, based on social media discussions. Furthermore, the pipeline identified mentions such as rupture, infection, pain, and fatigue as common self-reported issues among the public, as well as concerns about toxicity from silicone implants. Conclusions: Our study could inspire future studies on the suggested symptoms and factors of BII. Our study provides the first analysis and derived knowledge of BII from social media using natural language processing techniques and demonstrates the potential of using social media information to better understand similar emerging illnesses. %M 34847064 %R 10.2196/29768 %U https://medinform.jmir.org/2021/11/e29768 %U https://doi.org/10.2196/29768 %U http://www.ncbi.nlm.nih.gov/pubmed/34847064 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 10 %N 11 %P e29398 %T A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation %A Abdelkader,Wael %A Navarro,Tamara %A Parrish,Rick %A Cotoi,Chris %A Germini,Federico %A Linkins,Lori-Ann %A Iorio,Alfonso %A Haynes,R Brian %A Ananiadou,Sophia %A Chu,Lingyang %A Lokker,Cynthia %+ Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main St W, CRL-137, Hamilton, ON, L8S 4K1, Canada, 1 9055259140 ext 22208, lokkerc@mcmaster.ca %K bioinformatics %K machine learning %K evidence-based medicine %K literature retrieval %K medical informatics %K natural language processing %K NLP %K biomedical %K literature %K literature surveillance %K model development %D 2021 %7 29.11.2021 %9 Protocol %J JMIR Res Protoc %G English %X Background: A barrier to practicing evidence-based medicine is the rapidly increasing body of biomedical literature. Use of method terms to limit the search can help reduce the burden of screening articles for clinical relevance; however, such terms are limited by their partial dependence on indexing terms and usually produce low precision, especially when high sensitivity is required. Machine learning has been applied to the identification of high-quality literature with the potential to achieve high precision without sacrificing sensitivity. The use of artificial intelligence has shown promise to improve the efficiency of identifying sound evidence. Objective: The primary objective of this research is to derive and validate deep learning machine models using iterations of Bidirectional Encoder Representations from Transformers (BERT) to retrieve high-quality, high-relevance evidence for clinical consideration from the biomedical literature. Methods: Using the HuggingFace Transformers library, we will experiment with variations of BERT models, including BERT, BioBERT, BlueBERT, and PubMedBERT, to determine which have the best performance in article identification based on quality criteria. Our experiments will utilize a large data set of over 150,000 PubMed citations from 2012 to 2020 that have been manually labeled based on their methodological rigor for clinical use. We will evaluate and report on the performance of the classifiers in categorizing articles based on their likelihood of meeting quality criteria. We will report fine-tuning hyperparameters for each model, as well as their performance metrics, including recall (sensitivity), specificity, precision, accuracy, F-score, the number of articles that need to be read before finding one that is positive (meets criteria), and classification probability scores. Results: Initial model development is underway, with further development planned for early 2022. Performance testing is expected to star in February 2022. Results will be published in 2022. Conclusions: The experiments will aim to improve the precision of retrieving high-quality articles by applying a machine learning classifier to PubMed searching. International Registered Report Identifier (IRRID): DERR1-10.2196/29398 %M 34847061 %R 10.2196/29398 %U https://www.researchprotocols.org/2021/11/e29398 %U https://doi.org/10.2196/29398 %U http://www.ncbi.nlm.nih.gov/pubmed/34847061 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 11 %P e23101 %T Incorporating Domain Knowledge Into Language Models by Using Graph Convolutional Networks for Assessing Semantic Textual Similarity: Model Development and Performance Comparison %A Chang,David %A Lin,Eric %A Brandt,Cynthia %A Taylor,Richard Andrew %+ Yale Center for Medical Informatics, Yale University, Suite 501, 300 George St, New Haven, CT, United States, 1 2037854058, richard.taylor@yale.edu %K natural language processing %K graph neural networks %K National NLP Clinical Challenges %K bidirectional encoder representation from transformers %D 2021 %7 26.11.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Although electronic health record systems have facilitated clinical documentation in health care, they have also introduced new challenges, such as the proliferation of redundant information through the use of copy and paste commands or templates. One approach to trimming down bloated clinical documentation and improving clinical summarization is to identify highly similar text snippets with the goal of removing such text. Objective: We developed a natural language processing system for the task of assessing clinical semantic textual similarity. The system assigns scores to pairs of clinical text snippets based on their clinical semantic similarity. Methods: We leveraged recent advances in natural language processing and graph representation learning to create a model that combines linguistic and domain knowledge information from the MedSTS data set to assess clinical semantic textual similarity. We used bidirectional encoder representation from transformers (BERT)–based models as text encoders for the sentence pairs in the data set and graph convolutional networks (GCNs) as graph encoders for corresponding concept graphs that were constructed based on the sentences. We also explored techniques, including data augmentation, ensembling, and knowledge distillation, to improve the model’s performance, as measured by the Pearson correlation coefficient (r). Results: Fine-tuning the BERT_base and ClinicalBERT models on the MedSTS data set provided a strong baseline (Pearson correlation coefficients: 0.842 and 0.848, respectively) compared to those of the previous year’s submissions. Our data augmentation techniques yielded moderate gains in performance, and adding a GCN-based graph encoder to incorporate the concept graphs also boosted performance, especially when the node features were initialized with pretrained knowledge graph embeddings of the concepts (r=0.868). As expected, ensembling improved performance, and performing multisource ensembling by using different language model variants, conducting knowledge distillation with the multisource ensemble model, and taking a final ensemble of the distilled models further improved the system’s performance (Pearson correlation coefficients: 0.875, 0.878, and 0.882, respectively). Conclusions: This study presents a system for the MedSTS clinical semantic textual similarity benchmark task, which was created by combining BERT-based text encoders and GCN-based graph encoders in order to incorporate domain knowledge into the natural language processing pipeline. We also experimented with other techniques involving data augmentation, pretrained concept embeddings, ensembling, and knowledge distillation to further increase our system’s performance. Although the task and its benchmark data set are in the early stages of development, this study, as well as the results of the competition, demonstrates the potential of modern language model–based systems to detect redundant information in clinical notes. %M 34842531 %R 10.2196/23101 %U https://medinform.jmir.org/2021/11/e23101 %U https://doi.org/10.2196/23101 %U http://www.ncbi.nlm.nih.gov/pubmed/34842531 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 11 %P e29532 %T Introduction of Systematized Nomenclature of Medicine–Clinical Terms Coding Into an Electronic Health Record and Evaluation of its Impact: Qualitative and Quantitative Study %A Pankhurst,Tanya %A Evison,Felicity %A Atia,Jolene %A Gallier,Suzy %A Coleman,Jamie %A Ball,Simon %A McKee,Deborah %A Ryan,Steven %A Black,Ruth %+ NHS Foundation Trust, University Hospitals Birmingham, Mindelsohn Way, Birmingham, B15 2TG, United Kingdom, 44 7811357984, pankhurst.tanya@gmail.com %K coding standards %K clinical decision support %K Clinician led design %K clinician reported experience %K clinical usability %K data sharing %K diagnoses %K electronic health records %K electronic health record standards %K health data exchange %K health data research %K International Classification of Diseases version 10 (ICD-10) %K National Health Service Blueprint %K patient diagnoses %K population health %K problem list %K research %K Systematized Nomenclature Of Medicine–Clinical Terms (SNOMED-CT) %K use of electronic health data %K user-led design %D 2021 %7 23.11.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: This study describes the conversion within an existing electronic health record (EHR) from the International Classification of Diseases, Tenth Revision coding system to the SNOMED-CT (Systematized Nomenclature of Medicine–Clinical Terms) for the collection of patient histories and diagnoses. The setting is a large acute hospital that is designing and building its own EHR. Well-designed EHRs create opportunities for continuous data collection, which can be used in clinical decision support rules to drive patient safety. Collected data can be exchanged across health care systems to support patients in all health care settings. Data can be used for research to prevent diseases and protect future populations. Objective: The aim of this study was to migrate a current EHR, with all relevant patient data, to the SNOMED-CT coding system to optimize clinical use and clinical decision support, facilitate data sharing across organizational boundaries for national programs, and enable remodeling of medical pathways. Methods: The study used qualitative and quantitative data to understand the successes and gaps in the project, clinician attitudes toward the new tool, and the future use of the tool. Results: The new coding system (tool) was well received and immediately widely used in all specialties. This resulted in increased, accurate, and clinically relevant data collection. Clinicians appreciated the increased depth and detail of the new coding, welcomed the potential for both data sharing and research, and provided extensive feedback for further development. Conclusions: Successful implementation of the new system aligned the University Hospitals Birmingham NHS Foundation Trust with national strategy and can be used as a blueprint for similar projects in other health care settings. %M 34817387 %R 10.2196/29532 %U https://medinform.jmir.org/2021/11/e29532 %U https://doi.org/10.2196/29532 %U http://www.ncbi.nlm.nih.gov/pubmed/34817387 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 11 %P e26777 %T Natural Language Processing and Machine Learning Methods to Characterize Unstructured Patient-Reported Outcomes: Validation Study %A Lu,Zhaohua %A Sim,Jin-ah %A Wang,Jade X %A Forrest,Christopher B %A Krull,Kevin R %A Srivastava,Deokumar %A Hudson,Melissa M %A Robison,Leslie L %A Baker,Justin N %A Huang,I-Chan %+ Department of Epidemiology and Cancer Control, St. Jude Children's Research Hospital, MS 735, 262 Danny Thomas Pl, Memphis, TN, 38105, United States, 1 9015958369, I-Chan.Huang@STJUDE.ORG %K natural language processing %K machine learning %K PROs %K pediatric oncology %D 2021 %7 3.11.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Assessing patient-reported outcomes (PROs) through interviews or conversations during clinical encounters provides insightful information about survivorship. Objective: This study aims to test the validity of natural language processing (NLP) and machine learning (ML) algorithms in identifying different attributes of pain interference and fatigue symptoms experienced by child and adolescent survivors of cancer versus the judgment by PRO content experts as the gold standard to validate NLP/ML algorithms. Methods: This cross-sectional study focused on child and adolescent survivors of cancer, aged 8 to 17 years, and caregivers, from whom 391 meaning units in the pain interference domain and 423 in the fatigue domain were generated for analyses. Data were collected from the After Completion of Therapy Clinic at St. Jude Children’s Research Hospital. Experienced pain interference and fatigue symptoms were reported through in-depth interviews. After verbatim transcription, analyzable sentences (ie, meaning units) were semantically labeled by 2 content experts for each attribute (physical, cognitive, social, or unclassified). Two NLP/ML methods were used to extract and validate the semantic features: bidirectional encoder representations from transformers (BERT) and Word2vec plus one of the ML methods, the support vector machine or extreme gradient boosting. Receiver operating characteristic and precision-recall curves were used to evaluate the accuracy and validity of the NLP/ML methods. Results: Compared with Word2vec/support vector machine and Word2vec/extreme gradient boosting, BERT demonstrated higher accuracy in both symptom domains, with 0.931 (95% CI 0.905-0.957) and 0.916 (95% CI 0.887-0.941) for problems with cognitive and social attributes on pain interference, respectively, and 0.929 (95% CI 0.903-0.953) and 0.917 (95% CI 0.891-0.943) for problems with cognitive and social attributes on fatigue, respectively. In addition, BERT yielded superior areas under the receiver operating characteristic curve for cognitive attributes on pain interference and fatigue domains (0.923, 95% CI 0.879-0.997; 0.948, 95% CI 0.922-0.979) and superior areas under the precision-recall curve for cognitive attributes on pain interference and fatigue domains (0.818, 95% CI 0.735-0.917; 0.855, 95% CI 0.791-0.930). Conclusions: The BERT method performed better than the other methods. As an alternative to using standard PRO surveys, collecting unstructured PROs via interviews or conversations during clinical encounters and applying NLP/ML methods can facilitate PRO assessment in child and adolescent cancer survivors. %M 34730546 %R 10.2196/26777 %U https://www.jmir.org/2021/11/e26777 %U https://doi.org/10.2196/26777 %U http://www.ncbi.nlm.nih.gov/pubmed/34730546 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 11 %P e29120 %T Stroke Outcome Measurements From Electronic Medical Records: Cross-sectional Study on the Effectiveness of Neural and Nonneural Classifiers %A Zanotto,Bruna Stella %A Beck da Silva Etges,Ana Paula %A dal Bosco,Avner %A Cortes,Eduardo Gabriel %A Ruschel,Renata %A De Souza,Ana Claudia %A Andrade,Claudio M V %A Viegas,Felipe %A Canuto,Sergio %A Luiz,Washington %A Ouriques Martins,Sheila %A Vieira,Renata %A Polanczyk,Carisi %A André Gonçalves,Marcos %+ Computer Science Department, Universidade Federal de Minas Gerais, Avenue Antônio Carlos, 6627, Belo Horizonte, 31270-901, Brazil, 55 3134095860, mgoncalv@dcc.ufmg.br %K natural language processing %K stroke %K outcomes %K electronic medical records %K EHR %K electronic health records %K text processing %K data mining %K text classification %K patient outcomes %D 2021 %7 1.11.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: With the rapid adoption of electronic medical records (EMRs), there is an ever-increasing opportunity to collect data and extract knowledge from EMRs to support patient-centered stroke management. Objective: This study aims to compare the effectiveness of state-of-the-art automatic text classification methods in classifying data to support the prediction of clinical patient outcomes and the extraction of patient characteristics from EMRs. Methods: Our study addressed the computational problems of information extraction and automatic text classification. We identified essential tasks to be considered in an ischemic stroke value-based program. The 30 selected tasks were classified (manually labeled by specialists) according to the following value agenda: tier 1 (achieved health care status), tier 2 (recovery process), care related (clinical management and risk scores), and baseline characteristics. The analyzed data set was retrospectively extracted from the EMRs of patients with stroke from a private Brazilian hospital between 2018 and 2019. A total of 44,206 sentences from free-text medical records in Portuguese were used to train and develop 10 supervised computational machine learning methods, including state-of-the-art neural and nonneural methods, along with ontological rules. As an experimental protocol, we used a 5-fold cross-validation procedure repeated 6 times, along with subject-wise sampling. A heatmap was used to display comparative result analyses according to the best algorithmic effectiveness (F1 score), supported by statistical significance tests. A feature importance analysis was conducted to provide insights into the results. Results: The top-performing models were support vector machines trained with lexical and semantic textual features, showing the importance of dealing with noise in EMR textual representations. The support vector machine models produced statistically superior results in 71% (17/24) of tasks, with an F1 score >80% regarding care-related tasks (patient treatment location, fall risk, thrombolytic therapy, and pressure ulcer risk), the process of recovery (ability to feed orally or ambulate and communicate), health care status achieved (mortality), and baseline characteristics (diabetes, obesity, dyslipidemia, and smoking status). Neural methods were largely outperformed by more traditional nonneural methods, given the characteristics of the data set. Ontological rules were also effective in tasks such as baseline characteristics (alcoholism, atrial fibrillation, and coronary artery disease) and the Rankin scale. The complementarity in effectiveness among models suggests that a combination of models could enhance the results and cover more tasks in the future. Conclusions: Advances in information technology capacity are essential for scalability and agility in measuring health status outcomes. This study allowed us to measure effectiveness and identify opportunities for automating the classification of outcomes of specific tasks related to clinical conditions of stroke victims, and thus ultimately assess the possibility of proactively using these machine learning techniques in real-world situations. %M 34723829 %R 10.2196/29120 %U https://medinform.jmir.org/2021/11/e29120 %U https://doi.org/10.2196/29120 %U http://www.ncbi.nlm.nih.gov/pubmed/34723829 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 10 %P e29259 %T Transforming Anesthesia Data Into the Observational Medical Outcomes Partnership Common Data Model: Development and Usability Study %A Lamer,Antoine %A Abou-Arab,Osama %A Bourgeois,Alexandre %A Parrot,Adrien %A Popoff,Benjamin %A Beuscart,Jean-Baptiste %A Tavernier,Benoît %A Moussa,Mouhamed Djahoum %+ Univ. Lille, CHU Lille, ULR 2694 - METRICS: Évaluation des technologies de santé et des pratiques médicales, 2 place de Verdun, Lille, F-59000, France, 33 320626969, antoine.lamer@chru-lille.fr %K data reuse %K common data model %K Observational Medical Outcomes Partnership %K anesthesia %K data warehouse %K reproducible research %D 2021 %7 29.10.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Electronic health records (EHRs, such as those created by an anesthesia management system) generate a large amount of data that can notably be reused for clinical audits and scientific research. The sharing of these data and tools is generally affected by the lack of system interoperability. To overcome these issues, Observational Health Data Sciences and Informatics (OHDSI) developed the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to standardize EHR data and promote large-scale observational and longitudinal research. Anesthesia data have not previously been mapped into the OMOP CDM. Objective: The primary objective was to transform anesthesia data into the OMOP CDM. The secondary objective was to provide vocabularies, queries, and dashboards that might promote the exploitation and sharing of anesthesia data through the CDM. Methods: Using our local anesthesia data warehouse, a group of 5 experts from 5 different medical centers identified local concepts related to anesthesia. The concepts were then matched with standard concepts in the OHDSI vocabularies. We performed structural mapping between the design of our local anesthesia data warehouse and the OMOP CDM tables and fields. To validate the implementation of anesthesia data into the OMOP CDM, we developed a set of queries and dashboards. Results: We identified 522 concepts related to anesthesia care. They were classified as demographics, units, measurements, operating room steps, drugs, periods of interest, and features. After semantic mapping, 353 (67.7%) of these anesthesia concepts were mapped to OHDSI concepts. Further, 169 (32.3%) concepts related to periods and features were added to the OHDSI vocabularies. Then, 8 OMOP CDM tables were implemented with anesthesia data and 2 new tables (EPISODE and FEATURE) were added to store secondarily computed data. We integrated data from 5,72,609 operations and provided the code for a set of 8 queries and 4 dashboards related to anesthesia care. Conclusions: Generic data concerning demographics, drugs, units, measurements, and operating room steps were already available in OHDSI vocabularies. However, most of the intraoperative concepts (the duration of specific steps, an episode of hypotension, etc) were not present in OHDSI vocabularies. The OMOP mapping provided here enables anesthesia data reuse. %M 34714250 %R 10.2196/29259 %U https://www.jmir.org/2021/10/e29259 %U https://doi.org/10.2196/29259 %U http://www.ncbi.nlm.nih.gov/pubmed/34714250 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 10 %P e25378 %T Developing a RadLex-Based Named Entity Recognition Tool for Mining Textual Radiology Reports: Development and Performance Evaluation Study %A Tsuji,Shintaro %A Wen,Andrew %A Takahashi,Naoki %A Zhang,Hongjian %A Ogasawara,Katsuhiko %A Jiang,Gouqian %+ Department of Health Sciences Research, Department of Radiology, 200 First Street, SW, Rochester, MN , United States, 1 507 266 1327, Jiang.Guoqian@mayo.edu %K named entity recognition (NER) %K natural language processing (NLP) %K RadLex %K ontology %K stem term %D 2021 %7 29.10.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Named entity recognition (NER) plays an important role in extracting the features of descriptions such as the name and location of a disease for mining free-text radiology reports. However, the performance of existing NER tools is limited because the number of entities that can be extracted depends on the dictionary lookup. In particular, the recognition of compound terms is very complicated because of the variety of patterns. Objective: The aim of this study is to develop and evaluate an NER tool concerned with compound terms using RadLex for mining free-text radiology reports. Methods: We leveraged the clinical Text Analysis and Knowledge Extraction System (cTAKES) to develop customized pipelines using both RadLex and SentiWordNet (a general purpose dictionary). We manually annotated 400 radiology reports for compound terms in noun phrases and used them as the gold standard for performance evaluation (precision, recall, and F-measure). In addition, we created a compound terms–enhanced dictionary (CtED) by analyzing false negatives and false positives and applied it to another 100 radiology reports for validation. We also evaluated the stem terms of compound terms by defining two measures: occurrence ratio (OR) and matching ratio (MR). Results: The F-measure of cTAKES+RadLex+general purpose dictionary was 30.9% (precision 73.3% and recall 19.6%) and that of the combined CtED was 63.1% (precision 82.8% and recall 51%). The OR indicated that the stem terms of effusion, node, tube, and disease were used frequently, but it still lacks capturing compound terms. The MR showed that 71.85% (9411/13,098) of the stem terms matched with that of the ontologies, and RadLex improved approximately 22% of the MR from the cTAKES default dictionary. The OR and MR revealed that the characteristics of stem terms would have the potential to help generate synonymous phrases using the ontologies. Conclusions: We developed a RadLex-based customized pipeline for parsing radiology reports and demonstrated that CtED and stem term analysis has the potential to improve dictionary-based NER performance with regard to expanding vocabularies. %M 34714247 %R 10.2196/25378 %U https://www.jmir.org/2021/10/e25378 %U https://doi.org/10.2196/25378 %U http://www.ncbi.nlm.nih.gov/pubmed/34714247 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 10 %P e27396 %T Common Data Elements for Meaningful Stroke Documentation in Routine Care and Clinical Research: Retrospective Data Analysis %A Berenspöhler,Sarah %A Minnerup,Jens %A Dugas,Martin %A Varghese,Julian %+ Institute of Medical Informatics, Westfälische Wilhelms-University Münster, Albert Schweitzer Campus 1, Building A11, Münster, 48149, Germany, 49 251 83 55262, s.berenspoehler@t-online.de %K common data elements %K stroke %K documentation %D 2021 %7 12.10.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Medical information management for stroke patients is currently a very time-consuming endeavor. There are clear guidelines and procedures to treat patients having acute stroke, but it is not known how well these established practices are reflected in patient documentation. Objective: This study compares a variety of documentation processes regarding stroke. The main objective of this work is to provide an overview of the most commonly occurring medical concepts in stroke documentation and identify overlaps between different documentation contexts to allow for the definition of a core data set that could be used in potential data interfaces. Methods: Medical source documentation forms from different documentation contexts, including hospitals, clinical trials, registries, and international standards, regarding stroke treatment followed by rehabilitation were digitized in the operational data model. Each source data element was semantically annotated using the Unified Medical Language System. The concept codes were analyzed for semantic overlaps. A concept was considered common if it appeared in at least two documentation contexts. The resulting common concepts were extended with implementation details, including data types and permissible values based on frequent patterns of source data elements, using an established expert-based and semiautomatic approach. Results: In total, 3287 data elements were identified, and 1051 of these emerged as unique medical concepts. The 100 most frequent medical concepts cover 9.51% (100/1051) of all concept occurrences in stroke documentation, and the 50 most frequent concepts cover 4.75% (50/1051). A list of common data elements was implemented in different standardized machine-readable formats on a public metadata repository for interoperable reuse. Conclusions: Standardization of medical documentation is a prerequisite for data exchange as well as the transferability and reuse of data. In the long run, standardization would save time and money and extend the capabilities for which such data could be used. In the context of this work, a lack of standardization was observed regarding current information management. Free-form text fields and intricate questions complicate automated data access and transfer between institutions. This work also revealed the potential of a unified documentation process as a core data set of the 50 most frequent common data elements, accounting for 34% of the documentation in medical information management. Such a data set offers a starting point for standardized and interoperable data collection in routine care, quality management, and clinical research. %M 34636733 %R 10.2196/27396 %U https://medinform.jmir.org/2021/10/e27396 %U https://doi.org/10.2196/27396 %U http://www.ncbi.nlm.nih.gov/pubmed/34636733 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 10 %P e31980 %T Expressiveness of an International Semantic Standard for Wound Care: Mapping a Standardized Item Set for Leg Ulcers to the Systematized Nomenclature of Medicine–Clinical Terms %A Hüsers,Jens %A Przysucha,Mareike %A Esdar,Moritz %A John,Swen Malte %A Hübner,Ursula Hertha %+ University of Applied Sciences Osnabrück, Albrechtstr 30, Osnabrück, 49076, Germany, 49 5419692012, u.huebner@hs-osnabrueck.de %K wound care %K chronic wound %K chronic leg ulcer %K SNOMED CT %K health information exchange %K semantic interoperability %K terminology mapping %D 2021 %7 6.10.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Chronic health conditions are on the rise and are putting high economic pressure on health systems, as they require well-coordinated prevention and treatment. Among chronic conditions, chronic wounds such as cardiovascular leg ulcers have a high prevalence. Their treatment is highly interdisciplinary and regularly spans multiple care settings and organizations; this places particularly high demands on interoperable information exchange that can be achieved using international semantic standards, such as Systematized Nomenclature of Medicine–Clinical Terms (SNOMED CT). Objective: This study aims to investigate the expressiveness of SNOMED CT in the domain of wound care, and thereby its clinical usefulness and the potential need for extensions. Methods: A clinically consented and profession-independent wound care item set, the German National Consensus for the Documentation of Leg Wounds (NKDUC), was mapped onto the precoordinated concepts of the international reference terminology SNOMED CT. Before the mapping took place, the NKDUC was transformed into an information model that served to systematically identify relevant items. The mapping process was carried out in accordance with the ISO/TR 12300 formalism. As a result, the reliability, equivalence, and coverage rate were determined for all NKDUC items and sections. Results: The developed information model revealed 268 items to be mapped. Conducted by 3 health care professionals, the mapping resulted in moderate reliability (κ=0.512). Regarding the two best equivalence categories (symmetrical equivalence of meaning), the coverage rate of SNOMED CT was 67.2% (180/268) overall and 64.3% (108/168) specifically for wounds. The sections general medical condition (55/66, 83%), wound assessment (18/24, 75%), and wound status (37/57, 65%), showed higher coverage rates compared with the sections therapy (45/73, 62%), wound diagnostics (8/14, 57%), and patient demographics (17/34, 50%). Conclusions: The results yielded acceptable reliability values for the mapping procedure. The overall coverage rate shows that two-thirds of the items could be mapped symmetrically, which is a substantial portion of the source item set. Some wound care sections, such as general medical conditions and wound assessment, were covered better than other sections (wound status, diagnostics, and therapy). These deficiencies can be mitigated either by postcoordination or by the inclusion of new concepts in SNOMED CT. This study contributes to pushing interoperability in the domain of wound care, thereby responding to the high demand for information exchange in this field. Overall, this study adds another puzzle piece to the general knowledge about SNOMED CT in terms of its clinical usefulness and its need for further extensions. %M 34428171 %R 10.2196/31980 %U https://medinform.jmir.org/2021/10/e31980 %U https://doi.org/10.2196/31980 %U http://www.ncbi.nlm.nih.gov/pubmed/34428171 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 10 %P e30701 %T Improvement and Evaluation of the TOPCOP Taxonomy of Patient Portals: Taxonomy-Evaluation-Delphi (TED) Approach %A Glöggler,Michael %A Ammenwerth,Elske %+ Institute of Medical Informatics, UMIT – Private University for Health Sciences, Medical Informatics and Technology, Eduard-Wallnöfer-Zentrum 1, Hall in Tirol, 6060, Austria, 43 50 8648 ext 3809, michael.gloeggler@edu.umit-tirol.at %K taxonomy %K classification system %K patient portal %K EHR portal %K online EHR access %K evaluation %K Delphi study %K electronic health records %K digital health %K health information %K information management %K user perspectives %D 2021 %7 5.10.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Patient portals have been introduced in many countries over the last 10 years, but many health information managers still feel they have too little knowledge of patient portals. A taxonomy can help them to better compare and select portals. This has led us to develop the TOPCOP taxonomy for classifying and comparing patient portals. However, the taxonomy has not been evaluated by users. Objective: This study aimed to evaluate the taxonomy’s usefulness to support health information managers in comparing, classifying, defining a requirement profile for, and selecting patient portals and to improve the taxonomy where needed. Methods: We used a modified Delphi approach. We sampled a heterogeneous panel of 13 health information managers from 3 countries using the criterion sampling strategy. We conducted 4 anonymous survey rounds with qualitative and quantitative questions. In round 1, the panelists assessed the appropriateness of each dimension, and we collected new ideas to improve the dimensions. In rounds 2 and 3, the panelists iteratively evaluated the taxonomy that was revised based on round 1. In round 4, the panelists assessed the need for a taxonomy and the appropriateness of patient engagement as a distinguishing concept. Then, they compared 2 real portals with the final taxonomy and evaluated its usefulness for comparing portals, creating an initial requirement profile, and selecting patient portals. To determine group consensus, we applied the RAND/UCLA Appropriateness Method. Results: The final taxonomy consists of 25 dimensions with 65 characteristics. Five new dimensions were added to the original taxonomy, with 8 characteristics added to already existing dimensions. Group consensus was achieved on the need for such a taxonomy to compare portals, on patient engagement as an appropriate distinguishing concept, and on the comprehensibility of the taxonomy’s form. Further, consensus was achieved on the taxonomy’s usefulness for classifying and comparing portals, assisting users in better understanding portals, creating a requirement profile, and selecting portals. This allowed us to test the usefulness of the final taxonomy with the intended users. Conclusions: The TOPCOP taxonomy aims to support health information managers in comparing and selecting patient portals. By providing a standardized terminology to describe various aspects of patient portals independent of clinical setting or country, the taxonomy will also be useful for advancing research and evaluation of patient portals. %M 34403354 %R 10.2196/30701 %U https://www.jmir.org/2021/10/e30701 %U https://doi.org/10.2196/30701 %U http://www.ncbi.nlm.nih.gov/pubmed/34403354 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 9 %P e30161 %T Information Retrieval in an Infodemic: The Case of COVID-19 Publications %A Teodoro,Douglas %A Ferdowsi,Sohrab %A Borissov,Nikolay %A Kashani,Elham %A Vicente Alvarez,David %A Copara,Jenny %A Gouareb,Racha %A Naderi,Nona %A Amini,Poorya %+ Department of Radiology and Medical Informatics, University of Geneva, Campus Biotech G6-N3 - Chemin des Mines 9, Geneva, 1202, Switzerland, 41 022 379 0225, douglas.teodoro@unige.ch %K information retrieval %K multistage retrieval %K neural search %K deep learning %K COVID-19 %K coronavirus %K infodemic %K infodemiology %K literature %K online information %D 2021 %7 17.9.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: The COVID-19 global health crisis has led to an exponential surge in published scientific literature. In an attempt to tackle the pandemic, extremely large COVID-19–related corpora are being created, sometimes with inaccurate information, which is no longer at scale of human analyses. Objective: In the context of searching for scientific evidence in the deluge of COVID-19–related literature, we present an information retrieval methodology for effective identification of relevant sources to answer biomedical queries posed using natural language. Methods: Our multistage retrieval methodology combines probabilistic weighting models and reranking algorithms based on deep neural architectures to boost the ranking of relevant documents. Similarity of COVID-19 queries is compared to documents, and a series of postprocessing methods is applied to the initial ranking list to improve the match between the query and the biomedical information source and boost the position of relevant documents. Results: The methodology was evaluated in the context of the TREC-COVID challenge, achieving competitive results with the top-ranking teams participating in the competition. Particularly, the combination of bag-of-words and deep neural language models significantly outperformed an Okapi Best Match 25–based baseline, retrieving on average, 83% of relevant documents in the top 20. Conclusions: These results indicate that multistage retrieval supported by deep learning could enhance identification of literature for COVID-19–related questions posed using natural language. %M 34375298 %R 10.2196/30161 %U https://www.jmir.org/2021/9/e30161 %U https://doi.org/10.2196/30161 %U http://www.ncbi.nlm.nih.gov/pubmed/34375298 %0 Journal Article %@ 2368-7959 %I JMIR Publications %V 8 %N 9 %P e30827 %T Characteristics of Dimensional Psychopathology in Suicidal Patients With Major Psychiatric Disorders and Its Association With the Length of Hospital Stay: Algorithm Validation Study %A Lee,Dong Yun %A Park,Jimyung %A Noh,Jai Sung %A Roh,Hyun Woong %A Ha,Jae Ho %A Lee,Eun Young %A Son,Sang Joon %A Park,Rae Woong %+ Department of Biomedical Informatics, Ajou University School of Medicine, Gyeonggi-do, Suwon, 16499, Republic of Korea, 82 31 219 4471, rwpark99@gmail.com %K suicide %K computed phenotype %K natural language processing %K research domain criteria %K electronic health record %D 2021 %7 3.9.2021 %9 Original Paper %J JMIR Ment Health %G English %X Background: Suicide has emerged as a serious concern for public health; however, only few studies have revealed the differences between major psychiatric disorders and suicide. Recent studies have attempted to quantify research domain criteria (RDoC) into numeric scores to systematically use them in computerized methods. The RDoC scores were used to reveal the characteristics of suicide and its association with major psychiatric disorders. Objective: We intended to investigate the differences in the dimensional psychopathology among hospitalized suicidal patients and the association between the dimensional psychopathology of psychiatric disorders and length of hospital stay. Methods: This retrospective study enrolled hospitalized suicidal patients diagnosed with major psychiatric disorders (depression, schizophrenia, and bipolar disorder) between January 2010 and December 2020 at a tertiary hospital in South Korea. The RDoC scores were calculated using the patients’ admission notes. To measure the differences between psychiatric disorder cohorts, analysis of variance and the Cochran Q test were conducted and post hoc analysis for RDoC domains was performed with the independent two-sample t test. A linear regression model was used to analyze the association between the RDoC scores and sociodemographic features and comorbidity index. To estimate the association between the RDoC scores and length of hospital stay, multiple logistic regression models were applied to each psychiatric disorder group. Results: We retrieved 732 admissions for 571 patients (465 with depression, 73 with schizophrenia, and 33 with bipolar disorder). We found significant differences in the dimensional psychopathology according to the psychiatric disorders. The patient group with depression showed the highest negative RDoC domain scores. In the cognitive and social RDoC domains, the groups with schizophrenia and bipolar disorder scored higher than the group with depression. In the arousal RDoC domain, the depression and bipolar disorder groups scored higher than the group with schizophrenia. We identified significant associations between the RDoC scores and length of stay for the depression and bipolar disorder groups. The odds ratios (ORs) of the length of stay were increased because of the higher negative RDoC domain scores in the group with depression (OR 1.058, 95% CI 1.006-1.114) and decreased by higher arousal RDoC domain scores in the group with bipolar disorder (OR 0.537, 95% CI 0.285-0.815). Conclusions: This study showed the association between the dimensional psychopathology of major psychiatric disorders related to suicide and the length of hospital stay and identified differences in the dimensional psychopathology of major psychiatric disorders. This may provide new perspectives for understanding suicidal patients. %M 34477555 %R 10.2196/30827 %U https://mental.jmir.org/2021/9/e30827 %U https://doi.org/10.2196/30827 %U http://www.ncbi.nlm.nih.gov/pubmed/34477555 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 8 %P e23230 %T Automatic ICD-10 Coding and Training System: Deep Neural Network Based on Supervised Learning %A Chen,Pei-Fu %A Wang,Ssu-Ming %A Liao,Wei-Chih %A Kuo,Lu-Cheng %A Chen,Kuan-Chih %A Lin,Yu-Cheng %A Yang,Chi-Yu %A Chiu,Chi-Hao %A Chang,Shu-Chih %A Lai,Feipei %+ Department of Computer Science and Information Engineering, National Taiwan University, No 1, Sec 4, Roosevelt Road, Taipei, 10617, Taiwan, 886 0911126526, flai@ntu.edu.tw %K natural language processing %K deep learning %K International Classification of Diseases %K Recurrent Neural Network %K text classification %D 2021 %7 31.8.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: The International Classification of Diseases (ICD) code is widely used as the reference in medical system and billing purposes. However, classifying diseases into ICD codes still mainly relies on humans reading a large amount of written material as the basis for coding. Coding is both laborious and time-consuming. Since the conversion of ICD-9 to ICD-10, the coding task became much more complicated, and deep learning– and natural language processing–related approaches have been studied to assist disease coders. Objective: This paper aims at constructing a deep learning model for ICD-10 coding, where the model is meant to automatically determine the corresponding diagnosis and procedure codes based solely on free-text medical notes to improve accuracy and reduce human effort. Methods: We used diagnosis records of the National Taiwan University Hospital as resources and apply natural language processing techniques, including global vectors, word to vectors, embeddings from language models, bidirectional encoder representations from transformers, and single head attention recurrent neural network, on the deep neural network architecture to implement ICD-10 auto-coding. Besides, we introduced the attention mechanism into the classification model to extract the keywords from diagnoses and visualize the coding reference for training freshmen in ICD-10. Sixty discharge notes were randomly selected to examine the change in the F1-score and the coding time by coders before and after using our model. Results: In experiments on the medical data set of National Taiwan University Hospital, our prediction results revealed F1-scores of 0.715 and 0.618 for the ICD-10 Clinical Modification code and Procedure Coding System code, respectively, with a bidirectional encoder representations from transformers embedding approach in the Gated Recurrent Unit classification model. The well-trained models were applied on the ICD-10 web service for coding and training to ICD-10 users. With this service, coders can code with the F1-score significantly increased from a median of 0.832 to 0.922 (P<.05), but not in a reduced interval. Conclusions: The proposed model significantly improved the F1-score but did not decrease the time consumed in coding by disease coders. %M 34463639 %R 10.2196/23230 %U https://medinform.jmir.org/2021/8/e23230 %U https://doi.org/10.2196/23230 %U http://www.ncbi.nlm.nih.gov/pubmed/34463639 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 8 %P e20675 %T The Unified Medical Language System at 30 Years and How It Is Used and Published: Systematic Review and Content Analysis %A Jing,Xia %+ Department of Public Health Sciences, College of Behavioral, Social and Health Sciences, Clemson University, 511 Edwards Hall, Clemson, SC, 29634, United States, 1 8646563347, xjing@clemson.edu %K Unified Medical Language System %K systematic literature analysis %K biomedical informatics %K health informatics %D 2021 %7 27.8.2021 %9 Review %J JMIR Med Inform %G English %X Background: The Unified Medical Language System (UMLS) has been a critical tool in biomedical and health informatics, and the year 2021 marks its 30th anniversary. The UMLS brings together many broadly used vocabularies and standards in the biomedical field to facilitate interoperability among different computer systems and applications. Objective: Despite its longevity, there is no comprehensive publication analysis of the use of the UMLS. Thus, this review and analysis is conducted to provide an overview of the UMLS and its use in English-language peer-reviewed publications, with the objective of providing a comprehensive understanding of how the UMLS has been used in English-language peer-reviewed publications over the last 30 years. Methods: PubMed, ACM Digital Library, and the Nursing & Allied Health Database were used to search for studies. The primary search strategy was as follows: UMLS was used as a Medical Subject Headings term or a keyword or appeared in the title or abstract. Only English-language publications were considered. The publications were screened first, then coded and categorized iteratively, following the grounded theory. The review process followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Results: A total of 943 publications were included in the final analysis. Moreover, 32 publications were categorized into 2 categories; hence the total number of publications before duplicates are removed is 975. After analysis and categorization of the publications, UMLS was found to be used in the following emerging themes or areas (the number of publications and their respective percentages are given in parentheses): natural language processing (230/975, 23.6%), information retrieval (125/975, 12.8%), terminology study (90/975, 9.2%), ontology and modeling (80/975, 8.2%), medical subdomains (76/975, 7.8%), other language studies (53/975, 5.4%), artificial intelligence tools and applications (46/975, 4.7%), patient care (35/975, 3.6%), data mining and knowledge discovery (25/975, 2.6%), medical education (20/975, 2.1%), degree-related theses (13/975, 1.3%), digital library (5/975, 0.5%), and the UMLS itself (150/975, 15.4%), as well as the UMLS for other purposes (27/975, 2.8%). Conclusions: The UMLS has been used successfully in patient care, medical education, digital libraries, and software development, as originally planned, as well as in degree-related theses, the building of artificial intelligence tools, data mining and knowledge discovery, foundational work in methodology, and middle layers that may lead to advanced products. Natural language processing, the UMLS itself, and information retrieval are the 3 most common themes that emerged among the included publications. The results, although largely related to academia, demonstrate that UMLS achieves its intended uses successfully, in addition to achieving uses broadly beyond its original intentions. %M 34236337 %R 10.2196/20675 %U https://medinform.jmir.org/2021/8/e20675 %U https://doi.org/10.2196/20675 %U http://www.ncbi.nlm.nih.gov/pubmed/34236337 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 5 %N 8 %P e29500 %T Estimation of Psychological Distress in Japanese Youth Through Narrative Writing: Text-Based Stylometric and Sentiment Analyses %A Manabe,Masae %A Liew,Kongmeng %A Yada,Shuntaro %A Wakamiya,Shoko %A Aramaki,Eiji %+ Nara Institute of Science and Technology, 8916-5, Takayama-cho, Ikoma, Nara, Japan, 81 743 72 5250, aramaki@is.naist.jp %K psychological distress %K youth %K narratives %K natural language processing %K Japan %K mental health %K stress %K distress %K young adult %K teenager %K sentiment %D 2021 %7 12.8.2021 %9 Original Paper %J JMIR Form Res %G English %X Background: Internalizing mental illnesses associated with psychological distress are often underdetected. Text-based detection using natural language processing (NLP) methods is increasingly being used to complement conventional detection efforts. However, these approaches often rely on self-disclosure through autobiographical narratives that may not always be possible, especially in the context of the collectivistic Japanese culture. Objective: We propose the use of narrative writing as an alternative resource for mental illness detection in youth. Accordingly, in this study, we investigated the textual characteristics of narratives written by youth with psychological distress; our research focuses on the detection of psychopathological tendencies in written imaginative narratives. Methods: Using NLP tools such as stylometric measures and lexicon-based sentiment analysis, we examined short narratives from 52 Japanese youth (mean age 19.8 years, SD 3.1) obtained through crowdsourcing. Participants wrote a short narrative introduction to an imagined story before completing a questionnaire to quantify their tendencies toward psychological distress. Based on this score, participants were categorized into higher distress and lower distress groups. The written narratives were then analyzed using NLP tools and examined for between-group differences. Although outside the scope of this study, we also carried out a supplementary analysis of narratives written by adults using the same procedure. Results: Youth demonstrating higher tendencies toward psychological distress used significantly more positive (happiness-related) words, revealing differences in valence of the narrative content. No other significant differences were observed between the high and low distress groups. Conclusions: Youth with tendencies toward mental illness were found to write more positive stories that contained more happiness-related terms. These results may potentially have widespread implications on psychological distress screening on online platforms, particularly in cultures such as Japan that are not accustomed to self-disclosure. Although the mechanisms that we propose in explaining our results are speculative, we believe that this interpretation paves the way for future research in online surveillance and detection efforts. %M 34387556 %R 10.2196/29500 %U https://formative.jmir.org/2021/8/e29500 %U https://doi.org/10.2196/29500 %U http://www.ncbi.nlm.nih.gov/pubmed/34387556 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 8 %P e28229 %T A Fine-Tuned Bidirectional Encoder Representations From Transformers Model for Food Named-Entity Recognition: Algorithm Development and Validation %A Stojanov,Riste %A Popovski,Gorjan %A Cenikj,Gjorgjina %A Koroušić Seljak,Barbara %A Eftimov,Tome %+ Computer Systems Department, Jožef Stefan Institute, Jamova Cesta 39, Ljubljana, 1000, Slovenia, 386 1 477 3386, tome.eftimov@ijs.si %K food information extraction %K named-entity recognition %K fine-tuning BERT %K semantic annotation %K information extraction %K BERT %K bidirectional encoder representations from transformers %K natural language processing %K machine learning %D 2021 %7 9.8.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Recently, food science has been garnering a lot of attention. There are many open research questions on food interactions, as one of the main environmental factors, with other health-related entities such as diseases, treatments, and drugs. In the last 2 decades, a large amount of work has been done in natural language processing and machine learning to enable biomedical information extraction. However, machine learning in food science domains remains inadequately resourced, which brings to attention the problem of developing methods for food information extraction. There are only few food semantic resources and few rule-based methods for food information extraction, which often depend on some external resources. However, an annotated corpus with food entities along with their normalization was published in 2019 by using several food semantic resources. Objective: In this study, we investigated how the recently published bidirectional encoder representations from transformers (BERT) model, which provides state-of-the-art results in information extraction, can be fine-tuned for food information extraction. Methods: We introduce FoodNER, which is a collection of corpus-based food named-entity recognition methods. It consists of 15 different models obtained by fine-tuning 3 pretrained BERT models on 5 groups of semantic resources: food versus nonfood entity, 2 subsets of Hansard food semantic tags, FoodOn semantic tags, and Systematized Nomenclature of Medicine Clinical Terms food semantic tags. Results: All BERT models provided very promising results with 93.30% to 94.31% macro F1 scores in the task of distinguishing food versus nonfood entity, which represents the new state-of-the-art technology in food information extraction. Considering the tasks where semantic tags are predicted, all BERT models obtained very promising results once again, with their macro F1 scores ranging from 73.39% to 78.96%. Conclusions: FoodNER can be used to extract and annotate food entities in 5 different tasks: food versus nonfood entities and distinguishing food entities on the level of food groups by using the closest Hansard semantic tags, the parent Hansard semantic tags, the FoodOn semantic tags, or the Systematized Nomenclature of Medicine Clinical Terms semantic tags. %M 34383671 %R 10.2196/28229 %U https://www.jmir.org/2021/8/e28229 %U https://doi.org/10.2196/28229 %U http://www.ncbi.nlm.nih.gov/pubmed/34383671 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 8 %P e27842 %T Quality of Hospital Electronic Health Record (EHR) Data Based on the International Consortium for Health Outcomes Measurement (ICHOM) in Heart Failure: Pilot Data Quality Assessment Study %A Aerts,Hannelore %A Kalra,Dipak %A Sáez,Carlos %A Ramírez-Anguita,Juan Manuel %A Mayer,Miguel-Angel %A Garcia-Gomez,Juan M %A Durà-Hernández,Marta %A Thienpont,Geert %A Coorevits,Pascal %+ Research Programme on Biomedical Informatics, Hospital del Mar Medical Research Institute and Universitat Pompeu Fabra, C/ Dr Aiguader 88, Barcelona, 08003, Spain, 34 933 160 539, miguelangel.mayer@upf.edu %K data quality %K electronic health records %K heart failure %K value-based health insurance %K patient outcome assessment %D 2021 %7 4.8.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: There is increasing recognition that health care providers need to focus attention, and be judged against, the impact they have on the health outcomes experienced by patients. The measurement of health outcomes as a routine part of clinical documentation is probably the only scalable way of collecting outcomes evidence, since secondary data collection is expensive and error-prone. However, there is uncertainty about whether routinely collected clinical data within electronic health record (EHR) systems includes the data most relevant to measuring and comparing outcomes and if those items are collected to a good enough data quality to be relied upon for outcomes assessment, since several studies have pointed out significant issues regarding EHR data availability and quality. Objective: In this paper, we first describe a practical approach to data quality assessment of health outcomes, based on a literature review of existing frameworks for quality assessment of health data and multistakeholder consultation. Adopting this approach, we performed a pilot study on a subset of 21 International Consortium for Health Outcomes Measurement (ICHOM) outcomes data items from patients with congestive heart failure. Methods: All available registries compatible with the diagnosis of heart failure within an EHR data repository of a general hospital (142,345 visits and 12,503 patients) were extracted and mapped to the ICHOM format. We focused our pilot assessment on 5 commonly used data quality dimensions: completeness, correctness, consistency, uniqueness, and temporal stability. Results: We found high scores (>95%) for the consistency, completeness, and uniqueness dimensions. Temporal stability analyses showed some changes over time in the reported use of medication to treat heart failure, as well as in the recording of past medical conditions. Finally, the investigation of data correctness suggested several issues concerning the characterization of missing data values. Many of these issues appear to be introduced while mapping the IMASIS-2 relational database contents to the ICHOM format, as the latter requires a level of detail that is not explicitly available in the coded data of an EHR. Conclusions: Overall, results of this pilot study revealed good data quality for the subset of heart failure outcomes collected at the Hospital del Mar. Nevertheless, some important data errors were identified that were caused by fundamentally different data collection practices in routine clinical care versus research, for which the ICHOM standard set was originally developed. To truly examine to what extent hospitals today are able to routinely collect the evidence of their success in achieving good health outcomes, future research would benefit from performing more extensive data quality assessments, including all data items from the ICHOM standards set and across multiple hospitals. %M 34346902 %R 10.2196/27842 %U https://medinform.jmir.org/2021/8/e27842 %U https://doi.org/10.2196/27842 %U http://www.ncbi.nlm.nih.gov/pubmed/34346902 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 8 %P e25670 %T Construction of Genealogical Knowledge Graphs From Obituaries: Multitask Neural Network Extraction System %A He,Kai %A Yao,Lixia %A Zhang,JiaWei %A Li,Yufei %A Li,Chen %+ School of Computer Science and Technology, Xi’an Jiaotong University, Xianning West Road, 27th, Xi’an, 0086 710049, China, 86 158 0290 2703, cli@xjtu.edu.cn %K genealogical knowledge graph %K EHR %K information extraction %K genealogy %K neural network %D 2021 %7 4.8.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Genealogical information, such as that found in family trees, is imperative for biomedical research such as disease heritability and risk prediction. Researchers have used policyholder and their dependent information in medical claims data and emergency contacts in electronic health records (EHRs) to infer family relationships at a large scale. We have previously demonstrated that online obituaries can be a novel data source for building more complete and accurate family trees. Objective: Aiming at supplementing EHR data with family relationships for biomedical research, we built an end-to-end information extraction system using a multitask-based artificial neural network model to construct genealogical knowledge graphs (GKGs) from online obituaries. GKGs are enriched family trees with detailed information including age, gender, death and birth dates, and residence. Methods: Built on a predefined family relationship map consisting of 4 types of entities (eg, people’s name, residence, birth date, and death date) and 71 types of relationships, we curated a corpus containing 1700 online obituaries from the metropolitan area of Minneapolis and St Paul in Minnesota. We also adopted data augmentation technology to generate additional synthetic data to alleviate the issue of data scarcity for rare family relationships. A multitask-based artificial neural network model was then built to simultaneously detect names, extract relationships between them, and assign attributes (eg, birth dates and death dates, residence, age, and gender) to each individual. In the end, we assemble related GKGs into larger ones by identifying people appearing in multiple obituaries. Results: Our system achieved satisfying precision (94.79%), recall (91.45%), and F-1 measures (93.09%) on 10-fold cross-validation. We also constructed 12,407 GKGs, with the largest one made up of 4 generations and 30 people. Conclusions: In this work, we discussed the meaning of GKGs for biomedical research, presented a new version of a corpus with a predefined family relationship map and augmented training data, and proposed a multitask deep neural system to construct and assemble GKGs. The results show our system can extract and demonstrate the potential of enriching EHR data for more genetic research. We share the source codes and system with the entire scientific community on GitHub without the corpus for privacy protection. %M 34346903 %R 10.2196/25670 %U https://www.jmir.org/2021/8/e25670 %U https://doi.org/10.2196/25670 %U http://www.ncbi.nlm.nih.gov/pubmed/34346903 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 7 %P e28218 %T Head and Tail Entity Fusion Model in Medical Knowledge Graph Construction: Case Study for Pituitary Adenoma %A Fang,An %A Lou,Pei %A Hu,Jiahui %A Zhao,Wanqing %A Feng,Ming %A Ren,Huiling %A Chen,Xianlai %+ Big Data Institute, Central South University, 932 South Lushan Road, Changsha, 410083, China, 86 731 88879583, chenxianlai@csu.edu.cn %K knowledge graph %K pituitary adenoma %K entity fusion %K similarity calculation %D 2021 %7 22.7.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Pituitary adenoma is one of the most common central nervous system tumors. The diagnosis and treatment of pituitary adenoma remain very difficult. Misdiagnosis and recurrence often occur, and experienced neurosurgeons are in serious shortage. A knowledge graph can help interns quickly understand the medical knowledge related to pituitary tumor. Objective: The aim of this study was to develop a data fusion method suitable for medical data using data of pituitary adenomas integrated from different sources. The overall goal was to construct a knowledge graph for pituitary adenoma (KGPA) to be used for knowledge discovery. Methods: A complete framework suitable for the construction of a medical knowledge graph was developed, which was used to build the KGPA. The schema of the KGPA was manually constructed. Information of pituitary adenoma was automatically extracted from Chinese electronic medical records (CEMRs) and medical websites through a conditional random field model and newly designed web wrappers. An entity fusion method is proposed based on the head-and-tail entity fusion model to fuse the data from heterogeneous sources. Results: Data were extracted from 300 CEMRs of pituitary adenoma and 4 health portals. Entity fusion was carried out using the proposed data fusion model. The F1 scores of the head and tail entity fusions were 97.32% and 98.57%, respectively. Triples from the constructed KGPA were selected for evaluation, demonstrating 95.4% accuracy. Conclusions: This paper introduces an approach to fuse triples extracted from heterogeneous data sources, which can be used to build a knowledge graph. The evaluation results showed that the data in the KGPA are of high quality. The constructed KGPA can help physicians in clinical practice. %M 34057414 %R 10.2196/28218 %U https://medinform.jmir.org/2021/7/e28218 %U https://doi.org/10.2196/28218 %U http://www.ncbi.nlm.nih.gov/pubmed/34057414 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 5 %N 7 %P e28738 %T Studying How Individuals Who Express the Feeling of Loneliness in an Online Loneliness Forum Communicate in a Nonloneliness Forum: Observational Study %A Andy,Anietie %+ Penn Medicine, 3400 Civic Blvd, Philadelphia, PA, , United States, 1 202 486 4095, andyanietie@gmail.com %K loneliness %K Reddit %K nonloneliness %K mental health %K eHealth %K forum %K online forum %K communication %K natural language processing %K language %K linguistics %D 2021 %7 20.7.2021 %9 Original Paper %J JMIR Form Res %G English %X Background: Loneliness is a public health concern, and increasingly, individuals experiencing loneliness are seeking support on online forums, some of which focus on discussions around loneliness (loneliness forums). Some of these individuals may also seek support around loneliness on online forums not related to loneliness or well-being (nonloneliness forums). Hence, to design and implement appropriate and efficient online loneliness interventions, it is important to understand how individuals who express and seek support around loneliness on online loneliness forums communicate in nonloneliness forums; this could provide further insights into the support needs and concerns of these users. Objective: This study aims to explore how users who express the feeling of loneliness and seek support around loneliness on an online loneliness forum communicate in an online nonloneliness forum. Methods: A total of 2401 users who expressed loneliness in posts published on a loneliness forum on Reddit and had published posts in a nonloneliness forum were identified. Using latent Dirichlet allocation (a natural language processing algorithm); Linguistic Inquiry and Word Count (a psycholinguistic dictionary); and the word score–based language features valence, arousal, and dominance, the language use differences in posts published in the nonloneliness forum by these users compared to a control group of users who did not belong to any loneliness forum on Reddit were determined. Results: It was found that in posts published in the nonloneliness forum, users who expressed loneliness tend to use more words associated with the Linguistic Inquiry and Word Count categories on sadness (Cohen d=0.10) and seeking to socialize (Cohen d=0.114), and use words associated with valence (Cohen d=0.364) and dominance (Cohen d=0.117). In addition, they tend to publish posts related to latent Dirichlet allocation topics such as relationships (Cohen d=0.105) and family and friends and mental health (Cohen d=0.10). Conclusions: There are clear distinctions in language use in nonloneliness forum posts by users who express loneliness compared to a control group of users. These findings can help with the design and implementation of online interventions around loneliness. %M 34283026 %R 10.2196/28738 %U https://formative.jmir.org/2021/7/e28738 %U https://doi.org/10.2196/28738 %U http://www.ncbi.nlm.nih.gov/pubmed/34283026 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 7 %P e26714 %T A Biomedical Knowledge Graph System to Propose Mechanistic Hypotheses for Real-World Environmental Health Observations: Cohort Study and Informatics Application %A Fecho,Karamarie %A Bizon,Chris %A Miller,Frederick %A Schurman,Shepherd %A Schmitt,Charles %A Xue,William %A Morton,Kenneth %A Wang,Patrick %A Tropsha,Alexander %+ Renaissance Computing Institute, University of North Carolina at Chapel Hill, 100 Europa Drive, Suite 540, Chapel Hill, NC, 27517, United States, 1 919 445 9640, kfecho@copperlineprofessionalsolutions.com %K knowledge graph %K knowledge representation %K data exploration %K generalizability %K discovery %K open science %K immune-mediated disease %D 2021 %7 20.7.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Knowledge graphs are a common form of knowledge representation in biomedicine and many other fields. We developed an open biomedical knowledge graph–based system termed Reasoning Over Biomedical Objects linked in Knowledge Oriented Pathways (ROBOKOP). ROBOKOP consists of both a front-end user interface and a back-end knowledge graph. The ROBOKOP user interface allows users to posit questions and explore answer subgraphs. Users can also posit questions through direct Cypher query of the underlying knowledge graph, which currently contains roughly 6 million nodes or biomedical entities and 140 million edges or predicates describing the relationship between nodes, drawn from over 30 curated data sources. Objective: We aimed to apply ROBOKOP to survey data on workplace exposures and immune-mediated diseases from the Environmental Polymorphisms Registry (EPR) within the National Institute of Environmental Health Sciences. Methods: We analyzed EPR survey data and identified 45 associations between workplace chemical exposures and immune-mediated diseases, as self-reported by study participants (n= 4574), with 20 associations significant at P<.05 after false discovery rate correction. We then used ROBOKOP to (1) validate the associations by determining whether plausible connections exist within the ROBOKOP knowledge graph and (2) propose biological mechanisms that might explain them and serve as hypotheses for subsequent testing. We highlight the following three exemplar associations: carbon monoxide-multiple sclerosis, ammonia-asthma, and isopropanol-allergic disease. Results: ROBOKOP successfully returned answer sets for three queries that were posed in the context of the driving examples. The answer sets included potential intermediary genes, as well as supporting evidence that might explain the observed associations. Conclusions: We demonstrate real-world application of ROBOKOP to generate mechanistic hypotheses for associations between workplace chemical exposures and immune-mediated diseases. We expect that ROBOKOP will find broad application across many biomedical fields and other scientific disciplines due to its generalizability, speed to discovery and generation of mechanistic hypotheses, and open nature. %M 34283031 %R 10.2196/26714 %U https://medinform.jmir.org/2021/7/e26714 %U https://doi.org/10.2196/26714 %U http://www.ncbi.nlm.nih.gov/pubmed/34283031 %0 Journal Article %@ 2369-1999 %I JMIR Publications %V 7 %N 3 %P e27970 %T A Natural Language Processing–Assisted Extraction System for Gleason Scores: Development and Usability Study %A Yu,Shun %A Le,Anh %A Feld,Emily %A Schriver,Emily %A Gabriel,Peter %A Doucette,Abigail %A Narayan,Vivek %A Feldman,Michael %A Schwartz,Lauren %A Maxwell,Kara %A Mowery,Danielle %+ University of Pennsylvania Health System, 3400 Spruce Street, Philadelphia, PA, 19103, United States, 1 215 316 5151, Shun.Yu@pennmedicine.upenn.edu %K NLP %K Gleason score %K prostate cancer %K natural language processing %D 2021 %7 2.7.2021 %9 Original Paper %J JMIR Cancer %G English %X Background: Natural language processing (NLP) offers significantly faster variable extraction compared to traditional human extraction but cannot interpret complicated notes as well as humans can. Thus, we hypothesized that an “NLP-assisted” extraction system, which uses humans for complicated notes and NLP for uncomplicated notes, could produce faster extraction without compromising accuracy. Objective: The aim of this study was to develop and pilot an NLP-assisted extraction system to leverage the strengths of both human and NLP extraction of prostate cancer Gleason scores. Methods: We collected all available clinical and pathology notes for prostate cancer patients in an unselected academic biobank cohort. We developed an NLP system to extract prostate cancer Gleason scores from both clinical and pathology notes. Next, we designed and implemented the NLP-assisted extraction system algorithm to categorize notes into “uncomplicated” and “complicated” notes. Uncomplicated notes were assigned to NLP extraction and complicated notes were assigned to human extraction. We randomly reviewed 200 patients to assess the accuracy and speed of our NLP-assisted extraction system and compared it to NLP extraction alone and human extraction alone. Results: Of the 2051 patients in our cohort, the NLP system extracted a prostate surgery Gleason score from 1147 (55.92%) patients and a prostate biopsy Gleason score from 1624 (79.18%) patients. Our NLP-assisted extraction system had an overall accuracy rate of 98.7%, which was similar to the accuracy of human extraction alone (97.5%; P=.17) and significantly higher than the accuracy of NLP extraction alone (95.3%; P<.001). Moreover, our NLP-assisted extraction system reduced the workload of human extractors by approximately 95%, resulting in an average extraction time of 12.7 seconds per patient (vs 256.1 seconds per patient for human extraction alone). Conclusions: We demonstrated that an NLP-assisted extraction system was able to achieve much faster Gleason score extraction compared to traditional human extraction without sacrificing accuracy. %M 34255641 %R 10.2196/27970 %U https://cancer.jmir.org/2021/3/e27970 %U https://doi.org/10.2196/27970 %U http://www.ncbi.nlm.nih.gov/pubmed/34255641 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 6 %P e29667 %T A Word Pair Dataset for Semantic Similarity and Relatedness in Korean Medical Vocabulary: Reference Development and Validation %A Yum,Yunjin %A Lee,Jeong Moon %A Jang,Moon Joung %A Kim,Yoojoong %A Kim,Jong-Ho %A Kim,Seongtae %A Shin,Unsub %A Song,Sanghoun %A Joo,Hyung Joon %+ Korea University Research Institute for Medical Bigdata Science, Korea University Anam Hospital, 145 Anam-ro, Seoungbuk-gu, Seoul, 02841, Republic of Korea, 82 010 3476 0526, drjoohj@gmail.com %K medical word pair %K similarity %K relatedness %K word embedding %K fastText %K Korean %D 2021 %7 24.6.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: The fact that medical terms require special expertise and are becoming increasingly complex makes it difficult to employ natural language processing techniques in medical informatics. Several human-validated reference standards for medical terms have been developed to evaluate word embedding models using the semantic similarity and relatedness of medical word pairs. However, there are very few reference standards in non-English languages. In addition, because the existing reference standards were developed a long time ago, there is a need to develop an updated standard to represent recent findings in medical sciences. Objective: We propose a new Korean word pair reference set to verify embedding models. Methods: From January 2010 to December 2020, 518 medical textbooks, 72,844 health information news, and 15,698 medical research articles were collected, and the top 10,000 medical terms were selected to develop medical word pairs. Attending physicians (n=16) participated in the verification of the developed set with 607 word pairs. Results: The proportion of word pairs answered by all participants was 90.8% (551/607) for the similarity task and 86.5% (525/605) for the relatedness task. The similarity and relatedness of the word pair showed a high correlation (ρ=0.70, P<.001). The intraclass correlation coefficients to assess the interrater agreements of the word pair sets were 0.47 on the similarity task and 0.53 on the relatedness task. The final reference standard was 604 word pairs for the similarity task and 599 word pairs for relatedness, excluding word pairs with answers corresponding to outliers and word pairs that were answered by less than 50% of all the respondents. When FastText models were applied to the final reference standard word pair sets, the embedding models learning medical documents had a higher correlation between the calculated cosine similarity scores compared to human-judged similarity and relatedness scores (namu, ρ=0.12 vs with medical text for the similarity task, ρ=0.47; namu, ρ=0.02 vs with medical text for the relatedness task, ρ=0.30). Conclusions: Korean medical word pair reference standard sets for semantic similarity and relatedness were developed based on medical documents from the past 10 years. It is expected that our word pair reference sets will be actively utilized in the development of medical and multilingual natural language processing technology in the future. %M 34185005 %R 10.2196/29667 %U https://medinform.jmir.org/2021/6/e29667/ %U https://doi.org/10.2196/29667 %U http://www.ncbi.nlm.nih.gov/pubmed/34185005 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 6 %P e28277 %T Drug-Drug Interaction Predictions via Knowledge Graph and Text Embedding: Instrument Validation Study %A Wang,Meng %A Wang,Haofen %A Liu,Xing %A Ma,Xinyu %A Wang,Beilun %+ College of Design and Innovation, Tongji University, No. 281 Fuxin Road, Yangpu District, Shanghai, 200092, China, 86 139 1858 6855, carter.whfcarter@gmail.com %K drug-drug interactions %K knowledge graph %K natural language processing %D 2021 %7 19.6.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Minimizing adverse reactions caused by drug-drug interactions (DDIs) has always been a prominent research topic in clinical pharmacology. Detecting all possible interactions through clinical studies before a drug is released to the market is a demanding task. The power of big data is opening up new approaches to discovering various DDIs. However, these data contain a huge amount of noise and provide knowledge bases that are far from being complete or used with reliability. Most existing studies focus on predicting binary DDIs between drug pairs and ignore other interactions. Objective: Leveraging both drug knowledge graphs and biomedical text is a promising pathway for rich and comprehensive DDI prediction, but it is not without issues. Our proposed model seeks to address the following challenges: data noise and incompleteness, data sparsity, and computational complexity. Methods: We propose a novel framework, Predicting Rich DDI, to predict DDIs. The framework uses graph embedding to overcome data incompleteness and sparsity issues to make multiple DDI label predictions. First, a large-scale drug knowledge graph is generated from different sources. The knowledge graph is then embedded with comprehensive biomedical text into a common low-dimensional space. Finally, the learned embeddings are used to efficiently compute rich DDI information through a link prediction process. Results: To validate the effectiveness of the proposed framework, extensive experiments were conducted on real-world data sets. The results demonstrate that our model outperforms several state-of-the-art baseline methods in terms of capability and accuracy. Conclusions: We propose a novel framework, Predicting Rich DDI, to predict DDIs. Using rich DDI information, it can competently predict multiple labels for a pair of drugs across numerous domains, ranging from pharmacological mechanisms to side effects. To the best of our knowledge, this framework is the first to provide a joint translation-based embedding model that learns DDIs by integrating drug knowledge graphs and biomedical text simultaneously in a common low-dimensional space. The model also predicts DDIs using multiple labels rather than single or binary labels. Extensive experiments were conducted on real-world data sets to demonstrate the effectiveness and efficiency of the model. The results show our proposed framework outperforms several state-of-the-art baselines. %M 34185011 %R 10.2196/28277 %U https://medinform.jmir.org/2021/6/e28277/ %U https://doi.org/10.2196/28277 %U http://www.ncbi.nlm.nih.gov/pubmed/34185011 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 6 %P e27591 %T A National, Semantic-Driven, Three-Pillar Strategy to Enable Health Data Secondary Usage Interoperability for Research Within the Swiss Personalized Health Network: Methodological Study %A Gaudet-Blavignac,Christophe %A Raisaro,Jean Louis %A Touré,Vasundra %A Österle,Sabine %A Crameri,Katrin %A Lovis,Christian %+ Division of Medical Information Sciences, Geneva University Hospitals, Rue Gabrielle-Perret-Gentil 4, Geneva, 1205, Switzerland, 41 223726201, christophe.gaudet-blavignac@hcuge.ch %K interoperability %K clinical data reuse %K personalized medicine %D 2021 %7 24.6.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Interoperability is a well-known challenge in medical informatics. Current trends in interoperability have moved from a data model technocentric approach to sustainable semantics, formal descriptive languages, and processes. Despite many initiatives and investments for decades, the interoperability challenge remains crucial. The need for data sharing for most purposes ranging from patient care to secondary uses, such as public health, research, and quality assessment, faces unmet problems. Objective: This work was performed in the context of a large Swiss Federal initiative aiming at building a national infrastructure for reusing consented data acquired in the health care and research system to enable research in the field of personalized medicine in Switzerland. The initiative is the Swiss Personalized Health Network (SPHN). This initiative is providing funding to foster use and exchange of health-related data for research. As part of the initiative, a national strategy to enable a semantically interoperable clinical data landscape was developed and implemented. Methods: A deep analysis of various approaches to address interoperability was performed at the start, including large frameworks in health care, such as Health Level Seven (HL7) and Integrating Healthcare Enterprise (IHE), and in several domains, such as regulatory agencies (eg, Clinical Data Interchange Standards Consortium [CDISC]) and research communities (eg, Observational Medical Outcome Partnership [OMOP]), to identify bottlenecks and assess sustainability. Based on this research, a strategy composed of three pillars was designed. It has strong multidimensional semantics, descriptive formal language for exchanges, and as many data models as needed to comply with the needs of various communities. Results: This strategy has been implemented stepwise in Switzerland since the middle of 2019 and has been adopted by all university hospitals and high research organizations. The initiative is coordinated by a central organization, the SPHN Data Coordination Center of the SIB Swiss Institute of Bioinformatics. The semantics is mapped by domain experts on various existing standards, such as Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT), Logical Observation Identifiers Names and Codes (LOINC), and International Classification of Diseases (ICD). The resource description framework (RDF) is used for storing and transporting data, and to integrate information from different sources and standards. Data transformers based on SPARQL query language are implemented to convert RDF representations to the numerous data models required by the research community or bridge with other systems, such as electronic case report forms. Conclusions: The SPHN strategy successfully implemented existing standards in a pragmatic and applicable way. It did not try to build any new standards but used existing ones in a nondogmatic way. It has now been funded for another 4 years, bringing the Swiss landscape into a new dimension to support research in the field of personalized medicine and large interoperable clinical data. %M 34185008 %R 10.2196/27591 %U https://medinform.jmir.org/2021/6/e27591/ %U https://doi.org/10.2196/27591 %U http://www.ncbi.nlm.nih.gov/pubmed/34185008 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 6 %P e28247 %T A Novel Metric to Quantify the Effect of Pathway Enrichment Evaluation With Respect to Biomedical Text-Mined Terms: Development and Feasibility Study %A Qin,Xuan %A Yao,Xinzhi %A Xia,Jingbo %+ Hubei Key Lab of Agricultural Bioinformatics, College of Informatics, Huazhong Agricultural University, 1#, Lion Rock Street, Hongshan District, Hubei Province, Wuhan, 430070, China, 86 02787288509, xiajingbo.math@gmail.com %K pathway enrichment %K metric %K evaluation %K text mining %D 2021 %7 18.6.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Natural language processing has long been applied in various applications for biomedical knowledge inference and discovery. Enrichment analysis based on named entity recognition is a classic application for inferring enriched associations in terms of specific biomedical entities such as gene, chemical, and mutation. Objective: The aim of this study was to investigate the effect of pathway enrichment evaluation with respect to biomedical text-mining results and to develop a novel metric to quantify the effect. Methods: Four biomedical text mining methods were selected to represent natural language processing methods on drug-related gene mining. Subsequently, a pathway enrichment experiment was performed by using the mined genes, and a series of inverse pathway frequency (IPF) metrics was proposed accordingly to evaluate the effect of pathway enrichment. Thereafter, 7 IPF metrics and traditional P value metrics were compared in simulation experiments to test the robustness of the proposed metrics. Results: IPF metrics were evaluated in a case study of rapamycin-related gene set. By applying the best IPF metrics in a pathway enrichment simulation test, a novel discovery of drug efficacy of rapamycin for breast cancer was replicated from the data chosen prior to the year 2000. Our findings show the effectiveness of the best IPF metric in support of knowledge discovery in new drug use. Further, the mechanism underlying the drug-disease association was visualized by Cytoscape. Conclusions: The results of this study suggest the effectiveness of the proposed IPF metrics in pathway enrichment evaluation as well as its application in drug use discovery. %M 34142969 %R 10.2196/28247 %U https://medinform.jmir.org/2021/6/e28247 %U https://doi.org/10.2196/28247 %U http://www.ncbi.nlm.nih.gov/pubmed/34142969 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 6 %P e26892 %T Constructing High-Fidelity Phenotype Knowledge Graphs for Infectious Diseases With a Fine-Grained Semantic Information Model: Development and Usability Study %A Deng,Lizong %A Chen,Luming %A Yang,Tao %A Liu,Mi %A Li,Shicheng %A Jiang,Taijiao %+ Center of Systems Medicine, Institute of Basic Medical Sciences, Chinese Academy of Medical Sciences & Peking Union Medical College, #5 Dong Dan San Tiao, Dongcheng District, Beijing, 100005, China, 86 051262873781, taijiao@ibms.pumc.edu.cn %K knowledge graph %K knowledge granularity %K machine learning %K high-fidelity phenotyping %K phenotyping %K phenotype %K semantic %D 2021 %7 15.6.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Phenotypes characterize the clinical manifestations of diseases and provide important information for diagnosis. Therefore, the construction of phenotype knowledge graphs for diseases is valuable to the development of artificial intelligence in medicine. However, phenotype knowledge graphs in current knowledge bases such as WikiData and DBpedia are coarse-grained knowledge graphs because they only consider the core concepts of phenotypes while neglecting the details (attributes) associated with these phenotypes. Objective: To characterize the details of disease phenotypes for clinical guidelines, we proposed a fine-grained semantic information model named PhenoSSU (semantic structured unit of phenotypes). Methods: PhenoSSU is an “entity-attribute-value” model by its very nature, and it aims to capture the full semantic information underlying phenotype descriptions with a series of attributes and values. A total of 193 clinical guidelines for infectious diseases from Wikipedia were selected as the study corpus, and 12 attributes from SNOMED-CT were introduced into the PhenoSSU model based on the co-occurrences of phenotype concepts and attribute values. The expressive power of the PhenoSSU model was evaluated by analyzing whether PhenoSSU instances could capture the full semantics underlying the descriptions of the corresponding phenotypes. To automatically construct fine-grained phenotype knowledge graphs, a hybrid strategy that first recognized phenotype concepts with the MetaMap tool and then predicted the attribute values of phenotypes with machine learning classifiers was developed. Results: Fine-grained phenotype knowledge graphs of 193 infectious diseases were manually constructed with the BRAT annotation tool. A total of 4020 PhenoSSU instances were annotated in these knowledge graphs, and 3757 of them (89.5%) were found to be able to capture the full semantics underlying the descriptions of the corresponding phenotypes listed in clinical guidelines. By comparison, other information models, such as the clinical element model and the HL7 fast health care interoperability resource model, could only capture the full semantics underlying 48.4% (2034/4020) and 21.8% (914/4020) of the descriptions of phenotypes listed in clinical guidelines, respectively. The hybrid strategy achieved an F1-score of 0.732 for the subtask of phenotype concept recognition and an average weighted accuracy of 0.776 for the subtask of attribute value prediction. Conclusions: PhenoSSU is an effective information model for the precise representation of phenotype knowledge for clinical guidelines, and machine learning can be used to improve the efficiency of constructing PhenoSSU-based knowledge graphs. Our work will potentially shift the focus of medical knowledge engineering from a coarse-grained level to a more fine-grained level. %M 34128811 %R 10.2196/26892 %U https://www.jmir.org/2021/6/e26892 %U https://doi.org/10.2196/26892 %U http://www.ncbi.nlm.nih.gov/pubmed/34128811 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 6 %P e25028 %T A Determinants-of-Fertility Ontology for Detecting Future Signals of Fertility Issues From Social Media Data: Development of an Ontology %A Lee,Ji-Hyun %A Park,Hyeoun-Ae %A Song,Tae-Min %+ College of Nursing and Research Institute of Nursing Science, Seoul National University, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Republic of Korea, 82 27408827, hapark@snu.ac.kr %K ontology %K fertility %K public policy %K South Korea %K social media %K future %K infodemiology %K infoveillance %D 2021 %7 14.6.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: South Korea has the lowest fertility rate in the world despite considerable governmental efforts to boost it. Increasing the fertility rate and achieving the desired outcomes of any implemented policies requires reliable data on the ongoing trends in fertility and preparations for the future based on these trends. Objective: The aims of this study were to (1) develop a determinants-of-fertility ontology with terminology for collecting and analyzing social media data; (2) determine the description logics, content coverage, and structural and representational layers of the ontology; and (3) use the ontology to detect future signals of fertility issues. Methods: An ontology was developed using the Ontology Development 101 methodology. The domain and scope of the ontology were defined by compiling a list of competency questions. The terms were collected from Korean government reports, Korea’s Basic Plan for Low Fertility and Aging Society, a national survey about marriage and childbirth, and social media postings on fertility issues. The classes and their hierarchy were defined using a top-down approach based on an ecological model. The internal structure of classes was defined using the entity-attribute-value model. The description logics of the ontology were evaluated using Protégé (version 5.5.0), and the content coverage was evaluated by comparing concepts extracted from social media posts with the list of ontology classes. The structural and representational layers of the ontology were evaluated by experts. Social media data were collected from 183 online channels between January 1, 2011, and June 30, 2015. To detect future signals of fertility issues, 2 classes of the ontology, the socioeconomic and cultural environment, and public policy, were identified as keywords. A keyword issue map was constructed, and the defined keywords were mapped to identify future signals. R software (version 3.5.2) was used to mine for future signals. Results: A determinants-of-fertility ontology comprised 236 classes and terminology comprised 1464 synonyms of the 236 classes. Concept classes in the ontology were found to be coherently and consistently defined. The ontology included more than 90% of the concepts that appeared in social media posts on fertility policies. Average scores for all of the criteria for structural and representations layers exceeded 4 on a 5-point scale. Violence and abuse (socioeconomic and cultural factor) and flexible working arrangement (fertility policy) were weak signals, suggesting that they could increase rapidly in the future. Conclusions: The determinants-of-fertility ontology developed in this study can be used as a framework for collecting and analyzing social media data on fertility issues and detecting future signals of fertility issues. The future signals identified in this study will be useful for policy makers who are developing policy responses to low fertility. %M 34125068 %R 10.2196/25028 %U https://www.jmir.org/2021/6/e25028 %U https://doi.org/10.2196/25028 %U http://www.ncbi.nlm.nih.gov/pubmed/34125068 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 6 %P e28219 %T Extraction of Traditional Chinese Medicine Entity: Design of a Novel Span-Level Named Entity Recognition Method With Distant Supervision %A Jia,Qi %A Zhang,Dezheng %A Xu,Haifeng %A Xie,Yonghong %+ School of Computer and Communication Engineering, University of Science and Technology Beijing, 30 Xueyuan Road, Haidian District, Beijing, 100083, China, 86 010 62334547, xieyh@ustb.edu.cn %K traditional Chinese medicine %K named entity recognition %K span level %K distantly supervised %D 2021 %7 14.6.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Traditional Chinese medicine (TCM) clinical records contain the symptoms of patients, diagnoses, and subsequent treatment of doctors. These records are important resources for research and analysis of TCM diagnosis knowledge. However, most of TCM clinical records are unstructured text. Therefore, a method to automatically extract medical entities from TCM clinical records is indispensable. Objective: Training a medical entity extracting model needs a large number of annotated corpus. The cost of annotated corpus is very high and there is a lack of gold-standard data sets for supervised learning methods. Therefore, we utilized distantly supervised named entity recognition (NER) to respond to the challenge. Methods: We propose a span-level distantly supervised NER approach to extract TCM medical entity. It utilizes the pretrained language model and a simple multilayer neural network as classifier to detect and classify entity. We also designed a negative sampling strategy for the span-level model. The strategy randomly selects negative samples in every epoch and filters the possible false-negative samples periodically. It reduces the bad influence from the false-negative samples. Results: We compare our methods with other baseline methods to illustrate the effectiveness of our method on a gold-standard data set. The F1 score of our method is 77.34 and it remarkably outperforms the other baselines. Conclusions: We developed a distantly supervised NER approach to extract medical entity from TCM clinical records. We estimated our approach on a TCM clinical record data set. Our experimental results indicate that the proposed approach achieves a better performance than other baselines. %M 34125076 %R 10.2196/28219 %U https://medinform.jmir.org/2021/6/e28219 %U https://doi.org/10.2196/28219 %U http://www.ncbi.nlm.nih.gov/pubmed/34125076 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 6 %P e26598 %T Implementing Vertical Federated Learning Using Autoencoders: Practical Application, Generalizability, and Utility Study %A Cha,Dongchul %A Sung,MinDong %A Park,Yu-Rang %+ Department of Biomedical Systems Informatics, Yonsei University College of Medicine, 50-1 Yonsei-ro, Sinchon-dong, Seodaemun-gu, Seoul, 03722, Republic of Korea, 82 2 2228 2363, yurangpark@yuhs.ac %K federated learning %K vertically incomplete data %K privacy %K machine learning %K coding %K data %K performance %K model %K security %K training %K dataset %K unsupervised learning %K data sharing %K protection %D 2021 %7 9.6.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Machine learning (ML) is now widely deployed in our everyday lives. Building robust ML models requires a massive amount of data for training. Traditional ML algorithms require training data centralization, which raises privacy and data governance issues. Federated learning (FL) is an approach to overcome this issue. We focused on applying FL on vertically partitioned data, in which an individual’s record is scattered among different sites. Objective: The aim of this study was to perform FL on vertically partitioned data to achieve performance comparable to that of centralized models without exposing the raw data. Methods: We used three different datasets (Adult income, Schwannoma, and eICU datasets) and vertically divided each dataset into different pieces. Following the vertical division of data, overcomplete autoencoder-based model training was performed for each site. Following training, each site’s data were transformed into latent data, which were aggregated for training. A tabular neural network model with categorical embedding was used for training. A centrally based model was used as a baseline model, which was compared to that of FL in terms of accuracy and area under the receiver operating characteristic curve (AUROC). Results: The autoencoder-based network successfully transformed the original data into latent representations with no domain knowledge applied. These altered data were different from the original data in terms of the feature space and data distributions, indicating appropriate data security. The loss of performance was minimal when using an overcomplete autoencoder; accuracy loss was 1.2%, 8.89%, and 1.23%, and AUROC loss was 1.1%, 0%, and 1.12% in the Adult income, Schwannoma, and eICU dataset, respectively. Conclusions: We proposed an autoencoder-based ML model for vertically incomplete data. Since our model is based on unsupervised learning, no domain-specific knowledge is required in individual sites. Under the circumstances where direct data sharing is not available, our approach may be a practical solution enabling both data protection and building a robust model. %M 34106083 %R 10.2196/26598 %U https://medinform.jmir.org/2021/6/e26598 %U https://doi.org/10.2196/26598 %U http://www.ncbi.nlm.nih.gov/pubmed/34106083 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 6 %P e25968 %T A Care Knowledge Management System Based on an Ontological Model of Caring for People With Dementia: Knowledge Representation and Development Study %A Kim,Gyungha %A Jeon,Hwawoo %A Park,Sung Kee %A Choi,Yong Suk %A Lim,Yoonseob %+ The Research Center for Diagnosis, Treatment and Care System of Dementia, Korea Institute of Science and Technology, 5, Hwarang-ro 14-gil, Seongbuk-gu, Seoul, KS013, Republic of Korea, 82 10 3700 3867, yslim@kist.re.kr %K caregiver %K caregiver for person with dementia %K knowledge model %K ontology %K knowledge management %K semantic reasoning %D 2021 %7 8.6.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Caregivers of people with dementia find it extremely difficult to choose the best care method because of complex environments and the variable symptoms of dementia. To alleviate this care burden, interventions have been proposed that use computer- or web-based applications. For example, an automatic diagnosis of the condition can improve the well-being of both the person with dementia and the caregiver. Other interventions support the individual with dementia in living independently. Objective: The aim of this study was to develop an ontology-based care knowledge management system for people with dementia that will provide caregivers with a care guide suited to the environment and to the individual patient’s symptoms. This should also enable knowledge sharing among caregivers. Methods: To build the care knowledge model, we reviewed existing ontologies that contain concepts and knowledge descriptions relating to the care of those with dementia, and we considered dementia care manuals. The basic concepts of the care ontology were confirmed by experts in Korea. To infer the different care methods required for the individual dementia patient, the reasoning rules as defined in Semantic Web Rule Languages and Prolog were utilized. The accuracy of the care knowledge in the ontological model and the usability of the proposed system were evaluated by using the Pellet reasoner and OntOlogy Pitfall Scanner!, and a survey and interviews were conducted with caregivers working in care centers in Korea. Results: The care knowledge model contains six top-level concepts: care knowledge, task, assessment, person, environment, and medical knowledge. Based on this ontological model of dementia care, caregivers at a dementia care facility in Korea were able to access the care knowledge easily through a graphical user interface. The evaluation by the care experts showed that the system contained accurate care knowledge and a level of assessment comparable to normal assessment tools. Conclusions: In this study, we developed a care knowledge system that can provide caregivers with care guides suited to individuals with dementia. We anticipate that the system could reduce the workload of caregivers. %M 34100762 %R 10.2196/25968 %U https://www.jmir.org/2021/6/e25968 %U https://doi.org/10.2196/25968 %U http://www.ncbi.nlm.nih.gov/pubmed/34100762 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 5 %P e24803 %T An Attention Model With Transfer Embeddings to Classify Pneumonia-Related Bilingual Imaging Reports: Algorithm Development and Validation %A Park,Hyung %A Song,Min %A Lee,Eun Byul %A Seo,Bo Kyung %A Choi,Chang Min %+ Department of Pulmonary and Critical Care Medicine, Asan Medical Center, Olympic-ro 43-gil, Seoul, 05505, Republic of Korea, 82 2 3010 5902, ccm9607@gmail.com %K deep learning %K natural language process %K attention %K clinical data %K pneumonia %K classification %K medical imaging %K electronic health record %K machine learning %K model %D 2021 %7 17.5.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: In the analysis of electronic health records, proper labeling of outcomes is mandatory. To obtain proper information from radiologic reports, several studies were conducted to classify radiologic reports using deep learning. However, the classification of pneumonia in bilingual radiologic reports has not been conducted previously. Objective: The aim of this research was to classify radiologic reports into pneumonia or no pneumonia using a deep learning method. Methods: A data set of radiology reports for chest computed tomography and chest x-rays of surgical patients from January 2008 to January 2018 in the Asan Medical Center in Korea was retrospectively analyzed. The classification performance of our long short-term memory (LSTM)–Attention model was compared with various deep learning and machine learning methods. The area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve, sensitivity, specificity, accuracy, and F1 score for the models were compared. Results: A total of 5450 radiologic reports were included that contained at least one pneumonia-related word. In the test set (n=1090), our proposed model showed 91.01% (992/1090) accuracy (AUROCs for negative, positive, and obscure were 0.98, 0.97, and 0.90, respectively). The top 3 performances of the models were based on FastText or LSTM. The convolutional neural network–based model showed a lower accuracy 73.03% (796/1090) than the other 2 algorithms. The classification of negative results had an F1 score of 0.96, whereas the classification of positive and uncertain results showed a lower performance (positive F1 score 0.83; uncertain F1 score 0.62). In the extra-validation set, our model showed 80.0% (642/803) accuracy (AUROCs for negative, positive, and obscure were 0.92, 0.96, and 0.84, respectively). Conclusions: Our method showed excellent performance in classifying pneumonia in bilingual radiologic reports. The method could enrich the research on pneumonia by obtaining exact outcomes from electronic health data. %M 33820755 %R 10.2196/24803 %U https://medinform.jmir.org/2021/5/e24803 %U https://doi.org/10.2196/24803 %U http://www.ncbi.nlm.nih.gov/pubmed/33820755 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 5 %P e27228 %T Pathway-Driven Coordinated Telehealth System for Management of Patients With Single or Multiple Chronic Diseases in China: System Development and Retrospective Study %A Wang,Zheyu %A An,Jiye %A Lin,Hui %A Zhou,Jiaqiang %A Liu,Fang %A Chen,Juan %A Duan,Huilong %A Deng,Ning %+ Ministry of Education Key Laboratory of Biomedical Engineering, College of Biomedical Engineering and Instrument Science, Zhejiang University, 38 Zheda Rd, Zhouyiqing Bldg 512, Yuquan Campus, Hangzhou, 310027, China, 86 57122952693, zju.dengning@gmail.com %K chronic disease %K telehealth system %K integrated care %K pathway %K ontology %D 2021 %7 17.5.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Integrated care enhanced with information technology has emerged as a means to transform health services to meet the long-term care needs of patients with chronic diseases. However, the feasibility of applying integrated care to the emerging “three-manager” mode in China remains to be explored. Moreover, few studies have attempted to integrate multiple types of chronic diseases into a single system. Objective: The aim of this study was to develop a coordinated telehealth system that addresses the existing challenges of the “three-manager” mode in China while supporting the management of single or multiple chronic diseases. Methods: The system was designed based on a tailored integrated care model. The model was constructed at the individual scale, mainly focusing on specifying the involved roles and responsibilities through a universal care pathway. A custom ontology was developed to represent the knowledge contained in the model. The system consists of a service engine for data storage and decision support, as well as different forms of clients for care providers and patients. Currently, the system supports management of three single chronic diseases (hypertension, type 2 diabetes mellitus, and chronic obstructive pulmonary disease) and one type of multiple chronic conditions (hypertension with type 2 diabetes mellitus). A retrospective study was performed based on the long-term observational data extracted from the database to evaluate system usability, treatment effect, and quality of care. Results: The retrospective analysis involved 6964 patients with chronic diseases and 249 care providers who have registered in our system since its deployment in 2015. A total of 519,598 self-monitoring records have been submitted by the patients. The engine could generate different types of records regularly based on the specific care pathway. Results of the comparison tests and causal inference showed that a part of patient outcomes improved after receiving management through the system, especially the systolic blood pressure of patients with hypertension (P<.001 in all comparison tests and an approximately 5 mmHg decrease after intervention via causal inference). A regional case study showed that the work efficiency of care providers differed among individuals. Conclusions: Our system has potential to provide effective management support for single or multiple chronic conditions simultaneously. The tailored closed-loop care pathway was feasible and effective under the “three-manager” mode in China. One direction for future work is to introduce advanced artificial intelligence techniques to construct a more personalized care pathway. %M 33998999 %R 10.2196/27228 %U https://medinform.jmir.org/2021/5/e27228 %U https://doi.org/10.2196/27228 %U http://www.ncbi.nlm.nih.gov/pubmed/33998999 %0 Journal Article %@ 2368-7959 %I JMIR Publications %V 8 %N 5 %P e27113 %T Learning From Clinical Consensus Diagnosis in India to Facilitate Automatic Classification of Dementia: Machine Learning Study %A Jin,Haomiao %A Chien,Sandy %A Meijer,Erik %A Khobragade,Pranali %A Lee,Jinkook %+ Center for Economic and Social Research, University of Southern California, 635 Downey Way, VPD, Los Angeles, CA, 90089, United States, 1 626 554 3370, haomiaoj@usc.edu %K dementia %K Alzheimer disease %K machine learning %K artificial intelligence %K diagnosis %K classification %K India %K model %D 2021 %7 10.5.2021 %9 Original Paper %J JMIR Ment Health %G English %X Background: The Harmonized Diagnostic Assessment of Dementia for the Longitudinal Aging Study in India (LASI-DAD) is the first and only nationally representative study on late-life cognition and dementia in India (n=4096). LASI-DAD obtained clinical consensus diagnosis of dementia for a subsample of 2528 respondents. Objective: This study develops a machine learning model that uses data from the clinical consensus diagnosis in LASI-DAD to support the classification of dementia status. Methods: Clinicians were presented with the extensive data collected from LASI-DAD, including sociodemographic information and health history of respondents, results from the screening tests of cognitive status, and information obtained from informant interviews. Based on the Clinical Dementia Rating (CDR) and using an online platform, clinicians individually evaluated each case and then reached a consensus diagnosis. A 2-step procedure was implemented to train several candidate machine learning models, which were evaluated using a separate test set for predictive accuracy measurement, including the area under receiver operating curve (AUROC), accuracy, sensitivity, specificity, precision, F1 score, and kappa statistic. The ultimate model was selected based on overall agreement as measured by kappa. We further examined the overall accuracy and agreement with the final consensus diagnoses between the selected machine learning model and individual clinicians who participated in the clinical consensus diagnostic process. Finally, we applied the selected model to a subgroup of LASI-DAD participants for whom the clinical consensus diagnosis was not obtained to predict their dementia status. Results: Among the 2528 individuals who received clinical consensus diagnosis, 192 (6.7% after adjusting for sampling weight) were diagnosed with dementia. All candidate machine learning models achieved outstanding discriminative ability, as indicated by AUROC >.90, and had similar accuracy and specificity (both around 0.95). The support vector machine model outperformed other models with the highest sensitivity (0.81), F1 score (0.72), and kappa (.70, indicating substantial agreement) and the second highest precision (0.65). As a result, the support vector machine was selected as the ultimate model. Further examination revealed that overall accuracy and agreement were similar between the selected model and individual clinicians. Application of the prediction model on 1568 individuals without clinical consensus diagnosis classified 127 individuals as living with dementia. After applying sampling weight, we can estimate the prevalence of dementia in the population as 7.4%. Conclusions: The selected machine learning model has outstanding discriminative ability and substantial agreement with a clinical consensus diagnosis of dementia. The model can serve as a computer model of the clinical knowledge and experience encoded in the clinical consensus diagnostic process and has many potential applications, including predicting missed dementia diagnoses and serving as a clinical decision support tool or virtual rater to assist diagnosis of dementia. %M 33970122 %R 10.2196/27113 %U https://mental.jmir.org/2021/5/e27113 %U https://doi.org/10.2196/27113 %U http://www.ncbi.nlm.nih.gov/pubmed/33970122 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 5 %P e24678 %T Extracting Drug Names and Associated Attributes From Discharge Summaries: Text Mining Study %A Alfattni,Ghada %A Belousov,Maksim %A Peek,Niels %A Nenadic,Goran %+ Department of Computer Science, University of Manchester, Oxford Road, Manchester, , United Kingdom, 44 7478375084, gafattni@uqu.edu.sa %K information extraction %K electronic health records %K discharge summaries %K natural language processing %K medication prescriptions %D 2021 %7 5.5.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Drug prescriptions are often recorded in free-text clinical narratives; making this information available in a structured form is important to support many health-related tasks. Although several natural language processing (NLP) methods have been proposed to extract such information, many challenges remain. Objective: This study evaluates the feasibility of using NLP and deep learning approaches for extracting and linking drug names and associated attributes identified in clinical free-text notes and presents an extensive error analysis of different methods. This study initiated with the participation in the 2018 National NLP Clinical Challenges (n2c2) shared task on adverse drug events and medication extraction. Methods: The proposed system (DrugEx) consists of a named entity recognizer (NER) to identify drugs and associated attributes and a relation extraction (RE) method to identify the relations between them. For NER, we explored deep learning-based approaches (ie, bidirectional long-short term memory with conditional random fields [BiLSTM-CRFs]) with various embeddings (ie, word embedding, character embedding [CE], and semantic-feature embedding) to investigate how different embeddings influence the performance. A rule-based method was implemented for RE and compared with a context-aware long-short term memory (LSTM) model. The methods were trained and evaluated using the 2018 n2c2 shared task data. Results: The experiments showed that the best model (BiLSTM-CRFs with pretrained word embeddings [PWE] and CE) achieved lenient micro F-scores of 0.921 for NER, 0.927 for RE, and 0.855 for the end-to-end system. NER, which relies on the pretrained word and semantic embeddings, performed better on most individual entity types, but NER with PWE and CE had the highest classification efficiency among the proposed approaches. Extracting relations using the rule-based method achieved higher accuracy than the context-aware LSTM for most relations. Interestingly, the LSTM model performed notably better in the reason-drug relations, the most challenging relation type. Conclusions: The proposed end-to-end system achieved encouraging results and demonstrated the feasibility of using deep learning methods to extract medication information from free-text data. %M 33949962 %R 10.2196/24678 %U https://medinform.jmir.org/2021/5/e24678 %U https://doi.org/10.2196/24678 %U http://www.ncbi.nlm.nih.gov/pubmed/33949962 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 4 %P e24656 %T An Automatic Ontology-Based Approach to Support Logical Representation of Observable and Measurable Data for Healthy Lifestyle Management: Proof-of-Concept Study %A Chatterjee,Ayan %A Prinz,Andreas %A Gerdes,Martin %A Martinez,Santiago %+ Department of Information and Communication Technologies, Centre for e-Health, University of Agder, Jon Lilletuns Vei 9, Grimstad, 4879, Norway, 47 38141000, ayan.chatterjee@uia.no %K activity %K nutrition %K sensor %K questionnaire %K SSN %K ontology %K SNOMED CT %K eCoach %K personalized %K recommendation %K automated %K CDSS %K healthy lifestyle %K interoperability %K eHealth %K goal setting %K semantics %K simulation %K proposition %D 2021 %7 9.4.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Lifestyle diseases, because of adverse health behavior, are the foremost cause of death worldwide. An eCoach system may encourage individuals to lead a healthy lifestyle with early health risk prediction, personalized recommendation generation, and goal evaluation. Such an eCoach system needs to collect and transform distributed heterogenous health and wellness data into meaningful information to train an artificially intelligent health risk prediction model. However, it may produce a data compatibility dilemma. Our proposed eHealth ontology can increase interoperability between different heterogeneous networks, provide situation awareness, help in data integration, and discover inferred knowledge. This “proof-of-concept” study will help sensor, questionnaire, and interview data to be more organized for health risk prediction and personalized recommendation generation targeting obesity as a study case. Objective: The aim of this study is to develop an OWL-based ontology (UiA eHealth Ontology/UiAeHo) model to annotate personal, physiological, behavioral, and contextual data from heterogeneous sources (sensor, questionnaire, and interview), followed by structuring and standardizing of diverse descriptions to generate meaningful, practical, personalized, and contextual lifestyle recommendations based on the defined rules. Methods: We have developed a simulator to collect dummy personal, physiological, behavioral, and contextual data related to artificial participants involved in health monitoring. We have integrated the concepts of “Semantic Sensor Network Ontology” and “Systematized Nomenclature of Medicine—Clinical Terms” to develop our proposed eHealth ontology. The ontology has been created using Protégé (version 5.x). We have used the Java-based “Jena Framework” (version 3.16) for building a semantic web application that includes resource description framework (RDF) application programming interface (API), OWL API, native tuple store (tuple database), and the SPARQL (Simple Protocol and RDF Query Language) query engine. The logical and structural consistency of the proposed ontology has been evaluated with the “HermiT 1.4.3.x” ontology reasoner available in Protégé 5.x. Results: The proposed ontology has been implemented for the study case “obesity.” However, it can be extended further to other lifestyle diseases. “UiA eHealth Ontology” has been constructed using logical axioms, declaration axioms, classes, object properties, and data properties. The ontology can be visualized with “Owl Viz,” and the formal representation has been used to infer a participant’s health status using the “HermiT” reasoner. We have also developed a module for ontology verification that behaves like a rule-based decision support system to predict the probability for health risk, based on the evaluation of the results obtained from SPARQL queries. Furthermore, we discussed the potential lifestyle recommendation generation plan against adverse behavioral risks. Conclusions: This study has led to the creation of a meaningful, context-specific ontology to model massive, unintuitive, raw, unstructured observations for health and wellness data (eg, sensors, interviews, questionnaires) and to annotate them with semantic metadata to create a compact, intelligible abstraction for health risk predictions for individualized recommendation generation. %M 33835031 %R 10.2196/24656 %U https://www.jmir.org/2021/4/e24656 %U https://doi.org/10.2196/24656 %U http://www.ncbi.nlm.nih.gov/pubmed/33835031 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 3 %P e23983 %T A Framework (SOCRATex) for Hierarchical Annotation of Unstructured Electronic Health Records and Integration Into a Standardized Medical Database: Development and Usability Study %A Park,Jimyung %A You,Seng Chan %A Jeong,Eugene %A Weng,Chunhua %A Park,Dongsu %A Roh,Jin %A Lee,Dong Yun %A Cheong,Jae Youn %A Choi,Jin Wook %A Kang,Mira %A Park,Rae Woong %+ Department of Biomedical Informatics, Ajou University School of Medicine, 164, World cup-ro, Yeongtong-gu, Suwon-si, Gyeonggi-do, Suwon, 16499, Republic of Korea, 82 31 219 4471, veritas@ajou.ac.kr %K natural language processing %K search engine %K data curation %K data management %K common data model %D 2021 %7 30.3.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Although electronic health records (EHRs) have been widely used in secondary assessments, clinical documents are relatively less utilized owing to the lack of standardized clinical text frameworks across different institutions. Objective: This study aimed to develop a framework for processing unstructured clinical documents of EHRs and integration with standardized structured data. Methods: We developed a framework known as Staged Optimization of Curation, Regularization, and Annotation of clinical text (SOCRATex). SOCRATex has the following four aspects: (1) extracting clinical notes for the target population and preprocessing the data, (2) defining the annotation schema with a hierarchical structure, (3) performing document-level hierarchical annotation using the annotation schema, and (4) indexing annotations for a search engine system. To test the usability of the proposed framework, proof-of-concept studies were performed on EHRs. We defined three distinctive patient groups and extracted their clinical documents (ie, pathology reports, radiology reports, and admission notes). The documents were annotated and integrated into the Observational Medical Outcomes Partnership (OMOP)-common data model (CDM) database. The annotations were used for creating Cox proportional hazard models with different settings of clinical analyses to measure (1) all-cause mortality, (2) thyroid cancer recurrence, and (3) 30-day hospital readmission. Results: Overall, 1055 clinical documents of 953 patients were extracted and annotated using the defined annotation schemas. The generated annotations were indexed into an unstructured textual data repository. Using the annotations of pathology reports, we identified that node metastasis and lymphovascular tumor invasion were associated with all-cause mortality among colon and rectum cancer patients (both P=.02). The other analyses involving measuring thyroid cancer recurrence using radiology reports and 30-day hospital readmission using admission notes in depressive disorder patients also showed results consistent with previous findings. Conclusions: We propose a framework for hierarchical annotation of textual data and integration into a standardized OMOP-CDM medical database. The proof-of-concept studies demonstrated that our framework can effectively process and integrate diverse clinical documents with standardized structured data for clinical research. %M 33783361 %R 10.2196/23983 %U https://medinform.jmir.org/2021/3/e23983 %U https://doi.org/10.2196/23983 %U http://www.ncbi.nlm.nih.gov/pubmed/33783361 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 3 %P e23456 %T Natural Language Processing of Clinical Notes to Identify Mental Illness and Substance Use Among People Living with HIV: Retrospective Cohort Study %A Ridgway,Jessica P %A Uvin,Arno %A Schmitt,Jessica %A Oliwa,Tomasz %A Almirol,Ellen %A Devlin,Samantha %A Schneider,John %+ Department of Medicine, University of Chicago, 5841 S Maryland Ave, MC 5065, Chicago, IL, 60637, United States, 1 7737029185, jessica.ridgway@uchospitals.edu %K natural language processing %K HIV %K substance use %K mental illness %K electronic medical records %D 2021 %7 10.3.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Mental illness and substance use are prevalent among people living with HIV and often lead to poor health outcomes. Electronic medical record (EMR) data are increasingly being utilized for HIV-related clinical research and care, but mental illness and substance use are often underdocumented in structured EMR fields. Natural language processing (NLP) of unstructured text of clinical notes in the EMR may more accurately identify mental illness and substance use among people living with HIV than structured EMR fields alone. Objective: The aim of this study was to utilize NLP of clinical notes to detect mental illness and substance use among people living with HIV and to determine how often these factors are documented in structured EMR fields. Methods: We collected both structured EMR data (diagnosis codes, social history, Problem List) as well as the unstructured text of clinical HIV care notes for adults living with HIV. We developed NLP algorithms to identify words and phrases associated with mental illness and substance use in the clinical notes. The algorithms were validated based on chart review. We compared numbers of patients with documentation of mental illness or substance use identified by structured EMR fields with those identified by the NLP algorithms. Results: The NLP algorithm for detecting mental illness had a positive predictive value (PPV) of 98% and a negative predictive value (NPV) of 98%. The NLP algorithm for detecting substance use had a PPV of 92% and an NPV of 98%. The NLP algorithm for mental illness identified 54.0% (420/778) of patients as having documentation of mental illness in the text of clinical notes. Among the patients with mental illness detected by NLP, 58.6% (246/420) had documentation of mental illness in at least one structured EMR field. Sixty-three patients had documentation of mental illness in structured EMR fields that was not detected by NLP of clinical notes. The NLP algorithm for substance use detected substance use in the text of clinical notes in 18.1% (141/778) of patients. Among patients with substance use detected by NLP, 73.8% (104/141) had documentation of substance use in at least one structured EMR field. Seventy-six patients had documentation of substance use in structured EMR fields that was not detected by NLP of clinical notes. Conclusions: Among patients in an urban HIV care clinic, NLP of clinical notes identified high rates of mental illness and substance use that were often not documented in structured EMR fields. This finding has important implications for epidemiologic research and clinical care for people living with HIV. %M 33688848 %R 10.2196/23456 %U https://medinform.jmir.org/2021/3/e23456 %U https://doi.org/10.2196/23456 %U http://www.ncbi.nlm.nih.gov/pubmed/33688848 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 3 %P e22806 %T Contribution of Free-Text Comments to the Burden of Documentation: Assessment and Analysis of Vital Sign Comments in Flowsheets %A Yin,Zhijun %A Liu,Yongtai %A McCoy,Allison B %A Malin,Bradley A %A Sengstack,Patricia R %+ Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Ave., Suite 1475, Nashville, TN, 37203, United States, 1 615 936 3690, zhijun.yin@vanderbilt.edu %K electronic health system %K documentation burden %K flowsheets %K content analysis %K vital sign comments %K free text %D 2021 %7 4.3.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: Documentation burden is a common problem with modern electronic health record (EHR) systems. To reduce this burden, various recording methods (eg, voice recorders or motion sensors) have been proposed. However, these solutions are in an early prototype phase and are unlikely to transition into practice in the near future. A more pragmatic alternative is to directly modify the implementation of the existing functionalities of an EHR system. Objective: This study aims to assess the nature of free-text comments entered into EHR flowsheets that supplement quantitative vital sign values and examine opportunities to simplify functionality and reduce documentation burden. Methods: We evaluated 209,055 vital sign comments in flowsheets that were generated in the Epic EHR system at the Vanderbilt University Medical Center in 2018. We applied topic modeling, as well as the natural language processing Clinical Language Annotation, Modeling, and Processing software system, to extract generally discussed topics and detailed medical terms (expressed as probability distribution) to investigate the stories communicated in these comments. Results: Our analysis showed that 63.33% (6053/9557) of the users who entered vital signs made at least one free-text comment in vital sign flowsheet entries. The user roles that were most likely to compose comments were registered nurse, technician, and licensed nurse. The most frequently identified topics were the notification of a result to health care providers (0.347), the context of a measurement (0.307), and an inability to obtain a vital sign (0.224). There were 4187 unique medical terms that were extracted from 46,029 (0.220) comments, including many symptom-related terms such as “pain,” “upset,” “dizziness,” “coughing,” “anxiety,” “distress,” and “fever” and drug-related terms such as “tylenol,” “anesthesia,” “cannula,” “oxygen,” “motrin,” “rituxan,” and “labetalol.” Conclusions: Considering that flowsheet comments are generally not displayed or automatically pulled into any clinical notes, our findings suggest that the flowsheet comment functionality can be simplified (eg, via structured response fields instead of a text input dialog) to reduce health care provider effort. Moreover, rich and clinically important medical terms such as medications and symptoms should be explicitly recorded in clinical notes for better visibility. %M 33661128 %R 10.2196/22806 %U https://www.jmir.org/2021/3/e22806 %U https://doi.org/10.2196/22806 %U http://www.ncbi.nlm.nih.gov/pubmed/33661128 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 2 %P e21679 %T Lexicon Development for COVID-19-related Concepts Using Open-source Word Embedding Sources: An Intrinsic and Extrinsic Evaluation %A Parikh,Soham %A Davoudi,Anahita %A Yu,Shun %A Giraldo,Carolina %A Schriver,Emily %A Mowery,Danielle %+ Department of Biostatistics, Epidemiology, & Informatics, Institute for Biomedical Informatics, University of Pennsylvania, A206 Richards Hall,, 3700 Hamilton Walk, Philadelphia, PA, 19104-6021, United States, 1 215 746 6677, dlmowery@pennmedicine.upenn.edu %K natural language processing %K word embedding %K COVID-19 %K intrinsic %K open-source %K computation %K model %K prediction %K semantic %K syntactic %K pattern %D 2021 %7 22.2.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Scientists are developing new computational methods and prediction models to better clinically understand COVID-19 prevalence, treatment efficacy, and patient outcomes. These efforts could be improved by leveraging documented COVID-19–related symptoms, findings, and disorders from clinical text sources in an electronic health record. Word embeddings can identify terms related to these clinical concepts from both the biomedical and nonbiomedical domains, and are being shared with the open-source community at large. However, it’s unclear how useful openly available word embeddings are for developing lexicons for COVID-19–related concepts. Objective: Given an initial lexicon of COVID-19–related terms, this study aims to characterize the returned terms by similarity across various open-source word embeddings and determine common semantic and syntactic patterns between the COVID-19 queried terms and returned terms specific to the word embedding source. Methods: We compared seven openly available word embedding sources. Using a series of COVID-19–related terms for associated symptoms, findings, and disorders, we conducted an interannotator agreement study to determine how accurately the most similar returned terms could be classified according to semantic types by three annotators. We conducted a qualitative study of COVID-19 queried terms and their returned terms to detect informative patterns for constructing lexicons. We demonstrated the utility of applying such learned synonyms to discharge summaries by reporting the proportion of patients identified by concept among three patient cohorts: pneumonia (n=6410), acute respiratory distress syndrome (n=8647), and COVID-19 (n=2397). Results: We observed high pairwise interannotator agreement (Cohen kappa) for symptoms (0.86-0.99), findings (0.93-0.99), and disorders (0.93-0.99). Word embedding sources generated based on characters tend to return more synonyms (mean count of 7.2 synonyms) compared to token-based embedding sources (mean counts range from 2.0 to 3.4). Word embedding sources queried using a qualifier term (eg, dry cough or muscle pain) more often returned qualifiers of the similar semantic type (eg, “dry” returns consistency qualifiers like “wet” and “runny”) compared to a single term (eg, cough or pain) queries. A higher proportion of patients had documented fever (0.61-0.84), cough (0.41-0.55), shortness of breath (0.40-0.59), and hypoxia (0.51-0.56) retrieved than other clinical features. Terms for dry cough returned a higher proportion of patients with COVID-19 (0.07) than the pneumonia (0.05) and acute respiratory distress syndrome (0.03) populations. Conclusions: Word embeddings are valuable technology for learning related terms, including synonyms. When leveraging openly available word embedding sources, choices made for the construction of the word embeddings can significantly influence the words learned. %M 33544689 %R 10.2196/21679 %U https://medinform.jmir.org/2021/2/e21679 %U https://doi.org/10.2196/21679 %U http://www.ncbi.nlm.nih.gov/pubmed/33544689 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 2 %P e25530 %T Similarity-Based Unsupervised Spelling Correction Using BioWordVec: Development and Usability Study of Bacterial Culture and Antimicrobial Susceptibility Reports %A Kim,Taehyeong %A Han,Sung Won %A Kang,Minji %A Lee,Se Ha %A Kim,Jong-Ho %A Joo,Hyung Joon %A Sohn,Jang Wook %+ Department of Cardiology, Cardiovascular Center, Korea University College of Medicine, 73, Inchon-ro, Seongbuk-gu, 02841, Republic of Korea, 82 2 920 6411, drjoohj@gmail.com %K spelling correction %K natural language processing %K bacteria %K electronic health record %D 2021 %7 22.2.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Existing bacterial culture test results for infectious diseases are written in unrefined text, resulting in many problems, including typographical errors and stop words. Effective spelling correction processes are needed to ensure the accuracy and reliability of data for the study of infectious diseases, including medical terminology extraction. If a dictionary is established, spelling algorithms using edit distance are efficient. However, in the absence of a dictionary, traditional spelling correction algorithms that utilize only edit distances have limitations. Objective: In this research, we proposed a similarity-based spelling correction algorithm using pretrained word embedding with the BioWordVec technique. This method uses a character-level N-grams–based distributed representation through unsupervised learning rather than the existing rule-based method. In other words, we propose a framework that detects and corrects typographical errors when a dictionary is not in place. Methods: For detected typographical errors not mapped to Systematized Nomenclature of Medicine (SNOMED) clinical terms, a correction candidate group with high similarity considering the edit distance was generated using pretrained word embedding from the clinical database. From the embedding matrix in which the vocabulary is arranged in descending order according to frequency, a grid search was used to search for candidate groups of similar words. Thereafter, the correction candidate words were ranked in consideration of the frequency of the words, and the typographical errors were finally corrected according to the ranking. Results: Bacterial identification words were extracted from 27,544 bacterial culture and antimicrobial susceptibility reports, and 16 types of spelling errors and 914 misspelled words were found. The similarity-based spelling correction algorithm using BioWordVec proposed in this research corrected 12 types of typographical errors and showed very high performance in correcting 97.48% (based on F1 score) of all spelling errors. Conclusions: This tool corrected spelling errors effectively in the absence of a dictionary based on bacterial identification words in bacterial culture and antimicrobial susceptibility reports. This method will help build a high-quality refined database of vast text data for electronic health records. %M 33616536 %R 10.2196/25530 %U https://medinform.jmir.org/2021/2/e25530 %U https://doi.org/10.2196/25530 %U http://www.ncbi.nlm.nih.gov/pubmed/33616536 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 1 %P e25113 %T Disease Concept-Embedding Based on the Self-Supervised Method for Medical Information Extraction from Electronic Health Records and Disease Retrieval: Algorithm Development and Validation Study %A Chen,Yen-Pin %A Lo,Yuan-Hsun %A Lai,Feipei %A Huang,Chien-Hua %+ Department of Emergency Medicine, National Taiwan University Hospital, No 7, Chung Shan S Rd (Zhongshan S Rd), Zhongzheng District, Taipei City, Taiwan, 886 2 2312 3456, chhuang5940@ntu.edu.tw %K electronic health record %K EHR %K disease embedding %K disease retrieval %K emergency department %K concept %K extraction %K deep learning %K machine learning %K natural language processing %K NLP %D 2021 %7 27.1.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: The electronic health record (EHR) contains a wealth of medical information. An organized EHR can greatly help doctors treat patients. In some cases, only limited patient information is collected to help doctors make treatment decisions. Because EHRs can serve as a reference for this limited information, doctors’ treatment capabilities can be enhanced. Natural language processing and deep learning methods can help organize and translate EHR information into medical knowledge and experience. Objective: In this study, we aimed to create a model to extract concept embeddings from EHRs for disease pattern retrieval and further classification tasks. Methods: We collected 1,040,989 emergency department visits from the National Taiwan University Hospital Integrated Medical Database and 305,897 samples from the National Hospital and Ambulatory Medical Care Survey Emergency Department data. After data cleansing and preprocessing, the data sets were divided into training, validation, and test sets. We proposed a Transformer-based model to embed EHRs and used Bidirectional Encoder Representations from Transformers (BERT) to extract features from free text and concatenate features with structural data as input to our proposed model. Then, Deep InfoMax (DIM) and Simple Contrastive Learning of Visual Representations (SimCLR) were used for the unsupervised embedding of the disease concept. The pretrained disease concept-embedding model, named EDisease, was further finetuned to adapt to the critical care outcome prediction task. We evaluated the performance of embedding using t-distributed stochastic neighbor embedding (t-SNE) to perform dimension reduction for visualization. The performance of the finetuned predictive model was evaluated against published models using the area under the receiver operating characteristic (AUROC). Results: The performance of our model on the outcome prediction had the highest AUROC of 0.876. In the ablation study, the use of a smaller data set or fewer unsupervised methods for pretraining deteriorated the prediction performance. The AUROCs were 0.857, 0.870, and 0.868 for the model without pretraining, the model pretrained by only SimCLR, and the model pretrained by only DIM, respectively. On the smaller finetuning set, the AUROC was 0.815 for the proposed model. Conclusions: Through contrastive learning methods, disease concepts can be embedded meaningfully. Moreover, these methods can be used for disease retrieval tasks to enhance clinical practice capabilities. The disease concept model is also suitable as a pretrained model for subsequent prediction tasks. %M 33502324 %R 10.2196/25113 %U http://www.jmir.org/2021/1/e25113/ %U https://doi.org/10.2196/25113 %U http://www.ncbi.nlm.nih.gov/pubmed/33502324 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 1 %P e24594 %T Use of the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) for Processing Free Text in Health Care: Systematic Scoping Review %A Gaudet-Blavignac,Christophe %A Foufi,Vasiliki %A Bjelogrlic,Mina %A Lovis,Christian %+ Division of Medical Information Sciences, Geneva University Hospitals, Rue Gabrielle-Perret-Gentil 4, Geneva, 1205, Switzerland, 41 22 372 62 01, christophe.gaudet-blavignac@hcuge.ch %K SNOMED CT %K natural language processing %K scoping review %K terminology %D 2021 %7 26.1.2021 %9 Review %J J Med Internet Res %G English %X Background: Interoperability and secondary use of data is a challenge in health care. Specifically, the reuse of clinical free text remains an unresolved problem. The Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) has become the universal language of health care and presents characteristics of a natural language. Its use to represent clinical free text could constitute a solution to improve interoperability. Objective: Although the use of SNOMED and SNOMED CT has already been reviewed, its specific use in processing and representing unstructured data such as clinical free text has not. This review aims to better understand SNOMED CT's use for representing free text in medicine. Methods: A scoping review was performed on the topic by searching MEDLINE, Embase, and Web of Science for publications featuring free-text processing and SNOMED CT. A recursive reference review was conducted to broaden the scope of research. The review covered the type of processed data, the targeted language, the goal of the terminology binding, the method used and, when appropriate, the specific software used. Results: In total, 76 publications were selected for an extensive study. The language targeted by publications was 91% (n=69) English. The most frequent types of documents for which the terminology was used are complementary exam reports (n=18, 24%) and narrative notes (n=16, 21%). Mapping to SNOMED CT was the final goal of the research in 21% (n=16) of publications and a part of the final goal in 33% (n=25). The main objectives of mapping are information extraction (n=44, 39%), feature in a classification task (n=26, 23%), and data normalization (n=23, 20%). The method used was rule-based in 70% (n=53) of publications, hybrid in 11% (n=8), and machine learning in 5% (n=4). In total, 12 different software packages were used to map text to SNOMED CT concepts, the most frequent being Medtex, Mayo Clinic Vocabulary Server, and Medical Text Extraction Reasoning and Mapping System. Full terminology was used in 64% (n=49) of publications, whereas only a subset was used in 30% (n=23) of publications. Postcoordination was proposed in 17% (n=13) of publications, and only 5% (n=4) of publications specifically mentioned the use of the compositional grammar. Conclusions: SNOMED CT has been largely used to represent free-text data, most frequently with rule-based approaches, in English. However, currently, there is no easy solution for mapping free text to this terminology and to perform automatic postcoordination. Most solutions conceive SNOMED CT as a simple terminology rather than as a compositional bag of ontologies. Since 2012, the number of publications on this subject per year has decreased. However, the need for formal semantic representation of free text in health care is high, and automatic encoding into a compositional ontology could be a solution. %M 33496673 %R 10.2196/24594 %U http://www.jmir.org/2021/1/e24594/ %U https://doi.org/10.2196/24594 %U http://www.ncbi.nlm.nih.gov/pubmed/33496673 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 9 %N 1 %P e23104 %T Clinical Term Normalization Using Learned Edit Patterns and Subconcept Matching: System Development and Evaluation %A Kate,Rohit J %+ Department of Computer Science, University of Wisconsin-Milwaukee, 3200 N Cramer St, Milwaukee, WI, 53211, United States, 1 4142294264, katerj@uwm.edu %K clinical term normalization %K edit distance %K machine learning %K natural language processing %D 2021 %7 14.1.2021 %9 Original Paper %J JMIR Med Inform %G English %X Background: Clinical terms mentioned in clinical text are often not in their standardized forms as listed in clinical terminologies because of linguistic and stylistic variations. However, many automated downstream applications require clinical terms mapped to their corresponding concepts in clinical terminologies, thus necessitating the task of clinical term normalization. Objective: In this paper, a system for clinical term normalization is presented that utilizes edit patterns to convert clinical terms into their normalized forms. Methods: The edit patterns are automatically learned from the Unified Medical Language System (UMLS) Metathesaurus as well as from the given training data. The edit patterns are generalized sequences of edits that are derived from edit distance computations. The edit patterns are both character based as well as word based and are learned separately for different semantic types. In addition to these edit patterns, the system also normalizes clinical terms through the subconcepts mentioned within them. Results: The system was evaluated as part of the 2019 n2c2 Track 3 shared task of clinical term normalization. It obtained 80.79% accuracy on the standard test data. This paper includes ablation studies to evaluate the contributions of different components of the system. A challenging part of the task was disambiguation when a clinical term could be normalized to multiple concepts. Conclusions: The learned edit patterns led the system to perform well on the normalization task. Given that the system is based on patterns, it is human interpretable and is also capable of giving insights about common variations of clinical terms mentioned in clinical text that are different from their standardized forms. %M 33443483 %R 10.2196/23104 %U https://medinform.jmir.org/2021/1/e23104 %U https://doi.org/10.2196/23104 %U http://www.ncbi.nlm.nih.gov/pubmed/33443483 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 12 %P e22649 %T Detecting Miscoded Diabetes Diagnosis Codes in Electronic Health Records for Quality Improvement: Temporal Deep Learning Approach %A Rashidian,Sina %A Abell-Hart,Kayley %A Hajagos,Janos %A Moffitt,Richard %A Lingam,Veena %A Garcia,Victor %A Tsai,Chao-Wei %A Wang,Fusheng %A Dong,Xinyu %A Sun,Siao %A Deng,Jianyuan %A Gupta,Rajarsi %A Miller,Joshua %A Saltz,Joel %A Saltz,Mary %+ Department of Computer Science, Stony Brook University, 2212 Computer Science, Stony Brook, NY, 11794, United States, 1 631 632 8470, srashidian@cs.stonybrook.edu %K electronic health records %K diabetes %K deep learning %D 2020 %7 17.12.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Diabetes affects more than 30 million patients across the United States. With such a large disease burden, even a small error in classification can be significant. Currently billing codes, assigned at the time of a medical encounter, are the “gold standard” reflecting the actual diseases present in an individual, and thus in aggregate reflect disease prevalence in the population. These codes are generated by highly trained coders and by health care providers but are not always accurate. Objective: This work provides a scalable deep learning methodology to more accurately classify individuals with diabetes across multiple health care systems. Methods: We leveraged a long short-term memory-dense neural network (LSTM-DNN) model to identify patients with or without diabetes using data from 5 acute care facilities with 187,187 patients and 275,407 encounters, incorporating data elements including laboratory test results, diagnostic/procedure codes, medications, demographic data, and admission information. Furthermore, a blinded physician panel reviewed discordant cases, providing an estimate of the total impact on the population. Results: When predicting the documented diagnosis of diabetes, our model achieved an 84% F1 score, 96% area under the curve–receiver operating characteristic curve, and 91% average precision on a heterogeneous data set from 5 distinct health facilities. However, in 81% of cases where the model disagreed with the documented phenotype, a blinded physician panel agreed with the model. Taken together, this suggests that 4.3% of our studied population have either missing or improper diabetes diagnosis. Conclusions: This study demonstrates that deep learning methods can improve clinical phenotyping even when patient data are noisy, sparse, and heterogeneous. %M 33331828 %R 10.2196/22649 %U http://medinform.jmir.org/2020/12/e22649/ %U https://doi.org/10.2196/22649 %U http://www.ncbi.nlm.nih.gov/pubmed/33331828 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 22 %N 12 %P e18526 %T Transformation of Pathology Reports Into the Common Data Model With Oncology Module: Use Case for Colon Cancer %A Ryu,Borim %A Yoon,Eunsil %A Kim,Seok %A Lee,Sejoon %A Baek,Hyunyoung %A Yi,Soyoung %A Na,Hee Young %A Kim,Ji-Won %A Baek,Rong-Min %A Hwang,Hee %A Yoo,Sooyoung %+ Office of eHealth Research and Business, Seoul National University Bundang Hospital, 82 173rd Street, Gumi-ro, Bundang-gu, Seongnam, 13620, Republic of Korea, 82 317878980, yoosoo0@gmail.com %K common data model %K natural language processing %K oncology module %K colon cancer %K electronic health record %K oncology %K pathology %K clinical data %D 2020 %7 9.12.2020 %9 Original Paper %J J Med Internet Res %G English %X Background: Common data models (CDMs) help standardize electronic health record data and facilitate outcome analysis for observational and longitudinal research. An analysis of pathology reports is required to establish fundamental information infrastructure for data-driven colon cancer research. The Observational Medical Outcomes Partnership (OMOP) CDM is used in distributed research networks for clinical data; however, it requires conversion of free text–based pathology reports into the CDM’s format. There are few use cases of representing cancer data in CDM. Objective: In this study, we aimed to construct a CDM database of colon cancer–related pathology with natural language processing (NLP) for a research platform that can utilize both clinical and omics data. The essential text entities from the pathology reports are extracted, standardized, and converted to the OMOP CDM format in order to utilize the pathology data in cancer research. Methods: We extracted clinical text entities, mapped them to the standard concepts in the Observational Health Data Sciences and Informatics vocabularies, and built databases and defined relations for the CDM tables. Major clinical entities were extracted through NLP on pathology reports of surgical specimens, immunohistochemical studies, and molecular studies of colon cancer patients at a tertiary general hospital in South Korea. Items were extracted from each report using regular expressions in Python. Unstructured data, such as text that does not have a pattern, were handled with expert advice by adding regular expression rules. Our own dictionary was used for normalization and standardization to deal with biomarker and gene names and other ungrammatical expressions. The extracted clinical and genetic information was mapped to the Logical Observation Identifiers Names and Codes databases and the Systematized Nomenclature of Medicine (SNOMED) standard terminologies recommended by the OMOP CDM. The database-table relationships were newly defined through SNOMED standard terminology concepts. The standardized data were inserted into the CDM tables. For evaluation, 100 reports were randomly selected and independently annotated by a medical informatics expert and a nurse. Results: We examined and standardized 1848 immunohistochemical study reports, 3890 molecular study reports, and 12,352 pathology reports of surgical specimens (from 2017 to 2018). The constructed and updated database contained the following extracted colorectal entities: (1) NOTE_NLP, (2) MEASUREMENT, (3) CONDITION_OCCURRENCE, (4) SPECIMEN, and (5) FACT_RELATIONSHIP of specimen with condition and measurement. Conclusions: This study aimed to prepare CDM data for a research platform to take advantage of all omics clinical and patient data at Seoul National University Bundang Hospital for colon cancer pathology. A more sophisticated preparation of the pathology data is needed for further research on cancer genomics, and various types of text narratives are the next target for additional research on the use of data in the CDM. %M 33295294 %R 10.2196/18526 %U https://www.jmir.org/2020/12/e18526 %U https://doi.org/10.2196/18526 %U http://www.ncbi.nlm.nih.gov/pubmed/33295294 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 22 %N 12 %P e18767 %T Using Social Media Data to Understand Consumers' Information Needs and Emotions Regarding Cancer: Ontology-Based Data Analysis Study %A Lee,Jooyun %A Park,Hyeoun-Ae %A Park,Seul Ki %A Song,Tae-Min %+ College of Nursing, Seoul National University, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Republic of Korea, 82 2 740 8827, hapark@snu.ac.kr %K social media %K ontology %K cancer %K health information needs %K cancer information %K emotion %D 2020 %7 7.12.2020 %9 Original Paper %J J Med Internet Res %G English %X Background: Analysis of posts on social media is effective in investigating health information needs for disease management and identifying people’s emotional status related to disease. An ontology is needed for semantic analysis of social media data. Objective: This study was performed to develop a cancer ontology with terminology containing consumer terms and to analyze social media data to identify health information needs and emotions related to cancer. Methods: A cancer ontology was developed using social media data, collected with a crawler, from online communities and blogs between January 1, 2014 and June 30, 2017 in South Korea. The relative frequencies of posts containing ontology concepts were counted and compared by cancer type. Results: The ontology had 9 superclasses, 213 class concepts, and 4061 synonyms. Ontology-driven natural language processing was performed on the text from 754,744 cancer-related posts. Colon, breast, stomach, cervical, lung, liver, pancreatic, and prostate cancer; brain tumors; and leukemia appeared most in these posts. At the superclass level, risk factor was the most frequent, followed by emotions, symptoms, treatments, and dealing with cancer. Conclusions: Information needs and emotions differed according to cancer type. The observations of this study could be used to provide tailored information to consumers according to cancer type and care process. Attention should be paid to provision of cancer-related information to not only patients but also their families and the general public seeking information on cancer. %M 33284127 %R 10.2196/18767 %U http://www.jmir.org/2020/12/e18767/ %U https://doi.org/10.2196/18767 %U http://www.ncbi.nlm.nih.gov/pubmed/33284127 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 11 %P e22333 %T Automatic Structuring of Ontology Terms Based on Lexical Granularity and Machine Learning: Algorithm Development and Validation %A Luo,Lingyun %A Feng,Jingtao %A Yu,Huijun %A Wang,Jiaolong %+ School of Computer Science, University of South China, 28 West Changsheng Rd, Hengyang, 421001, China, 86 7348282473, luoly@usc.edu.cn %K ontology %K automatic structuring %K Foundational Model of Anatomy %K lexical granularity %K machine learning %D 2020 %7 25.11.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: As the manual creation and maintenance of biomedical ontologies are labor-intensive, automatic aids are desirable in the lifecycle of ontology development. Objective: Provided with a set of concept names in the Foundational Model of Anatomy (FMA), we propose an innovative method for automatically generating the taxonomy and the partonomy structures among them, respectively. Methods: Our approach comprises 2 main tasks: The first task is predicting the direct relation between 2 given concept names by utilizing word embedding methods and training 2 machine learning models, Convolutional Neural Networks (CNN) and Bidirectional Long Short-term Memory Networks (Bi-LSTM). The second task is the introduction of an original granularity-based method to identify the semantic structures among a group of given concept names by leveraging these trained models. Results: Results show that both CNN and Bi-LSTM perform well on the first task, with F1 measures above 0.91. For the second task, our approach achieves an average F1 measure of 0.79 on 100 case studies in the FMA using Bi-LSTM, which outperforms the primitive pairwise-based method. Conclusions: We have investigated an automatic way of predicting a hierarchical relationship between 2 concept names; based on this, we have further invented a methodology to structure a group of concept names automatically. This study is an initial investigation that will shed light on further work on the automatic creation and enrichment of biomedical ontologies. %M 33127601 %R 10.2196/22333 %U http://medinform.jmir.org/2020/11/e22333/ %U https://doi.org/10.2196/22333 %U http://www.ncbi.nlm.nih.gov/pubmed/33127601 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 11 %P e17964 %T Visualization Environment for Federated Knowledge Graphs: Development of an Interactive Biomedical Query Language and Web Application Interface %A Cox,Steven %A Ahalt,Stanley C %A Balhoff,James %A Bizon,Chris %A Fecho,Karamarie %A Kebede,Yaphet %A Morton,Kenneth %A Tropsha,Alexander %A Wang,Patrick %A Xu,Hao %+ Renaissance Computing Institute, University of North Carolina at Chapel Hill, 100 Europa Drive, Suite 540, Chapel Hill, NC, 27517, United States, 1 (919) 445 9640, scox@renci.org %K knowledge graphs %K clinical data %K biomedical data %K federation %K ontologies %K semantic harmonization %K visualization %K application programming interface %K translational science %K clinical practice %D 2020 %7 23.11.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Efforts are underway to semantically integrate large biomedical knowledge graphs using common upper-level ontologies to federate graph-oriented application programming interfaces (APIs) to the data. However, federation poses several challenges, including query routing to appropriate knowledge sources, generation and evaluation of answer subsets, semantic merger of those answer subsets, and visualization and exploration of results. Objective: We aimed to develop an interactive environment for query, visualization, and deep exploration of federated knowledge graphs. Methods: We developed a biomedical query language and web application interphase—termed as Translator Query Language (TranQL)—to query semantically federated knowledge graphs and explore query results. TranQL uses the Biolink data model as an upper-level biomedical ontology and an API standard that has been adopted by the Biomedical Data Translator Consortium to specify a protocol for expressing a query as a graph of Biolink data elements compiled from statements in the TranQL query language. Queries are mapped to federated knowledge sources, and answers are merged into a knowledge graph, with mappings between the knowledge graph and specific elements of the query. The TranQL interactive web application includes a user interface to support user exploration of the federated knowledge graph. Results: We developed 2 real-world use cases to validate TranQL and address biomedical questions of relevance to translational science. The use cases posed questions that traversed 2 federated Translator API endpoints: Integrated Clinical and Environmental Exposures Service (ICEES) and Reasoning Over Biomedical Objects linked in Knowledge Oriented Pathways (ROBOKOP). ICEES provides open access to observational clinical and environmental data, and ROBOKOP provides access to linked biomedical entities, such as “gene,” “chemical substance,” and “disease,” that are derived largely from curated public data sources. We successfully posed queries to TranQL that traversed these endpoints and retrieved answers that we visualized and evaluated. Conclusions: TranQL can be used to ask questions of relevance to translational science, rapidly obtain answers that require assertions from a federation of knowledge sources, and provide valuable insights for translational research and clinical practice. %M 33226347 %R 10.2196/17964 %U http://medinform.jmir.org/2020/11/e17964/ %U https://doi.org/10.2196/17964 %U http://www.ncbi.nlm.nih.gov/pubmed/33226347 %0 Journal Article %@ 2369-2960 %I JMIR Publications %V 6 %N 4 %P e21434 %T COVID-19 Surveillance in a Primary Care Sentinel Network: In-Pandemic Development of an Application Ontology %A de Lusignan,Simon %A Liyanage,Harshana %A McGagh,Dylan %A Jani,Bhautesh Dinesh %A Bauwens,Jorgen %A Byford,Rachel %A Evans,Dai %A Fahey,Tom %A Greenhalgh,Trisha %A Jones,Nicholas %A Mair,Frances S %A Okusi,Cecilia %A Parimalanathan,Vaishnavi %A Pell,Jill P %A Sherlock,Julian %A Tamburis,Oscar %A Tripathy,Manasa %A Ferreira,Filipa %A Williams,John %A Hobbs,F D Richard %+ Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Primary Care Building, Radcliffe Observatory Quarter, Woodstock Rd, Oxford, OX2 6GG, United Kingdom, 44 1865617283, simon.delusignan@phc.ox.ac.uk %K COVID-19 %K medical informatics %K sentinel surveillance %D 2020 %7 17.11.2020 %9 Original Paper %J JMIR Public Health Surveill %G English %X Background: Creating an ontology for COVID-19 surveillance should help ensure transparency and consistency. Ontologies formalize conceptualizations at either the domain or application level. Application ontologies cross domains and are specified through testable use cases. Our use case was an extension of the role of the Oxford Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC) to monitor the current pandemic and become an in-pandemic research platform. Objective: This study aimed to develop an application ontology for COVID-19 that can be deployed across the various use-case domains of the RCGP RSC research and surveillance activities. Methods: We described our domain-specific use case. The actor was the RCGP RSC sentinel network, the system was the course of the COVID-19 pandemic, and the outcomes were the spread and effect of mitigation measures. We used our established 3-step method to develop the ontology, separating ontological concept development from code mapping and data extract validation. We developed a coding system–independent COVID-19 case identification algorithm. As there were no gold-standard pandemic surveillance ontologies, we conducted a rapid Delphi consensus exercise through the International Medical Informatics Association Primary Health Care Informatics working group and extended networks. Results: Our use-case domains included primary care, public health, virology, clinical research, and clinical informatics. Our ontology supported (1) case identification, microbiological sampling, and health outcomes at an individual practice and at the national level; (2) feedback through a dashboard; (3) a national observatory; (4) regular updates for Public Health England; and (5) transformation of a sentinel network into a trial platform. We have identified a total of 19,115 people with a definite COVID-19 status, 5226 probable cases, and 74,293 people with possible COVID-19, within the RCGP RSC network (N=5,370,225). Conclusions: The underpinning structure of our ontological approach has coped with multiple clinical coding challenges. At a time when there is uncertainty about international comparisons, clarity about the basis on which case definitions and outcomes are made from routine data is essential. %M 33112762 %R 10.2196/21434 %U http://publichealth.jmir.org/2020/4/e21434/ %U https://doi.org/10.2196/21434 %U http://www.ncbi.nlm.nih.gov/pubmed/33112762 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 11 %P e18659 %T Characterizing Chronic Pain Episodes in Clinical Text at Two Health Care Systems: Comprehensive Annotation and Corpus Analysis %A Carlson,Luke A %A Jeffery,Molly M %A Fu,Sunyang %A He,Huan %A McCoy,Rozalina G %A Wang,Yanshan %A Hooten,William Michael %A St Sauver,Jennifer %A Liu,Hongfang %A Fan,Jungwei %+ Division of Digital Health Sciences, Department of Health Sciences Research, Mayo Clinic, 200 First Street SW, Rochester, MN, 55905, United States, 1 5075381191, fan.jung-wei@mayo.edu %K chronic pain %K guideline development %K knowledge representation %K corpus annotation %K content analysis %D 2020 %7 16.11.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Chronic pain affects more than 20% of adults in the United States and is associated with substantial physical, mental, and social burden. Clinical text contains rich information about chronic pain, but no systematic appraisal has been performed to assess the electronic health record (EHR) narratives for these patients. A formal content analysis of the unstructured EHR data can inform clinical practice and research in chronic pain. Objective: We characterized individual episodes of chronic pain by annotating and analyzing EHR notes for a stratified cohort of adults with known chronic pain. Methods: We used the Rochester Epidemiology Project infrastructure to screen all residents of Olmsted County, Minnesota, for evidence of chronic pain, between January 1, 2005, and September 30, 2015. Diagnosis codes were used to assemble a cohort of 6586 chronic pain patients; people with cancer were excluded. The records of an age- and sex-stratified random sample of 62 patients from the cohort were annotated using an iteratively developed guideline. The annotated concepts included date, location, severity, causes, effects on quality of life, diagnostic procedures, medications, and other treatment modalities. Results: A total of 94 chronic pain episodes from 62 distinct patients were identified by reviewing 3272 clinical notes. Documentation was written by clinicians across a wide spectrum of specialties. Most patients (40/62, 65%) had 1 pain episode during the study period. Interannotator agreement ranged from 0.78 to 1.00 across the annotated concepts. Some pain-related concepts (eg, body location) had 100% (94/94) coverage among all the episodes, while others had moderate coverage (eg, effects on quality of life) (55/94, 59%). Back pain and leg pain were the most common types of chronic pain in the annotated cohort. Musculoskeletal issues like arthritis were annotated as the most common causes. Opioids were the most commonly captured medication, while physical and occupational therapies were the most common nonpharmacological treatments. Conclusions: We systematically annotated chronic pain episodes in clinical text. The rich content analysis results revealed complexity of the chronic pain episodes and of their management, as well as the challenges in extracting pertinent information, even for humans. Despite the pilot study nature of the work, the annotation guideline and corpus should be able to serve as informative references for other institutions with shared interest in chronic pain research using EHRs. %M 33108311 %R 10.2196/18659 %U http://medinform.jmir.org/2020/11/e18659/ %U https://doi.org/10.2196/18659 %U http://www.ncbi.nlm.nih.gov/pubmed/33108311 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 22 %N 11 %P e20050 %T An Environmental Scan of Sex and Gender in Electronic Health Records: Analysis of Public Information Sources %A Lau,Francis %A Antonio,Marcy %A Davison,Kelly %A Queen,Roz %A Bryski,Katie %+ School of Health Information Science, University of Victoria, P.O. Box 1700 STN CSC, University of Victoria, Victoria, BC, V8W2Y2, Canada, 1 2504725131, fylau@uvic.ca %K sex %K gender %K electronic health records %K standards %K transgender persons %D 2020 %7 11.11.2020 %9 Original Paper %J J Med Internet Res %G English %X Background: Historically, the terms sex and gender have been used interchangeably as a binary attribute to describe a person as male or female, even though there is growing recognition that sex and gender are distinct concepts. The lack of sex and gender delineation in electronic health records (EHRs) may be perpetuating the inequities experienced by the transgender and gender nonbinary (TGNB) populations. Objective: This study aims to conduct an environmental scan to understand how sex and gender are defined and implemented in existing Canadian EHRs and current international health information standards. Methods: We examined public information sources on sex and gender definitions in existing Canadian EHRs and international standards communities. Definitions refer to data element names, code systems, and value sets in the descriptions of EHRs and standards. The study was built on an earlier environment scan by Canada Health Infoway, supplemented with sex and gender definitions from international standards communities. For the analysis, we examined the definitions for clarity, consistency, and accuracy. We also received feedback from a virtual community interested in sex-gender EHR issues. Results: The information sources consisted of public website descriptions of 52 databases and 55 data standards from 12 Canadian entities and 10 standards communities. There are variations in the definition and implementation of sex and gender in Canadian EHRs and international health information standards. There is a lack of clarity in some sex and gender concepts. There is inconsistency in the data element names, code systems, and value sets used to represent sex and gender concepts across EHRs. The appropriateness and adequacy of some value options are questioned as our societal understanding of sexual health evolves. Outdated value options raise concerns about current EHRs supporting the provision of culturally competent, safe, and affirmative health care. The limited options also perpetuate the inequities faced by the TGNB populations. The expanded sex and gender definitions from leading Canadian organizations and international standards communities have brought challenges in how to migrate these definitions into existing EHRs. We proposed 6 high-level actions, which are to articulate the need for this work, reach consensus on sex and gender concepts, reach consensus on expanded sex and gender definitions in EHRs, develop a coordinated action plan, embrace EHR change from socio-organizational and technical aspects to ensure success, and demonstrate the benefits in tangible terms. Conclusions: There are variations in sex and gender concepts across Canadian EHRs and the health information standards that support them. Although there are efforts to modernize sex and gender concept definitions, we need decisive and coordinated actions to ensure clarity, consistency, and competency in the definition and implementation of sex and gender concepts in EHRs. This work has implications for addressing the inequities of TGNB populations in Canada. %M 33174858 %R 10.2196/20050 %U http://www.jmir.org/2020/11/e20050/ %U https://doi.org/10.2196/20050 %U http://www.ncbi.nlm.nih.gov/pubmed/33174858 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 10 %P e20291 %T Building a Pharmacogenomics Knowledge Model Toward Precision Medicine: Case Study in Melanoma %A Kang,Hongyu %A Li,Jiao %A Wu,Meng %A Shen,Liu %A Hou,Li %+ Institute of Medical Information &Library, Chinese Academy of Medical Sciences/Peking Union Medical College, 3 Yabao Road, Chaoyang District, Beijing , China, 86 18910120178, hou.li@imicams.ac.cn %K pharmacogenomics %K knowledge model %K BERT–CRF model %K named entity recognition %K melanoma %D 2020 %7 21.10.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Many drugs do not work the same way for everyone owing to distinctions in their genes. Pharmacogenomics (PGx) aims to understand how genetic variants influence drug efficacy and toxicity. It is often considered one of the most actionable areas of the personalized medicine paradigm. However, little prior work has included in-depth explorations and descriptions of drug usage, dosage adjustment, and so on. Objective: We present a pharmacogenomics knowledge model to discover the hidden relationships between PGx entities such as drugs, genes, and diseases, especially details in precise medication. Methods: PGx open data such as DrugBank and RxNorm were integrated in this study, as well as drug labels published by the US Food and Drug Administration. We annotated 190 drug labels manually for entities and relationships. Based on the annotation results, we trained 3 different natural language processing models to complete entity recognition. Finally, the pharmacogenomics knowledge model was described in detail. Results: In entity recognition tasks, the Bidirectional Encoder Representations from Transformers–conditional random field model achieved better performance with micro-F1 score of 85.12%. The pharmacogenomics knowledge model in our study included 5 semantic types: drug, gene, disease, precise medication (population, daily dose, dose form, frequency, etc), and adverse reaction. Meanwhile, 26 semantic relationships were defined in detail. Taking melanoma caused by a BRAF gene mutation into consideration, the pharmacogenomics knowledge model covered 7 related drugs and 4846 triples were established in this case. All the corpora, relationship definitions, and triples were made publically available. Conclusions: We highlighted the pharmacogenomics knowledge model as a scalable framework for clinicians and clinical pharmacists to adjust drug dosage according to patient-specific genetic variation, and for pharmaceutical researchers to develop new drugs. In the future, a series of other antitumor drugs and automatic relation extractions will be taken into consideration to further enhance our framework with more PGx linked data. %M 33084582 %R 10.2196/20291 %U http://medinform.jmir.org/2020/10/e20291/ %U https://doi.org/10.2196/20291 %U http://www.ncbi.nlm.nih.gov/pubmed/33084582 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 4 %N 10 %P e16094 %T Coding Systems for Clinical Decision Support: Theoretical and Real-World Comparative Analysis %A Delvaux,Nicolas %A Vaes,Bert %A Aertgeerts,Bert %A Van de Velde,Stijn %A Vander Stichele,Robert %A Nyberg,Peter %A Vermandere,Mieke %+ Department of Public Health and Primary Care, Katholieke Universiteit Leuven, Kapucijnenvoer 33 Blok J - box 7001, Leuven, B-3000, Belgium, 32 16379885, nicolas.delvaux@kuleuven.be %K clinical decision support systems %K clinical coding %K medical informatics %K electronic health records %D 2020 %7 21.10.2020 %9 Original Paper %J JMIR Form Res %G English %X Background: Effective clinical decision support systems require accurate translation of practice recommendations into machine-readable artifacts; developing code sets that represent clinical concepts are an important step in this process. Many clinical coding systems are currently used in electronic health records, and it is unclear whether all of these systems are capable of efficiently representing the clinical concepts required in executing clinical decision support systems. Objective: The aim of this study was to evaluate which clinical coding systems are capable of efficiently representing clinical concepts that are necessary for translating artifacts into executable code for clinical decision support systems. Methods: Two methods were used to evaluate a set of clinical coding systems. In a theoretical approach, we extracted all the clinical concepts from 3 preventive care recommendations and constructed a series of code sets containing codes from a single clinical coding system. In a practical approach using data from a real-world setting, we studied the content of 1890 code sets used in an internationally available clinical decision support system and compared the usage of various clinical coding systems. Results: SNOMED CT and ICD-10 (International Classification of Diseases, Tenth Revision) proved to be the most accurate clinical coding systems for most concepts in our theoretical evaluation. In our practical evaluation, we found that International Classification of Diseases (Tenth Revision) was most often used to construct code sets. Some coding systems were very accurate in representing specific types of clinical concepts, for example, LOINC (Logical Observation Identifiers Names and Codes) for investigation results and ATC (Anatomical Therapeutic Chemical Classification) for drugs. Conclusions: No single coding system seems to fulfill all the needs for representing clinical concepts for clinical decision support systems. Comprehensiveness of the coding systems seems to be offset by complexity and forms a barrier to usability for code set construction. Clinical vocabularies mapped to multiple clinical coding systems could facilitate clinical code set construction. %M 33084593 %R 10.2196/16094 %U http://formative.jmir.org/2020/10/e16094/ %U https://doi.org/10.2196/16094 %U http://www.ncbi.nlm.nih.gov/pubmed/33084593 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 10 %P e18395 %T Phenotypically Similar Rare Disease Identification from an Integrative Knowledge Graph for Data Harmonization: Preliminary Study %A Zhu,Qian %A Nguyen,Dac-Trung %A Alyea,Gioconda %A Hanson,Karen %A Sid,Eric %A Pariser,Anne %+ Division of Pre-Clinical Innovation, National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), 9800 Medical Center Drive, Rockville, MD, 20850, United States, 1 301 4807841, qian.zhu@nih.gov %K GARD %K rare diseases %K phenotypical similarity %K data harmonization %D 2020 %7 2.10.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Although many efforts have been made to develop comprehensive disease resources that capture rare disease information for the purpose of clinical decision making and education, there is no standardized protocol for defining and harmonizing rare diseases across multiple resources. This introduces data redundancy and inconsistency that may ultimately increase confusion and difficulty for the wide use of these resources. To overcome such encumbrances, we report our preliminary study to identify phenotypical similarity among genetic and rare diseases (GARD) that are presenting similar clinical manifestations, and support further data harmonization. Objective: To support rare disease data harmonization, we aim to systematically identify phenotypically similar GARD diseases from a disease-oriented integrative knowledge graph and determine their similarity types. Methods: We identified phenotypically similar GARD diseases programmatically with 2 methods: (1) We measured disease similarity by comparing disease mappings between GARD and other rare disease resources, incorporating manual assessment; 2) we derived clinical manifestations presenting among sibling diseases from disease classifications and prioritized the identified similar diseases based on their phenotypes and genotypes. Results: For disease similarity comparison, approximately 87% (341/392) identified, phenotypically similar disease pairs were validated; 80% (271/392) of these disease pairs were accurately identified as phenotypically similar based on similarity score. The evaluation result shows a high precision (94%) and a satisfactory quality (86% F measure). By deriving phenotypical similarity from Monarch Disease Ontology (MONDO) and Orphanet disease classification trees, we identified a total of 360 disease pairs with at least 1 shared clinical phenotype and gene, which were applied for prioritizing clinical relevance. A total of 662 phenotypically similar disease pairs were identified and will be applied for GARD data harmonization. Conclusions: We successfully identified phenotypically similar rare diseases among the GARD diseases via 2 approaches, disease mapping comparison and phenotypical similarity derivation from disease classification systems. The results will not only direct GARD data harmonization in expanding translational science research but will also accelerate data transparency and consistency across different disease resources and terminologies, helping to build a robust and up-to-date knowledge resource on rare diseases. %M 33006565 %R 10.2196/18395 %U https://medinform.jmir.org/2020/10/e18395 %U https://doi.org/10.2196/18395 %U http://www.ncbi.nlm.nih.gov/pubmed/33006565 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 9 %P e19848 %T Chinese Clinical Named Entity Recognition in Electronic Medical Records: Development of a Lattice Long Short-Term Memory Model With Contextualized Character Representations %A Li,Yongbin %A Wang,Xiaohua %A Hui,Linhu %A Zou,Liping %A Li,Hongjin %A Xu,Luo %A Liu,Weihai %+ School of Medical Information Engineering, Zunyi Medical University, 6 Xuefu Road West, Xinpu New District., Zunyi, 563000, China, 86 18311545098, bynn456@126.com %K clinical named entity recognition %K ELMo %K lattice LSTM %K deep learning %K neural network %K sequence tagging %D 2020 %7 4.9.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Clinical named entity recognition (CNER), whose goal is to automatically identify clinical entities in electronic medical records (EMRs), is an important research direction of clinical text data mining and information extraction. The promotion of CNER can provide support for clinical decision making and medical knowledge base construction, which could then improve overall medical quality. Compared with English CNER, and due to the complexity of Chinese word segmentation and grammar, Chinese CNER was implemented later and is more challenging. Objective: With the development of distributed representation and deep learning, a series of models have been applied in Chinese CNER. Different from the English version, Chinese CNER is mainly divided into character-based and word-based methods that cannot make comprehensive use of EMR information and cannot solve the problem of ambiguity in word representation. Methods: In this paper, we propose a lattice long short-term memory (LSTM) model combined with a variant contextualized character representation and a conditional random field (CRF) layer for Chinese CNER: the Embeddings from Language Models (ELMo)-lattice-LSTM-CRF model. The lattice LSTM model can effectively utilize the information from characters and words in Chinese EMRs; in addition, the variant ELMo model uses Chinese characters as input instead of the character-encoding layer of the ELMo model, so as to learn domain-specific contextualized character embeddings. Results: We evaluated our method using two Chinese CNER datasets from the China Conference on Knowledge Graph and Semantic Computing (CCKS): the CCKS-2017 CNER dataset and the CCKS-2019 CNER dataset. We obtained F1 scores of 90.13% and 85.02% on the test sets of these two datasets, respectively. Conclusions: Our results show that our proposed method is effective in Chinese CNER. In addition, the results of our experiments show that variant contextualized character representations can significantly improve the performance of the model. %M 32885786 %R 10.2196/19848 %U http://medinform.jmir.org/2020/9/e19848/ %U https://doi.org/10.2196/19848 %U http://www.ncbi.nlm.nih.gov/pubmed/32885786 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 22 %N 8 %P e18350 %T Social Media Text Mining Framework for Drug Abuse: Development and Validation Study With an Opioid Crisis Case Analysis %A Nasralah,Tareq %A El-Gayar,Omar %A Wang,Yong %+ Supply Chain and Information Management Group, D’Amore-McKim School of Business, Northeastern University, 360 Huntington Ave, Boston, MA, 02115, United States, 1 617 373 4460, t.nasralah@northeastern.edu %K drug abuse %K social media %K infodemiology %K infoveillance %K text mining %K opioid crisis %D 2020 %7 13.8.2020 %9 Original Paper %J J Med Internet Res %G English %X Background: Social media are considered promising and viable sources of data for gaining insights into various disease conditions and patients’ attitudes, behaviors, and medications. They can be used to recognize communication and behavioral themes of problematic use of prescription drugs. However, mining and analyzing social media data have challenges and limitations related to topic deduction and data quality. As a result, we need a structured approach to analyze social media content related to drug abuse in a manner that can mitigate the challenges and limitations surrounding the use of such data. Objective: This study aimed to develop and evaluate a framework for mining and analyzing social media content related to drug abuse. The framework is designed to mitigate challenges and limitations related to topic deduction and data quality in social media data analytics for drug abuse. Methods: The proposed framework started with defining different terms related to the keywords, categories, and characteristics of the topic of interest. We then used the Crimson Hexagon platform to collect data based on a search query informed by a drug abuse ontology developed using the identified terms. We subsequently preprocessed the data and examined the quality using an evaluation matrix. Finally, a suitable data analysis approach could be used to analyze the collected data. Results: The framework was evaluated using the opioid epidemic as a drug abuse case analysis. We demonstrated the applicability of the proposed framework to identify public concerns toward the opioid epidemic and the most discussed topics on social media related to opioids. The results from the case analysis showed that the framework could improve the discovery and identification of topics in social media domains characterized by a plethora of highly diverse terms and lack of a commonly available dictionary or language by the community, such as in the case of opioid and drug abuse. Conclusions: The proposed framework addressed the challenges related to topic detection and data quality. We demonstrated the applicability of the proposed framework to identify the common concerns toward the opioid epidemic and the most discussed topics on social media related to opioids. %M 32788147 %R 10.2196/18350 %U https://www.jmir.org/2020/8/e18350 %U https://doi.org/10.2196/18350 %U http://www.ncbi.nlm.nih.gov/pubmed/32788147 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 4 %N 8 %P e16422 %T Occupation Coding of Job Titles: Iterative Development of an Automated Coding Algorithm for the Canadian National Occupation Classification (ACA-NOC) %A Bao,Hongchang %A Baker,Christopher J O %A Adisesh,Anil %+ Department of Computer Science, Faculty of Science, Applied Science and Engineering, University of New Brunswick, 100 Tucker Park Rd, Saint John, NB, E2K5E2, Canada, 1 (506) 648 2302, bakerc@unb.ca %K occupation coding %K automated coding %K occupational health %K job title %D 2020 %7 5.8.2020 %9 Original Paper %J JMIR Form Res %G English %X Background: In many research studies, the identification of social determinants is an important activity, in particular, information about occupations is frequently added to existing patient data. Such information is usually solicited during interviews with open-ended questions such as “What is your job?” and “What industry sector do you work in?” Before being able to use this information for further analysis, the responses need to be categorized using a coding system, such as the Canadian National Occupational Classification (NOC). Manual coding is the usual method, which is a time-consuming and error-prone activity, suitable for automation. Objective: This study aims to facilitate automated coding by introducing a rigorous algorithm that will be able to identify the NOC (2016) codes using only job title and industry information as input. Using manually coded data sets, we sought to benchmark and iteratively improve the performance of the algorithm. Methods: We developed the ACA-NOC algorithm based on the NOC (2016), which allowed users to match NOC codes with job and industry titles. We employed several different search strategies in the ACA-NOC algorithm to find the best match, including exact search, minor exact search, like search, near (same order) search, near (different order) search, any search, and weak match search. In addition, a filtering step based on the hierarchical structure of the NOC data was applied to the algorithm to select the best matching codes. Results: The ACA-NOC was applied to over 500 manually coded job and industry titles. The accuracy rate at the four-digit NOC code level was 58.7% (332/566) and improved when broader job categories were considered (65.0% at the three-digit NOC code level, 72.3% at the two-digit NOC code level, and 81.6% at the one-digit NOC code level). Conclusions: The ACA-NOC is a rigorous algorithm for automatically coding the Canadian NOC system and has been evaluated using real-world data. It allows researchers to code moderate-sized data sets with occupation in a timely and cost-efficient manner such that further analytics are possible. Initial assessments indicate that it has state-of-the-art performance and is readily extensible upon further benchmarking on larger data sets. %M 32755893 %R 10.2196/16422 %U https://formative.jmir.org/2020/8/e16422 %U https://doi.org/10.2196/16422 %U http://www.ncbi.nlm.nih.gov/pubmed/32755893 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 7 %P e17652 %T Temporal Expression Classification and Normalization From Chinese Narrative Clinical Texts: Pattern Learning Approach %A Pan,Xiaoyi %A Chen,Boyu %A Weng,Heng %A Gong,Yongyi %A Qu,Yingying %+ School of Business, Guangdong University of Foreign Studies, Faculty Building, Higher Education Mega Center, Guangdong University of Foreign Studies, Guangzhou, , China, 86 020 3932 8022, jessie.qu@gdufs.edu.cn %K Temporal expression extraction %K Temporal expression normalization %K Machine learning %K Heuristic rule %K Pattern learning %K Clinical text %D 2020 %7 27.7.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Temporal information frequently exists in the representation of the disease progress, prescription, medication, surgery progress, or discharge summary in narrative clinical text. The accurate extraction and normalization of temporal expressions can positively boost the analysis and understanding of narrative clinical texts to promote clinical research and practice. Objective: The goal of the study was to propose a novel approach for extracting and normalizing temporal expressions from Chinese narrative clinical text. Methods: TNorm, a rule-based and pattern learning-based approach, has been developed for automatic temporal expression extraction and normalization from unstructured Chinese clinical text data. TNorm consists of three stages: extraction, classification, and normalization. It applies a set of heuristic rules and automatically generated patterns for temporal expression identification and extraction of clinical texts. Then, it collects the features of extracted temporal expressions for temporal type prediction and classification by using machine learning algorithms. Finally, the features are combined with the rule-based and a pattern learning-based approach to normalize the extracted temporal expressions. Results: The evaluation dataset is a set of narrative clinical texts in Chinese containing 1459 discharge summaries of a domestic Grade A Class 3 hospital. The results show that TNorm, combined with temporal expressions extraction and temporal types prediction, achieves a precision of 0.8491, a recall of 0.8328, and a F1 score of 0.8409 in temporal expressions normalization. Conclusions: This study illustrates an automatic approach, TNorm, that extracts and normalizes temporal expression from Chinese narrative clinical texts. TNorm was evaluated on the basis of discharge summary data, and results demonstrate its effectiveness on temporal expression normalization. %M 32716307 %R 10.2196/17652 %U https://medinform.jmir.org/2020/7/e17652 %U https://doi.org/10.2196/17652 %U http://www.ncbi.nlm.nih.gov/pubmed/32716307 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 7 %P e16764 %T Embedding “Smart” Disease Coding Within Routine Electronic Medical Record Workflow: Prospective Single-Arm Trial %A Mangin,Dee %A Lawson,Jennifer %A Adamczyk,Krzysztof %A Guenter,Dale %+ Department of Family Medicine, McMaster University, David Braley Health Sciences Centre, 5th Floor, 100 Main Street West, Hamilton, ON, L8P 1H6, Canada, 1 905 525 9140 ext 21219, mangind@mcmaster.ca %K chronic disease management %K comorbidity %K problem list %K disease coding %K disease registry %K data improvement %K electronic medical record %K electronic health record %K practice-based research network %K population health %K primary care %K family medicine %D 2020 %7 27.7.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Electronic medical record (EMR) chronic disease measurement can help direct primary care prevention and treatment strategies and plan health services resource management. Incomplete data and poor consistency of coded disease values within EMR problem lists are widespread issues that limit primary and secondary uses of these data. These issues were shared by the McMaster University Sentinel and Information Collaboration (MUSIC), a primary care practice-based research network (PBRN) located in Hamilton, Ontario, Canada. Objective: We sought to develop and evaluate the effectiveness of new EMR interface tools aimed at improving the quantity and the consistency of disease codes recorded within the disease registry across the MUSIC PBRN. Methods: We used a single-arm prospective trial design with preintervention and postintervention data analysis to assess the effect of the intervention on disease recording volume and quality. The MUSIC network holds data on over 75,080 patients, 37,212 currently rostered. There were 4 MUSIC network clinician champions involved in gap analysis of the disease coding process and in the iterative design of new interface tools. We leveraged terminology standards and factored EMR workflow and usability into a new interface solution that aimed to optimize code selection volume and quality while minimizing physician time burden. The intervention was integrated as part of usual clinical workflow during routine billing activities. Results: After implementation of the new interface (June 25, 2017), we assessed the disease registry codes at 3 and 6 months (intervention period) to compare their volume and quality to preintervention levels (baseline period). A total of 17,496 International Classification of Diseases, 9th Revision (ICD9) code values were recorded in the disease registry during the 11.5-year (2006 to mid-2017) baseline period. A large gain in disease recording occurred in the intervention period (8516/17,496, 48.67% over baseline), resulting in a total of 26,774 codes. The coding rate increased by a factor of 11.2, averaging 1419 codes per month over the baseline average rate of 127 codes per month. The proportion of preferred ICD9 codes increased by 17.03% in the intervention period (11,007/17,496, 62.91% vs 7417/9278, 79.94%; χ21=819.4; P<.001). A total of 45.03% (4178/9278) of disease codes were entered by way of the new screen prompt tools, with significant increases between quarters (Jul-Sep: 2507/6140, 40.83% vs Oct-Dec: 1671/3148, 53.08%; χ21=126.2; P<.001). Conclusions: The introduction of clinician co-designed, workflow-embedded disease coding tools is a very effective solution to the issues of poor disease coding and quality in EMRs. The substantial effectiveness in a routine care environment demonstrates usability, and the intervention detail described here should be generalizable to any setting. Significant improvements in problem list coding within primary care EMRs can be realized with minimal disruption to routine clinical workflow. %M 32716304 %R 10.2196/16764 %U http://medinform.jmir.org/2020/7/e16764/ %U https://doi.org/10.2196/16764 %U http://www.ncbi.nlm.nih.gov/pubmed/32716304 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 22 %N 7 %P e20443 %T Ontological Organization and Bioinformatic Analysis of Adverse Drug Reactions From Package Inserts: Development and Usability Study %A Li,Xiaoying %A Lin,Xin %A Ren,Huiling %A Guo,Jinjing %+ Institute of Medical Information, Chinese Academy of Medical Sciences, 69 Dongdan N St, Beijing, 100005, China, 86 10 52328911, ren.huiling@imicams.ac.cn %K ontology %K adverse drug reactions %K package inserts %K information retrieval %K natural language processing %K bioinformatics %K drug %K adverse events %K machine-understandable knowledge %K clinical applications %D 2020 %7 20.7.2020 %9 Original Paper %J J Med Internet Res %G English %X Background: Licensed drugs may cause unexpected adverse reactions in patients, resulting in morbidity, risk of mortality, therapy disruptions, and prolonged hospital stays. Officially approved drug package inserts list the adverse reactions identified from randomized controlled clinical trials with high evidence levels and worldwide postmarketing surveillance. Formal representation of the adverse drug reaction (ADR) enclosed in semistructured package inserts will enable deep recognition of side effects and rational drug use, substantially reduce morbidity, and decrease societal costs. Objective: This paper aims to present an ontological organization of traceable ADR information extracted from licensed package inserts. In addition, it will provide machine-understandable knowledge for bioinformatics analysis, semantic retrieval, and intelligent clinical applications. Methods: Based on the essential content of package inserts, a generic ADR ontology model is proposed from two dimensions (and nine subdimensions), covering the ADR information and medication instructions. This is followed by a customized natural language processing method programmed with Python to retrieve the relevant information enclosed in package inserts. After the biocuration and identification of retrieved data from the package insert, an ADR ontology is automatically built for further bioinformatic analysis. Results: We collected 165 package inserts of quinolone drugs from the National Medical Products Administration and other drug databases in China, and built a specialized ADR ontology containing 2879 classes and 15,711 semantic relations. For each quinolone drug, the reported ADR information and medication instructions have been logically represented and formally organized in an ADR ontology. To demonstrate its usage, the source data were further bioinformatically analyzed. For example, the number of drug-ADR triples and major ADRs associated with each active ingredient were recorded. The 10 ADRs most frequently observed among quinolones were identified and categorized based on the 18 categories defined in the proposal. The occurrence frequency, severity, and ADR mitigation method explicitly stated in package inserts were also analyzed, as well as the top 5 specific populations with contraindications for quinolone drugs. Conclusions: Ontological representation and organization using officially approved information from drug package inserts enables the identification and bioinformatic analysis of adverse reactions caused by a specific drug with regard to predefined ADR ontology classes and semantic relations. The resulting ontology-based ADR knowledge source classifies drug-specific adverse reactions, and supports a better understanding of ADRs and safer prescription of medications. %M 32706718 %R 10.2196/20443 %U https://www.jmir.org/2020/7/e20443 %U https://doi.org/10.2196/20443 %U http://www.ncbi.nlm.nih.gov/pubmed/32706718 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 22 %N 7 %P e14591 %T Information Loss in Harmonizing Granular Race and Ethnicity Data: Descriptive Study of Standards %A Wang,Karen %A Grossetta Nardini,Holly %A Post,Lori %A Edwards,Todd %A Nunez-Smith,Marcella %A Brandt,Cynthia %+ Equity Research and Innovation Center, General Internal Medicine, Yale School of Medicine, 100 Church Street South, A200, New Haven, CT, 06520, United States, 1 203 785 5233, karen.wang@yale.edu %K continental population groups %K multiracial populations %K multiethnic groups %K data standards %K health status disparities %K race factors %K demography %D 2020 %7 20.7.2020 %9 Original Paper %J J Med Internet Res %G English %X Background: Data standards for race and ethnicity have significant implications for health equity research. Objective: We aim to describe a challenge encountered when working with a multiple–race and ethnicity assessment in the Eastern Caribbean Health Outcomes Research Network (ECHORN), a research collaborative of Barbados, Puerto Rico, Trinidad and Tobago, and the US Virgin Islands. Methods: We examined the data standards guiding harmonization of race and ethnicity data for multiracial and multiethnic populations, using the Office of Management and Budget (OMB) Statistical Policy Directive No. 15. Results: Of 1211 participants in the ECHORN cohort study, 901 (74.40%) selected 1 racial category. Of those that selected 1 category, 13.0% (117/901) selected Caribbean; 6.4% (58/901), Puerto Rican or Boricua; and 13.5% (122/901), the mixed or multiracial category. A total of 17.84% (216/1211) of participants selected 2 or more categories, with 15.19% (184/1211) selecting 2 categories and 2.64% (32/1211) selecting 3 or more categories. With aggregation of ECHORN data into OMB categories, 27.91% (338/1211) of the participants can be placed in the “more than one race” category. Conclusions: This analysis exposes the fundamental informatics challenges that current race and ethnicity data standards present to meaningful collection, organization, and dissemination of granular data about subgroup populations in diverse and marginalized communities. Current standards should reflect the science of measuring race and ethnicity and the need for multidisciplinary teams to improve evolving standards throughout the data life cycle. %M 32706693 %R 10.2196/14591 %U http://www.jmir.org/2020/7/e14591/ %U https://doi.org/10.2196/14591 %U http://www.ncbi.nlm.nih.gov/pubmed/32706693 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 7 %P e16008 %T Prediction of Medical Concepts in Electronic Health Records: Similar Patient Analysis %A Le,Nhat %A Wiley,Matthew %A Loza,Antonio %A Hristidis,Vagelis %A El-Kareh,Robert %+ Department of Computer Science & Engineering, University of California, Riverside, Winston Chung Hall 363, 900 University Ave., Riverside, CA, 92521, United States, 1 9518275639, nle020@ucr.edu %K consumer health information %K decision support techniques %K electronic health record %D 2020 %7 17.7.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Medicine 2.0—the adoption of Web 2.0 technologies such as social networks in health care—creates the need for apps that can find other patients with similar experiences and health conditions based on a patient’s electronic health record (EHR). Concurrently, there is an increasing number of longitudinal EHR data sets with rich information, which are essential to fulfill this need. Objective: This study aimed to evaluate the hypothesis that we can leverage similar EHRs to predict possible future medical concepts (eg, disorders) from a patient’s EHR. Methods: We represented patients’ EHRs using time-based prefixes and suffixes, where each prefix or suffix is a set of medical concepts from a medical ontology. We compared the prefixes of other patients in the collection with the state of the current patient using various interpatient distance measures. The set of similar prefixes yields a set of suffixes, which we used to determine probable future concepts for the current patient’s EHR. Results: We evaluated our methods on the Multiparameter Intelligent Monitoring in Intensive Care II data set of patients, where we achieved precision up to 56.1% and recall up to 69.5%. For a limited set of clinically interesting concepts, specifically a set of procedures, we found that 86.9% (353/406) of the true-positives are clinically useful, that is, these procedures were actually performed later on the patient, and only 4.7% (19/406) of true-positives were completely irrelevant. Conclusions: These initial results indicate that predicting patients’ future medical concepts is feasible. Effectively predicting medical concepts can have several applications, such as managing resources in a hospital. %M 32706678 %R 10.2196/16008 %U https://medinform.jmir.org/2020/7/e16008 %U https://doi.org/10.2196/16008 %U http://www.ncbi.nlm.nih.gov/pubmed/32706678 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 7 %P e17176 %T Data Integration in the Brazilian Public Health System for Tuberculosis: Use of the Semantic Web to Establish Interoperability %A Pellison,Felipe Carvalho %A Rijo,Rui Pedro Charters Lopes %A Lima,Vinicius Costa %A Crepaldi,Nathalia Yukie %A Bernardi,Filipe Andrade %A Galliez,Rafael Mello %A Kritski,Afrânio %A Abhishek,Kumar %A Alves,Domingos %+ Bioengineering Postgraduate Program of the São Carlos School of Engineering, University of São Paulo, Av. Trabalhador São-Carlense, 400, São Carlos, 13566-590, Brazil, 55 16 3373 9586, felipecp@alumni.usp.br %K health information systems %K tuberculosis %K ontology %K interoperability %K electronic health records %K semantic web %D 2020 %7 6.7.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Interoperability of health information systems is a challenge due to the heterogeneity of existing systems at both the technological and semantic levels of their data. The lack of existing data about interoperability disrupts intra-unit and inter-unit medical operations as well as creates challenges in conducting studies on existing data. The goal is to exchange data while providing the same meaning for data from different sources. Objective: To find ways to solve this challenge, this research paper proposes an interoperability solution for the tuberculosis treatment and follow-up scenario in Brazil using Semantic Web technology supported by an ontology. Methods: The entities of the ontology were allocated under the definitions of Basic Formal Ontology. Brazilian tuberculosis applications were tagged with entities from the resulting ontology. Results: An interoperability layer was developed to retrieve data with the same meaning and in a structured way enabling semantic and functional interoperability. Conclusions: Health professionals could use the data gathered from several data sources to enhance the effectiveness of their actions and decisions, as shown in a practical use case to integrate tuberculosis data in the State of São Paulo. %M 32628611 %R 10.2196/17176 %U https://medinform.jmir.org/2020/7/e17176 %U https://doi.org/10.2196/17176 %U http://www.ncbi.nlm.nih.gov/pubmed/32628611 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 6 %P e17650 %T Automatic Construction of a Depression-Domain Lexicon Based on Microblogs: Text Mining Study %A Li,Genghao %A Li,Bing %A Huang,Langlin %A Hou,Sibing %+ School of Information Technology & Management, University of International Business and Economics, Chaoyang District, Huixin East Street, Beijing, 100029, China, 86 1 343 978 8086, 01630@uibe.edu.cn %K depression detection %K depression diagnosis %K social media %K automatic construction %K domain-specific lexicon %K depression lexicon %K label propagation %D 2020 %7 23.6.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: According to a World Health Organization report in 2017, there was almost one patient with depression among every 20 people in China. However, the diagnosis of depression is usually difficult in terms of clinical detection owing to slow observation, high cost, and patient resistance. Meanwhile, with the rapid emergence of social networking sites, people tend to share their daily life and disclose inner feelings online frequently, making it possible to effectively identify mental conditions using the rich text information. There are many achievements regarding an English web-based corpus, but for research in China so far, the extraction of language features from web-related depression signals is still in a relatively primary stage. Objective: The purpose of this study was to propose an effective approach for constructing a depression-domain lexicon. This lexicon will contain language features that could help identify social media users who potentially have depression. Our study also compared the performance of detection with and without our lexicon. Methods: We autoconstructed a depression-domain lexicon using Word2Vec, a semantic relationship graph, and the label propagation algorithm. These two methods combined performed well in a specific corpus during construction. The lexicon was obtained based on 111,052 Weibo microblogs from 1868 users who were depressed or nondepressed. During depression detection, we considered six features, and we used five classification methods to test the detection performance. Results: The experiment results showed that in terms of the F1 value, our autoconstruction method performed 1% to 6% better than baseline approaches and was more effective and steadier. When applied to detection models like logistic regression and support vector machine, our lexicon helped the models outperform by 2% to 9% and was able to improve the final accuracy of potential depression detection. Conclusions: Our depression-domain lexicon was proven to be a meaningful input for classification algorithms, providing linguistic insights on the depressive status of test subjects. We believe that this lexicon will enhance early depression detection in people on social media. Future work will need to be carried out on a larger corpus and with more complex methods. %M 32574151 %R 10.2196/17650 %U http://medinform.jmir.org/2020/6/e17650/ %U https://doi.org/10.2196/17650 %U http://www.ncbi.nlm.nih.gov/pubmed/32574151 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 6 %P e17821 %T End-to-End Models to Imitate Traditional Chinese Medicine Syndrome Differentiation in Lung Cancer Diagnosis: Model Development and Validation %A Liu,Ziqing %A He,Haiyang %A Yan,Shixing %A Wang,Yong %A Yang,Tao %A Li,Guo-Zheng %+ School of Artifical Intelligence and Information Techology, Nanjing University of Chinese Medicine, Nanjing, China, 86 13405803341, taoyang1111@126.com %K traditional Chinese medicine %K syndrome differentiation %K lung cancer %K medical record %K deep learning %K model fusion %D 2020 %7 16.6.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Traditional Chinese medicine (TCM) has been shown to be an efficient mode to manage advanced lung cancer, and accurate syndrome differentiation is crucial to treatment. Documented evidence of TCM treatment cases and the progress of artificial intelligence technology are enabling the development of intelligent TCM syndrome differentiation models. This is expected to expand the benefits of TCM to lung cancer patients. Objective: The objective of this work was to establish end-to-end TCM diagnostic models to imitate lung cancer syndrome differentiation. The proposed models used unstructured medical records as inputs to capitalize on data collected for practical TCM treatment cases by lung cancer experts. The resulting models were expected to be more efficient than approaches that leverage structured TCM datasets. Methods: We approached lung cancer TCM syndrome differentiation as a multilabel text classification problem. First, entity representation was conducted with Bidirectional Encoder Representations from Transformers and conditional random fields models. Then, five deep learning–based text classification models were applied to the construction of a medical record multilabel classifier, during which two data augmentation strategies were adopted to address overfitting issues. Finally, a fusion model approach was used to elevate the performance of the models. Results: The F1 score of the recurrent convolutional neural network (RCNN) model with augmentation was 0.8650, a 2.41% improvement over the unaugmented model. The Hamming loss for RCNN with augmentation was 0.0987, which is 1.8% lower than that of the same model without augmentation. Among the models, the text-hierarchical attention network (Text-HAN) model achieved the highest F1 scores of 0.8676 and 0.8751. The mean average precision for the word encoding–based RCNN was 10% higher than that of the character encoding–based representation. A fusion model of the text-convolutional neural network, text-recurrent neural network, and Text-HAN models achieved an F1 score of 0.8884, which showed the best performance among the models. Conclusions: Medical records could be used more productively by constructing end-to-end models to facilitate TCM diagnosis. With the aid of entity-level representation, data augmentation, and model fusion, deep learning–based multilabel classification approaches can better imitate TCM syndrome differentiation in complex cases such as advanced lung cancer. %M 32543445 %R 10.2196/17821 %U https://medinform.jmir.org/2020/6/e17821 %U https://doi.org/10.2196/17821 %U http://www.ncbi.nlm.nih.gov/pubmed/32543445 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 6 %P e17353 %T Detecting and Filtering Immune-Related Adverse Events Signal Based on Text Mining and Observational Health Data Sciences and Informatics Common Data Model: Framework Development Study %A Yu,Yue %A Ruddy,Kathryn %A Mansfield,Aaron %A Zong,Nansu %A Wen,Andrew %A Tsuji,Shintaro %A Huang,Ming %A Liu,Hongfang %A Shah,Nilay %A Jiang,Guoqian %+ Department of Health Sciences Research, Mayo Clinic, 200 First St SW, Rochester, MN, 55905, United States, 1 507 284 2511, jiang.guoqian@mayo.edu %K immunotherapy/adverse effects %K drug-related side effects and adverse reactions %K pharmacovigilance %K adverse drug reaction reporting systems/standards %K text mining %D 2020 %7 12.6.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Immune checkpoint inhibitors are associated with unique immune-related adverse events (irAEs). As most of the immune checkpoint inhibitors are new to the market, it is important to conduct studies using real-world data sources to investigate their safety profiles. Objective: The aim of the study was to develop a framework for signal detection and filtration of novel irAEs for 6 Food and Drug Administration–approved immune checkpoint inhibitors. Methods: In our framework, we first used the Food and Drug Administration’s Adverse Event Reporting System (FAERS) standardized in an Observational Health Data Sciences and Informatics (OHDSI) common data model (CDM) to collect immune checkpoint inhibitor-related event data and conducted irAE signal detection. OHDSI CDM is a standard-driven data model that focuses on transforming different databases into a common format and standardizing medical terms to a common representation. We then filtered those already known irAEs from drug labels and literature by using a customized text-mining pipeline based on clinical text analysis and knowledge extraction system with Medical Dictionary for Regulatory Activities (MedDRA) as a dictionary. Finally, we classified the irAE detection results into three different categories to discover potentially new irAE signals. Results: By our text-mining pipeline, 490 irAE terms were identified from drug labels, and 918 terms were identified from the literature. In addition, of the 94 positive signals detected using CDM-based FAERS, 53 signals (56%) were labeled signals, 10 (11%) were unlabeled published signals, and 31 (33%) were potentially new signals. Conclusions: We demonstrated that our approach is effective for irAE signal detection and filtration. Moreover, our CDM-based framework could facilitate adverse drug events detection and filtration toward the goal of next-generation pharmacovigilance that seamlessly integrates electronic health record data for improved signal detection. %M 32530430 %R 10.2196/17353 %U http://medinform.jmir.org/2020/6/e17353/ %U https://doi.org/10.2196/17353 %U http://www.ncbi.nlm.nih.gov/pubmed/32530430 %0 Journal Article %@ 2562-7600 %I JMIR Publications %V 3 %N 1 %P e18501 %T Comparison of the Results of Manual and Automated Processes of Cross-Mapping Between Nursing Terms: Quantitative Study %A Torres,Fernanda Broering Gomes %A Gomes,Denilsen Carvalho %A Hino,Adriano Akira Ferreira %A Moro,Claudia %A Cubas,Marcia Regina %+ Graduate Program in Health Technology, Pontificia Universidade Católica do Paraná, Rua Imaculada Conceição 1155, Curitiba, 80215-901, Brazil, 55 41 3271 1657, ferbroering@yahoo.com.br %K health information interoperability %K nursing informatics %K controlled vocabulary %K standardized nursing terminology %K ehealth %D 2020 %7 9.6.2020 %9 Original Paper %J JMIR Nursing %G English %X Background: Cross-mapping establishes equivalence between terms from different terminology systems, which is useful for interoperability, updated terminological versions, and reuse of terms. Due to the number of terms to be mapped, this work can be extensive, tedious, and thorough, and it is susceptible to errors; this can be minimized by automated processes, which use computational tools. Objective: The aim of this study was to compare the results of manual and automated term mapping processes. Methods: In this descriptive, quantitative study, we used the results of two mapping processes as an empirical basis: manual, which used 2638 terms of nurses’ records from a university hospital in southern Brazil and the International Classification for Nursing Practice (ICNP); and automated, which used the same university hospital terms and the primitive terms of the ICNP through MappICNP, an algorithm based on rules of natural language processing. The two processes were compared via equality and exclusivity assessments of new terms of the automated process and of candidate terms. Results: The automated process mapped 569/2638 (21.56%) of the source bank’s terms as identical, and the manual process mapped 650/2638 (24.63%) as identical. Regarding new terms, the automated process mapped 1031/2638 (39.08%) of the source bank’s terms as new, while the manual process mapped 1251 (47.42%). In particular, manual mapping identified 101/2638 (3.82%) terms as identical and 429 (16.26%) as new, whereas the automated process identified 20 (0.75%) terms as identical and 209 (7.92%) as new. Of the 209 terms mapped as new by the automated process, it was possible to establish an equivalence with ICNP terms in 48 (23.0%) cases. An analysis of the candidate terms offered by the automated process to the 429 new terms mapped exclusively by the manual process resulted in 100 (23.3%) candidates that had a semantic relationship with the source term. Conclusions: The automated and manual processes map identical and new terms in similar ways and can be considered complementary. Direct identification of identical terms and the offering of candidate terms through the automated process facilitate and enhance the results of the mapping; confirmation of the precision of the automated mapping requires further analysis by researchers. %M 34345784 %R 10.2196/18501 %U https://nursing.jmir.org/2020/1/e18501/ %U https://doi.org/10.2196/18501 %U http://www.ncbi.nlm.nih.gov/pubmed/34345784 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 6 %P e17819 %T Predicting Onset of Dementia Using Clinical Notes and Machine Learning: Case-Control Study %A Hane,Christopher A %A Nori,Vijay S %A Crown,William H %A Sanghavi,Darshak M %A Bleicher,Paul %+ OptumLabs, Optum, 1 Main St, 10th Floor, Cambridge, MA, 02142, United States, 1 6126326432, christopher.hane@optum.com %K Alzheimer disease %K dementia %K health information systems %K machine learning %K natural language processing %K health information interoperability %D 2020 %7 3.6.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Clinical trials need efficient tools to assist in recruiting patients at risk of Alzheimer disease and related dementias (ADRD). Early detection can also assist patients with financial planning for long-term care. Clinical notes are an important, underutilized source of information in machine learning models because of the cost of collection and complexity of analysis. Objective: This study aimed to investigate the use of deidentified clinical notes from multiple hospital systems collected over 10 years to augment retrospective machine learning models of the risk of developing ADRD. Methods: We used 2 years of data to predict the future outcome of ADRD onset. Clinical notes are provided in a deidentified format with specific terms and sentiments. Terms in clinical notes are embedded into a 100-dimensional vector space to identify clusters of related terms and abbreviations that differ across hospital systems and individual clinicians. Results: When using clinical notes, the area under the curve (AUC) improved from 0.85 to 0.94, and positive predictive value (PPV) increased from 45.07% (25,245/56,018) to 68.32% (14,153/20,717) in the model at disease onset. Models with clinical notes improved in both AUC and PPV in years 3-6 when notes’ volume was largest; results are mixed in years 7 and 8 with the smallest cohorts. Conclusions: Although clinical notes helped in the short term, the presence of ADRD symptomatic terms years earlier than onset adds evidence to other studies that clinicians undercode diagnoses of ADRD. De-identified clinical notes increase the accuracy of risk models. Clinical notes collected across multiple hospital systems via natural language processing can be merged using postprocessing techniques to aid model accuracy. %M 32490841 %R 10.2196/17819 %U https://medinform.jmir.org/2020/6/e17819 %U https://doi.org/10.2196/17819 %U http://www.ncbi.nlm.nih.gov/pubmed/32490841 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 5 %P e16452 %T Challenges of Clustering Multimodal Clinical Data: Review of Applications in Asthma Subtyping %A Horne,Elsie %A Tibble,Holly %A Sheikh,Aziz %A Tsanas,Athanasios %+ Usher Institute, Edinburgh Medical School, University of Edinburgh, Nine Edinburgh Bio Quarter, 9 Little France Road, Edinburgh, EH16 4UX, United Kingdom, 44 1316517887, Elsie.Horne@ed.ac.uk %K asthma %K cluster analysis %K data mining %K machine learning %K unsupervised machine learning %D 2020 %7 28.5.2020 %9 Review %J JMIR Med Inform %G English %X Background: In the current era of personalized medicine, there is increasing interest in understanding the heterogeneity in disease populations. Cluster analysis is a method commonly used to identify subtypes in heterogeneous disease populations. The clinical data used in such applications are typically multimodal, which can make the application of traditional cluster analysis methods challenging. Objective: This study aimed to review the research literature on the application of clustering multimodal clinical data to identify asthma subtypes. We assessed common problems and shortcomings in the application of cluster analysis methods in determining asthma subtypes, such that they can be brought to the attention of the research community and avoided in future studies. Methods: We searched PubMed and Scopus bibliographic databases with terms related to cluster analysis and asthma to identify studies that applied dissimilarity-based cluster analysis methods. We recorded the analytic methods used in each study at each step of the cluster analysis process. Results: Our literature search identified 63 studies that applied cluster analysis to multimodal clinical data to identify asthma subtypes. The features fed into the cluster algorithms were of a mixed type in 47 (75%) studies and continuous in 12 (19%), and the feature type was unclear in the remaining 4 (6%) studies. A total of 23 (37%) studies used hierarchical clustering with Ward linkage, and 22 (35%) studies used k-means clustering. Of these 45 studies, 39 had mixed-type features, but only 5 specified dissimilarity measures that could handle mixed-type features. A further 9 (14%) studies used a preclustering step to create small clusters to feed on a hierarchical method. The original sample sizes in these 9 studies ranged from 84 to 349. The remaining studies used hierarchical clustering with other linkages (n=3), medoid-based methods (n=3), spectral clustering (n=1), and multiple kernel k-means clustering (n=1), and in 1 study, the methods were unclear. Of 63 studies, 54 (86%) explained the methods used to determine the number of clusters, 24 (38%) studies tested the quality of their cluster solution, and 11 (17%) studies tested the stability of their solution. Reporting of the cluster analysis was generally poor in terms of the methods employed and their justification. Conclusions: This review highlights common issues in the application of cluster analysis to multimodal clinical data to identify asthma subtypes. Some of these issues were related to the multimodal nature of the data, but many were more general issues in the application of cluster analysis. Although cluster analysis may be a useful tool for investigating disease subtypes, we recommend that future studies carefully consider the implications of clustering multimodal data, the cluster analysis process itself, and the reporting of methods to facilitate replication and interpretation of findings. %M 32463370 %R 10.2196/16452 %U http://medinform.jmir.org/2020/5/e16452/ %U https://doi.org/10.2196/16452 %U http://www.ncbi.nlm.nih.gov/pubmed/32463370 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 22 %N 5 %P e16795 %T An Informatics Framework to Assess Consumer Health Language Complexity Differences: Proof-of-Concept Study %A Yu,Biyang %A He,Zhe %A Xing,Aiwen %A Lustria,Mia Liza A %+ Florida State University, School of Information, 142 Collegiate Loop, Tallahassee, FL, 32306, United States, 1 850 644 5775, zhe@fsu.edu %K consumer health informatics %K readability %K digital divide %K health literacy %D 2020 %7 21.5.2020 %9 Original Paper %J J Med Internet Res %G English %X Background: The language gap between health consumers and health professionals has been long recognized as the main hindrance to effective health information comprehension. Although providing health information access in consumer health language (CHL) is widely accepted as the solution to the problem, health consumers are found to have varying health language preferences and proficiencies. To simplify health documents for heterogeneous consumer groups, it is important to quantify how CHLs are different in terms of complexity among various consumer groups. Objective: This study aimed to propose an informatics framework (consumer health language complexity [CHELC]) to assess the complexity differences of CHL using syntax-level, text-level, term-level, and semantic-level complexity metrics. Specifically, we identified 8 language complexity metrics validated in previous literature and combined them into a 4-faceted framework. Through a rank-based algorithm, we developed unifying scores (CHELC scores [CHELCS]) to quantify syntax-level, text-level, term-level, semantic-level, and overall CHL complexity. We applied CHELCS to compare posts of each individual on online health forums designed for (1) the general public, (2) deaf and hearing-impaired people, and (3) people with autism spectrum disorder (ASD). Methods: We examined posts with more than 4 sentences of each user from 3 health forums to understand CHL complexity differences among these groups: 12,560 posts from 3756 users in Yahoo! Answers, 25,545 posts from 1623 users in AllDeaf, and 26,484 posts from 2751 users in Wrong Planet. We calculated CHELCS for each user and compared the scores of 3 user groups (ie, deaf and hearing-impaired people, people with ASD, and the public) through 2-sample Kolmogorov-Smirnov tests and analysis of covariance tests. Results: The results suggest that users in the public forum used more complex CHL, particularly more diverse semantics and more complex health terms compared with users in the ASD and deaf and hearing-impaired user forums. However, between the latter 2 groups, people with ASD used more complex words, and deaf and hearing-impaired users used more complex syntax. Conclusions: Our results show that the users in 3 online forums had significantly different CHL complexities in different facets. The proposed framework and detailed measurements help to quantify these CHL complexity differences comprehensively. The results emphasize the importance of tailoring health-related content for different consumer groups with varying CHL complexities. %M 32436849 %R 10.2196/16795 %U https://www.jmir.org/2020/5/e16795 %U https://doi.org/10.2196/16795 %U http://www.ncbi.nlm.nih.gov/pubmed/32436849 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 4 %P e17125 %T A Deep Artificial Neural Network−Based Model for Prediction of Underlying Cause of Death From Death Certificates: Algorithm Development and Validation %A Falissard,Louis %A Morgand,Claire %A Roussel,Sylvie %A Imbaud,Claire %A Ghosn,Walid %A Bounebache,Karim %A Rey,Grégoire %+ Inserm (Institut National de la Santé et de la Recherche Médicale) - CépiDc (Centre d'epidémiologie sur les causes médicales de Décès), 80 Rue du Général Leclerc, Le Kremlin Bicêtre, 94270, France, 33 679649178, louis.falissard@gmail.com %K machine learning %K deep learning %K mortality statistics %K underlying cause of death %D 2020 %7 28.4.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Coding of underlying causes of death from death certificates is a process that is nowadays undertaken mostly by humans with potential assistance from expert systems, such as the Iris software. It is, consequently, an expensive process that can, in addition, suffer from geospatial discrepancies, thus severely impairing the comparability of death statistics at the international level. The recent advances in artificial intelligence, specifically the rise of deep learning methods, has enabled computers to make efficient decisions on a number of complex problems that were typically considered out of reach without human assistance; they require a considerable amount of data to learn from, which is typically their main limiting factor. However, the CépiDc (Centre d’épidémiologie sur les causes médicales de Décès) stores an exhaustive database of death certificates at the French national scale, amounting to several millions of training examples available for the machine learning practitioner. Objective: This article investigates the application of deep neural network methods to coding underlying causes of death. Methods: The investigated dataset was based on data contained from every French death certificate from 2000 to 2015, containing information such as the subject’s age and gender, as well as the chain of events leading to his or her death, for a total of around 8 million observations. The task of automatically coding the subject’s underlying cause of death was then formulated as a predictive modelling problem. A deep neural network−based model was then designed and fit to the dataset. Its error rate was then assessed on an exterior test dataset and compared to the current state-of-the-art (ie, the Iris software). Statistical significance of the proposed approach’s superiority was assessed via bootstrap. Results: The proposed approach resulted in a test accuracy of 97.8% (95% CI 97.7-97.9), which constitutes a significant improvement over the current state-of-the-art and its accuracy of 74.5% (95% CI 74.0-75.0) assessed on the same test example. Such an improvement opens up a whole field of new applications, from nosologist-level batch-automated coding to international and temporal harmonization of cause of death statistics. A typical example of such an application is demonstrated by recoding French overdose-related deaths from 2000 to 2010. Conclusions: This article shows that deep artificial neural networks are perfectly suited to the analysis of electronic health records and can learn a complex set of medical rules directly from voluminous datasets, without any explicit prior knowledge. Although not entirely free from mistakes, the derived algorithm constitutes a powerful decision-making tool that is able to handle structured medical data with an unprecedented performance. We strongly believe that the methods developed in this article are highly reusable in a variety of settings related to epidemiology, biostatistics, and the medical sciences in general. %M 32343252 %R 10.2196/17125 %U http://medinform.jmir.org/2020/4/e17125/ %U https://doi.org/10.2196/17125 %U http://www.ncbi.nlm.nih.gov/pubmed/32343252 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 8 %N 4 %P e17642 %T Using Natural Language Processing Techniques to Provide Personalized Educational Materials for Chronic Disease Patients in China: Development and Assessment of a Knowledge-Based Health Recommender System %A Wang,Zheyu %A Huang,Haoce %A Cui,Liping %A Chen,Juan %A An,Jiye %A Duan,Huilong %A Ge,Huiqing %A Deng,Ning %+ Ministry of Education Key Laboratory of Biomedical Engineering, College of Biomedical Engineering and Instrument Science, Zhejiang University, 38 Zheda Rd, Zhouyiqing Bldg 512, Yuquan Campus, Hangzhou, China, 86 571 2295 2693, zju.dengning@gmail.com %K health education %K ontology %K natural language processing %K chronic disease %K recommender system %D 2020 %7 23.4.2020 %9 Original Paper %J JMIR Med Inform %G English %X Background: Health education emerged as an important intervention for improving the awareness and self-management abilities of chronic disease patients. The development of information technologies has changed the form of patient educational materials from traditional paper materials to electronic materials. To date, the amount of patient educational materials on the internet is tremendous, with variable quality, which makes it hard to identify the most valuable materials by individuals lacking medical backgrounds. Objective: The aim of this study was to develop a health recommender system to provide appropriate educational materials for chronic disease patients in China and evaluate the effect of this system. Methods: A knowledge-based recommender system was implemented using ontology and several natural language processing (NLP) techniques. The development process was divided into 3 stages. In stage 1, an ontology was constructed to describe patient characteristics contained in the data. In stage 2, an algorithm was designed and implemented to generate recommendations based on the ontology. Patient data and educational materials were mapped to the ontology and converted into vectors of the same length, and then recommendations were generated according to similarity between these vectors. In stage 3, the ontology and algorithm were incorporated into an mHealth system for practical use. Keyword extraction algorithms and pretrained word embeddings were used to preprocess educational materials. Three strategies were proposed to improve the performance of keyword extraction. System evaluation was based on a manually assembled test collection for 50 patients and 100 educational documents. Recommendation performance was assessed using the macro precision of top-ranked documents and the overall mean average precision (MAP). Results: The constructed ontology contained 40 classes, 31 object properties, 67 data properties, and 32 individuals. A total of 80 SWRL rules were defined to implement the semantic logic of mapping patient original data to the ontology vector space. The recommender system was implemented as a separate Web service connected with patients' smartphones. According to the evaluation results, our system can achieve a macro precision up to 0.970 for the top 1 recommendation and an overall MAP score up to 0.628. Conclusions: This study demonstrated that a knowledge-based health recommender system has the potential to accurately recommend educational materials to chronic disease patients. Traditional NLP techniques combined with improvement strategies for specific language and domain proved to be effective for improving system performance. One direction for future work is to explore the effect of such systems from the perspective of patients in a practical setting. %M 32324148 %R 10.2196/17642 %U http://medinform.jmir.org/2020/4/e17642/ %U https://doi.org/10.2196/17642 %U http://www.ncbi.nlm.nih.gov/pubmed/32324148 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 4 %P e13917 %T Building a Semantic Health Data Warehouse in the Context of Clinical Trials: Development and Usability Study %A Lelong,Romain %A Soualmia,Lina F %A Grosjean,Julien %A Taalba,Mehdi %A Darmoni,Stéfan J %+ Department of Biomedical Informatics, Rouen University Hospital, D2IM, Porte 21, Cour Leschevin, Hôpital Charles-Nicolle, 1 Rue de Germont, Rouen, 76000, France, 33 232 888 898, romain.lelong@gmail.com %K data warehousing %K search engine %K semantics %K clinical trial %K patient selection %D 2019 %7 20.12.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: The huge amount of clinical, administrative, and demographic data recorded and maintained by hospitals can be consistently aggregated into health data warehouses with a uniform data model. In 2017, Rouen University Hospital (RUH) initiated the design of a semantic health data warehouse enabling both semantic description and retrieval of health information. Objective: This study aimed to present a proof of concept of this semantic health data warehouse, based on the data of 250,000 patients from RUH, and to assess its ability to assist health professionals in prescreening eligible patients in a clinical trials context. Methods: The semantic health data warehouse relies on 3 distinct semantic layers: (1) a terminology and ontology portal, (2) a semantic annotator, and (3) a semantic search engine and NoSQL (not only structured query language) layer to enhance data access performances. The system adopts an entity-centered vision that provides generic search capabilities able to express data requirements in terms of the whole set of interconnected conceptual entities that compose health information. Results: We assessed the ability of the system to assist the search for 95 inclusion and exclusion criteria originating from 5 randomly chosen clinical trials from RUH. The system succeeded in fully automating 39% (29/74) of the criteria and was efficiently used as a prescreening tool for 73% (54/74) of them. Furthermore, the targeted sources of information and the search engine–related or data-related limitations that could explain the results for each criterion were also observed. Conclusions: The entity-centered vision contrasts with the usual patient-centered vision adopted by existing systems. It enables more genericity in the information retrieval process. It also allows to fully exploit the semantic description of health information. Despite their semantic annotation, searching within clinical narratives remained the major challenge of the system. A finer annotation of the clinical texts and the addition of specific functionalities would significantly improve the results. The semantic aspect of the system combined with its generic entity-centered vision enables the processing of a large range of clinical questions. However, an important part of health information remains in clinical narratives, and we are currently investigating novel approaches (deep learning) to enhance the semantic annotation of those unstructured data. %M 31859675 %R 10.2196/13917 %U http://medinform.jmir.org/2019/4/e13917/ %U https://doi.org/10.2196/13917 %U http://www.ncbi.nlm.nih.gov/pubmed/31859675 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 4 %P e14325 %T Mapping ICD-10 and ICD-10-CM Codes to Phecodes: Workflow Development and Initial Evaluation %A Wu,Patrick %A Gifford,Aliya %A Meng,Xiangrui %A Li,Xue %A Campbell,Harry %A Varley,Tim %A Zhao,Juan %A Carroll,Robert %A Bastarache,Lisa %A Denny,Joshua C %A Theodoratou,Evropi %A Wei,Wei-Qi %+ Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Ave, Suite 1500, Nashville, TN, 37203, United States, 1 615 343 1956, wei-qi.wei@vumc.org %K electronic health record %K genome-wide association study %K phenome-wide association study %K phenotyping %K medical informatics applications %K data science %D 2019 %7 29.11.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: The phecode system was built upon the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) for phenome-wide association studies (PheWAS) using the electronic health record (EHR). Objective: The goal of this paper was to develop and perform an initial evaluation of maps from the International Classification of Diseases, 10th Revision (ICD-10) and the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) codes to phecodes. Methods: We mapped ICD-10 and ICD-10-CM codes to phecodes using a number of methods and resources, such as concept relationships and explicit mappings from the Centers for Medicare & Medicaid Services, the Unified Medical Language System, Observational Health Data Sciences and Informatics, Systematized Nomenclature of Medicine-Clinical Terms, and the National Library of Medicine. We assessed the coverage of the maps in two databases: Vanderbilt University Medical Center (VUMC) using ICD-10-CM and the UK Biobank (UKBB) using ICD-10. We assessed the fidelity of the ICD-10-CM map in comparison to the gold-standard ICD-9-CM phecode map by investigating phenotype reproducibility and conducting a PheWAS. Results: We mapped >75% of ICD-10 and ICD-10-CM codes to phecodes. Of the unique codes observed in the UKBB (ICD-10) and VUMC (ICD-10-CM) cohorts, >90% were mapped to phecodes. We observed 70-75% reproducibility for chronic diseases and <10% for an acute disease for phenotypes sourced from the ICD-10-CM phecode map. Using the ICD-9-CM and ICD-10-CM maps, we conducted a PheWAS with a Lipoprotein(a) genetic variant, rs10455872, which replicated two known genotype-phenotype associations with similar effect sizes: coronary atherosclerosis (ICD-9-CM: P<.001; odds ratio (OR) 1.60 [95% CI 1.43-1.80] vs ICD-10-CM: P<.001; OR 1.60 [95% CI 1.43-1.80]) and chronic ischemic heart disease (ICD-9-CM: P<.001; OR 1.56 [95% CI 1.35-1.79] vs ICD-10-CM: P<.001; OR 1.47 [95% CI 1.22-1.77]). Conclusions: This study introduces the beta versions of ICD-10 and ICD-10-CM to phecode maps that enable researchers to leverage accumulated ICD-10 and ICD-10-CM data for PheWAS in the EHR. %M 31553307 %R 10.2196/14325 %U http://medinform.jmir.org/2019/4/e14325/ %U https://doi.org/10.2196/14325 %U http://www.ncbi.nlm.nih.gov/pubmed/31553307 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 3 %P e14107 %T Common Data Elements for Acute Coronary Syndrome: Analysis Based on the Unified Medical Language System %A Kentgen,Markus %A Varghese,Julian %A Samol,Alexander %A Waltenberger,Johannes %A Dugas,Martin %+ Institute of Medical Informatics, University of Münster, Institut für Medizinische Informatik Münster, Albert-Schweitzer-Campus 1, Münster, 48149, Germany, 49 2518354714, julian.varghese@uni-muenster.de %K common data elements %K acute coronary syndrome %K documentation %K standardization %D 2019 %7 23.08.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: Standardization in clinical documentation can increase efficiency and can save time and resources. Objective: The objectives of this work are to compare documentation forms for acute coronary syndrome (ACS), check for standardization, and generate a list of the most common data elements using semantic form annotation with the Unified Medical Language System (UMLS). Methods: Forms from registries, studies, risk scores, quality assurance, official guidelines, and routine documentation from four hospitals in Germany were semantically annotated using UMLS. This allowed for automatic comparison of concept frequencies and the generation of a list of the most common concepts. Results: A total of 3710 forms items from 86 sources were semantically annotated using 842 unique UMLS concepts. Half of all medical concept occurrences were covered by 60 unique concepts, which suggests the existence of a core dataset of relevant concepts. Overlap percentages between forms were relatively low, hinting at inconsistent documentation structures and lack of standardization. Conclusions: This analysis shows a lack of standardized and semantically enriched documentation for patients with ACS. Efforts made by official institutions like the European Society for Cardiology have not yet been fully implemented. Utilizing a standardized and annotated core dataset of the most important data concepts could make export and automatic reuse of data easier. The generated list of common data elements is an exemplary implementation suggestion of the concepts to use in a standardized approach. %M 31444871 %R 10.2196/14107 %U http://medinform.jmir.org/2019/3/e14107/ %U https://doi.org/10.2196/14107 %U http://www.ncbi.nlm.nih.gov/pubmed/31444871 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 3 %P e13554 %T Core Data Elements in Acute Myeloid Leukemia: A Unified Medical Language System–Based Semantic Analysis and Experts’ Review %A Holz,Christian %A Kessler,Torsten %A Dugas,Martin %A Varghese,Julian %+ Institute of Medical Informatics, University of Münster, Institut für Medizinische Informatik Münster, Albert-Schweitzer-Campus 1, Münster, 48149, Germany, 49 2518354714, julian.varghese@uni-muenster.de %K common data elements %K UMLS %K acute myeloid leukemia %K medical informatics %D 2019 %7 12.08.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: For cancer domains such as acute myeloid leukemia (AML), a large set of data elements is obtained from different institutions with heterogeneous data definitions within one patient course. The lack of clinical data harmonization impedes cross-institutional electronic data exchange and future meta-analyses. Objective: This study aimed to identify and harmonize a semantic core of common data elements (CDEs) in clinical routine and research documentation, based on a systematic metadata analysis of existing documentation models. Methods: Lists of relevant data items were collected and reviewed by hematologists from two university hospitals regarding routine documentation and several case report forms of clinical trials for AML. In addition, existing registries and international recommendations were included. Data items were coded to medical concepts via the Unified Medical Language System (UMLS) by a physician and reviewed by another physician. On the basis of the coded concepts, the data sources were analyzed for concept overlaps and identification of most frequent concepts. The most frequent concepts were then implemented as data elements in the standardized format of the Operational Data Model by the Clinical Data Interchange Standards Consortium. Results: A total of 3265 medical concepts were identified, of which 1414 were unique. Among the 1414 unique medical concepts, the 50 most frequent ones cover 26.98% of all concept occurrences within the collected AML documentation. The top 100 concepts represent 39.48% of all concepts’ occurrences. Implementation of CDEs is available on a European research infrastructure and can be downloaded in different formats for reuse in different electronic data capture systems. Conclusions: Information management is a complex process for research-intense disease entities as AML that is associated with a large set of lab-based diagnostics and different treatment options. Our systematic UMLS-based analysis revealed the existence of a core data set and an exemplary reusable implementation for harmonized data capture is available on an established metadata repository. %M 31407666 %R 10.2196/13554 %U http://medinform.jmir.org/2019/3/e13554/ %U https://doi.org/10.2196/13554 %U http://www.ncbi.nlm.nih.gov/pubmed/31407666 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 3 %P e14499 %T Projection Word Embedding Model With Hybrid Sampling Training for Classifying ICD-10-CM Codes: Longitudinal Observational Study %A Lin,Chin %A Lou,Yu-Sheng %A Tsai,Dung-Jang %A Lee,Chia-Cheng %A Hsu,Chia-Jung %A Wu,Ding-Chung %A Wang,Mei-Chuen %A Fang,Wen-Hui %+ Department of Family and Community Medicine, Tri-Service General Hospital, National Defense Medical Center, No. 325, Section 2, Chenggong Road, Neihu District, Taipei, 11490, Taiwan, 886 02 87923100 ext 18448, rumaf.fang@gmail.com %K word embedding %K convolutional neural network %K artificial intelligence %K natural language processing %K electronic health records %D 2019 %7 23.7.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: Most current state-of-the-art models for searching the International Classification of Diseases, Tenth Revision Clinical Modification (ICD-10-CM) codes use word embedding technology to capture useful semantic properties. However, they are limited by the quality of initial word embeddings. Word embedding trained by electronic health records (EHRs) is considered the best, but the vocabulary diversity is limited by previous medical records. Thus, we require a word embedding model that maintains the vocabulary diversity of open internet databases and the medical terminology understanding of EHRs. Moreover, we need to consider the particularity of the disease classification, wherein discharge notes present only positive disease descriptions. Objective: We aimed to propose a projection word2vec model and a hybrid sampling method. In addition, we aimed to conduct a series of experiments to validate the effectiveness of these methods. Methods: We compared the projection word2vec model and traditional word2vec model using two corpora sources: English Wikipedia and PubMed journal abstracts. We used seven published datasets to measure the medical semantic understanding of the word2vec models and used these embeddings to identify the three–character-level ICD-10-CM diagnostic codes in a set of discharge notes. On the basis of embedding technology improvement, we also tried to apply the hybrid sampling method to improve accuracy. The 94,483 labeled discharge notes from the Tri-Service General Hospital of Taipei, Taiwan, from June 1, 2015, to June 30, 2017, were used. To evaluate the model performance, 24,762 discharge notes from July 1, 2017, to December 31, 2017, from the same hospital were used. Moreover, 74,324 additional discharge notes collected from seven other hospitals were tested. The F-measure, which is the major global measure of effectiveness, was adopted. Results: In medical semantic understanding, the original EHR embeddings and PubMed embeddings exhibited superior performance to the original Wikipedia embeddings. After projection training technology was applied, the projection Wikipedia embeddings exhibited an obvious improvement but did not reach the level of original EHR embeddings or PubMed embeddings. In the subsequent ICD-10-CM coding experiment, the model that used both projection PubMed and Wikipedia embeddings had the highest testing mean F-measure (0.7362 and 0.6693 in Tri-Service General Hospital and the seven other hospitals, respectively). Moreover, the hybrid sampling method was found to improve the model performance (F-measure=0.7371/0.6698). Conclusions: The word embeddings trained using EHR and PubMed could understand medical semantics better, and the proposed projection word2vec model improved the ability of medical semantics extraction in Wikipedia embeddings. Although the improvement from the projection word2vec model in the real ICD-10-CM coding task was not substantial, the models could effectively handle emerging diseases. The proposed hybrid sampling method enables the model to behave like a human expert. %R 10.2196/14499 %U http://medinform.jmir.org/2019/3/e14499/ %U https://doi.org/10.2196/14499 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 3 %P e13562 %T Identification of Knee Osteoarthritis Based on Bayesian Network: Pilot Study %A Sheng,Bo %A Huang,Liang %A Wang,Xiangbin %A Zhuang,Jie %A Tang,Lihua %A Deng,Chao %A Zhang,Yanxin %+ Department of Exercise Sciences, The University of Auckland, 4703906, Newmarket, Auckland,, New Zealand, 64 99236859, yanxin.zhang@auckland.ac.nz %K osteoarthritis %K knee %K classification %K health services for the aged %K physical fitness %K Bayesian network %D 2019 %7 18.07.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: Early identification of knee osteoarthritis (OA) can improve treatment outcomes and reduce medical costs. However, there are major limitations among existing classification or prediction models, including abstract data processing and complicated dataset attributes, which hinder their applications in clinical practice. Objective: The aim of this study was to propose a Bayesian network (BN)–based classification model to classify people with knee OA. The proposed model can be treated as a prescreening tool, which can provide decision support for health professionals. Methods: The proposed model’s structure was based on a 3-level BN structure and then retrained by the Bayesian Search (BS) learning algorithm. The model’s parameters were determined by the expectation-maximization algorithm. The used dataset included backgrounds, the target disease, and predictors. The performance of the model was evaluated based on classification accuracy, area under the curve (AUC), specificity, sensitivity, positive predictive value (PPV), and negative predictive value (NPV); it was also compared with other well-known classification models. A test was also performed to explore whether physical fitness tests could improve the performance of the proposed model. Results: A total of 249 elderly people between the ages of 60 and 80 years, living in the Kongjiang community (Shanghai), were recruited from April to September 2007. A total of 157 instances were adopted as the dataset after data preprocessing. The experimental results showed that the results of the proposed model were higher than, or equal to, the mean scores of other classification models: .754 for accuracy, .78 for AUC, .78 for specificity, and .73 for sensitivity. The proposed model provided .45 for PPV and .92 for NPV at the prevalence of 20%. The proposed model also showed a significant improvement when compared with the traditional BN model: 6.3% increase in accuracy (from .709 to .754), 4.0% increase in AUC (from .75 to .78), 6.8% increase in specificity (from .73 to .78), 5.8% increase in sensitivity (from .69 to .73), 15.4% increase in PPV (from .39 to .45), and 2.2% increase in NPV (from .90 to .92). Furthermore, the test results showed that the performance of the proposed model could be largely enhanced through physical fitness tests in 3 evaluation indices: 10.6% increase in accuracy (from .682 to .754), 16.4% increase in AUC (from .67 to .78), and 30.0% increase in specificity (from .60 to .78). Conclusions: The proposed model presents a promising method to classify people with knee OA when compared with other classification models and the traditional BN model. It could be implemented in clinical practice as a prescreening tool for knee OA, which would not only improve the quality of health care for elderly people but also reduce overall medical costs. %M 31322132 %R 10.2196/13562 %U http://medinform.jmir.org/2019/3/e13562/ %U https://doi.org/10.2196/13562 %U http://www.ncbi.nlm.nih.gov/pubmed/31322132 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 3 %P e14797 %T Influence of Scribes on Patient-Physician Communication in Primary Care Encounters: Mixed Methods Study %A Danak,Shivang U %A Guetterman,Timothy C %A Plegue,Melissa A %A Holmstrom,Heather L %A Kadri,Reema %A Duthler,Alexander %A Yoo,Anne %A Buis,Lorraine R %+ Department of Family Medicine, University of Michigan, 1018 Fuller Street, Ann Arbor, MI, 48104, United States, 1 734 998 7120, buisl@umich.edu %K electronic health records %K documentation %K medical informatics %D 2019 %7 11.07.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: With the increasing adoption of electronic health record (EHR) systems, documentation-related burdens have been increasing for health care providers. Recent estimates indicate that primary care providers spend about one-half of their workdays interacting with the EHR, of which about half is focused on clerical tasks. To reduce documentation burdens associated with the EHR, health care systems and physician practices are increasingly implementing medical scribes to assist providers with real-time documentation. Scribes are typically unlicensed paraprofessionals who assist health care providers by documenting notes electronically under the direction of a licensed practitioner or physician in real time. Despite the promise of scribes, few studies have investigated their effect on clinical encounters, particularly with regard to patient-provider communication. Objective: The purpose of this quasi-experimental pilot study was to understand how scribes affect patient-physician communication in primary care clinical encounters. Methods: We employed a convergent mixed methods design and included a sample of three physician-scribe pairs and 34 patients. Patients’ clinical encounters were randomly assigned to a scribe or nonscribe group. We conducted patient surveys focused on perceptions of patient-provider communication and satisfaction with encounters, video recorded clinical encounters, and conducted physician interviews about their experiences with scribes. Results: Overall, the survey results revealed that patients across both arms reported very high satisfaction of communication with their physician, their physician’s use of the EHR, and their care, with very little variability. Video recording analysis supported patient survey data by demonstrating high measures of communication among physicians in both scribed and nonscribed encounters. Furthermore, video recordings revealed that the presence of scribes had very little effect on the clinical encounter. Conclusions: From the patient’s perspective, scribes are an acceptable addition to clinical encounters. Although they do not have much impact on patients’ perceptions of satisfaction and their impact on the clinical encounter itself was minimal, their potential to reduce documentation-related burden on physicians is valuable. Physicians noted important issues related to scribes, including important considerations for implementing scribe programs, the role of scribes in patient interactions, how physicians work with scribes, characteristics of good scribes, and the role of scribes in physician workflow. %M 31298218 %R 10.2196/14797 %U http://medinform.jmir.org/2019/3/e14797/ %U https://doi.org/10.2196/14797 %U http://www.ncbi.nlm.nih.gov/pubmed/31298218 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 21 %N 6 %P e12847 %T Visibility of Community Nursing Within an Administrative Health Classification System: Evaluation of Content Coverage %A Block,Lorraine J %A Currie,Leanne M %A Hardiker,Nicholas R %A Strudwick,Gillian %+ School of Nursing, University of British Columbia, T201-2211 Wesbrook Mall, Vancouver, BC, V6T 2B5, Canada, 1 604 822 7417, lori.block@ubc.ca %K World Health Organization %K classification %K nursing informatics %K medical informatics %K data collection %K terminology %K community health services %K standardized nursing terminology %D 2019 %7 26.06.2019 %9 Original Paper %J J Med Internet Res %G English %X Background: The World Health Organization is in the process of developing an international administrative classification for health called the International Classification of Health Interventions (ICHI). The purpose of ICHI is to provide a tool for supporting intervention reporting and analysis at a global level for policy development and beyond. Nurses represent the largest resource carrying out clinical interventions in any health system. With the shift in nursing care from hospital to community settings in many countries, it is important to ensure that community nursing interventions are present in any international health information system. Thus, an investigation into the extent to which community nursing interventions were covered in ICHI was needed. Objective: The objectives of this study were to examine the extent to which International Classification for Nursing Practice (ICNP) community nursing interventions were represented in the ICHI administrative classification system, to identify themes related to gaps in coverage, and to support continued advancements in understanding the complexities of knowledge representation in standardized clinical terminologies and classifications. Methods: This descriptive study used a content mapping approach in 2 phases in 2018. A total of 187 nursing intervention codes were extracted from the ICNP Community Nursing Catalogue and mapped to ICHI. In phase 1, 2 coders completed independent mapping activities. In phase 2, the 2 coders compared each list and discussed concept matches until consensus on ICNP-ICHI match and on mapping relationship was reached. Results: The initial percentage agreement between the 2 coders was 47% (n=88), but reached 100% with consensus processes. After consensus was reached, 151 (81%) of the community nursing interventions resulted in an ICHI match. A total of 36 (19%) of community nursing interventions had no match to ICHI content. A total of 100 (53%) community nursing interventions resulted in a broader ICHI code, 9 (5%) resulted in a narrower ICHI code, and 42 (23%) were considered equivalent. ICNP concepts that were not represented in ICHI were thematically grouped into the categories family and caregivers, death and dying, and case management. Conclusions: Overall, the content mapping yielded similar results to other content mapping studies in nursing. However, it also found areas of missing concept coverage, difficulties with interterminology mapping, and further need to develop mapping methods. %M 31244480 %R 10.2196/12847 %U https://www.jmir.org/2019/6/e12847/ %U https://doi.org/10.2196/12847 %U http://www.ncbi.nlm.nih.gov/pubmed/31244480 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 21 %N 6 %P e12876 %T Mining of Textual Health Information from Reddit: Analysis of Chronic Diseases With Extracted Entities and Their Relations %A Foufi,Vasiliki %A Timakum,Tatsawan %A Gaudet-Blavignac,Christophe %A Lovis,Christian %A Song,Min %+ Department of Library and Information Science, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul, 120-749, Republic of Korea, 82 22123 2405, min.song@yonsei.ac.kr %K social media %K chronic disease %K data mining %D 2019 %7 13.6.2019 %9 Original Paper %J J Med Internet Res %G English %X Background: Social media platforms constitute a rich data source for natural language processing tasks such as named entity recognition, relation extraction, and sentiment analysis. In particular, social media platforms about health provide a different insight into patient’s experiences with diseases and treatment than those found in the scientific literature. Objective: This paper aimed to report a study of entities related to chronic diseases and their relation in user-generated text posts. The major focus of our research is the study of biomedical entities found in health social media platforms and their relations and the way people suffering from chronic diseases express themselves. Methods: We collected a corpus of 17,624 text posts from disease-specific subreddits of the social news and discussion website Reddit. For entity and relation extraction from this corpus, we employed the PKDE4J tool developed by Song et al (2015). PKDE4J is a text mining system that integrates dictionary-based entity extraction and rule-based relation extraction in a highly flexible and extensible framework. Results: Using PKDE4J, we extracted 2 types of entities and relations: biomedical entities and relations and subject-predicate-object entity relations. In total, 82,138 entities and 30,341 relation pairs were extracted from the Reddit dataset. The most highly mentioned entities were those related to oncological disease (2884 occurrences of cancer) and asthma (2180 occurrences). The relation pair anatomy-disease was the most frequent (5550 occurrences), the highest frequent entities in this pair being cancer and lymph. The manual validation of the extracted entities showed a very good performance of the system at the entity extraction task (3682/5151, 71.48% extracted entities were correctly labeled). Conclusions: This study showed that people are eager to share their personal experience with chronic diseases on social media platforms despite possible privacy and security issues. The results reported in this paper are promising and demonstrate the need for more in-depth studies on the way patients with chronic diseases express themselves on social media platforms. %M 31199327 %R 10.2196/12876 %U http://www.jmir.org/2019/6/e12876/ %U https://doi.org/10.2196/12876 %U http://www.ncbi.nlm.nih.gov/pubmed/31199327 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 21 %N 6 %P e13456 %T Sentiment Analysis of Social Media on Childhood Vaccination: Development of an Ontology %A On,Jeongah %A Park,Hyeoun-Ae %A Song,Tae-Min %+ College of Nursing, Seoul National University, 103, Daehak-ro, Jongno-gu, Seoul, 03080, Republic of Korea, 82 27408827, hapark@snu.ac.kr %K social media %K vaccination %K health information interoperability %K semantics %D 2019 %7 7.6.2019 %9 Original Paper %J J Med Internet Res %G English %X Background: Although vaccination rates are above the threshold for herd immunity in South Korea, a growing number of parents have expressed concerns about the safety of vaccines. It is important to understand these concerns so that we can maintain high vaccination rates. Objective: The aim of this study was to develop a childhood vaccination ontology to serve as a framework for collecting and analyzing social data on childhood vaccination and to use this ontology for identifying concerns about and sentiments toward childhood vaccination from social data. Methods: The domain and scope of the ontology were determined by developing competency questions. We checked if existing ontologies and conceptual frameworks related to vaccination can be reused for the childhood vaccination ontology. Terms were collected from clinical practice guidelines, research papers, and posts on social media platforms. Class concepts were extracted from these terms. A class hierarchy was developed using a top-down approach. The ontology was evaluated in terms of description logics, face and content validity, and coverage. In total, 40,359 Korean posts on childhood vaccination were collected from 27 social media channels between January and December 2015. Vaccination issues were identified and classified using the second-level class concepts of the ontology. The sentiments were classified in 3 ways: positive, negative or neutral. Posts were analyzed using frequency, trend, logistic regression, and association rules. Results: Our childhood vaccination ontology comprised 9 superclasses with 137 subclasses and 431 synonyms for class, attribute, and value concepts. Parent’s health belief appeared in 53.21% (15,709/29,521) of posts and positive sentiments appeared in 64.08% (17,454/27,236) of posts. Trends in sentiments toward vaccination were affected by news about vaccinations. Posts with parents’ health belief, vaccination availability, and vaccination policy were associated with positive sentiments, whereas posts with experience of vaccine adverse events were associated with negative sentiments. Conclusions: The childhood vaccination ontology developed in this study was useful for collecting and analyzing social data on childhood vaccination. We expect that practitioners and researchers in the field of childhood vaccination could use our ontology to identify concerns about and sentiments toward childhood vaccination from social data. %M 31199290 %R 10.2196/13456 %U http://www.jmir.org/2019/6/e13456/ %U https://doi.org/10.2196/13456 %U http://www.ncbi.nlm.nih.gov/pubmed/31199290 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 11 %N 1 %P e9695 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2019 %7 ..2019 %9 %J Online J Public Health Inform %G English %X ObjectiveThe purpose of this study is to review the extant literature for evidence on the validity of ICD-9-CM and -10-CM codes for the purpose of identifying cases of chlamydia, gonorrhea, and syphilis.IntroductionAdministrative data refers to data generated during the processes of health care. These data are a rich source of patient health information, including diagnoses and problem lists, laboratory and diagnostic tests, and medications. Established standards are used to code each data into the appropriate coding systems. The International Classification of Diseases, Ninth and Tenth Revisions, Clinical Modification (ICD-9-CM and ICD-10-CM) codes are the coding standard for diagnoses and have been frequently used to identify cases for the creation of cohorts in examining care delivery, screening, prevalence, and risk factors1,2. However, while some studies have assessed the validity and reliability of ICD-CM codes to identify various conditions such as cerebral palsy and rheumatoid arthritis3,4, the evidence for using ICD codes to accurately identify sexually transmitted infection (STI) cases is largely unexamined. The purpose of this study is to review the extant literature for evidence on the validity of ICD codes for identifying cases of chlamydia, gonorrhea, and syphilis. Our findings will inform efforts to improve the use of administrative data for STI-related health service and surveillance researches.MethodsOur systematic review followed a protocol consistent with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA). We comprehensively searched PubMed and Scopus databases for peer-reviewed articles published before February 2018. Articles were identified with search terms related to our STIs of interest (chlamydia, syphilis or gonorrhea), pelvic inflammatory disease (PID), administrative codes, and validation studies. PID was included as 33%-50% of PID cases are due to chlamydia or gonorrhea5. Only empirical publications appearing in peer-reviewed English language journals were included. Further, we excluded articles classified as letters to the editor, policy briefs, perspectives, commentaries, summaries of future research plans, and grey literature. Additionally, articles without abstracts were also excluded. The screening process used by our review is outlined in Figure 1. Briefly, all articles were subjected to a two-step screening process. First, we reviewed articles based on title and abstract. We eliminated studies that did not focus on STIs or on validation in the context of STIs. Articles were included if they focused on any combination of the STIs of interest, or on PID, and were validation studies on diagnostic testing or administrative codes. Second, selected articles were then reviewed in full to identify studies which included the STIs of interest, assessed and listed ICD-9-CM or -10-CM codes, and measured validity. The snowball technique was used on included articles, whereby we reviewed all references found in the references of the included articles.ResultsOur search strategy identified 1,754 articles to be screened by title and abstract. Of these, only five (0.29%) articles met the initial inclusion criteria. After full text review, only two articles6,7 met the final inclusion criteria to be included in the systematic review. Both articles focused on PID with no assessment of syphilis. They utilized ICD-9-CM codes to identify cases with PID and performed chart reviews to determine true PID status. Results of both articles found positive predictive value (PPV) of PID to be between 18%–79%. Only one article7 examined the PPV of chlamydia (56%; 5/9 cases) and gonorrhea (100%; 4/4 cases) separately.ConclusionsWe identified just two studies that evaluated the validity of ICD codes in identifying the STIs of interest. Both studies focused on PID cases in which chlamydia and gonorrhea diagnoses and tests might be documented. Additionally, since both studies were published before 2015, neither evaluates the use of administrative data following the U.S. transition to ICD-10 codes. Given these findings, further studies are required to examine the predictive value of ICD-9 and -10 codes for all three diseases in the general population.References1. Tao G, Zhang CX. HIV Testing of Commercially Insured Patients Diagnosed With Sexually Transmitted Diseases. Sexually Transmitted Diseases. 2008;35(1):43-6.2. Evans HE, Mercer CH, Rait G, et al. Trends in HIV testing and recording of HIV status in the UK primary care setting: a retrospective cohort study 1995-2005. Sexually Transmitted Infections. 2009.3. Oskoui M, Ng P, Dorais M, et al. Accuracy of administrative claims data for cerebral palsy diagnosis: a retrospective cohort study. CMAJ open. 2017 Jul 18;5(3):E570-e5.4. Sauer BC, Teng C-C, Accortt NA, et al. Models solely using claims-based administrative data are poor predictors of rheumatoid arthritis disease activity. Arthritis Research & Therapy. 2017 2017/05/08;19(1):86.5. Haggerty CL, Ness RB. Epidemiology, pathogenesis and treatment of pelvic inflammatory disease. Expert review of anti-infective therapy. 2006 Apr;4(2):235-47.6. Satterwhite CL, Yu O, Raebel MA, et al. Detection of Pelvic Inflammatory Disease: Development of an Automated Case-Finding Algorithm Using Administrative Data. Infect Dis Obstet Gynecol. 2011;2011:7.7. Ratelle S, Yokoe D, Blejan C, et al. Predictive value of clinical diagnostic codes for the CDC case definition of pelvic inflammatory disease (PID): implications for surveillance. Sex Transm Dis. 2003 Nov;30(11):866-70. %R 10.5210/ojphi.v11i1.9695 %U %U https://doi.org/10.5210/ojphi.v11i1.9695 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 2 %P e12704 %T Development of a Consumer Health Vocabulary by Mining Health Forum Texts Based on Word Embedding: Semiautomatic Approach %A Gu,Gen %A Zhang,Xingting %A Zhu,Xingeng %A Jian,Zhe %A Chen,Ken %A Wen,Dong %A Gao,Li %A Zhang,Shaodian %A Wang,Fei %A Ma,Handong %A Lei,Jianbo %+ Center for Medical Informatics, Peking University, No. 1 Administration Building, Peking University Health Science Center, Haidian Dist, Beijing,, China, 86 (010)82805906, jblei@hsc.pku.edu.cn %K consumer health vocabulary %K word embedding %K representation learning %K natural language processing %K consumer health information %K ontology enrichment %D 2019 %7 23.05.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: The vocabulary gap between consumers and professionals in the medical domain hinders information seeking and communication. Consumer health vocabularies have been developed to aid such informatics applications. This purpose is best served if the vocabulary evolves with consumers’ language. Objective: Our objective is to develop a method for identifying and adding new terms to consumer health vocabularies, so that it can keep up with the constantly evolving medical knowledge and language use. Methods: In this paper, we propose a consumer health term–finding framework based on a distributed word vector space model. We first learned word vectors from a large-scale text corpus and then adopted a supervised method with existing consumer health vocabularies for learning vector representation of words, which can provide additional supervised fine tuning after unsupervised word embedding learning. With a fine-tuned word vector space, we identified pairs of professional terms and their consumer variants by their semantic distance in the vector space. A subsequent manual review of the extracted and labeled pairs of entities was conducted to validate the results generated by the proposed approach. The results were evaluated using mean reciprocal rank (MRR). Results: Manual evaluation showed that it is feasible to identify alternative medical concepts by using professional or consumer concepts as queries in the word vector space without fine tuning, but the results are more promising in the final fine-tuned word vector space. The MRR values indicated that on an average, a professional or consumer concept is about 14th closest to its counterpart in the word vector space without fine tuning, and the MRR in the final fine-tuned word vector space is 8. Furthermore, the results demonstrate that our method can collect abbreviations and common typos frequently used by consumers. Conclusions: By integrating a large amount of text information and existing consumer health vocabularies, our method outperformed several baseline ranking methods and is effective for generating a list of candidate terms for human review during consumer health vocabulary development. %M 31124461 %R 10.2196/12704 %U http://medinform.jmir.org/2019/2/e12704/ %U https://doi.org/10.2196/12704 %U http://www.ncbi.nlm.nih.gov/pubmed/31124461 %0 Journal Article %@ 2368-7959 %I JMIR Publications %V 6 %N 5 %P e13498 %T Adverse Childhood Experiences Ontology for Mental Health Surveillance, Research, and Evaluation: Advanced Knowledge Representation and Semantic Web Techniques %A Brenas,Jon Hael %A Shin,Eun Kyong %A Shaban-Nejad,Arash %+ Department of Pediatrics, Oak Ridge National Laboratory Center for Biomedical Informatics, University of Tennessee Health Science Center, 50 N Dunlap Street, R492, Memphis, TN, 38103, United States, 1 901 287 5836, ashabann@uthsc.edu %K ontologies %K mental health surveillance %K adverse childhood experiences %K semantics %K computational psychiatry %D 2019 %7 21.05.2019 %9 Original Paper %J JMIR Ment Health %G English %X Background: Adverse Childhood Experiences (ACEs), a set of negative events and processes that a person might encounter during childhood and adolescence, have been proven to be linked to increased risks of a multitude of negative health outcomes and conditions when children reach adulthood and beyond. Objective: To better understand the relationship between ACEs and their relevant risk factors with associated health outcomes and to eventually design and implement preventive interventions, access to an integrated coherent dataset is needed. Therefore, we implemented a formal ontology as a resource to allow the mental health community to facilitate data integration and knowledge modeling and to improve ACEs’ surveillance and research. Methods: We use advanced knowledge representation and semantic Web tools and techniques to implement the ontology. The current implementation of the ontology is expressed in the description logic ALCRIQ(D), a sublogic of Web Ontology Language (OWL 2). Results: The ACEs Ontology has been implemented and made available to the mental health community and the public via the BioPortal repository. Moreover, multiple use-case scenarios have been introduced to showcase and evaluate the usability of the ontology in action. The ontology was created to be used by major actors in the ACEs community with different applications, from the diagnosis of individuals and predicting potential negative outcomes that they might encounter to the prevention of ACEs in a population and designing interventions and policies. Conclusions: The ACEs Ontology provides a uniform and reusable semantic network and an integrated knowledge structure for mental health practitioners and researchers to improve ACEs’ surveillance and evaluation. %M 31115344 %R 10.2196/13498 %U http://mental.jmir.org/2019/5/e13498/ %U https://doi.org/10.2196/13498 %U http://www.ncbi.nlm.nih.gov/pubmed/31115344 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 2 %P e13590 %T Fast Healthcare Interoperability Resources, Clinical Quality Language, and Systematized Nomenclature of Medicine—Clinical Terms in Representing Clinical Evidence Logic Statements for the Use of Imaging Procedures: Descriptive Study %A Odigie,Eseosa %A Lacson,Ronilda %A Raja,Ali %A Osterbur,David %A Ip,Ivan %A Schneider,Louise %A Khorasani,Ramin %+ Center for Evidence-Based Imaging, Brigham and Women's Hospital, 20 Kent Street, Brookline, MA, 02445, United States, 1 6175259712, rlacson@rics.bwh.harvard.edu %K knowledge representation %K guidelines %K evidence-based medicine %K clinical decision support %D 2019 %7 13.05.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: Evidence-based guidelines and recommendations can be transformed into “If-Then” Clinical Evidence Logic Statements (CELS). Imaging-related CELS were represented in standardized formats in the Harvard Medical School Library of Evidence (HLE). Objective: We aimed to (1) describe the representation of CELS using established Systematized Nomenclature of Medicine—Clinical Terms (SNOMED CT), Clinical Quality Language (CQL), and Fast Healthcare Interoperability Resources (FHIR) standards and (2) assess the limitations of using these standards to represent imaging-related CELS. Methods: This study was exempt from review by the Institutional Review Board as it involved no human subjects. Imaging-related clinical recommendations were extracted from evidence sources and translated into CELS. The clinical terminologies of CELS were represented using SNOMED CT and the condition-action logic was represented in CQL and FHIR. Numbers of fully and partially represented CELS were tallied. Results: A total of 765 CELS were represented in the HLE as of December 2018. We were able to fully represent 137 of 765 (17.9%) CELS using SNOMED CT, CQL, and FHIR. We were able to represent terms using SNOMED CT in the temporal component for action (“Then”) statements in CQL and FHIR in 755 of 765 (98.7%) CELS. Conclusions: CELS were represented as shareable clinical decision support (CDS) knowledge artifacts using existing standards—SNOMED CT, FHIR, and CQL—to promote and accelerate adoption of evidence-based practice. Limitations to standardization persist, which could be minimized with an add-on set of standard terms and value sets and by adding time frames to the CQL framework. %M 31094359 %R 10.2196/13590 %U http://medinform.jmir.org/2019/2/e13590/ %U https://doi.org/10.2196/13590 %U http://www.ncbi.nlm.nih.gov/pubmed/31094359 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 2 %P e12596 %T Identifying Clinical Terms in Medical Text Using Ontology-Guided Machine Learning %A Arbabi,Aryan %A Adams,David R %A Fidler,Sanja %A Brudno,Michael %+ Department of Computer Science, University of Toronto, 10 King's College Road, SF 3303, Toronto, ON, M5S 3G4, Canada, 1 4169782589, brudno@cs.toronto.edu %K concept recognition %K medical text mining %K biomedical ontologies %K machine learning %K phenotyping %K human phenotype ontology %D 2019 %7 10.05.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: Automatic recognition of medical concepts in unstructured text is an important component of many clinical and research applications, and its accuracy has a large impact on electronic health record analysis. The mining of medical concepts is complicated by the broad use of synonyms and nonstandard terms in medical documents. Objective: We present a machine learning model for concept recognition in large unstructured text, which optimizes the use of ontological structures and can identify previously unobserved synonyms for concepts in the ontology. Methods: We present a neural dictionary model that can be used to predict if a phrase is synonymous to a concept in a reference ontology. Our model, called the Neural Concept Recognizer (NCR), uses a convolutional neural network to encode input phrases and then rank medical concepts based on the similarity in that space. It uses the hierarchical structure provided by the biomedical ontology as an implicit prior embedding to better learn embedding of various terms. We trained our model on two biomedical ontologies—the Human Phenotype Ontology (HPO) and Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT). Results: We tested our model trained on HPO by using two different data sets: 288 annotated PubMed abstracts and 39 clinical reports. We achieved 1.7%-3% higher F1-scores than those for our strongest manually engineered rule-based baselines (P=.003). We also tested our model trained on the SNOMED-CT by using 2000 Intensive Care Unit discharge summaries from MIMIC (Multiparameter Intelligent Monitoring in Intensive Care) and achieved 0.9%-1.3% higher F1-scores than those of our baseline. The results of our experiments show high accuracy of our model as well as the value of using the taxonomy structure of the ontology in concept recognition. Conclusion: Most popular medical concept recognizers rely on rule-based models, which cannot generalize well to unseen synonyms. In addition, most machine learning methods typically require large corpora of annotated text that cover all classes of concepts, which can be extremely difficult to obtain for biomedical ontologies. Without relying on large-scale labeled training data or requiring any custom training, our model can be efficiently generalized to new synonyms and performs as well or better than state-of-the-art methods custom built for specific ontologies. %M 31094361 %R 10.2196/12596 %U http://medinform.jmir.org/2019/2/e12596/ %U https://doi.org/10.2196/12596 %U http://www.ncbi.nlm.nih.gov/pubmed/31094361 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 21 %N 4 %P e12776 %T Developing a Physical Activity Ontology to Support the Interoperability of Physical Activity Data %A Kim,Hyeoneui %A Mentzer,Jessica %A Taira,Ricky %+ School of Nursing, Duke University, 307 Trent Drive, Durham, NC, 27710, United States, 1 919 684 7534, hyeoneui.kim@duke.edu %K exercise %K leisure activities %K health information interoperability %K terminology as topic %D 2019 %7 23.04.2019 %9 Original Paper %J J Med Internet Res %G English %X Background: Physical activity data provides important information on disease onset, progression, and treatment outcomes. Although analyzing physical activity data in conjunction with other clinical and microbiological data will lead to new insights crucial for improving human health, it has been hampered partly because of the large variations in the way the data are collected and presented. Objective: The aim of this study was to develop a Physical Activity Ontology (PACO) to support structuring and standardizing heterogeneous descriptions of physical activities. Methods: We prepared a corpus of 1140 unique sentences collected from various physical activity questionnaires and scales as well as existing standardized terminologies and ontologies. We extracted concepts relevant to physical activity from the corpus using a natural language processing toolkit called Multipurpose Text Processing Tool. The target concepts were formalized into an ontology using Protégé (version 4). Evaluation of PACO was performed to ensure logical and structural consistency as well as adherence to the best practice principles of building an ontology. A use case application of PACO was demonstrated by structuring and standardizing 36 exercise habit statements and then automatically classifying them to a defined class of either sufficiently active or insufficiently active using FaCT++, an ontology reasoner available in Protégé. Results: PACO was constructed using 268 unique concepts extracted from the questionnaires and assessment scales. PACO contains 225 classes including 9 defined classes, 20 object properties, 1 data property, and 23 instances (excluding 36 exercise statements). The maximum depth of classes is 4, and the maximum number of siblings is 38. The evaluations with ontology auditing tools confirmed that PACO is structurally and logically consistent and satisfies the majority of the best practice rules of ontology authoring. We showed in a small sample of 36 exercise habit statements that we could formally represent them using PACO concepts and object properties. The formal representation was used to infer a patient activity status category of sufficiently active or insufficiently active using the FaCT++ reasoner. Conclusions: As a first step toward standardizing and structuring heterogeneous descriptions of physical activities for integrative data analyses, PACO was constructed based on the concepts collected from physical activity questionnaires and assessment scales. PACO was evaluated to be structurally consistent and compliant to ontology authoring principles. PACO was also demonstrated to be potentially useful in standardizing heterogeneous physical activity descriptions and classifying them into clinically meaningful categories that reflect adequacy of exercise. %M 31012864 %R 10.2196/12776 %U http://www.jmir.org/2019/4/e12776/ %U https://doi.org/10.2196/12776 %U http://www.ncbi.nlm.nih.gov/pubmed/31012864 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 21 %N 3 %P e13249 %T Genomic Common Data Model for Seamless Interoperation of Biomedical Data in Clinical Practice: Retrospective Study %A Shin,Seo Jeong %A You,Seng Chan %A Park,Yu Rang %A Roh,Jin %A Kim,Jang-Hee %A Haam,Seokjin %A Reich,Christian G %A Blacketer,Clair %A Son,Dae-Soon %A Oh,Seungbin %A Park,Rae Woong %+ Department of Biomedical Informatics, Ajou University School of Medicine, 206 World cup-ro, Yeongtong-gu, Suwon, 16499, Republic of Korea, 82 312194471, veritas@ajou.ac.kr %K high-throughput nucleotide sequencing %K databases, genetic %K multicenter study %K patient privacy %K data visualization %D 2019 %7 26.03.2019 %9 Original Paper %J J Med Internet Res %G English %X Background: Clinical sequencing data should be shared in order to achieve the sufficient scale and diversity required to provide strong evidence for improving patient care. A distributed research network allows researchers to share this evidence rather than the patient-level data across centers, thereby avoiding privacy issues. The Observational Medical Outcomes Partnership (OMOP) common data model (CDM) used in distributed research networks has low coverage of sequencing data and does not reflect the latest trends of precision medicine. Objective: The aim of this study was to develop and evaluate the feasibility of a genomic CDM (G-CDM), as an extension of the OMOP-CDM, for application of genomic data in clinical practice. Methods: Existing genomic data models and sequencing reports were reviewed to extend the OMOP-CDM to cover genomic data. The Human Genome Organisation Gene Nomenclature Committee and Human Genome Variation Society nomenclature were adopted to standardize the terminology in the model. Sequencing data of 114 and 1060 patients with lung cancer were obtained from the Ajou University School of Medicine database of Ajou University Hospital and The Cancer Genome Atlas, respectively, which were transformed to a format appropriate for the G-CDM. The data were compared with respect to gene name, variant type, and actionable mutations. Results: The G-CDM was extended into four tables linked to tables of the OMOP-CDM. Upon comparison with The Cancer Genome Atlas data, a clinically actionable mutation, p.Leu858Arg, in the EGFR gene was 6.64 times more frequent in the Ajou University School of Medicine database, while the p.Gly12Xaa mutation in the KRAS gene was 2.02 times more frequent in The Cancer Genome Atlas dataset. The data-exploring tool GeneProfiler was further developed to conduct descriptive analyses automatically using the G-CDM, which provides the proportions of genes, variant types, and actionable mutations. GeneProfiler also allows for querying the specific gene name and Human Genome Variation Society nomenclature to calculate the proportion of patients with a given mutation. Conclusions: We developed the G-CDM for effective integration of genomic data with standardized clinical data, allowing for data sharing across institutes. The feasibility of the G-CDM was validated by assessing the differences in data characteristics between two different genomic databases through the proposed data-exploring tool GeneProfiler. The G-CDM may facilitate analyses of interoperating clinical and genomic datasets across multiple institutions, minimizing privacy issues and enabling researchers to better understand the characteristics of patients and promote personalized medicine in clinical practice. %M 30912749 %R 10.2196/13249 %U http://www.jmir.org/2019/3/e13249/ %U https://doi.org/10.2196/13249 %U http://www.ncbi.nlm.nih.gov/pubmed/30912749 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 7 %N 1 %P e11487 %T SNOMED CT Concept Hierarchies for Computable Clinical Phenotypes From Electronic Health Record Data: Comparison of Intensional Versus Extensional Value Sets %A Chu,Ling %A Kannan,Vaishnavi %A Basit,Mujeeb A %A Schaeflein,Diane J %A Ortuzar,Adolfo R %A Glorioso,Jimmie F %A Buchanan,Joel R %A Willett,Duwayne L %+ University of Texas Southwestern Medical Center, 5323 Harry Hines Boulevard, Dallas, TX, 75390, United States, 1 214 648 1303, Ling.Chu@UTSouthwestern.edu %K SNOMED CT %K value sets %K clinical phenotypes %K population health %K pragmatic clinical study %D 2019 %7 16.01.2019 %9 Original Paper %J JMIR Med Inform %G English %X Background: Defining clinical phenotypes from electronic health record (EHR)–derived data proves crucial for clinical decision support, population health endeavors, and translational research. EHR diagnoses now commonly draw from a finely grained clinical terminology—either native SNOMED CT or a vendor-supplied terminology mapped to SNOMED CT concepts as the standard for EHR interoperability. Accordingly, electronic clinical quality measures (eCQMs) increasingly define clinical phenotypes with SNOMED CT value sets. The work of creating and maintaining list-based value sets proves daunting, as does insuring that their contents accurately represent the clinically intended condition. Objective: The goal of the research was to compare an intensional (concept hierarchy-based) versus extensional (list-based) value set approach to defining clinical phenotypes using SNOMED CT–encoded data from EHRs by evaluating value set conciseness, time to create, and completeness. Methods: Starting from published Centers for Medicare and Medicaid Services (CMS) high-priority eCQMs, we selected 10 clinical conditions referenced by those eCQMs. For each, the published SNOMED CT list-based (extensional) value set was downloaded from the Value Set Authority Center (VSAC). Ten corresponding SNOMED CT hierarchy-based intensional value sets for the same conditions were identified within our EHR. From each hierarchy-based intensional value set, an exactly equivalent full extensional value set was derived enumerating all included descendant SNOMED CT concepts. Comparisons were then made between (1) VSAC-downloaded list-based (extensional) value sets, (2) corresponding hierarchy-based intensional value sets for the same conditions, and (3) derived list-based (extensional) value sets exactly equivalent to the hierarchy-based intensional value sets. Value set conciseness was assessed by the number of SNOMED CT concepts needed for definition. Time to construct the value sets for local use was measured. Value set completeness was assessed by comparing contents of the downloaded extensional versus intensional value sets. Two measures of content completeness were made: for individual SNOMED CT concepts and for the mapped diagnosis clinical terms available for selection within the EHR by clinicians. Results: The 10 hierarchy-based intensional value sets proved far simpler and faster to construct than exactly equivalent derived extensional value set lists, requiring a median 3 versus 78 concepts to define and 5 versus 37 minutes to build. The hierarchy-based intensional value sets also proved more complete: in comparison, the 10 downloaded 2018 extensional value sets contained a median of just 35% of the intensional value sets’ SNOMED CT concepts and 65% of mapped EHR clinical terms. Conclusions: In the EHR era, defining conditions preferentially should employ SNOMED CT concept hierarchy-based (intensional) value sets rather than extensional lists. By doing so, clinical guideline and eCQM authors can more readily engage specialists in vetting condition subtypes to include and exclude, and streamline broad EHR implementation of condition-specific decision support promoting guideline adherence for patient benefit. %M 30664458 %R 10.2196/11487 %U http://medinform.jmir.org/2019/1/e11487/ %U https://doi.org/10.2196/11487 %U http://www.ncbi.nlm.nih.gov/pubmed/30664458 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 6 %N 4 %P e52 %T Identifying Principles for the Construction of an Ontology-Based Knowledge Base: A Case Study Approach %A Jing,Xia %A Hardiker,Nicholas R %A Kay,Stephen %A Gao,Yongsheng %+ Department of Social and Public Health, College of Health Sciences and Professions, Ohio University, Grover Center W357, Athens, OH, 45701, United States, 1 7405930750, xia.xjing@gmail.com %K cystic fibrosis %K knowledge base %K knowledge representation %K molecular genetics information %K ontology %K OntoKBCF %K phenotypes %D 2018 %7 21.12.2018 %9 Original Paper %J JMIR Med Inform %G English %X Background: Ontologies are key enabling technologies for the Semantic Web. The Web Ontology Language (OWL) is a semantic markup language for publishing and sharing ontologies. Objective: The supply of customizable, computable, and formally represented molecular genetics information and health information, via electronic health record (EHR) interfaces, can play a critical role in achieving precision medicine. In this study, we used cystic fibrosis as an example to build an Ontology-based Knowledge Base prototype on Cystic Fibrobis (OntoKBCF) to supply such information via an EHR prototype. In addition, we elaborate on the construction and representation principles, approaches, applications, and representation challenges that we faced in the construction of OntoKBCF. The principles and approaches can be referenced and applied in constructing other ontology-based domain knowledge bases. Methods: First, we defined the scope of OntoKBCF according to possible clinical information needs about cystic fibrosis on both a molecular level and a clinical phenotype level. We then selected the knowledge sources to be represented in OntoKBCF. We utilized top-to-bottom content analysis and bottom-up construction to build OntoKBCF. Protégé-OWL was used to construct OntoKBCF. The construction principles included (1) to use existing basic terms as much as possible; (2) to use intersection and combination in representations; (3) to represent as many different types of facts as possible; and (4) to provide 2-5 examples for each type. HermiT 1.3.8.413 within Protégé-5.1.0 was used to check the consistency of OntoKBCF. Results: OntoKBCF was constructed successfully, with the inclusion of 408 classes, 35 properties, and 113 equivalent classes. OntoKBCF includes both atomic concepts (such as amino acid) and complex concepts (such as “adolescent female cystic fibrosis patient”) and their descriptions. We demonstrated that OntoKBCF could make customizable molecular and health information available automatically and usable via an EHR prototype. The main challenges include the provision of a more comprehensive account of different patient groups as well as the representation of uncertain knowledge, ambiguous concepts, and negative statements and more complicated and detailed molecular mechanisms or pathway information about cystic fibrosis. Conclusions: Although cystic fibrosis is just one example, based on the current structure of OntoKBCF, it should be relatively straightforward to extend the prototype to cover different topics. Moreover, the principles underpinning its development could be reused for building alternative human monogenetic diseases knowledge bases. %M 30578220 %R 10.2196/medinform.9979 %U http://medinform.jmir.org/2018/4/e52/ %U https://doi.org/10.2196/medinform.9979 %U http://www.ncbi.nlm.nih.gov/pubmed/30578220 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 20 %N 10 %P e274 %T Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study %A Varghese,Julian %A Sandmann,Sarah %A Dugas,Martin %+ Institute of Medical Informatics, University of Münster, Albert-Schweitzer-Campus 1, Gebäude A11, Münster, 48149, Germany, 49 2518354714, julian.varghese@ukmuenster.de %K clinical coding %K health information interoperability %K Unified Medical Language System %K eligibility criteria %D 2018 %7 15.10.2018 %9 Original Paper %J J Med Internet Res %G English %X Background: Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System by the National Library of Medicine is the largest clinical terminology system for medical coders and Natural Language Processing tools. However, the abundance of ambiguous codes leads to low rates of uniform coding among different coders. Objective: The objective of our study was to measure uniform coding among different medical experts in terms of interrater reliability and analyze the effect on interrater reliability using an expert- and Web-based code suggestion system. Methods: We conducted a quasi-experimental study in which 6 medical experts coded 602 medical items from structured quality assurance forms or free-text eligibility criteria of 20 different clinical trials. The medical item content was selected on the basis of mortality-leading diseases according to World Health Organization data. The intervention comprised using a semiautomatic code suggestion tool that is linked to a European information infrastructure providing a large medical text corpus of >300,000 medical form items with expert-assigned semantic codes. Krippendorff alpha (Kalpha) with bootstrap analysis was used for the interrater reliability analysis, and coding times were measured before and after the intervention. Results: The intervention improved interrater reliability in structured quality assurance form items (from Kalpha=0.50, 95% CI 0.43-0.57 to Kalpha=0.62 95% CI 0.55-0.69) and free-text eligibility criteria (from Kalpha=0.19, 95% CI 0.14-0.24 to Kalpha=0.43, 95% CI 0.37-0.50) while preserving or slightly reducing the mean coding time per item for all 6 coders. Regardless of the intervention, precoordination and structured items were associated with significantly high interrater reliability, but the proportion of items that were precoordinated significantly increased after intervention (eligibility criteria: OR 4.92, 95% CI 2.78-8.72; quality assurance: OR 1.96, 95% CI 1.19-3.25). Conclusions: The Web-based code suggestion mechanism improved interrater reliability toward moderate or even substantial intercoder agreement. Precoordination and the use of structured versus free-text data elements are key drivers of higher interrater reliability. %M 30322834 %R 10.2196/jmir.9644 %U http://www.jmir.org/2018/10/e274/ %U https://doi.org/10.2196/jmir.9644 %U http://www.ncbi.nlm.nih.gov/pubmed/30322834 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 6 %N 3 %P e42 %T Validating a Framework for Coding Patient-Reported Health Information to the Medical Dictionary for Regulatory Activities Terminology: An Evaluative Study %A Brajovic,Sonja %A Blaser,David A %A Zisk,Meaghan %A Caligtan,Christine %A Okun,Sally %A Hall,Marni %A Pamer,Carol A %+ PatientsLikeMe, 160 Second Street, Cambridge, MA, 02142, United States, 1 781 718 6382, dblaser@patientslikeme.com %K adverse drug events %K Food and Drug Administration %K MedDRA %K patient-generated health data %K PatientsLikeMe %K vocabulary, controlled %K data curation %D 2018 %7 21.08.2018 %9 Original Paper %J JMIR Med Inform %G English %X Background: The availability of and interest in patient-generated health data (PGHD) have grown steadily. Patients describe medical experiences differently compared with how clinicians or researchers would describe their observations of those same experiences. Patients may find nonserious, known adverse drug events (ADEs) to be an ongoing concern, which impacts the tolerability and adherence. Clinicians must be vigilant for medically serious, potentially fatal ADEs. Having both perspectives provides patients and clinicians with a complete picture of what to expect from drug therapies. Multiple initiatives seek to incorporate patients’ perspectives into drug development, including PGHD exploration for pharmacovigilance. The Food and Drug Administration (FDA) Adverse Event Reporting System contains case reports of postmarketing ADEs. To facilitate the analysis of these case reports, case details are coded using the Medical Dictionary for Regulatory Activities (MedDRA). PatientsLikeMe is a Web-based network where patients report, track, share, and discuss their health information. PatientsLikeMe captures PGHD through free-text and structured data fields. PatientsLikeMe structured data are coded to multiple medical terminologies, including MedDRA. The standardization of PatientsLikeMe PGHD enables electronic accessibility and enhances patient engagement. Objective: The aim of this study is to retrospectively review PGHD for symptoms and ADEs entered by patients on PatientsLikeMe and coded by PatientsLikeMe to MedDRA terminology for concordance with regulatory-focused coding practices. Methods: An FDA MedDRA coding expert retrospectively reviewed a data file containing verbatim patient-reported symptoms and ADEs and PatientsLikeMe-assigned MedDRA terms to determine the medical accuracy and appropriateness of the selected MedDRA terms, applying the International Council for Harmonisation MedDRA Term Selection: Points to Consider (MTS:PTC) guides. Results: The FDA MedDRA coding expert reviewed 3234 PatientsLikeMe-assigned MedDRA codes and patient-reported verbatim text. The FDA and PatientsLikeMe were concordant at 97.09% (3140/3234) of the PatientsLikeMe-assigned MedDRA codes. The 2.91% (94/3234) discordant subset was analyzed to identify reasons for differences. Coding differences were attributed to several reasons but mostly driven by PatientsLikeMe’s approach of assigning a more general MedDRA term to enable patient-to-patient engagement, while the FDA assigned a more specific medically relevant term. Conclusions: PatientsLikeMe MedDRA coding of PGHD was generally comparable to how the FDA would code similar data, applying the MTS:PTC principles. Discordant coding resulted from several reasons but mostly reflected a difference in purpose. The MTS:PTC coding principles aim to capture the most specific reported information about an ADE, whereas PatientsLikeMe may code patient-reported symptoms and ADEs to more general MedDRA terms to support patient engagement among a larger group of patients. This study demonstrates that most verbatim reports of symptoms and ADEs collected by a PGHD source, such as the PatientsLikeMe platform, could be reliably coded to MedDRA terminology by applying the MTS:PTC guide. Regarding all secondary use of novel data, understanding coding and standardization principles applied to these data types are important. %M 30131314 %R 10.2196/medinform.9878 %U http://medinform.jmir.org/2018/3/e42/ %U https://doi.org/10.2196/medinform.9878 %U http://www.ncbi.nlm.nih.gov/pubmed/30131314 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 20 %N 8 %P e254 %T Developing Embedded Taxonomy and Mining Patients’ Interests From Web-Based Physician Reviews: Mixed-Methods Approach %A Li,Jia %A Liu,Minghui %A Li,Xiaojun %A Liu,Xuan %A Liu,Jingfang %+ School of Management, Shanghai University, 99 Shangda Road, Shanghai, 200444, China, 86 21 66133790 ext 803, jingfangliu2014@hotmail.com %K labeled-LDA %K physicians %K topic modeling %K topic taxonomy %K Web-based review %D 2018 %7 16.08.2018 %9 Original Paper %J J Med Internet Res %G English %X Background: Web-based physician reviews are invaluable gold mines that merit further investigation. Although many studies have explored the text information of physician reviews, very few have focused on developing a systematic topic taxonomy embedded in physician reviews. The first step toward mining physician reviews is to determine how the natural structure or dimensions is embedded in reviews. Therefore, it is relevant to develop the topic taxonomy rigorously and systematically. Objective: This study aims to develop a hierarchical topic taxonomy to uncover the latent structure of physician reviews and illustrate its application for mining patients’ interests based on the proposed taxonomy and algorithm. Methods: Data comprised 122,716 physician reviews, including reviews of 8501 doctors from a leading physician review website in China (haodf.com), collected between 2007 and 2015. Mixed methods, including a literature review, data-driven-based topic discovery, and human annotation were used to develop the physician review topic taxonomy. Results: The identified taxonomy included 3 domains or high-level categories and 9 subtopics or low-level categories. The physician-related domain included the categories of medical ethics, medical competence, communication skills, medical advice, and prescriptions. The patient-related domain included the categories of the patient profile, symptoms, diagnosis, and pathogenesis. The system-related domain included the categories of financing and operation process. The F-measure of the proposed classification algorithm reached 0.816 on average. Symptoms (Cohen d=1.58, Δu=0.216, t=229.75, and P<.001) are more often mentioned by patients with acute diseases, whereas communication skills (Cohen d=−0.29, Δu=−0.038, t=−42.01, and P<.001), financing (Cohen d=−0.68, Δu=−0.098, t=−99.26, and P<.001), and diagnosis and pathogenesis (Cohen d=−0.55, Δu=−0.078, t=−80.09, and P<.001) are more often mentioned by patients with chronic diseases. Patients with mild diseases were more interested in medical ethics (Cohen d=0.25, Δu 0.039, t=8.33, and P<.001), operation process (Cohen d=0.57, Δu 0.060, t=18.75, and P<.001), patient profile (Cohen d=1.19, Δu 0.132, t=39.33, and P<.001), and symptoms (Cohen d=1.91, Δu=0.274, t=62.82, and P<.001). Meanwhile, patients with serious diseases were more interested in medical competence (Cohen d=−0.99, Δu=−0.165, t=−32.58, and P<.001), medical advice and prescription (Cohen d=−0.65, Δu=−0.082, t=−21.45, and P<.001), financing (Cohen d=−0.26, Δu=−0.018, t=−8.45, and P<.001), and diagnosis and pathogenesis (Cohen d=−1.55, Δu=−0.229, t=−50.93, and P<.001). Conclusions: This mixed-methods approach, integrating literature reviews, data-driven topic discovery, and human annotation, is an effective and rigorous way to develop a physician review topic taxonomy. The proposed algorithm based on Labeled-Latent Dirichlet Allocation can achieve impressive classification results for mining patients’ interests. Furthermore, the mining results reveal marked differences in patients’ interests across different disease types, socioeconomic development levels, and hospital levels. %M 30115610 %R 10.2196/jmir.8868 %U http://www.jmir.org/2018/8/e254/ %U https://doi.org/10.2196/jmir.8868 %U http://www.ncbi.nlm.nih.gov/pubmed/30115610 %0 Journal Article %@ 2369-2960 %I JMIR Publications %V 4 %N 2 %P e10218 %T A Surveillance Infrastructure for Malaria Analytics: Provisioning Data Access and Preservation of Interoperability %A Al Manir,Mohammad Sadnan %A Brenas,Jon Haël %A Baker,Christopher JO %A Shaban-Nejad,Arash %+ Oak Ridge National Laboratory Center for for Biomedical Informatics, Department of Pediatrics, The University of Tennessee Health Science Center, 50 N Dunlap Street, R492, Memphis, TN, 38103, United States, 1 901 287 5836, ashabann@uthsc.edu %K malaria surveillance %K global health %K interoperability %K change management %K Web services %K population health intelligence %D 2018 %7 15.06.2018 %9 Original Paper %J JMIR Public Health Surveill %G English %X Background: According to the World Health Organization, malaria surveillance is weakest in countries and regions with the highest malaria burden. A core obstacle is that the data required to perform malaria surveillance are fragmented in multiple data silos distributed across geographic regions. Furthermore, consistent integrated malaria data sources are few, and a low degree of interoperability exists between them. As a result, it is difficult to identify disease trends and to plan for effective interventions. Objective: We propose the Semantics, Interoperability, and Evolution for Malaria Analytics (SIEMA) platform for use in malaria surveillance based on semantic data federation. Using this approach, it is possible to access distributed data, extend and preserve interoperability between multiple dynamic distributed malaria sources, and facilitate detection of system changes that can interrupt mission-critical global surveillance activities. Methods: We used Semantic Automated Discovery and Integration (SADI) Semantic Web Services to enable data access and improve interoperability, and the graphical user interface-enabled semantic query engine HYDRA to implement the target queries typical of malaria programs. We implemented a custom algorithm to detect changes to community-developed terminologies, data sources, and services that are core to SIEMA. This algorithm reports to a dashboard. Valet SADI is used to mitigate the impact of changes by rebuilding affected services. Results: We developed a prototype surveillance and change management platform from a combination of third-party tools, community-developed terminologies, and custom algorithms. We illustrated a methodology and core infrastructure to facilitate interoperable access to distributed data sources using SADI Semantic Web services. This degree of access makes it possible to implement complex queries needed by our user community with minimal technical skill. We implemented a dashboard that reports on terminology changes that can render the services inactive, jeopardizing system interoperability. Using this information, end users can control and reactively rebuild services to preserve interoperability and minimize service downtime. Conclusions: We introduce a framework suitable for use in malaria surveillance that supports the creation of flexible surveillance queries across distributed data resources. The platform provides interoperable access to target data sources, is domain agnostic, and with updates to core terminological resources is readily transferable to other surveillance activities. A dashboard enables users to review changes to the infrastructure and invoke system updates. The platform significantly extends the range of functionalities offered by malaria information systems, beyond the state-of-the-art. %M 29907554 %R 10.2196/10218 %U http://publichealth.jmir.org/2018/2/e10218/ %U https://doi.org/10.2196/10218 %U http://www.ncbi.nlm.nih.gov/pubmed/29907554 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 6 %N 2 %P e37 %T Development and Validation of a Functional Behavioural Assessment Ontology to Support Behavioural Health Interventions %A Merlo,Gianluca %A Chiazzese,Giuseppe %A Taibi,Davide %A Chifari,Antonella %+ Istituto per le Tecnologie Didattiche, Consiglio Nazionale delle Ricerche, Via Ugo La Malfa 153, Palermo, 90146, Italy, 39 0916809207, gianluca.merlo@itd.cnr.it %K ontology %K behavioral interventions %K functional behavioral assessment %K eHealth care %K evidence-based practice %D 2018 %7 31.05.2018 %9 Original Paper %J JMIR Med Inform %G English %X Background: In the cognitive-behavioral approach, Functional Behavioural Assessment is one of the most effective methods to identify the variables that determine a problem behavior. In this context, the use of modern technologies can encourage the collection and sharing of behavioral patterns, effective intervention strategies, and statistical evidence about antecedents and consequences of clusters of problem behaviors, encouraging the designing of function-based interventions. Objective: The paper describes the development and validation process used to design a specific Functional Behavioural Assessment Ontology (FBA-Ontology). The FBA-Ontology is a semantic representation of the variables that intervene in a behavioral observation process, facilitating the systematic collection of behavioral data, the consequential planning of treatment strategies and, indirectly, the scientific advancement in this field of study. Methods: The ontology has been developed deducing concepts and relationships of the ontology from a gold standard and then performing a machine-based validation and a human-based assessment to validate the Functional Behavioural Assessment Ontology. These validation and verification processes were aimed to verify how much the ontology is conceptually well founded and semantically and syntactically correct. Results: The Pellet reasoner checked the logical consistency and the integrity of classes and properties defined in the ontology, not detecting any violation of constraints in the ontology definition. To assess whether the ontology definition is coherent with the knowledge domain, human evaluation of the ontology was performed asking 84 people to fill in a questionnaire composed by 13 questions assessing concepts, relations between concepts, and concepts’ attributes. The response rate for the survey was 29/84 (34.52%). The domain experts confirmed that the concepts, the attributes, and the relationships between concepts defined in the FBA-Ontology are valid and well represent the Functional Behavioural Assessment process. Conclusions: The new ontology developed could be a useful tool to design new evidence-based systems in the Behavioral Interventions practices, encouraging the link with other Linked Open Data datasets and repositories to provide users with new models of eHealth focused on the management of problem behaviors. Therefore, new research is needed to develop and implement innovative strategies to improve the poor reproducibility and translatability of basic research findings in the field of behavioral assessment. %R 10.2196/medinform.7799 %U http://medinform.jmir.org/2018/2/e37/ %U https://doi.org/10.2196/medinform.7799 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 10 %N 1 %P e8359 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2018 %7 ..2018 %9 %J Online J Public Health Inform %G English %X Objective:Previous research identified data gaps between traditional paper-based STI notifiable condition reporting and pilot electronic initial case reporting (eICR) relying on Continuity of Care Documents (CCDs) exported from our clinical partner’s electronic health record (EHR) software.1 Structured data capture is needed for automatic processing of eICR data imported into public health repositories and surveillance systems, similar to electronic laboratory reporting (ELR). Coding data gaps (between paper and electronic case reports) using standardized vocabularies will allow integration of additional questions into EHR or other data collection systems and may allow creation of standard Clinical Data Architecture (CDA) templates, Logical Observation Identifiers Names and Codes (LOINC) panels, or Fast Healthcare Interoperability Resources (FHIR) resources. Furthermore, identifying data gaps can inform improvements to other standards including nationwide standardization efforts for notifiable conditions.Introduction:Under the CDC STD Surveillance Network (SSuN) Part B grant, WA DOH is testing eICR of sexually transmitted infections (STI) with a clinical partner. Existing standard vocabulary codes were identified to represent previously-identified information gaps, or the need for new codes or concepts was identified.Methods:In prior work, CCDs were securely received from our clinical partner and then analyzed for gaps compared to the existing paper-based STI case reporting form.1 Now, codes associated with gap concepts were identified in standard vocabularies such as LOINC and SNOMED CT. Standards were searched using online browsers offered by the standards development organizations (SDO) to identify potential codes, which were reviewed with public health epidemiologists and clinicians. Gaps were listed, and wording and intent was compared to standard codes including accessory information found in the SDO browsers and a final table of recommended codes was produced.Results:Acceptable congruity between currently used case reporting questions and coded vocabularies was found for the majority of data gaps previously identified. Where data need was incongruous with standard coded vocabularies, new codes or concepts could be proposed to the SDO.Pregnancy status is often missing from CDA documents but is well-conceptualized in both LOINC and SNOMED CT systems under several codes, including any current laboratory tests for pregnancy. HIV status is similarly well-conceptualized in both LOINC and SNOMED CT both as a problem list item well as thru a variety of laboratory tests. However, problem list EHR models lack standard inclusion of date of last HIV test or dates of current pregnancy as an associated coded data element which is desirable for public health.Information about the case patient’s sexual partners, need for partner STI treatment, and partner treatment completed is lacking in standard CCD documents.1 Number of current sexual partners [requiring treatment] – has a SNOMED-CT code, but lacks a match in LOINC coding system. This gap identified the need for exploration of how information about sexual partners can best be represented in interoperability artifacts, including the most useful division of information content between the information model and the standard terminology. Needed concept could resemble ‘record target’ found in HL7 version 3.0 and would allow data to be provided without specifying additional codes. Exploration of information model options or new codes is recommended.Many codes are possible for site of infection and specific symptoms but the overall concepts of ‘symptomatic infection’ and ‘site of infection’ as coded elements would need to be added.Conclusions:This coded element gap analysis found that most information requested in an STI case report can be found in a CCD. Gaps can be addressed by using existing standard terms except for concepts about sexual partners that might be better addressed by exploring the information model rather than through the addition of standard codes. Standards use will facilitate complete case reporting using CDA or FIHR, for example, within a ‘blue button’ or other system with functionality for exchanging additional information about notifiable condition case patients. Collaboration with clinicians, public health practitioners, informaticians, and EHR vendors in will help determine how these concepts might best be modeled. Understanding data gaps involves working closely with a broad range of stakeholders, to understand why gaps exist and how well proposed solutions will meet stakeholder needs. %R 10.5210/ojphi.v10i1.8359 %U %U https://doi.org/10.5210/ojphi.v10i1.8359 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 6 %N 1 %P e7 %T Representation of Time-Relevant Common Data Elements in the Cancer Data Standards Repository: Statistical Evaluation of an Ontological Approach %A Chen,Henry W %A Du,Jingcheng %A Song,Hsing-Yi %A Liu,Xiangyu %A Jiang,Guoqian %A Tao,Cui %+ School of Biomedical Informatics, The University of Texas Health Science Center at Houston, 7000 Fannin, Suite 600 Houston, Texas 77030, Houston, TX,, United States, 1 7135003981, cui.tao@uth.tmc.edu %K common data elements %K database management systems %K database %K time %K biomedical ontology %D 2018 %7 22.02.2018 %9 Original Paper %J JMIR Med Inform %G English %X Background: Today, there is an increasing need to centralize and standardize electronic health data within clinical research as the volume of data continues to balloon. Domain-specific common data elements (CDEs) are emerging as a standard approach to clinical research data capturing and reporting. Recent efforts to standardize clinical study CDEs have been of great benefit in facilitating data integration and data sharing. The importance of the temporal dimension of clinical research studies has been well recognized; however, very few studies have focused on the formal representation of temporal constraints and temporal relationships within clinical research data in the biomedical research community. In particular, temporal information can be extremely powerful to enable high-quality cancer research. Objective: The objective of the study was to develop and evaluate an ontological approach to represent the temporal aspects of cancer study CDEs. Methods: We used CDEs recorded in the National Cancer Institute (NCI) Cancer Data Standards Repository (caDSR) and created a CDE parser to extract time-relevant CDEs from the caDSR. Using the Web Ontology Language (OWL)–based Time Event Ontology (TEO), we manually derived representative patterns to semantically model the temporal components of the CDEs using an observing set of randomly selected time-related CDEs (n=600) to create a set of TEO ontological representation patterns. In evaluating TEO’s ability to represent the temporal components of the CDEs, this set of representation patterns was tested against two test sets of randomly selected time-related CDEs (n=425). Results: It was found that 94.2% (801/850) of the CDEs in the test sets could be represented by the TEO representation patterns. Conclusions: In conclusion, TEO is a good ontological model for representing the temporal components of the CDEs recorded in caDSR. Our representative model can harness the Semantic Web reasoning and inferencing functionalities and present a means for temporal CDEs to be machine-readable, streamlining meaningful searches. %M 29472179 %R 10.2196/medinform.8175 %U http://medinform.jmir.org/2018/1/e7/ %U https://doi.org/10.2196/medinform.8175 %U http://www.ncbi.nlm.nih.gov/pubmed/29472179 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 20 %N 1 %P e26 %T A Natural Language Processing System That Links Medical Terms in Electronic Health Record Notes to Lay Definitions: System Development Using Physician Reviews %A Chen,Jinying %A Druhl,Emily %A Polepalli Ramesh,Balaji %A Houston,Thomas K %A Brandt,Cynthia A %A Zulman,Donna M %A Vimalananda,Varsha G %A Malkani,Samir %A Yu,Hong %+ Department of Quantitative Health Sciences, University of Massachusetts Medical School, 368 Plantation Street, Worcester, MA, 01605, United States, 1 774 455 3527, jinying.chen@umassmed.edu %K electronic health records %K natural language processing %K consumer health informatics %K usability testing %K computer software %D 2018 %7 22.01.2018 %9 Original Paper %J J Med Internet Res %G English %X Background: Many health care systems now allow patients to access their electronic health record (EHR) notes online through patient portals. Medical jargon in EHR notes can confuse patients, which may interfere with potential benefits of patient access to EHR notes. Objective: The aim of this study was to develop and evaluate the usability and content quality of NoteAid, a Web-based natural language processing system that links medical terms in EHR notes to lay definitions, that is, definitions easily understood by lay people. Methods: NoteAid incorporates two core components: CoDeMed, a lexical resource of lay definitions for medical terms, and MedLink, a computational unit that links medical terms to lay definitions. We developed innovative computational methods, including an adapted distant supervision algorithm to prioritize medical terms important for EHR comprehension to facilitate the effort of building CoDeMed. Ten physician domain experts evaluated the user interface and content quality of NoteAid. The evaluation protocol included a cognitive walkthrough session and a postsession questionnaire. Physician feedback sessions were audio-recorded. We used standard content analysis methods to analyze qualitative data from these sessions. Results: Physician feedback was mixed. Positive feedback on NoteAid included (1) Easy to use, (2) Good visual display, (3) Satisfactory system speed, and (4) Adequate lay definitions. Opportunities for improvement arising from evaluation sessions and feedback included (1) improving the display of definitions for partially matched terms, (2) including more medical terms in CoDeMed, (3) improving the handling of terms whose definitions vary depending on different contexts, and (4) standardizing the scope of definitions for medicines. On the basis of these results, we have improved NoteAid’s user interface and a number of definitions, and added 4502 more definitions in CoDeMed. Conclusions: Physician evaluation yielded useful feedback for content validation and refinement of this innovative tool that has the potential to improve patient EHR comprehension and experience using patient portals. Future ongoing work will develop algorithms to handle ambiguous medical terms and test and evaluate NoteAid with patients. %M 29358159 %R 10.2196/jmir.8669 %U http://www.jmir.org/2018/1/e26/ %U https://doi.org/10.2196/jmir.8669 %U http://www.ncbi.nlm.nih.gov/pubmed/29358159 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 19 %N 12 %P e417 %T Text Simplification Using Consumer Health Vocabulary to Generate Patient-Centered Radiology Reporting: Translation and Evaluation %A Qenam,Basel %A Kim,Tae Youn %A Carroll,Mark J %A Hogarth,Michael %+ Department of Radiological Sciences, College of Applied Medical Sciences, King Saud University, P.O. Box 10219, Riyadh, 11432, Saudi Arabia, 966 114693567, bqenam@ksu.edu.sa %K consumer health information %K vocabulary %K radiology %K electronic health records %K natural language processing %D 2017 %7 18.12.2017 %9 Original Paper %J J Med Internet Res %G English %X Background: Radiology reporting is a clinically oriented form of documentation that reflects critical information for patients about their health care processes. Realizing its importance, many medical institutions have started providing radiology reports in patient portals. The gain, however, can be limited because of medical language barriers, which require a way for customizing these reports for patients. The open-access, collaborative consumer health vocabulary (CHV) is a terminology system created for such purposes and can be the basis of lexical simplification processes for clinical notes. Objective: The aim of this study was to examine the comprehensibility and suitability of CHV in simplifying radiology reports for consumers. This was done by characterizing the content coverage and the lexical similarity between the terms in the reports and the CHV-preferred terms. Methods: The overall procedure was divided into the following two main stages: (1) translation and (2) evaluation. The translation process involved using MetaMap to link terms in the reports to CHV concepts. This is followed by replacing the terms with CHV-preferred terms using the concept names and sources table (MRCONSO) in the Unified Medical Language System (UMLS) Metathesaurus. In the second stage, medical terms in the reports and general terms that are used to describe medical phenomena were selected and evaluated by comparing the words in the original reports with the translated ones. The evaluation includes measuring the content coverage, investigating lexical similarity, and finding trends in missing concepts. Results: Of the 792 terms selected from the radiology reports, 695 of them could be mapped directly to CHV concepts, indicating a content coverage of 88.5%. A total of 51 of the concepts (53%, 51/97) that could not be mapped are names of human anatomical structures and regions, followed by 28 anatomical descriptions and pathological variations (29%, 28/97). In addition, 12 radiology techniques and projections represented 12% of the unmapped concepts, whereas the remaining six concepts (6%, 12/97) were physiological descriptions. The rate of lexical similarity between the CHV-preferred terms and the terms in the radiology reports was approximately 72.6%. Conclusions: The CHV covered a high percentage of concepts found in the radiology reports, but unmapped concepts are associated with areas that are commonly found in radiology reporting. CHV terms also showed a high percentage of lexical similarity with terms in the reports, which contain a myriad of medical jargon. This suggests that many CHV terms might not be suitable for lay consumers who would not be facile with radiology-specific vocabulary. Therefore, further patient-centered content changes are needed of the CHV to increase its usefulness and facilitate its integration into consumer-oriented applications. %M 29254915 %R 10.2196/jmir.8536 %U http://www.jmir.org/2017/12/e417/ %U https://doi.org/10.2196/jmir.8536 %U http://www.ncbi.nlm.nih.gov/pubmed/29254915 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 5 %N 4 %P e49 %T Standard Anatomic Terminologies: Comparison for Use in a Health Information Exchange–Based Prior Computed Tomography (CT) Alerting System %A Beitia,Anton Oscar %A Lowry,Tina %A Vreeman,Daniel J %A Loo,George T %A Delman,Bradley N %A Thum,Frederick L %A Slovis,Benjamin H %A Shapiro,Jason S %+ Department of Emergency Medicine, Icahn School of Medicine at Mount Sinai, Box 1620,, One Gustave L. Levy Pl, New York, NY, 10029, United States, 1 2128248047, anton.beitia@mountsinai.org %K tomography, x-ray computed %K health information exchange %K radiation dosage %K terminology %K anatomy, regional %D 2017 %7 14.12.2017 %9 Original Paper %J JMIR Med Inform %G English %X Background: A health information exchange (HIE)–based prior computed tomography (CT) alerting system may reduce avoidable CT imaging by notifying ordering clinicians of prior relevant studies when a study is ordered. For maximal effectiveness, a system would alert not only for prior same CTs (exams mapped to the same code from an exam name terminology) but also for similar CTs (exams mapped to different exam name terminology codes but in the same anatomic region) and anatomically proximate CTs (exams in adjacent anatomic regions). Notification of previous same studies across an HIE requires mapping of local site CT codes to a standard terminology for exam names (such as Logical Observation Identifiers Names and Codes [LOINC]) to show that two studies with different local codes and descriptions are equivalent. Notifying of prior similar or proximate CTs requires an additional mapping of exam codes to anatomic regions, ideally coded by an anatomic terminology. Several anatomic terminologies exist, but no prior studies have evaluated how well they would support an alerting use case. Objective: The aim of this study was to evaluate the fitness of five existing standard anatomic terminologies to support similar or proximate alerts of an HIE-based prior CT alerting system. Methods: We compared five standard anatomic terminologies (Foundational Model of Anatomy, Systematized Nomenclature of Medicine Clinical Terms, RadLex, LOINC, and LOINC/Radiological Society of North America [RSNA] Radiology Playbook) to an anatomic framework created specifically for our use case (Simple ANatomic Ontology for Proximity or Similarity [SANOPS]), to determine whether the existing terminologies could support our use case without modification. On the basis of an assessment of optimal terminology features for our purpose, we developed an ordinal anatomic terminology utility classification. We mapped samples of 100 random and the 100 most frequent LOINC CT codes to anatomic regions in each terminology, assigned utility classes for each mapping, and statistically compared each terminology’s utility class rankings. We also constructed seven hypothetical alerting scenarios to illustrate the terminologies’ differences. Results: Both RadLex and the LOINC/RSNA Radiology Playbook anatomic terminologies ranked significantly better (P<.001) than the other standard terminologies for the 100 most frequent CTs, but no terminology ranked significantly better than any other for 100 random CTs. Hypothetical scenarios illustrated instances where no standard terminology would support appropriate proximate or similar alerts, without modification. Conclusions: LOINC/RSNA Radiology Playbook and RadLex’s anatomic terminologies appear well suited to support proximate or similar alerts for commonly ordered CTs, but for less commonly ordered tests, modification of the existing terminologies with concepts and relations from SANOPS would likely be required. Our findings suggest SANOPS may serve as a framework for enhancing anatomic terminologies in support of other similar use cases. %M 29242174 %R 10.2196/medinform.8765 %U http://medinform.jmir.org/2017/4/e49/ %U https://doi.org/10.2196/medinform.8765 %U http://www.ncbi.nlm.nih.gov/pubmed/29242174 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 5 %N 4 %P e42 %T Ranking Medical Terms to Support Expansion of Lay Language Resources for Patient Comprehension of Electronic Health Record Notes: Adapted Distant Supervision Approach %A Chen,Jinying %A Jagannatha,Abhyuday N %A Fodeh,Samah J %A Yu,Hong %+ Department of Quantitative Health Sicences, University of Massachusetts Medical School, 368 Plantation Street, Worcester, MA, 01605, United States, 1 774 455 3527, jinying.chen@umassmed.edu %K electronic health records %K natural language processing %K lexical entry selection %K transfer learning %K information extraction %D 2017 %7 31.10.2017 %9 Original Paper %J JMIR Med Inform %G English %X Background: Medical terms are a major obstacle for patients to comprehend their electronic health record (EHR) notes. Clinical natural language processing (NLP) systems that link EHR terms to lay terms or definitions allow patients to easily access helpful information when reading through their EHR notes, and have shown to improve patient EHR comprehension. However, high-quality lay language resources for EHR terms are very limited in the public domain. Because expanding and curating such a resource is a costly process, it is beneficial and even necessary to identify terms important for patient EHR comprehension first. Objective: We aimed to develop an NLP system, called adapted distant supervision (ADS), to rank candidate terms mined from EHR corpora. We will give EHR terms ranked as high by ADS a higher priority for lay language annotation—that is, creating lay definitions for these terms. Methods: Adapted distant supervision uses distant supervision from consumer health vocabulary and transfer learning to adapt itself to solve the problem of ranking EHR terms in the target domain. We investigated 2 state-of-the-art transfer learning algorithms (ie, feature space augmentation and supervised distant supervision) and designed 5 types of learning features, including distributed word representations learned from large EHR data for ADS. For evaluating ADS, we asked domain experts to annotate 6038 candidate terms as important or nonimportant for EHR comprehension. We then randomly divided these data into the target-domain training data (1000 examples) and the evaluation data (5038 examples). We compared ADS with 2 strong baselines, including standard supervised learning, on the evaluation data. Results: The ADS system using feature space augmentation achieved the best average precision, 0.850, on the evaluation set when using 1000 target-domain training examples. The ADS system using supervised distant supervision achieved the best average precision, 0.819, on the evaluation set when using only 100 target-domain training examples. The 2 ADS systems both performed significantly better than the baseline systems (P<.001 for all measures and all conditions). Using a rich set of learning features contributed to ADS’s performance substantially. Conclusions: ADS can effectively rank terms mined from EHRs. Transfer learning improved ADS’s performance even with a small number of target-domain training examples. EHR terms prioritized by ADS were used to expand a lay language resource that supports patient EHR comprehension. The top 10,000 EHR terms ranked by ADS are available upon request. %M 29089288 %R 10.2196/medinform.8531 %U http://medinform.jmir.org/2017/4/e42/ %U https://doi.org/10.2196/medinform.8531 %U http://www.ncbi.nlm.nih.gov/pubmed/29089288 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 6 %N 10 %P e196 %T Knowledge Management Framework for Emerging Infectious Diseases Preparedness and Response: Design and Development of Public Health Document Ontology %A Zhang,Zhizun %A Gonzalez,Mila C %A Morse,Stephen S %A Venkatasubramanian,Venkat %+ Fu Foundation School of Engineering and Applied Science, Department of Chemical Engineering, Columbia University, 500 W. 120th St., Mudd 801, New York, NY,, United States, 1 212 854 2487, venkat@columbia.edu %K EIDs %K public health %K systems engineering %K knowledge representation %K teleological function %K knowledge management %K ontology %K semantic reasoning %D 2017 %7 11.10.2017 %9 Original Paper %J JMIR Res Protoc %G English %X Background: There are increasing concerns about our preparedness and timely coordinated response across the globe to cope with emerging infectious diseases (EIDs). This poses practical challenges that require exploiting novel knowledge management approaches effectively. Objective: This work aims to develop an ontology-driven knowledge management framework that addresses the existing challenges in sharing and reusing public health knowledge. Methods: We propose a systems engineering-inspired ontology-driven knowledge management approach. It decomposes public health knowledge into concepts and relations and organizes the elements of knowledge based on the teleological functions. Both knowledge and semantic rules are stored in an ontology and retrieved to answer queries regarding EID preparedness and response. Results: A hybrid concept extraction was implemented in this work. The quality of the ontology was evaluated using the formal evaluation method Ontology Quality Evaluation Framework. Conclusions: Our approach is a potentially effective methodology for managing public health knowledge. Accuracy and comprehensiveness of the ontology can be improved as more knowledge is stored. In the future, a survey will be conducted to collect queries from public health practitioners. The reasoning capacity of the ontology will be evaluated using the queries and hypothetical outbreaks. We suggest the importance of developing a knowledge sharing standard like the Gene Ontology for the public health domain. %M 29021130 %R 10.2196/resprot.7904 %U http://www.researchprotocols.org/2017/10/e196/ %U https://doi.org/10.2196/resprot.7904 %U http://www.ncbi.nlm.nih.gov/pubmed/29021130 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 5 %N 3 %P e34 %T An Ontology to Improve Transparency in Case Definition and Increase Case Finding of Infectious Intestinal Disease: Database Study in English General Practice %A de Lusignan,Simon %A Shinneman,Stacy %A Yonova,Ivelina %A van Vlymen,Jeremy %A Elliot,Alex J %A Bolton,Frederick %A Smith,Gillian E %A O'Brien,Sarah %+ Section of Clinical Medicine and Ageing, Department of Clinical and Experimental Medicine, University of Surrey, Leggett Building, 2nd Floor, Guildford, GU2 7XH, United Kingdom, 44 (0)1483 683089, s.lusignan@surrey.ac.uk %K dysentery %K enteritis %K enterobacteriaceae %K enterocolitis %K gastritis %K gastroenteritis %K intestinal diseases %K medical records systems, computerized %K norovirus %K primary health care %D 2017 %7 28.09.2017 %9 Original Paper %J JMIR Med Inform %G English %X Background: Infectious intestinal disease (IID) has considerable health impact; there are 2 billion cases worldwide resulting in 1 million deaths and 78.7 million disability-adjusted life years lost. Reported IID incidence rates vary and this is partly because terms such as “diarrheal disease” and “acute infectious gastroenteritis” are used interchangeably. Ontologies provide a method of transparently comparing case definitions and disease incidence rates. Objective: This study sought to show how differences in case definition in part account for variation in incidence estimates for IID and how an ontological approach provides greater transparency to IID case finding. Methods: We compared three IID case definitions: (1) Royal College of General Practitioners Research and Surveillance Centre (RCGP RSC) definition based on mapping to the Ninth International Classification of Disease (ICD-9), (2) newer ICD-10 definition, and (3) ontological case definition. We calculated incidence rates and examined the contribution of four supporting concepts related to IID: symptoms, investigations, process of care (eg, notification to public health authorities), and therapies. We created a formal ontology using ontology Web language. Results: The ontological approach identified 5712 more cases of IID than the ICD-10 definition and 4482 more than the RCGP RSC definition from an initial cohort of 1,120,490. Weekly incidence using the ontological definition was 17.93/100,000 (95% CI 15.63-20.41), whereas for the ICD-10 definition the rate was 8.13/100,000 (95% CI 6.70-9.87), and for the RSC definition the rate was 10.24/100,000 (95% CI 8.55-12.12). Codes from the four supporting concepts were generally consistent across our three IID case definitions: 37.38% (3905/10,448) (95% CI 36.16-38.5) for the ontological definition, 38.33% (2287/5966) (95% CI 36.79-39.93) for the RSC definition, and 40.82% (1933/4736) (95% CI 39.03-42.66) for the ICD-10 definition. The proportion of laboratory results associated with a positive test result was 19.68% (546/2775). Conclusions: The standard RCGP RSC definition of IID, and its mapping to ICD-10, underestimates disease incidence. The ontological approach identified a larger proportion of new IID cases; the ontology divides contributory elements and enables transparency and comparison of rates. Results illustrate how improved diagnostic coding of IID combined with an ontological approach to case definition would provide a clearer picture of IID in the community, better inform GPs and public health services about circulating disease, and empower them to respond. We need to improve the Pathology Bounded Code List (PBCL) currently used by laboratories to electronically report results. Given advances in stool microbiology testing with a move to nonculture, PCR-based methods, the way microbiology results are reported and coded via PBCL needs to be reviewed and modernized. %M 28958989 %R 10.2196/medinform.7641 %U http://medinform.jmir.org/2017/3/e34/ %U https://doi.org/10.2196/medinform.7641 %U http://www.ncbi.nlm.nih.gov/pubmed/28958989 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 19 %N 9 %P e292 %T The User Knows What to Call It: Incorporating Patient Voice Through User-Contributed Tags on a Participatory Platform About Health Management %A Chen,Annie T %A Carriere,Rachel M %A Kaplan,Samantha Jan %+ Department of Biomedical Informatics and Medical Education, University of Washington School of Medicine, 850 Republican Street, Box 358047, C238, Seattle, WA, 98109, United States, 1 206 221 9218, atchen@uw.edu %K collaborative tagging %K folksonomy %K knowledge organization %K self-management %K body listening %K body awareness %D 2017 %7 07.09.2017 %9 Original Paper %J J Med Internet Res %G English %X Background: Body listening, described as the act of paying attention to the body’s signals and cues, can be an important component of long-term health management. Objective: The aim of this study was to introduce and evaluate the Body Listening Project, an innovative effort to engage the public in the creation of a public resource—to leverage collective wisdom in the health domain. This project involved a website where people could contribute their experiences of and dialogue with others concerning body listening and self-management. This article presents an analysis of the tags contributed, with a focus on the value of these tags for knowledge organization and incorporation into consumer-friendly health information retrieval systems. Methods: First, we performed content analysis of the tags contributed, identifying a set of categories and refining the relational structure of the categories to develop a preliminary classification scheme, the Body Listening and Self-Management Taxonomy. Second, we compared the concepts in the Body Listening and Self-Management Taxonomy with concepts that were automatically identified from an extant health knowledge resource, the Unified Medical Language System (UMLS), to better characterize the information that participants contributed. Third, we employed visualization techniques to explore the concept space of the tags. A correlation matrix, based on the extent to which categories tended to be assigned to the same tags, was used to study the interrelatedness of the taxonomy categories. Then a network visualization was used to investigate structural relationships among the categories in the taxonomy. Results: First, we proposed a taxonomy called the Body Listening and Self-Management Taxonomy, with four meta-level categories: (1) health management strategies, (2) concepts and states, (3) influencers, and (4) health-related information behavior. This taxonomy could inform future efforts to organize knowledge and content of this subject matter. Second, we compared the categories from this taxonomy with the UMLS concepts that were identified. Though the UMLS offers benefits such as speed and breadth of coverage, the Body Listening and Self-Management Taxonomy is more consumer-centric. Third, the correlation matrix and network visualization demonstrated that there are natural areas of ambiguity and semantic relatedness in the meanings of the concepts in the Body Listening and Self-Management Taxonomy. Use of these visualizations can be helpful in practice settings, to help library and information science practitioners understand and resolve potential challenges in classification; in research, to characterize the structure of the conceptual space of health management; and in the development of consumer-centric health information retrieval systems. Conclusions: A participatory platform can be employed to collect data concerning patient experiences of health management, which can in turn be used to develop new health knowledge resources or augment existing ones, as well as be incorporated into consumer-centric health information systems. %M 28882809 %R 10.2196/jmir.7673 %U http://www.jmir.org/2017/9/e292/ %U https://doi.org/10.2196/jmir.7673 %U http://www.ncbi.nlm.nih.gov/pubmed/28882809 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 19 %N 7 %P e259 %T Ontology-Based Approach to Social Data Sentiment Analysis: Detection of Adolescent Depression Signals %A Jung,Hyesil %A Park,Hyeoun-Ae %A Song,Tae-Min %+ College of Nursing, Seoul National University, Daehak-ro 103, Jongno-gu, Seoul, 03080, Republic Of Korea, 82 27408827, hapark@snu.ac.kr %K ontology %K adolescent %K depression %K data mining %K social media data %D 2017 %7 24.07.2017 %9 Original Paper %J J Med Internet Res %G English %X Background: Social networking services (SNSs) contain abundant information about the feelings, thoughts, interests, and patterns of behavior of adolescents that can be obtained by analyzing SNS postings. An ontology that expresses the shared concepts and their relationships in a specific field could be used as a semantic framework for social media data analytics. Objective: The aim of this study was to refine an adolescent depression ontology and terminology as a framework for analyzing social media data and to evaluate description logics between classes and the applicability of this ontology to sentiment analysis. Methods: The domain and scope of the ontology were defined using competency questions. The concepts constituting the ontology and terminology were collected from clinical practice guidelines, the literature, and social media postings on adolescent depression. Class concepts, their hierarchy, and the relationships among class concepts were defined. An internal structure of the ontology was designed using the entity-attribute-value (EAV) triplet data model, and superclasses of the ontology were aligned with the upper ontology. Description logics between classes were evaluated by mapping concepts extracted from the answers to frequently asked questions (FAQs) onto the ontology concepts derived from description logic queries. The applicability of the ontology was validated by examining the representability of 1358 sentiment phrases using the ontology EAV model and conducting sentiment analyses of social media data using ontology class concepts. Results: We developed an adolescent depression ontology that comprised 443 classes and 60 relationships among the classes; the terminology comprised 1682 synonyms of the 443 classes. In the description logics test, no error in relationships between classes was found, and about 89% (55/62) of the concepts cited in the answers to FAQs mapped onto the ontology class. Regarding applicability, the EAV triplet models of the ontology class represented about 91.4% of the sentiment phrases included in the sentiment dictionary. In the sentiment analyses, “academic stresses” and “suicide” contributed negatively to the sentiment of adolescent depression. Conclusions: The ontology and terminology developed in this study provide a semantic foundation for analyzing social media data on adolescent depression. To be useful in social media data analysis, the ontology, especially the terminology, needs to be updated constantly to reflect rapidly changing terms used by adolescents in social media postings. In addition, more attributes and value sets reflecting depression-related sentiments should be added to the ontology. %M 28739560 %R 10.2196/jmir.7452 %U http://www.jmir.org/2017/7/e259/ %U https://doi.org/10.2196/jmir.7452 %U http://www.ncbi.nlm.nih.gov/pubmed/28739560 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 5 %N 2 %P e12 %T Effective Information Extraction Framework for Heterogeneous Clinical Reports Using Online Machine Learning and Controlled Vocabularies %A Zheng,Shuai %A Lu,James J %A Ghasemzadeh,Nima %A Hayek,Salim S %A Quyyumi,Arshed A %A Wang,Fusheng %+ Department of Biomedical Informatics, Stony Brook University, 2211 Computer Science, Stony Brook, NY, 11794-8330, United States, 1 6316327528, fusheng.wang@stonybrook.edu %K information extraction %K natural language processing %K controlled vocabulary %K electronic medical records %D 2017 %7 09.05.2017 %9 Original Paper %J JMIR Med Inform %G English %X Background: Extracting structured data from narrated medical reports is challenged by the complexity of heterogeneous structures and vocabularies and often requires significant manual effort. Traditional machine-based approaches lack the capability to take user feedbacks for improving the extraction algorithm in real time. Objective: Our goal was to provide a generic information extraction framework that can support diverse clinical reports and enables a dynamic interaction between a human and a machine that produces highly accurate results. Methods: A clinical information extraction system IDEAL-X has been built on top of online machine learning. It processes one document at a time, and user interactions are recorded as feedbacks to update the learning model in real time. The updated model is used to predict values for extraction in subsequent documents. Once prediction accuracy reaches a user-acceptable threshold, the remaining documents may be batch processed. A customizable controlled vocabulary may be used to support extraction. Results: Three datasets were used for experiments based on report styles: 100 cardiac catheterization procedure reports, 100 coronary angiographic reports, and 100 integrated reports—each combines history and physical report, discharge summary, outpatient clinic notes, outpatient clinic letter, and inpatient discharge medication report. Data extraction was performed by 3 methods: online machine learning, controlled vocabularies, and a combination of these. The system delivers results with F1 scores greater than 95%. Conclusions: IDEAL-X adopts a unique online machine learning–based approach combined with controlled vocabularies to support data extraction for clinical reports. The system can quickly learn and improve, thus it is highly adaptable. %M 28487265 %R 10.2196/medinform.7235 %U http://medinform.jmir.org/2017/2/e12/ %U https://doi.org/10.2196/medinform.7235 %U http://www.ncbi.nlm.nih.gov/pubmed/28487265 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 9 %N 1 %P e7688 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2017 %7 ..2017 %9 %J Online J Public Health Inform %G English %X IntroductionThe ICD-9 codes for acute respiratory illness (ARI) andpneumonia/influenza (P&I) are commonly used in ARI surveillance;however, few studies evaluate the accuracy of these codes or theimportance of ICD-9 position. We reviewed ICD-9 codes reportedamong patients identified through severe acute respiratory infection(SARI) surveillance to compare medical record documentation withmedical coding and evaluated ICD-9 codes assigned to patients withinfluenza detections.MethodsThe Minnesota Department of Health (MDH) conducted SARIsurveillance at three hospitals. All hospitalized patients withsubmission of a physician-ordered upper respiratory specimens(e.g., sputum, throat or nasopharyngeal swabs) were enrolled.A medical chart review was conducted to identify those meetingSARI criteria, defined as patients admitted to an inpatient ward withnew onset of respiratory symptoms or acute exacerbation of chronicrespiratory conditions. Enrolled patients who did not meet the SARIcriteria were categorized as non-SARI. Residual material from theupper respiratory specimens were submitted to MDH for influenzatesting by RT-PCR. Demographic and clinical data, including upto eight ICD-9 codes, were collected through the medical recordreview. Patients with an ICD-9 code indicating ARI (460 to 466)or P&I (480 to 488) were defined as having an ARI/P&I code. Wecompared the frequency of ARI/P&I codes by SARI clinical criteriaand influenza detection and evaluated the position of the reportedARI/P&I code.ResultsFrom May 2013 through August 2015, we enrolled 5,950patients, of which 4,449 (75%) met SARI criteria and 1501 did not(non-SARI). An ARI/P&I code in any position was found in 61%(2705) of SARI vs. 16% (241) of non-SARI patients (odds ratio [OR]8.1, 95% confidence interval [CI] 7.0-9.4); an ARI/P&I code in thefirst position was found in 40% of SARI vs 7% of non-SARI patients(OR=8.6, 95% CI 7.0-10.5). Among SARI patients with at least oneARI/P&I code, 66% had their first or only ARI/P&I code in the 1stposition, 25% in the 2ndposition, and 6% in the 3rdposition. Foridentification of SARI, sensitivity/specificity was 61%/84% for ARI/P&I codes in any position and 40%/93% for ARI/P&I codes in the 1stposition. Among SARI patients, codes for pneumonia (486) and acutebronchiolitis (466.11, 466.19) were commonly reported. The mostfrequent codes among SARI patients without an ARI/P&I code werefever (780.6), acute respiratory failure (518.81), and asthma (493.92)(Table). Influenza was detected among 8% (351) of SARI patients.An ARI/P&I code in any position was more common in influenza-positive vs. influenza-negative SARI patients (77% vs 59%, OR 2.4,95% CI 1.8-3.1). An ARI/P&I code in the 1stposition was slightlymore common in influenza-positive vs -negative patients though notsignificant (44% vs 40%).ConclusionsAmong patients from whom a respiratory specimen was collected,administrative data identified those meeting SARI with moderatesensitivity and high specificity, and with lower sensitivity but greaterspecificity when limited to the 1stICD-9 position. Pneumonia andacute bronchiolitis ICD-9 codes were frequent ARI/P&I codes amongSARI patients. Further investigation is needed to determine the valueof including additional ICD-9 codes, such as respiratory distress andacute asthma exacerbation, in identifying SARI. %R 10.5210/ojphi.v9i1.7688 %U %U https://doi.org/10.5210/ojphi.v9i1.7688 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 5 %N 1 %P e4 %T Ontology-Driven Search and Triage: Design of a Web-Based Visual Interface for MEDLINE %A Demelo,Jonathan %A Parsons,Paul %A Sedig,Kamran %+ Purdue Polytechnic Institute, Department of Computer Graphics Technology, Purdue University, Knoy Hall, 401 N Grant St, West Lafayette, IN, 47907, United States, 1 765 494 0511, parsonsp@purdue.edu %K MEDLINE %K user-computer interface %K information storage and retrieval %K medical informatics %K PubMed %D 2017 %7 02.02.2017 %9 Original Paper %J JMIR Med Inform %G English %X Background: Diverse users need to search health and medical literature to satisfy open-ended goals such as making evidence-based decisions and updating their knowledge. However, doing so is challenging due to at least two major difficulties: (1) articulating information needs using accurate vocabulary and (2) dealing with large document sets returned from searches. Common search interfaces such as PubMed do not provide adequate support for exploratory search tasks. Objective: Our objective was to improve support for exploratory search tasks by combining two strategies in the design of an interactive visual interface by (1) using a formal ontology to help users build domain-specific knowledge and vocabulary and (2) providing multi-stage triaging support to help mitigate the information overload problem. Methods: We developed a Web-based tool, Ontology-Driven Visual Search and Triage Interface for MEDLINE (OVERT-MED), to test our design ideas. We implemented a custom searchable index of MEDLINE, which comprises approximately 25 million document citations. We chose a popular biomedical ontology, the Human Phenotype Ontology (HPO), to test our solution to the vocabulary problem. We implemented multistage triaging support in OVERT-MED, with the aid of interactive visualization techniques, to help users deal with large document sets returned from searches. Results: Formative evaluation suggests that the design features in OVERT-MED are helpful in addressing the two major difficulties described above. Using a formal ontology seems to help users articulate their information needs with more accurate vocabulary. In addition, multistage triaging combined with interactive visualizations shows promise in mitigating the information overload problem. Conclusions: Our strategies appear to be valuable in addressing the two major problems in exploratory search. Although we tested OVERT-MED with a particular ontology and document collection, we anticipate that our strategies can be transferred successfully to other contexts. %M 28153818 %R 10.2196/medinform.6918 %U http://medinform.jmir.org/2017/1/e4/ %U https://doi.org/10.2196/medinform.6918 %U http://www.ncbi.nlm.nih.gov/pubmed/28153818 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 8 %N 3 %P e7052 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2016 %7 ..2016 %9 %J Online J Public Health Inform %G English %X The health care system in China is facing a multitude of challenges owing to the changing demographics of the country, the evolving economics of health care, and the emerging epidemiology of health as well as diseases. China’s many national health care policies are documented in Chinese text documents. It is necessary to map the policies synoptically, systemically, and systematically to discover their emphases and biases, assess them, and modify them in the future. Using a logically constructed ontology of health care policies based on the common bodies of knowledge as a lens, we map the current policies to reveal their ‘bright’, ‘light’, and ‘blind/blank’ spots. The ontological map will help (a) develop a roadmap for future health care policies in China, and (b) compare and contrast China’s health care policies with other countries’. %M 28210417 %R 10.5210/ojphi.v8i3.7052 %U %U https://doi.org/10.5210/ojphi.v8i3.7052 %U http://www.ncbi.nlm.nih.gov/pubmed/28210417 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 4 %N 2 %P e20 %T Evaluating Health Information Systems Using Ontologies %A Eivazzadeh,Shahryar %A Anderberg,Peter %A Larsson,Tobias C %A Fricker,Samuel A %A Berglund,Johan %+ Department of Health Science, Blekinge Institute of Technology, Blekinge Tekniska Högskola, Karlskrona, 371 79, Sweden, 46 765628829, shahryar.eivazzadeh@bth.se %K health information systems %K ontologies %K evaluation %K technology assessment %K biomedical %D 2016 %7 16.06.2016 %9 Original Paper %J JMIR Med Inform %G English %X Background: There are several frameworks that attempt to address the challenges of evaluation of health information systems by offering models, methods, and guidelines about what to evaluate, how to evaluate, and how to report the evaluation results. Model-based evaluation frameworks usually suggest universally applicable evaluation aspects but do not consider case-specific aspects. On the other hand, evaluation frameworks that are case specific, by eliciting user requirements, limit their output to the evaluation aspects suggested by the users in the early phases of system development. In addition, these case-specific approaches extract different sets of evaluation aspects from each case, making it challenging to collectively compare, unify, or aggregate the evaluation of a set of heterogeneous health information systems. Objectives: The aim of this paper is to find a method capable of suggesting evaluation aspects for a set of one or more health information systems—whether similar or heterogeneous—by organizing, unifying, and aggregating the quality attributes extracted from those systems and from an external evaluation framework. Methods: On the basis of the available literature in semantic networks and ontologies, a method (called Unified eValuation using Ontology; UVON) was developed that can organize, unify, and aggregate the quality attributes of several health information systems into a tree-style ontology structure. The method was extended to integrate its generated ontology with the evaluation aspects suggested by model-based evaluation frameworks. An approach was developed to extract evaluation aspects from the ontology that also considers evaluation case practicalities such as the maximum number of evaluation aspects to be measured or their required degree of specificity. The method was applied and tested in Future Internet Social and Technological Alignment Research (FI-STAR), a project of 7 cloud-based eHealth applications that were developed and deployed across European Union countries. Results: The relevance of the evaluation aspects created by the UVON method for the FI-STAR project was validated by the corresponding stakeholders of each case. These evaluation aspects were extracted from a UVON-generated ontology structure that reflects both the internally declared required quality attributes in the 7 eHealth applications of the FI-STAR project and the evaluation aspects recommended by the Model for ASsessment of Telemedicine applications (MAST) evaluation framework. The extracted evaluation aspects were used to create questionnaires (for the corresponding patients and health professionals) to evaluate each individual case and the whole of the FI-STAR project. Conclusions: The UVON method can provide a relevant set of evaluation aspects for a heterogeneous set of health information systems by organizing, unifying, and aggregating the quality attributes through ontological structures. Those quality attributes can be either suggested by evaluation models or elicited from the stakeholders of those systems in the form of system requirements. The method continues to be systematic, context sensitive, and relevant across a heterogeneous set of health information systems. %M 27311735 %R 10.2196/medinform.5185 %U http://medinform.jmir.org/2016/2/e20/ %U https://doi.org/10.2196/medinform.5185 %U http://www.ncbi.nlm.nih.gov/pubmed/27311735 %0 Journal Article %@ 2291-9694 %I Gunther Eysenbach %V 4 %N 1 %P e5 %T Industry and Occupation in the Electronic Health Record: An Investigation of the National Institute for Occupational Safety and Health Industry and Occupation Computerized Coding System %A Schmitz,Matthew %A Forst,Linda %+ Environmental and Occupational Health Sciences, School of Public Health, University of Illinois at Chicago, 2121 W. Taylor, MC 922, Chicago, IL, 60612, United States, 1 312 3553826, Forst-L@uic.edu %K medical informatics %K occupation code %K industry code %K NIOCCS %K occupational health %K occupation+electronic health record %D 2016 %7 15.02.2016 %9 Original Paper %J JMIR Med Inform %G English %X Background: Inclusion of information about a patient’s work, industry, and occupation, in the electronic health record (EHR) could facilitate occupational health surveillance, better health outcomes, prevention activities, and identification of workers’ compensation cases. The US National Institute for Occupational Safety and Health (NIOSH) has developed an autocoding system for “industry” and “occupation” based on 1990 Bureau of Census codes; its effectiveness requires evaluation in conjunction with promoting the mandatory addition of these variables to the EHR. Objective: The objective of the study was to evaluate the intercoder reliability of NIOSH’s Industry and Occupation Computerized Coding System (NIOCCS) when applied to data collected in a community survey conducted under the Affordable Care Act; to determine the proportion of records that are autocoded using NIOCCS. Methods: Standard Occupational Classification (SOC) codes are used by several federal agencies in databases that capture demographic, employment, and health information to harmonize variables related to work activities among these data sources. There are 359 industry and occupation responses that were hand coded by 2 investigators, who came to a consensus on every code. The same variables were autocoded using NIOCCS at the high and moderate criteria level. Results: Kappa was .84 for agreement between hand coders and between the hand coder consensus code versus NIOCCS high confidence level codes for the first 2 digits of the SOC code. For 4 digits, NIOCCS coding versus investigator coding ranged from kappa=.56 to .70. In this study, NIOCCS was able to achieve production rates (ie, to autocode) 31%-36% of entered variables at the “high confidence” level and 49%-58% at the “medium confidence” level. Autocoding (production) rates are somewhat lower than those reported by NIOSH. Agreement between manually coded and autocoded data are “substantial” at the 2-digit level, but only “fair” to “good” at the 4-digit level. Conclusions: This work serves as a baseline for performance of NIOCCS by investigators in the field. Further field testing will clarify NIOCCS effectiveness in terms of ability to assign codes and coding accuracy and will clarify its value as inclusion of these occupational variables in the EHR is promoted. %M 26878932 %R 10.2196/medinform.4839 %U http://medinform.jmir.org/2016/1/e5/ %U https://doi.org/10.2196/medinform.4839 %U http://www.ncbi.nlm.nih.gov/pubmed/26878932 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 7 %N 2 %P e6010 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2015 %7 ..2015 %9 %J Online J Public Health Inform %G English %X Integration of disparate information from electronic health records, clinical data warehouses, birth certificate registries and other public health information systems offers great potential for clinical care, public health practice, and research. Such integration, however, depends on correctly matching patient-specific records using demographic identifiers. Without standards for these identifiers, record linkage is complicated by issues of structural and semantic heterogeneity.Objectives: Our objectives were to develop and validate an ontology to: 1) identify components of identity and events subsequent to birth that result in creation, change, or sharing of identity information; 2) develop an ontology to facilitate data integration from multiple healthcare and public health sources; and 3) validate the ontology’s ability to model identity-changing events over time.Methods: We interviewed domain experts in area hospitals and public health programs and developed process models describing the creation and transmission of identity information among various organizations for activities subsequent to a birth event. We searched for existing relevant ontologies. We validated the content of our ontology with simulated identity information conforming to scenarios identified in our process models.Results: We chose the Simple Event Model (SEM) to describe events in early childhood and integrated the Clinical Element Model (CEM) for demographic information. We demonstrated the ability of the combined SEM-CEM ontology to model identity events over time.Conclusion: The use of an ontology can overcome issues of semantic and syntactic heterogeneity to facilitate record linkage. %M 26392849 %R 10.5210/ojphi.v7i2.6010 %U %U https://doi.org/10.5210/ojphi.v7i2.6010 %U http://www.ncbi.nlm.nih.gov/pubmed/26392849 %0 Journal Article %@ 2291-9694 %I Gunther Eysenbach %V 2 %N 2 %P e23 %T Enabling Online Studies of Conceptual Relationships Between Medical Terms: Developing an Efficient Web Platform %A Albin,Aaron %A Ji,Xiaonan %A Borlawsky,Tara B %A Ye,Zhan %A Lin,Simon %A Payne,Philip RO %A Huang,Kun %A Xiang,Yang %+ Department of Biomedical Informatics, The Ohio State University, 250 Lincoln Tower, 1800 Canon Drive, Columbus, OH, 43210, United States, 1 614 366 5066, yxiang@bmi.osu.edu %K UMLS %K ontology %K conceptual relationships %D 2014 %7 07.10.2014 %9 Original Paper %J JMIR Med Inform %G English %X Background: The Unified Medical Language System (UMLS) contains many important ontologies in which terms are connected by semantic relations. For many studies on the relationships between biomedical concepts, the use of transitively associated information from ontologies and the UMLS has been shown to be effective. Although there are a few tools and methods available for extracting transitive relationships from the UMLS, they usually have major restrictions on the length of transitive relations or on the number of data sources. Objective: Our goal was to design an efficient online platform that enables efficient studies on the conceptual relationships between any medical terms. Methods: To overcome the restrictions of available methods and to facilitate studies on the conceptual relationships between medical terms, we developed a Web platform, onGrid, that supports efficient transitive queries and conceptual relationship studies using the UMLS. This framework uses the latest technique in converting natural language queries into UMLS concepts, performs efficient transitive queries, and visualizes the result paths. It also dynamically builds a relationship matrix for two sets of input biomedical terms. We are thus able to perform effective studies on conceptual relationships between medical terms based on their relationship matrix. Results: The advantage of onGrid is that it can be applied to study any two sets of biomedical concept relations and the relations within one set of biomedical concepts. We use onGrid to study the disease-disease relationships in the Online Mendelian Inheritance in Man (OMIM). By crossvalidating our results with an external database, the Comparative Toxicogenomics Database (CTD), we demonstrated that onGrid is effective for the study of conceptual relationships between medical terms. Conclusions: onGrid is an efficient tool for querying the UMLS for transitive relations, studying the relationship between medical terms, and generating hypotheses. %M 25600290 %R 10.2196/medinform.3387 %U http://medinform.jmir.org/2014/2/e23/ %U https://doi.org/10.2196/medinform.3387 %U http://www.ncbi.nlm.nih.gov/pubmed/25600290 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 5 %N 1 %P e4440 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2013 %7 ..2013 %9 %J Online J Public Health Inform %G English %X NYC emergency departments (ED) visits were categorized into NYC residential, NYC PO Box or commercial building, commuters into NYC, and out-of-town visitors by patient reported ZIP code. Spatial and temporal trends in total ED visits by home ZIP code type were analyzed to describe patterns in each group, and determine how such information might inform daily syndromic surveillance activities. Of the approximately 4 million ED visits in NYC during 2011, the number of visits by commuters and out-of-town visitors were 125,236 (3.1%) and 45,158 (1.1%) respectively. Out-of town visitors were found to mostly go to hospitals in Manhattan rather than the outer boroughs. While the seasonal trends and day-of-week patterns for the NYC residents and the commuters appear to be fairly similar, temporal trends for NYC residents and out-of-town visitors were found to be different. Out-of-town visitors represents a unique subset of the ED population and our results suggest that including a separate analyses of total ED and syndromic visits by out-of-town visitors might provide additional information that could prove useful to daily syndromic surveillance activities. %R 10.5210/ojphi.v5i1.4440 %U %U https://doi.org/10.5210/ojphi.v5i1.4440 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 5 %N 1 %P e4447 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2013 %7 ..2013 %9 %J Online J Public Health Inform %G English %X Electronic Laboratory Reporting (ELR) has the potential to be more accurate, timely, and cost-effective. However, the continuing use of non-standard, local codes to represent laboratory test results complicates the use of ELR data in public health practice. Use of structured and standardized coding system(s) to support the concepts represented by local codes improves the computational characteristics of ELR data. We examined the use of LOINC and SNOMED CT codes for coding laboratory tests in hospital laboratory reports. We found that the hospitals more frequently used LOINC codes than SNOMED CT in reporting test results. %R 10.5210/ojphi.v5i1.4447 %U %U https://doi.org/10.5210/ojphi.v5i1.4447 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 5 %N 1 %P e4452 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2013 %7 ..2013 %9 %J Online J Public Health Inform %G English %X There is increasing demand for ways to use syndromic surveillance data for population health surveillance. The authors developed a nontraumatic oral health classification that could provide timely burden estimates of oral health-related visits to North Carolina (NC) emergency departments (EDs) using BioSense syndromic data. A combination of literature review, input by subject matter experts, and analysis of syndromic data was employed to create a classification that used select chief complaint text and ICD-9-CM codes for visit inclusion and exclusion criteria. Visit estimates created using this classification could contribute to policy decisions aimed at reducing this unnecessary burden on NC EDs. %R 10.5210/ojphi.v5i1.4452 %U %U https://doi.org/10.5210/ojphi.v5i1.4452