This paper is in the following e-collection/theme issue:

Natural Language Processing (692) Artificial Intelligence (1371) Electronic Health Records (1053)

Published on in Vol 7, No 3 (2019): Jul-Sep

Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)–Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study

Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)–Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study

Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)–Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study

Authors of this article:

Fei Li1, 2, 3 Author Orcid Image ;   Yonghao Jin1 Author Orcid Image ;   Weisong Liu1, 2, 3 Author Orcid Image ;   Bhanu Pratap Singh Rawat4 Author Orcid Image ;   Pengshan Cai4 Author Orcid Image ;   Hong Yu1, 2, 3, 4 Author Orcid Image
Article Authors Cited by (99) Tweetations (5) Metrics

Journals

  1. Xu D, Gopale M, Zhang J, Brown K, Begoli E, Bethard S. Unified Medical Language System resources improve sieve-based generation and Bidirectional Encoder Representations from Transformers (BERT)–based ranking for concept normalization. Journal of the American Medical Informatics Association 2020;27(10):1510 View
  2. Park B, Afzal M, Hussain J, Abbas A, Lee S. Automatic Identification of High Impact Relevant Articles to Support Clinical Decision Making Using Attention-Based Deep Learning. Electronics 2020;9(9):1364 View
  3. Li L, Wang P, Yan J, Wang Y, Li S, Jiang J, Sun Z, Tang B, Chang T, Wang S, Liu Y. Real-world data medical knowledge graph: construction and applications. Artificial Intelligence in Medicine 2020;103:101817 View
  4. Colicchio T, Dissanayake P, Cimino J. Formal representation of patients’ care context data: the path to improving the electronic health record. Journal of the American Medical Informatics Association 2020;27(11):1648 View
  5. Kang H, Li J, Wu M, Shen L, Hou L. Building a Pharmacogenomics Knowledge Model Toward Precision Medicine: Case Study in Melanoma. JMIR Medical Informatics 2020;8(10):e20291 View
  6. Jim H, Hoogland A, Brownstein N, Barata A, Dicker A, Knoop H, Gonzalez B, Perkins R, Rollison D, Gilbert S, Nanda R, Berglund A, Mitchell R, Johnstone P. Innovations in research and clinical care using patient‐generated health data. CA: A Cancer Journal for Clinicians 2020;70(3):182 View
  7. Kersloot M, van Putten F, Abu-Hanna A, Cornet R, Arts D. Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies. Journal of Biomedical Semantics 2020;11(1) View
  8. Qin Q, Zhao S, Liu C, Hassanien A. A BERT‐BiGRU‐CRF Model for Entity Recognition of Chinese Electronic Medical Records. Complexity 2021;2021(1) View
  9. Tahayori B, Chini‐Foroush N, Akhlaghi H. Advanced natural language processing technique to predict patient disposition based on emergency triage notes. Emergency Medicine Australasia 2021;33(3):480 View
  10. Kim Y, Lee J, Choi S, Lee J, Kim J, Seok J, Joo H. Validation of deep learning natural language processing algorithm for keyword extraction from pathology reports in electronic health records. Scientific Reports 2020;10(1) View
  11. Ji Y, Zhou Z, Liu H, Davuluri R, Kelso J. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics 2021;37(15):2112 View
  12. Lu Z, Wang J, Li X. Revealing Opinions for COVID-19 Questions Using a Context Retriever, Opinion Aggregator, and Question-Answering Model: Model Development Study. Journal of Medical Internet Research 2021;23(3):e22860 View
  13. Lee H, Kang J, Yeo J. Medical Specialty Recommendations by an Artificial Intelligence Chatbot on a Smartphone: Development and Deployment. Journal of Medical Internet Research 2021;23(5):e27460 View
  14. Mitra A, Rawat B, McManus D, Yu H. Relation Classification for Bleeding Events From Electronic Health Records Using Deep Learning Systems: An Empirical Study. JMIR Medical Informatics 2021;9(7):e27527 View
  15. Emmert-Streib F. Grand Challenges for Artificial Intelligence in Molecular Medicine. Frontiers in Molecular Medicine 2021;1 View
  16. Percha B. Modern Clinical Text Mining: A Guide and Review. Annual Review of Biomedical Data Science 2021;4(1):165 View
  17. Miftahutdinov Z, Kadurin A, Kudrin R, Tutubalina E, Wren J. Medical concept normalization in clinical trials with drug and disease representation learning. Bioinformatics 2021;37(21):3856 View
  18. Kumar V, Reforgiato Recupero D, Helaoui R, Riboni D. K-LM: Knowledge Augmenting in Language Models Within the Scholarly Domain. IEEE Access 2022;10:91802 View
  19. Tamine L, Goeuriot L. Semantic Information Retrieval on Medical Texts. ACM Computing Surveys 2022;54(7):1 View
  20. Krishnan R, Rajpurkar P, Topol E. Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering 2022;6(12):1346 View
  21. Mitchell J, Szepietowski P, Howard R, Reisman P, Jones J, Lewis P, Fridley B, Rollison D. A Question-and-Answer System to Extract Data From Free-Text Oncological Pathology Reports (CancerBERT Network): Development Study. Journal of Medical Internet Research 2022;24(3):e27210 View
  22. Caskey J, McConnell I, Oguss M, Dligach D, Kulikoff R, Grogan B, Gibson C, Wimmer E, DeSalvo T, Nyakoe-Nyasani E, Churpek M, Afshar M. Identifying COVID-19 Outbreaks From Contact-Tracing Interview Forms for Public Health Departments: Development of a Natural Language Processing Pipeline. JMIR Public Health and Surveillance 2022;8(3):e36119 View
  23. Nair L, Shivani M, Jo Cheriyan S. Enabling Remote School Education using Knowledge Graphs and Deep Learning Techniques. Procedia Computer Science 2022;215:618 View
  24. Park W, Siddiqui I, Chakraborty C, Qureshi N, Shin D. Scarcity-aware spam detection technique for big data ecosystem. Pattern Recognition Letters 2022;157:67 View
  25. Guo S, Yang W, Han L, Song X, Wang G. A multi-layer soft lattice based model for Chinese clinical named entity recognition. BMC Medical Informatics and Decision Making 2022;22(1) View
  26. Kariampuzha W, Alyea G, Qu S, Sanjak J, Mathé E, Sid E, Chatelaine H, Yadaw A, Xu Y, Zhu Q. Precision information extraction for rare disease epidemiology at scale. Journal of Translational Medicine 2023;21(1) View
  27. Kanaparthi V. Examining Natural Language Processing Techniques in the Education and Healthcare Fields. International Journal of Engineering and Advanced Technology 2022;12(2):8 View
  28. Li L, Zhai Y, Gao J, Wang L, Hou L, Zhao J. Stacking-BERT model for Chinese medical procedure entity normalization. Mathematical Biosciences and Engineering 2022;20(1):1018 View
  29. Kawazoe Y, Shimamoto K, Shibata D, Shinohara E, Kawaguchi H, Yamamoto T. Impact of a Clinical Text–Based Fall Prediction Model on Preventing Extended Hospital Stays for Elderly Inpatients: Model Development and Performance Evaluation. JMIR Medical Informatics 2022;10(7):e37913 View
  30. Brisk R, Bond R, Finlay D, McLaughlin J, Piadlo A, McEneaney D. WaSP-ECG: A Wave Segmentation Pretraining Toolkit for Electrocardiogram Analysis. Frontiers in Physiology 2022;13 View
  31. Olthof A, van Ooijen P, Cornelissen L. Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance. Journal of Medical Systems 2021;45(10) View
  32. Wan C, Ge X, Wang J, Zhang X, Yu Y, Hu J, Liu Y, Ma H. Identification and Impact Analysis of Family History of Psychiatric Disorder in Mood Disorder Patients With Pretrained Language Model. Frontiers in Psychiatry 2022;13 View
  33. Wang K, Wang Z, Usman M. Deep Learning Models and Social Governance Guided by Fair Policies. Scientific Programming 2022;2022:1 View
  34. Nassiri K, Akhloufi M. Transformer models used for text-based question answering systems. Applied Intelligence 2023;53(9):10602 View
  35. Li I, Pan J, Goldwasser J, Verma N, Wong W, Nuzumlalı M, Rosand B, Li Y, Zhang M, Chang D, Taylor R, Krumholz H, Radev D. Neural Natural Language Processing for unstructured data in electronic health records: A review. Computer Science Review 2022;46:100511 View
  36. Segura-Bedmar I, Camino-Perdones D, Guerrero-Aspizua S. Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts. BMC Bioinformatics 2022;23(1) View
  37. Piao C, Lv M, Wang S, Zhou R, Wang Y, Wei J, Liu J. Multi-objective data enhancement for deep learning-based ultrasound analysis. BMC Bioinformatics 2022;23(1) View
  38. Colicchio T, Liang W, Dissanayake P, Do Rosario C, Cimino J. Physicians' perceptions about a semantically integrated display for chart review: A Multi-Specialty survey. International Journal of Medical Informatics 2022;163:104788 View
  39. Bannour N, Wajsbürt P, Rance B, Tannier X, Névéol A. Privacy-preserving mimic models for clinical named entity recognition in French. Journal of Biomedical Informatics 2022;130:104073 View
  40. Frei J, Soto-Rey I, Kramer F, Banerjee I. DrNote: An open medical annotation service. PLOS Digital Health 2022;1(8):e0000086 View
  41. Richter-Pechanski P, Geis N, Kiriakou C, Schwab D, Dieterich C. Automatic extraction of 12 cardiovascular concepts from German discharge letters using pre-trained language models. DIGITAL HEALTH 2021;7 View
  42. Lin C, Hsu K, Liang C, Lee T, Liou C, Lee J, Peng T, Shih C, Fann Y. A disease-specific language representation model for cerebrovascular disease research. Computer Methods and Programs in Biomedicine 2021;211:106446 View
  43. Vyas S, Shabaz M, Pandit P, Parvathy L, Ofori I, AL-Farga A. Integration of Artificial Intelligence and Blockchain Technology in Healthcare and Agriculture. Journal of Food Quality 2022;2022:1 View
  44. Khurshid S, Reeder C, Harrington L, Singh P, Sarma G, Friedman S, Di Achille P, Diamant N, Cunningham J, Turner A, Lau E, Haimovich J, Al-Alusi M, Wang X, Klarqvist M, Ashburner J, Diedrich C, Ghadessi M, Mielke J, Eilken H, McElhinney A, Derix A, Atlas S, Ellinor P, Philippakis A, Anderson C, Ho J, Batra P, Lubitz S. Cohort design and natural language processing to reduce bias in electronic health records research. npj Digital Medicine 2022;5(1) View
  45. Li S, Hickey G, Lander M, Kanwar M. Artificial Intelligence and Mechanical Circulatory Support. Heart Failure Clinics 2022;18(2):301 View
  46. Frei J, Kramer F. German Medical Named Entity Recognition Model and Data Set Creation Using Machine Translation and Word Alignment: Algorithm Development and Validation. JMIR Formative Research 2023;7:e39077 View
  47. López-García G, Jerez J, Ribelles N, Alba E, Veredas F. Explainable clinical coding with in-domain adapted transformers. Journal of Biomedical Informatics 2023;139:104323 View
  48. Roitero K, Portelli B, Popescu M, Mea V. DiLBERT: Cheap Embeddings for Disease Related Medical NLP. IEEE Access 2021;9:159714 View
  49. Shoeibi A, Moridian P, Khodatars M, Ghassemi N, Jafari M, Alizadehsani R, Kong Y, Gorriz J, Ramírez J, Khosravi A, Nahavandi S, Acharya U. An overview of deep learning techniques for epileptic seizures detection and prediction based on neuroimaging modalities: Methods, challenges, and future works. Computers in Biology and Medicine 2022;149:106053 View
  50. Zhang A, Xing L, Zou J, Wu J. Shifting machine learning for healthcare from development to deployment and from models to data. Nature Biomedical Engineering 2022;6(12):1330 View
  51. Alzubi R, Alzoubi H, Katsigiannis S, West D, Ramzan N. Automated Detection of Substance-Use Status and Related Information from Clinical Text. Sensors 2022;22(24):9609 View
  52. Wu X, Duan J, Pan Y, Li M. Medical Knowledge Graph: Data Sources, Construction, Reasoning, and Applications. Big Data Mining and Analytics 2023;6(2):201 View
  53. Park C, Yi P, Al Yousif H, Wang K. Machine vs. Radiologist-Based Translations of RadLex: Implications for Multi-language Report Interoperability. Journal of Digital Imaging 2022;35(3):660 View
  54. Jupin-Delevaux É, Djahnine A, Talbot F, Richard A, Gouttard S, Mansuy A, Douek P, Si-Mohamed S, Boussel L. BERT-based natural language processing analysis of French CT reports: Application to the measurement of the positivity rate for pulmonary embolism. Research in Diagnostic and Interventional Imaging 2023;6:100027 View
  55. Zhang Y, Li X, Liu Y, Li A, Yang X, Tang X. A Multilabel Text Classifier of Cancer Literature at the Publication Level: Methods Study of Medical Text Classification. JMIR Medical Informatics 2023;11:e44892 View
  56. Ding J, Qiao Y, Zhang L. Plant disease prescription recommendation based on electronic medical records and sentence embedding retrieval. Plant Methods 2023;19(1) View
  57. Wang M, Sushil M, Miao B, Butte A. Bottom-up and top-down paradigms of artificial intelligence research approaches to healthcare data science using growing real-world big data. Journal of the American Medical Informatics Association 2023;30(7):1323 View
  58. Chen Y, Li X, Li A, Li Y, Yang X, Lin Z, Yu S, Tang X. A Deep Learning Model for the Normalization of Institution Names by Multisource Literature Feature Fusion: Algorithm Development Study. JMIR Formative Research 2023;7:e47434 View
  59. Houssein E, Mohamed R, Ali A. Heart disease risk factors detection from electronic health records using advanced NLP and deep learning techniques. Scientific Reports 2023;13(1) View
  60. Wang B, Xie Q, Pei J, Chen Z, Tiwari P, Li Z, Fu J. Pre-trained Language Models in Biomedical Domain: A Systematic Survey. ACM Computing Surveys 2024;56(3):1 View
  61. Frei J, Frei-Stuber L, Kramer F. GERNERMED++: Semantic annotation in German medical NLP through transfer-learning, translation and word alignment. Journal of Biomedical Informatics 2023;147:104513 View
  62. Unger S, Raak C, Ostermann T. Reliability and Performance of the Online Literature Database CAMbase after Changing from a Semantic Search to a Score Ranking Algorithm. SN Computer Science 2023;4(5) View
  63. Wornow M, Xu Y, Thapa R, Patel B, Steinberg E, Fleming S, Pfeffer M, Fries J, Shah N. The shaky foundations of large language models and foundation models for electronic health records. npj Digital Medicine 2023;6(1) View
  64. Kumari N, Singh P. Hindi Text Summarization Using Sequence to Sequence Neural Network. ACM Transactions on Asian and Low-Resource Language Information Processing 2023;22(10):1 View
  65. Ahmad P, Liu Y, Khan K, Jiang T, Burhan U. BIR: Biomedical Information Retrieval System for Cancer Treatment in Electronic Health Record Using Transformers. Sensors 2023;23(23):9355 View
  66. Vithanage D, Yu P, Wang L, Deng C. Contextual Word Embedding for Biomedical Knowledge Extraction: a Rapid Review and Case Study. Journal of Healthcare Informatics Research 2024;8(1):158 View
  67. Wang L, Ma Y, Bi W, Lv H, Li Y. An Entity Extraction Pipeline for Medical Text Records Using Large Language Models: Analytical Study. Journal of Medical Internet Research 2024;26:e54580 View
  68. Yang T, Sucholutsky I, Jen K, Schonlau M. exKidneyBERT: a language model for kidney transplant pathology reports and the crucial role of extended vocabularies. PeerJ Computer Science 2024;10:e1888 View
  69. Chen C, Chang C. Effectiveness of AI-assisted game-based learning on science learning outcomes, intrinsic motivation, cognitive load, and learning behavior. Education and Information Technologies 2024;29(14):18621 View
  70. Yang H, Zhu D, He S, Xu Z, Liu Z, Zhang W, Cai J. Enhancing psychiatric rehabilitation outcomes through a multimodal multitask learning model based on BERT and TabNet: An approach for personalized treatment and improved decision-making. Psychiatry Research 2024;336:115896 View
  71. Zeinali N, Albashayreh A, Fan W, White S. Symptom-BERT: Enhancing Cancer Symptom Detection in EHR Clinical Notes. Journal of Pain and Symptom Management 2024;68(2):190 View
  72. Tran H, Yang Z, Yao Z, Yu H. BioInstruct: instruction tuning of large language models for biomedical natural language processing. Journal of the American Medical Informatics Association 2024;31(9):1821 View
  73. Nerella S, Bandyopadhyay S, Zhang J, Contreras M, Siegel S, Bumin A, Silva B, Sena J, Shickel B, Bihorac A, Khezeli K, Rashidi P. Transformers and large language models in healthcare: A review. Artificial Intelligence in Medicine 2024;154:102900 View
  74. Preuss N, Alshehri A, You F. Large language models for life cycle assessments: Opportunities, challenges, and risks. Journal of Cleaner Production 2024;466:142824 View
  75. Huisman T, Huisman T. Artificial Intelligence in Newborn Medicine. Newborn 2024;3(2):96 View
  76. Marchena Sekli G. The research landscape on generative artificial intelligence: a bibliometric analysis of transformer-based models. Kybernetes 2024 View
  77. Meshkin H, Zirkle J, Arabidarrehdor G, Chaturbedi A, Chakravartula S, Mann J, Thrasher B, Li Z. Harnessing large language models’ zero-shot and few-shot learning capabilities for regulatory research. Briefings in Bioinformatics 2024;25(5) View
  78. Han P, Li X, Zhang Z, Zhong Y, Gu L, Hua Y, Li X. CMCN: Chinese medical concept normalization using continual learning and knowledge-enhanced. Artificial Intelligence in Medicine 2024;157:102965 View
  79. Panagides R, Fu S, Jung S, Singh A, Eluvathingal Muttikkal R, Broad R, Meakem T, Hamilton R. Enhancing Literature Review Efficiency: A Case Study on Using Fine-Tuned BERT for Classifying Focused Ultrasound-Related Articles. AI 2024;5(3):1670 View
  80. Attai K, Ekpenyong M, Amannah C, Asuquo D, Ajuga P, Obot O, Johnson E, John A, Maduka O, Akwaowo C, Uzoka F. Enhancing the Interpretability of Malaria and Typhoid Diagnosis with Explainable AI and Large Language Models. Tropical Medicine and Infectious Disease 2024;9(9):216 View
  81. Renc P, Jia Y, Samir A, Was J, Li Q, Bates D, Sitek A. Zero shot health trajectory prediction using transformer. npj Digital Medicine 2024;7(1) View
  82. Chang E, Sung S. Use of SNOMED CT in Large Language Models: Scoping Review. JMIR Medical Informatics 2024;12:e62924 View
  83. He R, Sarwal V, Qiu X, Zhuang Y, Zhang L, Liu Y, Chiang J. Generative AI models in time varying biomedical data: a systematic review (Preprint). Journal of Medical Internet Research 2024 View
  84. Cho H, Jun T, Kim Y, Kang H, Ahn I, Gwon H, Kim Y, Seo J, Choi H, Kim M, Han J, Kee G, Park S, Ko S. Task-Specific Transformer-Based Language Models in Health Care: Scoping Review. JMIR Medical Informatics 2024;12:e49724 View

Books/Policy Documents

  1. Blinov P, Avetisian M, Kokh V, Umerenkov D, Tuzhilin A. Artificial Intelligence in Medicine. View
  2. Zhu R, Tu X, Huang J. Data Analytics in Biomedical Engineering and Healthcare. View
  3. Lin H, Yang L, Wang P. Trends and Applications in Information Systems and Technologies. View
  4. Jia G, Zhu W, Tang J, Zhang W. HCI in Business, Government and Organizations. View
  5. Guo S, Han L, Yang W. Clinical Chinese Named Entity Recognition in Natural Language Processing. View
  6. Arideh M, Taboada M. New Trends in Disruptive Technologies, Tech Ethics and Artificial Intelligence. View
  7. Arideh M, Taboada M, Martínez D. Proceedings of the Future Technologies Conference (FTC) 2023, Volume 3. View