This paper is in the following e-collection/theme issue:

Natural Language Processing (917) Generative Language Models Including ChatGPT (708) AI Language Models in Health Care (202) Chatbots and Conversational Agents (740) Artificial Intelligence (2129) Formative Evaluation of Digital Health Interventions (3229)

Published on in Vol 12 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/57674, first published .
Data Set and Benchmark (MedGPTEval) to Evaluate Responses From Large Language Models in Medicine: Evaluation Development and Validation

Data Set and Benchmark (MedGPTEval) to Evaluate Responses From Large Language Models in Medicine: Evaluation Development and Validation

Data Set and Benchmark (MedGPTEval) to Evaluate Responses From Large Language Models in Medicine: Evaluation Development and Validation

Authors of this article:

Jie Xu1 Author Orcid Image ;   Lu Lu1 Author Orcid Image ;   Xinwei Peng1 Author Orcid Image ;   Jiali Pang1 Author Orcid Image ;   Jinru Ding1 Author Orcid Image ;   Lingrui Yang2 Author Orcid Image ;   Huan Song3, 4 Author Orcid Image ;   Kang Li3, 4 Author Orcid Image ;   Xin Sun2 Author Orcid Image ;   Shaoting Zhang1 Author Orcid Image
Article Authors Cited by (12) Tweetations Metrics

Journals

  1. Rewthamrongsris P, Burapacheep J, Trachoo V, Porntaveetus T. Accuracy of Large Language Models for Infective Endocarditis Prophylaxis in Dental Procedures. International Dental Journal 2025;75(1):206 View
  2. Andrew A, Tizzard E. Large language models for improving cancer diagnosis and management in primary health care settings. Journal of Medicine, Surgery, and Public Health 2024:100157 View
  3. Chang Y, Yin J, Li J, Liu C, Cao L, Lin S. Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework. Journal of Medical Systems 2024;48(1) View
  4. Kreso A, Boban Z, Kabic S, Rada F, Batistic D, Barun I, Znaor L, Kumric M, Bozic J, Vrdoljak J. Using large language models as decision support tools in emergency ophthalmology. International Journal of Medical Informatics 2025;199:105886 View
  5. Wei B, Yao L, Hu X, Hu Y, Rao J, Ji Y, Dong Z, Duan Y, Wu X. Evaluating the Effectiveness of Large Language Models in Providing Patient Education for Chinese Patients With Ocular Myasthenia Gravis: Mixed Methods Study. Journal of Medical Internet Research 2025;27:e67883 View
  6. Liu Y, Shi C, Wu L, Lin X, Chen X, Zhu Y, Tan H, Zhang W. Development and Validation of a Large Language Model–Based System for Medical History-Taking Training: Prospective Multicase Study on Evaluation Stability, Human-AI Consistency, and Transparency. JMIR Medical Education 2025;11:e73419 View
  7. Torous J, Ledley K, Gorban C, Strudwick G, Schwarz J, Choudhary S, Emerson M, Patriquin M, Dempsey A, Bantjes J, Ospina-Pinillos L, Hornick J, Kochhar S. Accelerating Digital Mental Health: The Society of Digital Psychiatry’s Three-Pronged Road Map for Education, Digital Navigators, and AI. JMIR Mental Health 2025;12:e84501 View
  8. Dwyer B, Flathers M, Sano A, Dempsey A, Cipriani A, Gazi A, Hill B, Gorban C, Rodriguez C, Stromeyer C, King D, Rozenblit E, Strudwick G, Linardon J, Cheong J, Firth J, Herpertz J, Schwarz J, Truong K, Emerson M, Paulus M, Patriquin M, Hua Y, Choudhary S, Siddals S, Pinillos L, Bantjes J, Scheuller S, Xu X, Duckworth K, Gillison D, Wood M, Torous J. Mindbench.ai: an actionable platform to evaluate the profile and performance of large language models in a mental healthcare context. NPP—Digital Psychiatry and Neuroscience 2025;3(1) View
  9. Chang Q, Chen F, Chen Y, Cheng L, Dong D, Dong J, Feng X, Ge J, He J, He Y, He Z, Ji H, Jiang X, Jiang Z, Li N, Li P, Li Y, Liu B, Liu J, Lyu H, Min D, Qi W, Shen X, Sheng B, Sun J, Sun Y, Tian B, Wang K, Wang L, Wang L, Wang W, Wang Y, Wang Y, Wang Z, Weng J, Wei J, Wu G, Wu X, Xiao Y, Xu Y, Yan P, Ye Z, Yin W, Zhang C, Zhang D, Zhang P, Zhang W, Zhang X, Zhao S, Zhao Y, Zhou S, Zhou X, Zhu B, Zhu L, Zhu Z. 2025 Expert consensus on retrospective evaluation of large language model applications in clinical scenarios. Intelligent Medicine 2025 View
  10. Singh S, Alyakin A, Alber D, Stryker J, Tong A, Sangwon K, Goff N, De La Paz M, Hernandez-Rovira M, Park K, Leuthardt E, Oermann E. The pitfalls of multiple-choice questions in generative AI and medical education. Scientific Reports 2025;15(1) View

Books/Policy Documents

  1. Xu H, Xue T, Liu D, Zhang F, Westin C, Kikinis R, O’Donnell L, Cai W. Foundation Models for General Medical AI. View
  2. Geathers J, Hicke Y, Chan C, Rajashekar N, Young S, Sewell J, Cornes S, Kizilcec R, Shung D. Artificial Intelligence in Education. View