This paper is in the following e-collection/theme issue:

Natural Language Processing (692) Clinical Informatics (1022) Development and Evaluation of Research Methods, Instruments and Tools (481)

Published on in Vol 9, No 7 (2021): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/20492, first published .
Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology

Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology

Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology

Authors of this article:

Lea Canales1 Author Orcid Image ;   Sebastian Menke2 Author Orcid Image ;   Stephanie Marchesseau2 Author Orcid Image ;   Ariel D’Agostino2 Author Orcid Image ;   Carlos del Rio-Bermudez2 Author Orcid Image ;   Miren Taberna2 Author Orcid Image ;   Jorge Tello2 Author Orcid Image
Article Authors Cited by (43) Tweetations (17) Metrics

Journals

  1. Montoto C, Gisbert J, Guerra I, Plaza R, Pajares Villarroya R, Moreno Almazán L, López Martín M, Domínguez Antonaya M, Vera Mendoza I, Aparicio J, Martínez V, Tagarro I, Fernandez-Nistal A, Canales L, Menke S, Gomollón F. Evaluation of Natural Language Processing for the Identification of Crohn Disease–Related Variables in Spanish Electronic Health Records: A Validation Study for the PREMONITION-CD Project. JMIR Medical Informatics 2022;10(2):e30345 View
  2. Yusufov M, Pirl W, Braun I, Tulsky J, Lindvall C. Natural Language Processing for Computer-Assisted Chart Review to Assess Documentation of Substance use and Psychopathology in Heart Failure Patients Awaiting Cardiac Resynchronization Therapy. Journal of Pain and Symptom Management 2022;64(4):400 View
  3. Díez J, Cabrera L, Iglesias P, Benavent M, Argüello G, López G, Parralejo A, Leal J. Prevalencia de cáncer en pacientes con hipotiroidismo: análisis mediante herramientas de big data. Endocrinología, Diabetes y Nutrición 2023;70:50 View
  4. Wai E. CORR Insights®: Can We Geographically Validate a Natural Language Processing Algorithm for Automated Detection of Incidental Durotomy Across Three Independent Cohorts From Two Continents?. Clinical Orthopaedics & Related Research 2022;480(9):1776 View
  5. Poveda J, Bretón-Romero R, Del Rio-Bermudez C, Taberna M, Medrano I. How can artificial intelligence optimize value-based contracting?. Journal of Pharmaceutical Policy and Practice 2022;15(1) View
  6. Giachelle F, Irrera O, Silvello G. MedTAG: a portable and customizable annotation tool for biomedical documents. BMC Medical Informatics and Decision Making 2021;21(1) View
  7. Hens D, Wyers L, Claeys K. Validation of an Artificial Intelligence driven framework to automatically detect red flag symptoms in screening for rare diseases in electronic health records: hereditary transthyretin amyloidosis polyneuropathy as a key example. Journal of the Peripheral Nervous System 2023;28(1):79 View
  8. Segura T, Medrano I, Collazo S, Maté C, Sguera C, Del Rio-Bermudez C, Casero H, Salcedo I, García-García J, Alcahut-Rodríguez C, Aquino J, Casadevall D, Donaire D, Marin-Corral J, Menke S, Polo N, Taberna M. Symptoms timeline and outcomes in amyotrophic lateral sclerosis using artificial intelligence. Scientific Reports 2023;13(1) View
  9. Gomollón F, Gisbert J, Guerra I, Plaza R, Pajares Villarroya R, Moreno Almazán L, López Martín M, Domínguez Antonaya M, Vera Mendoza M, Aparicio J, Martínez V, Tagarro I, Fernández-Nistal A, Lumbreras S, Maté C, Montoto C. Clinical characteristics and prognostic factors for Crohn’s disease relapses using natural language processing and machine learning: a pilot study. European Journal of Gastroenterology & Hepatology 2022;34(4):389 View
  10. González-Juanatey C, Anguita-Sánchez M, Barrios V, Núñez-Gil I, Gómez-Doblas J, García-Moll X, Lafuente-Gormaz C, Rollán-Gómez M, Peral-Disdier V, Martínez-Dolz L, Rodríguez-Santamarta M, Viñolas-Prat X, Soriano-Colomé T, Muñoz-Aguilera R, Plaza I, Curcio-Ruigómez A, Orts-Soler E, Segovia J, Fanjul V, Cequier Á. Major Adverse Cardiovascular Events in Coronary Type 2 Diabetic Patients: Identification of Associated Factors Using Electronic Health Records and Natural Language Processing. Journal of Clinical Medicine 2022;11(20):6004 View
  11. Izquierdo J, Rodríguez J, Almonacid C, Benavent M, Arroyo-Espliguero R, Agustí A. Real-life burden of hospitalisations due to COPD exacerbations in Spain. ERJ Open Research 2022;8(3):00141-2022 View
  12. González-Juanatey C, Anguita-Sá́nchez M, Barrios V, Núñez-Gil I, Gómez-Doblas J, García-Moll X, Lafuente-Gormaz C, Rollán-Gómez M, Peral-Disdie V, Martínez-Dolz L, Rodríguez-Santamarta M, Viñolas-Prat X, Soriano-Colomé T, Muñoz-Aguilera R, Plaza I, Curcio-Ruigómez A, Orts-Soler E, Segovia J, Maté C, Cequier Á, Pizzi C. Assessment of medical management in Coronary Type 2 Diabetic patients with previous percutaneous coronary intervention in Spain: A retrospective analysis of electronic health records using Natural Language Processing. PLOS ONE 2022;17(2):e0263277 View
  13. Navarro-Compán V, Ermann J, Poddubnyy D. A glance into the future of diagnosis and treatment of spondyloarthritis. Therapeutic Advances in Musculoskeletal Disease 2022;14 View
  14. Díez J, Cabrera L, Iglesias P, Benavent M, López G, Argüello G, Parralejo A, López-Velázquez A. Carcinoma de tiroides en personas mayores: caracterización mediante herramientas de big data. Endocrinología, Diabetes y Nutrición 2023;70(3):179 View
  15. Valdés Sanz N, García-Layana A, Colas T, Moriche M, Montero Moreno M, Ciprandi G. Clinical Characterization of Inpatients with Acute Conjunctivitis: A Retrospective Analysis by Natural Language Processing and Machine Learning. Applied Sciences 2022;12(23):12352 View
  16. Calleja Panero J, de la Poza G, Hidalgo L, Aguilera Sancho-Tello M, Torras X, Santos de Lamadrid R, Maté C, Sánchez Antolín G. Patient journey of individuals tested for HCV in Spain: LiverTAI, a retrospective analysis of EHRs through natural language processing. Gastroenterología y Hepatología 2023;46(7):491 View
  17. Díez J, Cabrera L, Iglesias P, Benavent M, López G, Argüello G, Parralejo A, López-Velázquez A. Thyroid carcinoma in elderly people: Characterization using big data tools. Endocrinología, Diabetes y Nutrición (English ed.) 2023;70(3):179 View
  18. Calleja Panero J, de la Poza G, Hidalgo L, Aguilera Sancho-Tello M, Torras X, Santos de Lamadrid R, Maté C, Sánchez Antolín G. Patient journey of individuals tested for HCV in Spain: LiverTAI, a retrospective analysis of EHRs through natural language processing. Gastroenterología y Hepatología (English Edition) 2023;46(7):491 View
  19. Loscertales J, Abrisqueta-Costa P, Gutierrez A, Hernández-Rivas J, Andreu-Lapiedra R, Mora A, Leiva-Farré C, López-Roda M, Callejo-Mellén Á, Álvarez-García E, García-Marco J. Real-World Evidence on the Clinical Characteristics and Management of Patients with Chronic Lymphocytic Leukemia in Spain Using Natural Language Processing: The SRealCLL Study. Cancers 2023;15(16):4047 View
  20. Park J, Djelassi M, Chima D, Hernandez R, Poroshin V, Iliescu A, Domalik D, Southall N. Validation of a Natural Language Machine Learning Model for Safety Literature Surveillance. Drug Safety 2024;47(1):71 View
  21. Díez J, Cabrera L, Iglesias P, Benavent M, Argüello G, López G, Parralejo A, Leal J. Prevalence of cancer in patients with hypothyroidism: Analysis using big data tools. Endocrinología, Diabetes y Nutrición (English ed.) 2023;70:50 View
  22. Sabrie N, Khan R, Jogendran R, Scaffidi M, Bansal R, Gimpaya N, Youssef M, Forbes N, Mosko J, Berzin T, Lightfoot D, Grover S. Performance of natural language processing in identifying adenomas from colonoscopy reports: a systematic review and meta-analysis. iGIE 2023;2(3):350 View
  23. Calleja-Panero J, Esteban Mur R, Jarque I, Romero-Gómez M, Group S, García Labrador L, González Calvo J. Chronic liver disease-associated severe thrombocytopenia in Spain: Results from a retrospective study using machine learning and natural language processing. Gastroenterología y Hepatología 2024;47(3):236 View
  24. Morena D, Lumbreras S, Rodríguez J, Campos C, Castillo M, Benavent M, Izquierdo J. Chronic Respiratory Diseases as a Risk Factor for Herpes Zoster Infection. Archivos de Bronconeumología 2023;59(12):797 View
  25. Iglesias P, Arias J, López G, Romero I, Díez J. Primary Hyperparathyroidism and Cardiovascular Disease: An Association Study Using Clinical Natural Language Processing Systems and Big Data Analytics. Journal of Clinical Medicine 2023;12(21):6718 View
  26. Iglesias P, Benavent M, López G, Arias J, Romero I, Díez J. Hyperthyroidism and cardiovascular disease: an association study using big data analytics. Endocrine 2023;83(2):405 View
  27. González-Juanatey C, Anguita-Sánchez M, Barrios V, Núñez-Gil I, Gómez-Doblas J, García-Moll X, Lafuente-Gormaz C, Rollán-Gómez M, Peral-Disdier V, Martínez-Dolz L, Rodríguez-Santamarta M, Viñolas-Prat X, Soriano-Colomé T, Muñoz-Aguilera R, Plaza I, Curcio-Ruigómez A, Orts-Soler E, Segovia-Cubero J, Fanjul V, Marín-Corral J, Cequier Á, SAVANA Research Group . Impact of Advanced Age on the Incidence of Major Adverse Cardiovascular Events in Patients with Type 2 Diabetes Mellitus and Stable Coronary Artery Disease in a Real-World Setting in Spain. Journal of Clinical Medicine 2023;12(16):5218 View
  28. Muñoz A, Souto J, Lecumberri R, Obispo B, Sanchez A, Aparicio J, Aguayo C, Gutierrez D, Palomo A, Fanjul V, del Rio-Bermudez C, Viñuela-Benéitez M, Hernández-Presa M. Development of a predictive model of venous thromboembolism recurrence in anticoagulated cancer patients using machine learning. Thrombosis Research 2023;228:181 View
  29. Morena D, Izquierdo J, Rodríguez J, Cuesta J, Benavent M, Perralejo A, Rodríguez J. The Clinical Profile of Patients with COPD Is Conditioned by Age. Journal of Clinical Medicine 2023;12(24):7595 View
  30. Benavent D, Muñoz-Fernández S, De la Morena I, Fernández-Nebro A, Marín-Corral J, Castillo Rosa E, Taberna M, Sanabra C, Sastre C. Using natural language processing to explore characteristics and management of patients with axial spondyloarthritis and psoriatic arthritis treated under real-world conditions in Spain: SpAINET study. Therapeutic Advances in Musculoskeletal Disease 2023;15 View
  31. Larrainzar-Garijo R, Fernández-Tormos E, Collado-Escudero C, Alcantud Ibáñez M, Oñorbe-San Francisco F, Marin-Corral J, Casadevall D, Donaire-Gonzalez D, Martínez-Sanchez L, Cabal-Hierro L, Benavent D, Brañas F. Predictive model for a second hip fracture occurrence using natural language processing and machine learning on electronic health records. Scientific Reports 2024;14(1) View
  32. Román Ivorra J, Trallero-Araguas E, Lopez Lasanta M, Cebrián L, Lojo L, López-Muñíz B, Fernández-Melon J, Núñez B, Silva-Fernández L, Veiga Cabello R, Ahijado P, De la Morena Barrio I, Costas Torrijo N, Safont B, Ornilla E, Restrepo J, Campo A, Andreu J, Díez E, López Robles A, Bollo E, Benavent D, Vilanova D, Luján Valdés S, Castellanos-Moreira R. Prevalence and clinical characteristics of patients with rheumatoid arthritis with interstitial lung disease using unstructured healthcare data and machine learning. RMD Open 2024;10(1):e003353 View
  33. de Sequera P, Arias J, Quiroga B, Benavent M, Procaccini F, Romero I, López G, Diez J, Ortiz A. Cardiovascular risk assessment: Missing albuminuria contributing to gender inequality. Nefrología 2024 View
  34. Meczner A, Cohen N, Qureshi A, Reza M, Sutaria S, Blount E, Bagyura Z, Malak T. Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics. JMIR Formative Research 2024;8:e49907 View
  35. Calleja-Panero J, Esteban Mur R, Jarque I, Romero-Gómez M, Group S, García Labrador L, González Calvo J. Chronic liver disease-associated severe thrombocytopenia in Spain: Results from a retrospective study using machine learning and natural language processing. Gastroenterología y Hepatología (English Edition) 2024;47(3):236 View
  36. Benavent D, Benavent-Núñez M, Marin-Corral J, Arias-Manjón J, Navarro-Compán V, Taberna M, Salcedo I, Peiteado D, Carmona L, de Miguel E. Natural language processing to identify and characterize spondyloarthritis in clinical practice. RMD Open 2024;10(2):e004302 View
  37. Cosín-Sales J, Anguita M, Suárez C, Arias-Cabrales C, Martínez-Sanchez L, Arumi D, Fernández de Cabo S. Oral anticoagulant treatment in atrial fibrillation: the AFIRMA real-world study using natural language processing and machine learning. Revista Clínica Española (English Edition) 2024;224(7):428 View
  38. Cosín-Sales J, Anguita M, Suárez C, Arias-Cabrales C, Martínez-Sanchez L, group S, Arumi D, Fernández de Cabo S. Tratamiento anticoagulante oral en la fibrilación auricular: AFIRMA, el estudio de vida real realizado mediante procesamiento de lenguaje natural y aprendizaje automático. Revista Clínica Española 2024;224(7):428 View
  39. Iglesias P, Arias J, López G, Romero I, Díez J. Integration of big data analytics in the investigation of the relationship between acromegaly and cancer. Endocrinología, Diabetes y Nutrición 2024;71(8):324 View
  40. Antón M, Molina M, Pérez J, Pina S, Tapiador N, De La Calle B, Martínez M, Ortega P, Ruspaggiari M, Tudela C, Conejo M, Leno P, López M, Marhuenda C, Arias-Cabrales C, Maisonobe P, Herrera A, Candau E. Botulinum Toxin Type A (BoNT-A) Use for Post-Stroke Spasticity: A Multicenter Study Using Natural Language Processing and Machine Learning. Toxins 2024;16(8):340 View
  41. Muñoz Martín A, Lecumberri R, Souto J, Obispo B, Sanchez A, Aparicio J, Aguayo C, Gutierrez D, García Palomo A, Benavent D, Taberna M, Viñuela-Benéitez M, Arumi D, Hernández-Presa M. Prediction model for major bleeding in anticoagulated patients with cancer-associated venous thromboembolism using machine learning and natural language processing. Clinical and Translational Oncology 2024 View
  42. Iglesias P, Arias J, López G, Romero I, Díez J. Integration of big data analytics in the investigation of the relationship between acromegaly and cancer. Endocrinología, Diabetes y Nutrición (English ed.) 2024;71(8):324 View
  43. Cosín-Sales J, Anguita Sánchez M, Suárez C, Arias-Cabrales C, Martínez-Sanchez L, Savana Research Group S, Arumi D, Fernández de Cabo S. Safety and Effectiveness of Oral Anticoagulants in Atrial Fibrillation: Real-World Insights Using Natural Language Processing and Machine Learning. Journal of Clinical Medicine 2024;13(20):6226 View