This paper is in the following e-collection/theme issue:

Machine Learning (1465) Nosocomial Infections (20) Infectious Diseases (non-STD/STI) (1575)

Published on in Vol 8, No 7 (2020): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/15182, first published .
Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study

Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study

Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study

Authors of this article:

Mark P Sendak1 Author Orcid Image ;   William Ratliff1 Author Orcid Image ;   Dina Sarro2 Author Orcid Image ;   Elizabeth Alderton2 Author Orcid Image ;   Joseph Futoma3, 4 Author Orcid Image ;   Michael Gao1 Author Orcid Image ;   Marshall Nichols1 Author Orcid Image ;   Mike Revoir1 Author Orcid Image ;   Faraz Yashar3 Author Orcid Image ;   Corinne Miller2 Author Orcid Image ;   Kelly Kester2 Author Orcid Image ;   Sahil Sandhu5 Author Orcid Image ;   Kristin Corey1, 6 Author Orcid Image ;   Nathan Brajer1, 6 Author Orcid Image ;   Christelle Tan1, 6 Author Orcid Image ;   Anthony Lin1, 6 Author Orcid Image ;   Tres Brown7 Author Orcid Image ;   Susan Engelbosch7 Author Orcid Image ;   Kevin Anstrom8 Author Orcid Image ;   Madeleine Clare Elish9 Author Orcid Image ;   Katherine Heller3, 10 Author Orcid Image ;   Rebecca Donohoe11 Author Orcid Image ;   Jason Theiling11 Author Orcid Image ;   Eric Poon7, 12 Author Orcid Image ;   Suresh Balu1, 6 Author Orcid Image ;   Armando Bedoya7, 13 Author Orcid Image ;   Cara O'Brien12 Author Orcid Image
Article Authors Cited by (99) Tweetations (37) Metrics

Journals

  1. Sandhu S, Lin A, Brajer N, Sperling J, Ratliff W, Bedoya A, Balu S, O'Brien C, Sendak M. Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study. Journal of Medical Internet Research 2020;22(11):e22421 View
  2. Jia Y, Lawton T, Burden J, McDermid J, Habli I. Safety-driven design of machine learning for sepsis treatment. Journal of Biomedical Informatics 2021;117:103762 View
  3. Shung D, Sung J. Challenges of developing artificial intelligence‐assisted tools for clinical medicine. Journal of Gastroenterology and Hepatology 2021;36(2):295 View
  4. Santus E, Marino N, Cirillo D, Chersoni E, Montagud A, Santuccione Chadha A, Valencia A, Hughes K, Lindvall C. Artificial Intelligence–Aided Precision Medicine for COVID-19: Strategic Areas of Research and Development. Journal of Medical Internet Research 2021;23(3):e22453 View
  5. Beldhuis I, Marapin R, Jiang Y, Simões de Souza N, Georgiou A, Kaufmann T, Castela Forte J, van der Horst I. Cognitive biases, environmental, patient and personal factors associated with critical care decision making: A scoping review. Journal of Critical Care 2021;64:144 View
  6. Smith M, Adelaine S, Bednarz L, Patterson B, Pothof J, Liao F. Predictive Solutions in Learning Health Systems: The Critical Need to Systematize Implementation of Prediction to Action to Intervention. NEJM Catalyst 2021;2(5) View
  7. Wu M, Du X, Gu R, Wei J. Artificial Intelligence for Clinical Decision Support in Sepsis. Frontiers in Medicine 2021;8 View
  8. Patel B, Steinberg E, Pfohl S, Shah N. Learning decision thresholds for risk stratification models from aggregate clinician behavior. Journal of the American Medical Informatics Association 2021;28(10):2258 View
  9. Marabelli M, Newell S, Handunge V. The lifecycle of algorithmic decision-making systems: Organizational choices and ethical challenges. The Journal of Strategic Information Systems 2021;30(3):101683 View
  10. Gonem S, Taylor A, Figueredo G, Forster S, Quinlan P, Garibaldi J, McKeever T, Shaw D. Dynamic early warning scores for predicting clinical deterioration in patients with respiratory disease. Respiratory Research 2022;23(1) View
  11. Singh H, Mhasawade V, Chunara R, Pollard T. Generalizability challenges of mortality risk prediction models: A retrospective analysis on a multi-center database. PLOS Digital Health 2022;1(4):e0000023 View
  12. Nandi M, Anton M, Lyle J. Cardiovascular waveforms - can we extract more from routine signals?. JRSM Cardiovascular Disease 2022;11 View
  13. Galsgaard A, Doorschodt T, Holten A, Müller F, Ploug Boesen M, Maas M. Artificial intelligence and multidisciplinary team meetings; a communication challenge for radiologists' sense of agency and position as spider in a web?. European Journal of Radiology 2022;155:110231 View
  14. Sendak M, Vidal D, Trujillo S, Singh K, Liu X, Balu S. Editorial: Surfacing best practices for AI software development and integration in healthcare. Frontiers in Digital Health 2023;5 View
  15. Su Z, He L, Jariwala S, Zheng K, Chen Y. "What is Your Envisioned Future?": Toward Human-AI Enrichment in Data Work of Asthma Care. Proceedings of the ACM on Human-Computer Interaction 2022;6(CSCW2):1 View
  16. van der Meijden S, de Hond A, Thoral P, Steyerberg E, Kant I, Cinà G, Arbous M. Intensive Care Unit Physicians’ Perspectives on Artificial Intelligence–Based Clinical Decision Support Tools: Preimplementation Survey Study. JMIR Human Factors 2023;10:e39114 View
  17. Dutta S, McEvoy D, Rubins D, Dighe A, Filbin M, Rhee C. Clinical decision support improves blood culture collection before intravenous antibiotic administration in the emergency department. Journal of the American Medical Informatics Association 2022;29(10):1705 View
  18. Chen J, Baxter S, van den Brandt A, Lieu A, Camp A, Do J, Welsbie D, Moghimi S, Christopher M, Weinreb R, Zangwill L. Usability and Clinician Acceptance of a Deep Learning-Based Clinical Decision Support Tool for Predicting Glaucomatous Visual Field Progression. Journal of Glaucoma 2023;32(3):151 View
  19. Sharma M, Savage C, Nair M, Larsson I, Svedberg P, Nygren J. Artificial Intelligence Applications in Health Care Practice: Scoping Review. Journal of Medical Internet Research 2022;24(10):e40238 View
  20. Schwartz J, Moy A, Rossetti S, Elhadad N, Cato K. Response to: Looking for clinician involvement under the wrong lamp post: the need for collaboration measures. Journal of the American Medical Informatics Association 2021;28(11):2543 View
  21. Sikstrom L, Maslej M, Hui K, Findlay Z, Buchman D, Hill S. Conceptualising fairness: three pillars for medical algorithms and health equity. BMJ Health & Care Informatics 2022;29(1):e100459 View
  22. Chan S, Lee J, Ong M, Siddiqui F, Graves N, Ho A, Liu N. Implementation of Prediction Models in the Emergency Department from an Implementation Science Perspective—Determinants, Outcomes, and Real-World Impact: A Scoping Review. Annals of Emergency Medicine 2023;82(1):22 View
  23. Chomutare T, Tejedor M, Svenning T, Marco-Ruiz L, Tayefi M, Lind K, Godtliebsen F, Moen A, Ismail L, Makhlysheva A, Ngo P. Artificial Intelligence Implementation in Healthcare: A Theory-Based Scoping Review of Barriers and Facilitators. International Journal of Environmental Research and Public Health 2022;19(23):16359 View
  24. ter Avest E, Carenzo L, Lendrum R, Christian M, Lyon R, Coniglio C, Rehn M, Lockey D, Perkins Z. Advanced interventions in the pre-hospital resuscitation of patients with non-compressible haemorrhage after penetrating injuries. Critical Care 2022;26(1) View
  25. Zhang M, Gao Q, Gupta S. Online Course Model of Social and Political Education Using Deep Learning. Computational Intelligence and Neuroscience 2022;2022:1 View
  26. Bai E, Song S, Fraser H, Ranney M. A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care. Applied Clinical Informatics 2022;13(01):056 View
  27. Zając H, Li D, Dai X, Carlsen J, Kensing F, Andersen T. Clinician-Facing AI in the Wild: Taking Stock of the Sociotechnical Challenges and Opportunities for HCI. ACM Transactions on Computer-Human Interaction 2023;30(2):1 View
  28. Otaigbe I. Scaling up artificial intelligence to curb infectious diseases in Africa. Frontiers in Digital Health 2022;4 View
  29. Sandhu S, Sendak M, Ratliff W, Knechtle W, Fulkerson W, Balu S. Accelerating health system innovation: principles and practices from the Duke Institute for Health Innovation. Patterns 2023;4(4):100710 View
  30. Mainz J, Munch L, Bjerring J, Godtfredsen S. Why algorithmic speed can be more important than algorithmic accuracy. Clinical Ethics 2023;18(2):161 View
  31. Tulk Jesso S, Kelliher A, Sanghavi H, Martin T, Henrickson Parker S. Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review. Frontiers in Psychology 2022;13 View
  32. Stern A, Goldfarb A, Minssen T, Price II W. AI Insurance: How Liability Insurance Can Drive the Responsible Adoption of Artificial Intelligence in Health Care. NEJM Catalyst 2022;3(4) View
  33. McDermott M, Nestor B, Szolovits P. Clinical Artificial Intelligence. Clinics in Laboratory Medicine 2023;43(1):29 View
  34. Lui T, Cheung K, Leung W. Machine learning models in the prediction of 1-year mortality in patients with advanced hepatocellular cancer on immunotherapy: a proof-of-concept study. Hepatology International 2022;16(4):879 View
  35. Henry K, Adams R, Parent C, Soleimani H, Sridharan A, Johnson L, Hager D, Cosgrove S, Markowski A, Klein E, Chen E, Saheed M, Henley M, Miranda S, Houston K, Linton R, Ahluwalia A, Wu A, Saria S. Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nature Medicine 2022;28(7):1447 View
  36. Morris A, Horvat C, Stagg B, Grainger D, Lanspa M, Orme J, Clemmer T, Weaver L, Thomas F, Grissom C, Hirshberg E, East T, Wallace C, Young M, Sittig D, Suchyta M, Pearl J, Pesenti A, Bombino M, Beck E, Sward K, Weir C, Phansalkar S, Bernard G, Thompson B, Brower R, Truwit J, Steingrub J, Hiten R, Willson D, Zimmerman J, Nadkarni V, Randolph A, Curley M, Newth C, Lacroix J, Agus M, Lee K, deBoisblanc B, Moore F, Evans R, Sorenson D, Wong A, Boland M, Dere W, Crandall A, Facelli J, Huff S, Haug P, Pielmeier U, Rees S, Karbing D, Andreassen S, Fan E, Goldring R, Berger K, Oppenheimer B, Ely E, Pickering B, Schoenfeld D, Tocino I, Gonnering R, Pronovost P, Savitz L, Dreyfuss D, Slutsky A, Crapo J, Pinsky M, James B, Berwick D. Computer clinical decision support that automates personalized clinical care: a challenging but needed healthcare delivery strategy. Journal of the American Medical Informatics Association 2022;30(1):178 View
  37. Shashikumar S, Wardi G, Malhotra A, Nemati S. Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”. npj Digital Medicine 2021;4(1) View
  38. Nazir T, Mushhood Ur Rehman M, Asghar M, Kalia J. Artificial intelligence assisted acute patient journey. Frontiers in Artificial Intelligence 2022;5 View
  39. Ehrmann D, Harish V, Morgado F, Rosella L, Johnson A, Mema B, Mazwi M. Ignorance Isn't Bliss: We Must Close the Machine Learning Knowledge Gap in Pediatric Critical Care. Frontiers in Pediatrics 2022;10 View
  40. Arbelaez Ossa L, Rost M, Lorenzini G, Shaw D, Elger B. A smarter perspective: Learning with and from AI-cases. Artificial Intelligence in Medicine 2023;135:102458 View
  41. Ulloa M, Rothrock B, Ahmad F, Jacobs M. Invisible clinical labor driving the successful integration of AI in healthcare. Frontiers in Computer Science 2022;4 View
  42. Harris S, Bonnici T, Keen T, Lilaonitkul W, White M, Swanepoel N. Clinical deployment environments: Five pillars of translational machine learning for health. Frontiers in Digital Health 2022;4 View
  43. Bernstam E, Shireman P, Meric‐Bernstam F, N. Zozus M, Jiang X, Brimhall B, Windham A, Schmidt S, Visweswaran S, Ye Y, Goodrum H, Ling Y, Barapatre S, Becich M. Artificial intelligence in clinical and translational science: Successes, challenges and opportunities. Clinical and Translational Science 2022;15(2):309 View
  44. Kashyap S, Morse K, Patel B, Shah N. A survey of extant organizational and computational setups for deploying predictive models in health systems. Journal of the American Medical Informatics Association 2021;28(11):2445 View
  45. Sendak M, Gao M, Ratliff W, Nichols M, Bedoya A, O’Brien C, Balu S. Looking for clinician involvement under the wrong lamp post: The need for collaboration measures. Journal of the American Medical Informatics Association 2021;28(11):2541 View
  46. Verma A, Pou-Prom C, McCoy L, Murray J, Nestor B, Bell S, Mourad O, Fralick M, Friedrich J, Ghassemi M, Mamdani M. Developing and Validating a Prediction Model For Death or Critical Illness in Hospitalized Adults, an Opportunity for Human-Computer Collaboration. Critical Care Explorations 2023;5(5):e0897 View
  47. van der Vegt A, Scott I, Dermawan K, Schnetler R, Kalke V, Lane P. Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework. Journal of the American Medical Informatics Association 2023;30(7):1349 View
  48. van der Vegt A, Scott I, Dermawan K, Schnetler R, Kalke V, Lane P. Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework. Journal of the American Medical Informatics Association 2023;30(9):1503 View
  49. Ghosh P, Posner K, Hyland S, van Cleve W, Bristow M, Long D, Palla K, Nair B, Fong C, Pauldine R, Vavilala M, O'Hara K. Framing Machine Learning Opportunities for Hypotension Prediction in Perioperative Care: A Socio-technical Perspective. ACM Transactions on Computer-Human Interaction 2023;30(5):1 View
  50. Aslan A, Permana B, Harris P, Naidoo K, Pienaar M, Irwin A. The Opportunities and Challenges for Artificial Intelligence to Improve Sepsis Outcomes in the Paediatric Intensive Care Unit. Current Infectious Disease Reports 2023;25(11):243 View
  51. Nair M, Andersson J, Nygren J, Lundgren L. Barriers and Enablers for Implementation of an Artificial Intelligence–Based Decision Support Tool to Reduce the Risk of Readmission of Patients With Heart Failure: Stakeholder Interviews. JMIR Formative Research 2023;7:e47335 View
  52. Rigg J, Doyle O, McDonogh N, Leavitt N, Ali R, Son A, Kreter B. Finding undiagnosed patients with hepatitis C virus: an application of machine learning to US ambulatory electronic medical records. BMJ Health & Care Informatics 2023;30(1):e100651 View
  53. Ahmed M, Spooner B, Isherwood J, Lane M, Orrock E, Dennison A. A Systematic Review of the Barriers to the Implementation of Artificial Intelligence in Healthcare. Cureus 2023 View
  54. Price W, Sendak M, Balu S, Singh K. Enabling collaborative governance of medical AI. Nature Machine Intelligence 2023;5(8):821 View
  55. Davis S, Matheny M, Balu S, Sendak M. A framework for understanding label leakage in machine learning for health care. Journal of the American Medical Informatics Association 2023;31(1):274 View
  56. Nghiem J, Adler D, Estrin D, Livesey C, Choudhury T. Understanding Mental Health Clinicians’ Perceptions and Concerns Regarding Using Passive Patient-Generated Health Data for Clinical Decision-Making: Qualitative Semistructured Interview Study. JMIR Formative Research 2023;7:e47380 View
  57. Susanto A, Lyell D, Widyantoro B, Berkovsky S, Magrabi F. Effects of machine learning-based clinical decision support systems on decision-making, care delivery, and patient outcomes: a scoping review. Journal of the American Medical Informatics Association 2023;30(12):2050 View
  58. Verma A, Trbovich P, Mamdani M, Shojania K. Grand rounds in methodology: key considerations for implementing machine learning solutions in quality improvement initiatives. BMJ Quality & Safety 2024;33(2):121 View
  59. Antweiler D, Albiez D, Bures D, Hosters B, Jovy-Klein F, Nickel K, Reibel T, Schramm J, Sander J, Antons D, Diehl A. Einsatz von KI-basierten Anwendungen durch Krankenhauspersonal: Aufgabenprofile und Qualifizierungsbedarfe. Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz 2024;67(1):66 View
  60. van Velzen M, de Graaf-Waar H, Ubert T, van der Willigen R, Muilwijk L, Schmitt M, Scheper M, van Meeteren N. 21st century (clinical) decision support in nursing and allied healthcare. Developing a learning health system: a reasoned design of a theoretical framework. BMC Medical Informatics and Decision Making 2023;23(1) View
  61. Tang K, Xiang Y, Tian J, Hou J, Chen X, Wang X, Zhong Z. Machine Learning-Based Morphological and Mechanical Prediction of Kirigami-Inspired Active Composites. International Journal of Mechanical Sciences 2023:108956 View
  62. Terranova C, Cestonaro C, Fava L, Cinquetti A. AI and professional liability assessment in healthcare. A revolution in legal medicine?. Frontiers in Medicine 2024;10 View
  63. Boussina A, Shashikumar S, Malhotra A, Owens R, El-Kareh R, Longhurst C, Quintero K, Donahue A, Chan T, Nemati S, Wardi G. Impact of a deep learning sepsis prediction model on quality of care and survival. npj Digital Medicine 2024;7(1) View
  64. Macrae C. Managing risk and resilience in autonomous and intelligent systems: Exploring safety in the development, deployment, and use of artificial intelligence in healthcare. Risk Analysis 2024 View
  65. Kamran F, Tjandra D, Heiler A, Virzi J, Singh K, King J, Valley T, Wiens J. Evaluation of Sepsis Prediction Models before Onset of Treatment. NEJM AI 2024;1(3) View
  66. Kwong J, Nickel G, Wang S, Kvedar J. Integrating artificial intelligence into healthcare systems: more than just the algorithm. npj Digital Medicine 2024;7(1) View
  67. Pushkaran A, Arabi A. From understanding diseases to drug design: can artificial intelligence bridge the gap?. Artificial Intelligence Review 2024;57(4) View
  68. Younas A, Reynolds S. Leveraging Artificial Intelligence for Expediting Implementation Efforts. Creative Nursing 2024;30(2):111 View
  69. Yamamoto Y, Muñoz A, Sandström K. Practical Aspects of Designing a Human-centred AI System in Manufacturing. Procedia Computer Science 2024;232:2626 View
  70. Boag W, Hasan A, Kim J, Revoir M, Nichols M, Ratliff W, Gao M, Zilberstein S, Samad Z, Hoodbhoy Z, Ali M, Khan N, Patel M, Balu S, Sendak M. The algorithm journey map: a tangible approach to implementing AI solutions in healthcare. npj Digital Medicine 2024;7(1) View
  71. Balagopalan A, Baldini I, Celi L, Gichoya J, McCoy L, Naumann T, Shalit U, van der Schaar M, Wagstaff K, Badawi O. Machine learning for healthcare that matters: Reorienting from technical novelty to equitable impact. PLOS Digital Health 2024;3(4):e0000474 View
  72. Rehman A, Li M, Wu B, Ali Y, Rasheed S, Shaheen S, Liu X, Luo R, Zhang J. Role of Artificial Intelligence in Revolutionizing Drug Discovery. Fundamental Research 2024 View
  73. Petrella R. The AI Future of Emergency Medicine. Annals of Emergency Medicine 2024;84(2):139 View
  74. Cheng R, Aggarwal A, Chakraborty A, Harish V, McGowan M, Roy A, Szulewski A, Nolan B. Implementation considerations for the adoption of artificial intelligence in the emergency department. The American Journal of Emergency Medicine 2024;82:75 View
  75. Botha N, Ansah E, Segbedzi C, Dumahasi V, Maneen S, Kodom R, Tsedze I, Akoto L, Atsu F. Artificial intelligent tools: evidence-mapping on the perceived positive effects on patient-care and confidentiality. BMC Digital Health 2024;2(1) View
  76. Griffin A, Wang K, Leung T, Facelli J. Recommendations to promote fairness and inclusion in biomedical AI research and clinical use. Journal of Biomedical Informatics 2024;157:104693 View
  77. Rajagopal A, Ayanian S, Ryu A, Qian R, Legler S, Peeler E, Issa M, Coons T, Kawamoto K. Machine Learning Operations in Health Care: A Scoping Review. Mayo Clinic Proceedings: Digital Health 2024;2(3):421 View
  78. Cary M, Bessias S, McCall J, Pencina M, Grady S, Lytle K, Economou‐Zavlanos N. Empowering nurses to champion Health equity & BE FAIR: Bias elimination for fair and responsible AI in healthcare. Journal of Nursing Scholarship 2024 View
  79. Kamel Rahimi A, Pienaar O, Ghadimi M, Canfell O, Pole J, Shrapnel S, van der Vegt A, Sullivan C. Implementing AI in Hospitals to Achieve a Learning Health System: Systematic Review of Current Enablers and Barriers. Journal of Medical Internet Research 2024;26:e49655 View
  80. Nair M, Svedberg P, Larsson I, Nygren J, Six S. A comprehensive overview of barriers and strategies for AI implementation in healthcare: Mixed-method design. PLOS ONE 2024;19(8):e0305949 View
  81. Botha N, Segbedzi C, Dumahasi V, Maneen S, Kodom R, Tsedze I, Akoto L, Atsu F, Lasim O, Ansah E. Artificial intelligence in healthcare: a scoping review of perceived threats to patient rights and safety. Archives of Public Health 2024;82(1) View
  82. Handra J, James H, Mbilinyi A, Moller-Hansen A, Andrade J, Deyell M, Hague C, Hawkins N, Ho K, Hu R, Leipsic J, O'Riley C, Tam R. The Role of Machine Learning in the Detection of Cardiac Fibrosis from Electrocardiogram: A Scoping Review (Preprint). JMIR Cardio 2024 View

Books/Policy Documents

  1. Dube S. An Intuitive Exploration of Artificial Intelligence. View
  2. Ehrmann D, Assadi A, Eytan D, Goodfellow S, Goodwin A, Greer R, Schwartz S, Mazwi M. Pediatric and Congenital Cardiology, Cardiac Surgery and Intensive Care. View
  3. Wright M. Clinical Decision Support and Beyond. View
  4. Wu C, Mathur P. Artificial Intelligence in Clinical Practice. View
  5. Marabelli M. AI, Ethics, and Discrimination in Business. View
  6. Manjunath P, Lerner B, Dunn T. Artificial Intelligence in Medicine. View