AI in Anesthesia: From Simulation to Real-Time Operating Room Analysis

PI-Technion, Assist. Prof. Shlomi Laufer,
PI-Rambam, Assoc. Prof. Aeyal Raz, MD, Dr. Fadi Mahameed, MD

Objectives and Clinical Need

Anesthesiology involves continuous monitoring, rapid decision-making, and precise execution of critical tasks under time pressure. While simulation-based training plays an important role in preparing clinicians for rare and high-risk scenarios, it captures only part of the complexity of real clinical practice. In the operating room, anesthesiologists must manage dynamic environments, interact with multiple systems, and coordinate actions while maintaining constant awareness of the patient’s physiological state.
There is a growing clinical need for tools that can objectively analyze anesthesiologists’ behavior, both in simulation and in real surgical settings. Such systems could support training and assessment, improve adherence to clinical protocols, and provide insights into how clinicians interact with monitoring systems, medications, and workflows in practice. However, current approaches rely largely on manual observation, which is labor-intensive, subjective, and difficult to scale.

S6

Our joint work between the Technion and Rambam Health Care Campus addresses these challenges by developing AI-driven methods for analyzing anesthesiology practice across both simulation environments and real operating rooms. The goal is to move toward automated, objective, and interpretable analysis of clinical behavior, capturing not only what actions are performed, but also how they are executed over time.

 

Database

To support this research, we collected a comprehensive multimodal dataset that includes both high-fidelity simulation sessions and real-world operating room recordings.

Data collection in the simulation setting was a joint effort between the Technion, Rambam Health Care Campus, and the Israel Society of Anesthesiology. Together, we established an annual, IRB-approved workshop designed to prepare residents for their board examinations. This initiative serves as the foundation of our dataset. To date, we have collected data from more than 150 residents across 17 different medical centers.

During these simulation sessions, anesthesia residents manage acute clinical scenarios such as anaphylaxis, bradycardia, opioid overdose, hypoglycemia, and cardiac arrest. The scenarios are designed by clinical experts and provide controlled yet realistic representations of critical events.

In addition, and as a central component of this project, we deployed a multi-camera system in the operating room. A total of seven cameras were installed to continuously record the work of anesthesiologists during surgery. Using this setup, we collected data from more than 100 surgical procedures, capturing real clinical workflows under natural conditions.

The dataset includes synchronized multimodal data, combining video, audio, and patient monitor signals. This enables analysis of clinician behavior from multiple perspectives, including physical actions, verbal communication, and interaction with medical devices.

 

Research Results

Automated Analysis of Anesthesiology Practice

A central contribution of this research is the development of methods for automatically analyzing anesthesiologists’ behavior from multimodal data. Using both simulation and operating room recordings, we extract structured representations of clinical activity and quantify how clinicians interact with their environment over time.

Based on video data, we analyze several key aspects of anesthesiology practice. This includes observation patterns of the patient monitor, capturing how often and when clinicians attend to vital signs during different phases of care. We also study how anesthesiologists prepare and administer drugs, identifying the sequence and timing of actions involved in medication handling.

To better understand clinical workflows, we apply process mining techniques to the induction phase of anesthesia. This allows us to model typical action sequences, identify variations in practice, and compare observed behavior to expected clinical protocols. By representing clinical activity as structured processes, we can systematically analyze efficiency, consistency, and deviations across cases.

 

Multimodal Assessment and Behavioral Modeling

In parallel, we developed multimodal frameworks that combine video, audio, and physiological data to assess clinician performance. These models capture multiple aspects of behavior, including communication, attention to monitoring systems, movement within the operating room, and interaction with equipment.

Our results show that combining multiple modalities provides a more robust and informative representation of clinical behavior than any single source alone. These methods enable continuous, data-driven assessment of performance and open the door to automated feedback systems that can support training and quality improvement.

 

Insights from Simulation-Based Analysis

Building on our earlier work in simulation, we demonstrated that speech and behavioral signals can be used to automatically identify clinical actions and complete task-based checklists. We also showed that simulation-based assessments may contain inherent biases related to prior exposure to simulation environments, highlighting the importance of objective, data-driven evaluation methods.

These findings complement our operating room analysis by providing controlled settings in which specific behaviors can be studied in detail, while the OR data ensures that our methods generalize to real clinical practice.

 

Overall Impact

Together, these contributions establish a comprehensive framework for analyzing anesthesiology practice across both simulation and real operating room environments. By leveraging multimodal data and advanced AI methods, our work enables objective, scalable, and interpretable analysis of clinical behavior.

This research provides new insights into how anesthesiologists monitor patients, administer treatments, and follow clinical workflows in practice. It supports the development of improved training methods, more reliable assessment tools, and data-driven approaches to quality assurance.

Looking forward, these methods lay the foundation for real-time clinical support systems that can assist anesthesiologists during surgery, enhance situational awareness, and improve patient safety.

 

Publications

  • Gershov S, Mahameed F, Raz A, Laufer S. More Than Meets the Eye: Analyzing Anesthesiologists’ Visual Attention in the Operating Room Using Deep Learning Models. International Workshop on PRedictive Intelligence In MEdicine, 2023, p. 253–64.
    https://doi.org/10.1007/978-3-031-46005-0_22
  • Gershov S, Raz A, Karpas E, Laufer S. Towards an autonomous clinical decision support system. Eng Appl Artif Intell.
    https://doi.org/10.1016/J.ENGAPPAI.2023.107215
  • Gershov S, Mahameed F, Raz A, Laufer S. Inherent bias in simulation-based assessment. Br J Anaesth 2025; Gershov, S., Mahameed, F., Raz, A., & Laufer, S. (2025). Inherent bias in simulation-based assessment. British journal of anaesthesia, 134(5), 1531–1533.
    https://doi.org/10.1016/j.bja.2024.10.044
  • Gershov S, Mahameed F, Raz A, Laufer S. Towards accurate and interpretable competency-based assessment: enhancing clinical competency assessment through multimodal AI and anomaly detection. NPJ Digit Med 2026.
    https://doi.org/10.1038/s41746-025-02299-2
  • Nissan A, Mahameed F, Gershov S, Raz A, Laufer S. Efficient computer vision pipeline for automated anesthetic injection documentation. Computer Assisted Surgery 2025;30.
    https://doi.org/10.1080/24699322.2025.2582020
  • Gershov S, Braunold D, Spektor R, Ioscovich A, Raz A, Laufer S. Automating medical simulations. J Biomed Inform 2023;
    https://doi.org/10.1016/j.jbi.2023.104446

 

Audio pipeline for automatic checklist completion

Figure 5. Audio pipeline for automatic checklist completion in anesthesiology simulation.