Intelligent Digital Surgeon – Impulse InnoCH

Intelligent Digital Surgeon

The Intelligent Digital Surgeon is an embodied virtual agent within MAGES medical training platform. Its main objective is to identify and analyze the immersed trainee’s behavioral model and provide personalized real-time feedback, assessment, and recommendations like a real surgical instructor.

Based on an optimal behavioral model, the IDS will present the training module’s actions using the correct gestures. Medical-specific operations will be performed using physics-based 3D representation of deformable soft body objects (skin, tissue, etc.) to achieve high-realism and immersion. The proposed system will adopt a distributed software architecture taking advantage of edge-cloud resources and 5G networking to enable ultra-low latency untethered VR. The IDS’s optimal behavioral model, based on virtual gesture data sets, will be derived at the Hospital of Geneva (HUG).

A deep learning model will be derived to assist the feedback decision engine. A distributed VR system architecture will be designed utilizing edge computing and 5G networks. Furthermore, MAGES system will embed methods of cutting and tearing of physics-based deformable surfaces that advance state-of-the-art.


Scientific goals:

UNIGE together with HUG and the support of MIRALab sarl will provide a dedicated intelligent digital surgeon (IDS) that is to be used for many different applications as:

1. An animatable digital surgeon

2. A dataset containing the right gestures for performing incisions and sutures

3. A dataset containing the trainee’s gestures

4. A dataset containing the surgeon’ advices to the trainee’s wrong gestures

5. A hand gestures recognition module

6. An analytical tool to compare the gestures done by the trainee and the right gestures from the dataset.

7. An analytical tool that will advise the trainee and show her/him how to perform the right gestures.

Modeling of the animatable digital surgeon

Arms and hands rigging