Intelligent Digital Surgeon – Impulse InnoCH

Intelligent Digital Surgeon

This project aims to design and model a 3D digital surgeon in the existing ORamaVR’s MAGES SDK platform to guide trainees in virtual surgery.  The sms MIRALab sarl is responsible of creating and animating in Real-Time the virtual 3D surgeon. The team at UNIGE-CUI will use the animatable digital surgeon to identify the trainee’s gestures by recognizing and analyzing the trainee’s hand/arm gestures and provides personalized assessment, real-time feedback, instructions, and recommendations.

The digital surgeon’s optimal behavioural model, based on real gestures, will be video recorded with the support of the Hospital of Geneva (HUG). A deep learning model will be derived from recording skilled real surgeons’ gestures to assist the feedback decision engine of the digital surgeon by the UnIversity of Geneva (UNIGE). Furthermore, MAGES system will embed methods of cutting and tearing of physics-based deformable surfaces.

 

Scientific goals:

UNIGE together with HUG and the support of MIRALab sarl will provide a dedicated intelligent digital surgeon (IDS) that is to be used for many different applications as:

1. An animatable digital surgeon

2. A dataset containing the right gestures for performing incisions and sutures

3. A dataset containing the trainee’s gestures

4. A dataset containing the surgeon’ advices to the trainee’s wrong gestures

5. A hand gestures recognition module

6. An analytical tool to compare the gestures done by the trainee and the right gestures from the dataset.

7. An analytical tool that will advise the trainee and show her/him how to perform the right gestures.

Modeling of the animatable digital surgeon

Arms and hands rigging