Clinical use of augmented reality, mixed reality, three-dimensional-navigation and artificial intelligence in liver surgery

Abstract

A precise knowledge of intra-parenchymal vascular and biliary architecture and the location of lesions in relation to the complex anatomy is indispensable to perform liver surgery. Therefore, virtual three-dimensional (3D)-reconstruction models from computed tomography/magnetic resonance imaging scans of the liver might be helpful for visualization. Augmented reality, mixed reality and 3D-navigation could transfer such 3D-image data directly into the operation theater to support the surgeon. This review examines the literature about the clinical and intraoperative use of these image guidance techniques in liver surgery and provides the reader with the opportunity to learn about these techniques. Augmented reality and mixed reality have been shown to be feasible for the use in open and minimally invasive liver surgery. 3D-navigation facilitated targeting of intraparenchymal lesions. The existing data is limited to small cohorts and description about technical details e.g., accordance between the virtual 3D-model and the real liver anatomy. Randomized controlled trials regarding clinical data or oncological outcome are not available. Up to now there is no intraoperative application of artificial intelligence in liver surgery. The usability of all these sophisticated image guidance tools has still not reached the grade of immersion which would be necessary for a widespread use in the daily surgical routine. Although there are many challenges, augmented reality, mixed reality, 3D-navigation and artificial intelligence are emerging fields in hepato-biliary surgery.

Key Words: Augmented reality, Mixed reality, 3D, Navigation, Artificial intelligence, Liver surgery, Liver resection, Image guided surgery

Core Tip: Virtual three-dimensional (3D)-reconstruction models from computed tomography/magnetic resonance imaging scans of the liver might be helpful for visualization during liver surgery. Augmented reality, mixed reality and 3D-navigation could transfer such 3D-image data directly into the operation theater. Augmented reality and mixed reality have been shown to be feasible for the use in open and in minimally invasive liver surgery. 3D-navigation facilitated targeting of intraparenchymal lesions. Randomized controlled trials regarding clinical data or oncological outcome are not available. Up to now there is no intraoperative application of artificial intelligence in liver surgery. The usability of all these sophisticated image guidance tools has still not reached the grade of immersion which would be necessary for a widespread use in the daily surgical routine.

INTRODUCTION

The surgical liver anatomy is defined not only by external landmarks but more important by its three-dimensional (3D) intra-parenchymal vascular and biliary architecture. It shows a high-grade of variation in each individual, making liver anatomy even more complex. In addition, liver lesions are often located intraparenchymally, which makes them invisible for the surgeon. Therefore, a high grade of anatomical knowledge before and during hepato-biliary surgery is directly related to the post-operative success and outcome for the patients[1]. Especially spatial, 3D-orientation is of utmost importance in the liver: (1) For pre-surgical localization of intrahepatic lesion; (2) For exact planning of the resection line; and (3) For intraoperative identification of the lesions and orientation during the parenchyma dissection. Hence hepato-biliary surgeons have been ambitious to use computer and image guidance techniques to facilitate preoperative planning and intraoperative procedures. Computer-assisted 3D-segmentation and -reconstruction techniques have helped to transfer 2-dimensional (2D) images, slices, of the liver from a computed tomography (CT)- or magnetic resonance imaging (MRI) -scan back to a 3D structure familiar to the surgeon’s perception of the real anatomy. First applications of segmentation and virtual 3D-reconstruction of the liver dated from the early 90s of the last century[2,3]. Summarized under the term “virtual hepatectomy” this 3D-preoperative liver segmentation technique has improved outcome after major liver resection and living liver donation. It has become a standard procedure at specialized liver centers[4,5]. The next step was to transfer the preoperative reconstructed 3D-images into the operative theater- followed by early applications of intraoperative navigation with stereotactic systems[6]. The additional 3D-image information was presented on a secondary screen and the surgeon has to mentally merge the real live situation and the virtual 3D reconstruction of the liver. In the 1990s years the challenges became even greater[7] with the introduction of laparoscopic liver surgery. “Virtuality” has emerged to liver surgery: Performing the laparoscopic operation only according to a displayed 2D image. Years later passive-polarizing 3D display techniques reintroduced spatial orientation into minimally invasive surgery and has shown to improve the surgical performance[8,9].

“Augmented reality” (AR) or “mixed reality” (MR) is created by superimposing the virtual 3D model of the liver on the laparoscopic screen or directly on the liver. At this point the fusion between image data and real-world anatomy took place - which was performed up to that point in the surgeon’s mind. AR/MR should facilitate this procedure and so the surgical process. A key factor to achieve this is calibration and registration, which means to match the 3D liver model and the real liver to create AR/MR. This is still a major source of error[10]. Artificial intelligence (AI) might be the next step in liver surgery. It has the potential to help the surgeon to identify anatomical structures. One novel way to integrate AI in liver surgery could be achieved by automatic tissue recognition according to the laparoscopic image and image fusion with the virtual 3D model.

Aim of this review is to evaluate the clinical usage of AR and MR, 3D-navigation and AI in liver surgery.

For the comprehensive literature review utilizing MEDLINE (PubMed) was performed using the search terms “mixed reality liver”, ”augmented reality liver”, “navigation liver”, “artificial intelligence surgery“ and “artificial intelligence liver” (publication date from January 1991 until January 2021). Only articles in English language were considered. Review articles were excluded. The query retrieved in total 450 publications. Duplicates were identified by Endnote leaving 433 citations for review. The headlines and abstracts of those citation were reviewed manually. Finally, 44 citations were considered relevant to the topic.

TECHNIQUES TO CREATE AR AND MR IN OPEN LIVER SURGERY

While using AR/MR the first step is to perform the segmentation and reconstruction of a virtual 3D liver model out of the 2D-CT/MRI scan. After that this 3D model must be superimposed intraoperatively onto the liver. Therefore, a registration and calibration process must be performed: Anatomical landmarks of the liver must be identified and then matched to the corresponding points on the virtual 3D model. The accuracy between 3D model and the real-life anatomical structures is determined by the precision of this registration process. Anatomical landmarks on the surface of the liver and/or vascular structures defined by intraoperative ultrasound can be obtained[10] for the registration process.

In open liver surgery AR and MR could be realized using different techniques: (1) The virtual 3D model is projected on the surface of the liver or the abdominal wall; (2) The liver is visualized through a scope and displayed on a secondary screen (“open laparoscopy”). On that screen the virtual 3D model is superimposed on the image of the real liver. Using this technique, the surgeon has to look away from the operative field to use the AR/MR model; (3) the 3D model is superimposed on a semi-transparent display, which is placed between the surgeon and the operative space. The surgeon has to look through this semi-transparent display to see the real liver and to perform the surgery; (4) The liver is visualized through the camera of a tablet pc and the virtual 3D-model was then superimposed onto the liver image on the tablet’s screen; and (5) A so called “hologram” was created on head-mounted semitransparent display. In this setting the surgeon could see the real liver through the semitransparent display (which is worn like glasses) and the “hologram” was superimposed on the semitransparent display using it as a projection screen.

AR AND MR IN OPEN LIVER SURGERY

Visualizing the liver through a scope was a first step of AR/MR in open liver surgery. Onda et al[11] described two cases of liver resection (right hepatectomy and partial hepatectomy), where this technique has been successfully used. However the technique was time consuming: 10 hr for preoperative planning and 3D-model reconstruction, one hour for the intraoperative setup and 1-2 min for the registration process. Data on clinical outcome were not available[11]. Okamoto et al[12] used to create AR/MR with the open scope technique and via a so-called see-through display, which is mounted directly between the surgeon and the operative space[12]. Two hepato-biliary procedures were reported with this technique (bile duct resection, right hepatectomy). Operation time and blood loss were 245 min/242 mL and 530 min/1329 mL respectively. The scope technique to create AR/MR was also used to identify disappeared colorectal liver metastasis after chemotherapy. In three patient this AR/MR technique was used to find and finally resect the tissue of the disappeared metastasis[13]. Using a tablet pc is an easy, state-of-the-art video-based variation of the scope see-through AR/MR technique in open liver surgery. One case is described using this AR technique to perform a left hepatectomy and hepatico-jejunostomy with complex biliary reconstruction for hilar cholangio-carcinoma[14]. Yasuda et al[15] used a comparable technique with a tablet pc as display “in” the operative field combined with the open-scope technique. In a series of eight patients they described an accuracy/registration error between the 3D virtual model and the real liver of 1 mm to 11 mm. Data regarding clinical outcome parameters were not available. Still an unsolved problem using AR/MR and 3D-navigation is the high grade of deformation of the liver during open surgery. The superimposed images could not follow this deformation and the error between the 3D model and real anatomy increases during the process of parenchyma dissection. Golse et al[16] have recently described an AR technique during open liver surgery using a marker less non-rigid registration system. They showed in four patients that registration was possible and the 3D model could be superimposed on the liver following some deformation.

Lately head-mounted semitransparent displays (e.g., Hololens) have been introduced to open liver surgery. With this technique the surgeon can see a so called “hologram” superimposed on the real world and handle it via gesture recognition without the need of an input device (e.g., touchpad or touch screen). It is right now not possible with this technique to match the hologram directly on the real liver - in fact the hologram is projected somewhere in the visual field of the user. A first study evaluated the use of the hololens regarding anatomical identification of liver lesions. Pelani et al[17] could show in an out-of-the-operation-room study including 28 surgeons, that the correct identification of a simulated liver lesion could be performed in 6 s with the Hololens compared to 24 s using the 2D-CT scan of the liver. Saito et al[18] described the intraoperative use of the hologram technique. Here the hologram was superimposed above the operative field. In the first patient with more than 20 colorectal liver metastasis the 3D hologram of the liver was used to identify the liver lesions and to visualize the parenchyma dissection line. In the second case the hologram was used to facilitate the identification of a complex hilar anatomy in order to perform the glisonnean pedicel approach in a patient with an HCC. In this case multiple contributors of the surgical procedures have worn the hololens at the same time (Table 1).

Table 1 Augmented and mixed reality in open liver surgery.

Ref.	No of procedures	Technique	Key outcomes
Onda et al[11], 2013	2 liver resections	Open stereo-scope, AR created on a passive -polarizing 3D display	Open scope technique feasible, 10 hr pre-op image preparation, 1 h intraoperative setup, 1-2 min for registration process
Okamoto et al[12], 2013	2 HPB procedures	Video see-through display	Position of virtual 3D model and organ image closely corresponded, registration error 5 mm
Ntourakis et al[13], 2016	3 patients with 4 disappeared CRLM	Open stereo-scope, AR created on video screen, registration performed by an additional computer technician	AR helped to detect disappeared all metastases, R0, planned security margin 1 cm, registration time within 6 min
Tang et al, 2017[14]	1 patient	AR created on a tablet pc as see-trough display	Feasible, improved vision compared to video based AR system
Yasuda et al[15], 2018	7 patients including minor and major liver resections	Open scope technique combined with AR created on a tablet pc with infrared sensor	Tablet pc method feasible, registration error 1-11 mm
Saito et al[18], 2020	2 HPB procedures	3D hologram on head mounted display	Feasible, orientation improved, multiple surgeons used the technique at the same time, hologram reduced task load

CRLM: Colorectal liver metastasis; HPB: Hepato-biliary; AR: Augmented reality; 3D: Three-dimensional; R0: R0 Resection.

AR AND MR IN LAPAROSCOPIC LIVER SURGERY

In laparoscopic surgery the real-world 3D appearance is transferred into a virtual 2D image on a screen. This leads to a loss of spatial orientation, which is a major challenge. Therefore, anatomical orientation is aggravated. With the use of 3D laparoscopic systems spatial orientation was reintroduced to minimally invasive surgery. This accelerated complex laparoscopic procedures and facilitated them[8,9]. AR and MR could provide precious additional information about the liver anatomy and localization of intrahepatic lesions on the virtual image. Image projection on the abdominal surface for trocar positioning and anatomical orientation was the first level of AR in laparoscopic liver surgery[19]. Volonté et al[19] described in a study with four patients the use of the projection technique: The 3D-modell was projected on the abdominal wall. This early version of AR was used to visualize the anatomy and to place the trocar ports for laparoscopic approaches. In a clinical study on 24 patients this AR image projection technique on the abdominal wall resulted in less deviation between the planned trocar position and the real trocar positions[20]. The next step of AR in minimally invasive surgery was similar to the use in open liver surgery: To place additional image information on the display. The surgeon could see the laparoscopic image and the reconstructed virtual 3D model at the same time on the same screen - but without image fusion[21]. This was followed by image fusion of the virtual 3D model and the laparoscopic image of the liver. The registration and matching process of both to create AR is crucial. As in open surgery this relied on a manual registration by the surgeon. In a feasibility study Schneider et al[22] could show that semi-automatic registration of a superimposed 3D model was feasible in 16 out of 18 patients. This facilitated and speeded the process up, but with lower precision compared to the standard manual registration algorithm. Kang et al[23] described an AR system in an in-vivo porcine model, which could superimpose the intraoperative laparoscopic ultrasound image on the real liver. Therefore they used a stereotactic navigation system and 3D laparoscopic imaging system. In 2015 one case of a trans-thoracic minimally invasive liver resection guided by AR was described. Here the registration process and fusion of the virtual 3D model and the liver anatomy was performed by a specialized computer scientist to ensure accuracy by using visible landmarks on the liver surface corresponding to the virtual 3D model[24].

Robotic platforms for surgery have the potential to integrate multiple additional information into the operation field in the view of the surgeon. Right now, the integration of ultrasound and indocyanine green (ICG) imaging are standard features of robotic surgical platforms. Pessaux et al[25] described in 2015 three cases of a liver segmentectomy supported by superimposed 3D models of the liver. The registration and image fusion were again manually performed by a computer scientist with the help of an additional video mixer[15,25]. Automatic compensation of the laparoscopic motion during AR is another new feature: The location of the 3D model was adapted to the changed perspective of the laparoscope during the resection. In a series of 10 patients this led to an accuracy of 5 mm between the virtual 3D model and the real anatomic position of the liver[26] (Table 2).

Table 2 Augmented and mixed reality in minimally invasive liver surgery.

Ref.	No of procedures	Technique	Key outcomes
Volonté et al[19], 2011	4 procedures	Projection of the virtual 3D model on the body surface	Anatomical orientation and trocar placement improved
López-Mir et al[20], 2013	12 procedures	Projection of the virtual 3D model on the body surface	lower deviation between planned and actual trocar positions using AR
Pessaux et al[25], 2015	2 robotic liver resections	Virtual 3D model superimposed on console display, registration performed manually by a computer scientist	AR and registration process feasible, time to create AR 8 min
Schneider et al[22], 2020	18 laparoscopic liver resections	Passive polarizing 3D laparoscope, optical tracking of the laparoscope, semi-automatic registration	semiautomatic registration an image fusion achieved in 16/18 manual registration vs semiautomatic accuracy 11 mm vs 14 mm

AR: Augmented reality, 3D: Three-dimensional.

AR AND MR FOR 3D NAVIGATION

Preoperative use of a virtual 3D models for planning followed by intraoperative use via AR for orientation leads to the next level of image-guided liver surgery: Intraoperative navigation. A navigation systems should not only visualize the anatomy but also guide the surgeon through the resection and show correlated to the used surgical instruments the location of important anatomical structures, at best before they were visible.

Early versions of navigation systems from the 2000s years often based on intraoperative ultrasound. They were able to guide a needle for thermal ablation into liver lesions[27]. The combination of the ultrasound technique with 3D virtual reconstruction of the liver and stereotactic navigation systems, already known from neurosurgery, followed after that[28]. Beller et al[29] described the clinical use of a navigation system for open liver surgery. The system was based on optical electromagnetic tracking: Marker shields must be placed on the instruments, which were scanned by a camera system placed above the operative space The system used 3D virtual image reconstruction of the liver, matched the 3D image with intraoperative ultrasound and could show the position of the used instruments during liver parenchyma transaction on the virtual 3D image and the ultrasound image[29]. In this early study 32 navigated liver resection were compared to 32 conventional liver resections. The authors could show that in the navigation group the planned dissection line could be maintained with an accuracy of 5 mm. Also, the rate of R1-resection was significantly reduced in the navigation group[29]. The navigation technique was optimized during the following years[30]. Peterhans et al[10] developed a stereotactic navigation system for open liver surgery. This system superimposes the position of the instruments and the ultrasound image on the virtual 3D liver model on a secondary screen. The first clinical evaluation of this system was performed on 9 patients undergoing oncologic liver resection. The optimized workflow of the system resulted in short landmark definition and acquisition times of just one minute, which has made the navigation system ready to use in the operation theatre[10]. The largest cohort of patients that underwent liver resection supported by a 3D navigation system was published by the group from Bern/Switzerland with 65 patients over a period of four years. They combined 3D-navigated liver resection and 3D-navigated thermal ablation in order to perform parenchyma sparing treatment instead of formal major anatomical liver resections. The technical accuracy, matching the virtual 3D model and the real liver, could be optimized to 4.5 mm deviation. They also described a new technique of landmark acquisition and registration: The landmarks on the liver surface were combined with intrahepatic vascular structures acquired by ultrasound[31].

In the following years the electro-magnetic navigation technique was transferred to minimally invasive liver surgery[28,32] and later also combined with AR. On the laparoscopic image of the real liver the 3D virtual model was superimposed and the surgical instruments were tracked and could be navigated in this AR environment[33]. Thenceforth AR, MR and navigation techniques tread a parallel development path[13].

Spatial orientation is especially important in laparoscopic thermal ablation of liver lesions. Tinguely et al[34] showed in a cohort of 54 patients, which were treated with pure laparoscopic 3D navigated microwave ablations a registration accuracy of 8.1mm. Yet, the early local recurrence rate in this cohort was high with 32%. Thomas et al[35] described an optimized system for laparoscopic ultrasound navigated microwave ablation lately. With this navigation tool novices could achieve an accuracy and a speed in targeting defined liver lesion comparable to expert surgeons. In a cohort of 27 patients Aoki et al[36] described the use of a laparoscopic navigation system with instrument tracking. This system displays the position of the instrument on the reconstructed 2D-CT image. As a result of the use of the navigation system a low median tumor margin (R0-Resection) of 9 mm could be achieved. The latest development combining AR and stereotactic 3D navigation in laparoscopic liver surgery was described by Prevost et al[26]. Their navigation system could create an AR overlay of the intrahepatic structures directly around the stereotactic tracked dissection instrument. Ten patients could be successfully operated with the system, showing a calibration time of 9 min for the navigation system with a registration error of 9.2 mm (Figures 1 and 2)[26]. Organ deformation may reduce the precision of the registration and navigation process during the surgical procedure. Updating the navigation information by intraoperative real-time CT image acquisition, using injected fiducials could further minimize the registration error and increase precision in a pre-clinical setting[37] (Table 3).

Figure 1 The use of augmented reality during laparoscopic liver resection using a 3D passive polarizing display technique. The complete virtual three-dimensional model of the liver is visible on a second screen (right picture). On the main screen augmented reality is created (left picture). Citation: Prevost GA, Eigl B, Paolucci I, Rudolph T, Peterhans M, Weber S, Beldi G, Candinas D, Lachenmayer A, Efficiency, Accuracy and Clinical Applicability of a New Image-Guided Surgery System in 3D Laparoscopic Liver Surgery. J Gastrointest Surg 2020; 24(10): 2251-2258, Copyright © The Author(s) 2020, Published by Springer Nature[26].

Figure 2 Directly on the laparoscopic three-dimensional image of the liver there is only that part of the virtual three-dimensional model superimposed on an area, which is relevant for the parenchyma dissection during that phase of the operation. At the area, where the lesion is located and the parenchyma dissection will be performed, the virtual three-dimensional model is matched around the tracked/navigated dissection tool. Citation: Prevost GA, Eigl B, Paolucci I, Rudolph T, Peterhans M, Weber S, Beldi G, Candinas D, Lachenmayer A, Efficiency, Accuracy and Clinical Applicability of a New Image-Guided Surgery System in 3D Laparoscopic Liver Surgery. J Gastrointest Surg 2020, 24(10), 2251-2258, Copyright © The Author(s) 2020, Published by Springer Nature[26].

Table 3 Augmented and mixed reality for 3D Navigation.

Ref.	Number of procedures	Technique	Key outcomes
Beller et al[29], 2007	33 open liver resections	Stereotactic optical navigation system, combined with a virtual 3D model and ultrasound, dissection device tracked and navigated on ultrasound image	Navigation successful in 32/33 cases, difference between projected and actual vascular dissection lever 6mm, R0 resection in 30 cases
Peterhans et al[10], 2011	9 open liver resections	Stereotactic navigation system, combined with a virtual 3D model and ultrasound, landmark acquisition on the liver surface, dissection device tracked and navigated on the virtual 3D model	Navigation successful in all cases, median accuracy 6.3 mm
Banz et al[32], 2016	65 open liver resections	Stereotactic optical navigation system, combined with a virtual 3D model and ultrasound, dissection device tracked and navigated on the virtual 3D model, landmark acquisition with ultrasound possible	Combination of 3 d navigated resection and thermal ablation in 16 patients, accuracy optimized to 4.5 ± 3.6 mm
Tinguely et al[35], 2017	54 laparoscopic image guided microwave ablation	Laparoscopic stereotactic navigation system, combined with a virtual 3D model, landmark acquisition on the liver surface, ablation device tracked and navigated on the virtual 3D model, standard 2D laparoscopic display	Registration time 4:38 min, accuracy 8.1 ± 2.8 mm, early local recurrence rate 32%
Aoki et al[37], 2021	27 laparoscopic liver lesions	virtual real-time CT-guided volume navigation, electromagnetic tracking of the surgical instruments displayedon the preoperatively acquired CT images	Registration time < 2 min, registration error 12 mm, histologic resection margin 9 mm
Prevost et al[26], 2020	10 laparoscopic liver resections	stereotactic augmented reality navigation, virtual 3D liver model superimposed on the real liver with a 3D laparoscopic system, instruments tracked	Registration time 8:50 min, registration error 9.2 mm, facilitates to find disappeared liver lesions

AR: Augmented reality; 3D: Three-dimensional.

FLUORESCENCE GUIDED NAVIGATION TECHNOLOGY AND ROBOTIC PLATFORMS

During the last 10 years the use of real time-fluorescence technique with ICG has been established in open and laparoscopic liver surgery. By easy-to-see intraoperative green fluorescence it could facilitate evaluating the liver anatomy[38], visualize tumor lesions[39] and optimize segmental and subsegmental anatomical resections as well parenchyma dissection in major liver surgery[40,41]. Compared to the above mentioned navigation systems, ICG is more an intraoperative staining technique. It visualizes liver parenchyma or lesions directly through an optical system and “navigates” the surgeon during the operation. Fusion of real time-fluorescence imaging with pre-operative CT-or MRI-data combined with the intraoperative view to create AR would be a further step in navigation technique. Here robotic surgical platforms may become a game-changer, because they create a 3D minimally invasive surgical environment with real-time fluorescence and ultrasound imaging in one display. Adding a virtual 3D model of the liver from preoperative image data, intraoperative navigation could lead to the next level of immersion.

AI

Deformation of the liver tissue is still a major issue for precise registration and the substantial use of navigation and image superimposition during surgery. Convolutional neural networks are able to learn soft tissue behavior, which could be transferred to surgical navigation[42]. Elastic surface based-matching registration algorithms may reduce registration errors[43]. Unfortunately up to now there is no clinical intraoperative use of AI in liver surgery. Aspects of machine learning are integrated in the AR/MR and navigation systems. But automated registration and recognition of anatomical structures of the liver is not available for clinical use up to now.

DISCUSSION

Due to the invisibility of intrahepatic vascular anatomy during surgery and the high variability, preoperative analyzes of the anatomy and planning of the resection is essential in liver surgery. Therefore, there is a high need for image guidance in hepato-biliary surgery. The use of preoperative 3D virtual reconstruction image techniques have evidence-based optimized the outcome after major liver surgery[1]. The next step of using image guidance was to transfer the 3D image of the liver into the operation theater. The feasibility of AR, MR and intraoperative 3D-navigation has been proven up to now, but the majority of the systems are still in an experimental status. The scenario for clinical use-cases in hepato-biliary surgery is not clearly defined up to now. It is still not clear under which circumstances the use of intraoperative AR and MR or navigation leads to a benefit - for the surgeon to facilitate the operative procedure or for the patient to optimize his outcome?

Minimized safety margins, increased R0-rates, increased number of potential treatable lesion, minimized blood loss, shorter operation time, “visualization” of disappeared liver metastasis, precise sub-segmental anatomical resections, flattened learning curve of complex procedures could be theoretically optimized by the usage of intraoperative AR, MR and 3D navigation in hepato-biliary surgery.

These factors should be evaluated systematically and addressed clearly with high-quality studies, which have not been conducted up to now.

Another important issue is the usability of the virtual 3D technique. The intraoperative use of AR/MR and 3D-navigation changes the workflow during liver resection. It is important that the surgeon feels comfortable with the system and is not limited by the technique, so a high grade of usability is mandatory. This is still a major drawback of the available systems: Additional secondary screens are needed (displays, tablet pc or head-mounted display), secondary cameras above the operative field, marker shields have to be placed on the instruments, registration and calibration must be performed manually and the technique in general is often limited to certain anatomic areas of the liver. Systematic data about the usability is still missing in scientific literature. The low grade of usability and the high cost of image guidance systems (200000 euro to 600000 euro for infrastructure plus additional running costs) limit the further development right now. Thinking about navigation the image of driving a car comes into our mind: A navigation system should tell us where to go, show us the shortest and easiest way to our goal - and where and when the driver should be careful. AR/MR and 3D-navigation in liver surgery have not reached this level of immersion right now. If this is really necessary during surgical procedures could be discussed. It could be enough to support the surgeon with some additional information during cardinal steps of a procedure. AI support is up to now not available in hepato-biliary surgery in the operating theater. Many procedures while using AR and 3D-navigation could be facilitated with AI in the future. Especially the problem of soft tissue deformation, which is omnipresent in liver surgery, could be approached by AI techniques.

CONCLUSION

Although there are still many challenges, AR, MR, 3D-navigation and AI are emerging fields in hepato-biliary surgery. The benefit of these sophisticated computerized image guidance techniques should be measured by its impact on clinically relevant outcome parameters in the future. As shown by the huge effort that was made by hepato-biliary surgeons in the past in this field, these techniques will be further developed over the next years.