Mixed reality for visualization of orthopedic surgical anatomy

Abstract

In the modern era, preoperative planning is substantially facilitated by artificial reality technologies, which permit a better understanding of patient anatomy, thus increasing the safety and accuracy of surgical interventions. In the field of orthopedic surgery, the increase in safety and accuracy improves treatment quality and orthopedic patient outcomes. Artificial reality technologies, which include virtual reality (VR), augmented reality (AR), and mixed reality (MR), use digital images obtained from computed tomography or magnetic resonance imaging. VR replaces the user’s physical environment with one that is computer generated. AR and MR have been defined as technologies that permit the fusing of the physical with the virtual environment, enabling the user to interact with both physical and virtual objects. MR has been defined as a technology that, in contrast to AR, enables users to visualize the depth and perspective of the virtual models. We aimed to shed light on the role that MR can play in the visualization of orthopedic surgical anatomy. The literature suggests that MR could be a valuable tool in orthopedic surgeon’s hands for visualization of the anatomy. However, we remark that confusion exists in the literature concerning the characteristics of MR. Thus, a more clear description of MR is needed in orthopedic research, so that the potential of this technology can be more deeply understood.

Key Words: Orthopedic surgery; Mixed reality; Anatomy; Augmented reality; Three-dimensional visualization technologies; Artificial reality technologies

Core Tip: Mixed reality could be a valuable tool in orthopedic surgeon’s hands for visualization of anatomy, but a more clear description of this technology is needed in the orthopedic literature.

INTRODUCTION

In the modern era, surgical planning is substantially facilitated by artificial reality technologies, which permit a better understanding of patient anatomy, thus increasing safety and accuracy[1]. Among artificial reality technologies, virtual reality (VR) has been defined as a technology that completely replaces the user’s physical environment with one that is computer generated[2]. Augmented reality (AR) enables specific devices, to fuse digital models with physical objects and allow for interaction with both[3]. MR, like AR, permits fusing of physical with virtual environment, but in contrast to AR, enables users to visualize depth and perspective in the virtual models[2,4]. The models are derived from preoperative images, obtained by computed tomography (CT) or magnetic resonance imaging[5]. Of these technologies, VR and AR allow for adequate visualization of orthopedic surgical anatomy, thus facilitating the performance of several types of orthopedic interventions[5]. The technologies provide surgeons with the ability to visualize patient data in real time, improve preoperative planning, and offer accuracy in performance of interventions, thus leading to upgrades of treatment quality and orthopedic patient outcomes[5]. We aimed to shed light on the role that MR can play in the perception of orthopedic surgical anatomy. We consider that, in contrast with VR and AR, the confusion that exists in the literature impedes the understanding of the value of this technology for the visualization of anatomy in orthopedic surgical procedures.

MIXED REALITY AND VISUALIZATION OF ORTHOPEDIC SURGICAL ANATOMY

In a review of the literature on the implementation of VR, AR, and MR in orthopedics Verhey et al[5] stated that similar to AR, an MR system produces stereoscopic images formed by combining the real world with three-dimensional (3D) virtual models[5]. It was also stated that in MR systems, virtual objects are not simply projected on real ones, as in AR, but the user can interact with both the real and digital objects. The definition, provided by Verhey et al[5], is different from that provided by Moro et al[3], according to which, AR does allow for interaction. Also, Verhey et al[5] argued that both MR systems and AR, produce stereoscopic images. Stereoscopic visualization has been defined as the combined view of two digital images seen separately by each eye, using special devices[6]. In contrast, monoscopic visualization comprises digital objects that can be three-dimensionally rotated but are projected on a two-dimensional screen[6]. According to the aforementioned definitions of stereoscopic visualization and AR, it can be noted that stereopsis is not an essential characteristic of AR, thus there is a disagreement with Verhey et al[5].

Condino et al[7] described an MR-based orthopedic surgery simulator for which hip arthroplasty was chosen as a benchmark for evaluation. The authors performed quantitative tests to “estimate the accuracy of the system by evaluating the perceived position of AR targets”. According to Condino et al[7], the results of their study supported the use of MR to develop a simulator for orthopedic surgery. However, as can be concluded by the aforementioned purpose of the study, Condino et al[7] did not distinguish MR from AR.

Gregory et al[8] reported a case of a patient who underwent reverse shoulder arthroplasty performed with the aid of an MR headset. The authors noted that the system enabled accurate visualization of the patient’s anatomy, which was beneficial for the safety of the procedure. A postoperative CT scan confirmed the satisfactory position of the prosthesis, and the patient experienced no peri- or postoperative complications (Table 1). Nevertheless, in the introduction of their article, the authors stated that AR is commonly referred to as MR, thus they did not differentiate the two technologies.

Table 1 Studies of implementation of mixed reality for visualization of orthopedic surgical anatomy, with patient outcomes.

Ref.	Operation	Effects of MR on visualization of orthopedic surgical anatomy	Patient outcomes
Gregory et al[8]	Reverse shoulder arthroplasty	Accurate visualization of the patient’s anatomy	A postoperative CT scan confirmed the adequate position of the prosthesis, while the patient experienced no peri- or postoperative complications
Wei et al[10]	Percutaneous kyphoplasty to treat an osteoporotic vertebral compression fracture with intravertebral vacuum cleft	The surgeon could obtain accurate guidance to the intravertebral vacuum cleft area during the operation	Vertebral height improvement, cement diffusion and pain relief were significantly better in the MR group in comparison with the traditional C-arm fluoroscopy group
Gu et al[13]	Lumbar pedicle-screw placement	The implantation accuracy with the use of MR was significantly higher in comparison with traditional screw placement	Significantly less bleeding and operative time, faster recovery, significantly better pain scores at 1 month postoperatively with MR, in comparison with traditional screw placement
Lei et al[14]	Total hip arthroplasty	The patient's virtual bone, as well as the other anatomical structures, were accurately superimposed on the patient’s body	The range of motion for the hip joint was within the normal range, while the patient’s recovery was reported to be good and he was discharged without obvious surgical complications

CT: Computed tomography; MR: Mixed reality.

Wu et al[9] reported a case of a patient with traumatic high paraplegia who underwent a complicated cervical spine fracture procedure with the use of MR technology. The authors noted that the MR system enabled the surgeon to clearly visualize the anatomy in the operative field, and that CT with 3D reconstruction could not adequately depict neuronal and vascular components around the fracture. However, in the introduction, the authors defined MR as “the merging of the real world and the virtual world,” and did not explain the difference between MR and AR.

Wei et al[10] evaluated the clinical outcome of MR-assisted percutaneous kyphoplasty to treat an osteoporotic vertebral compression fracture with intravertebral vacuum cleft. It was concluded that percutaneous kyphoplasty assisted by MR provided the surgeon with accurate guidance to the intravertebral vacuum cleft area during the operation. A group of patients who underwent MR-assisted percutaneous kyphoplasty to treat an osteoporotic vertebral compression fracture with intravertebral vacuum cleft was compared with a group who underwent the same procedure with traditional C-arm fluoroscopy instead of MR. Vertebral height improvement, cement diffusion, and pain relief were significantly improved by MR assistance (Table 1). The authors stated that MR is a combination of AR and VR and that it permits accurate combination of virtual objects with the real world, without further explanation.

Wu et al[11] assessed the safety and accuracy of pedicle-screw placement in a 3D printed model of an upper cervical spine fracture under MR-based navigation. The authors noted that MR could effectively help surgeons visualize intraoperative anatomy, especially in complex cases involving the upper cervical spine. The authors highlighted the advantages of MR, which “generates computer graphics onto the holographic display of real scenes”, and cited a study by Volonté et al[12]. However, Volonté et al[12] dealt with AR and not MR technology.

A study by Gu et al[13] included patients who were randomly divided in two groups. The first with MR-based lumbar pedicle-screw placement and the second with traditional screw placement. The implantation accuracy was significantly better in the first group than in the second one. Also, there was significantly less bleeding, shorter operative time, and faster recovery in the first group. One month postoperatively, the pain scores were significantly better in the first than in the second group (Table 1). The authors defined MR as a technology that combines virtual with physical objects, without further clarification.

Lei et al[14] performed a complicated total hip arthroplasty combining 3D printing technology with MR. It was noted that the virtual bone and other anatomical structures were accurately superimposed on the patient’s body. Postoperatively, the range of motion for the hip joint was within the normal range, the patient’s recovery was reported to be good, and he was discharged without obvious surgical complications (Table 1). The authors stated that “the unsatisfied accuracy of registration in MR technology is an urgent problem yet to be resolved” and cited an article by Fida et al[15]. However, Fida et al[15] reviewed the use of AR in open surgery, and both Lei et al[14] and Fida et al[15] used the terms “AR” and “MR” interchangeably.

CONCLUSION

According to the literature, MR can be a valuable tool in the orthopedic surgeon’s hands for visualization of anatomy. Although the two technologies are distinct, the interchangeable use of the terms “AR” and “MR” in the orthopedic surgery literature does not permit researchers and surgeons to extract safe conclusions about the possible superiority of AR or MR. Because MR has been defined as a technology that provides depth and perspective in the virtual environment, in contrast to AR[2,4], it seems that the two technologies may have different values in perceiving orthopedic surgical anatomy. The literature suggests that the two technologies may have different anatomy teaching potential[16]. Currently, there is a lack of research to permit comparison between AR and MR in terms of their value in orthopedic surgical practice. The possible difference between the value of two technologies needs further investigation, which should proceed with a clear description of the technology under investigation and with differentiation between AR and MR.