Field Of Vision Open Access
Copyright ©2012 Baishideng Publishing Group Co., Limited. All rights reserved.
World J Gastroenterol. Aug 21, 2012; 18(31): 4086-4090
Published online Aug 21, 2012. doi: 10.3748/wjg.v18.i31.4086
Three-dimensional image reconstruction in capsule endoscopy
Anastasios Koulaouzidis, Centre for Liver and Digestive Disorders, The Royal Infirmary of Edinburgh, Scotland EH16 4SA, United Kingdom
Alexandros Karargyris, National Library of Medicine, National Institutes of Health, Bethesda, MD 20814, United States
Author contributions: Koulaouzidis A and Karargyris A collected the material, discussed the topic and wrote the manuscript.
Correspondence to: Dr. Anastasios Koulaouzidis, MD, MRCP, FEBG, Centre for Liver and Digestive Disorders, The Royal Infirmary of Edinburgh, 51 Little France Crescent, Edinburgh, Scotland EH16 4SA, United Kingdom. akoulaouzidis@hotmail.com
Telephone: +44-131-2421126 Fax: +44-131-2421618
Received: June 20, 2012
Revised: July 13, 2012
Accepted: July 18, 2012
Published online: August 21, 2012

Abstract

To date, limited research has been carried out in developing methods and materials that offer three-dimensional (3-D) representation of the digestive tract. In the field of capsule endoscopy (CE), hardware approaches have been developed that provide real time both 3-D information and texture using an infrared projector and a complementary metal oxide semiconductor camera. The major drawbacks of this system are its size, power consumption and packaging issues. A software approach to approximate a 3-D representation of digestive tract surface utilising current CE technology has been proposed. The algorithm utilizes the Shape from Shading technique and seem to provide promising results for polypoid structures and angioectasias. Further clinical evaluation is currently under way.

Key Words: Capsule endoscopy; Three-dimensional; Reconstruction; Angioectasias; Software



INVITED COMMENTARY ON HOT ARTICLES

I read with great interest the recent paper by Karargyris et al[1] on the use of elastic video interpolation in the three-dimensional (3-D) reconstruction of the digestive wall in capsule endoscopy (CE) videos. The article presents promising 3-D approaches in CE; due to its technical nature though it may be slightly difficult for the general gastroenterologist. Therefore, I invited its first author to help me present the clinically relevant points. These authors believe that proper consideration should be given for its further clinical application in CE.

Wireless CE, a milestone in minimally invasive investigation of the gastrointestinal tract, was made possible by the advent of power efficient/low cost image sensors based on complementary metal oxide semiconductor (CMOS) technology[2,3]. Upon the 1950’s debut of the world’s first gastro-camera, few could have envisaged that, just half a century later, imaging of the human digestive tract would become wireless[4-6]. However, this spectacular advent in medical endoscopy is not without its limitations. Let us not forget that a complete capsule platform comprises six fundamental modules: locomotion, vision, telemetry, localisation, power and diagnostic/tissue manipulation tools[1,7]. To present, due to space constraints, the majority of commercially available capsules include only a subset of the aforementioned modules[7]. Furthermore, power limitation is a major hurdle which current CE technology has yet to overcome[1,5,7].

Since the capsule needs 6-8 h to traverse through the small-bowel[7,8], cameras within the currently marketed capsule endoscopes work at a capture rate of 2-3 frames per second (fps) in order to comply with power requirements[7]. Nonetheless, this has an adverse effect on the smoothness of motion between consecutive frames and creates a visually unpleasant effect to the human eye[1,7]. Furthermore, shape is an important element in human perception, yet CE suffers, unlike other diagnostic modalities i.e., Computed tomography, magnetic resonance imaging, from lack of 3-D information[1]. Practically, this can only be feasible with the use of new generation devices that have yet to be realised.

3-D technology is currently in use e.g., a magnetometer can provide not only acceleration values on the three axes but also the 3-D orientation of the device (Figure 1A)[9]. Commercial time-of-flight range cameras (i.e., Microsoft’s Kinect Project, Figure 1B)[10] already exist in the market and in the near future this may be further improved and miniaturised for use inside a capsule endoscope. These cameras offer information on depth and colour. We should not forget that 3-D guidance systems are already used for endoscopic surgeries offering 3-D position information of the sensor (Figure 1C). Therefore, using the acquired information (orientation, acceleration, depth values, position etc.) from these miniature sensors in conjunction with sophisticated registration software algorithms, an accurate 3-D representation of the digestive tract could be created successfully.

Figure 1
Figure 1 Various sensors which are commercially available[12]. A: Miniature magnetometers offer orientation and acceleration information, [Bertda Services (SEA) Pte. Ltd.]; B: Three-dimensional (3-D) range camera used in a widely commercial product; C: 3-D guidance system used in endoscopy devices (inition).
3-D representation for endoscopy

To date, limited research has been carried out in developing methods and materials that offer 3-D representation of the digestive tract. For conventional endoscopy systems, stereo technology has been introduced to capture stereo images and to create depth information and therefore 3-D reconstruction of digestive structures. However, due to issues with size, such systems have not been widely accepted[1,5]. Likewise, in CE there has been a hardware approach that provides in real time both 3-D information and texture using an infrared projector and a CMOS camera. The major drawbacks of this system are its size, power consumption and packaging issues[11]. Therefore, in order to tackle the problem of the current hardware limitations, Karargyris et al[1,12] proposed the use of a software approach to approximate a 3-D representation of digestive tract surface utilising current CE technology.

Shape from shading

The Shape from Shading (SfS) technique is a member of a family of “shape recovery” algorithms called shape-from-X techniques[1,13]. Essentially, SfS algorithms try to recover the shape of objects by using the gradual variation of shading (Figure 2)[14]. SfS techniques can be divided into four groups or approaches: minimisation, propagation, local, and linear approaches[13,14].

Figure 2
Figure 2 The Shape from Shading flow[12,13]. Capturing a surface using a camera removes depth information. Shape from Shading techniques try to reproduce the missing depth information from a given image.

Karargyris et al[1] propose using a specific sub-category of SfS methods: the Tsai’s method (or linear approach)[1,15] because (1) it produces good results for spherical surfaces, which in fact is the case with most shapes of digestive tract pathology e.g., polyps; (2) it is very fast; and (3) it “behaves” relatively well with specular surfaces (surfaces with mirror-like reflection of light). Of note, in CE the light source axis and the miniature camera are basically aligned. Tsai’s method works well on smooth objects and spherical structures[15]. Additionally, Tsai’s method takes into consideration only the light direction, which in our case is 0o degrees (since the lighting direction and that of the camera are parallel).

A correction on the camera distortion is not performed because there is insufficient data on the capsule camera optics’ specifications. The result of applying Tsai’s method in a CE frame is given in Figure 3. Figure 3A shows a polyp in a conventional 2-D CE frame, whereas in Figure 3B the same polyp in presented using the 3-D software. The SfS approach gives promising outcomes, especially for visualizing polyps and vascular lesions, but less so for ulcers or lymphangiectasia (Figure 4). In the first 2 cases, the 3-D result is rather exciting, supporting the argument for further evaluation of this technique for use in clinical practice. Additionally, a 3-D representation may be helpful in the design of more accurate and robust computer-aided detection algorithms, incorporating other image enhancement tools e.g., virtual chromoendoscopy (FICE)[16] or colour (blue) mode[17] analysis of CE videos instead of still CE frames. However, prior to any software integration, further qualitative assessment and accuracy assessment should take place (work currently undertaken by these authors).

Figure 3
Figure 3 Original capsule endoscopy and 3-D represented image depicting a protrusion. A: Original capsule endoscopy image depicting a protrusion (polyp); B: Its 3-D represented image processed by the proposed Shape from Shading approach.
Figure 4
Figure 4 Two-dimensional capsule endoscopy images (upper panel) and three-dimensional representation of above structures is seen (lower panel). A1, A2: P2 angioectasia; B1, B2: P1 angiectasia; C1, C2: Lymphoid hyperplasia with superficial ulceration; D1, D2: Aphthous ulcers; E1, E2: Serpiginous ulcers; F1, F2: Nodular lymphangiectasia; G1, G2: Another capsule endoscope still inside the small-bowel.

Conversely, in cases of sudden large intensity change, Tsai’s method fails because it takes into consideration the preservation of intensity gradient. However, in non-artificial images such as CE frames, the brightness (image pixel values) transitions smoothly with no abrupt changes. Technically however, one can notice in Figure 5 the presence of highlights, hence false information about the shape. Highlights are essentially linear combinations of specular and diffuse reflection (light reflected at various angles) components of the surface.

Figure 5
Figure 5 3-D representation of a single video capsule image with highlights removal (arrows).

Many objects in the real world are dielectric and homogeneous, hence displaying both types of reflections. Most digestive structures fall into this category[18]. When the light beams fall on to such an object, some of them reflect back immediately creating the specular reflection, while the rest of the beams first penetrate the object and then reflect creating the diffuse reflection. Along with 3-D representation algorithm a highlight suppression scheme is applied to the original CE images to produce desired better results, whilst maintaining the shapes and structures of the digestive tract. Figure 5 shows the successful highlight removal without affecting the shapes of other objects. It has to be mentioned that highlights from light reflections on the surface of the digestive tract are still an open problem not only for a 3-D representation but also for traditional CE review.

In conclusion, the software by Karargyris et al[1] offers a new potential to reform and enhance the currently existing reading software in capsule endoscopy by improving lesion demarcation and highlighting the textural features of ulcers, angioectasias and polyps[5,19]. Further work is needed to prove its clinical validity but the idea of a 3-D aid (with the present level of CE technology) seems, at least for this author, not only captivating but promising as well. However, it should not be forgotten that true 3-D capability requires dual video images, although the inclusion of two cameras within the shell of a capsule endoscopy might be unwieldy at present[5].

Footnotes

Peer reviewers: Ferruccio Bonino, MD, PhD, Professor of Gastroenterology, Department of Internal Medicine, University Hospital of Pisa, University Hospital of Pisa, University of Pisa, Via Roma 67, 56124 Pisa, Italy; Naoaki Sakata, MD, PhD, Division of Hepato-Biliary Pancreatic Surgery, Tohoku University Graduate School of Medicine, 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574, Japan; Andrzej S Tarnawski, MD, PhD, DSc (Med), Professor of Medicine, Chief Gastroenterology, VA Long Beach Health Care System, University of California, Irvine, CA, 5901 E. Seventh Str., Long Beach, CA 90822, United States

S- Editor Gou SX L- Editor A E- Editor Zhang DN

References
1.  Karargyris A, Bourbakis N. Three-dimensional reconstruction of the digestive wall in capsule endoscopy videos using elastic video interpolation. IEEE Trans Med Imaging. 2011;30:957-971.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 46]  [Cited by in F6Publishing: 26]  [Article Influence: 2.0]  [Reference Citation Analysis (0)]
2.  Liedlgruber M, Uhl A. Computer-aided decision support systems for endoscopy in the gastrointestinal tract: a review. IEEE Rev Biomed Eng. 2011;4:73-88.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 118]  [Cited by in F6Publishing: 82]  [Article Influence: 6.8]  [Reference Citation Analysis (0)]
3.  Muñoz-Navas M. Capsule endoscopy. World J Gastroenterol. 2009;15:1584-1586.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in CrossRef: 38]  [Cited by in F6Publishing: 25]  [Article Influence: 1.7]  [Reference Citation Analysis (0)]
4.  Fireman Z. Capsule endoscopy: Future horizons. World J Gastrointest Endosc. 2010;2:305-307.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in CrossRef: 8]  [Cited by in F6Publishing: 7]  [Article Influence: 0.5]  [Reference Citation Analysis (0)]
5.  Fisher LR, Hasler WL. New vision in video capsule endoscopy: current status and future directions. Nat Rev Gastroenterol Hepatol. 2012;9:392-405.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 50]  [Cited by in F6Publishing: 56]  [Article Influence: 4.7]  [Reference Citation Analysis (0)]
6.  Swain P. At a watershed? Technical developments in wireless capsule endoscopy. J Dig Dis. 2010;11:259-265.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 11]  [Cited by in F6Publishing: 10]  [Article Influence: 0.7]  [Reference Citation Analysis (0)]
7.  Ciuti G, Menciassi A, Dario P. Capsule endoscopy: from current achievements to open challenges. IEEE Rev Biomed Eng. 2011;4:59-72.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 273]  [Cited by in F6Publishing: 123]  [Article Influence: 10.3]  [Reference Citation Analysis (0)]
8.  Westerhof J, Koornstra JJ, Hoedemaker RA, Sluiter WJ, Kleibeuker JH, Weersma RK. Diagnostic yield of small bowel capsule endoscopy depends on the small bowel transit time. World J Gastroenterol. 2012;18:1502-1507.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in CrossRef: 35]  [Cited by in F6Publishing: 36]  [Article Influence: 3.0]  [Reference Citation Analysis (0)]
9.   Available from: http: //en.wikipedia.org/wiki/ Magnetometer.  [PubMed]  [DOI]  [Cited in This Article: ]
10.   Available from: http: //en.wikipedia.org/wiki/Kinect.  [PubMed]  [DOI]  [Cited in This Article: ]
11.  Kolar A, Romain O, Ayoub J, Faura D, Viateur S, Granado B, Graba T. A system for an accurate 3D reconstruction in Video Endoscopy Capsule. EURASIP J on Emb Sys. 2009;2009:716317.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 13]  [Cited by in F6Publishing: 14]  [Article Influence: 0.9]  [Reference Citation Analysis (0)]
12.  Karargyris A, Karargyris O, Bourbakis N.  3D Representation of the Digestive Tract Surface in Wireless Capsule Endoscopy Videos. In: Proceedings of the 2010; IEEE xplore, 2010: 279-280.  [PubMed]  [DOI]  [Cited in This Article: ]
13.  Prados E, Faugeras OF A rigorous and realistic Shape from Shading method and some of its applications. France: INRIA 2004; .  [PubMed]  [DOI]  [Cited in This Article: ]
14.  Zhang R, Tsai PS, Cryer JE, Shah M. Shape-from-Shading: A Survey. IEEE Trans Pattern Anal Mach Intell. 1999;21:690-706.  [PubMed]  [DOI]  [Cited in This Article: ]
15.  Tsai PS, Shah M. Shape from shading using linear approximation. Image and Vision Computing. 1994;12:487-498.  [PubMed]  [DOI]  [Cited in This Article: ]
16.  Gupta T, Ibrahim M, Deviere J, Van Gossum A. Evaluation of Fujinon intelligent chromo endoscopy-assisted capsule endoscopy in patients with obscure gastroenterology bleeding. World J Gastroenterol. 2011;17:4590-4595.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in CrossRef: 21]  [Cited by in F6Publishing: 30]  [Article Influence: 2.3]  [Reference Citation Analysis (0)]
17.  Krystallis C, Koulaouzidis A, Douglas S, Plevris JN. Chromoendoscopy in small bowel capsule endoscopy: Blue mode or Fuji Intelligent Colour Enhancement? Dig Liver Dis. 2011;43:953-957.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 24]  [Cited by in F6Publishing: 33]  [Article Influence: 2.5]  [Reference Citation Analysis (0)]
18.  van der Zee P Measurement and modeling of the optical properties of human tissue in the near infrared. London: University of London 1992; .  [PubMed]  [DOI]  [Cited in This Article: ]
19.  Lewis BS. Expanding role of capsule endoscopy in inflammatory bowel disease. World J Gastroenterol. 2008;14:4137-4141.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in CrossRef: 42]  [Cited by in F6Publishing: 36]  [Article Influence: 2.3]  [Reference Citation Analysis (0)]