1
|
Blair J, Stephen B, Brown B, McArthur S, Gorman D, Forbes A, Pottier C, McAlorum J, Dow H, Perry M. Photometric stereo data for the validation of a structural health monitoring test rig. Data Brief 2024; 53:110164. [PMID: 38375140 PMCID: PMC10875225 DOI: 10.1016/j.dib.2024.110164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 12/22/2023] [Accepted: 01/31/2024] [Indexed: 02/21/2024] Open
Abstract
Photometric stereo uses images of objects illuminated from various directions to calculate surface normals which can be used to generate 3D meshes of the object. Such meshes can be used by engineers to estimate damage of a concrete surface, or track damage progression over time to inform maintenance decisions. This dataset [1] was collected to quantify the uncertainty in a photometric stereo test rig through both the comparison with a well characterised method (coordinate measurement machine) and experiment virtualisation. Data was collected for 9 real objects using both the test rig and the coordinate measurement machine. These objects range from clay statues to damaged concrete slabs. Furthermore, synthetic data for 12 objects was created via virtual renders generated using Blender (3D software) [2]. The two methods of data generation allowed the decoupling of the physical rig (used to light and photograph objects) and the photometric stereo algorithm (used to convert images and lighting information into 3D meshes). This data can allow users to: test their own photometric stereo algorithms, with specialised data created for structural health monitoring applications; provide an industrially relevant case study to develop and test uncertainty quantification methods on test rigs for structural health monitoring of concrete; or develop data processing methodologies for the alignment of scaled, translated, and rotated data.
Collapse
Affiliation(s)
- Jennifer Blair
- Institute for Energy and Environment, Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, G1 1XJ, UK
- National Physical Laboratory, Teddington, TW11 0LW, UK
| | - Bruce Stephen
- Institute for Energy and Environment, Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, G1 1XJ, UK
| | - Blair Brown
- Institute for Energy and Environment, Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, G1 1XJ, UK
| | - Stephen McArthur
- Institute for Energy and Environment, Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, G1 1XJ, UK
| | - David Gorman
- National Physical Laboratory, Teddington, TW11 0LW, UK
| | | | | | - Jack McAlorum
- Department of Civil and Environmental Engineering, University of Strathclyde, Glasgow, G1 1XJ, UK
| | - Hamish Dow
- Department of Civil and Environmental Engineering, University of Strathclyde, Glasgow, G1 1XJ, UK
| | - Marcus Perry
- Department of Civil and Environmental Engineering, University of Strathclyde, Glasgow, G1 1XJ, UK
| |
Collapse
|
2
|
Nguyen AH, Wang Z. Time-Distributed Framework for 3D Reconstruction Integrating Fringe Projection with Deep Learning. SENSORS (BASEL, SWITZERLAND) 2023; 23:7284. [PMID: 37631820 PMCID: PMC10458373 DOI: 10.3390/s23167284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 08/07/2023] [Accepted: 08/18/2023] [Indexed: 08/27/2023]
Abstract
In recent years, integrating structured light with deep learning has gained considerable attention in three-dimensional (3D) shape reconstruction due to its high precision and suitability for dynamic applications. While previous techniques primarily focus on processing in the spatial domain, this paper proposes a novel time-distributed approach for temporal structured-light 3D shape reconstruction using deep learning. The proposed approach utilizes an autoencoder network and time-distributed wrapper to convert multiple temporal fringe patterns into their corresponding numerators and denominators of the arctangent functions. Fringe projection profilometry (FPP), a well-known temporal structured-light technique, is employed to prepare high-quality ground truth and depict the 3D reconstruction process. Our experimental findings show that the time-distributed 3D reconstruction technique achieves comparable outcomes with the dual-frequency dataset (p = 0.014) and higher accuracy than the triple-frequency dataset (p = 1.029 × 10-9), according to non-parametric statistical tests. Moreover, the proposed approach's straightforward implementation of a single training network for multiple converters makes it more practical for scientific research and industrial applications.
Collapse
Affiliation(s)
- Andrew-Hieu Nguyen
- Neuroimaging Research Branch, National Institute on Drug Abuse, National Institutes of Health, Baltimore, MD 21224, USA;
| | - Zhaoyang Wang
- Department of Mechanical Engineering, The Catholic University of America, Washington, DC 20064, USA
| |
Collapse
|
3
|
Nguyen AH, Ly KL, Lam VK, Wang Z. Generalized Fringe-to-Phase Framework for Single-Shot 3D Reconstruction Integrating Structured Light with Deep Learning. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23094209. [PMID: 37177413 PMCID: PMC10181406 DOI: 10.3390/s23094209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 04/16/2023] [Accepted: 04/18/2023] [Indexed: 05/15/2023]
Abstract
Three-dimensional (3D) shape acquisition of objects from a single-shot image has been highly demanded by numerous applications in many fields, such as medical imaging, robotic navigation, virtual reality, and product in-line inspection. This paper presents a robust 3D shape reconstruction approach integrating a structured-light technique with a deep learning-based artificial neural network. The proposed approach employs a single-input dual-output network capable of transforming a single structured-light image into two intermediate outputs of multiple phase-shifted fringe patterns and a coarse phase map, through which the unwrapped true phase distributions containing the depth information of the imaging target can be accurately determined for subsequent 3D reconstruction process. A conventional fringe projection technique is employed to prepare the ground-truth training labels, and part of its classic algorithm is adopted to preserve the accuracy of the 3D reconstruction. Numerous experiments have been conducted to assess the proposed technique, and its robustness makes it a promising and much-needed tool for scientific research and engineering applications.
Collapse
Affiliation(s)
- Andrew-Hieu Nguyen
- Department of Mechanical Engineering, The Catholic University of America, Washington, DC 20064, USA
- Neuroimaging Research Branch, National Institute on Drug Abuse, National Institutes of Health, Baltimore, MD 21224, USA
| | - Khanh L Ly
- Department of Biomedical Engineering, The Catholic University of America, Washington, DC 20064, USA
| | - Van Khanh Lam
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC 20012, USA
| | - Zhaoyang Wang
- Department of Mechanical Engineering, The Catholic University of America, Washington, DC 20064, USA
| |
Collapse
|
4
|
Casula G, Fais S, Cuccuru F, Bianchi MG, Ligas P. Diagnostic Process of an Ancient Colonnade Using 3D High-Resolution Models with Non-Invasive Multi Techniques. SENSORS (BASEL, SWITZERLAND) 2023; 23:3098. [PMID: 36991808 PMCID: PMC10051703 DOI: 10.3390/s23063098] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Revised: 02/15/2023] [Accepted: 03/08/2023] [Indexed: 06/19/2023]
Abstract
Here, an avant-garde study of three ancient Doric columns of the precious, ancient Romanesque church of Saints Lorenzo and Pancrazio in the historical town center of Cagliari (Italy) is presented based on the integrated application of different non-destructive testing methods. The limitations of each methodology are overcome by the synergistic application of these methods, affording an accurate, complete 3D image of the studied elements. Our procedure begins with a macroscopic in situ analysis to provide a preliminary diagnosis of the conditions of the building materials. The next step is laboratory tests, in which the porosity and other textural characteristics of the carbonate building materials are studied by optical and scanning electron microscopy. After this, a survey with a terrestrial laser scanner and close-range photogrammetry is planned and executed to produce accurate high-resolution 3D digital models of the entire church and the ancient columns inside. This was the main objective of this study. The high-resolution 3D models allowed us to identify architectural complications occurring in historical buildings. The 3D reconstruction with the above metric techniques was indispensable for planning and carrying out the 3D ultrasonic tomography, which played an important role in detecting defects, voids, and flaws within the body of the studied columns by analyzing the propagation of the ultrasonic waves. The high-resolution 3D multiparametric models allowed us to obtain an extremely accurate picture of the conservation state of the studied columns in order to locate and characterize both shallow and internal defects in the building materials. This integrated procedure can aid in the control of the spatial and temporal variations in the materials' properties and provides information on the process of deterioration in order to allow adequate restoration solutions to be developed and the structural health of the artefact to be monitored.
Collapse
Affiliation(s)
- Giuseppe Casula
- Istituto Nazionale di Geofisica e Vulcanologia (INGV)—Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy;
| | - Silvana Fais
- Department of Environmental Civil Engineering and Architecture (DICAAR), University of Cagliari, Via Marengo 2, 09123 Cagliari, Italy; (S.F.); (F.C.); (P.L.)
- Consorzio Interuniversitario Nazionale per l’Ingegneria delle Georisorse, CINIGEO, Palazzo Baleani, Corso Vittorio Emanuele II 244, 00186 Roma, Italy
- National Research Council of Italy (CNR)—Institute of Environmental Geology and Geoengineering (IGAG), Via Marengo 2, 09123 Cagliari, Italy
| | - Francesco Cuccuru
- Department of Environmental Civil Engineering and Architecture (DICAAR), University of Cagliari, Via Marengo 2, 09123 Cagliari, Italy; (S.F.); (F.C.); (P.L.)
| | - Maria Giovanna Bianchi
- Istituto Nazionale di Geofisica e Vulcanologia (INGV)—Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy;
| | - Paola Ligas
- Department of Environmental Civil Engineering and Architecture (DICAAR), University of Cagliari, Via Marengo 2, 09123 Cagliari, Italy; (S.F.); (F.C.); (P.L.)
| |
Collapse
|
5
|
Sanao H, Yingjie S, Ming L, Jingwei Q, Ke X. Underwater 3D reconstruction using a photometric stereo with illuminance estimation. APPLIED OPTICS 2023; 62:612-619. [PMID: 36821264 DOI: 10.1364/ao.476003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Accepted: 12/11/2022] [Indexed: 06/18/2023]
Abstract
Underwater image sensing is directly affected by the change in illuminance towards a camera caused by the refraction of light in different media. In this study, the convergence of a near-field point light source in water is analyzed using a light propagation model. The photometric stereo (PS) formula is determined based on an accurate estimation of the illuminance entering the camera. An underwater PS system is designed to verify the proposed method's feasibility. The experimental results demonstrate improved accuracy in normal calculation. This helps achieve accurate underwater 3D reconstruction of objects that is suitable for underwater surface microdefect detection.
Collapse
|
6
|
Cao Y, Ding B, Chen J, Liu W, Guo P, Huang L, Yang J. Photometric-Stereo-Based Defect Detection System for Metal Parts. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22218374. [PMID: 36366075 PMCID: PMC9655976 DOI: 10.3390/s22218374] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 10/27/2022] [Accepted: 10/27/2022] [Indexed: 05/27/2023]
Abstract
Automated inspection technology based on computer vision is now widely used in the manufacturing industry with high speed and accuracy. However, metal parts always appear in high gloss or shadow on the surface, resulting in the overexposure of the captured images. It is necessary to adjust the light direction and view to keep defects out of overexposure and shadow areas. However, it is too tedious to adjust the position of the light direction and view the variety of parts' geometries. To address this problem, we design a photometric-stereo-based defect detection system (PSBDDS), which combines the photometric stereo with defect detection to eliminate the interference of highlights and shadows. Based on the PSBDDS, we introduce a photometric-stereo-based defect detection framework, which takes images captured in multiple directional lights as input and obtains the normal map through the photometric stereo model. Then, the detection model uses the normal map as input to locate and classify defects. Existing learning-based photometric stereo methods and defect detection methods have achieved good performance in their respective fields. However, photometric stereo datasets and defect detection datasets are not sufficient for training and testing photometric-stereo-based defect detection methods, thus we create a photometric stereo defect detection (PSDD) dataset using our PSBDDS to eliminate gaps between learning-based photometric stereo and defect detection methods. Furthermore, experimental results prove the effectiveness of the proposed PSBBD and PSDD dataset.
Collapse
Affiliation(s)
- Yanlong Cao
- State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
- Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Binjie Ding
- State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
- Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Jingxi Chen
- State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
- Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Wenyuan Liu
- State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
- Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Pengning Guo
- State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
- Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Liuyi Huang
- Zhejiang Academy of Special Equipment Science, Zhejiang University, Hangzhou 310023, China
- Key Laboratory of Special Equipment Safety Testing Technology of Zhejiang Province, Hangzhou 310023, China
| | - Jiangxin Yang
- State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
- Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province, School of Mechanical Engineering, Zhejiang University, Hangzhou 310058, China
| |
Collapse
|
7
|
Karami A, Menna F, Remondino F. Combining Photogrammetry and Photometric Stereo to Achieve Precise and Complete 3D Reconstruction. SENSORS (BASEL, SWITZERLAND) 2022; 22:8172. [PMID: 36365869 PMCID: PMC9654855 DOI: 10.3390/s22218172] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Revised: 10/17/2022] [Accepted: 10/19/2022] [Indexed: 06/16/2023]
Abstract
Image-based 3D reconstruction has been employed in industrial metrology for micro-measurements and quality control purposes. However, generating a highly-detailed and reliable 3D reconstruction of non-collaborative surfaces is still an open issue. In this paper, a method for generating an accurate 3D reconstruction of non-collaborative surfaces through a combination of photogrammetry and photometric stereo is presented. On one side, the geometric information derived with photogrammetry is used in areas where its 3D measurements are reliable. On the other hand, the high spatial resolution capability of photometric stereo is exploited to acquire a finely detailed topography of the surface. Finally, three different approaches are proposed to fuse both geometric information and high frequency details. The proposed method is tested on six different non-collaborative objects with different surface characteristics. To evaluate the accuracy of the proposed method, a comprehensive cloud-to-cloud comparison between reference data and 3D points derived from the proposed fusion methods is provided. The experiments demonstrated that, despite correcting global deformation up to an average RMSE of less than 0.1 mm, the proposed method recovers the surface topography at the same high resolution as the photometric stereo.
Collapse
Affiliation(s)
- Ali Karami
- 3D Optical Metrology (3DOM) Unit, Bruno Kessler Foundation (FBK), 38123 Trento, Italy
- Department of Information Engineering and Computer Science, University of Trento, 38123 Trento, Italy
| | - Fabio Menna
- 3D Optical Metrology (3DOM) Unit, Bruno Kessler Foundation (FBK), 38123 Trento, Italy
| | - Fabio Remondino
- 3D Optical Metrology (3DOM) Unit, Bruno Kessler Foundation (FBK), 38123 Trento, Italy
| |
Collapse
|
8
|
A Novel Method to Inspect 3D Ball Joint Socket Products Using 2D Convolutional Neural Network with Spatial and Channel Attention. SENSORS 2022; 22:s22114192. [PMID: 35684808 PMCID: PMC9185281 DOI: 10.3390/s22114192] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/20/2022] [Revised: 05/27/2022] [Accepted: 05/29/2022] [Indexed: 01/25/2023]
Abstract
Product defect inspections are extremely important for industrial manufacturing processes. It is necessary to develop a special inspection system for each industrial product due to their complexity and diversity. Even though high-precision 3D cameras are usually used to acquire data to inspect 3D objects, it is hard to use them in real-time defect inspection systems due to their high price and long processing time. To address these problems, we propose a product inspection system that uses five 2D cameras to capture all inspection parts of the product and a deep learning-based 2D convolutional neural network (CNN) with spatial and channel attention (SCA) mechanisms to efficiently inspect 3D ball joint socket products. Channel attention (CA) in our model detects the most relevant feature maps while spatial attention (SA) finds the most important regions in the extracted feature map of the target. To build the final SCA feature vector, we concatenated the learned feature vectors of CA and SA because they complement each other. Thus, our proposed CNN with SCA provides high inspection accuracy as well as it having the potential to detect small defects of the product. Our proposed model achieved 98% classification accuracy in the experiments and proved its efficiency on product inspection in real-time.
Collapse
|
9
|
Photometric Stereo-Based Defect Detection System for Steel Components Manufacturing Using a Deep Segmentation Network. SENSORS 2022; 22:s22030882. [PMID: 35161628 PMCID: PMC8838491 DOI: 10.3390/s22030882] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/30/2021] [Revised: 01/19/2022] [Accepted: 01/21/2022] [Indexed: 11/23/2022]
Abstract
This paper presents an automatic system for the quality control of metallic components using a photometric stereo-based sensor and a customized semantic segmentation network. This system is designed based on interoperable modules, and allows capturing the knowledge of the operators to apply it later in automatic defect detection. A salient contribution is the compact representation of the surface information achieved by combining photometric stereo images into a RGB image that is fed to a convolutional segmentation network trained for surface defect detection. We demonstrate the advantage of this compact surface imaging representation over the use of each photometric imaging source of information in isolation. An empirical analysis of the performance of the segmentation network on imaging samples of materials with diverse surface reflectance properties is carried out, achieving Dice performance index values above 0.83 in all cases. The results support the potential of photometric stereo in conjunction with our semantic segmentation network.
Collapse
|
10
|
Song Z, Song Z, Ye Y. Eliminating the Effect of Reflectance Properties on Reconstruction in Stripe Structured Light System. SENSORS 2020; 20:s20226564. [PMID: 33212938 PMCID: PMC7698391 DOI: 10.3390/s20226564] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/18/2020] [Revised: 10/30/2020] [Accepted: 11/04/2020] [Indexed: 11/21/2022]
Abstract
The acquisition of the geometry of general scenes is related to the interplay of surface geometry, material properties and illumination characteristics. Surface texture and non-Lambertian reflectance properties degrade the reconstruction results by structured light technique. Existing structured light techniques focus on different coding strategy and light sources to improve reconstruction accuracy. The hybrid system consisting of a structured light technique and photometric stereo combines the depth value with normal information to refine the reconstruction results. In this paper, we propose a novel hybrid system consisting of stripe-based structured light and photometric stereo. The effect of surface texture and non-Lambertian reflection on stripe detection is first concluded. Contrary to existing fusion strategy, we propose an improved method for stripe detection to reduce the above factor’s effects on accuracy. The reconstruction problem for general scene comes down to using reflectance properties to improve the accuracy of stripe detection. Several objects, including checkerboard, metal-flat plane and free-form objects with complex reflectance properties, were reconstructed to validate our proposed method, which illustrates the effectiveness on improving the reconstruction accuracy of complex objects. The three-step phase-shifting algorithm was implemented and the reconstruction results were given and also compared with ours. In addition, our proposed framework provides a new feasible scheme for solving the ongoing problem of the reconstruction of complex objects with variant reflectance. The problem can be solved by subtracting the non-Lambertian components from the original grey values of stripe to improve the accuracy of stripe detection. In the future, based on stripe structured light technique, more general reflection models can be used to model different types of reflection properties of complex objects.
Collapse
Affiliation(s)
- Zhao Song
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China; (Z.S.); (Y.Y.)
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - Zhan Song
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China; (Z.S.); (Y.Y.)
- Mechanical and Automation Engineering Department, The Chinese University of Hong Kong, Hong Kong 3947700, China
- Correspondence:
| | - Yuping Ye
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China; (Z.S.); (Y.Y.)
- University of Chinese Academy of Sciences, Beijing 100049, China
| |
Collapse
|
11
|
Yu C, Lee SW. Deep Photometric Stereo Network with Multi-Scale Feature Aggregation. SENSORS 2020; 20:s20216261. [PMID: 33153006 PMCID: PMC7675179 DOI: 10.3390/s20216261] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/08/2020] [Revised: 10/30/2020] [Accepted: 11/02/2020] [Indexed: 11/21/2022]
Abstract
We present photometric stereo algorithms robust to non-Lambertian reflection, which are based on a convolutional neural network in which surface normals of objects with complex geometry and surface reflectance are estimated from a given set of an arbitrary number of images. These images are taken from the same viewpoint under different directional illumination conditions. The proposed method focuses on surface normal estimation, where multi-scale feature aggregation is proposed to obtain a more accurate surface normal, and max pooling is adopted to obtain an intermediate order-agnostic representation in the photometric stereo scenario. The proposed multi-scale feature aggregation scheme using feature concatenation is easily incorporated into existing photometric stereo network architectures. Our experiments were performed with a DiLiGent photometric stereo benchmark dataset consisting of ten real objects, and they demonstrated that the accuracies of our calibrated and uncalibrated photometric stereo approaches were improved over those of baseline methods. In particular, our experiments also demonstrated that our uncalibrated photometric stereo outperformed the state-of-the-art method. Our work is the first to consider the multi-scale feature aggregation in photometric stereo, and we showed that our proposed multi-scale fusion scheme estimated the surface normal accurately and was beneficial to improving performance.
Collapse
Affiliation(s)
- Chanki Yu
- Department of Media Technology, Graduate School of Media, Sogang University, Seoul 04107, Korea;
| | - Sang Wook Lee
- Department of Media Technology, Graduate School of Media, Sogang University, Seoul 04107, Korea;
- Department of Art & Technology, School of Media, Arts and Science, Sogang University, Seoul 04107, Korea
- Correspondence: ; Tel.: +82-2-705-8902
| |
Collapse
|
12
|
Shandro BM, Emrith K, Slabaugh G, Poullis A, Smith ML. Optical imaging technology in colonoscopy: Is there a role for photometric stereo? World J Gastrointest Endosc 2020; 12:138-148. [PMID: 32477448 PMCID: PMC7243575 DOI: 10.4253/wjge.v12.i5.138] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/29/2020] [Revised: 05/08/2020] [Accepted: 05/12/2020] [Indexed: 02/06/2023] Open
Abstract
Colonoscopy screening for the detection and removal of colonic adenomas is central to efforts to reduce the morbidity and mortality of colorectal cancer. However, up to a third of adenomas may be missed at colonoscopy, and the majority of post-colonoscopy colorectal cancers are thought to arise from these. Adenomas have three-dimensional surface topographic features that differentiate them from adjacent normal mucosa. However, these topographic features are not enhanced by white light colonoscopy, and the endoscopist must infer these from two-dimensional cues. This may contribute to the number of missed lesions. A variety of optical imaging technologies have been developed commercially to enhance surface topography. However, existing techniques enhance surface topography indirectly, and in two dimensions, and the evidence does not wholly support their use in routine clinical practice. In this narrative review, co-authored by gastroenterologists and engineers, we summarise the evidence for the impact of established optical imaging technologies on adenoma detection rate, and review the development of photometric stereo (PS) for colonoscopy. PS is a machine vision technique able to capture a dense array of surface normals to render three-dimensional reconstructions of surface topography. This imaging technique has several potential clinical applications in colonoscopy, including adenoma detection, polyp classification, and facilitating polypectomy, an inherently three-dimensional task. However, the development of PS for colonoscopy is at an early stage. We consider the progress that has been made with PS to date and identify the obstacles that need to be overcome prior to clinical application.
Collapse
Affiliation(s)
- Benjamin M Shandro
- Department of Gastroenterology, St George's University Hospitals NHS Foundation Trust, London SW17 0QT, United Kingdom
| | - Khemraj Emrith
- Centre for Machine Vision, University of the West of England, Bristol BS16 1QY, United Kingdom
| | - Gregory Slabaugh
- Department of Computer Science, City, University of London, London EC1V 0HB, United Kingdom
| | - Andrew Poullis
- Department of Gastroenterology, St George's University Hospitals NHS Foundation Trust, London SW17 0QT, United Kingdom
| | - Melvyn L Smith
- Centre for Machine Vision, University of the West of England, Bristol BS16 1QY, United Kingdom
| |
Collapse
|