Online volume rendering of incrementally accumulated LSCEM images for superficial oral cancer detection

Abstract

Laser scanning confocal endomicroscope (LSCEM) has emerged as an imaging modality which provides non-invasive, in vivo imaging of biological tissue on a microscopic scale. Scientific visualizations for LSCEM datasets captured by current imaging systems require these datasets to be fully acquired and brought to a separate rendering machine. To extend the features and capabilities of this modality, we propose a system which is capable of performing realtime visualization of LSCEM datasets. Using field-programmable gate arrays, our system performs three tasks in parallel: (1) automated control of dataset acquisition; (2) imaging-rendering system synchronization; and (3) realtime volume rendering of dynamic datasets. Through fusion of LSCEM imaging and volume rendering processes, acquired datasets can be visualized in realtime to provide an immediate perception of the image quality and biological conditions of the subject, further assisting in realtime cancer diagnosis. Subsequently, the imaging procedure can be improved for more accurate diagnosis and reduce the need for repeating the process due to unsatisfactory datasets.

Key Words: Confocal endomicroscope; Field-programmable gate arrays; Incrementally accumulated volume rendering; Realtime; Online cancer detection

INTRODUCTION

A variety of medical imaging modalities such as computed tomography (CT) scanning[1], ultrasound imaging[2] and magnetic resonance imaging (MRI)[3] are used to capture digital images of biological tissue and structures. The laser scanning confocal endomicroscope (LSCEM)[4] is an emerging modality used to perform non-invasive, in vivo scanning at a microscopic scale beneath the tissue surface. This technique is also useful to capture volumetric datasets by capturing slices at progressive depth levels. The LSCEM which is an extended variation of the laser scanning confocal microscope (LSCM)[5,6], has a miniature probe to perform in vivo imaging on live tissue in hard to reach areas.

Typically, LSCM is able to penetrate and section a specimen up to 50 µm or more[7], which enables image acquisition at different depths, subsequently permitting volumetric dataset capturing. The image quality is much improved compared to other widefield microscopy techniques[6] due to the isolation from background fluorescence signals. Furthermore, this modality enables live in vivo imaging[8], minimizing patient’s agony of biopsy and avoiding flaws arising due to physical tissue cutting and specimen staining.

However, the procedural tasks for undergoing LSCEM imaging involve consumption of valuable time and resources. Firstly, current LSCEM imaging systems require manual control fully operated by the user. Secondly, the presence of typical imaging flaws such as poor dataset quality, which can only be realized after imaging ceases, will require the procedure to be re-performed. Apart from that, the lack of a realtime volume visualization system limits the flexibility for changes on-the-spot during imaging.

Performing diagnoses for diseases and medical abnormalities is a complex operation which requires trained expertise to provide an accurate outcome. However, having experienced diagnosticians alone is not enough, as the help of additional imaging and computing systems are also crucial. There has been discussion about the need for interactive visualization of LSCM datasets[9]. Consequently, it is meaningful to explore various ideas and methods proposed to view 3D datasets as 2D images. Scientific visualization of medical datasets has always been important to aid practitioners relate changes across acquired data images and perceive them visually for more precise analyses. These relations and changes are normally difficult or impossible to realize under traditional methods of reading separate data. It is also clinically useful if the captured slices can be perceived from arbitrary viewing angles or highlight desired features. These functions introduce user interaction and further promote the effectiveness of visualization.

In view of that, we present an online visualization system which performs volume rendering on-the-fly while LSCEM imaging is being carried out. In our aim to improve the imaging procedure, this system should perform the required tasks without alterations to its pre-existing features and performances.

PREVIOUS WOKE

The LSCEM is a useful tool used to visualize microscopic tissue structures to a high level of detail. Since the advent of LSCEM for medical-based applications, many procedures have used this modality to obtain volume datasets[4,10-11]. Further improvements on the obtained datasets involved work including image processing methods. Some of the works include deconvolution methods[12-14] to mitigate image degradation[15], and volume reconstruction[16] for the misalignment problem. Recently LSCM has also been combined with reconfigurable computing to provide realtime imaging for instant diagnosis and tissue assessment[17].

Initial attempts to visualize LSCM datasets using volume rendering is presented by Sakas et al[18]. The authors discussed methods to render LSCM datasets, including surface reconstruction[19], maximum intensity projection (MIP)[20], and using illumination models. A specially developed visualization system known as proteus[21] is used to visualize time dependent LSCM data, focusing on the analysis of chromatin condensation and decondensation during mitosis. From the conducted experiments the authors concluded that there was no all around visualization technique at the time of publishing, and that various techniques should be offered, each with its own strengths and weaknesses.

Fast visualization of LSCM datasets using the ray casting algorithm has shown the capability to provide interactive examination of datasets, improving the understanding of LSCM imaging. Further enhancement using pre-processing for intensity compensation and image deconvolution to enhance the dataset is also useful.

Volume rendering is the study of displaying 3D objects onto a 2D screen. A generalized rendering equation[22] is initially used to model physical properties of light transport. This equation describes the way objects are viewed in reality, enabling computing systems to simulate these phenomena. Subsequently, this equation is used as a theoretical basis for varying model derivations[23]. One of the most commonly used methods for rendering objects is the ray tracing algorithm. The algorithm can be used to model different objects, including polynomial surfaces[24], fractal surfaces and shapes[25], as well as volumetric densities[26].

Initially, CT scanned datasets are visualized using multiplanar reconstruction[27], which provides cross sectional representations using arbitrary configurations. The VolVis framework[28-29] is used to visualize MRI and confocal cell imaging datasets. This system is further extended to include global illumination and shadow casting[30]. A multidimensional transfer function[31] is used to render the human head and brain datasets into meaningful imaging, enabling the human user to intuitively identify different tissue material and boundaries. An open source toolkit known as the medical imaging interaction toolkit (MITK)[32], promotes interactivity and image analysis specifically for visualization of medical datasets. This system is subsequently included in the development of a software platform for medical image processing and analysis[33], with the aim of incorporating medical image processing algorithms into a complete framework.

MATERIALS AND METHODS

Overview

The involved imaging device in our project is the Optiscan FIVE1 Endomicroscope (Optiscan Imaging Ltd., Victoria, Australia). This device is capable of scanning an area of 475 μm × 475 μm with up to 1024 × 1024 resolution. The scanning time for each slice with the highest resolution is approximately 1.4 s. Operating the LSCEM device requires at least one person. Functional interfaces include: (1) directing the probe to the desired imaging area; (2) controlling image scanning using a footswitch control panel; and (3) managing the dataset through the imaging software on a PC.

The conventional imaging procedure on the device is commenced by applying fluorescent material to the tissue. After absorption, the tissue will emit light signals when laser light is applied. Capture trigger and imaging depth are fully operated manually by the user. To capture a full volumetric dataset, the operator has to repetitively increment depths and capture slices individually. This results in a prolonged imaging session, where tissue or probe movements may be introduced to compromise the dataset accuracy. Also, such controls are tedious and repetitive, so an automated system is desired.

We built our prototype system on a field-programmable gate array (FPGA) chip. The Celoxica RC340 FPGA Development board is the main platform we use to program the FPGA. FPGAs are future oriented building bricks that allow logic circuits to be implemented in any fashion. System-on-Chip designs can thus be built on a single device using FPGAs. FPGAs are also well-known for the reconfigurable feature, which allows different designs to be implemented on the same chip. This promotes lower design costs and ease of prototyping. In our system, we integrate automated imaging control, realtime rendering and task allocation into a single device, which makes FPGAs suitable as they do not adhere to any fixed computational or programming models.

The FPGA is responsible for running three core tasks in parallel. Firstly, to efficiently couple the imaging and rendering systems reliably, task allocation and synchronization is maintained by the FPGA. Secondly, the FPGA acts as an automated controller to replace the existing footswitch for dataset acquisition. Finally, the entire volume rendering system is built on the FPGA due to support for full algorithm flexibility.

The image-ordered ray casting algorithm is used for rendering. This algorithm casts rays from each pixel on the screen towards the dataset and is an iteration process which visits each pixel on the viewing screen. Along each ray, points of a constant distance are sampled. Finally sampling points on each ray are combined in a compositing module and the output is used as the display pixel. Various compositing methods are present, commonly MIP, averaging, nearest-point and alpha compositing.

System design

We designed the imaging-rendering system as shown in Figure 1. We developed the novel volume renderer using reconfigurable hardware for experimentation flexibility and cheaper cost for replicating prototypes. Through a handheld probe the Optiscan FIVE1 imaging device captures endomicroscopic images at incremental steps within the tissue depth to build up volumetric data. This is achieved by detecting the light signals emitted from the fluorescent chemical applied to the biological tissue. These signals, when captured by confining to a single focal plane, form an image slice. Consecutive slices captured across increasing depth will thus form a volume dataset.

Figure 1 The system architecture.

Default operations involve using the provided footswitch for depth control, calibration and triggering the capture process. The supplied configuration requires the user to increment capture depths slice-by-slice and send instructions to manually progress to the next slice and trigger a capture. This poses disadvantages which include prolonged capture time and complications that arise, due to change of state and condition of the tissue. Requiring the user to hold the probe while performing stepping controls on the footswitch will introduce distortion due to movements, which is significant at a microscopic scale.

Apart from that, to perform continuous online rendering, scanning along the z-depth direction must be continuous and progress incrementally deeper into the tissue. In response, we developed an alternative hardware to substitute the footswitch[34]. The main task of the relay switchbox hardware is to replace the manual controls of the footswitch by sending automated signals at constant intervals to increment the z-depth value every time a slice capture is complete. The user is no longer required to manually increment plane depths and initiate capture with this substitute.

For this design, the FPGA is required to fulfill the following requirements: (1) capability to perform all the rendering computations; (2) provide realtime control signals over the imaging device; and (3) support automated scanning and capturing controls.

In the LSCEM device, the captured data are buffered in an internal memory while the scanning point moves across the subject in a raster scanning fashion. We forward an entire slice after each completed capture via the USB to our rendering device, ensuring that the rendered result is continuously updated as soon as the slice is acquired. Once the first slice is received, rendering is initiated. An electronic controller manages the rendering and imaging tasks, so the operations of the imaging device do not affect the renderer and vice versa. Rendering is performed in conjunction with acquisition, where after each capture the relay switchbox instructs the probe to progress to the next slice automatically.

The imaging-rendering arrangement is illustrated in Figure 2. Conventional practices require the full dataset to be completed and imaging to be ceased before rendering in an alternative engine. We allocate these two processes to be performed concurrently, so that immediate results can be shown as soon as a slice is captured. The rendering and imaging operations are performed in parallel, and the imaging always captures an additional slice to be rendered for the next cycle.

Figure 2 Imaging-rendering allocation. A: Conventional procedure; B: Simultaneous dataset capture and rendering.

The process is initiated with a slice capture by the imaging system, while the renderer is on standby as the dataset is empty. After a complete slice is captured, it is forwarded and stored in the SDRAM of the FPGA system. Rendering proceeds by retrieving required voxels for computation from the SDRAM. The SDRAM as a shared resource between imaging and rendering must be allocated to be mutually exclusive to avoid access conflicts.

The operations repeat until the final slice as determined by the user, when capturing is ceased. Rendering can still be performed on the existing dataset until it is terminated by the user.

OPTIMIZATIONS

To accommodate for the realtime requirements of the system, we have also employed several custom designed optimizations on the FPGA rendering system.

Pre-computing ray parameters for ray casting traversal

This optimization eliminates repetitive matrix transformation computations for sampling coordinates. In this case, matrix multiplications are reduced to simple arithmetic addition and subtraction operations. The ray parameters are represented by a coordinate for the ray origin and a vector for the ray direction.

Cubic memory organization for dataset storage

We store the dataset voxels according to a customized arrangement in the SDRAM as shown in Figure 3. This is due to trilinear interpolation for sampling voxels, which requires eight memory accesses to the SDRAM. Storing dataset voxels in a cubic fashion enables fast retrieval of these points for computations in a single cycle due to the 64-bit data bus. Using this technique, sampling points require less memory access for more efficient operations.

Figure 3 Cubic memory organization: Voxels are arranged in memory as cubes.

Accumulative rendering mode

We introduce the accumulated rendering mode in this system. Rather than revisiting the whole dataset when a new slice is updated, the previous rendering output is directly combined with the new slice instead. Through this computation, load is limited to only two slices, regardless of the size of the dataset. Modifications to the existing compositing methods will be required, and we would anticipate further refinement and utilize it to our advantage. We perform compositing for accumulative rendering using the following formula.

To accommodate the accumulative rendering mode, we use four framebuffers to store intermediate slices. This novel design is illustrated in Figure 4. We use four SRAM blocks as the framebuffers, two of which facilitate compositing and the remaining two as display buffers. This is used to overcome the problem of simultaneous read/write accesses, prohibited by the use of a single data bus.

Figure 4 Framebuffers for accumulative rendering mode.

During operation, the first slice only involves memory writing, where one SRAM from each framebuffer is accessed to store the acquired slice. In all other stages, the SRAM blocks interchange between memory reading and memory writing to ensure continuous flow, providing simultaneous reading and writing of rendering results. This is significant in the case of parallel pipelined coding, where memory read and write operations are continuous at each cycle, while the operations are only repeated once for each pixel. In single data placement systems, the read and write operation has to be time-multiplexed, resulting in longer time required for processing.

Optimization limitations

The introduction of these optimization techniques speeds up the rendering computation, but at the expense of several limitations. In online mode, a constant viewing configuration is employed. Viewing angles and distances cannot be adjusted in this mode, as pre-computation of ray parameters assumes that the rays are always fixed in such a manner. This limitation is further reinforced with the method of combined slices, as a view change will require the whole dataset to be re-traversed instead of only visiting the existing rendering and the new slice.

Rendering is also performed only for parallel projections, which assume that all casted rays have the same direction. Further extensions can be done to include perspective projection ray casting through modification of the system and additional memory requirements to store different ray parameters for each pixel.

Also, cubic memory scheme organizes memory cubes continuously instead of interleaved. This introduces duplications of the voxels in memory, thus increasing the memory requirements to approximately eight times dataset size. However, the capacities supported by modern hardware and our system of 256 Mbytes on-board memory can easily satisfy these requirements.

Accumulative rendering mode is currently supported only for orthographic projection rendering. The viewing direction is always assumed to be perpendicular to the slices for accumulative rendering. Further refinement of this design can enable views from arbitrary directions. As this technique only renders two slices, cubic memory is not employed for rendering under this mode, but is used for rendering under arbitrary view configurations.

EXPERIMENTAL RESULTS AND DISCUSSION

The prototype system has been built and deployed with our article-titleorators at the National Cancer Center of Singapore (NCCS) for clinical experiments. A photo of the system is shown in Figure 5. This system includes the Optiscan FIVE1 LSCEM imaging device, the Celoxica RC340 FPGA Development board, the relay switchbox hardware, and user interactivity tools.

Figure 5 The Laser scanning confocal endomicroscope imaging-rendering prototype system.

We show the realtime online rendering results for a mouse tongue dataset in Figure 6. The rendering results are live results obtained during the imaging experiment. For rendering performance, the rendering system has to at least be able to perform faster than the rate slices are acquired in order to be meaningful. This rate is approximately 1.4 s per slice. The maximum number of slices is dependent on the significance of signal degradation as the focal plane advances deeper into the tissue. For our case, the maximal is around 40 slices.

Figure 6 Online rendering of the mouse tongue surface across increasing number of slices. A: 2 slices; B: 10 slices; C: 15 slices; D: 20 slices; E: 25 slices; F: 30 slices; G: 35 slices; H: 38 slices.

The rendering results in Figure 6 show two different views. The orthographic projection renders the dataset by setting the viewing plane parallel to the acquired slices, under accumulative rendering mode. This shows perceived results viewed through consecutive depths. At the bottom right corner is a parallel projection with a specific viewing configuration, which shows an overview of the current dataset. Observations can be made through these different views across increasing thicknesses to the dataset.

The rendering will provide immediate perception of the acquired datasets at high quality. This is aimed towards improving the imaging procedure and further enhancing the features and capabilities of LSCEM imaging. With this current platform, further extensions can be included, such as image processing techniques to provide automated diagnosis of cancer stages[17].

The entire imaging procedure is now simplified, and can be fully operated by a single user. The user can adjust the probe either by a handheld device or through a fixed stand. Capture controls are fully automated, where the user is only required to initiate and terminate imaging controls. This enhances the imaging time as well as requiring less attention from the user to operate different controls at the same time.

CONCLUSION

We have presented our preliminary system to perform online rendering side-by-side with LSCEM imaging. By having an immediate knowledge of the dataset quality as well as the biological tissue conditions, alterations can be made on the spot. This will introduce the opportunity to change imaging conditions or medical decisions according to the online rendering results. This work is also motivated by the need to realize the quality of the captured datasets in realtime to reduce excessive time required for offline rendering.

The LSCEM procedure is also simplified and only one user is required to fully operate the system. Using electronic automation to substitute the footswitch as manual controls, the user is not required to repetitively control the imaging sequences to capture a full volumetric dataset. This reduces the time and attention required to perform LSCEM imaging.

We have also shown the FPGA as a viable solution for designing a system which can run the required tasks in parallel. Providing automated controls to substitute the prevailing manual capture, we make use of our custom built hardware to interface with the imaging system. Also the FPGA is able to provide synchronized controls between imaging and rendering, which is crucial in ensuring a stable operation. The full flexibility offered by FPGAs is important, considering future developments for embedded rendering systems to co-exist with imaging devices.

We would extend this work further by incorporating more sophisticated rendering procedures such as perspective projection rendering and interactive view changes in realtime.