Deep learning driven colorectal lesion detection in gastrointestinal endoscopic and pathological imaging

Abstract

Colorectal cancer has the second highest incidence of malignant tumors and is the fourth leading cause of cancer deaths in China. Early diagnosis and treatment of colorectal cancer will lead to an improvement in the 5-year survival rate, which will reduce medical costs. The current diagnostic methods for early colorectal cancer include excreta, blood, endoscopy, and computer-aided endoscopy. In this paper, research on image analysis and prediction of colorectal cancer lesions based on deep learning is reviewed with the goal of providing a reference for the early diagnosis of colorectal cancer lesions by combining computer technology, 3D modeling, 5G remote technology, endoscopic robot technology, and surgical navigation technology. The findings will supplement the research and provide insights to improve the cure rate and reduce the mortality of colorectal cancer.

Key Words: Deep learning; Artificial intelligence; Image analysis; Endoscopic; Colorectal lesions; Colorectal cancer

Core Tip: The development of computer technology has promoted the progress of medical treatment. Artificial intelligence (AI) has been gradually applied in the medical field and achieved good results. The detection of colorectal lesions in the conventional gastrointestinal endoscopy is difficult, the diagnosis time is long, and there is often the problem of missed diagnosis and misdiagnosis. AI is a good aid for doctors. In this review, we summarize the application of AI in the detection of colorectal lesions in recent years, in order to provide reference for the follow-up development and research.

INTRODUCTION

Colorectal cancer (CRC) is one of the most common human cancers[1]. According to the latest cancer data survey in China, the incidence rates of CRC rank third and fourth and the mortality rates rank fifth and fourth among male and female cancers, respectively[2]. The cure rate of early CRC is more than 90%[3,4]; thus, early detection, early diagnosis, and early treatment are very important for reducing the incidence rate and mortality of CRC. However, in real clinical practice, the early discovery of CRC is very limited. Huang et al[5] summarized the recent progress of early diagnosis of CRC, which is based on approaches that include excreta, blood, computer-aided endoscopy, and enteroscopy evaluations. They indicated that artificial intelligence (AI)-assisted endoscopy, which adopts facial recognition technology based on AI, can quickly identify abnormal conditions based on analyses of images of the colorectal area, thus providing a timely warning to avoid nontumor polypectomy. In addition, this method has a high accuracy and sensitivity, which indicates the application prospects of computer-aided endoscopy in early CRC diagnosis. In recent years, the rapid development of AI technology has provided us with a new computer-based screening approach[6]. It is hoped to be able to detect, analyze, and classify colonic polyps automatically through the rapid and high-precision processing of endoscopic images via AI to distinguish tumor polyps that need to be removed from nontumor polyps that do not need to be removed and improve the early detection rate of tumors. All these processes require image analysis technology based on deep learning.

In this minireview, we briefly introduce the principles of deep learning-based image analysis in predicting colorectal lesions, as well as the recent progress and clinical application effects of deep learning techniques. Further, the shortcomings of existing studies and future research directions are summarized with a view to providing ideas for the next step of research.

PRINCIPLES OF DEEP LEARNING

Origin and development of deep learning

Deep learning is derived from research on artificial neural networks (Figure 1) and based on the combination of low-level features, the superposition of higher-level abstract feature attributes, and the classification of perceptual objects[7]. Its depth is mainly reflected in the multiple transformations of target recognition features from shallow to deep, and the abstract features are calculated in the deep neural network by stacking multilayer nonlinear mapping to aid in the classification[8]. The development process of deep learning is as follows: (1) Since Hopfield et al[9] proposed a neural network model with a complete theoretical basis in 1982, the problem of multilayer perceptron training for deep learning was solved and gradually formed a basic mature deep learning model in recent years; (2) In 2006, Hinton et al[10] pointed out that a “multihidden-layer neural network has better feature learning ability” and formally proposed the concept of deep learning; and (3) In recent years, deep learning has been widely used in the analysis and prediction of medical images. In March 2019, Wei et al[11] published an article on applying a deep learning network to a histological model of lung adenocarcinoma sections and performing pathological classification, and they obtained a model that could help pathologists more efficiently classify lung adenocarcinoma models.

Figure 1 Phase description.

Origin and development of convolutional neural networks

Convolutional neural networks (CNNs) or deep convolutional neural networks (DCNNs) are most commonly used deep learning models for image processing and analysis. Typically, one of the most popularly used CNNs is the recurrent neural network (RNN) model that has various network architectures, e.g., long short-term memory (LSTM) networks that are a type of recurrent neural network capable of learning order dependence in sequence prediction, Bidirectional-LSTM, and gated recurrent units that are a gating mechanism in recurrent neural networks introduced by Kyunghyun Cho and colleagues in 2014. Different CNNs deal with different problems such as object recognition, target detection and tracking, and medical image segmentation and classification.

A typical CNN is an in-depth learning architecture that has excellent performance in image target recognition[12]. A CNN is a series of methods to reduce the dimensionality of the image recognition problem with a large amount of data and extract data features effectively, and it is widely used in image and video recognition, recommendation systems, and natural language processing[13]. The first CNN was proposed by Waibel et al[14] in 1987, and it was first applied to handwritten font recognition by Lecun et al[15] in 1989 . Data have shown that convolution layers based on deep learning will make great achievements in image recognition, speech recognition, and computer vision, including the prediction of colonic polyps. The most classical pattern of CNNs applied to image classification was constructed by Lecun et al[16], who built a more complete CNN. Since then, other application research based on CNNs has been performed. In 2003, Microsoft developed optical character reading using a CNN[17]. In 2004, Garcia et al[18] applied a CNN to facial recognition . In 2014, Abdel-Hamid et al[19] applied a CNN to speech recognition.

While CNNs are widely introduced into various computer vision tasks, e.g., face and facial recognition, text image extraction and recognition, and 3D reconstruction, they are still a relatively new analysis method and technique in the field of medical image analysis (Figure 2). Because of specific medical applications of CNNs, there are relatively few studies that have focused on medical image analysis. The research procedure including arriving at results and translating these results to clinical verification is generally long, which results in few studies on CNN driven medical image analysis. Currently, many researchers have employed CNNs to various clinical tasks of medical image processing and analysis. By applying CNNs to clinical data analysis, the diagnostic yield can be improved as well as clinical outcomes can be enhanced[20-22].

Figure 2 Origin and development of convolutional neural network.

DEEP LEARNING DRIVEN COLORECTAL LESION ANALYSIS

At present, deep learning CNNs are widely used in speech recognition[23], face recognition[24,25], and behavior recognition[26]. In clinical image analysis, these networks are also widely used in feature recognition classification or image segmentation model construction[27,28]. Deep learning-based image analysis plays a key role in improving the diagnostic accuracy of many clinical ailments[29]. First, in terms of medical image segmentation, a deep learning algorithm combined with other methods can analyze the heart, cervical cancer, and colon polyps[30]. In deep learning-based image recognition, benign and malignant focal lesions can be classified, thus indicating the diagnostic accuracy and sensitivity but also the superior specificity[31]. Image analysis has achieved good results in colonoscopy. Optical biopsy can be used as a diagnostic method to accurately predict the histological changes of 5-mm or smaller polyps[32]. Recent studies have successfully used automated image analysis techniques to accurately predict histopathology based on images captured by endoscopy and magnifying endoscopy[33].

For image diagnoses of clinical colorectal lesions, high-resolution endoscopy, fluorescence imaging, enhanced endoscopy, and other advanced technologies are available to improve the detection rate of tumors under endoscopy; however, the inspection level of endoscopists is important to fully exploit these advanced technologies[34-36]. The application of deep learning in computer-aided detection improves the detection rate of early CRC and polyps and the inspection quality of optical biopsy, reduces the influence of doctors’ inspection level on the inspection results, improves the detection rate of tumors, reduces the rates of missed and misdiagnosed cases, and improves the quality of endoscopy[37].

Endoscopic video colorectal lesion detection

The detection of colonic polyps has become one of the most important fields in the application of AI deep learning for the detection of colorectal endoscopy. A large number of studies have shown that the detection rate of polyps is related to cancer risk. Summers et al[38] have shown that computer-aided detection can help inexperienced clinicians because of its sensitivity in detecting polyps; thus, it can balance the gap between different levels of endoscopic physicians and improve the accuracy of diagnosis. Corley et al[39] assessed the association of polyp detection rates with CRC risk and cancer-related deaths diagnosed 6 mo to 10 years after colonoscopy and concluded that polyp detection rates were negatively correlated with the risk of interstitial CRC, advanced interphase cancer, and fatal interphase cancer. Computer-aided detection has great advantages in terms of improving the detection rate of polyps and reducing the cost of examination.

In this review, we investigate important research on computer-aided detection of colon diseases in recent years (Table 1). Computer-aided detection systems for colon diseases in gastrointestinal endoscopy have been developed continuously since 2003 and numerous studies have been published in the literature. In 2003, Karkanis et al[40] extracted color wavelet features to test the performance of computer-aided colon tumor detection and found that its specificity was as high as 97% and sensitivity was as high as 90%, which is a very significant breakthrough in this field. Recently, Misawa et al[41] developed an AI system based on modeled deep learning. The system could detect 94% of polyps, and the false-positive detection rate was 60%, which verifies the feasibility of the detection system. This retrospective analysis confirmed that computer-aided detection can indeed play a great role in the diagnosis of colonic diseases.

Table 1 Summary of important studies of computer-aided endoscopic colorectal lesion detection.

Ref.	Methods and data	Important results	Limitation and drawback
Karkanis et al[40], 2003	Endoscopic video tumor detection by color wavelet covariance, supported by linear discriminant analysis, 66 patients with 95 polyps	Specificity 90% and sensitivity 97%	It is not enough stable to classify different types of colorectal polyps
Misawa et al[41], 2018	An AI-assisted CADe system using 3D CNNs, 155 polyp-positive videos with 391 polyp-negative	Sensitivity 90.0%, specificity 63.3%, and accuracy 76.5%	Further machine and deep learning and prospective evaluations are mandatory
Urban et al[42], 2018	CNNs; 8641 hand-labeled images with 4088 unique polyps	AUC of 0.991 and accuracy of 96.4%	Unknown effects of CNNs on inspection behavior by colonoscopists, anonymous and unidentified natural or endoscopic videos
Mori et al[43], 2018	Retrospective analysis: An AI system by machine learning, 144 diminutive polyps (≤ 5 mm)	Sensitivity 98%, specificity 71%, accuracy 81%, positive 67%, and negative 98%	Insufficient endoscopic video image data
Yamada et al[44], 2020	Retrospective analysis: A deep learning driven system using a Single Shot Multibox Detector for capsule endoscopic colon lesions detection, 15933 training images and 4784 testing images	AUC 0.902, sensitivity 79.0%, specificity 87.0%, accuracy 83.9%, and at a probability cutoff of 0.348	It was a retrospective study that only used the selected images, while it also did not consider pathological diagnoses and the clinical utility of the AI model has not been evaluated

AI: Artificial intelligence; CNN: Convolutional neural network; AUC: Area under the curve.

Previous clinical data showed that in a group of 8641 colonoscopy images containing 4088 unique polyps, deep learning could locate and identify polyps in real time, and the accuracy rate in colonoscopy screening was approximately 96%[42]. At present, Japan has developed EndoBrain[43], which uses AI to analyze the blood vessels and cell structure of the lesion site and shows the tumor probability in an instant, and it has been used to identify tumor or nontumor polyps. To better distinguish the invasion depth of lesions and reduce the misdiagnosis rate, various dyes must be used on the mucous membrane, which limits the current routine application of this technology. Yamada et al[44] developed an AI system that uses deep learning to automatically detect such lesions in CCE images, and after training with 15933 CCE images and assessing 4784 images, the sensitivity, specificity, and accuracy of this system were 79.0%, 87.0%, and 83.9%, respectively. The effectiveness of AI technology was demonstrated in 324 patients in a randomized controlled trial by Wu et al[45], and compared to the control group, the blind spot rate was reduced (5.68% vs 22.46%, P < 0.001).

Computer-aided endoscopic detection has important potential in the field of colon diseases and is under continuous research and development. Its high specificity and sensitivity can help to improve the detection rate of various diseases and help doctors judge the condition. However, the existing studies still have shortcomings, such as the system is not stable enough, the classification is not complete enough, it is influenced by the operator and the subject, more clinical practice proof is needed, etc. Therefore, future studies will identify a more robust classification scheme and the developed system can be enhanced with a classifier fusion scheme to identify different types of colorectal polyps. More importantly, future randomized studies could directly address the overall value (quality vs cost) of CNN by examining the impact of CNN on colonoscopy time, pathology cost, ADR, polyps per procedure, surveillance-associated polyps per procedure, and surveillance-unassociated polyps per procedure (e.g., normal and lymphatic aggregates).

Pathological image colorectal lesion detection

Among colonic mucosal diseases, optical biopsy can accurately identify the activity degree of ulcerative colitis and the nature of ulcerative colitis-related intraepithelial neoplasia and colonic polyps. Optical biopsy is a new endoscopic diagnosis technology represented by confocal microscopy, and its principle is similar to that of confocal microscopy[46] according to the use of advanced imaging technology combined with an existing classification system. However, the precision of optical biopsy is based on the professional knowledge of the operators.

Although optical biopsy technology depends on the professional knowledge of the operator, the development of computer-aided technology in recent years can aid in a more accurate diagnosis (Table 2). At present, AI technology can realize automatic optical biopsy, which is mainly based on the extraction of image features of colonic lesions, sorting and then inputting the input layer into the computer system for deep learning, sorting the results in the output layer, and then finally outputting the diagnosis results (Figure 3). Although outstanding achievements have not yet been made in image analysis and predictions of colorectal lesions by deep learning in China, Tamaki et al[47] proposed a new combination of local features and sampling schemes and tested 908 narrow band imaging (NBI) images. The system achieved a recognition rate of 96% for 10-fold cross validation on a real dataset of 908 NBI images collected during actual colonoscopies, and the rate was 93% for a separate test dataset. Then, Nao Ito and other Japanese scholars[48] published relevant research results of endoscopic diagnosis in support of a cT1b CRC deep learning system. The accuracy of the CNN in this study was 81.2%, which showed that the effect of CNN examinations is equivalent to the judgment of clinicians in endoscopic diagnosis. The skill of deep learning of surgical navigation is expected to be applied to assistant clinicians in endoscopic examinations. In another study[49], the authors created an autonomous computational system to classify endoscopy findings and showed that autonomous classification of endoscopic images with AI technology is possible. The overall accuracy for the benign classifier was 80.8%. The binary classifier correctly identified 92.0% of the malignant-premalignant lesions, with an overall accuracy of 93.0%. However, better network implementations and larger datasets are needed to improve the classifier’s accuracy. Zachariah et al[50] demonstrated the feasibility of in situ diagnosis of colorectal polyps using CNN. Their model exceeded PIVI thresholds for both “resect and discard” and “diagnose and leave” strategies independent of NBI use. Point-of-care adenoma detection rates and surveillance recommendations are potential added benefits. The study of Shahidi et al[51] provided the first description of a potential future application of AI, in which AI can help in the arbitration between endoscopists and pathologists when discordant diagnoses occur. The study results were consistent with the endoscopic diagnosis in 577 (89.6%) lesions. Concerning discordant endoscopic and pathologic diagnoses, the results were consistent with the endoscopic diagnosis in 168 (90.3%) lesions. Of those lesions identified on pathology as normal mucosa, 90 (90.9%) were consistent with the endoscopic diagnosis.

Figure 3 Computer-aided diagnosis of colonic lesions.

Table 2 Important studies of computer-aided pathological prediction of colorectal lesions.

Ref.	Methods and data	Important results	Limitation and drawback
Tamaki et al[47], 2013	A new combination of local features and sampling tested on 908 NBI images	A recognition rate of 96% for 10-fold cross validation and a rate of 93% for separate data	Without investigation on robustness, motion blur, focus, window size, color bleeding, and highlight areas
Ito et al[48], 2018	Use AlexNet to diagnose cT1b, 190 colorectal lesion images from 41 patient cases	Sensitivity 67.5%, specificity 89.0%, accuracy 81.2%, and AUC 0.871	Insufficient pathological images to build CNNs
Zachariah et al[50], 2020	A CNNs model using TensorFlow and ImageNet, 6223 images with 80% train and 20% test, processing at 77 frames per second	Negative 97% among diminutive rectum/ rectosigmoid polyps, surveillance interval 93%. In fresh validation, NPV 97% and surveillance interval 94%	Retrospective study using offline images
Shahidi et al[51], 2020	An established real-time AI clinical decision support solution to resolve endoscopic and pathologic discrepancies, 644 images with colorectal lesions ≤ 3 mm	CDSS was consistent with the endoscopic diagnosis in 577 (89.6%) lesions	Inevitable CDSS optimization, given the increasingly used deep learning for development of current AI platforms, manifesting in AI’s ability to adapt with increasing data exposure

AI: Artificial intelligence; CNN: Convolutional neural network; AUC: Area under the curve; CDSS: Clinical decision support system.

Based on these studies, we reveal the application value and development prospect of AI in optical biopsy. Of course, there are also some limitations, such as inadequate images, many influences, inconsistent sizes, color differences, and others. In the future, new computer hardware, algorithms, and multicenter model cross-validations are needed to improve the accuracy of diagnosis for clinical use. In addition, the combined and joint applications of colon disease big data and AI computer systems are also essential. The application of AI requires analysis, classification, and deep learning based on a large amount of data. The larger the amount of data, the higher the accuracy of the final learning system. However, at present, we still lack the support of such a large amount of data. If such a large amount of data can be collected, the application of AI in optical biopsy will make great progress.

CONCLUSION

With the development of science and technology and the improvement of deep learning algorithms, AI will be applied in many fields in the future[52]. At the same time, with the improvement of computer technology and the increase in image data, the application of image analysis and prediction based on deep learning for clinical colorectal lesions will be gradually increased and the accuracy of diagnosis will be significantly improved. The application of AI in clinical work will greatly reduce the workload of clinicians.

At present, deep learning algorithms have shown good benefit for histopathological diagnosis in the context of tumor risk stratification[53]. Recent applications have focused on the most common types of cancer, such as breast, prostate, and lung cancer. At present, research on the application of image analysis of colorectal lesions based on deep learning has been able to distinguish the pathological types of colorectal polyps and improve the detection rate of polyps. We hope to be able to collect colorectal disease images (including endoscopic ultrasound images) and image data through computer-aided technology of relevant depth learning combined with the advantages of clinical big data and develop an image processing method that can determine the invasion depth of colorectal disease and construct three-dimensional images in the process of endoscopic colonoscopy to guide further diagnosis and treatment. Computer-aided diagnosis (CADx) has been used for cancer staging and invasion depth estimation[54], and Kubota et al[55] developed another CADx for the automatic diagnosis of gastric cancer invasion depth. AI systems help determine whether additional surgery is needed after endoscopic resection of T1 CRC by predicting lymph node metastasis[56]. According to the bronchoscope navigation system researched by Luo and others, an operation navigation system of the digestive tract can be developed to realize the localization and treatment of lesions[57]. At present, an image-based navigation strategy is proposed in Van Der Stap[58] to realize the automation of the flexible endoscope, and a framework composed of robot steering and cavity concentration is proposed to realize the automation of the colonoscope[59]. Perhaps in the future, endoscopic surgery will present operations similar to that of a surgical Da Vinci robot. For example, a soft endoscopic robot was developed to replace hard endoscopic surgery for surgical transanal resection of tumors. In addition, with the development of new materials and computer technology, printing digestive tract models with 3D materials and using 5G remote technology to improve the efficiency of diagnosis and follow-up could increase the detection rate of early CRC[60,61] (Figure 4).

Figure 4 Future development directions. AI: Artificial intelligence.

Visual processing of computer images and videos has achieved excellent results, thus showing its superiority in clinical medical diagnosis and examination and resolving the gap between doctors at different levels. Further research and popularization will be of great significance for the diagnosis and treatment of colorectal lesions and could significantly reduce the incidence rate of CRC and improve patient survival rates.