Optical diagnosis of colorectal polyps using convolutional neural networks

Abstract

Colonoscopy remains the gold standard investigation for colorectal cancer screening as it offers the opportunity to both detect and resect pre-malignant and neoplastic polyps. Although technologies for image-enhanced endoscopy are widely available, optical diagnosis has not been incorporated into routine clinical practice, mainly due to significant inter-operator variability. In recent years, there has been a growing number of studies demonstrating the potential of convolutional neural networks (CNN) to enhance optical diagnosis of polyps. Data suggest that the use of CNNs might mitigate the inter-operator variability amongst endoscopists, potentially enabling a “resect and discard“ or ”leave in“ strategy to be adopted in real-time. This would have significant financial benefits for healthcare systems, avoid unnecessary polypectomies of non-neoplastic polyps and improve the efficiency of colonoscopy. Here, we review advances in CNN for the optical diagnosis of colorectal polyps, current limitations and future directions.

Key Words: Artificial intelligence, Deep learning, Convolutional neural networks, Computer aided diagnosis, Optical diagnosis, Colorectal polyps

Core Tip: A convolutional neural network (CNN) is a specific type of artificial intelligence deep learning. These networks may play an important role in the coming years in assisting endoscopists to optically diagnose colorectal polyps. CNNs can mitigate the inter-operator variability amongst endoscopists, potentially enabling a “resect and discard” or “leave in” strategy to be adopted. This would improve the efficiency of colonoscopy, reduce healthcare costs and reduce adverse events for patients by avoiding unnecessary resections of non-neoplastic polyps. In this article, we expand on the most relevant studies in this field and discuss limitations and future directions that will determine fulfilment of the potential of CNN in the optical diagnosis of colorectal polyps.

INTRODUCTION

Colorectal cancer (CRC) is the third most commonly diagnosed cancer worldwide[1] and thus, a significant burden on global healthcare systems. Most CRCs develop in a relatively predictable, stepwise sequence from mutation-accumulating neoplastic polyps, such as adenomas and sessile serrated lesions (SSL)[2]. Current evidence-based societal guidelines unequivocally accept colonoscopy to be the gold standard tool for screening of CRC[3]. Colonoscopy offers the opportunity to both detect and resect neoplastic polyps[4] and its implementation, especially as part of bowel cancer screening programs, has been linked to a significant reduction in the incidence of the CRC and CRC-related mortality[5].

Over 90% of polyps detected at colonoscopy are either small (6-9 mm) or diminutive (≤ 5 mm), entities that are thought to harbour a very low risk for developing into CRC[6]. Furthermore, almost half of these polyps are non-neoplastic in nature; and frequently hyperplastic[7]. Accurate differentiation of neoplastic from non-neoplastic polyps can prevent the unnecessary resection of the latter, avoiding an intervention which is not cost-effective and which carries risks of significant morbidity[8].

Recent years have seen significant research activity in the use of artificial intelligence (AI), particularly convolutional neural networks (CNN), to optically diagnose colorectal polyps. The field is gaining increasing momentum. The aim of this review article is to summarise and critically appraise the available medical literature related to advances in CNN for optical diagnosis of colorectal polyps and highlight the field’s current limitations and future directions.

OPTICAL DIAGNOSIS

The term “optical diagnosis” refers to the use of advanced imaging techniques for real-time, in-vivo polyp characterisation and evaluation to guide therapeutic decisions[9]. Accurate optical diagnosis of diminutive polyps would enable identification of hyperplastic polyps in the rectosigmoid region, where they are commonly found, and allow the endoscopist to confidently take a “diagnose and leave” approach instead of resecting the lesion. Equally, for diminutive adenomas, accurate optical diagnosis would prompt the endoscopist to remove the lesion on the spot and discard the specimen without the need for histological assessment (“resect and discard”strategy)[9].

The American Society of Gastrointestinal Endoscopy established the Preservation and Incorporation of Valuable endoscopic Innovations (PIVI) to provide thresholds that are required of endoscopic technology in order to implement a “resect and discard”(PIVI 1) and “diagnose and leave” (PIVI 2) strategy[9]. PIVI 1 requires ≥ 90% concordance in post-polypectomy surveillance intervals when comparing the combination of optical diagnosis for diminutive adenomas with histopathology assessment of all other polyps against decisions based solely on histopathology evaluation of all identified polyps[10]. PIVI 2 requires a technology to achieve a negative predictive value (NPV) of ≥ 90% for diminutive adenomatous polyps in the rectosigmoid region[9].

There has been extensive research in image enhanced endoscopy (IEE), such as narrow band imaging (NBI), to assist endoscopists in optical diagnosis to characterise diminutive polyps[11-13]. Using IEE, expert endoscopists in academic centres have consistently demonstrated an optical diagnosis accuracy that exceeds PIVI thresholds[14-16], however, studies have often found community and non-expert endoscopists to fall short of these minimal thresholds[17]. An example is the multi-centre DISCARD-2 study which evaluated the optical diagnosis accuracy of 28 community endoscopists using NBI. Disappointingly, the endoscopists’ optical diagnosis derived colonoscopy surveillance intervals only matched 68% of the histopathology derived intervals[18]. Although widely available, technologies for optical diagnosis has not been incorporated into routine clinical practice with one of the main barriers being the inter-operator variability amongst endoscopists[19].

WHAT IS A CONVOLUTIONAL NEURAL NETWORK?

AI is the ability of computers to perform tasks that traditionally require human intelligence (Figure 1)[20]. Machine learning (ML) is a subset of AI, whereby computers continuously learn from data without explicit human programming[21]. This can be used to predicate a polyp’s histology. ML models can be trained using unsupervised or supervised techniques. Unsupervised learning is when the input and output data are not paired. Supervised ML is more labour intensive as it requires paired input and output data for training. An example of a supervised ML model for optical diagnosis is to annotate a bounding box around a polyp (input data), commonly referred to as a region of interest, and label it with the histology of the polyp (output data). The model automatically learns to extract features that allow it to differentiate polyp subtypes and output a diagnosis based on the histology classification system it was trained with but the annotation process is time consuming for the clinician.

Figure 1 The relationship between convolutional neural networks, deep learning, machine learning and artificial intelligence.

Deep learning is a subset of ML, whereby algorithms use multiple layers within a neural network[22], mimicking the human brain, to extract high level features from input data. CNNs are the most commonly used network in the application of deep learning to optically diagnose polyps. They provide an objective output, bypassing the human inter and intra-operator variability, and can develop classification algorithms without exhaustive effort as they do not require human-crafted feature extraction or extensive pre-processing of data[23].

Building a CNN model typically involves three separate datasets; a training set, a validation set and a test set[24]. The training set is used to develop the model so that it predicts a label (e.g., adenomatous or hyperplastic polyp for polyp characterisation) based on features extracted from the endoscopic image by the algorithm itself. The validation set is used to avoid over-fitting into the training dataset through fine tuning of the hyperparameters of the model. Finally, the testing set is used as an independent dataset to evaluate the generalisability of the CNN. With smaller datasets, cross-validation can be used to assess the model’s robustness. In cross-validation, the data is split into equal parts (e.g., 4 parts), with one part held out as a validation dataset. This process is repeated multiple times, with the results of each split eventually pooled together to decide how robust the model is[24]. CNNs evaluated using cross-validation should still be assessed against an independent test set to examine their generalisability[24].

CONVOLUTIONAL NEURAL NETWORKS AND OPTICAL DIAGNOSIS

It is only in the last few years that the use of CNNs in optical diagnosis of colorectal polyps has been extensively investigated, with various studies emerging (Table 1). Many of these studies have in fact demonstrated the capability of CNNs to surpass the PIVI 2 threshold in order to support a “leave in” strategy for rectosigmoid hyperplastic polyps (Table 2). This was first demonstrated by Chen et al[25], who used a single centre, retrospective, still image dataset of 2157 polyps to train a CNN and reported a sensitivity for identifying adenomas of 96.3% , specificity 78.1%, and NPV of 91.5% when evaluating a test set of 284 colonic and rectal diminutive adenomatous and hyperplastic polyps. Using colonic diminutive polyps is a common strategy to assess against PIVI 2 due to difficulties in obtaining large datasets of diminutive rectosigmoid polyps. An important limitation of this study is that it used magnified narrow-band imaging (NBI) data. This recently developed modality is not yet readily available in most endoscopy departments, although it will become more widely used with time.

Table 1 Summary of the studies on convolutional neural network algorithms for the optical diagnosis of colorectal polyps.

Ref.	Study design (training/testing)	Multi-centre study	Dataset	Image quality	Classification system	Lesion number (training/testing)	SSL excluded	Endoscopic processor	Image modality (training)	Real-time capability
Komeda et al[37]	Retrospective	Single	Video	Not specified	Adenoma/non-adenoma	Not specified/10	No	Not specified	WLI, NBI, chromoendoscopy	Not specified
Chen et al[25]	Retrospective/prospective	Single	Still	HQ	Hyperplastic/neoplastic	2157/284	Yes	Olympus 260 + 290	Magnified NBI	Real-time (approximately 450 ms)
Byrne et al[23]	Retrospective/prospective	Single	Video	All images	NICE Type 1/NICE Type 2	220/125	Yes	Olympus 190	NBI-NF	Real-time ( approximately 50 ms)
Zachariah et al[26]	Prospective	Two	Still	Adequate and HQ	Adenomatous/serrated polyp	5278/634	No	Olympus 190 (90%), 180 (7%), Pentax i10(3%)	WLI, NBI, i-SCAN	Real-time ( approximately 13 ms)
Ozawa et al[38]	Retrospective/prospective	Single	Still	HQ	Hyperplastic/adenomatous/SSL/CRC/other	WLI: 17566/783 NBI: 2865/290	No	Olympus 260 + 290	WLI, NBI	Real-time (approximately 20 ms)
Jin et al[31]	Retrospective/prospective	Single	Still	HQ	Hyperplastic/adenomatous	2150/300	Yes	Olympus 290	NBI-NF	Real-time (approximately 10 ms)
Song et al[39]	Retrospective/prospective	Single	Still	HQ	Serrated polyp/benign adenoma/MSM/DSMC	624/545	No	Olympus 290	NBI-NF	Real-time ( approximately 20-40 ms)
Rodriguez-Diaz et al[28]	Retrospective/prospective	Two	Still	Not specified	Neoplastic (adenomas, CRC)/non-neoplastic (hyperplastic, normal)	607/280	Training: Yes Testing: No	Olympus 190	NBI-NF, NBI (digital magnification)	Real-time (approximately 100 ms)
van der Zander et al[27]	Retrospective/prospective	Not specified	Still	HQ	Benign (hyperplastic)/pre-malignant (adenomatous, SSL, T1 CRC)	398/60	No	Fujifilm, Pentax	WLI, BLI, i-SCAN	Real-time (approximately 14.8 ms)

SSL: Sessile serrated lesion; WLI: White light imaging; BLI: Blue light imaging; NBI: Narrow band imaging; NBI-NF: Narrow band imaging–near focus; NICE: NBI International Colorectal Endoscopic; HQ: High-quality; CRC: Colorectal cancer; MSMC: Mucosal or superficial submucosal cancer; DSMC: Deep submucosal cancer.

Table 2 Summary of the per-polyp results of studies on convolutional neural network algorithms for the optical diagnosis of colorectal polyps (cross-validation results not included).

Ref.	Image Modality (testing)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	Accuracy for neoplasia (%)	PIVI 1 achieved (%)	PIVI 2 achieved (%)
Komeda et al[37]	Not specified	-	-	-	-	70	-	-
Chen et al[25]	Magnified NBI	96.3	78.1	89.6	91.5	90.1	-	Yes (91.5)
Byrne et al[23]	NBI-NF	98	83	90	97	94	-	Yes (97)
Zachariah et al[26]	NBI	-	-	-	96.5	93.1	Yes (98.3)	Yes (96.5)
	WLI	-	-	-	88.9	92.8	Yes (90.8)	No (88.9)
Ozawa et al[38]1	NBI	97	-	84	88	-	-	-
	WLI	98	-	85	88	-	-	-
Jin et al	NBI-NF	83.3	91.7	93.3	78.6	86.7	-	-
Song et al[39]	NBI-NF (test set 1)	84.1	74	88.3	67.7	-	-	-
	NBI-NF (test set 2)	88.5	72.1	88.6	84.7	-	-	-
Rodriguez-Diaz et al[28]	NBI-NF (90%) + NBI (10%)	95	88	-	93	-	Yes (94 (20/90 LC))	Yes (98 (6/68 LC))
van der Zander et al[27]	WLI + BLI	95.6	93.3	97.7	87.5	95.0	-	No (87.5)

¹Per frame analysis reported only.

WLI: White light imaging; BLI: Blue light imaging; NBI: Narrow band imaging; NBI-NF: Narrow band imaging–near focus; PIVI: Preservation and Incorporation of Valuable endoscopic Innovations; PPV: Positive predictor value; NPV: Negative predictor value; LC: Low-confidence.

Byrne et al[23] further advanced the field by training a CNN with NBI-near focus (NBI-NF) which is more commonly used in Europe and North America. It was trained with 220 polyp positive videos and when tested against 125 diminutive polyps which were collected prospectively, the model diagnosed 106 polyps with high confidence, achieving a sensitivity for identifying NBI International Colorectal Endoscopic (NICE) type 1 polyps of 98%, specificity 83% and NPV of 97%. A novelty worth highlighting in this study was the use of images derived from videos, an approach that reduces selection bias compared to retrospective still images as endoscopists usually capture high quality polyp views that are free from motion blur and surface artifact. An additional advantage of this CNN is that it simplified the clinical workflow as it automatically diagnoses polyps without requiring a still image of the polyp to be captured. Limitations of the study are that SSLs, normal tissue and lymphoid aggregates were excluded from the final analysis and the videos used to train and test the CNN were captured from colonoscopies performed by a single expert endoscopist and hence, potentially less generalisable to novice users.

The most commonly used imaging modalities amongst community endoscopists are white light imaging (WLI) and NBI without magnification. Using a large retrospective still image training set of 5278 polyps and tested against 634 polyps, Zachariah et al[26]’s CNN fell short of PIVI 2 in WLI (NPV of 88.9% and accuracy 92.8%) but achieved the threshold in NBI without magnification (NPV of 90.8% and accuracy 93.1%). This study advanced the field as it demonstrated the capabilities of CNNs to optically diagnose polyps in standard NBI modality and also to differentiate adenomas from serrated polyps through the inclusion of SSLs in its dataset.

Whilst the majority of CNNs have been trained and tested using Olympus data, studies are emerging using data from other manufacturers. van der Zander et al[27] recently developed a CNN using Fujifilm data in high definition white light (HDWL) and blue light imaging (BLI). The CNN was more efficacious when it used a unique multimodal imaging approach where it combined both HDWL and BLI images of the same polyp in its decision process compared to a single imaging modality. When evaluated against 60 prospectively collected diminutive polyps, it did not reach the PIVI 2 threshold with a NPV of 87.5% but did achieve an optical diagnosis accuracy of 95% (sensitivity for identifying pre-malignant polyps 95.6% and specificity 93.3%) and demonstrated superiority to both expert and novice endoscopists in human benchmark testing.

In comparison to PIVI 2, there are fewer studies evaluating the performance of CNNs against PIVI 1. The CNN presented in Zachariah et al[26] reached PIVI 1 thresholds in both WLI and NBI with normal magnification, achieving concordance with histology-based colonoscopy surveillance intervals in 90.9% and 98.3% of patients, for each respective modality. Rodrigues-Diaz et al[28] used a single centre retrospective still image dataset to train a CNN with 607 polyps and tested against 90 diminutive polyps where it achieved a high confidence diagnosis in 78% of cases, with a 94% agreement with histology-based colonoscopy surveillance intervals. Tested against 68 rectosigmoid polyps, the model diagnosed 88% of polyps with high confidence, achieving PIVI 2 thresholds with a NPV of 97%.

There is also potential to expand the use of optical diagnosis CNNs outside of the ”resect and discard” and “leave in strategy”. A dilemma that can complicate issuing post-polypectomy surveillance intervals is discrepancies between endoscopic and histological diagnosis and classification of polyps with tissue fragmentation in the specimen retrieval process playing an important role. Shahidi et al[29]’s proof of concept study used a CNN to resolve discrepancies in polyps ≤ 3 mm in size. Tested against 900 polyps that were ≤ 3 mm and optically diagnosed as adenomatous by an expert endoscopist, the CNN diagnosed the adenomas with high confidence in 644 polyps, with 256 polyps deemed to be of sub-optimal imaging quality. However, of these high confidence diagnoses, the pathologists diagnosed 15.4% as normal mucosa, 13.2% as hyperplastic polyp and 0.3% as SSL. In this context, a CNN could help to mitigate against the risk of under-surveillance.

Whilst CNN’s diagnostic accuracy excels in many studies, without real-time capabilities, they would have no clinical utility. Prior to the era of deep learning, computer aided diagnosis algorithms lacked real-time capability, but most CNNs do not share this problem and often process data at a rate that exceeds the 25 frames per second that is generated in a video recording of a colonoscopy procedure. Given the excellent performance in ex-vivo studies and the real-time capabilities displayed by CNNs, the future appears promising for their integration in colonoscopy.

TRANSPARENCY OF CONVOLUTIONAL NEURAL NETWORKS

The complexity of CNN models’ decision process is often referred to as a “black box” and represents an important barrier to its acceptance by both clinicians and patients[30]. Opening the ‘’black box’’ to display the raw features which informed the CNN’s decision is important for transparency especially from a safety standpoint[28]. Transparency can help identify biases within the neural network and aid root-cause analyses in cases of patient harm, for example, if a neoplastic polyp that subsequently develops into a CRC is originally misdiagnosed as non-neoplastic by the CNN model.

For polyp characterisation, important steps have been taken to open the black box. Jin et al[31] developed a CNN that generated a coloured heat map, overlaid to the polyp, to help the endoscopist comprehend the specific aspects of the image that contributed to the CNN’s prediction (Figure 2). This could help the endoscopist to decide which information is relevant and which decisions are truly based on appropriate image analysis. If, for example, the heatmap is overlaid to normal mucosa, then the endoscopist would quickly be able to appreciate this and disregard the CNN’s diagnosis.

Figure 2 Illustration of coloured heatmaps, overlaid to the polyp, which demonstrates the regions that most likely contributed to the convolutional neural networks’s diagnosis. A, B, C, D: Original narrow band imaging (NBI) of polyps; a, b, c, d: Coloured heatmap overlaid on the NBI image; Red: Higher probability that this region informed the convolutional neural networks (CNN)’s diagnosis; Blue: Lower probability that this region informed the CNN’s diagnosis. Images adapted and modified with permission from the publisher[31]. Citation: Jin EH, Lee D, Bae JH, Kang HY, Kwak MS, Seo JY, Yang JI, Yang SY, Lim SH, Yim JY, Lim JH, Chung GE, Chung SJ, Choi JM, Han YM, Kang SJ, Lee J, Chan Kim H, Kim JS. Improved Accuracy in Optical Diagnosis of Colorectal Polyps Using Convolutional Neural Networks. Gastroenterology 2020; 158(8): 2169-2179. Copyright© The Authors 2020. Published by Elsevier.

More recently, in order to further enhance CNN transparency, Rodriguez-Diaz et al[28] developed a colour coded segmentation model (Figure 3). In this model, the CNN divides the polyp into distinct segments to allow the endoscopist to identify the specific regions within the image that is informing the CNN’s decision. The CNN predicts the histology of each subregion of the segmented polyp, with high confidence neoplastic diagnoses coloured in red, high confidence non-neoplastic in green, and low confidence/indeterminate diagnoses in yellow, with the final predication resulting from an aggregate of all the analysed regions. The end result is a detailed spatial colour coded histology map of the polyp surface, which the endoscopist can visualise and incorporate into their decision process[28], enhancing the interpretability of this CNN model in comparison to others. However, an important limitation to this advanced CNN is that it currently lacks the ability to operate at a video rate.

Figure 3 Spatial colour coded histology map which allows the user to visualise the sub-regions of the polyp surface that contributed to the convolutional neural networks’s decision process. A: Hyperplastic polyps; B: Adenomatous polyps; C: Sessile serrated lesions; Red: High-confidence neoplastic diagnosis; Green: High-confidence non-neoplastic diagnosis; Yellow: Indeterminate or low-confidence diagnosis. Adapted from Ref. [28]. Citation: Rodriguez-Diaz E, Baffy G, Lo WK, Mashimo H, Vidyarthi G, Mohapatra SS, Singh SK. Real-time artificial intelligence-based histologic classification of colorectal polyps with augmented visualization. Gastrointest Endosc 2021; 93: 662-670. Copyright© The Authors 2021. Published by Elsevier.

Further research in the interpretability of CNN models is required to improve its acceptance[32] and accelerate its translation to clinical practise.

LIMITATIONS AND FUTURE DIRECTIONS

Despite the promise shown by CNNs this far, it is crucial to recognise that there are various limitations that need to be overcome before they can become part of the endoscopic clinical workflow. The most significant limitations are the reliance on retrospective datasets[33], which are inherently subject to selection bias, and the lack of prospective studies and randomised controlled trials[34]. Most studies train and test CNNs using high quality images of polyps, free from “noise” such as motion blur and polyp surface artifact (e.g., mucus, stool or blood). The extent to which CNNs pre-clinical results are reproducible in the real-world setting, where ‘noise’ is frequently encountered, remains to be seen.

To the best of our knowledge, there have been no prospective randomised controlled clinical trials evaluating optical diagnosis CNN in-vivo. This is partly due to clinical trials being time consuming and expensive, and an alternative pragmatic approach could be the use of a benchmark test in the form a publicly available external dataset to compare different CNN models[35]. No such datasets currently exist for polyp characterisation and therefore the generalisability of CNN models remains poorly understood. Generalisability refers to the CNN performance with different endoscope models and clinical settings from the site that the data was generated to train the CNN. To date, only one study[36] has evaluated generalisability, and this was limited to a small testing set of 69 polyp images from two population cohorts (Australian and Japanese) using two separate endoscope manufactures (Olympus and Fujifilm). Despite the small test-set, this study highlighted the concerns of generalisability as the operator area under the curve fell from 94.3% for the internal set, to 84.5% and 90.3% for the external testing sets (NBI and BLI respectively).

Another important limitation is that studies often exclude polyps that are not adenomas or hyperplastic polyps, restricting the possible classification outputs of CNNs. This, in turn, limits their clinical utility as polyps such as SSL and inflammatory polyps would be misclassified due to limitations in the initial training phase of the CNNs when the categorisation system is established.

Research in this field is likely to continue to expand and future directions to consider include: (1) Guidelines to identify the role of CNNs in the clinical workflow, specifically, whether it is a second reader, a concurrent reader or a provider of an independent diagnosis[30]; (2) Prospective multi-centre randomised clinical trials; (3) Publicly available external datasets for benchmark testing and evaluation of the generalisability of CNN models in different clinical settings and population cohorts; and (4) Acquiring datasets inclusive of all polyp sub-types to advance CNN classification systems.

CONCLUSION

In summary, this is an exciting time for the endoscopy community. CNNs diagnostic performance has excelled in ex-vivo studies and in human benchmarking testing. CNNs are likely to be a key adjunct in optically diagnosing polyps and have renewed optimism that implementation of a “resect and discard” and “leave in” strategy is feasible due to the potential to alleviate the inter-operator variability amongst endoscopists. This would bring significant financial benefits to healthcare systems, avoid unnecessary polypectomies of non-neoplastic polyps and improve the efficiency of colonoscopy. However, prospective multi-centre randomised controlled trials and publicly available datasets for benchmark testing are required to further evaluate the efficacy and generalisability of CNNs. Furthermore, with these models now emerging in endoscopy units, it’s imperative that guidelines are developed to establish their role in the clinical workflow.