Artificial intelligence technologies for the detection of colorectal lesions: The future is now

Abstract

Several studies have shown a significant adenoma miss rate up to 35% during screening colonoscopy, especially in patients with diminutive adenomas. The use of artificial intelligence (AI) in colonoscopy has been gaining popularity by helping endoscopists in polyp detection, with the aim to increase their adenoma detection rate (ADR) and polyp detection rate (PDR) in order to reduce the incidence of interval cancers. The efficacy of deep convolutional neural network (DCNN)-based AI system for polyp detection has been trained and tested in ex vivo settings such as colonoscopy still images or videos. Recent trials have evaluated the real-time efficacy of DCNN-based systems showing promising results in term of improved ADR and PDR. In this review we reported data from the preliminary ex vivo experiences and summarized the results of the initial randomized controlled trials.

Key Words: Endoscopy; Colonoscopy; Screening; Surveillance; Technology; Quality; Artificial intelligence

Core Tip: The use of artificial intelligence (AI) in colonoscopy has been gaining popularity in current times. At first, the efficacy of deep convolutional neural network (DCNN)-based AI system for polyp detection has been tested in ex vivo settings such as still images or videos from colonoscopies. Recent trials have evaluated the real-time efficacy of DCNN-based systems in improving adenoma detection rate and polyp detection rate. In this review we reported all the preliminary ex vivo experiences and summarized the promising results of the initial randomized controlled trials.

INTRODUCTION

Colorectal cancer (CRC) remains one of the leading causes of mortality among neoplastic diseases in the world[1]. Adequate colonoscopy based CRC screening programs have proved to be the key to reduce the risk of mortality, by early diagnosis of existing CRC and detection of pre-cancerous lesions[2-4]. Nevertheless, long-term effectiveness of colonoscopy is influenced by a range of variables that make it far from a perfect tool[5]. The effectiveness of a colonoscopy mainly depends on its quality, which in turn is dependent on the skill and expertise of the endoscopist. In fact, several studies have shown a significant adenoma miss rate of 24%-35%, especially in patients with diminutive adenomas[6,7]. These data are in line with interval cancers incidence (I-CRC), defined as the percentage of cancers diagnosed after a screening program and before the intended surveillance duration, of approximately 3%-5%[8,9].

Adenoma detection rate (ADR), defined as the proportion of patients in which at least one adenoma is detected (> 30% in men and 20% in women), along with adequate bowel preparation rate (> 85% of all colonoscopies), cecal intubation rate (> 95% in screening colonoscopies) and withdrawal time > 6 min, have been identified as quality metrics in screening and diagnostic colonoscopies, to reduce the I-CRC incidence[5,10,11]. Increase in ADR by 1% has shown to decrease the risk of incidence of CRC by 3%[9].

Innovations such as virtual and dye-spray chromoendoscopy and add on devices may help in improving ADR, particularly in low detectors[12-14]. However, all these strategies are operator-dependent tools requiring a learning curve. Further, individual experience and preference, may influence their use and efficacy.

The development of the artificial intelligence (AI) applications in the medical field has grown in interest in the past decade. Its performance on increasing automatic polyp and adenoma detection has shown promising results in order to achieve an higher ADR[15]. The use of computer aided diagnosis (CAD) for detection and further characterization of polyps had initially been studied in ex vivo studies but in the last few years, with the advancement in computer aided technology and emergence of deep learning algorithms, use of AI during colonoscopy has been achieved and more studies have been undertaken[16].

The aim of this review is to provide an overview on the progress of AI, with deep learning technologies, with experiences from initial ex vivo studies to real time adenoma detection from most recent studies.

AI

AI is the result of the evolution of general software systems that provide an input and obtain an output through an algorithm. Machine Learning is the ability of a program, to learn, after an adequate training, from data that were initially entered, in order to obtain a model that can cope with scenarios that had not specifically been instructed for.

In gastroenterology, AI could be applied to tasks and clinical concerns faced by endoscopists every day. For instance, the human eye is capable of capturing only a fixed number of frames or images per second. AI can help the endoscopist to highlight a specific region of interest which needs a closer examination for identification of polyps, or can assist with categorizing polyps as hyperplastic versus adenomatous polyp, thus eventually improving the ADR[17].

In the endoscopic field, this innovative technology uses two principal Machine learning methods.

Handcrafted knowledge

In these systems, engineers create a set of rules that describe knowledge in a well-defined field. It is, historically, the first approach to artificial intelligence that, in the 1980s, led to the development of the first Expert Systems based on an approach "If … Then". The systems that fall into this category "reason" on a very specific problem, have no ability to learn and a poor ability to reason in conditions of uncertainty, when they do not have all the elements to make the decision. Indeed object characteristics are extracted and selected manually and are used to create a model capable of categorizing them through algorithms. With regards to the evaluation of polyps, it will record a series of fixed parameters such as shape, size and texture, alone or in combination, from polyp image datasets in order to differentiate a polyp from the normal mucosa.

Deep learning

In deep learning, large artificial neural networks receive algorithms and increasing amounts of data, constantly enhancing the ability to “think” and “learn”. The adjective “deep” refers to the many levels that the neural network accumulates over time, improving performance proportionally to the depth of the network. Although most of the current deep learning is performed with human supervision, the goal is the creation of neural networks that can self-train and “learn” autonomously. Similar to our biological brain which tries to formulate an answer to a question by deducing a logical hypothesis and arrive at a solution for a problem, deep learning sets neural connections in motion (exactly as the human mind does), improving its performance through continuous learning using the convolutional neural network (CNN), mathematical-computer calculation models based on the functioning of biological neural networks.

An artificial neural network receives external signals on a layer of input nodes (processing units), each of which is connected with numerous internal nodes, organized in several levels. Each node processes the received signals and transmits the result to subsequent nodes working in parallel (Figure 1). They need a system training phase that fixes the weights of individual neurons and this phase can take a long time, if the number of records and variables to be analyzed are exceptionally large. As a result, the network success significantly depends on the creator’s experience[18].

Figure 1 Convolutional neural network design.

The following would be a solid example of how a deep neural network works with the visual recognition of the patterns. In polyp or adenoma detection, the neurons of the first layer could learn to recognize the edges, the neurons in the second layer could learn to recognize elementary shapes, for example the round shape created by the edges. The third layer would recognize even more complex forms as a 3D structure, the fourth would recognize further details as a granular pattern and so on.

EX VIVO STUDIES

Several ex vivo studies on AI have been published in the past 20 years (Table 1).

Table 1 Ex vivo studies.

Ref.	Country	Algorithm	Number of images/videos	Outcomes
Karkanis et al[19], 2003	Greece	Hand-crafted	60 videos	Sensitivity 94%, specificity 99%
Maroulis et al[34], 2003	Greece	Hand-crafted	2809 video frame	Accuracy > 95%
Jerebko et al[35], 2006	United States	Hand-crafted	56 images	Sensitivity 84%
Hwang et al[21], 2007	United States	Hand-crafted	8621 video frame	Per-polyp sensitivity 96%
Park et al[36], 2012	United States	Hand-crafted	35 videos, > 1 million frames	AUROC 0.89
Wang et al[37], 2014	United States	Hand-crafted	46 video file	Sensitivity 81,4%
Bernal et al[38], 2015	Spain	Hand-crafted	612 video frame	PPV 70%
Tajbakhsh et al[39], 2015	United States	Hand-crafted	19400 video frame (property), 300 video frame in CVC-ColonDB	Sensitivity on property database 48%, sensitivity in CVC-ColonDB 88%
Wang et al[20], 2015	United States	Hand-crafted	53 videos	Per-polyp senstivity 97.7%
Geetha et al[40], 2016	India	Hand-crafted	Still images 703 frames	Sensitivity 95%, specificity 97%
Fernández-Esparrach et al[22], 2016	Spain	Hand-crafted	25 videos	Sensitivity 70.4%, specificity 72.4%
Angermann et al[41], 2017	France	Hand-crafted	18 video with 10924 frames	100% per-polyp sensitivity PPV 50%
Park et al[42], 2016	United States	CNN	562 images	Sensitivity 86%, specificity 85%
Billah et al[43], 2017	Bangladesh	CNN	14000 still images	Sensitivity 99%, Specificity 99%
Yu et al[44], 2017	China	CNN	18 videos	Sensitivity 71%, PPV 88%
Zhang et al[23], 2017	China	CNN	150 random + 30 NBI images	Sensitivity 98%, PPV 99%, AUROC 1, Accuracy 86%
Urban et al[25], 2018	United States	CNN	ImageNet 1.2 mil, 53588 imges from videos	90% sensitivity
Misawa et al[24], 2018	Japan	CNN	135 video clips	Per-polyp sensitivity 94%, per-frame sensitivity 90%, specificity 63.3%, accuracy 76.5%
Pogorelov et al[45], 2018	Norway	CNN	1359 to 11954 frame from still images	Sensitivity 75%, specificity 94%
Yamada et al[46], 2018	Japan	CNN	4840 video images	Sensitivity 97%, specificity 99%, AUROC 0.975
Zhu et al[47], 2018	China	CNN	616 still images	Sensitivity 89%, 92% classification accuracy
Hassan et al[26], 2019	Italy	CNN	338 videos, 1.5 milion frames	Sensitivity per lesion 99.7%
Ahmad et al[48], 2019	England	CNN	24596 video frames	Sensitivity 85%, specificity 93%
Eelbode et al[49], 2019	Belgium	CNN	758 frames of still imges	Sensitiviy 92%, specificity 85%
Ka-Luen Lui et al[50], 2019	China	CNN	6 unedited videos	Per-polyp sensitivity 100%, per frame sensitivity 98.3%, specificity 99.7%, AUROC 0.99
Misawa et al[51], 2019	Japan	CNN	64 videos	86% sensitivity
Shichijo et al[52], 2019	Japan	CNN	1233 still images	Per-polyp sensitivity 100%, per-image semsitivity 99%, 76% PPV
Ozawa et al[53], 2020	Japan	CNN	7077 images	92% sensitivity, 86% PPV, accuracy 83%

USA: United States of America; CNN: Convolutional neural network; PPV: Positive predictive value; AUROC: Area under the receiver operating characteristic.

In the early 2000s, Karkanis et al[19] developed the first algorithm based on color analysis. They chose 180 polyp sample images and then randomly analyzed 1200 polyp frames from a 5-10 s extract of 60 videos. Algorithm results were promising with a 93.6% sensitivity and 99.3% specificity for polyp, although with the limitation of long processing time and the still images[19]. To address these issues, Wang et al[20] introduced a new algorithm called “Polyp-Alert”, that by using edge detection, succeeded in a near real-time video analysis at 10 frames per second from a sample of 43 polyp shots extrapolated randomly from 53 videos. They reported a 97.7% per-polyp sensitivity making this algorithm one of the first to be able to compete with real time speed. Many others methods have been proposed such as the one focusing on elliptical shape features from Hwang et al[21] or the window median depth of valleys accumulation energy maps system by Fernàndez-Esparrach et al[22], which have been able to detect a specific area of the image containing a polyp as it perceives polyps as mucosal protrusions with precise boundaries. As already pointed out, a steppingstone in the progress of AI was the advent of CNN and deep learning for CAD of polyps. The real innovation is tied to the fact that CAD systems can recognize polypoid and non-polypoid features without a continuous external input, after an initial training. In 2017, Zhang et al[23] devised an algorithm able to outperform expert endoscopists in polyp detection accuracy (86% vs 74%). They also reported a 98% sensitivity using a dataset which contained millions of naturalistic images in addition to images of polyps. Misawa et al[24] had trained the CNN with a dataset of 1.8 million of frames of polyps from 73 colonoscopy videos, with a total of 155 polyps. Each frame was retrospectively evaluated by two expert endoscopists before being included in the dataset. The sensitivity achieved was 90% with a specificity of 63.3% for the frame-based analysis[24]. Another interesting work was published in 2018 by Urban et al[25]. The group pre-trained the algorithm with a dataset of 8641 colonoscopy images from about 2000 patients achieving a 96.4% of accuracy. Moreover the authors confirmed the CNN value, empowering the polyps detection potential of any senior endoscopist. Further, they also showed the possibility to reduce the miss rate, including 11 videos with 73 polyps deliberately missed by the endoscopist because of a fast withdrawal. The CNN identified 67 of the 73 polyps with a false positive rate of 5%. Another comparison between the human brain and the AI was made by Hassan and colleagues with a new system, the GI-Genius (Medtronic)[26]. It uses a dataset of 2684 histologically confirmed neoplastic polyps manually annotated by expert endoscopists and listed in 1.5 million frames. They quantified the reaction time in polyp detection for AI and five expert endoscopists were asked to observe 338 video clips and to press a button, once a polyp was identified. The overall sensitivity per lesion was 99.7%. The AI anticipated the detection against endoscopists in 82% of cases. They concluded that this result is probably due to the variability of endoscopist expertise, a greater detection of hyperplastic polyps in the AI group and benefits of AI outweighing the limitations of human beings such as fatigue and distraction.

Data from main ex vivo experiences on AI systems are summarized in Table 1.

IN VIVO STUDIES

There is currently a lack of robust data on AI application in real time colonoscopy and its utility compared to ex vivo studies, but recently there have been more studies published. Registration data on different CAD systems are summarized in Table 2.

Table 2 Artificial intelligence system country approval.

Artificial intelligence system	Country
GI-Genius (Medtronic)	European Union, Australia, Israel, South Arabia
CAD-Eye (Fuji)	European Union
Discovery (Pentax)	European Union
Endobrain-EYE (Olympus)	Japan
Wision-AI	China

Klare et al[27] published a prospective study on CAD (KoloPal) to aid with polyp detection in high definition (HD) white light endoscopy in 55 patients. Fifty-five HD-white light colonoscopies were carried out by experienced endoscopists that through a verbal signal communicated the presence of any polyp. In parallel, another independent operator observed the examination on two other screens projecting images with and without the automated polyp detection software (APDS). The system highlighted with a green ring, particular areas of interest chosen by a combination of color, structure, textures, and motion, with a delay of 50-ms. Comparing APDS and endoscopists, polyp detection rate (PDR) (50.9% vs 56.4%) and ADR (29.1% vs 30.9%) were comparable. In particular a good performance was observed for larger (≥ 10 mm) and Paris morphology 0-Ip and 0-Is polyps. Smaller and flat polyp morphology had insufficient polyp detection rates by APDS. However, no polyp was detected by the APDS before detected by endoscopists, probably because of the software delay[27].

In the last year, there have been randomized controlled trials (RCT) published on the use of AI in colonoscopy (Table 3). In 2019 the first prospective RCT on real time automatic detection system using deep neural networks was conducted in China. Wang et al[15] enrolled 1058 patients undergoing routine colonoscopy. They were randomly assigned to either the CAD assisted colonoscopy group or the control group. They reported a significantly higher ADR in CAD group (29.1% vs 20.3%), primarily due to increase in detection of diminutive adenomas without a significant difference in large adenomas. The detection of hyperplastic polyps was also significantly higher in CAD group (43.6% vs 34.9%), which could potentially minimize the resection of these polyps and adverse events. There was also a higher mean number of adenomas detected (0.53 vs 0.31) in the CAD group[15].

Table 3 In vivo randomized control trials characteristics.

Ref.	Country	CAD system	CAD system aim	Number of patients		ADR (%)	ADR (%)
				WL	CAD	WL	CAD
Wang et al[15], 2019	China	EndoScreener	Detection	536	522	20.3	28.9
Wang et al[54], 2020	China	EndoScreener	Detection	478	484	28	34.1
Gong et al[30], 2020	China	ENDOANGEL	Quality	318	324	8	16
Repici et al[31], 2020	Italy	GI-Genius	Detection	344	341	40.4	54.8
Liu et al[28], 2020	China	Henan Xuanweitang Medical Information technology Co. Ltd.	Detection	518	508	23.9	39.2
Su et al[29], 2020	China	-	Detection; quality	315	308	16.5	28.9

WL: White light (control group); CAD: Computer aided diagnosis; ADR: Adenoma detection rate.

Liu et al[28] published their experience from China and reported that the average number of polyps detected in the CAD group were higher (0.87 vs 0.57, P < 0.001). CAD group also achieved an ADR of 39% compared to 23% in the control group. The detection power was particularly improved for sessile serrated lesions and diminutive adenomas[28]. Su et al[29] developed an automatic quality control system (AQCS) using deep CNN and randomized 659 patients between AI and control groups. They reported significantly higher ADR, PDR, mean number of adenomas and mean number of polyps in the AQCS group[29].

The most recent eastern RCT was by Gong et al[30] using another CAD technology called ENDOANGEL. Apart from its help in polyp detection, ENDOANGEL technology was also trained for recording, and possibly improving, colonoscopy quality indicators such as cecal intubation or withdrawal time by real-time signaling during the procedure[30]. They reported a significant improvement in ADR but one of the principal limitations of this study is the lack of external validity due to very low ADR rates in both groups (17% vs 8%, CAD vs control group). In countries like China where the incidence of CRC is lower[1], it may be acceptable, but it’s performance in Western countries where rates are much higher these such low ADR rates would indicate a low quality colonoscopy[5] .

As a matter of fact Repici et al[31] recently investigated the role of the GI-Genius CADe system (Figure 2) among expert endoscopists in a western-setting. The ADR of 40.4% of the control group was further increased up to 54.8% using the CADe system. Diminutive adenomas (≤ 5 mm) were detected in a significantly higher proportion of subjects in the CADe group (33.7%) than in the control group (26.5%), as were adenomas of 6–9 mm (10.6% vs 5.8%), regardless of morphology or location. Based on colonoscopy videos of such a trial, the authors developed a “false positive” classification aiming to standardize future reports on this topic for a better insight on AI systems[32].

Figure 2 GI-Genius computer aided polyp detection system in high definition white light, and virtual chromoendoscopy with blue light imaging and linked color imaging. A: High definition white light; B: Virtual chromoendoscopy with blue light imaging; C: Virtual chromoendoscopy with linked color imaging.

FUTURE NEEDS

The utility of AI has come a long way with computed assisted technology making significant strides in colonoscopy and improving the outcomes for quality metrics. Initial studies published were retrospective in nature, based primarily on feeding still images to the software. This could potentially introduce selection bias in polyp selection and hence influence the outcomes. Following these, there have been several prospective studies, designed for both polyp detection and characterization which reduce the possibility of bias and help with better interpretation of the functioning and accuracy of the CAD system. In the last year, we have had few RCTs published on this subject, mainly from China. They have shown improvement in quality metrics like ADR, withdrawal times and other outcomes like PDR, mean adenomas and polyps per person[33]. We believe that the future studies should be directed towards these goals: it is important to investigate in future studies, the utility of AI in the diagnosis and characterization of flat lesions and sessile serrated lesions which are known to have significant malignant potential. Only one of the RCTs has been performed in a multi-center setting and we need more RCTs performed in various centers across the world for generalizability and reproducibility of the results. Also several different CAD systems have been investigated including laser-induced fluorescent spectroscopy, endocytoscopy, narrow band imaging and chromoendoscopy but an ideal AI system combined with high definition white light endoscopy, aiding with polyp detection and characterization, is yet to be found and this is a work in progress currently. Moreover, the impact of false positive (futile activations) on endoscopists behave was only reported by a single experience and will deserve further insights. The impact of the ability of CAD in assisting with accurate prediction of polyp surveillance intervals also need to be investigated with longitudinal studies including follow-up data.

CONCLUSION

Performance of a high-quality colonoscopy is essential in preventing the incidence of colorectal cancer. Significant progress has been made in the field of AI assisted colonoscopy, especially with the advent of deep CNN, which helps in overcoming the limitations of a traditional colonoscopy related to technical variations by operators and human errors. Early evidences on AI application in colonoscopy have shown it to be an effective tool in increasing efficacy in adenoma detection. RCT’s investigating these quality metrics have been published recently and more are in progress. However, the role of the endoscopist and in particular his abilities and experience cannot be overshadowed: (1) The detection ability of AI systems is dependent on the inspection of the mucosa exposed by the endoscopist during the scope withdrawal, and an adequate technique and the quality of bowel preparation are essential for its effective operating; and (2) The improving in detection seems to involve even hyperplastic polyps with low malignant potential and the endoscopist should be able to make a decision on which need to be resected and which do not. However, the ability of optical diagnosis is still suboptimal compared to histopathological evaluation, obtaining valid results only in specific settings[55]. Also in this field, CNN systems are being developed in order to further assist the endoscopist[56]. These would be key factors in deciding the efficacy and success of AI assistance and also play an important role with cost-benefit related outcomes in the future. Longitudinal follow-up and performance of AI in different study populations is essential in future studies to study its impact and generalizability of its use in clinical practice.