Biophysics inspired artificial intelligence for colorectal cancer characterization

Abstract

Over the last ten years artificial intelligence (AI) methods have begun to pervade even the most common everyday tasks such as email filtering and mobile banking. While the necessary quality and safety standards may have understandably slowed the introduction of AI to healthcare when compared with other industries, we are now beginning to see AI methods becoming more available to the clinician in select settings. In this paper we discuss current AI methods as they pertain to gastrointestinal procedures including both gastroenterology and gastrointestinal surgery. The current state of the art for polyp detection in gastroenterology is explored with a particular focus on deep leaning, its strengths, as well as some of the factors that may limit its application to the field of surgery. The use of biophysics (utilizing physics to study and explain biological phenomena) in combination with more traditional machine learning is also discussed and proposed as an alternative approach that may solve some of the challenges associated with deep learning. Past and present uses of biophysics inspired AI methods, such as the use of fluorescence guided surgery to aid in the characterization of colorectal lesions, are used to illustrate the role biophysics-inspired AI can play in the exciting future of the gastrointestinal proceduralist.

Key Words: Gastroenterology; Artificial intelligence; Gastrointestinal surgery; Deep learning; Biophysics; Machine learning

Core Tip: In this piece we provide an overview of current state of the art in gastroenterology and gastrointestinal surgery. We discuss current deep learning artificial intelligence methods for colorectal lesion detection and characterization as well as exploring biophysics inspired artificial intelligence methods and the potential role they can play in the future of gastroenterological practice.

INTRODUCTION

One of the most fulfilling yet challenging aspects of medical practice revolves around the art of correct decision-making. Current training models within medicine address this through experiential learning and graded autonomy over time along with sub specialization in order for an individual to reach competency and, ideally, mastery within their chosen field. Despite now widespread use of decision support systems in areas such a manufacturing and business, automated decision support for the modern clinician in clinical practice remains in its infancy. The last ten years have seen all medical specialties introduce artificial intelligence (AI) methods as a topic for research and increasingly it is beginning to impact clinical practice as supportive data accrues. Success rates have however varied with areas such as radiology (chest X-ray and mammogram interpretation) and ophthalmology (retinal disease progression) emerging as early beneficiaries[1-3]. Increasingly interest is developing regarding the application of these principles to gastrointestinal disease and its interventions.

GASTROINTESTINAL INTERVENTION

The practice of gastroenterological endoscopy has also seen promising developments regarding in situ determination of colonic lesions through AI methods culminating in the recent launch of commercially approved, AI software (GI-Genius, Medtronic, MN, United States) to aid in the detection of colorectal polyps at colonoscopy[4]. Commencing in 2003, initial endeavour in this domain involved early computer-aided detection software performing post hoc analysis on static images (“The Colorectal Lesion Detector System” by Maroulis et al[5] and Karkanis et al[6]). The state-of-the art thereafter quickly progressed to post-hoc video, and subsequently real-time video, analysis. Recently published trials have shown significantly improved polyp detection rates with these technologies along with indicators of a potential ability to characterise lesions (hyperplastic vs adenoma) in some cases[7,8]. The aforementioned studies all employ deep learning (DL), a subset of machine learning within AI that emerged in the mid-2010s, as their modus operandi[9]. DL capitalizes upon recent advances in computing capabilities to implement learning algorithms consisting of many networked layers of interconnected processing units known as neurons, arranged as neural networks. For colonoscopy, DL architectures best suited to image recognition such as convolutional neural networks are most applicable.

The GI Genius currently represents the “state of the art” accessible to the practicing clinician today. This “intelligent endoscopy module” acts as an adjunct to the gastroenterologist during a colonoscopy to highlight regions with visual characteristics consistent with different types of mucosal abnormalities and so stops short of being an autonomous polyp detection tool and provides no characterization. Powered by closed, selective datasets which may not be representative of general practice (e.g., in terms of bowel prep, withdrawal times, etc.) the module neither records nor reports its findings but instead presents areas of the screen for the endoscopist to interpret their significance including whether to biopsy, resect or disregard. Surface feature detection learned from these datasets has yet to prove its performance in real world practice in particular regarding accuracy (a typical colonoscopy comprises 50000 frames so even a tiny false positive frame rate could generate significant distraction), explainability (“telling you the settings used in the machine”) and interpretability (“why should I believe it?”). While pertinent to Food and Drug Administration approval, these considerations are particularly import when expanding into the concept of lesion characterization and, even more so, true decision support. Like all DL, system performance depends on the assumption that all possible future polyps are represented by previously encountered polyps upon which the system learned. Nevertheless, this system is truly groundbreaking in its existence as a commercial product and opens up the possibilities for AI integration at scale, including articulation of the value proposition of digital assistance as the norm.

Gastrointestinal surgery, an allied field, is also comprised of sequential steps each requiring numerous operator-led decisions but has unique decision-support challenges and unique barriers to AI implementation (although it too is increasingly delivered via image-driven minimally invasive approaches, whether by standard or robotic-assisted laparoscopy). DL as an AI method, while highly efficient at tasks such as image recognition, does not as readily lend itself surgical video where the landscape is more complex with less hallmarks available to exploit during structure differentiation (for example ureter and vascular identification during dissection vs lesion detection on plain film X-ray). Current AI methods within surgical practice are limited to tasks such as instrument detection or segmentation of procedures into their procedural phases with little AI assistance currently available in the more intricate components of surgery such as tissue identification or classification progressively during dissection[10,11]. In general, surgical procedures are not as easily represented by individual static images like the other specialties mentioned thus far and while video provides a deeper situational understanding for the experienced operator, it makes artificially intelligent interpretation much more challenging. Combined with this increased complexity, the datasets required to train current DL systems for surgery (i.e., many thousands of recorded surgical cases) do not currently exist in volumes comparable to endoscopic polyp images, retinal photographs, or mammograms.

BIOPHYSICS

Biophysics-inspired approaches to AI in surgery may present an alternative, or perhaps even better, a complimentary/synergistic approach to the current DL strategies in both gastrointestinal endoscopy and surgery. The term “biophysics” was first proposed by Karl Pearson in 1892 as an all-encompassing term to describe the application of physics principals to describe biological phenomena[12]. It would not be until the 1950s however, following significant advancements in physical measurement techniques, that the potential contributions of biophysics within the field of medicine could be realized. Initial endeavours sought to understand and describe biological phenomena such as haemoglobin dissociation and cell-cell interactions and structure[13,14]. The field then progressed to more complex tasks such as computerized simulation of blood flow and tissue perfusion using biological compartment models contingent on vascular parameters such as vascular density, perfusion rate and permeability[15]. More recent still, the combining of AI methods with biophysics principles has resulted in paradigm shifts in areas such as the study of protein folding and structure and promises to modify drug research processes[16,17]. It is now possible to study and predict protein-protein interactions by combining existing knowledge of protein structural biology and biophysics with machine learning in order to make predictions about the behaviour of previously undocumented proteins[18]. This technology has many potential uses including advancing understanding of inflammatory signaling processes, the search for cancer driving mutations and in new drug discovery. Currently, mechanisms of drug development include processes such as “target deconvolution” whereby the potential new agent, once identified, must be screened against all the known proteins in the human body. This laborious and resource intensive task aims to identify potential drug benefits and importantly to identify any potential off-target effects that may be undesirable. Harnessing the power of AI in conjunction with biophysics, researchers are now able to use computational modelling to simulate the physical interactions between molecules and potential target proteins[17]. Furthermore, comprehensive databanks of human proteins (the proteome) now exist with which to evaluate any new drugs. This permits in silico creation of a full pharmacological profile of any given drug molecule.

While biophysics inspired approaches such as those mentioned have numerous benefits, it is worth noting however that such methodology can only be employed in cases where an in-depth mechanistic understanding of all involved elements has been achieved. Therefore, utilization is limited to fields with a strong human understanding of relevant biological and physio-chemical components and furthermore efforts may be derailed where incorrect perceptions of what is true exist.

NEARINFRARED ENDOLAPAROSCOPY

In contemporary, gastrointestinal surgical practice, the advent of nearinfrared (NIR) endolaparoscopy (combining conventional endoscopic and/or minimally invasive laparoscopic techniques with NIR imaging) provides great scope for development of such AI algorithms. This technology utilizes an extended electromagnetic illumination wavelength (up to 800 nm) to detect exogenous agents capable of fluorescence (there is no background biological fluorescence at these wavelengths)[19]. Such agents can be profiled dynamically as well as absolutely (presence/absence) to garner information regarding the biological features of the tissue, including disease. This has already proven useful clinically in visual determination of intestinal perfusion during surgery where the NIR imagery is presented alongside the white light appearances, but interpretation remains qualitative by the surgeon. Via AI methods however, the added information provided by NIR tissue assessment over standard white light viewing can be combined with our existing understanding of tissue biology to enhance the proceduralist’s understanding of the field in front of them and to assist them in their task. To date indocyanine green (ICG) represents the most successful NIR agent upon which AI recommending systems have been based (and indeed it remains the only currently approved NIR fluorophore although others are in development) and it is likely that development of such decision support systems will in turn enable broader AI development across more standard surgical imagery[20]. However, for now, ICG in combination with NIR provides an excellent test-case to describe the application of biophysics-inspired AI for gastrointestinal interventions (both endoscopy and image-guided surgery).

NIR-ICG TISSUE PERFUSION

ICG is a fluorescent dye used extensively within the now established field of fluorescent guided surgery[21]. When given intravenously it remains within the vasculature with a half-life of 2.5-3 min and can be seen using a near infra-red camera[22]. It is currently used as a subjective decision-making adjunct in tasks such as anatomical delineation (biliary anatomy) and tissue physiology assessment (colorectal anastomosis formation and gastric conduit formation post oesophagectomy)[23-25]. It has also been used to assist in the identification of solid organ tumours however the non-selective nature of the dye leaves this staining method vulnerable to false negatives secondary to accumulation in other areas of pathology such as inflammation[26,27].

Intra-operative ICG perfusion angiograms to assist operator decision making during colon transection and anastomosis represents the most successful utilization of ICG in gastrointestinal surgery to date. Trials assessing subjective surgeon interpretation of these angiograms have been equivocal in their conclusions with respect to reducing complication rates and overall patient benefit however[28-30]. Numerous groups have set about quantifying these perfusion angiograms using time-fluorescence curves with the aim of reducing subjectivity of interpretation and ideally automating it entirely through computer vision and AI[31-35]. Son et al[36], in their landmark paper demonstrated the perfusion patterns seen during quantitative tracking of ICG colonic angiograms and subsequently analyzed these curves. They concluded that measurable parameters within these time-fluorescence curves such as the fluorescence slope, time from first fluorescence increase to half maximum value (T_{1/2 Max}) and time ratio (T_1/2max/T_max) could be used to detect areas of insufficient perfusion and reduce anastomotic complications[36]. Building on this work, Park et al[34], recently described an AI based real-time microcirculation analysis system (AIRAM) capable of generating more accurate and consistent perfusion assessment results when compared to the original parameter-based method described above. Using a corpus of 50 training videos the authors developed an unsupervised learning algorithm that identified 25 distinct colonic perfusion patterns during ICG inflow and outflow. Each perfusion pattern was then assigned a “risk level” or “assessment of adequacy of perfusion” (safe, intermediate, and dangerous) based on each pattern’s performance using a simulator of colonic circulation. Subsequent testing on 15 unseen videos demonstrated comparable results between the original parameter-based methodology and the AI algorithm with a computer processing time of less than 50 s.

Capitalizing on the well described biophysical differences in perfusion characteristics between malignant and benign tissues (abnormal angiogenesis such as capillary sprouting and increased interstitial pressures) and demonstrating these differences in colorectal lesions using ICG to create unique signatures, we have recently shown too that these signatures can be used to discriminate tissue accurately using traditional machine learning techniques without the need for large volumes of data and DL[37,38]. Blood flow, as well as the active and passive uptake of substances, are altered in malignant tissues. While these differences in fluorescence appearance can at times be appreciable to the human eye on screen, they are subtle, occur at different rates across the full field of view and transpire over several minutes. This complexity, along with the known variability between individuals to interpret intra-operative fluorescence footage, certainly requires the need for computer vision to interpret these differences[39].

To develop this biophysics inspired AI recommender, a commercially available Pinpoint Endoscopic Fluorescence Imaging System (Stryker Corp, Kalamazoo, MI, United States) was used to interrogate lesions within the distal colon transanally following intravenous administration of ICG. A bespoke fluorescence intensity tracker was then used to map intensity changes (representing blood inflow and outflow through tissue) on the multi-spectral intra-operative videos obtained. Tissues in these training videos with known pathology (healthy tissue, benign tissue, malignant tissue) were chosen as “regions of interest” and the fluorescent intensity changes tracked within these tissues to create “perfusion signatures” for each tissue type. The data created from these training videos were then taken and fitted to a parametric curve derived from a biophysical model of in vivo perfusion. A supervised machine learning based classification model was designed using these training perfusion signatures and then applied to tissue signatures of previously unseen videos in real time. Using this method the algorithm was able to successfully discriminate between healthy, cancerous and benign tissue in a pilot study of 20 patients with 95% accuracy[37,38]. Such systems employed clinically, providing objective feedback to the endoscopic operator, would permit either immediate local resection in the case of early disease or prompt appropriate, expedited referral for definitive surgical management in the case of more advanced disease. We are currently exploring the validity of this approach in non-colonic tumours with early results demonstrating generalizability of principle across tissue types and also in applying this working prototype to flexible endoscopic systems for more proximal colonic lesions.

CONCLUSION

Biophysics-inspired AI has numerous strengths over DL approaches when it comes to healthcare (Table 1). The transmutability of biophysics inspired AI is not seen in DL methods where for example, data sets collected to train colon cancer identification algorithms likely will not translate to other cancer types. In addition to solving the issue of training data volumes, biophysics inspired principles also provide answers to the “black box” concerns of DL use in medicine. Using DL methods, conclusions drawn by the system, while potentially accurate, are not “explainable” (the ability of the parameters used to justify the result) given the complexity of the algorithms used. This raises ethical dilemmas and accountability concerns where AI is used to direct patient treatment such as the decision to remove tissue or leave in-situ. Along with increased explainability, biophysics inspired modelling facilitates better system interpretability (the ability to associate a cause to an effect). The interrogation of tissue, through fluorescence guided surgery, allows artificially intelligent analysis of the fundamental properties of the tissue itself over pattern recognition within segmented images[40]. Furthermore, the detection and interpretation of these discrete tissue signals is likely less prone to the bias seen with other AI methods where deficiencies in the trainings data used, such as under-representation of particular conditions or people, negatively impacts the pattern detection capabilities of the AI method[41].

Table 1 Artificial intelligence methods in healthcare: A comparison of biophysics inspired machine learning and deep learning methods.

Criteria	Biophysics inspired machine learning	Deep learning
Principle	Identification of discriminating features within data set prior to system training based on already proven biophysical properties	Discriminating features/patterns in data discovered through analysis of large databanks
Training corpus for system to accurately assess unseen cases	Small to moderate data cohorts	Large training data corpuses required
Explainability	Settings, e.g., parameter description and number, used in algorithms are easily described	Complex algorithms utilizing numerous parameters and hyperparameters to control the learning process mean such algorithms often poorly understood
Interpretability	Conclusions reached are easily appreciated and can be explained logically by an appropriately trained individual	Human comprehension of sophisticated algorithm predictions/results may be difficult (including for experts in the field)
Generalizability	Accurate extrapolation of results to unseen cases as well as adaptation of such systems to other similar uses	High degree of specialization within DL systems makes adaptation to other similar uses difficult
Bias	Well described, transparent and biophysics-based features help reduce or identify bias within such systems	Bias within training datasets may be perpetuated by DL systems through subtle mechanisms that may even be imperceptible to humans

DL: Deep learning.

For these reasons, biophysics represents a core, although as yet underutilized, element of the next AI move in gastroenterology. This may be as a means to compensate for the apparent lack of video data that exists to train DL models or to augment DL methods by unlocking another realm of tissue specific information that is imparted by analysis of ICG behaviour within tissues. The extra information gleaned from the tissues’ biology, combined with AI methods, lay the blueprint for the creation of full field of view topographic maps that are biologically representative of each individual lesion and potentially even facilitate automation of procedures using fluorescent signal guidance.