Clinical impact of artificial intelligence-based solutions on imaging of the pancreas and liver

Abstract

Artificial intelligence (AI) has experienced substantial progress over the last ten years in many fields of application, including healthcare. In hepatology and pancreatology, major attention to date has been paid to its application to the assisted or even automated interpretation of radiological images, where AI can generate accurate and reproducible imaging diagnosis, reducing the physicians’ workload. AI can provide automatic or semi-automatic segmentation and registration of the liver and pancreatic glands and lesions. Furthermore, using radiomics, AI can introduce new quantitative information which is not visible to the human eye to radiological reports. AI has been applied in the detection and characterization of focal lesions and diffuse diseases of the liver and pancreas, such as neoplasms, chronic hepatic disease, or acute or chronic pancreatitis, among others. These solutions have been applied to different imaging techniques commonly used to diagnose liver and pancreatic diseases, such as ultrasound, endoscopic ultrasonography, computerized tomography (CT), magnetic resonance imaging, and positron emission tomography/CT. However, AI is also applied in this context to many other relevant steps involved in a comprehensive clinical scenario to manage a gastroenterological patient. AI can also be applied to choose the most convenient test prescription, to improve image quality or accelerate its acquisition, and to predict patient prognosis and treatment response. In this review, we summarize the current evidence on the application of AI to hepatic and pancreatic radiology, not only in regard to the interpretation of images, but also to all the steps involved in the radiological workflow in a broader sense. Lastly, we discuss the challenges and future directions of the clinical application of AI methods.

Key Words: Artificial intelligence, Machine learning, Deep learning, Imaging, Liver, Pancreas

Core Tip: The gastroenterology field is changing with the application of artificial intelligence (AI) solutions capable of assisting and even automating the interpretation of radiological images (ultrasound, endoscopic ultrasound, computerized tomography, magnetic resonance imaging, and positron emission tomography), generating accurate and reproducible diagnoses. AI can further be applied to other steps of the radiological workflow beyond image interpretation, including test selection, image quality improvement, acceleration of image acquisition, and prediction of patient prognosis and outcome. We herein discuss the current evidence, challenges, and future directions on the application of AI to hepatic and pancreatic radiology.

INTRODUCTION

Malignant tumors of the liver and pancreas are among the most common and lethal types of cancer. According to the recent GLOBOCAN 2020 data[1], liver and pancreas are the 6^th and 12^th most common sites for primary cancer, with 905677 and 495773 new cases in 2020, respectively. However, they also represent the 3^rd and 7^th neoplasia with the highest mortality, causing 830180 and 466003 deaths worldwide in 2020, respectively. If taken combined, cancer at the liver or pancreas thus represents the 5^th most incident and the second most lethal one.

Cancer at these locations account for almost as many deaths as cases. Five-year survival rates are 20% for liver cancer[2] and as low as 11% for pancreatic cancer[3], making them two of the cancer sites with the poorest prognosis. Other non-oncologic diseases affecting these organs are also highly prevalent, such as diffuse liver disease, including chronic liver disease, which affects tens of millions of people globally and represents a substantial socioeconomic burden[4].

Clinical outcomes of patients with these types of disease depend on a variety of factors, including stage and disease extension as assessed by imaging, and correct election of treatment. Thus, there is an unmet need for new tools capable of assisting specialists in early detection, characterization, and management of these diseases.

In recent years, artificial intelligence (AI) has shown promise in different areas of healthcare. The evaluation of medical images by machine learning (ML) approaches is a leading research field which, in gastroenterology, has applications in automatic analysis of different types of images, such as radiology, pathology, and endoscopy studies[5].

The first applications of AI to radiology have been dominated by anatomic locations such as the brain or the breast. Image analysis of abdominal organs, such as the liver and pancreas, are more challenging. Magnetic resonance imaging (MRI) in these locations, especially at 3 T, is prone to motion and field inhomogeneity artifacts, which are aggravated by larger fields of view[6]. As a result, advances in automatic analyses of abdominal images have gathered comparatively less attention. Nonetheless, the application of AI in liver and pancreas imaging is also gaining increasing interest (Figure 1). The goal of this review is to summarize the current experience on the use of AI to assist radiologists in their workflow, acquisition, and interpretation of medical images of the liver and pancreas.

Figure 1 PubMed results by year using the search terms. A and B: Artificial intelligence radiology (top) and artificial intelligence AND (liver OR pancreas) (bottom).

AI IN RADIOLOGY: BASIC PRINCIPLES

Artificial intelligence is expected to revolutionize the medical field, deeply impacting the hospital and clinical settings by potentially improving diagnostic accuracy, treatment delivery, and allowing a more personalized medical care[7]. Radiology will arguably be one the most changed areas of medicine because of AI implementation in its workflows, as the information-rich images generated in this field are an excellent source of data for the development of AI algorithms. Broadly, the term AI refers to a wide range of technologies and computing processes capable of imitating human intelligence to extract information from input data to solve a problem. This rapidly evolving area has a vocabulary of its own (Figure 2) that can be daunting to those not familiar with the field, including terms that are oftentimes used as synonyms to AI, such as ML.

Figure 2 Relation between artificial intelligence and related subdisciplines, neural network architectures, and/or techniques. ANN: Artificial neural network; FCN: Fully convolutional network; CNN: Convolutional neural network; GAN: Generation adversarial network.

ML is actually a subset of AI consisting of those methods capable of training a computer system to perform a given task based on provided information or experience without explicit programming, thus conferring machines the ability to learn[8]. The aim of ML is to predict an output based on a given input (a training dataset). Common ML applications in radiology include classification, image segmentation, regression, and clustering[9]. ML can be sub-divided into supervised and unsupervised learning[10]. In supervised learning, the most common type used in medical research, the algorithm is trained with labeled examples (i.e., the correct output for these training data, known as ground truth, is already known). Among the methods employed in supervised learning, random forest (RF), and specially, support vector machine (SVM), are powerful algorithms frequently used for the classification of images[7], including image segmentation. Conversely, in unsupervised learning, the ground truth is not known, as the algorithm is trained with unlabeled data that must be classified by the algorithm itself.

Artificial neural networks (ANNs), named after their brain-inspired structure and functioning process, can be trained via both supervised and unsupervised ML. In these ANNs, input information flows through a variable number of layers composed of artificial neurons, joined by weighted connectors, that process the data to obtain an output that matches the ground truth as closely as possible. Generative adversarial networks (GANs) are an example of ANN trained via unsupervised learning. GANs include two networks: One which creates new data based on input examples (i.e., generator), and one which distinguishes between different types of data (i.e., discriminator)[11]. These networks can be used to produce realistic, synthetic images as a strategy for data augmentation[12]. Similarly, the structure of convolutional neural networks (CNNs), a type of ANN specially designed for computer vision tasks, is based on that of the animal visual cortex. Typically used in image recognition and classification, in CNNs the input information is filtered and analyzed through a convolutional layer, and the size of the resulting image is subsequently reduced by a pooling layer. This two-step process will be repeated as many times as layers integrate the CNN, with a final step in which an ANN will classify the image (Figure 3). Fully convolutional networks (FCNs, a type of ANN that only performs the convolution step) are the basis for U-net, a modified architecture that consists of a contracting path including several convolutional and pooling layers to capture context, followed by a symmetric expanding path including a number of up-sampling and convolutional layers to enable accurate localization. U-net is a popular network for the development of automatic segmentation algorithms, as it requires relatively small datasets for algorithm training[13].

Figure 3 Diagram of a convolutional neural network used for the classification of a focal liver lesion in a computerized tomography image. HCC: Hepatocellular carcinoma; CT: Computed tomography.

Deep learning (DL) is a section of ML that utilizes multi-layered ANNs, referred to as deep neural networks (DNN), allowing the exploration of more complex data[14]. DL algorithms are gaining attention and raising considerable enthusiasm thanks to their scalability, easy accessibility, and ability to extract relevant information from the data without further indications other than input data. The recently developed nnU-Net, a publicly available DL-based segmentation tool capable of automatically configuring itself, has set a new state-of-the-art standard thanks to the systematization of the configuration process, which used to be a manual, complicated, and oftentimes limited task in previous approaches[15]. Improvement of the computational resources and the development of cloud technologies are also contributing to the application of DL architectures in a wide variety of research fields beyond medicine[14].

Closely related to the development of AI, the term radiomics refers to the computational extraction (via ML and DL algorithms) of quantitative data from radiological image features[16]. A particularly useful and valuable application of radiomics is the analysis of radiologic textures, defined as the differences in the grayscale intensities in the area of interest, which have been associated with intratumor heterogeneity[17] and that can potentially provide clinically relevant information that otherwise would remain unknown.

IMAGE ACQUISITION

The ultimate aim of computerized tomography (CT) and MRI is to unveil clinically relevant information; thus, the importance of this information relies heavily on the quality of the image. For CT, radiation dose is a parameter as important as image quality, and both are closely related to acquisition and reconstruction times. Iterative reconstruction (IR) algorithms[18] are the current technique of choice to transform the raw data into a 3D volume presented as an anatomical image. These algorithms generate an image estimate that is projected forward into a synthetic sinogram; subsequently, this image estimate is iteratively rectified by comparison with the real raw data sinogram until the algorithm’s predefined endpoint condition is met, resulting in enhanced image quality and thus allowing an important dose reduction[19]. DL reconstruction algorithms (DLR) are currently being developed with the aim to further improve image quality, therefore further reducing radiation doses. Compared to IR algorithms, DLR algorithms trained with low-dose data offer an improved signal-to-noise (SNR) ratio, as demonstrated by the U-net-based CNN developed by Jin et al[20], thus facilitating the detection of lesions of any kind and the increased use of low-dose imaging. Currently, there are two commercially available DLRs: TrueFidelity (GE Healthcare, Chicago, IL, United States) and AiCE (Canon Medical Systems, Otawara, Japan). Akagi et al[21] employed AiCE in their study and reported improved contrast-to-noise ratio and image quality in CT images, compared to images created with a hybrid IR algorithm. Although the preliminary results are exciting, further validation for these DLR algorithms is required, and real dose reduction in the clinical setting has yet to be demonstrated.

An important setback of MRI is the long acquisition time, forcing the patient to lay still for a relatively long period and with any movement affecting the quality of the image. One way to reduce acquisition time is compressed sensing, based on the idea that if signal information is only present in a small portion of pixels, that sparsity can be used to reconstruct a high-definition image from considerably less collected data (undersampling). Kaga et al[22] evaluated the usefulness of the Compressed SENSE algorithm (Philips, Amsterdam, The Netherlands) in MRI of the abdomen using diffusion weighted images (DWIs) and reported a significantly improved image noise and contour of the liver and pancreas and higher apparent diffusion coefficient values, thus offering superior image quality compared to parallel imaging (PI)-DWI[22].

AI applications have also been designed to automate MRI and CT protocol selection with the aim to standardize workflows and increase effectiveness in the radiology setting. The selection of an appropriate imaging protocol requires taking into account factors including the type of procedure, clinical indication, and the patient’s medical history. The increasing incorporation of electronic medical records and other digital content has opened opportunities for the application of natural language processing (NLP) methods to extract structured data from unstructured radiology reports. López-Úbeda et al[23] developed an NLP-based classification system for automated protocol assignment that offered an overall accuracy of 92.25% for the CT and 86.91% for the MRI datasets. This system has already been successfully implemented and is currently in use at the HT Médica centers.

Information about the respiration of the patient can be used for functional studies, overall monitoring, or motion compensation during the performance of an MRI. Typically, breathing is measured via belts or nasal sensors that can potentially alter the raw MRI data. Using adaptive intelligence, the laser-based VitalEye system (Philips) registers a contactless continuous respiratory signal, with up to 50 body locations analyzed simultaneously and in real time, thus producing a more robust respiratory trace compared to traditional respiratory belts[24]. Moreover, as soon as the patient is lying on the table, the BioMatrix Respiratory Sensors (Siemens AG, Munich, Germany) embedded in the spinal coil produce a local magnetic field that changes with the variation of lung volume during breathing. These changes are registered, and the breathing pattern is integrated to optimize image quality[25]. By standardizing and accelerating the workflow, these advances allow technicians and radiologists to concentrate on the patient.

IMAGE ANALYSIS

Segmentation of liver and pancreas

Image analysis has experimented a huge progression with the advent of AI, and especially with DL, reaching state-of-art performances in many biomedical image analysis tasks[26-28] (Table 1). Among them, segmentation is one of the most important in radiology. For instance, accurate pancreas segmentation has applications in surgical planning, assessment of diabetes, and detection and analysis of pancreatic tumors[29]. Another key application of organ and lesion contouring is treatment volume calculation for radiotherapy planning. However, boundary delimitation of anatomical structures in medical images remains a challenge due to their complexity, particularly in the upper abdominal cavity, where there are constant changes in the position of the different organs with the respiratory cycle, as well as the occurrence of anatomical variants and pathological changes of organs[30].

Table 1 Works proposed for automated image analysis.

Image analysis	Anatomical area	Modality	AI model	Ref.
Segmentation	Pancreas	MRI	CNN	[33,34,110]
			UDCGAN	[111]
			3D-Unet	[112]
	Liver	CT	SSC (no AI)	[36]
			PA (Atlas-no AI)	[39]
		MRI	CNN	[37,38,42,113]
			GAN	[43]
Registration	Liver	CT, MRI	CNN	[47]
			SG-DIR (no AI)	[48]
			Cycle-GAN + UR-Net	[46]
		4D-MRI	Non-rigid	[49]

MRI: Magnetic resonance imaging; CT: Computerized tomography; 4D-MRI: Four-dimensional magnetic resonance imaging; CNN: Convolutional neural network; UDCGAN: U-Type densely connected generation adversarial network; SCC: Sparse shape composition; AI: Artificial intelligence; PA: Probabilistic atlas; GAN: Generation adversarial network; SG-DIR: Structure-guided deformable image registration; UR-Net: Unsupervised registration network.

The intersubject variability and complexity of the pancreas make segmentation of this organ a demanding task. Segmentation of pancreatic cancer lesions is particularly challenging because of their limited contrast and blurred boundaries against the background pancreatic parenchyma in CT and MR images[31]. In addition, other factors such as body mass index, visceral abdominal fat, volume of the pancreas, standard deviation of CT attenuation within pancreas, and median and average CT attenuation in the immediate neighborhood of the pancreas may affect segmentation accuracy[29,32].

These problems lead to high segmentation uncertainty and inaccurate results. To tackle these problems, Zheng et al[33] proposed a 2D, DL-based method that describes the uncertain regions of pancreatic MR images based on shadowed sets theory. It demonstrated high accuracy, with a dice similarity coefficient (DSC) of 73.88% on a cancer MRI dataset and 84.37% on the National Institutes of Health (NIH) Pancreas dataset (which contains 82 CT scans of healthy pancreas), respectively. The same authors reported[34] a more sophisticated 2.5D network that benefits from multi-level slice interaction. They surpassed state-of-art performances in the NIH dataset, with a DSC of 86.21% ± 4.37%, sensitivity of 87.49% ± 6.38%, and a specificity of 85.11% ± 6.49%.

The liver is also a popular target for automated segmentation algorithms. Automatic segmentation of this organ is regarded as somewhat less challenging than that of the pancreas, with reported DSC scores typically in the > 0.90 range[35].

Li et al[36] presented a liver segmentation method from abdominal CT volumes for both healthy and pathological tissues, based on the level set and sparse shape composition (SSC) method. The experiments, performed using public databases SILVER07 and 3Dicardb, showed good results, with mean ASD, RMSD, MSD, VOE, and RVD of 0.9 mm, 1.8 mm, 19.4 mm, 5.1%, and 0.1%, respectively. Moreover, Winther et al[37] used a 3D DNN for automatic liver segmentation along with a Gd-EOB-DTPA-enhanced liver MR images dataset. Results show an intraclass correlation coefficient (ICC) of 0.987, DSC of 96.7% ± 1.9%, and a Hausdorff distance of 24.9 mm ± 14.7 mm compared with two expert readers who corresponded to an ICC of 0.973 and a DSC of 95.2% ± 2.8%. Finally, Mohagheghi et al[38] used a CNN but further incorporated prior knowledge. The model learnt the global shape information as prior knowledge by using a convolutional denoising auto-encoder; then, this knowledge was used to define a loss function and combine it with the Dice loss in the main segmentation model. This model with prior knowledge improved the performance of the 3D U-Net model and reached a DSC of 97.62% segmenting CT images of the Silver07-liver dataset.

Organ segmentation is even more challenging in pediatric patients studied with CT, as it is acquired at a low dose to minimize harmful radiation to children, thus having a lower SNR. Nakayama et al[39] proposed a liver segmentation algorithm for pediatric CT scans using a patient-specific level set distribution model to generate a probabilistic atlas, obtaining a DSC index of 88.21% in the segmentation. This approach may be useful for low dose studies in general, i.e., also in the adult population.

Algorithms for automatic segmentation of the liver using MR images have proven equally efficient. For instance, Bobo et al[40] used a 2D FCN architecture to segment livers on T2-weighted MR images with a DSC score of 0.913. In a recent paper, Saunders et al[41] systematically analyzed the performance of different types of MR images in the training of CNN for liver segmentation, using a 3D U-net architecture. Water and fat images outperformed other modalities, such as T2* images, with a DSC of 0.94.

Conversely, high-quality automatic segmentation of liver lesions is not an easy task, since the low contrast between tumors and healthy liver parenchyma in CT images, their inhomogeneity, and its complexity pose a challenge for liver tumor segmentation. In addition, motion-induced phase errors due to peristaltic and respiratory movements negatively affect image quality and assessment of liver lesions in MR images. A 3D CNN was used by Meng et al[42] where a special three-dimensional dual path multiscale convolutional neural network (TDP-CNN) was designed for liver tumor segmentation. Results achieved in the LiTS public dataset were a DSC of 68.9%, Hausdorff distance of 7.96 mm, and average distance of 1.07 mm for liver tumor segmentation and a DSC of 96.5%, Hausdorff distance of 29.162 mm, and average distance of 0.197 mm for liver segmentation. A different approach for liver tumor segmentation was proposed by Chen et al[43]. In this work, an adversarial densely connected network algorithm was trained and evaluated using the Liver Tumor Segmentation challenge dataset. Results revealed an average Dice score of 68.4% and ASD, MSD, VOE, and RVD of 21 mm, 124 mm, 0.46%, and 0.73%, respectively.

Automatic contouring of hepatic tumor volumes has also been reported using CT scans, a modified SegNet CNN[44], and dynamic contrast enhanced (DCE)-MRI images in a U-net-like architecture[45], for example.

Some medical imaging vendors incorporate solutions for liver segmentation and hepatic lesion characterization integrated in the proprietary radiologist’s workflow. For instance, the Liver Analysis research application from Siemens Healthcare (Erlangen, Germany) aims to provide AI support for liver MRI and CT reading. The tool includes DL-based algorithms for automatic segmentation of the whole liver, functional liver segments, and other abdominal organs like the spleen and kidneys (Figure 4A). It also features an AI method to automatically detect and segment focal liver lesions, providing lesion diameter, volume, and 3D contours (Figure 4B).

Figure 4 In-house experience on liver assessment with artificial intelligence. Magnetic resonance studies of a patient with liver focal lesions (liver hemangiomas), processed with the Liver Analysis research application from Siemens Healthcare. A: Automatic segmentation of the whole liver, liver segments, and other abdominal organs; B: Automatic detection, segmentation, and measurement of the two liver hemangiomas.

Registration

Medical image registration seeks to find an optimal spatial transformation that best aligns the underlying anatomical structures. Medical image registration is used in many clinical applications such as image guidance systems (IGS), motion tracking, segmentation, dose accumulation, image reconstruction, etc[28]. In clinical practice, image registration is a major problem in image-guided liver interventions, especially for the soft-tissues, where organ shape changes occurring between pre-procedural and intra-procedural imaging pose significant challenges[46]. Schneider et al[47] showed how semi-automatic registration in IGS may improve patient safety by enabling 3D visualization of critical intra- and extra-hepatic structures. A novel IGS (SmartLiver) offering augmented reality visualization was developed to provide intuitive visualization by using DL algorithms for semi-automatic image registration. Results showed a mean registration accuracy of 10.9 mm ± 4.2 mm (manual) vs 13.9 mm ± 4.4 mm (semi-automatic), hence significantly improving the manual registration. Kuznetsova et al[48] assessed the performance of structure-guided deformable image registration (SG-DIR) relative to rigid registration and DIR using TG-132 recommendations for 14 patients with liver tumors to whom stereotactic body radiation therapy (SBRT) was applied. The median DSC for rigid registration was 88% and 89% for DIR, and 90% for both SG-DIR using liver contours only and using liver structures along with anatomical landmarks. However, most of the existing volumetric registration algorithms are not suitable for the intra-procedural stage, as they involve time-consuming optimization. In the report by Wei et al[46], a fast MR-CT image registration method was proposed for overlaying pre-procedural MR (pMR) and pre-procedural CT (pCT) images onto an intra-procedural CT (iCT) image to guide thermal ablation of liver tumors. This method, consisting of four DL-based modules and one conventional ANTs registration module, showed higher Dice ratios (around 7% improvement) over tumors and compatible Dice ratios over livers. However, its main advantage was the computational time cost of around 7 s in the intra-procedural stage, which is only 0.1% runtime in the conventional way (i.e., ANTs).

Treatment planning concepts using the mid-ventilation and internal-target volume concept are based on the extent of tumor motion between expiration and inspiration. Therefore, four-dimensional (4D) imaging is required to provide the necessary information about the individual respiration-associated motion pattern. Weick et al[49] proposed a method to increase the image quality of the end-expiratory and end-inspiratory phases of retrospective respiratory self-gated 4D MRI data sets using two different non-rigid image registration schemes for improved target delineation of moving liver tumors. In the first scheme, all phases were registered directly (dir-Reg), while in the second next neighbors were successively registered until the target was reached (nn-Reg). Results showed that the Median dir-Reg coefficient of variation of all regions of interest (ROIs) was 5.6% lower for expiration and 7.0% lower for inspiration compared with nn-Reg. Statistically significant differences were found in all comparisons.

DIAGNOSIS

Two decades ago, the methods proposed for ML-based diagnosis required manually extracting the features from the images. This tedious step has been partially relieved with the irruption of CNNs. However, techniques such as radiomics are still in use to try to improve the performance of novel AI methods for medical diagnosis. Radiomics concerns the high throughput extracting of comprehensible features from radiological images that can be further analyzed by ML algorithms for classification or regression tasks. In this section, different methods proposed for liver and pancreas imaging diagnosis are reviewed (Table 2).

Table 2 Summary of works based on artificial intelligence for automated diagnosis of pancreas and hepatobiliary system diseases.

Anatomical area	Modality	AI model	What is diagnosed?	Ref.
Liver	Scintiscan	ANN	Chronic hepatitis and cirrhosis	[114]
	CT	ANN	HCC, intra-hepatic peripheral cholangiocarcinoma, hemangioma, metastases	[52]
		CNN	HCC, malignant liver tumors, indeterminate mases, hemangiomas, cysts	[53]
			Liver fibrosis	[50,115]
		SVM	Cirrhosis and HCC	[51]
			Malignant liver tumors	[54]
		KNN, SVM, RF	HCC	[116]
	MRI	CNN	HCC	[55]
			Simple cyst, cavernous hemangioma, FNH, HCC, ICC	[56,57]
		Extremely randomized trees	Adenomas, cysts, hemangiomas, HCC, metastases	[58]
	US	PNN	Benign and malignant focal liver lesions	[65]
		SVM	Fatty liver	[68]
			HCC	[66]
		CNN	Focal liver lesions: Angioma, metastasis, HCC, cyst, FNH	[67]
			Liver fibrosis stages	[69]
Biliary system	MRI	ANN	Cholangiocarcinoma	[59,60]
			Lymph node status in ICC	[117]
Pancreas	CT	Hybrid SVM-RF	Pancreas cancer	[76]
		SVM	Serous cystic neoplasms	[72]
		CNN	IPMN, mucinous cystic neoplasm, serous cystic neoplasm, solid pseudopapillary tumor	[73]
	MRI	SVM	IPMN	[78]
	US	ANN	Chronic pancreatitis, pancreatic adenocarcinoma	[81]
		CNN	Malignancy in IPMN	[82]
			Autoimmune pancreatitis, pancreatic ductal adenocarcinoma, chronic pancreatitis	[83]

CT: Computerized tomography; MRI: Magnetic resonance imaging; US: Ultrasound; ANN: Artificial neural network; CNN: Convolutional neural network; SVM: Support vector machine; KNN: K-nearest neighbors; RF: Random forest; PNN: Probabilistic neural network; HCC: Hepatocellular carcinoma; FNH: Focal nodular hyperplasia; ICC: Intrahepatic cholangiocarcinoma; IPMN: Intra-ductal papillary mucinous neoplasm.

Liver-CT

Starting with chronic liver disease, Choi et al[50] presented a CNN model for staging liver fibrosis from contrast-enhanced CT images. Before using the CT image as input for the CNN, the liver is segmented. The testing dataset included 891 patients and the CNN achieved a staging accuracy of 79.4% and an AUC of 96%, 97%, and 95% for diagnosing significant fibrosis, advanced fibrosis, and cirrhosis, respectively. A different approach was proposed by Nayak et al[51], where SVM was used instead of CNN for aiding in the diagnosis of cirrhosis and hepatocellular carcinoma (HCC) from multi-phase abdomen CT. Features were extracted from the segmented liver in all the phases, which were previously registered. Using 5-fold cross validation, they reported an accuracy of 86.9% and 81% for the detection of cirrhosis and HCC, respectively.

There are also several reports exploring the role of DL in the characterization of focal liver lesions (Figure 5). In this sense, Matake et al[52] applied an ANN to assist in the diagnosis of hepatic mases using clinical and radiological parameters extracted from CT images. The authors used 120 cases of liver diseases and implemented a leave-one-out cross-validation method for training and testing the ANN, reporting an AUC of 96.1%. Also using CT images, Yasaka et al[53] used a CNN for the differentiation of five different types of liver masses from contrast-enhanced CT. For testing, they used 100 liver mass images, reporting an accuracy of 84%. Similarly, Khan and Narejo[54] proposed Fuzzy Linguistic Constant (FLC) to enhance low contrast CT images of the liver before training a SVM to distinguish between cancerous or non-cancerous lesions. The reported classification accuracy was 98.3%. The proposed method also showed the ability to automatically segment the tumor with an improved detection rate of 78% and a precision value of 60%.

Figure 5 Computerized tomography scan of a 61-year-old male patient with colon carcinoma and liver metastases. The intensity histograms of regions with and without metastases are different; hence, the first order radiomics features[109], which are based on the intensity histogram will potentially be different.

Liver and biliary system MRI

Techniques concerning MR images have also been developed for the diagnosis and classification of focal liver lesions (Figure 6). Zhou et al[55] proposed a method using a novel CNN to grade HCC from DWIs. They applied a 2D CNN to log maps generated from different b-value images. In their work, they reported a validation AUC of 83% using 40 cases. A CNN was also trained by Hamm et al[56] and Wang et al[57] to classify six different focal hepatic lesions from T1-weighted MR images in the postcontrast phase. They used 60 cases for testing and reported a sensitivity and specificity of 90% and 98%, respectively. In the second part of their study, they transformed it into an “interpretable” DL system by analyzing the relative contributions of specific imaging features to its predictions in order to shed light on the factors involved in the network’s decision-making process. Finally, DCE-MRI and T2-weighted MRI, together with risk factor features, were applied to build an extremely randomized trees classifier for focal liver lesions[58], achieving an overall accuracy of 77%.

Figure 6 Sixty-seven-year-old patient with pancreatic carcinoma and liver metastases treated with chemotherapy. The Digital Oncology Companion (Siemens Healthineers, Germany) artificial intelligence-based prototype automatically segments liver, portal and hepatic vessels, lesions, and surrounding anatomical structures. From left to right: screenshots of the segmented liver, vessels, and lesions; and generated 3D models.

Some advancements have also been reached in the automatic diagnosis of lesions in the biliary system from MR cholangiopancreatography (MRCP) sequences. Logeswaran[59,60] trained an ANN classifier for assisting in the diagnosis of cholangiocarcinoma. He utilized 55 MRCP studies for testing and reported an accuracy of 94% when differentiating healthy and tumor images and of 88% in multi-disease tests.

MRI is a superior technique in the evaluation of chronic liver disease in comparison with CT, but making the most of it requires considerable skills and optimization at the acquisition, post-processing, and interpretation phases[61]. AI has proved useful to assist radiologists in the MR-guided diagnosis and grading of these diseases, including liver fibrosis and non-alcoholic fatty liver disease[62].

Radiomics studies have been proposed to aid in the diagnosis of liver fibrosis. Kato et al[63] performed texture analysis of the liver parenchyma processed by an ANN to detect and grade hepatic fibrosis, with varying success depending on the type of MR sequence used (AUC of 0.801, 0.597, and 0.525 for gadolinium-enhanced equilibrium phase, T1-weighted, and T2-weighted images, respectively).

Later, Hectors et al[64] developed a DL algorithm for liver fibrosis staging using gadolinium enhancement sequences acquired in the hepatobiliary phase, which showed good to excellent diagnostic performance, comparable to that of MR elastography.

Liver-US

Ultrasound (US) and endoscopic ultrasonography (EUS) are commonly used in the diagnostic work-up of several pancreatic and liver lesions. AI-based solutions have also been applied to US images in the assessment of focal and diffuse liver diseases in order to enhance their diagnostic capabilities. Acharya et al[65] suggested a method for aiding in the diagnosis of focal liver lesions from liver US images. The authors extracted features from US images and trained several classifiers, obtaining the highest AUC (94.1%) using a probabilistic neural network (PNN) classifier. Another approach is shown in Yao et al[66], where a radiomics analysis was established for the diagnosis and clinical behavior prediction of HCC, showing an AUC of 94% for benign and malignant classification. Rightly, CNN architectures have also been developed for US images as in the report by Schmauch et al[67], where a CNN was employed to help in the diagnosis of focal liver lesions from US images. The authors used a dataset composed by 367 2D US images for training and another dataset from 177 patients for testing, reporting a mean score of 89.1%.

There is limited experience in the use of AI with US images with regards to diffuse liver disease. Li et al[68] used a SVM classifier to help in the diagnosis of fatty liver from US images. Input features were computed from ROIs selected by examiners. A total of 93 images were used for training and testing using leave-one-out cross-validation. The authors reported an 84% accuracy for normal livers and 97.1% for fatty livers. Moreover, a mix of radiomics features and DL techniques were used with two-dimensional shear waver elastography (2D-SWE) for assessing liver fibrosis stages in Wang et al[69]. Results reached AUCs of 97% for cirrhosis, 98% for advanced fibrosis, and 85% for significant fibrosis.

Pancreas CT and PET/CT

The role of AI in the detection of pancreatic lesions from CT has been extensively investigated. Pancreatic cancer detection is a challenging task for radiologists and its improvement is a hot research topic. Chen et al[70] developed a DL-based tool including a segmentation CNN and a 5-CNN classifier for the detection of pancreatic cancer lesions, with a special focus on lesions smaller than 2 cm, in abdominal CT scans. Their model was able to distinguish between cancer and control scans with an AUC of 0.95, 89.7% sensitivity, and 92.8% specificity. Sensitivity for the detection of lesions smaller than 2 cm was 74.7%[70]. Still focused on the identification of lesions smaller than 2 cm, Alves et al[71] proposed an automatic framework for pancreatic ductal adenocarcinoma (PDAC) detection based on state-of-the-art DL models. They trained an nnUnet (nnUnet_T) on a dataset including contrast-enhanced CT scans from 119 PDAC patients and 123 healthy individuals for automatic lesion detection and segmentation. Additionally, two other nnUnets were trained to investigate the impact of anatomy integration, with nnUnet_TP segmenting both the pancreas and the tumor and nnUnet_MS segmenting the pancreas, tumor, and adjacent anatomical structures. All three networks were compared on an open access external dataset, with nnUnet_MS offering the best results with an AUC of 0.91 for the entire dataset and of 0.88 for lesions smaller than 2 cm[71]. Several studies have focused on the role of AI-based solutions in the detection of pancreatic cystic lesions. Wei et al[72] presented a ML-based computer-aided diagnosis system to help in the diagnosis of pancreas serous cystic neoplasms from CT images. They extracted radiomic features from manual ROIs outlining the peripheral margin of each neoplasm. After selecting the most important features by using least absolute shrinkage selection operator regression, they trained a SVM classifier by a 5-fold cross validation with 200 patients. The authors used a validation cohort of 60 patients and reported and AUC of 83.7%, a sensitivity of 66.7%, and a specificity of 81.8%. Along the same lines, Li et al[73] also proposed a computer-aided framework for early differential diagnosis of pancreatic cysts without pre-segmenting the lesions by using densely connected convolutional networks (Dense-Net). In this approach, saliency maps were integrated in the framework to assist physicians to understand the decisions of the DL methods. Accuracy reported on a cohort of 206 patients with four pathologically confirmed subtypes of pancreatic cysts was 72.8%, significantly higher than the baseline of 48.1% according to the authors. Park et al[74] developed a 3D nnU-Net-based model for the automatic diagnosis of solid and cystic pancreatic neoplasms on abdominal CT scans. The model was trained on CT scans (852 patients) from both patients who underwent resection for pancreatic lesions and subjects without any pancreatic abnormalities, and performance was evaluated using receiver operating characteristic analysis in a temporally independent cohort (test set 1, including 603 patients) and a temporally and spatially independent cohort (test set 2, including 589 patients). This approach showed a remarkable capacity to identify solid and cystic pancreatic lesions on CT, with an AUC of 0.91 for the test set 1 and 0.87 for the test set 2. Furthermore, it offered a high sensitivity in the identification of solid lesions of any size (98%-100%) and cystic lesions of at least 1 cm (92%-93%)[74].

In the pursuit of more accurate models, some authors have combined CT images with other biomarkers, such as molecular markers or multimodal images. For example, Qiao et al[75] used CT scans and serum tumor markers (including serum carbohydrate antigens 50, 199, and 242) to train different types of networks (CNN, FCN, and U-Net) to diagnose pancreatic cancer with high sensitivity and specificity. Li et al[76] also used a hybrid SVM-RF model to classify normal and pancreas cancer from PET/CT images. First, they segmented the pancreas from CT images and registered the CT and PET series, then they extracted features from the segmented ROI in both types of studies. The authors tested the model using 10-fold cross validation with 80 cases and achieved 96.47% accuracy, 95.23% sensitivity, and 97.51% specificity.

Pancreas-MRI

MR is the technique of election for the assessment of complex pancreatic conditions. Thus, its association with AI is regarded as promising to help radiologists in diagnostic dilemmas regarding this organ. For instance, radiomics has been proposed as a way to predict the malignant potential of pancreatic cystic lesions, differentiating benign cysts from those likely to transform into pancreatic cancer[77].

There is limited experience with the use of AI in the detection of focal lesions with pancreatic MR studies. Corral et al[78] proposed the use of SVM to classify intraductal papillary mucinous neoplasms (IPMN). First, features were extracted using a CNN from T2-weighted and post-contrast T1-weighted MR images. For validation, authors used 10-fold cross-validation using 139 cases. They achieved an AUC of 78%. Kaissis et al[79] also developed a supervised ML algorithm which predicted the above-versus-below median overall survival of patients with pancreatic ductal adenocarcinoma, with 87% sensitivity and 80% specificity, using preoperative DWIs.

Lastly, the generation of synthetic MR images of pancreatic neuroendocrine tumors (PNET) has been explored using GANs. This data augmentation technique can alleviate the relative low abundance of these type of pancreatic tumors in order to train AI models. Gao and Wang then used the synthetic images to evaluate the performance of a CNN in the prediction of PNET grading on contrast-enhanced images[80].

Pancreas-EUS

Application of AI to EUS has focused on the differentiation of focal pancreatic lesions. In this sense, Săftoiu et al[81] developed an ANN to help in the difficult differentiation between PDAC and focal chronic pancreatitis (CP) with EUS-elastography. They included 258 patients in the study and reported 84.27% testing accuracy using 10-fold cross-validation. In addition, Kuwahara et al[82] used a CNN to assist in the distinction between benign and malignant IPMNs of the pancreas from EUS images. For testing, the authors used images from 50 patients, obtaining an AUC of 98% and sensitivity, specificity, and accuracy values of 95.7%, 92.6%, and 94%, respectively. Finally, in the report by Marya et al[83] an EUS-based CNN model was trained to differentiate autoimmune pancreatitis (AIP) from PDAC, CP, and normal pancreas (NP). Results obtained from 583 patients (146 AIP, 292 PDAC, 72 CP, and 73 NP) demonstrated a sensitivity of 99% and a specificity of 98% to distinguish between AIP and NP, 94% and 71% for AIP and CP, and 90% and 93% for AIP and PDAC. Furthermore, the sensitivity and specificity to distinguish AIP from all study conditions (i.e., PDAC, CP, and NP) were 90% and 85%, respectively. In view of these results, the application of AI to EUS in the assessment of focal pancreatic lesions is promising, although limited due to the short number of available databases for algorithm training and validation[84].

TREATMENT PREDICTION

Prediction of treatment response and patient outcome based on AI is a very appealing idea which has been explored in a number of liver and pancreatic diseases, particularly in patients with HCC (Table 3).

Table 3 Summary of the works proposed to predict patient prognosis using artificial intelligence.

Anatomical area	Pathology	Modality	AI model	What is prognosed?	Ref.
Liver	HCC	CT	ANN	Progression of hepatectomized patients with HCC	[85]
			CNN	Early recurrence of HCC	[88]
				Response to transarterial chemoembolization for patients with intermediate-stage HCC	[89]
			LASSO Cox regression	Early recurrence of HCC	[90]
				Recurrence of HCC after liver transplantation	[91]
				Recurrence of HCC after resection	[118]
		MRI	LR, RF	Response to intra-arterial treatment of HCC	[86,87]
		US	CNN, SVM	Response to transarterial chemoembolization for patients with HCC	[119]
Biliary system	Liver metastases, HCC, cholangiocarcinoma	CT	CNN	Prediction of hepatobiliary toxicity after liver SBRT	[92]
Pancreas	Postoperative pancreatic fistula	CT	RepTree	Prediction of postoperative pancreas fistulas after pancreatoduodenectomy	[93]

HCC: Hepatocellular carcinoma; CT: Computerized tomography; MRI: Magnetic resonance imaging; US: Ultrasound; ANN: Artificial neural network; CNN: Convolutional neural network; LR: Logistic regression; RF: Random forest; SVM: Support vector machine; SBRT: Stereotactic body radiotherapy.

The idea of using ML to predict the prognosis of patients with HCC emerged decades ago. Already in 1995 the progression of hepatectomized patients with HCC was analyzed using ANN[85]. Liver volume, which was measured in CT studies, was used, among others, as an input parameter. Fifty-four example cases were used to train an ANN composed of three layers, and the model was successfully used to predict the prognosis of 11 patients. Nevertheless, the model was not tested with enough cases to determine its usefulness in actual clinical activity. However, the rise of AI has prompted many more works to be developed in the last few years. The response to intra-arterial treatment of HCC prior to intervention has been predicted using ML[86,87]. Specifically, logistic regression (LR) and RF models were trained with 35 patients using features extracted from clinical data and the segmentations of liver and liver lesions in a contrast-enhanced 3D fat-suppressed spoiled gradient-echo T1-weighted sequence in the arterial phase. Both trained models predicted treatment response with an overall accuracy of 78% (62.5% sensitivity, 82.1% specificity). Other authors tried to predict the early recurrence of HCC employing a CNN model based on the combination of CT images and clinical data[88]. They used 10-fold cross-validation with data from 167 patients and reported an AUC of 0.825. A RestNet CNN model was also trained for preoperative response prediction of patients with intermediate-stage HCC undergoing transarterial chemoembolization[89]. The model used the segmented ROI of the tumor area in a CT study as input. The training cohort included 162 patients and the two validation cohorts included 89 and 138 patients, respectively. The authors reported an accuracy of 85.1% and 82.8% in the two evaluation datasets.

Radiomics has also been applied to predict treatment response of HCC to different therapies based on studies of several imaging modalities. The early recurrence of HCC after curative treatment was evaluated using an LR model based on radiomics features[90], which were extracted from manually delineated peritumoral areas in CT images. They used 109 patients for training and 47 patients for validation, reporting an AUC of 0.79 with the validation dataset. Guo et al[91] also predicted the recurrence of HCC after liver transplantation. For that purpose, authors extracted radiomic features from ROIs delineated around the lesion in arterial-phase CT images. Then, they combined clinical risk factors and radiomic features to build a multivariable Cox regression model. The authors used a training dataset of 93 patients and a validation dataset of 40 patients and they reported a C-index of 0.789 in the validation dataset.

ML models have also been used to predict hepatobiliary toxicity after liver SBRT[92]. The authors built a CNN model which was previously pretrained using CT images of human organs. Then, using transfer learning, the model was trained with liver SBRT cases. They used 125 patients for training and validation using a 20-fold cross-validation approach, reporting an AUC of 0.79.

Regarding the pancreas, postoperative pancreatic fistulas were predicted using ML-based texture analysis[93] performed to extract features from ROIs segmented in non-contrast CT images. Then, after dimension reduction, several ML classifiers were built using Auto-WEKA 2.0, obtaining the best results using a REPTree classifier. The authors used 10-fold cross-validation using data from 110 patients, and reported an AUC of 0.95, sensitivity of 96%, and specificity of 98%.

DISCUSSION

In recent years, a large number of AI-based solutions have been developed with the aim of easing and streamlining the radiologist’s workflow. Many of these tools are focused on imaging of the liver, biliary system, and pancreas. The developed tools range from improving image quality to the prediction of the patient’s prognosis after treatment. The literature shows that many AI-based solutions targeting liver and pancreas imaging allow for improved disease detection and characterization, lower inter-reader variability, and increased diagnostic efficiency. A key factor for their success in the clinical setting is to attain a seamless integration in the radiologist’s workflow, requiring minimal additional work by the radiologist and adding significant value to the radiologist’s work. In this sense, it is crucial that there is a fluid collaboration between the radiologists, technicians, and bioengineers in charge of the tools.

Image analysis and processing are transversal parts of most AI methods described in this review. Improving their performance is thus a key task. Unfortunately, some image processing techniques such as registration are still time-consuming, hence making the incorporation of some of these procedures in clinical practice unfeasible. Some new methods are arising to minimize this impact[94], especially in critical applications like image IGS. Semi-automatic or even automatic segmentation is another important step that some of the AI tools may incorporate for diagnosis or prognosis purposes[95]. Therefore, it is of paramount importance for these algorithms to achieve a high level of performance.

The literature reports many applications of AI to aid in the detection and characterization of pancreatic and liver focal lesions using a variety of imaging modalities as input, either single (e.g., T1-weighted MRI) or in combination with other techniques and data (e.g., T2-weighted and DCE-MRI plus risk factors). In chronic liver disease, radiomics-based tools have been developed to assist in the diagnosis and grading of hepatic fibrosis, among others. These models have been built using different imaging modalities, such as MRI or US.

With regard to the prognosis of liver, biliary or pancreatic diseases, tools based on radiological information have hardly been developed. Many of these tools are focused on the prognosis of HCC based on information extracted from CT[96]. In this field of research, literature shows a clear trend toward integrating genetic information[97-101]. There are also studies that try to include variables extracted from clinical data and laboratory values[102,103]. In a scenario that advances towards integrated diagnosis, increasing volumes of data of different nature are available. This should allow for the generation of more accurate predictive models of clinical prognosis using information from many sources.

For the AI-based tools developed to be used in daily clinical practice, they must obtain regulatory clearance, such as Food and Drug Administration (FDA) approval in the United States or CE marking in Europe. Despite the explosive production of such tools in the last years, to date only a small group of them have obtained this approval. One of the main problems is the lack of appropriately annotated data. Without large datasets of properly labeled studies, the performance of data-hungry algorithms like CNNs will not be sufficient to be massively deployed in clinical environments. Furthermore, algorithms demand diverse data, such as multi-centric and multi-vendor, to avoid selection biases that would challenge their implementation in a real-world environment[104]. Another limitation of most AI-based tools found today is that they are aimed at a very concrete application (narrow AI applications), within a specific imaging modality, rather than being valid for a wide range of tasks at the radiologist’s work practice.

Yet, the general attitude of radiology staff toward AI is positive. In a recent survey, European radiographers declared excitement about AI (83%), although only 8% had been taught on this matter in their qualification studies[105].

In another survey, European radiologists regarded the outcomes of AI algorithms for diagnostic purposes as generally reliable (75.7%), and algorithms for workload prioritization as very helpful (23.4%) or moderately helpful (62.2%) to reduce the workload of the medical staff[106].

The sentiment of gastroenterologists toward AI is also generally favorable, with a wide majority of United Kingdom[107] and European[108] specialists perceiving it as beneficial to key aspects of their clinical practice. Their main concerns according to these studies are related to algorithm bias, lack of guidelines, and potential increase in procedural times and operator dependence.

CONCLUSION

The rapid advance of AI is already transforming the gastrointestinal field with the development of applications aimed to assist and streamline image diagnosis. Traditional diagnostic imaging techniques such as US, EUS, CT, MRI, and PET/CT are already benefitting from a variety of AI algorithms that can perform automatic or semi-automatic segmentation and registration of the liver and pancreas and their lesions, aid the diagnosis and characterization of pancreatic and liver focal lesions and diffuse illnesses, improve image quality, accelerate image acquisition, and anticipate treatment response and patient prognosis. Moreover, with the use of radiomics, AI can add quantitative information previously undetected by radiologists to radiological reports. The massive adoption of AI in radiology of pancreatic and liver diseases is still incipient, but irreversible, and the sector is clearly moving in this direction. Advances in the field, such as the availability of regulatory cleared, robust algorithms trained and validated multicentrically, increased awareness on AI by the medical staff, and access to products that seamlessly integrate with their workflow should pave the way for a rapid adoption of AI in the clinical practice, impacting the outcomes of hepatic and pancreatic patients for the better.