Deep learning based radiomics for gastrointestinal cancer diagnosis and treatment: A minireview

Table 1 Summary of studies using deep-learning-based radiomics for esophageal cancer

Ref.	Imaging	Study design	Study aim	DL model	Dataset	Outcomes
Takeuchi et al[23], 2021	CT	Retrospective	Detection of esophageal cancer	VGG16	1646 CT images (1500 images for training and validation, 146 for testing)	Accuracy: 84.2%; F value: 74.2%; Sensitivity: 71.1%; Specificity: 90%) in test set
Hu et al[24], 2021	CT	Retrospective	Evaluation of response to NCRT to ESCC	ResNet50	231 patients (161 in training cohort, 70 in testing cohort)	AUC: 0.805; C-index: 0.805; Accuracy: 77.1%; Sensitivity: 83.9%; Specificity: 71.8%) for the testing cohort
Ypsilantis et al[25], 2015	PET	Retrospective	Prediction of response to NAC in patients with esophageal cancer	3S-CNN	107 patients	Sensitivity: 80.7%; Specificity: 81.6%; Accuracy: 73.4%
Amyar et al[26], 2019	PET	Retrospective	Prediction of response to radio-chemotherapy in patients with esophageal cancer	3D RPET-NET	97 patients	Accuracy: 75.0%; Sensitivity: 76.0%; Specificity: 74.0%; AUC: 0.74
Li et al[27], 2021	CT	Retrospective	Prediction of treatment response to CCRT among patients with locally advanced TESCC	ResNet34	306 patients (203 in training cohort, 103 in validation cohort)	AUC: 0.833; PPV: 100%
Wang et al[28], 2022	CT	Retrospective	Prediction of survival rates for patients with esophageal cancer after 3 yr with chemoradiotherapy	DenseNet- 169	154 patients (116 in training cohort, 38 in validation cohort)	AUC: 0.942; C-index: 0.784
Yang et al[29], 2019	PET	Retrospective	Identification of esophageal cancer patients with poor prognosis	3D-CNN based on ResNet18	1107 scans	AUC: 0.738
Gong et al[30], 2022	CECT	Retrospective	Prediction of LRFS in esophageal cancer patients after 1 yr of definitive chemoradiotherapy	3D-Densenet	397 patients	C-index: 0.76
Wu et al[31], 2019	CT	Retrospective	Prediction of LN status of patients with ESCC	CNN-F	411 patients	C-index: 0.840

CT: Computed tomography; PET: Positron emission tomography; CECT: Contrast-enhanced computed tomography; NCRT: Neoadjuvant chemoradiotherapy; NAC: Neoadjuvant chemotherapy; ESCC: Esophageal squamous cell carcinoma; CCRT: Concurrent chemo-radiation therapy; TESCC: Thoracic esophageal squamous cell carcinoma; LRFS: Local recurrence-free survival; LN: Lymph node; CNN-F: Convolutional neural network-fast; AUC: Area under the receiver operating characteristic curve; PPV: Positive predictive value; DLR: Deep-learning-based radiomics.

Table 2 Summary of studies using deep-learning-based radiomics for gastric cancer

Ref.	Imaging	Study design	Study aim	DL model	Dataset	Outcomes
Cui et al[32], 2022	CT	Retrospective	Prediction of response to NAC in patients with LAGC	DenseNet-121	719 patients	C-index: 0.829
Li et al[33], 2022	CT	Retrospective	Diagnosis and prediction of chemotherapy response to SRCC patients	Modified U-Net	855 patients (598 in training cohort; 257 in testing cohort)	For diagnosis, AUC: 0.786; accuracy: 71.6%; sensitivity: 77.3%; specificity: 69.2% for testing cohort
Tan et al[34], 2020	CT	Retrospective	Prediction of response to chemotherapy in patients with gastric cancer	V-Net	116 patients	Mean AUC: 0.728 (testing cohort); 0.828 (validation cohort) when using semi-segmentation
Hao et al[35], 2021	CT	Retrospective	Prediction of OS and PFS after gastrectomy; evaluation of effects of variables on survival prediction	Attention-guided VAE	1061 patients (743 for training; 318 for testing)	C-index of OS: 0.783; C-index of PFS: 0.770 when only using postoperative variables
Zhang et al[36], 2021	CT	Retrospective	Prediction of OS risks of patients with gastric cancer	MMF-FPN	640 patients (337 in training set; 181 in validation set; 122 in test set)	C-index: 0.76; hazard ratio: 9.46 in test set
Zhang et al[37], 2020	CT	Retrospective	Prediction of early recurrence of patients with AGC	DCNNs	669 patients	AUC: 0.806; accuracy: 0.723; sensitivity: 0.827; specificity: 0.667
Guan et al[38], 2022	CT	Retrospective	Prediction of preoperative status of LNM of gastric cancer patients	ResNet50-RF	347 patients (242 for training; 105 for testing)	AUC: 0.9803; accuracy: 98.10%; sensitivity: 98.39%; specificity: 0.9767% for testing of ResNet50-RF
Dong et al[39], 2020	CT	Retrospective	Prediction of the number of LNM in LAGC	DenseNet-201	730 patients	C-index: 0.822 in validation set
Li et al[40], 2020	CT	Retrospective	Prediction of LNM and prognosis in gastric cancer patients	DCNNs	204 patients (136 in training set, 68 in test set)	AUC: 0.82 in test set; C-index of OS: 0.67; C-index of PFS: 0.64
Jin et al[41], 2021	CT	Retrospective	Prediction of LNM status in LN stations of gastric cancer patients	ResNet-18	1699 patients	Median AUC: 0.876; median Sensitivity: 0.743; median Specificity: 0.936 in validation cohort
Sun et al[42], 2020	CT	Stage I: Retrospective; stage II: Validation	Prediction of serosa invasion of AGC patients	DCNNs	572 patients (252 in training set, 176 in test set I, 144 in test set II)	AUC: 0.87; accuracy: 80%; sensitivity: 0.73; specificity: 0.85 in test set I. AUC: 0.90; accuracy: 85%; sensitivity: 0.75; specificity: 0.93 in test set II
Li et al[43], 2022	CT	Retrospective	Evaluation of lymphovascular invasion of localized gastric cancer patients	SqueezeNet, ResNet50, Inception V3, VGG19, DeepLoc	1062 patients (728 for training, 334 for testing)	AUC: 0.725; sensitivity: 73.2%; specificity: 60.3%; accuracy: 71.0% for radiomics GRISK model (final model) in testing cohort

CT: Computed tomography; NAC: Neoadjuvant chemotherapy; LAGC: Locally advanced gastric cancer; SRCC: Signet ring cell carcinoma; OS: Overall survival; PFS: Progression-free survival; AGC: Advanced gastric cancer; LNM: Lymph node metastasis; LN: Lymph node; DCNNs: Deep convolutional neural networks; MMN-FPN: Multi-focus and multi-level fusion feature pyramid network; UC: Area under the receiver operating characteristic curve; C-index: Harrell’s concordance index; GRISK: Gastric Risk; RF: Random forest; VAE: Variational auto-encoder; DLR: Deep-learning-based radiomics.

Table 3 Summary of studies using deep-learning-based radiomics for liver cancer

Ref.	Imaging	Study design	Study aim	DL model	Dataset	Outcomes
Ding et al[44], 2021	CT	Retrospective	Evaluation of HCC differentiation	VGG19	1234 patients (799 in training cohort, 248 in validation cohort; 187 in independent testing cohort)	AUC: 0.8042; accuracy: 72.73%; sensitivity: 70.75%; specificity: 75.31% in testing cohort for the fused DLRs model
Peng et al[45], 2020	CT	Retrospective	Prediction of different treatment responses to TACE in HCC patients	ResNet50	789 patients (562 in training cohort; 89 and 138 in validation cohorts 1 and 2)	Accuracy: 85.1% in validation cohort 1; accuracy: 82.8% in validation cohort 2
Peng et al[46], 2021	CT	Retrospective	Prediction of initial response to TACE in HCC patients	CNN	310 patients (139 in training cohort; 171 in validation cohort)	AUC: 0.994
Wei et al[47], 2021	CT	Retrospective	Prediction of OS of HCC patients treated with SBRT	CNN	167 patients	C-index: 0.650 in cross validation
Liu et al[48], 2020	US	Retrospective	Prediction of PFS of HCC patients treated with RFA or surgical resection	CNN	214 RFA patients (149 for training; 65 for validation), 205 SR patients (144 for training; 61 for validation)	C-index of RFA: 0.726; C-index of surgical resection: 0.726
Wang et al[49], 2019	CT	Retrospective	Prediction of early recurrence of HCC patients	ResNet	167 patients	AUC of best model: 0.825
Wang et al[50], 2020	CT	Retrospective	Prediction of early recurrence of HCC patients	ResNet	167 patients	For the best model with joint loss function, AUC: 0.8331; accuracy: 80.49%
He et al[51], 2021	MRI and pathological data	Retrospective	Evaluation of HCC recurrence risk of liver transplantation recipients	U-net, CapsNet	109 patients (87 for training; 22 for testing)	Total accuracy: 82%; recall: 80%; precision: 89%; AUC: 0.87; F-1 score: 84%
Jiang et al[52], 2021	CT	Retrospective	Prediction of microvascular invasion status of HCC patients	3D-CNN	405 patients (324 in training set, 81 in validation set)	AUC: 0.906; sensitivity: 75.7%; specificity: 93.2%; accuracy: 85.2%; F-1 score: 87.2% in validation set
Wang et al[53], 2022	CT	Retrospective	Prediction of microvascular invasion status of HCC patients	Transformer, CNN	138 patients	For arterial phase images in validation set, AUC: 0.9223; Average accuracy: 86.78%
Fu et al[54], 2021	CT	Retrospective	Prediction of macrovascular invasion status in HCC patients	Modified U-Net	366 patients (281 in training cohort, 85 in validation cohort)	AUC: 0.836 in validation cohort

CT: Computed tomography; MRI: Magnetic resonance imaging; US: Ultrasound; HCC: Hepatocellular carcinoma; TACE: Transarterial chemoembolization; PFS: Progression-free survival; RFA: Radiofrequency ablation; SBRT: Stereotactic body radiation therapy; CNN: Convolutional neural network; AUC: Area under the receiver operating characteristic curve; C-index: Harrell’s concordance index; DLR: Deep-learning-based radiomics.

Table 4 Summary of studies using deep-learning-based radiomics for pancreatic cancer

Ref.	Imaging	Study design	Study aim	DL model	Dataset	Outcomes
Ziegelmayer et al[55], 2020	CT	Retrospective	Identification of PDAC and AIP	VGG19	86 patients (44 AIP patients and 42 PDAC patients)	Sensitivity: 89%; specificity: 83%; AUC: 0.90
Liao et al[56], 2022	CT	Retrospective	Identification of PDAC, non-cancerous pancreatic diseases and normal pancreas	CNN	3120 images (1872 for training, 624 for validation, 624 for testing)	Sensitivity: 89.9%; specificity: 91.3%; AUC: 0.960 when distinguishing between PDAC and control group
Tong et al[57], 2022	US	Retrospective	Identification of PDAC and CP	ResNet-50	558 patients	AUC: 0.967; sensitivity: 87.2%; specificity: 100%
Watson et al[58]	CT	Retrospective	Prediction of pathologic response of PDAC patients to NAC	LeNet	81 patients (65 for training and validation; 16 for testing)	AUC: 0.785; brier score: 0.174; sensitivity: 81.4%; specificity: 60.4% in test set of hybrid deep learning model
Muhammad et al[59], 2018	CT	Retrospective	Evaluation of survival hazard of PDAC patients	AlexNet	159 patients	C-index: 0.76; hazard ratio: 9.46 in test set
Zhang et al[60], 2020	CT	Retrospective	Evaluation of survival probability of PDAC patients	CNN	520 patients	IPA: 11.81%, C-index: 0.651 in testing cohort
Zhang et al[61], 2020	CT	Retrospective	Prediction of OS of PDAC patients; Evaluation of risk scores to distinguish patients with high or low risk	CNN	98 patients (68 in training cohort; 30 in testing cohort)	AUC: 0.81; hazard ratio: 1.86
Zhang et al[62], 2021	CT	Retrospective	Prediction of 2-yr OS of resectable PDAC patients	CNN	98 patients (68 in training cohort; 30 in testing cohort)	AUC: 0.84; specificity: 68%; sensitivity: 91%
Yao et al[63], 2021	CT	Retrospective	Prediction of survival risk and tumor resection margin of resectable PDAC patients	CNN	205 patients	C-index: 0.705 for survival prediction; balanced accuracy: 73.6%, sensitivity: 81.3%, specificity: 65.9% for resection margin prediction
Yao et al[63], 2021	CT	Retrospective	Prediction of survival risk and tumor resection margin of resectable PDAC patients	CNN	1209 patients	C-index: 0.667 for survival prediction; balanced accuracy: 67.1%; sensitivity: 59.8%; specificity: 74.3% for resection margin prediction
An et al[64], 2022	CT	Retrospective	Prediction of LNM status and OS in PDAC patients	ResNet-18	148 patients (88 in training cohort, 25 in validation cohort, 35 in testing cohort)	For combined model, AUC: 0.92; accuracy: 86%; sensitivity: 94%; specificity: 78% in testing cohort
Li et al[65], 2019	CT	Retrospective	Prediction of HMGA2 and C-MYC gene expression status of PDAC patients;Prediction of survival time of patients	CNN	111 patients	Average AUC score: 0.90; accuracy: 95%; sensitivity: 92%; specificity: 98% in C-MYC test with deep features selected by Doctor B; average AUC score: 0.91; accuracy: 88%; sensitivity: 89%; specificity: 88% in HMGA2 test with deep features selected by Doctor B

CT: Computed tomography; US: Ultrasound; PDAC: Pancreatic ductal adenocarcinoma; AIP: Autoimmune pancreatitis; CP: Chronic pancreatitis; NAC: Neoadjuvant chemotherapy; LNM: Lymph node metastasis; OS: Overall survival; AUC: Area under the receiver operating characteristic curve; C-index: Harrell’s concordance index; CNN: Convolutional neural network; IPA: Index of prediction accuracy; DLR: Deep-learning-based radiomics.

Table 5 Summary of studies using deep-learning-based radiomics for colorectal cancer

Ref.	Imaging	Study design	Study aim	DL model	Dataset	Outcomes
Wu et al[67], 2020	CT	Retrospective	Predicting KRAS status in patients with CRC	CNN	Primary cohort: 279 patients; validation cohort: 119 patients	C-index of 0.815 for the primary cohort and 0.832 for the validation cohort
Wei et al[68], 2021	CT	Retrospective	Predicting the response to chemotherapy in CRLM	ResNet10	192 patients	AUC of DLR: 0.820; AUC of HCR: 0.598
Zhang et al[69], 2021	MRI	Retrospective	Predicting the MSI status of CRC	MobileNetV2	491 patients	Accuracy: 85.4%; AUC: 0.868
Fu et al[70], 2020	MRI	Retrospective	Predicting NCRT response in patients with LARC	VGG19	43 patients	AUC of DLR: 0.73; AUC of HCR: 0.64
Liu et al[71], 2021	MRI	Retrospective	Predicting the distant metastasis of LARC patients receiving NCRT	ResNet18	235 patients	C-index of 0.747 and AUC of 0.894 in the validation cohort
Lu et al[72], 2021	CT	Retrospective	Prediction of early on-treatment response in mCRC	CNN + RNN	1028 patients	C-index: 0.649
Ding et al[73], 2020	MRI	Retrospective	Prediction of metastatic LN in CRC	Faster RCNN	545 patients	AUC for training: 0.862; AUC for validation: 0.920
Zhao et al[74], 2022	CT	Retrospective	Prediction of metastatic LN in CRC	Autoencoder	423 patients	AUC for training: 0.81; AUC for validation: 0.73; AUC for testing: 0.77
Li et al[75], 2020	MRI	Retrospective	Classification of CRC LN Metastasis images	AlexNet	3364 samples (1646 positive; 1718 negative)	Accuracy: 75.83%; AUC: 0.7941

CRC: Colorectal cancer; CT: Computed tomography; MRI: Magnetic resonance imaging; CRLM: Colorectal liver metastases; MSI: Microsatellite instability; NCRT: Neoadjuvant chemoradiotherapy; LARC: Locally advanced rectal cancer; mCRC: Metastatic colorectal cancer; LN: Lymph node; CNN: Convolutional neural network; RNN: Recurrent neural network; AUC: Area under the receiver operating characteristic curve; C-index: Harrell’s concordance index; DLR: Deep-learning-based radiomics; HCR: Handcrafted radiomics; Faster R-CNN: Faster region-based convolutional neural network.