Predictive value of machine learning models for lymph node metastasis in gastric cancer: A two-center study

Abstract

BACKGROUND

Gastric cancer is one of the most common malignant tumors in the digestive system, ranking sixth in incidence and fourth in mortality worldwide. Since 42.5% of metastatic lymph nodes in gastric cancer belong to nodule type and peripheral type, the application of imaging diagnosis is restricted.

AIM

To establish models for predicting the risk of lymph node metastasis in gastric cancer patients using machine learning (ML) algorithms and to evaluate their predictive performance in clinical practice.

METHODS

Data of a total of 369 patients who underwent radical gastrectomy at the Department of General Surgery of Affiliated Hospital of Xuzhou Medical University (Xuzhou, China) from March 2016 to November 2019 were collected and retrospectively analyzed as the training group. In addition, data of 123 patients who underwent radical gastrectomy at the Department of General Surgery of Jining First People’s Hospital (Jining, China) were collected and analyzed as the verification group. Seven ML models, including decision tree, random forest, support vector machine (SVM), gradient boosting machine, naive Bayes, neural network, and logistic regression, were developed to evaluate the occurrence of lymph node metastasis in patients with gastric cancer. The ML models were established following ten cross-validation iterations using the training dataset, and subsequently, each model was assessed using the test dataset. The models’ performance was evaluated by comparing the area under the receiver operating characteristic curve of each model.

RESULTS

Among the seven ML models, except for SVM, the other ones exhibited higher accuracy and reliability, and the influences of various risk factors on the models are intuitive.

CONCLUSION

The ML models developed exhibit strong predictive capabilities for lymph node metastasis in gastric cancer, which can aid in personalized clinical diagnosis and treatment.

Key Words: Machine learning; Prediction model; Gastric cancer; Lymph node metastasis

Core Tip: The purpose of this study was to explore the performance of machine learning based models for the risk assessment of lymph node metastasis in patients with gastric cancer. We used seven different methods to analyze our data. After training, the algorithm with the highest average area under the receiver operating characteristic curve was selected as the optimal algorithm.

INTRODUCTION

Gastric cancer is one of the most common malignant tumors in the digestive system, ranking sixth in the world in incidence and fourth in mortality[1]. At present, gastric cancer typically is managed with comprehensive treatment that includes surgery. However, the overall 5-year survival rate remains below 50%[2]. In the Tumor-Node-Metastasis staging system of the American Joint Committee on Cancer, N represents the number of lymph node metastases, which is itself an independent factor in predicting the overall survival rate of gastric cancer patients[3]. However, there are some difficulties in the exploration of lymph nodes in patients with gastric cancer, such as multiple regional lymph nodes located in the abdominal cavity, which are not easy to explore preoperatively. In addition, 42.5% of metastatic lymph nodes in gastric cancer belong to nodule type and peripheral type, restricting the application of imaging diagnosis[4,5].

Artificial intelligence refers to the ability of machines to independently replicate typical human intellectual processes[6]. Artificial intelligence has various applications in the medical field, encompassing image processing, computer vision, machine learning (ML), artificial neural networks (ANNs), and convolutional neural networks (CNNs). ML can assist physicians in interpreting clinical data through the computer-aided diagnostic (CAD) system. The CAD can be categorized into three stages: Feature recognition, feature extraction, and clinical reasoning. It is feasible to feed variables related to gastric cancer lymph node metastasis into the system and develop a risk model for lymph node metastasis of gastric cancer using a more advanced ML-based algorithm[7,8]. ML algorithms play crucial roles in assisting diagnosis and predicting prognosis by processing a large amount of complex medical data[9,10]. A clinical prediction model can be proposed and optimized through the training dataset, and subsequently examined through the external validation dataset to determine its external validity and adaptability to other patients[11,12]. The clinical utility of ML within the realm of artificial intelligence is increasingly attracting clinicians’ attention, and it is also applied to help diagnose and treat various clinical diseases, including gastric cancer.

MATERIALS AND METHODS

Study subjects

A total of 369 patients who underwent radical gastrectomy at the Department of General Surgery of the Affiliated Hospital of Xuzhou Medical University (Xuzhou, China) from March 2016 to November 2019 were enrolled as the training group, and 123 patients who underwent radical gastrectomy at the Department of General Surgery of Jining First People’s Hospital (Jining, China) were enrolled as the verification group. The inclusion criteria were as follows: (1) Newly diagnosed gastric cancer patients with complete medical records; (2) radical resection for primary gastric cancer was performed in either of the two hospitals, and lymph node metastasis was confirmed by imaging and pathology; and (3) no anti-tumor therapy, such as radiotherapy or chemotherapy, was performed preoperatively. The exclusion criteria were: (1) Combination with other malignant tumors; (2) preoperative complications of other infectious diseases, blood system diseases, autoimmune diseases, and other diseases that could affect inflammatory indicators; (3) recently or currently receiving anti-inflammatory or immunosuppressive therapy; (4) preoperative blood transfusion treatment; (5) severe liver and kidney dysfunction; and (6) incomplete clinical data (Figure 1).

Figure 1 Flow chart. GBM: Gradient boosting machine.

Observational indicators

Clinical data, such as patient name, age, gender, and other clinicopathological data, including routine blood parameters, tumor location, maximum tumor diameter, depth of invasion, and the presence or absence of lymph node metastasis, were collected from all patients. Blood samples were collected in the morning on an empty stomach on the day after admission to determine neutrophil count, platelet count, monocyte count, and lymphocyte count using the Sysmex XE-2100 Automatic Blood Analyzer. Carcinoembryonic antigen (CEA) level in the blood was also measured. The pan-immune-inflammation value (PIV) and CEA level were utilized to establish clinical prediction models. PIV was calculated as (neutrophil count × platelet count × monocyte count)/lymphocyte count.

Statistical analysis

Continuous variables are expressed as the mean ± SD, and categorical variables are presented as percentages. LR was employed to identify the independent risk factors associated with lymph node metastasis in gastric cancer patients. This analysis allowed for the calculation of odds ratios (ORs) and their corresponding 95% confidence intervals. An OR greater than 1 indicated that the variable was a positive risk factor affecting the outcome, while an OR less than 1 suggested that the variable was a negative risk factor influencing the outcome. Statistical significance was defined as a P value of less than 0.05. The statistical analyses and modeling procedures were carried out using SPSS 20.0 software (IBM, Armonk, NY, United States) and R-Studio 25.0 software (R Foundation for Statistical Computing, Vienna, Austria). Several packages were utilized to train models and draw relevant graphs, with the caret package applied for training and validating ML models. In addition to the fundamental linear model (linear LR), seven ML models were fitted, including LR, random forest (RF), gradient boosting machine (GBM), decision tree (DT), support vector machine (SVM), naive Bayes (NB), and multi-layer perceptron (MLP), as illustrated in Figure 2.

Figure 2 Prediction performance evaluation of each model. SVM: Support vector machine; AUC: Area under the receiver operating characteristic curve; MLP: Multi-layer perceptron.

The training dataset was combined with the validation dataset, and seven ML algorithms were employed to establish prediction models. LR is a classification algorithm designed to establish a relationship between a feature and the probability of a specific outcome. Rather than using LR for estimating class probability, it employs S-shaped functions for modeling[13,14]. DT is primarily utilized for classification tasks. It begins at the root node to split the dataset based on the most informative feature, creating decision points that segment the data into distinct classes[15]. RF is an extension of the DT method and functions as an ensemble approach. It generates multiple DTs, with the majority vote from these trees determining the final class prediction of the model[16,17]. MLP is an ML algorithm inspired by biological neural networks. ANNs consist of interconnected nodes that communicate through connections[18,19]. SVM classifies data by defining boundaries that separate classes. The optimization process aims to maximize the margin between these class boundaries. While SVM generally outperforms LR, its computational complexity may lead to longer training time during model development[20,21]. GBM is a boosting technique that serves as a numerical optimization algorithm for constructing additive models that minimize loss functions[22,23]. NB is a straightforward classification algorithm that calculates the probability of each category’s occurrence given the item to be classified. The item is assigned to the category with the highest probability[24,25].

Performance evaluation of the models involved various metrics, including accuracy, recall, and other indicators. The primary indicator for predicting binary classification results was the area under the receiver operating characteristic curve (AUC). This metric varies from 0 to 1, with higher values signifying a superior performance. Additionally, for models with two outcomes, the area under the accuracy-recall curve was utilized, illustrating the trade-off between true accuracy and positive predictive value, and the F1 score, defined as the harmonic mean of recall and accuracy. The models underwent 10-fold cross-validation on the training dataset and then assessed for their performance on the test dataset. According to the optimal model, a network estimator was developed to facilitate disease prediction using patient data. This estimator enables surgeons to assess the risk of lymph node metastasis in gastric cancer patients.

RESULTS

Baseline clinical data in the training group and verification group

The comparison of clinical data between the two groups is presented in Table 1. Gender, age, tumor location, and surgical method exhibited no significant differences between the two groups (P > 0.05). In the training dataset, the proportion of patients with total gastrectomy, neurovascular invasion, and maximum tumor diameter > 5 cm was significantly higher in patients with lymph node metastasis than in those without (P < 0.05). In the verification dataset, the number of patients who were aged > 60 years old and had neurovascular invasion and maximum tumor diameter > 5 cm was significantly greater in patients with lymph node metastasis than in those without (P < 0.05).

Table 1 Comparison of clinical data between the two groups.

Clinical data	Training set		t/Z/χ2	P value	Validation set		t/Z/χ2	P value
	No lymph node metastasis (n = 141)	Lymph node metastasis (n = 228)			No lymph node metastasis (n = 51)	Lymph node metastasis (n = 72)
Gender			1.017	0.313			1.126	0.289
Male	99 (70.2)	171 (75.0)			33 (64.7)	53 (73.6)
Female	42 (29.8)	57 (25.0)	0		18 (35.3)	19 (26.4)
Age (yr)			0.015	0.901			4.729	0.030
≤ 60	64 (45.4)	105 (46.1)			27 (52.9)	24 (33.3)
> 60	77 (54.6)	123 (53.9)			24 (47.1)	48 (66.7)
Mode of operation			7.816	0.005			3.578	0.059
Partial gastrectomy	113 (80.1)	152 (66.7)			43 (84.3)	50 (69.4)
Total gastrectomy	28 (19.9)	76 (33.3)			8 (15.7)	22 (30.6)
Tumor invasion depth			-11.022	< 0.001			-7.114	< 0.001
T1	61 (43.3)	13 (5.7)			30 (58.8)	4 (5.6)
T2	42 (29.8)	22 (9.6)			13 (25.5)	13 (18.1)
T3	21 (14.9)	64 (28.1)			6 (11.8)	27 (37.5)
T4	17 (12.1)	129 (56.6)			2 (3.9)	28 (38.9)
Tumor site			0.716	0.699			0.392	0.822
Gastric body	24 (17.0)	32 (14.0)			18 (35.3)	22 (30.6)
Gastric antrum	73 (51.8)	126 (55.3)			26 (51.0)	38 (52.8)
Gastric cardia	44 (31.2)	70 (30.7)			7 (13.7)	12 (16.7)
Nerve or vascular invasion			128.649	< 0.001			54.772	< 0.001
No	108 (76.6)	39 (17.1)			42 (82.4)	11 (15.3)
Yes	33 (23.4)	189 (82.9)			9 (17.6)	61 (84.7)
Maximum tumor diameter			38.634	< 0.001			8.323	0.004
≤ 5 cm	122 (86.5)	126 (55.3)			46 (90.2)	49 (68.1)
> 5 cm	19 (13.5)	102 (44.7)			5 (9.8)	23 (31.9)
PIV	132.00 (80.73, 226.80)	190.72 (106.49, 311.44)	-3.606	< 0.001	149.43 (91.73, 217.49)	173.59 (102.20, 274.73)	-1.586	0.113
CEA	2.47 (1.53, 3.58)	2.90 (1.82, 6.87)	-3.189	0.001	2.65 (1.47, 3.95)	4.91 (1.97, 9.02)	-2.331	0.020

PIV: Pan-immune-inflammation value; CEA: Carcinoembryonic antigen.

The results of Mann-Whitney U test revealed that there were no statistically significant differences in the depth of infiltration, PIV, or CEA level between the two groups (P > 0.05). It was found that the depth of infiltration and CEA level in patients with lymph node metastasis were significantly higher than those in patients without (P < 0.05). In the training dataset, the infiltration depth, PIV, and CEA level in patients with lymph node metastasis were significantly greater than those in patients without (P < 0.05).

Evaluation of predictive performance of each model

In order to compare the predictive performance of the seven ML-based models, this study employed ten-fold cross-validation and utilized the AUC value, validated on the test dataset, as the primary metric for assessing their performance. As shown in Table 2 and Figure 2, the GBM model exhibited the best performance in predicting the occurrence of lymph node metastasis in gastric cancer patients, with an average AUC of 0.927. In this study, a web-based online estimator, along with feature importance (Figure 3) and Shapley Additive Explanations (SHAP) summary plot (Figure 4), was developed based on the GBM model. Feature importance enables the visualization of the model’s internal results, highlighting the significance of specific variables within the model. Utilizing the optimal GBM model, we have developed a web-based risk calculator (https://gastric.shinyapps.io/gbm4Lymph). By entering the clinical characteristics of patients with gastric cancer and lymph node metastasis, healthcare professionals can predict the risk of lymph node development in these patients (Figure 5).

Figure 3 Feature importance. SHAP: Shapley Additive Explanations; CEA: Carcinoembryonic antigen; PIV: Pan-immune-inflammation value.

Figure 4 Shapley Additive Explanations summary plot. SHAP: Shapley Additive Explanations; CEA: Carcinoembryonic antigen; PIV: Pan-immune-inflammation value.

Figure 5 Web risk calculator. SHAP: Shapley Additive Explanations; CEA: Carcinoembryonic antigen; PIV: Pan-immune-inflammation value.

Table 2 Prediction performance evaluation of each model.

Model	AUC	Accuracy	Kappa	Sensitivity (recall rates)	Specificity
DT	0.824	0.821	0.638	0.806	0.843
RF	0.923	0.854	0.702	0.847	0.882
SVM	0.721	0.585	0.000	0.750	0.547
GBM	0.927	0.870	0.734	0.875	0.863
NB	0.914	0.821	0.640	0.861	0.843
MLP	0.907	0.837	0.665	0.882	0.824
LR	0.898	0.821	0.636	0.806	0.882

AUC: Area under the receiver operating characteristic curve; DT: Decision tree; RF: Random forest; SVM: Support vector machine; GBM: Gradient boosting machine; NB: Naive bayes; LR: Logistic regression; MLP: Multi-layer perceptron.

DISCUSSION

As a result of the limited early detection of gastric cancer, over 50% of patients are diagnosed at advanced stages or with metastasis. At present, surgery is the main method for the treatment of gastric cancer, and lymph node metastasis is regarded as the main factor affecting the stage, grade, and survival rate of gastric cancer[26,27]. Therefore, early prediction of the occurrence of lymph node metastasis is vital. To date, several scholars have concentrated on lymph node metastasis in gastric cancer, while few studies have developed tools to provide accurate predictions. Therefore, the development of precise predictive models is essential to facilitate collaborative decision-making for clinicians and patients. The continuous advancement of artificial intelligence in the field of clinical research has led to the introduction of innovative approaches.

ML represents an evolving frontier in the field of medicine, drawing substantial resources to connect computer science and statistical analysis with medical challenges. ML has the capacity to effectively handle extensive, diverse, and intricate medical data. Consequently, the implementation of ML techniques in medicine is widely regarded as the cornerstone of future endeavors in biomedical research, personalized medicine, and computer-aided diagnosis[28,29]. Specifically, the operational framework of ML involves development of algorithms to execute numerous tasks, refining the algorithms iteratively to optimize performance. Ultimately, this process yields a model that establishes connections between multiple variables and target outcomes. In the present study, clinical data were collected, and ML algorithms were employed to develop a model for assessing the risk of lymph node metastasis in gastric cancer. By leveraging multiple variables, clinicians can employ this AI-driven approach to select more efficacious treatment strategies[30-33].

In this study, in addition to some clinicopathological data, hematological indicators, namely, immunoinflammatory factors (PIV and CEA), were utilized to develop the prediction models. PIV is a novel blood-based biomarker that integrates different subsets of peripheral blood immune cells, neutrophils, platelets, monocytes, and lymphocytes. As PIV has the potential to comprehensively represent patients’ immunity and systemic inflammation, it may potentially serve as a robust predictor in advanced cancer patients undergoing cytotoxic chemotherapy, immunotherapy, and targeted therapy. It has been previously demonstrated that PIV is mainly dependent on neutrophil-to-lymphocyte ratio, platelet-to-lymphocyte ratio, lymphocyte-to-monocyte ratio, and other indicators in predicting cancer prognosis[34,35]. CEA is a widely used serum tumor marker in clinical practice, particularly in the early screening of various types of cancer, and elevation of its elevation is also regarded as an independent risk factor for poor prognosis of gastric cancer[36]. Development of a model based on combination of clinicopathological data with hematological suggestions can better reflect the physiological and pathological changes of patients with gastric cancer during the disease, making the model more representative.

Using ML, seven models were established for comparative analysis, utilizing the AUC as the benchmark for assessment. The outcomes are summarized as follows: The AUC for the DT model was 0.824, the RF model yielded an AUC of 0.923, the AUC for SVM was 0.721, and the GBM model demonstrated an AUC of 0.927. The NB model’s AUC stood at 0.914, while the NNET model’s AUC reached 0.907. The results of the seven models indicated that the GBM model displayed the most reliable performance, while SVM exhibited the least promising results. Furthermore, a feature importance table was developed based on the highly effective GBM model, which highlighted that factors, such as nerve or vascular invasion, CEA level, maximum tumor diameter, PIV, age, and tumor site, were significant contributors to the occurrence of lymph node metastasis.

Using the best-performing GBM model, feature importance assessment was conducted. This analysis highlighted the significance of specific indicators within the model, providing new insights into the model’s structure. To understand the relationship between the direction of lymph node metastasis in gastric cancer and the importance of its main predictors, a SHAP summary plot was drawn. This method was utilized to explain the predictions of ML models. SHAP-Beeswarm diagrams, a common visualization tool in SHAP method, display the effect of each feature on the predicted results. The horizontal axis of the plot represents the SHAP value, indicating the contribution of each feature to the predicted result, while the vertical axis represents the feature name. Each data point in the diagram represents a sample, with its horizontal position indicating the sample’s influence on the prediction result. Data points closer to the left side of the graph negatively impact the result, while those closer to the right side positively impact the result. The vertical position of the data point represents the feature name, with each feature having a corresponding vertical position.

According to the optimal GBM model, a web-based risk calculator was developed. By inputting patients’ clinical characteristics, it can directly predict the probability of lymph node metastasis in patients with gastric cancer. This tool is user-friendly and straightforward, making it accessible for healthcare practitioners. It serves as a valuable resource in diagnosis and treatment, providing significant support for clinicians.

CONCLUSION

In summary, based on the clinicopathological data of 492 gastric cancer patients in two centers, ML algorithms were utilized to establish clinical models and conduct cross-validation, and AUC values were finally compared to draw conclusions. In addition to SVM, other ML models have exhibited promising accuracy and reliability, as well as better predictive value for gastric cancer lymph node metastasis. Among them, GBM outperformed the others, with the highest predictive value and accuracy. This study demonstrated that ML could reveal the potential of clinical data to reflect disease conditions, thereby assisting clinicians in evaluating patients’ conditions and making more informed treatment decisions.

ARTICLE HIGHLIGHTS

Research background

Gastric cancer is one of the most common malignant tumors of the digestive system, ranking sixth in incidence and fourth in mortality worldwide. Machine learning (ML) represents an evolving frontier in the field of medicine, drawing substantial resources to connect computer science and statistical analysis with medical challenges. ML has the capacity to effectively handle extensive, diverse, and intricate medical data. Consequently, the implementation of ML techniques in medicine is widely regarded as the cornerstone of future endeavors in biomedical research, personalized medicine, and computer-aided diagnosis.

Research motivation

Using machine learning-based models to predict lymph node metastasis of gastric cancer is helpful to individualized diagnosis and treatment of gastric cancer patients.

Research objectives

Based on the clinicopathological data of 492 gastric cancer patients in two centers, we used ML algorithms to establish clinical models and conduct cross-validation, and finally compared the area under the receiver operating characteristic curve to draw conclusions. In addition to support vector machine, other ML models have good accuracy and reliability, and have better predictive value for gastric cancer lymph node metastasis. Among them, gradient boosting machine (GBM) has the best performance and the highest predictive value and accuracy. Through this study, ML can dig out the ability of clinical data to reflect disease, which can help clinicians evaluate patients' conditions and make better treatment decisions.

Research methods

Seven machine algorithm models were built with data from two centers, and then their performance was evaluated. Based on GBM model, a web-based online estimator and Shapley Additive Explanations summary plot were established.

Research results

ML can tap into the ability of clinical data to reflect disease, which can help clinicians assess patients' conditions and make better treatment decisions.

Research conclusions

ML algorithms have been used to establish an optimal prediction model for lymph node metastasis in gastric cancer, which is helpful for clinical risk stratification and individualized diagnosis and treatment of gastric cancer patients.

Research perspectives

In the future, multi-center data are needed to verify the external applicability of our model.