Published online Nov 21, 2023. doi: 10.3748/wjg.v29.i43.5804
Peer-review started: August 26, 2023
First decision: September 18, 2023
Revised: October 7, 2023
Accepted: November 3, 2023
Article in press: November 3, 2023
Published online: November 21, 2023
Surgical resection is the primary treatment for hepatocellular carcinoma (HCC). However, studies indicate that nearly 70% of patients experience HCC recurrence within five years following hepatectomy. The earlier the recurrence, the worse the prognosis. Current studies on postoperative recurrence primarily rely on posto
To identify key variables in pre-operative clinical and imaging data using ma
The demographic and clinical data of 371 HCC patients were collected for this retrospective study. These data were randomly divided into training and test sets at a ratio of 8:2. The training set was analyzed, and key feature variables with predictive value for early HCC recurrence were selected to construct six different machine learning prediction models. Each model was evaluated, and the best-performing model was selected for interpreting the importance of each variable. Finally, an online calculator based on the model was generated for daily clinical practice.
Following machine learning analysis, eight key feature variables (age, intratumoral arteries, alpha-fetoprotein, pre-operative blood glucose, number of tumors, glucose-to-lymphocyte ratio, liver cirrhosis, and pre-operative platelets) were selected to construct six different prediction models. The XGBoost model outperformed other models, with the area under the receiver operating characteristic curve in the training, validation, and test datasets being 0.993 (95% confidence interval: 0.982-1.000), 0.734 (0.601-0.867), and 0.706 (0.585-0.827), respectively. Calibration curve and decision curve analysis indicated that the XGBoost model also had good predictive performance and clinical application value.
The XGBoost model exhibits superior performance and is a reliable tool for predicting early postoperative HCC recurrence. This model may guide surgical strategies and postoperative individualized medicine.
Core Tip: The current study aimed at employing machine learning techniques to select imaging and pre-operative clinical characteristic variables, to which the clinicians were easily accessible, to develop six different risk prediction models for early postoperative recurrence of hepatocellular carcinoma (HCC). We compared the sensitivity and specificity of these models in detecting patients at high risk of early postoperative recurrence of HCC. In addition, to increase the feasibility and applicability of the constructed model, we generated a calculator online based on the predictive model to help clinicians apply it in their daily medical practice.
- Citation: Zhang YB, Yang G, Bu Y, Lei P, Zhang W, Zhang DY. Development of a machine learning-based model for predicting risk of early postoperative recurrence of hepatocellular carcinoma. World J Gastroenterol 2023; 29(43): 5804-5817
- URL: https://www.wjgnet.com/1007-9327/full/v29/i43/5804.htm
- DOI: https://dx.doi.org/10.3748/wjg.v29.i43.5804
Liver cancer is one of the most common malignant tumors globally, and its incidence ranks 5th in various malignant tumors. According to statistics, there were about 905700 new cases of liver cancer worldwide in 2020, and about 830200 patients with liver cancer died. The incidence and mortality of liver cancer are expected to increase by > 50% in 2040[1]. Hepatocellular carcinoma (HCC) is the primary type of liver cancer. Currently, surgical resection is still the main treatment for HCC. However, nearly 70% of patients experience HCC recurrence within five years after hepatectomy[2,3], which is the main factor affecting patients' survival rate after surgery. HCC recurrence can be classified into early (≤ 2 years) or late recurrence (> 2 years after HCC hepatectomy). A study has shown that the earlier the recurrence of HCC, the worse the prognosis, and the overall survival rate of patients with early recurrence of HCC tends to be lower than those with late recurrence[4]. For patients with recurrent HCC, radical treatment is more effective than palliative therapies in prolonging their survival[5,6]. For patients at high risk of potentially postoperative recurrence, neoadjuvant therapy, appropriate intraoperative margin expansion, and early postoperative use of targeted therapies and immunotherapies may improve their prognosis[7]. However, the lack of specific tumor biomarkers and clinical features makes it challenging to identify the patients at high risk of postoperative recurrence before surgery. Therefore, developing pre-operative non-invasive predictive methods will be highly significant in identifying patients at high risk of postoperative recurrence and precise management of those patients by closely monitoring and individualized treatment on time.
Currently, there are many prediction models to predict the early recurrence of HCC in the clinic[8-10]. However, these models are usually constructed based on linear assumptions, which may not model the complex, multidimensional, and nonlinear relationships among different predictor variables that may exist. Their predictive value is greatly limited[11,12]. In recent years, machine learning has great potential for research in the field of disease prediction because machine learning can better capture the relationships between nonlinearities, deeply mine the hidden relationships in massive data through various algorithms, and build treatment models. These models can efficiently and accurately predict the prognosis of diseases and guide clinical treatment. Previous studies have used artificial intelligence techniques and machine learning algorithms to extract imaging features from pre-operative computed tomography (CT) or magnetic resonance imaging (MRI) images and clinical data to predict the risk of liver cancer recurrence[13,14]. Although these studies make superior predictions, these constructed models remain not highly interpretable and could not be widely applied in most medical scenarios, a common problem with machine learning research.
This study aimed to utilize machine learning techniques to select easily accessible imaging and pre-operative clinical characteristic variables. The goal was to develop six different risk prediction models for the early postoperative re
The demographic and clinical data of age, gender, pathology, radiological images, hepatectomy, and postoperative follow-up of 578 patients with primary liver cancer at Ningxia Medical University General Hospital from January 2017 to June 2021 were retrospectively collected according to the inclusion and exclusion criteria. After excluding those with non-primary liver cancer, incomplete data, receiving neoadjuvant therapy, or liver cancer rupture, the remaining cases were selected for this study, randomized by a simple number generated with a computer, and divided into training and test sets at a ratio of 8:2. The training set of data was used to construct the machine learning model, and the test set of data was used to validate the model performance. The study was approved by the Ethics Committee of the General Hospital of Ningxia Medical University and individual patients provided written informed consent. The study was performed per the Declaration of Helsinki, and the study design followed the TRIPOD statement[15] (Multifactorial predictive reporting norms for individual prognosis or diagnosis). The process of this study is shown in Figure 1.
The inclusion criteria included: (1) Postoperatively pathologically confirmed primary HCC (the first treatment option was radical hepatectomy); (2) undergoing pre-operative abdominal contrast-enhanced MRI examinations; and (3) complete clinical, pathological, and imaging data and postoperative follow-up. The exclusion criteria were: (1) Posto
The early liver cancer recurrence was defined according to a previous report[16]. Briefly, a new tumor appeared in the liver within two years after surgical resection of the primary liver cancer. The new tumors were evaluated by radiological imaging or pathological examination. The time to recurrence was defined as the period from the surgical resection of primary liver cancer to the appearance of a new tumor in the liver.
Patients were followed postoperatively in our outpatient service from monthly to every 3 mo in the first-year post-surgical resection of the primary liver cancer. The patients were tested for serum liver functions and alpha-fetoprotein (AFP) levels, chest CT scanning, and abdominal contrast enhanced CT or MRI. Subsequently, the patients who survived were followed every 6 mo starting from the second year after surgery. This follow-up continued until April 2023 or until tumor recurrence, loss of patient follow-up, or death.
The characteristic variables included: (1) Patient’s demographic characteristics and pre-operative laboratory tests, such as gender, age, AFP, blood platelet count (PLT), blood glucose, HBeAg, HBsAg, monocyte to lymphocyte ratio (MLR), γ-glutamyl transferase (GGT) to lymphocyte ratio (GLR), PLT to lymphocyte count ratio (PLR), alkaline phosphatase (ALP) to lymphocyte count ratio (ALR), lymphocyte to monocyte count ratio (LMR), fibrinogen to albumin level ratio (FAR), systemic immune-inflammation index (SII; neutrophil count × PLT/lymphocyte count), ALP, and albumin-bilirubin (ALBI) score; and (2) pre-operative imaging features (abdominal contrast-enhanced MRI): Tumor size, tumor number, homogeneous tumor enhancement, intratumoral necrosis, tumor envelope, whether tumor margin is regular, intratumoral arteries, tumor adjacency to major vessels, ascitic fluid, and liver cirrhosis (Figure 2).
The data in the training set were analyzed using the receiver operating characteristic (ROC) curve to determine the best cut-off points for the continuous variable values GLR, PLR, ALR, LMR, FAR, and SII to divide the patients into two groups (as shown in Figure 3), where the best cut-off value was 0.285 [95% confidence interval (95%CI): 0.482-0.601] for MLR; 32.16 (95%CI: 0.526- 0.642) for GLR; 138.18 (95%CI: 0.494-0.61) for PLR; 80.29 (95%CI: 0.465-0.583) for ALR; 5.33 (95%CI: 0.40-0.517) for LMR; 0.07 (95%CI: 0.484-0.6) for FAR; and 312.71 (95%CI: 0.507-0.624) for SII in this population. Z-score normalization of continuous variables was performed. When the proportion of missing values was less than 10%, the mean value was used for interpolation, and the case was deleted when the proportion of missing values of the variable was greater than 10%.
All statistical analyses were performed using the R 4.0.2 (R Foundation for Statistical Computing, Vienna, Austria) and Python language. Categorical variables are expressed as n (%) and were analyzed by the chi-square test or Fisher's exact test. Continuous variables are expressed as the mean ± SD or the median and interquartile range and were analyzed by the independent-sample t-test or the Mann–Whitney U test based on their distribution. A P value of < 0.05 was considered statistically significant.
The most significant feature variables were chosen from the intersection of the three methods used for important feature analysis. Six different machine learning models were constructed based on these feature sets. The hyperparameters for each model were determined through a grid search to select the best parameters and perform a five-fold cross-validation on the training data set. Cross-validation provided a more accurate evaluation of model performance through metrics from multiple experiments. The predictive performance of the model was validated using discriminative and calibration methods, and the clinical usefulness of the model was evaluated by clinical decision curve analysis (DCA). After selection, the importance of each characteristic in the best model was interpreted using the SHAP package in Python to provide insight into the relationships among them.
According to the inclusion and exclusion criteria, 207 out of 578 cases were excluded, and the remaining 371 patients were finally included for this study. Their demographic and clinical characteristics are shown in Table 1. Among them, 219 (59.02%) patients suffered from early HCC recurrence. Those 371 patients were randomized at a ratio of 8:2 into the training and test sets. As a result, 296 patients were in the training set, including 221 males and 75 females, of whom 172 (58.11%) experienced early HCC recurrence; 75 patients in the test set, including 59 males and 16 females, of whom 47 had early HCC recurrence (62.67%). No statistically significant difference was observed in any measure tested between the training and test groups (P > 0.05 for all; Table 1).
| Variable | Overall (n = 371) | No HCC recurrence (n = 152) | HCC recurrence (n = 219) | P value | Training set (n = 296) | Test set (n = 75) | P value | |
| Liver cirrhosis, n (%) | No | 85 (22.911) | 28 (18.421) | 57 (26.027) | 0.086 | 85 (22.911) | 69 (23.311) | 0.716 |
| Yes | 286 (77.089) | 124 (81.579) | 162 (73.973) | 227 (76.689) | 59 (78.667) | |||
| Tumor size, n (%), cm | ≤ 2 | 42 (11.321) | 16 (10.526) | 26 (11.872) | 0.027 | 29 (9.797) | 13 (17.333) | 0.308 |
| 2-5 | 168 (45.283) | 78 (51.316) | 90 (41.096) | 137 (46.284) | 31 (41.333) | |||
| 5-10 | 109 (29.380) | 46 (30.263) | 63 (28.767) | 87 (29.392) | 22 (29.333) | |||
| ≥ 10 | 52 (14.016) | 12 (7.895) | 40 (18.265) | 43 (14.527) | 9 (12.000) | |||
| Tumor number, n (%) | 1 | 285 (76.819) | 128 (84.211) | 157 (71.689) | 0.005 | 229 (77.365) | 56 (74.667) | 0.621 |
| ≥ 2 | 86 (23.181) | 24 (15.789) | 62 (28.311) | 67 (22.635) | 19 (25.333) | |||
| Homogeneous tumor enhancement, n (%) | No | 297 (80.054) | 119 (78.289) | 178 (81.279) | 0.479 | 238 (80.405) | 59 (78.667) | 0.736 |
| Yes | 74 (19.946) | 33 (21.711) | 41 (18.721) | 58 (19.595) | 16 (21.333) | |||
| Intratumoral necrosis, n (%) | No | 142 (38.275) | 69 (45.395) | 73 (33.333) | 0.019 | 113 (38.176) | 29 (38.667) | 0.938 |
| Yes | 229 (61.725) | 83 (54.605) | 146 (66.667) | 183 (61.824) | 46 (61.333) | |||
| Tumor envelope intactness, n (%) | No | 145 (39.084) | 60 (39.474) | 85 (38.813) | 0.898 | 117 (39.527) | 28 (37.333) | 0.728 |
| Yes | 226 (60.916) | 92 (60.526) | 134 (61.187) | 179 (60.473) | 47 (62.667) | |||
| Regular tumor margin, n (%) | No | 176 (47.439) | 73 (48.026) | 103 (47.032) | 0.850 | 139 (46.959) | 37 (49.333) | 0.713 |
| Yes | 195 (52.561) | 79 (51.974) | 116 (52.968) | 157 (53.041) | 38 (50.667) | |||
| Intratumoral arteries, n (%) | No | 174 (46.900) | 89 (58.553) | 85 (38.813) | < 0.001 | 135 (45.608) | 39 (52.000) | 0.322 |
| Yes | 197 (53.100) | 63 (41.447) | 134 (61.187) | 161 (54.392) | 36 (48.000) | |||
| Adjacency to major vessels, n (%) | No | 216 (58.221) | 93 (61.184) | 123 (56.164) | 0.335 | 171 (57.770) | 45 (60.000) | 0.727 |
| Yes | 155 (41.779) | 59 (38.816) | 96 (43.836) | 125 (42.230) | 30 (40.000) | |||
| Ascitic fluid, n (%) | No | 262 (70.620) | 106 (69.737) | 156 (71.233) | 0.756 | 207 (69.932) | 55 (73.333) | 0.564 |
| Yes | 109 (29.380) | 46 (30.263) | 63 (28.767) | 89 (30.068) | 20 (26.667) | |||
| Age, n (%), yr | < 60 | 227 (61.186) | 85 (55.921) | 142 (64.840) | 0.083 | 187 (63.176) | 40 (53.333) | 0.118 |
| ≥ 60 | 144 (38.814) | 67 (44.079) | 77 (35.160) | 109 (36.824) | 35 (46.667) | |||
| Gender, n (%) | Male | 280 (75.472) | 118 (77.632) | 162 (73.973) | 0.421 | 221 (74.662) | 59 (78.667) | 0.472 |
| Female | 91 (24.528) | 34 (22.368) | 57 (26.027) | 75 (25.338) | 16 (21.333) | |||
| HBsAg, n (%) | Negative | 101 (27.224) | 41 (26.974) | 60 (27.397) | 0.928 | 76 (25.676) | 25 (33.333) | 0.183 |
| Positive | 270 (72.776) | 111 (73.026) | 159 (72.603) | 220 (74.324) | 50 (66.667) | |||
| HBeAg, n (%) | Negative | 289 (77.898) | 123 (80.921) | 166 (75.799) | 0.242 | 229 (77.365) | 60 (80.000) | 0.623 |
| Positive | 82 (22.102) | 29 (19.079) | 53 (24.201) | 67 (22.635) | 15 (20.000) | |||
| ALBI, n (%), grade | 1 | 67 (18.059) | 26 (17.105) | 41 (18.721) | 0.907 | 54 (18.243) | 13 (17.333) | 0.710 |
| 2 | 288 (77.628) | 119 (78.289) | 169 (77.169) | 228 (77.027) | 60 (80.000) | |||
| 3 | 16 (4.313) | 7 (4.605) | 9 (4.110) | 14 (4.730) | 2 (2.667) | |||
| MLR, n (%) | < 0.285 | 171 (46.092) | 75 (49.342) | 96 (43.836) | 0.295 | 140 (47.297) | 31 (41.333) | 0.355 |
| ≥ 0.285 | 200 (53.908) | 77 (50.658) | 123 (56.164) | 156 (52.703) | 44 (58.667) | |||
| GLR, n (%) | < 32.16 | 190 (51.213) | 92 (60.526) | 98 (44.749) | 0.003 | 152 (51.351) | 38 (50.667) | 0.916 |
| ≥ 32.16 | 181 (48.787) | 60 (39.474) | 121 (55.251) | 144 (48.649) | 37 (49.333) | |||
| PLR, n (%) | < 138.18 | 300 (80.863) | 132 (86.842) | 168 (76.712) | 0.015 | 241 (81.419) | 59 (78.667) | 0.588 |
| ≥ 138.18 | 71 (19.137) | 20 (13.158) | 51 (23.288) | 55 (18.581) | 16 (21.333) | |||
| ALP, n (%) | < 80.29 | 280 (75.472) | 121 (79.605) | 159 (72.603) | 0.123 | 222 (75.000) | 58 (77.333) | 0.675 |
| ≥ 80.29 | 91 (24.528) | 31 (20.395) | 60 (27.397) | 74 (25.000) | 17 (22.667) | |||
| LMR, n (%) | < 5.33 | 82 (22.102) | 28 (18.421) | 54 (24.658) | 0.155 | 67 (22.635) | 15 (20.000) | 0.623 |
| ≥ 5.33 | 289 (77.898) | 124 (81.579) | 165 (75.342) | 229 (77.365) | 60 (80.000) | |||
| FAR, n (%) | < 0.07 | 220 (59.299) | 96 (63.158) | 124 (56.621) | 0.208 | 174 (58.784) | 46 (61.333) | 0.688 |
| ≥ 0.07 | 151 (40.701) | 56 (36.842) | 95 (43.379) | 122 (41.216) | 29 (38.667) | |||
| SII, n (%) | < 312.71 | 198 (53.369) | 93 (61.184) | 105 (47.945) | 0.012 | 157 (53.041) | 41 (54.667) | 0.801 |
| ≥ 312.71 | 173 (46.631) | 59 (38.816) | 114 (52.055) | 139 (46.959) | 34 (45.333) | |||
| AFP, n (%), ng/mL | < 20 | 171 (46.092) | 81 (53.289) | 90 (41.096) | 0.012 | 138 (46.622) | 33 (44.000) | 0.857 |
| 20-400 | 86 (23.181) | 37 (24.342) | 49 (22.374) | 69 (23.311) | 17 (22.667) | |||
| ≥ 400 | 114 (30.728) | 34 (22.368) | 80 (36.530) | 89 (30.068) | 25 (33.333) | |||
| PLT, median (IQR), 109/L | 168.000 (113.000, 209.000) | 154.000 (112.000, 200.000) | 171.000 (118.000, 216.000) | 0.110 | 165.000 (111.000, 208.000) | 178.000 (133.000, 220.000) | -1.451 | |
| Blood glucose, median (IQR), mmol/L | 4.720 (4.340, 5.310) | 4.840 (4.390, 5.660) | 4.660 (4.300, 5.170) | 0.009 | 4.710 (4.350, 5.310) | 4.770 (4.300, 5.310) | -0.066 |
A total of 26 potential characteristic variables were selected for this study. Their importance was analyzed using LASSO regression, XGBoost, and random forest (RF). Ten variables with the highest importance were ranked in descending order. Eight key variables were identified after screening the intersection of the variables using the three models. In no particular order, these variables were age, intratumoral arteries, AFP, blood glucose, number of tumors, GLR, liver cirrhosis, and PLT (Figure 4).
We eventually constructed six machine learning models, and calculated the area under the ROC curve (AUROC) in both the training set and validation set (Figure 5A and B) and generated the forest plots of the area under the curve (AUC) values of the six predictive models in the training set (Figure 6). The XGBoost analyses indicated larger and better AUROCs in the training, validation, and test sets of 0.993 (95%CI: 0.982-1.000), 0.734 (95%CI: 0.601-0.867), and 0.706 (95%CI: 0.585-0.827), respectively (Figure 7). After the five-fold cross-validation, the AUC had an SD of 0.002 for the XGBoost model, the smallest value among the six models, indicating that the XGBoost model had the most stable performance (Figure 6) and that XGBoost achieved lower and better Brier scores (Figure 5C). The DCA results revealed that the XGBoost model was the best diagnostic tool and had good clinical utility compared to other models (Figure 5D). In comparison with individual models (Tables 2 and 3), although the RF model had better indicators for each evaluation in the training dataset, the XGBoost model was an optimal model after a five-fold cross-validation, considering that the RF model might have overfitting. The XGBoost model achieved an AUC of 0.706 (95%CI: 0.585-0.827), accuracy of 0.64, sensitivity of 0.857, specificity of 0.545, and F1 score of 0.766 in the test dataset. The optimal critical value for the prediction probability of the XGBoost model was 55.0 %, according to the Youden index.
| Model | AUC (95%CI) | Accuracy (95%CI) | Sensitivity (95%CI) | Specificity (95%CI) | F1 score (95%CI) |
| XGBoost | 0.993 (0.984-1.000) | 0.974 (0.969-0.979) | 0.980 (0.971-0.988) | 0.976 (0.966-0.986) | 0.981 (0.977-0.985) |
| GNB | 0.692 (0.624-0.761) | 0.671 (0.660-0.683) | 0.754 (0.689-0.820) | 0.566 (0.486-0.646) | 0.728 (0.705-0.750) |
| MLP | 0.574 (0.501-0.648) | 0.563 (0.516-0.611) | 0.517 (0.274-0.761) | 0.637 (0.410-0.863) | 0.549 (0.411-0.686) |
| SVM | 0.308 (0.240-0.376) | 0.453 (0.391-0.514) | 0.204 (-0.182-0.591) | 0.800 (0.413-1.187) | NaN |
| Logistic | 0.702 (0.634-0.770) | 0.677 (0.657-0.696) | 0.747 (0.654-0.841) | 0.589 (0.499-0.679) | 0.727 (0.689-0.765) |
| AdaBoost | 0.814 (0.761-0.868) | 0.725 (0.701-0.748) | 0.661 (0.585-0.737) | 0.825 (0.773-0.876) | 0.737 (0.700-0.773) |
| Model | AUC (95%CI) | Accuracy (95%CI) | Sensitivity (95%CI) | Specificity (95%CI) | F1 score (95%CI) |
| XGBoost | 0.734 (0.601-0.867) | 0.683 (0.658-0.708) | 0.662 (0.596-0.729) | 0.782 (0.717-0.847) | 0.694 (0.636-0.752) |
| GNB | 0.657 (0.512-0.802) | 0.618 (0.583-0.653) | 0.647 (0.443-0.851) | 0.662 (0.464-0.861) | 0.641 (0.535-0.748) |
| MLP | 0.548 (0.396-0.699) | 0.514 (0.452-0.575) | 0.575 (0.398-0.752) | 0.627 (0.491-0.764) | 0.577 (0.471-0.682) |
| SVM | 0.363 (0.217-0.509) | 0.446 (0.381-0.511) | 0.452 (0.035-0.870) | 0.617 (0.188-1.046) | NaN |
| Logistic | 0.661 (0.517-0.806) | 0.632 (0.602-0.661) | 0.797 (0.709-0.886) | 0.525 (0.414-0.637) | 0.728 (0.706-0.751) |
| AdaBoost | 0.649 (0.504-0.793) | 0.618 (0.569-0.667) | 0.591 (0.330-0.852) | 0.780 (0.610-0.950) | 0.617 (0.403-0.831) |
The XGBoost model was further analyzed using the SHAP software package, the results unveiled the importance and impact of each feature in the sample, and the importance from the highest to lowest ranked as blood glucose, PLT, intratumoral arteries, AFP, liver cirrhosis, GLR, age, and number of tumors.
Analysis of the XGBoost model using the SHAP package indicated that each feature in the sample had both positive and negative impacts. The relationship between the magnitude of the eigenvalues and the predicted impact is shown in Figure 8. For applying the XGBoost model, the optimal cut-off point for the model's prediction probability was 55%. If the model predicted > 55%, it would indicate that patients undergoing hepatectomy are at higher risk of early postoperative HCC recurrence. In addition, we generated a calculator based on the model online for clinicians to apply this model in their daily practice (available at: http://152.136.184.184:5050/).
Early HCC recurrence is one of the main factors shortening the survival of postoperative HCC patients. Given that most HCC patients experience HCC recurrence within 5 years post-resection of their primary tumors[2], prevention, early diagnosis, and precise treatment will be critical for limiting postoperative recurrence and improving the survival of HCC patients. This study found an early HCC recurrence rate of 59.02% in this population. While most cases with early HCC recurrence usually come from the hidden micrometastasis of the initial HCC in the body, the late HCC recurrence appears two years after resection. It is commonly from new polyclonal origins of HCC lesions that may completely differ from the initial liver cancer. Actually, early HCC recurrence is usually associated with more aggressive features and a worse prognosis. A previous study has suggested that adjuvant therapy for HCC patients at risk of early recurrence may prolong their survival[17]. However, identifying the patients at high risk of postoperative recurrence before surgery is complicated because of the lack of specific tumor biomarkers and clinical features. Therefore, clinical prediction models will be valuable for predicting early HCC recurrence. However, most available clinical prediction models are often constructed based on the assumption of linear relationships and usually fail to reflect the nature of complex, multidimensional, nonlinear relationships between different predictor variables in the pathogenesis of HCC to affect their predictive performance. Machine learning has shown great potential in analyses of data with nonlinear relationships, and in general, machine learning models outperform traditional regression models in predicting the development of HCC[18].
In this study, we analyzed pre-operative imaging and clinical indicators that were easily accessible and non-invasive and selected eight important feature variables using machine learning algorithms, including pre-operative blood glucose value, PLT, intratumoral artery mass, methemoglobin, cirrhosis, GLR, age, and number of tumors. Subsequently, we generated six machine learning prediction models and validated their value in predicting the risk of early postoperative HCC recurrence. After randomly dividing the independent test set, we applied hyperparameter tuning in the training dataset with five-fold cross-validation and validated them in the test dataset. The results indicated that the XGBoost model exhibited better discriminative and predictive value than other models constructed. The calibration plots for the XGBoost model also displayed an excellent consistency between predictions and actual observations, and the DCA unveiled that the XGBoost model exhibited excellent clinical application. The model was, therefore, selected as the optimal model with an optimal cut-off value of 55% for the prediction probability. If a patient has a predicted probability of greater than 55%, the patient will have an increased risk of early HCC recurrence after primary tumor resection. Therefore, the prediction model may be valuable for predicting HCC patients for their risk of early postoperative HCC recurrence. Accordingly, to make the prediction model more convenient for clinical practice, we generated a model-based calculator online for physicians to use this model to predict early postoperative HCC recurrence. This will significantly improve the efficiency of the practical application of the prediction model.
To ensure better performance and clinical interpretation of the predictive model, we used the shape model to analyze its interpretation after determining the importance of the relationship between each characteristic variable and early postoperative HCC recurrence. The results indicated that the pre-operative blood glucose value was the variable with the highest importance, consistent with previous studies[19-21]. Hyperglycemia has now been recognized as a risk factor for the development and prognosis of HCC. Shi et al[19] found that HCC cells employed glycolysis to support their rapid growth in hypoxic and oxidative stress in the body. On the other hand, hyperglycemia can enhance the growth, invasion, and metastasis of HCC cells by promoting the expression of vascular endothelial growth factor[20]. Similarly, PLT also influenced the early HCC recurrence more, possibly by a positive feedback regulation[22]. Elevated pre-operative PLT are independently associated with increased tumor load, extrahepatic recurrence and metastasis, and poor prognoses of HCC[23]. In turn, HCC cells can promote platelet generation by secreting platelet-generating hormones, and the increased PLT further provide favorable conditions for HCC cell proliferation and metastasis[24,25]. Intratumoral arterial features on pre-operative imaging are a significant risk factor for early postoperative HCC recurrence. A previous study has shown that the distribution and number of blood vessels supplying the tumor can impact the prognosis of HCC patients[26]. Multiple small arteries supplying blood to the tumor may reflect high pressure inside the tumor. The high-pressure environment tends to cause the tumor cells to shed and form microvascular invasion in the surrounding vessels, which may even spread to other locations in the liver, increasing the risk of early postoperative HCC recurrence. Moreover, inflammatory factors are essential for the tumor microenvironment, and high levels of GGT will lead to high cysteine levels in vivo, which influence tumor development regarding redox regulation and drug resistance[27]. GGT is a new potential prognostic indicator of inflammation in HCC[28], while lymphocytes are crucial for the systemic and local immune responses, including anti-tumor immunity[29]. A decrease in the number of lymphocytes can result in a weakened immune function, which may enhance angiogenesis and extracellular matrix remodeling, creating a mutagenic environment for tumor cells and promoting tumorigenesis and development[30]. Indeed, GLR has been shown to be valuable for predicting early postoperative HCC recurrence in a population of 606 patients[31]. Therefore, these vital feature variables included in the best mode are critical for predicting early postoperative HCC recurrence.
In summary, the predictive model constructed in this study may help clinicians predict the risk of early postoperative HCC recurrence in individual HCC patients and better manage them before, during, and after surgery. Our findings may open new avenues for investigating therapeutic strategies for postoperative recurrence of HCC.
We acknowledge that the current study has limitations. First, our study was conducted with a relatively small sample size in a single center and did not include external validation from other centers. Therefore, future studies are needed to validate the value of this model in a larger patient population across multiple centers. Second, the optimal thresholds for the inflammation-related ratio in this study may need to be re-evaluated due to the limited sample size. Considering these limitations, we plan to expand the sample size through a multicenter study in the future and compare our model with other prediction models to further verify its reliability.
Compared to other models, the XGBoost model demonstrated superior performance and emerged as the best predictive measurement model. This predictive model is easily accessible in daily clinical practice and may serve as a crucial tool in guiding postoperative follow-up and individualized medicine for HCC patients.
Surgical resection is still the main treatment for hepatocellular carcinoma (HCC). HCC recurrence is the main factor affecting patients' survival rate after surgery. Developing pre-operative non-invasive predictive methods will be highly significant in identifying patients at high risk of postoperative recurrence and precise management of those patients by closely monitoring and individualized treatment on time.
To develop a new risk prediction model for the early postoperative recurrence of HCC and enhance the feasibility and applicability of the constructed model.
This study aimed to identify key variables in pre-operative clinical and imaging data using machine learning algorithms to construct multiple risk prediction models for early postoperative recurrence of HCC.
The demographic and clinical data of 371 HCC patients were collected and analyzed, and the key feature variables were selected to construct six different machine learning prediction models. Each model was evaluated, and the best-performing model was selected for interpreting the importance of each variable. Finally, an online calculator based on the model was generated for daily clinical practice.
Following machine learning analysis, eight key feature variables were selected to construct six different prediction models. The XGBoost model outperformed other models, with the area under the receiver operating characteristic curve in the training, validation, and test datasets being 0.993 [95% confidence interval (95%CI): 0.982-1.000], 0.734 (95%CI: 0.601-0.867), and 0.706 (95%CI: 0.585-0.827), respectively. Calibration curve and decision curve analysis indicated that the XGBoost model also had good predictive performance and clinical application value.
The XGBoost model exhibits superior performance and is a reliable tool for predicting early postoperative HCC recurrence. This model may guide surgical strategies and postoperative individualized medicine.
A multicenter study with large samples should be conducted in the future, and comparing our model with other prediction models is needed to further verify its reliability.
Provenance and peer review: Unsolicited article; Externally peer reviewed.
Peer-review model: Single blind
Specialty type: Gastroenterology and hepatology
Country/Territory of origin: China
Peer-review report’s scientific quality classification
Grade A (Excellent): 0
Grade B (Very good): B, B
Grade C (Good): C
Grade D (Fair): 0
Grade E (Poor): 0
P-Reviewer: Mijwil MM, Iraq; Tsoulfas G, Greece S-Editor: Lin C L-Editor: Wang TQ P-Editor: Xu ZH
| 1. | Rumgay H, Arnold M, Ferlay J, Lesi O, Cabasag CJ, Vignat J, Laversanne M, McGlynn KA, Soerjomataram I. Global burden of primary liver cancer in 2020 and predictions to 2040. J Hepatol. 2022;77:1598-1606. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 17] [Cited by in F6Publishing: 347] [Article Influence: 173.5] [Reference Citation Analysis (0)] |
| 2. | Llovet JM, Kelley RK, Villanueva A, Singal AG, Pikarsky E, Roayaie S, Lencioni R, Koike K, Zucman-Rossi J, Finn RS. Hepatocellular carcinoma. Nat Rev Dis Primers. 2021;7:6. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1343] [Cited by in F6Publishing: 2489] [Article Influence: 829.7] [Reference Citation Analysis (1)] |
| 3. | Niu ZS, Wang WH, Niu XJ. Recent progress in molecular mechanisms of postoperative recurrence and metastasis of hepatocellular carcinoma. World J Gastroenterol. 2022;28:6433-6477. [PubMed] [DOI] [Cited in This Article: ] [Cited by in CrossRef: 2] [Cited by in F6Publishing: 1] [Article Influence: 0.5] [Reference Citation Analysis (3)] |
| 4. | Wei T, Zhang XF, Bagante F, Ratti F, Marques HP, Silva S, Soubrane O, Lam V, Poultsides GA, Popescu I, Grigorie R, Alexandrescu S, Martel G, Workneh A, Guglielmi A, Hugh T, Lv Y, Aldrighetti L, Pawlik TM. Early Versus Late Recurrence of Hepatocellular Carcinoma After Surgical Resection Based on Post-recurrence Survival: an International Multi-institutional Analysis. J Gastrointest Surg. 2021;25:125-133. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 19] [Cited by in F6Publishing: 19] [Article Influence: 6.3] [Reference Citation Analysis (0)] |
| 5. | Bednarsch J, Czigany Z, Heij LR, Amygdalos I, Heise D, Bruners P, Ulmer TF, Neumann UP, Lang SA. The role of re-resection in recurrent hepatocellular carcinoma. Langenbecks Arch Surg. 2022;407:2381-2391. [PubMed] [DOI] [Cited in This Article: ] [Reference Citation Analysis (0)] |
| 6. | Criss CR, Makary MS. Salvage locoregional therapies for recurrent hepatocellular carcinoma. World J Gastroenterol. 2023;29:413-424. [PubMed] [DOI] [Cited in This Article: ] [Cited by in CrossRef: 1] [Cited by in F6Publishing: 1] [Article Influence: 1.0] [Reference Citation Analysis (8)] |
| 7. | Llovet JM, De Baere T, Kulik L, Haber PK, Greten TF, Meyer T, Lencioni R. Locoregional therapies in the era of molecular and immune treatments for hepatocellular carcinoma. Nat Rev Gastroenterol Hepatol. 2021;18:293-313. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 174] [Cited by in F6Publishing: 370] [Article Influence: 123.3] [Reference Citation Analysis (0)] |
| 8. | Wang XH, Liao B, Hu WJ, Tu CX, Xiang CL, Hao SH, Mao XH, Qiu XM, Yang XJ, Yue X, Kuang M, Peng BG, Li SQ. Novel Models Predict Postsurgical Recurrence and Overall Survival for Patients with Hepatitis B Virus-Related Solitary Hepatocellular Carcinoma ≤10 cm and Without Portal Venous Tumor Thrombus. Oncologist. 2020;25:e1552-e1561. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 9] [Cited by in F6Publishing: 9] [Article Influence: 2.3] [Reference Citation Analysis (0)] |
| 9. | Chan AWH, Berhane S, Cucchetti A, Johnson PJ. Reply to: Correspondence concerning "Development of pre and post-operative models to predict early recurrence of hepatocellular carcinoma after surgical resection". J Hepatol. 2019;70:573-574. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1] [Cited by in F6Publishing: 1] [Article Influence: 0.2] [Reference Citation Analysis (0)] |
| 10. | Yao LQ, Chen ZL, Feng ZH, Diao YK, Li C, Sun HY, Zhong JH, Chen TH, Gu WM, Zhou YH, Zhang WG, Wang H, Zeng YY, Wu H, Wang MD, Xu XF, Pawlik TM, Lau WY, Shen F, Yang T. Correction to: Clinical Features of Recurrence After Hepatic Resection for Early-Stage Hepatocellular Carcinoma and Long-Term Survival Outcomes of Patients with Recurrence: A Multi-institutional Analysis. Ann Surg Oncol. 2022;29:5206. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1] [Cited by in F6Publishing: 2] [Article Influence: 1.0] [Reference Citation Analysis (0)] |
| 11. | Haug CJ, Drazen JM. Artificial Intelligence and Machine Learning in Clinical Medicine, 2023. N Engl J Med. 2023;388:1201-1208. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1] [Cited by in F6Publishing: 162] [Article Influence: 162.0] [Reference Citation Analysis (1)] |
| 12. | Zeng J, Zeng J, Lin K, Lin H, Wu Q, Guo P, Zhou W, Liu J. Development of a machine learning model to predict early recurrence for hepatocellular carcinoma after curative resection. Hepatobiliary Surg Nutr. 2022;11:176-187. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 8] [Cited by in F6Publishing: 16] [Article Influence: 8.0] [Reference Citation Analysis (0)] |
| 13. | Wu C, Yu S, Zhang Y, Zhu L, Chen S, Liu Y. CT-Based Radiomics Nomogram Improves Risk Stratification and Prediction of Early Recurrence in Hepatocellular Carcinoma After Partial Hepatectomy. Front Oncol. 2022;12:896002. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1] [Cited by in F6Publishing: 6] [Article Influence: 3.0] [Reference Citation Analysis (0)] |
| 14. | Yan M, Zhang X, Zhang B, Geng Z, Xie C, Yang W, Zhang S, Qi Z, Lin T, Ke Q, Li X, Wang S, Quan X. Deep learning nomogram based on Gd-EOB-DTPA MRI for predicting early recurrence in hepatocellular carcinoma after hepatectomy. Eur Radiol. 2023;33:4949-4961. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1] [Reference Citation Analysis (0)] |
| 15. | Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1604] [Cited by in F6Publishing: 1625] [Article Influence: 180.6] [Reference Citation Analysis (0)] |
| 16. | European Association for the Study of the Liver. EASL Clinical Practice Guidelines: Management of hepatocellular carcinoma. J Hepatol. 2018;69:182-236. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 3934] [Cited by in F6Publishing: 4962] [Article Influence: 827.0] [Reference Citation Analysis (0)] |
| 17. | He W, Peng B, Tang Y, Yang J, Zheng Y, Qiu J, Zou R, Shen J, Li B, Yuan Y. Nomogram to Predict Survival of Patients With Recurrence of Hepatocellular Carcinoma After Surgery. Clin Gastroenterol Hepatol. 2018;16:756-764.e10. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 33] [Cited by in F6Publishing: 34] [Article Influence: 5.7] [Reference Citation Analysis (0)] |
| 18. | Singal AG, Mukherjee A, Elmunzer BJ, Higgins PD, Lok AS, Zhu J, Marrero JA, Waljee AK. Machine learning algorithms outperform conventional regression models in predicting development of hepatocellular carcinoma. Am J Gastroenterol. 2013;108:1723-1730. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 200] [Cited by in F6Publishing: 168] [Article Influence: 15.3] [Reference Citation Analysis (1)] |
| 19. | Shi DY, Xie FZ, Zhai C, Stern JS, Liu Y, Liu SL. The role of cellular oxidative stress in regulating glycolysis energy metabolism in hepatoma cells. Mol Cancer. 2009;8:32. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 85] [Cited by in F6Publishing: 90] [Article Influence: 6.0] [Reference Citation Analysis (0)] |
| 20. | Li W, Liu H, Qian W, Cheng L, Yan B, Han L, Xu Q, Ma Q, Ma J. Hyperglycemia aggravates microenvironment hypoxia and promotes the metastatic ability of pancreatic cancer. Comput Struct Biotechnol J. 2018;16:479-487. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 25] [Cited by in F6Publishing: 37] [Article Influence: 6.2] [Reference Citation Analysis (0)] |
| 21. | De Matteis S, Ragusa A, Marisi G, De Domenico S, Casadei Gardini A, Bonafè M, Giudetti AM. Aberrant Metabolism in Hepatocellular Carcinoma Provides Diagnostic and Therapeutic Opportunities. Oxid Med Cell Longev. 2018;2018:7512159. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 76] [Cited by in F6Publishing: 95] [Article Influence: 15.8] [Reference Citation Analysis (0)] |
| 22. | Lai Q, Vitale A, Manzia TM, Foschi FG, Levi Sandri GB, Gambato M, Melandro F, Russo FP, Miele L, Viganò L, Burra P, Giannini EG; Associazione Italiana per lo Studio del Fegato (AISF) HCC Special Interest Group. Platelets and Hepatocellular Cancer: Bridging the Bench to the Clinics. Cancers (Basel). 2019;11. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 16] [Cited by in F6Publishing: 25] [Article Influence: 5.0] [Reference Citation Analysis (0)] |
| 23. | Liu PH, Hsu CY, Su CW, Huang YH, Hou MC, Rich NE, Fujiwara N, Hoshida Y, Singal AG, Huo TI. Thrombocytosis is associated with worse survival in patients with hepatocellular carcinoma. Liver Int. 2020;40:2522-2534. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 13] [Cited by in F6Publishing: 17] [Article Influence: 4.3] [Reference Citation Analysis (0)] |
| 24. | Tsilimigras DI, Mehta R, Aldrighetti L, Poultsides GA, Maithel SK, Martel G, Shen F, Koerkamp BG, Endo I, Pawlik TM; International Intrahepatic Cholangiocarcinoma Study Group. Development and Validation of a Laboratory Risk Score (LabScore) to Predict Outcomes after Resection for Intrahepatic Cholangiocarcinoma. J Am Coll Surg. 2020;230:381-391.e2. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 14] [Cited by in F6Publishing: 20] [Article Influence: 5.0] [Reference Citation Analysis (0)] |
| 25. | Ramadori P, Klag T, Malek NP, Heikenwalder M. Platelets in chronic liver disease, from bench to bedside. JHEP Rep. 2019;1:448-459. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 22] [Cited by in F6Publishing: 37] [Article Influence: 7.4] [Reference Citation Analysis (0)] |
| 26. | Erstad DJ, Tanabe KK. Prognostic and Therapeutic Implications of Microvascular Invasion in Hepatocellular Carcinoma. Ann Surg Oncol. 2019;26:1474-1493. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 139] [Cited by in F6Publishing: 230] [Article Influence: 46.0] [Reference Citation Analysis (0)] |
| 27. | Hanigan MH. Gamma-glutamyl transpeptidase: redox regulation and drug resistance. Adv Cancer Res. 2014;122:103-141. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 141] [Cited by in F6Publishing: 170] [Article Influence: 18.9] [Reference Citation Analysis (0)] |
| 28. | Zhou B, Zhan C, Wu J, Liu J, Zhou J, Zheng S. Prognostic significance of preoperative gamma-glutamyltransferase to lymphocyte ratio index in nonfunctional pancreatic neuroendocrine tumors after curative resection. Sci Rep. 2017;7:13372. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 12] [Cited by in F6Publishing: 12] [Article Influence: 1.7] [Reference Citation Analysis (0)] |
| 29. | Salman T, Kazaz SN, Varol U, Oflazoglu U, Unek IT, Kucukzeybek Y, Alacacioglu A, Atag E, Semiz HS, Cengiz H, Oztop I, Tarhan MO. Prognostic Value of the Pretreatment Neutrophil-to-Lymphocyte Ratio and Platelet-to-Lymphocyte Ratio for Patients with Neuroendocrine Tumors: An Izmir Oncology Group Study. Chemotherapy. 2016;61:281-286. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 36] [Cited by in F6Publishing: 41] [Article Influence: 5.1] [Reference Citation Analysis (0)] |
| 30. | Qin L, Li C, Xie F, Wang Z, Wen T. Are inflammation-based markers useful in patients with hepatocellular carcinoma and clinically significant portal hypertension after liver resection? Biosci Trends. 2020;14:297-303. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1] [Cited by in F6Publishing: 1] [Article Influence: 0.3] [Reference Citation Analysis (0)] |
| 31. | Li S, Xu W, Liao M, Zhou Y, Weng J, Ren L, Yu J, Liao W, Huang Z. The Significance of Gamma-Glutamyl Transpeptidase to Lymphocyte Count Ratio in the Early Postoperative Recurrence Monitoring and Prognosis Prediction of AFP-Negative Hepatocellular Carcinoma. J Hepatocell Carcinoma. 2021;8:23-33. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 5] [Cited by in F6Publishing: 15] [Article Influence: 5.0] [Reference Citation Analysis (0)] |