Gastric cancer is a common disease with > 1 million cases worldwide in 2020, ranking fifth in incidence and fourth in mortality [1]. In Korea, due to screening gastroscopy, ~70% of the newly diagnosed gastric cancers are early gastric cancer (EGC), for which endoscopic resection (ER) represents an excellent treatment. Indeed, ER has a comparable overall survival to surgery, as well as additional benefits in terms of complications and hospital stay.

The most important factor in determining the initial treatment of EGC is lymph node metastasis (LNM). The absolute indication for ER for EGC is differentiated-type mucosal adenocarcinoma with a diameter ≤ 2 cm and without ulcerative findings [2,3]. The expanded criteria of ER are as follows: (1) differentiated-type mucosal cancer with a diameter > 2 cm and without ulceration, (2) differentiated-type mucosal cancer with ulceration and < 3 cm in diameter, (3) undifferentiated-type mucosal cancer with < 2 cm in diameter and without ulceration, and (4) differentiated-type SM1 submucosal cancer (< 500 μm from the muscularis mucosa) with < 3 cm in diameter. Recently, the Japanese Gastric Cancer Association (JGCA) published the 2nd edition of guidelines for ER of EGC, and endoscopic curability after ER was classified into four categories eCuraA, eCuraB, eCuraC-1, and eCuraC-2 [4]. Among them, eCuraC-2 requires surgical treatment because of the high risk of LNM. However, both the expanded criteria and the eCura system also tend to include ER indications too strictly, which has led to a demand for more sophisticated criteria. To accurately predict the subgroup in which endoscopic submucosal dissection (ESD) is feasible, it is important to identify risk factors for LNM in EGC [5–7].

The logistic regression (LR) model is the most well-known predictive model in the medical field [8,9]. Several studies have performed LR analysis for intramucosal gastric signet ring cell (SRC) cancer and have created a risk score system or nomogram before verifying it with a validation set [10,11]. Recently, the research on predictive models using artificial intelligence-driven machine learning methods in various fields of medicine [12–15]. One study compared several machine learning methods in poorly differentiated intramucosal gastric cancer [16]. Another study reported that XGBoost and support vector machine (SVM) models improved the prediction rate of non-curative resection in ESD patients [17].

Among machine learning methods, those belonging to the decision tree family are considered superior to other methods [18]. Decision tree methods have advantages in explainability, as well as improved performance through bagging and boosting methods. Bagging is a method to make predictions through voting after constructing each tree model by extracting the training data differently through bootstrap sampling several times. The random forest model is representative of bagging tree model [19]. Among boosting methods, which are considered to have superior performance compared to the bagging method, a gradient boosting machine (GBM) is a representative method of learning by assigning a higher weight to misclassified samples, while XGBoost and lightGBM are models known to implement this method. In particular, the XGBoost model shows an advantage in explaining the prediction process because it can identify the features that have a strong influence on the model prediction among each feature using a method called SHapley Additive exPlanations (SHAP) [20,21].

However, no previous research has developed and validated a model for predicting LNM in EGC using the latest high-performance machine learning techniques and applied it to clinical practice. Therefore, we constructed a model for predicting LNM in EGC using LR and GBM models, evaluating the performance in internal and external validation cohorts and comparing it to the expanded criteria.

Here, we evaluated the performance of prediction models including GBM to predict LNM in EGC. To the best of our knowledge, this is the first study to conduct external validation of a model for predicting LNM in EGC using a machine learning method in patients with EGC who underwent ESD followed by gastrectomy. Our findings suggest that the predictive model implemented using machine learning can accurately predict LNM risk in EGC. We also performed analysis using machine learning methods other than LR and GBM including SVM, Random Forest, Lasso, and Elastic Net models (S3 Fig.). To summarize, GBM, LR, and Lasso showed comparable performances to LR and GBM, while SVM and Elastic Net showed lower performance results.

To date, several studies conducted a retrospective analysis of clinicopathological data for patients who underwent gastrectomy to explore the risk factors of LNM in EGC using LR [22–24]. Recently, one study conducted on 2,348 patients in five major tertiary medical centers comparing six machine learning technologies reported that XGBoost showed the best performance to predict LNM in EGC [25]. In contrast, we constructed a model for predicting LNM in EGC using the GBM method with all pathologic data. Only one study has used the GBM method to evaluate a model for predicting the risk of death within 10 years in patients with colorectal cancer, in which the AUROC and accuracy were acceptable at 0.84 and 0.83, respectively [26]. Another study employed a deep learning method to create a model to determine the risk of LNM in T1 colorectal cancer [27]. In the current study, unlike previous studies, only patients who were followed up after ER were used to estimate LNM, and the model was developed by dividing the patient group into training and validation cohorts. Patients who had undergone gastrectomy as an initial treatment for EGC were included in the internal validation set (set 1), while those who underwent ESD as the initial treatment were used as the external validation set (set 2) to compare the performance of the GBM method with the existing JGCA guidelines (conventional expanded criteria and recent eCura system).

LVI, depth, size, ulcer, and differentiation have been consistently considered risk factors for LNM in EGC [5,6]. Except for LVI, the other four features are included in the classic expanded criteria. Even when analyzed using univariate and multivariate LR in this study, the results were consistent with those of previous studies, as shown in Table 2. Additionally, in the GBM method, the features that had a significant influence on predicting LNM in the predictive model, in order of importance, were LVI, depth, differentiation, size, age, location, and ulcer (Fig. 2), which is consistent with the LR model. Specifically, the younger the age, the higher the probability of LNM, which is consistent with the results of the LR model. In contrast, perineural invasion and sex had less influence than the other features.

Both GBM and LR models showed better performance than JGCA guidelines in terms of AUPRC (Table 3). The GBM method showed high performance (AUROC: 0.867 and 0.943 in set 1 and set 2, respectively), but the LR model showed the highest accuracy in set 1 (0.617, 0.566, and 0.555 in LR, GBM, and beyond the expanded criteria, respectively). When the sensitivity was matched to the value of the expanded criteria (0.922) to minimize missing patients with LNM by maximizing the sensitivity to detect LNM, the specificities of LR, GBM, and beyond the expanded criteria were 0.574, 0.516, and 0.502, respectively. However, patients who receive ER as an initial treatment (set 2) is a clinical situation that requires the prediction of LNM risk, and in set 2, the GBM model showed the highest accuracy (0.803, 0.810, and 0.796 for LR, GBM, and eCuraC-2, respectively). When the sensitivity was matched to that of eCuraC-2 (0.958), the specificities of LR, GBM, and eCuraC-2 were 0.796, 0.803, and 0.788, respectively. Despite a slight difference, the specificity of GBM was higher than the upper limit (0.790) of the specificity 95% CI of eCuraC-2.

Set 2 comprised two heterogeneous groups of patients who were followed up after ESD (n=425) and those who underwent gastrectomy within 6 months after complete but non-curative resection (n=123). In clinical practice, it is difficult to decide whether additional surgery should be performed in cases that are beyond the expanded criteria or eCuraC-2, but show complete resection or horizontal margin positive. In such cases, additional surgery should be performed according to the current guidelines; however, the LNM risk is low and unnecessary surgery can be avoided if an individualized approach can be applied. Therefore, subgroup analysis was performed for patients who underwent gastrectomy within 6 months of ESD (n=123), and GBM showed better performance than eCuraC-2 (accuracy: 0.333 and 0.268 in GBM and eCuraC-2, respectively). When the sensitivity was equal to that of eCuraC-2, the specificity of the GBM model was significantly higher (0.190 vs. 0.110). In the clinical setting, the GBM model may be more useful to determine LNM risk and the need for additional surgery (S1 Table, S2 Fig.).

The patients in set 2 had a shallow depth, good differentiation, and small tumor size compared to the training set and set 1 (Table 1). The LR, GBM, and JGCA guidelines outperformed set 2 compared to set 1, which can be attributed to the lower prevalence of LNM in set 2 (Table 3). Moreover, 70 patients (12.8%) with metachronous recurrence were included in set 2 because metachronous recurrence was considered separate to LNM of the primary lesion [28]; selection bias may occur when patients with metachronous recurrence are excluded.

This study has several limitations. First, despite the relatively large number of subjects, this was a single-center retrospective study, and data supplementation through collaborative research with other institutions is warranted in the future. Second, GBM model was trained and validated on the class imbalanced data, where the minority class of LNM positive accounts for as few as 12.5% and 4.3% in set 1 and set 2, respectively. To overcome the class imbalance, data-level methods for machine learning were conducted by oversampling using Synthetic Minority Over-sampling Technique (SMOTE). However, when the GBM analysis was performed with SMOTE applied, a significant decrease in precision and specificity was observed both in set 1 and set 2 (data not shown). Third, rare pathologic subtypes including GCLS, micropapillary adenocarcinoma, and mucinous adenocarcinoma were included in the analyses. When these minor pathologic subtypes were all excluded and reanalyzed, the performances were not substantially different but appears to be slightly worse (S4 Table). Fourth, in set 2, after initial ESD, lymph node and distant metastases were confirmed in the follow-up group through CT. In this study, LNM was confirmed on CT in one of 425 patients (0.24%) in the follow-up group after ESD. Since most studies used the data of patients who underwent surgery only, few studies were found that evaluated patients who underwent ESD and followed up without surgery for the prediction of LNM risk. One recent study reported that the incidence LNM confirmed on CT was 0.27% (4 of 1,491) in such patients, which is comparable to our findings [29]. However, although CT is currently the optimal means to confirm metastasis, the sensitivity of CT for microscopic lymph node metastases may not be sufficient. Fifth, our proposed predictive model is a bit complex and has limitations in applying in clinical practice.

Nevertheless, our study has several strengths. First, the model was tested as an external validation set in the patient group that received ESD as the initial treatment, with the aim to develop a model to assist clinical decision-making in such patients. That is, if the risk of LNM can be accurately predicted from clinical and pathological findings after ESD, unnecessary additional surgery can be avoided. Second, by applying the JGCA guidelines (beyond expanded criteria and eCuraC-2), which are used to decide the need for additional surgery after ESD, to the internal and external validation sets, their effectiveness was analyzed and compared to that of the predictive models using machine learning methods. Third, we used the GBM (the best performing machine learning method), conducted preprocessing (e.g., feature selection and label encoding), and fine-tuned hyperparameters to maximize the performance and prevent overfitting to ultimately develop a predictive model that can be applied to actual practice [30].

In conclusion, the GBM model showed a good performance that was comparable with the JGCA guideline criteria (conventional expanded criteria and recent eCura system) in predicting LNM risk in EGC in both internal and external validation sets. Thus, the GBM model may represent a substitute for the Japanese eCura system in clinical practice. Further studies are warranted to clarify the usefulness of the GBM model.