We appreciate the authors’ interest and commentary to highlight this important paper regarding the ability of radiomics to predict immunotherapy response in advanced cancer. As we know, there are multiple proven methods used in machine learning for radiomics analysis. Various feature selection methods and prediction models can be employed and yield sound data analysis and precise, accurate, and robust results. In our study, we have developed a pipeline for bioinformatic analysis that has been validated both preclinically and clinically for the processing of radiomics data1 derived from both MRI and CT imaging.

The commenting authors mention, ‘the investigators may adjust the algorithm’s hyperparameters and try again until satisfactory performance is achieved. Since many changes are made to make the model more accurate for the validation data, overfitting may occur.’ While some investigators building tree-based classification algorithms (decision trees, random forests, and eXtreme Gradient Boosting (XGBoost)) use hyperparameter tuning to minimize a certain type of loss function or the classification error rate, we have not. In our case, while building our radiomics models, we have not performed any grid search for tuning of the algorithm’s hyperparameters. XGBoost parameters are divided into general parameters (booster), booster parameters (eta, min_child_weight, max_depth, max_leaf_nodes, gamma, max_delta_step, subsample, etc), and learning task parameters (objective, eval_metric, and seed).2 Keeping in mind the small size of our dataset, we relied on the default values for major parameters from the XGBoost algorithm (booster=gbtree, eta=0.3, max_depth=6, objective = “binary logistic”, and eval_metric=error).

The commenting authors also bring up sample size, tumor type heterogeneity, feature selection stability, and the use of leave-one-out cross-validation (LOOCV). As we acknowledge in our original article, a limitation of our study is the small sample size (N=57); however, we were still able to robustly predict immunotherapy response, thus demonstrating the feasibility of this method in advanced cancer. While we acknowledge the tumor type heterogeneity inherent in advanced rare cancers and while not every tumor histological subtype had responders, the advanced tumor group as per compliance with standard United States Food and Drug Administration (FDA) clinical trial protocol demonstrated that key radiomics features irrespective of tumor type predicted response to immunotherapy. Although it was not the primary aim of this study, we are able to see that tumor as a whole harbor specific radiomics features irrespective of tumor type that can robustly help with patient stratification. Regarding feature selection and predictive modeling, feature selection gains particular importance while working with high dimensional datasets where small-n-large-p problems exist; and choosing the best feature selection method to overcome this problem is crucial. The LASSO feature selection method addresses the small-n-large-p issues by applying a shrinkage (regularization) process,3 through penalizing the coefficients of the regression variables and shrinking some of them to zero.4 In doing so, this method reduces the variance without any significant increase of the bias; and hence, this is most useful in cases where there are small number of observations and many (radiomics) features. Moreover, the LASSO feature selection method helps eliminate irrelevant features not associated with the response variable, reduces overfitting, and successfully handles multicollinearity.5 Furthermore, LASSO, as a regularization method, is known to handle multicollinearity very well in small datasets when compared with other feature selection methods.6 The XGBoost, a tree boosting algorithm used in machine learning, is the most widely recognized classification algorithm and is extensively used as it enables one to do the parallel computation, cross-validation, regularization, tree pruning, and missing value imputation (if needed).2 For building our radiomics classification models, we have chosen an ensemble modeling approach where we used least absolute shrinkage and selection operator (LASSO) followed by XGBoost for feature selection and model building, respectively. Feature selection methods like minimum redundancy maximum relevance method and wrapper methods (forward and backward elimination) have been previously tested and implemented as a part of validating our pipeline during its development phase, and we have noted that the LASSO feature selection method resulted in higher predictive accuracy and the selection of more meaningful features when compared with others and was best in handling multicollinearity of the features.

Leave-one-out cross-validation is an extension of the k-fold cross-validation, where k is equal to the number of samples in the dataset. LOOCV is best when working with small datasets and when the estimation of the model performance is critical.7 Since our dataset is small (N=57 patients), we applied LOOCV as the cross-validation while building our radiomics models. Multiple studies have shown that LASSO for feature selection and XGBoost for classification offer good performance in predictive modeling.8 LASSO feature selection, XGBoost, and LOOCV were also used and confirmed in the landmark paper published by Zinn et al1 establishing a causal linkage between radiomics and genomics. Furthermore, this method and pipeline have been validated in both MRI and CT.1 In addition, immune-related Response Evaluation Criteria in Solid Tumors (irRECIST) adapted the concept of measurement similar to that of Response Evaluation Criteria in Solid Tumors (RECIST). So, we expect to have shared radiomics features extracted in patients assessed by irRECIST and RECIST in our radiomics models.

As per the authors’ suggestion, we have checked the multicollinearity of the features and performed 10-fold cross-validation as an alternative to LOOCV to see the predictive performance of our radiomics models. The results for multicollinearity identified by variance inflation factor (VIF) on the top 10 radiomics features used to build the RECIST and irRECIST radiomic models are given in table 1. High VIFs (>10) indicate multicollinearity; this makes interpretation of the contribution of an individual feature challenging but does not necessarily impact predictive performance. Tree-based algorithms such as XGBoost are particularly robust to multi-collinearity. Furthermore, using the 10-fold cross-validation over 10 iterations, the predictive model using the top 10 out of the total 44 features identified from LASSO feature selection to predict RECIST response resulted in a radiomics model that had an accuracy, sensitivity, and specificity (89.47%, 91.89%, and 85%, respectively) (figure 1A). Similarly, the predictive model using the top 10 out of the total of 56 features identified after LASSO feature selection to predict irRECIST response with 10-fold cross-validation over 10 iterations resulted in a radiomics model that had an accuracy, sensitivity, and specificity (87.72%, 90.91%, and 83.33%, respectively) (figure 1B).

In conclusion, we thank the authors for their comments, and we feel there are several ways to look at the radiomics and our approach has been one of these approaches. More studies are needed to solidify and validate the findings.