The U.S. Food and Drug Administration (FDA) has considered large durable treatment effects on tumor burden and large effects on progression-free survival (PFS) in randomized controlled trials (RCTs) to predict effects on overall survival (OS), a direct measure of clinical benefit.2 Because these endpoints can be evaluated earlier than OS to characterize a drug’s efficacy, and these endpoints are not confounded by crossover, clinical trials routinely assess treatment effects on PFS and objective response rate (ORR). Identifying the optimal algorithm to detect changes in tumor burden that correlate best with OS is particularly important in the regulatory setting to ensure that new, safe, and effective therapies are accessible to patients as soon as possible.

The criteria to characterize treatment effects on tumor have evolved to maintain or improve accuracy while limiting administrative costs and other burdens. The first generally accepted bidimensional criteria, the WHO response criteria (1981),3 was replaced by unidimensional Response Evaluation Criteria in Solid Tumours (RECIST) (2000)4; the current standard is RECIST V.1.1,5 a widely used, standardized algorithm for characterizing tumor response and tumor progression in clinical trials. Investigators, the pharmaceutical industry, and regulatory agencies accept ORR, duration of response (DOR), and PFS as assessed by conventional response criteria (eg, RECIST V.1.1) as valid measures of clinically meaningful changes in tumor burden, describing treatment effects supporting some of the new drug approvals.6

However, RECIST V.1.1 does not capture the atypical patterns of tumor response described with ipilimumab and anti-PD-1/PD-L1 antibodies. These atypical responses include initial increase in tumor size followed by a clinically important reduction in tumor burden (‘tumor flare’/pseudoprogression) or initial reduction in tumor size with appearance of new lesions that subsequently regress. To account for atypical tumor responses, several modified response criteria have been proposed, including the immune-related response criteria,7 immune-related RECIST,8 immune-modified RECIST,9 and immune RECIST.10 These criteria differ in their consideration of new lesions (ie, whether new lesions indicate progression and/or incorporation of new lesions in the measurement of tumor burden), requirements for confirmation of progression, and use of bidimensional or unidimensional measurements of tumor lesions (online supplementary table 1).

Evidence to support whether these novel response criteria better assess outcomes in patients in trials evaluating immunotherapeutics (as compared with RECIST V.1.1) is limited. To provide insight into the relative performance of iRECIST, as immune-based response criteria are increasingly employed in cancer immunotherapy trials, we conducted a retrospective assessment of response (ORR, best overall response (BOR), and PFS) according to iRECIST and RECIST V.1.1.

Kaplan-Meier25 methods were used to estimate iPFS/PFS and OS. The association between iPFS/PFS and OS was evaluated using a weighted linear regression model with weights equal to the sample size of each comparison. The strength of this association was measured by the coefficient of determination (R²) from the weighted linear regression model, where values close to 1 represent strong association and those close to 0 represent lack of association. Estimated treatment effects of iPFS, PFS and OS were calculated as the log of the HR from an unstratified Cox proportional hazards model with study arm as the covariate. The analysis for this paper was generated using SAS software V.9.4. Survival figures were generated using R V.3.4.3 and RStudio V.1.1.456.

Among the 8170 randomized patients, 4802 (59%) were randomized to receive an anti-PD-1/PD-L1 antibody; 642 (8%) were randomized to receive an anti-CTLA-4 antibody; and 2728 (33%) were randomized to receive chemotherapy. The as-treated population included 7920 patients receiving at least one dose of study-specified therapy, comprising 4751 (60%) anti-PD-1/PD-L1 antibody-treated patients, 613 (8%) anti-CTLA-4 antibody-treated patients, and 2556 (32%) chemotherapy-treated patients (figure 1).

ORR was 31.5% by iRECIST and 30.5% by RECIST V.1.1 for anti-PD-1/PD-L1 antibody-treated patients, for a difference in ORR of 1%. The differences in ORR by iRECIST and RECIST were even smaller for those treated with an anti-CTLA-4 antibody (19.7% vs 19.2%) or chemotherapy (15.2% vs 15%) (table 1). There were 232 (4.9%) of the 4751 anti-PD-1/PD-L1 antibody-treated patients who achieved an iCR or iPR (n=133 (2.8%)) or iSD (n=99 (2.1%)) after RECIST V.1.1-defined progression (table 2). Of these 133 patients with iPR/iCR, 85 (64%) also achieved a CR or PR prior to their RECIST V.1.1-defined progression, whereas 48 (36%) patients had stable (n=11) or progressive disease (n=37) as their best response by RECIST V.1.1 (ie, iRECIST-only responses) (table 3). The median DOR among all responding patients by iRECIST was 10.1 months (range: 0–33.4) and that by RECIST V.1.1 was 9.2 months (range: 0–33.3) (table 1). Among the 138 patients with a DOR that differed based on response criteria, the median iDOR was 14.1 months (range: 0–33.4) and the median DOR was 6.1 months (range: 0.9–24.9) (online supplementary figure 1).

Among the 37 patients with iRECIST responses whose BOR by RECIST V.1.1 was PD, evidence of PD by RECIST V.1.1 was based on an increase in existing lesions in 57% of patients (target lesions increase in 11 patients (30%); increase in non-target lesions in 9 patients (24%), and increase in non-target plus new lesions in 1 patient (3%)) and on an appearance of new lesions in the other 43%. The spider plots summarizing the change in tumor burden over time by type of progression are presented in figure 2A–C. As illustrated in the spider plots, all PD events occurred within 14 weeks of initiation of treatment with a median time to iUPD of 8 weeks (min, max: 1, 14).

In the analyses of PFS, 62% of those randomized to anti-PD-1/PD-L1 antibodies were identified as having PD per RECIST V.1.1 and 59% per iRECIST. Of the 2832 patients with PD per iRECIST, 61% had iUPD assigned at the time of PD per RECIST and had no further disease assessments; 18% had iUPD with subsequent disease assessments that did not confirm PD (ie, iCPD not assigned); and 21% were subsequently identified as having iCPD. In total, there were 232 patients with PD by RECIST V.1.1 who had a longer duration of iPFS based on subsequent determination of iSD/iPR/iCR at a later timepoint, resulting in either censoring at the last assessment if ongoing (n=116, figure 3A) or in a later PFS event (n=116, figure 3B) based on iCPD (n=12), iUPD at the last assessment (n=93), or death (n=11); specifically, the respective median iPFS and median PFS durations were 18.7 (min, max: 1.3, 36.1) and 5.4 (95% CI 4.0 to 6.2) months in the iRECIST-censored subgroup and were 10.1 (95% CI 8.4 to 10.9) and 2.8 (95% CI 2.6 to 4.1) months in the subgroup with PFS events documented later by iRECIST. In the overall PFS analyses, the estimated median iPFS was 4.2 months (95% CI 4.0 to 4.4) compared with an estimated median PFS of 3.9 months (95% CI 3.6 to 4.1).

In the analyses of OS by response status in all randomized patients and in anti-PD-1/PD-L1-treated patients (figure 4A,B), patients with responses only per iRECIST appear to have similar survival initially compared with those achieving PR or CR under RECIST V.1.1. Among the anti-PD-1/PD-L1-treated group, patients who achieved iSD after RECIST V.1.1 progression appeared to have similar survival initially to those achieving SD by RECIST with divergence of the curves at later timepoints. The estimated median OS time was 12.1 months (95% CI 8.9 to NR) for those with iSD only and 16.2 months (95% CI 14.9 to 18.2) for those with SD per RECIST V.1.1.

The correlations of PFS per RECIST V.1.1 and per iRECIST with OS were analyzed in the evaluation of PFS (RECIST V.1.1) and iPFS (iRECIST) as a surrogate endpoint for OS. There was a minimal increase in R² value for the association of iPFS and OS compared with that for PFS and OS (R²=0.277 vs R²=0.260) (online supplementary figure 2 a, b).

Trials intended to demonstrate the efficacy of anticancer drugs and biologics commonly evaluate tumor measurement based endpoints—such as ORR with prolonged durations of response and PFS—as the primary outcome measure and, in the context of the clinical setting of the disease, the magnitude of the treatment effect, and the risk–benefit profile, may support an accelerated or regular approval.2 There is intense interest in the oncology community for use of immune-based response criteria rather than the conventional criteria, given the observed pattern of atypical responses and that conventional criteria are reported to underestimate the ORR for cancer immunotherapeutics by up to 15%.26

Notably, the incremental percentage of immune-based responders by iRECIST is lower than that in previous reports using other immune-based criteria27 as our analysis did not consider patients with iSD following RECIST V.1.1 PD as responders, since this is not clear evidence of a drug effect and may represent the natural history of the disease in that patient.26 These analyses conducted across 14 clinical trials provides a large database in which ORR per iRECIST was similar to that per RECIST V.1.1 (31.5% vs 30.5%), a finding that was consistent across tumor types, with 2.8% of anti-PD-1/PD-L1 antibody-treated patients achieving an iCR or iPR after progression based on RECIST V.1.1. This includes the approximately 1% of patients with early RECIST V.1.1 PD who appear to derive benefit based on subsequent durable tumor responses as illustrated in online supplementary figure 1 and Figure 2 a-c. Additionally, some patients with RECIST V.1.1 responses experience iUPD followed by iSD, thereby increasing the durability of the response. Taken together, durability of response assessed by iRECIST led to a longer median DOR, a difference that was modest in the overall analysis population (~1 month) but potentially meaningful within the evaluation of specific therapeutic categories, for example, the anti-CTLA-4 antibody subgroup (~3 months increase in median DOR with iRECIST). Thus, while RECIST V.1.1 appears to capture most of the treatment effect based on ORR for anti-PD-1/PD-L1 antibodies, the use of this response criteria may result in early termination of effective treatment in a limited number of patients who may also exhibit prolonged survival similar to that observed in patients with RECIST V.1.1 response.

Patients with prolonged iSD, while not considered in the assessment of ORR, could substantially impact the assessment of treatment effects on PFS. Some patients with initial disease progression followed by disease stability are considered early progressors by RECIST but prolonged SD by iRECIST, thus improving the correlation between PFS and survival. Among anti-PD-1/PD-L1 antibody-treated patients, the estimated median iPFS was 4.2 months (95% CI 4.0 to 4.4), and the estimated median PFS was 3.9 months (95% CI 3.6 to 4.1) by RECIST V.1.1. Ultimately, PFS is a time-to-event tumor measurement-based endpoint that requires a control arm, typically in a randomized trial, to interpret the meaningfulness of a treatment effect on this outcome. The associations between iPFS and OS and between PFS and OS were weak and similar. Thus, on a trial-level basis, both tumor response criteria yield similar results. However, conclusions regarding the minimal differences in R² value when evaluating the utility of iPFS as a surrogate for OS in immunotherapy trials compared with PFS are limited by the fact that this analysis included multiple cancer subtypes (melanoma, NSCLC, RCC, and HNSCC), and PFS per RECIST V.1.1 has not been validated as a surrogate for survival in all of these cancer types. Additionally, not all studies had mature OS follow-up, potentially affecting these results.

Although this analysis includes a large number of studies and patient data, it is limited by its retrospective nature and the fact that none of the trials were conducted according to iRECIST. Collection of non-target lesion tumor data in case report forms does not provide adequate information for characterization of response status by iRECIST since subsequent increase in size, which is used to confirm disease progression, was not collected after first designation of ‘unequivocal progression’. In our analyses, we took a conservative approach, assigning confirmation if unequivocal progression was indicated at the next assessment. Additionally, measurements for new lesions were not always recorded, thus confirmation based on an increase in lesion size could not always be made. Finally, iRECIST requires that if a patient discontinues prior to confirmation (iCPD), the reason for discontinuation should be captured in case report forms, as discontinuation is medically appropriate in some settings. The number of patients without confirmation of disease progression in our analysis is considerable; 79% of patients with iUPD were not confirmed as of the data cut-off date for the study or prior to leaving the trial, including the 61% of patients with PD per RECIST V.1.1 and no further disease assessments. The effects of missing data regarding confirmation of iUPD on determination of iPFS have not been assessed in this retrospective analysis.

The use of immune-based tumor response criteria as the primary assessment of tumor measurement-based endpoints has been hampered in part by the multiple response criteria that have been proposed and variations therein, complexity of individual criteria, differential application in randomized trials where immune-based response criteria are applied only to the experimental arm (immunotherapy) and not the control arm (eg, a chemotherapeutic regimen), and increasing reliance on single-arm trials to identify clinically meaningful treatment effects on ORR and DOR. Employing novel response criteria is particularly challenging for single-arm trials as the assessment of treatment effects relies on historic controls with cross-study comparisons confounded by use of disparate response criteria. While adoption of iRECIST in immunotherapeutic trials may address some of the limitations of RECIST V.1.1 with respect to patient management—limitations that are often overcome on a case-by-case basis in current protocols—the question remains whether the additional burden in data collection (eg, measurements of new lesions, assessment of non-target growth from the previous assessment, and confirmation of progression) and data evaluation (eg, programming patient response post initial RECIST V.1.1 progression and a more complicated algorithm regarding what constitutes a confirmation of progression) is outweighed by any impact that prospective use of iRECIST will have on the interpretation of tumor measurement-based endpoints such as ORR and PFS. While the magnitudes of treatment effects on ORR and PFS, as well as associations of PFS with OS, as calculated by iRECIST or RECIST V.1.1 appear similar, the substantial proportion of missing data following RECIST V.1.1 progression in this retrospective analysis limits firm conclusions of the utility of iRECIST for evaluation of immunotherapeutics in trials intended to support marketing authorization.

In prospective trials proposing the use of an immune-based tumor response criteria such as iRECIST for the determination of the primary endpoint, the trial design, trial conduct, and analysis plan would need to minimize and address potential bias introduced with the use of such criteria. Given its widespread acceptance, iRECIST has the potential to unify how data are collected and evaluated across both single-arm and randomized trials, providing a new standard for tumor response assessment across trials.