Mutation calls generated using Multi-Center Mutation calling in Multiple Cancers (MC3) WES data from TCGA project were used for this analysis.37 Variants that overlapped with the CCDS, using bedtools (-wa option)38 were extracted from the publicly available mc3.v0.2.8.PUBLIC.maf file (https://gdc.cancer.gov/about-data/publications/pancanatlas). Finally, the data were filtered for any overlap or redundancy using the ‘merge’ function. The consortium created a final bed file that covered 32.102 Mb of the genome after intersecting the data found in the MAF files and filtering for any overlap or redundancy (see online supplementary methods). The final bed file size was used as the denominator for calculating WES TMB in this study. Three different consortium laboratories independently calculated WES TMB using the same dataset and analytical methodology with 100% concordance.

Ten thousand two hundred ninety-five tumor samples with matched normal initially composed part of the cohort. Only samples with at least one variant which PASSED variant review filter were used (see online supplementary methods for variant quality filters). Low quality samples based on variant filters and those with low purity were also removed from further analysis. The remaining cases (n=8291) were randomly assigned to training (n=4157) and validation (n=4134) datasets with similar median candidate mutations and cancer types (online supplementary figure 2). Participants, though not required, could use the ‘training’ set for their own algorithm or parameter testing. However, all analyses described herein were conducted using the validation dataset.

The evaluations reported in the present study are those comparing panel TMB to WES TMB on the validation set, with no adjustments made to the panel TMB algorithms once the validation set analyses began. All analyses focused on tumors for which WES TMB was ≤40 because >98% of the TCGA dataset tumors investigated had TMB ≤40 in the TCGA dataset and all members of the consortium agreed that this range would have the greatest relevance for clinical decision-making. Of the 4134 tumors initially represented in the validation set, 4065 remained after excluding those with WES TMB >40. All results were blinded to the entire consortium, with the exception of the project statistician and data manager (LMMS and DMM) who were regarded as neutral parties not affiliated with any of the participating laboratories.

Statistical analyses interrogated the relationship between WES TMB and panel TMB values. The first analysis focused on the combined data from all 32 tumor types. Spearman’s R correlation values were calculated, and scatterplots and difference plots were created to assess linearity of the relationship between panel TMB and WES TMB and to evaluate whether variance of panel TMB was constant across the range of WES TMB values.

Next, the 32 tumors were divided into three strata according to the number of samples within each tumor type that had TMB values spanning the range 0–40 mut/Mb (see online supplementary methods and figure 3). Stratum 1 contained eight tumor types (see online supplementary table 1A—stratum 1) displaying a good distribution of TMB values spanning the range of interest (0–40 mut/Mb). Seventy-seven per cent of samples (1257/1627) had TMB ≤10 mut/Mb, 19% (306/1627) had TMB 10–40 mut/Mb and 4% (64/1627) had TMB ≥40 mut/Mb and were thus eliminated from further analyses. Stratum 2 was represented by 11 tumor types (see online supplementary table 1B—stratum 2) whose samples had generally low TMB values (≤10 mut/Mb, 98%, 1723/1754), and only 1.5% (26/1754) of samples had TMB 10–40 mut/Mb. Only five samples (0.29%) had TMB ≥40 mut/Mb and were thus eliminated from further analyses. Stratum 3 was represented by 13 tumor types (see online supplementary table 1C—stratum 3) whose samples had very low TMB values (≤5 mut/Mb, 99.5%, 749/753) and only 4 samples (0.5%) had samples with TMB between 5 and 10 mut/Mb. Regression modeling using weighted least squares was implemented to account for the heteroscedasticity in errors, referring to the variability in panel TMB values about the fitted regression line, which was observed to increase with the mean and with WES TMB. This modeling was conducted for all strata, although we focused on stratum 1 considering strata 2 and 3 provided less stable and unreliable estimates due to the large number of samples that concentrated in the lower end of the TMB range.

For each regression, the mean panel TMB was modeled as a simple linear function of the WES TMB, and five different models for the error variance were considered (see online supplementary methods). Restricted maximum likelihood analysis using the gls function available in the R package nlme was performed to estimate the model parameters and select a best fitting variance structure based on minimum Akaike information and Bayesian information criteria.

The whole exome analysis of the TCGA MC3 validation dataset used an agreed on methodology to calculate the WES TMB values, termed the Uniform TMB Calculation Method (see online supplementary table 2). The goal of phase I of this harmonization study is to assess the theoretical variability across panels. Given that the participating panels were at different stages of development and had different sensitivity levels, the consortium decided to use the Uniform TMB Calculation Method, which would enable the selection of high-quality variants that all laboratories were able to assess as part of their panels. The consortium created a custom bed file covering 32.102 Mb of the genome which was used to calculate the reference WES TMB values. The calculated WES TMB values comprised the reference dataset for this study. The uniform method for analysis of WES TMB included minimum thresholds for median target coverage (median 300X as this was identified as the point where sensitivity for the lower allele frequency variants drops drastically) (see online supplementary figure 4), variant allele frequency (≥0.05), read depth (≥25) and variant count (≥3), and synonymous variants were excluded.

Each participating laboratory calculated TMB from the subset of the exome restricted to the genes covered by their targeted panel and using their own unique bioinformatics pipeline (panel TMB). If available, the laboratory’s bioinformatics analysis has been reported in table 1.

The panel-derived TMB datasets were sent to a neutral third party (DMM) who assigned coded identifiers to the laboratories to mask which laboratory contributed each dataset. All subsequent data analyses were conducted by LMMS and DMM. Participating laboratories were not involved in the analyses and were not provided the key to the coded lab identifiers.

Eleven academic and commercial laboratories with targeted gene panels in different stages of development participated in this study (table 1). The size of the coding region used to estimate TMB from these gene panels ranged between 0.80 and 1.72 Mb. And the number of genes in each of the gene panels ranged between 324 and 607 genes. All participating laboratories included exonic somatic non-synonymous, frameshift and splice site variants and short indels when estimating TMB. Eight panels (8/11, 73%) also included synonymous variants in their estimation. Each laboratory used their own bioinformatics algorithms and workflows, which were optimized using the sequencing methods, mutation types and filters that best suited their own panel specifications. Since the participating panels were in different stages of development, only a few had published panel performance characteristics (table 1).

The WES TMB values were calculated using the TCGA MC3 Mutation Annotation Format (MAF) validation dataset and an agreed on methodology (see ‘Whole exome analysis’ section and online supplementary table 2). The panel TMB values on the same validation dataset were estimated by down-sampling to the regions covered by each of the laboratories’ panels and applying their own bioinformatics algorithms. To prevent the misinterpretation of this study’s results as an interlab performance study, all laboratories agreed for the results to be blinded with respect to the lab generating each dataset.

First, all 32 cancer types in the TCGA MC3 dataset were investigated together using weighted linear regression analysis (generalized least squares, see ‘Methods’ section). Some variation was observed across panels, with Spearman’s rank correlation values (R) ranging from 0.79 to 0.88, and slope values ranging from 0.87 to 1.47 (figure 1A, online supplementary figure 5). Eight laboratories (73%) had slope values >1, demonstrating an overestimation of TMB. Panel factors that may influence TMB overestimation were not assessed due to the blinded study design but may have included the type of mutations counted for the panel TMB value (eg, synonymous alterations included in panel TMB that were excluded from the WES estimation), among others.

A limitation of analyzing all cancer types together is the variable distribution of TMB across different cancer types, with some cancer types displaying large dynamic ranges of TMB values up to several hundred mutations per Mb and others with very limited distributions with very few samples reaching 20 mutations per Mb (see online supplementary figure 3). To account for this limitation, cancer types were categorized into strata by their distribution of WES TMB values. Stratum 1 (n=1563 samples with <40 mut/Mb) had samples with a good distribution of WES TMB values covering 0–40 mut/Mb, which enabled a more robust regression analysis across a clinically relevant TMB range. The eight cancer types in stratum 1 were: bladder urothelial carcinoma (BLCA, n=195), colon adenocarcinoma (COAD, n=128), head and neck squamous cell carcinoma (HNSC, n=232), lung adenocarcinoma (LUAD, n=228), lung squamous cell carcinoma (LUSC, n=228), skin cutaneous melanoma (SKCM, n=166), stomach adenocarcinoma (STAD, n=189) and uterine corpus endometrial carcinoma (UCEC, n=197).

Regression analyses restricted to stratum 1 tumors revealed an association between WES TMB and panel TMB similar to that for all cancer types analyzed together (Spearman’s R: 0.81–0.90 and slope 0.80–1.32, figure 1B, per laboratory online supplementary figure 6 and table 3). The slopes calculated when stratum 1 tumors were analyzed were consistently lower than when all cancers were analyzed. The greatest differences in slope values when comparing slopes estimated for all cancers and for stratum 1 tumors only, were observed for labs 8 (all cancers 1.47 vs stratum 1 1.32) and 9 (all cancers 1.24 vs stratum 1 1.1) (both ∆ 0.15), while labs 4 (all cancers 0.904 vs stratum 1 0.897) and 2 (all cancers 1.087 vs stratum 1 1.076) had the least differences (∆ 0.007 and 0.01, respectively). When stratum 1 tumors were analyzed, only six laboratories (55%) reported overestimation of TMB with slope values >1.

Regression analyses with stratum 2 and 3 were not robust, as the WES TMB values did not adequately cover the entire clinically meaningful range (see online supplementary figures 7 and 8, and table 3).

Lastly, the eight cancers in stratum 1 were analyzed separately. UCEC, BLCA and COAD had the broadest range of slope values (UCEC: range 0.755–1.602, ∆ 0.847; BLCA: range 1.042–1.79, ∆ 0.748; COAD: range 0.75–1.486, ∆ 0.736) (figure 2, online supplementary table 4), and most laboratories consistently overestimated these cancer types, with BLCA as the only cancer type for which all 11 laboratories (100%) consistently overestimated their panel TMB values relative to WES TMB. Conversely, LUAD, LUSC and HNSC had the tightest range of slope values with no consistent bias to overestimating or underestimating TMB (LUAD: range 0.817–1.135, ∆ 0.318; LUSC: range 0.741–1.099, ∆ 0.358; HNSC: range 0.854–1.244, ∆ 0.39).

Prediction limits for the observed panel TMB at fixed WES TMB (5, 10, 15 and 20 mut/Mb) were calculated to quantify the variability around the regression line at those selected WES TMB values. The limits were designed to capture approximately 95% of the panel TMB values expected to be observed at a given WES TMB. Some laboratories had consistently tighter (narrower) prediction intervals, while others demonstrated more variability (wider intervals), but for all laboratories, the prediction intervals became tighter with decreasing WES TMB value, indicating greater variability in panel TMB at larger WES TMB values (figure 3). Generally, the prediction intervals observed for each participating laboratory were similar, with laboratories demonstrating intervals that spanned as small as ±4.7 mut/Mb or as large as ±12.3 mut/Mb when the WES TMB was 10 mut/Mb, which is a TMB threshold that has been previously used to define a TMB-high cohort using NGS panels.33 When prediction limits were assessed by strata, the variability of the intervals was very large for cancer types in strata 2 and 3 compared with stratum 1 because most TMB values for the cancers in these strata accumulate in the lower end of the TMB spectrum, thus resulting in more uncertainty in the fitted regression lines and wide scatter in panel TMB values around those lines (see online supplementary figure 9). When the eight stratum 1 cancers were analyzed separately, prediction intervals at the discreet value of WES TMB=10 mut/Mb were observed to be wider for BLCA and UCEC, while LUAD, LUSC, HNSC and SKCM had the tightest intervals (figure 4). This is similar to the observed variation in fitted regression lines for BLCA and UCEC across laboratories (figure 2). The theoretical variability around the regression was also seen to increase (wider intervals) with increasing TMB value in individual cancer types (see online supplementary figure 10).

The current practice of reporting WES-derived TMB values as number of somatic mutations, while panel-derived TMB values are reported as a density of somatic mutations per Mb of genomic region covered by the panel (mut/Mb), precludes the aggregate analysis of TMB being derived from WES or targeted panels, especially since the size of the exome interrogated using different platforms may not be consistent. Reporting TMB as mutations/megabase (mut/Mb) in order to keep these values consistent and comparable is recommended by the consortium.

Size of targeted gene panel, technical sensitivity of the assay and pre-analytical and analytical variables are known to contribute to variability in panel-based TMB estimates.33 The same in silico data were used by every participating laboratory, which created a theoretical setting that focused the investigation on potential sources of variability that are unique to the technical specifications of the panel (eg, size and composition), and the bioinformatics approaches of each laboratory (eg, mutation types counted and germline and hotspot mutation filtering). Some of these factors cannot be easily modified and standardized across laboratories as panel assays are, for the most part, proprietary and have been designed to optimize their respective technical specifications and conditions. However, harmonization of TMB estimates may be achieved across laboratories by ensuring that the analytical validation studies for each panel follow a standard approach including alignment of panel TMB values to an external reference standard. Recommendations for analyzing accuracy, precision and sensitivity of TMB values to tumor content when used both as a continuous score and a categorical call have been proposed by the consortium (table 2). These recommendations will ensure that regardless of the type of panel or bioinformatics pipeline a laboratory decides to use, TMB estimates are held to a standard of acceptable reliability.

Comparison to WES TMB is currently the most recognized way to determine accuracy of panel TMB. However, it should be noted that differences in performance between panel TMB and WES TMB are to be expected based on differences in coverage depth between the two methods, with typically greater depth and higher variability observed in panels.

Universal reference standards, with TMB values spanning a clinically relevant range (eg, 0–40 mut/Mb), represent a promising tool to achieve alignment or calibration in order to ensure consistency of the TMB estimation across platforms, regardless of known sources of variability. An ideal reference standard for TMB estimation should be generated from a renewable source and its TMB values should be calculated using WES. To mitigate differences resulting from comparisons using multiple different WES assays, a universally accepted, predefined bioinformatics pipeline and statistical methods should be implemented. A calibration curve generated using the reference standard should be used to normalize and compare across panels, which should promote alignment and aid in the analytical validation of panel TMB values.