Predictive Utility of Marketed Volumetric Software Tools in Subjects at Risk for Alzheimer Disease: Do Regions Outside the Hippocampus Matter?

Discussion

Our study, with the complete set of raw volumetric measures available in 2 fully automated, commercially available brain volumetric software packages, confirms previous studies documenting the prognostic utility of hippocampal volume in patients with MCI. Indeed, hippocampal volumes provided the highest prognostic value of all individual regions available in both the NQ and NR volumetric reports. Multivariable analysis, including the use of a cross-validated machine-based learning classifier algorithm, which incorporated other brain regions available in these software packages, did not provide additional predictive value compared with a model based on just hippocampal volumes. The results of our study are in agreement with previous studies that have found hippocampal volumes to be most predictive of conversion of MCI to AD, with little added benefit from additional brain regions. For example, a large study of patients in the ADNI using semiautomated methods reported that hippocampal volumes were the most predictive of conversion compared with other regional and whole-brain measures.²⁹ Automated volumetric measurements of the hippocampus also had high predictive values for conversion.² A 2-year clinical follow-up study of patients with MCI with manual methods demonstrated that baseline hippocampal volumes had high predictive value.⁵ Despite the differences in segmentation techniques (manual, semiautomated, and automated) among the 3 studies, all these studies reported that hippocampal volumes were the most predictive of MCI conversion.

On the other hand, a number of previous studies have implicated brain regions other than the hippocampus as either adding to the prognostic value of hippocampal volume or having prognostic value in their own right, though direct comparison with our study is limited due to differences in methodologies and primary outcomes. In one study using manual methods, a bivariate model of hippocampal volumes and follow-up changes to either ventricular volume or whole-brain volume was shown to have prognostic value in predicting conversion of MCI to AD.⁷ Another study with automated methods found that amygdala and caudate nucleus volumes were predictive of MCI conversion, whereas hippocampal volumes were not.¹² A separate study with automated methods reported that deep gray matter structures, including the amygdala, thalamus, putamen, and nucleus accumbens, were predictive of conversion of MCI to AD, in addition to the hippocampus.³⁰ Temporal horn volumes were shown to be more predictive than hippocampal volumes in one study using semiautomated methods,³¹ whereas the left lateral temporal lobe and left parietal cortex were the most predictive factors in another study using automated methods.³²

A likely reason for discrepancies with these studies is that volume loss in AD-affected regions is correlated with that of the hippocampus, and these collinear effects are therefore lost in the multivariable model when one accounts for hippocampal volume. For example, in our study, volume of the hippocampus was highly correlated with that of the amygdala (r = 0.71) and cortical gray matter (r = 0.52) for NQ, and amygdala (r = 0.75) and temporal lobe (r = 0.57) for NR. Another explanation for the differences in results, compared with our study, could be the use of different segmentation methods. In our study, we used an automated segmentation method with NQ and NR, which may be less accurate in measuring smaller and deeper structures compared with manual or semiautomated segmentation methods.

Also of note in this study, the HOC score, a composite index of hippocampal and temporal horn volume, did not outperform hippocampal volume alone. The HOC score has been advocated as a more accurate measurement of hippocampal tissue loss, which accounts for individual variations in hippocampal size by accounting for temporal horn volume. In our study population, temporal horn volume was somewhat collinear with hippocampal volume (r = −0.33), and we did not find the HOC to be more predictive of conversion than hippocampal volume alone, suggesting that the additional variance added by temporal horn volume might be redundant information and/or noise. Although temporal horn volume was a significant predictor of conversion in the univariable logistic regression model, it lost significance when hippocampal volume was added to the model and negligibly increased the AUC. Nevertheless, our results do not exclude the possibility that combining these 2 measures may be helpful in the preclinical or mild dementia phases of the disease spectrum.

Our study has several limitations. It was theoretically possible that the ROC area for hippocampal volume might have been larger if hippocampal volumes were normalized to intracranial volume and adjusted for age.³³ Such corrections might adjust for bias across subjects since ICV-normalized and age-adjusted values are already available in the volumetric reports. Nevertheless, for our primary analysis, we chose to use the raw values, in keeping with our purpose of comparing single-versus-combined measures within the same subjects, which should not be affected by normalizing by intracranial volume, a practice that could introduce additional noise into the measures. Although age may differentially affect various brain regions, we did not have age-adjusted values available for most regions in the NQ volumetric report and therefore did not include these measures in our primary analysis. Even so, secondary analyses of hippocampal volume with intracranial volume normalization, as well as including age as a covariate in the linear model, did not yield additional prognostic efficacy. Of note, 1 group, the Coalition Against Major Diseases, reported higher AUC values for the prediction of MCI conversion to AD (in 2 years) in their de novo analysis, based on automated hippocampal volume obtained by several methods: 0.7565 (LEAP; Learning Embeddings for Atlas Propagation, http://www.ixico.com/additional-information/leap-analysis), 0.7516 (NeuroQuant), 0.7536 (FreeSurfer; http://surfer.nmr.mgh.harvard.edu), and 0.7290 (HMAPS; Hippocampus Multi-Atlas Propagation and Segmentation).³⁴ However, the same committee also reported a range of AUC values between 0.60 and 0.77 based on their literature review, and our results are within this range.

Both the NQ and NR programs provide a large number of data values (55 for the general morphometry report through NQ, 140 for the NR report) that we did not fully include in our machine-based learning classifier program. Increasing the number of features can hurt the performance of a classifier, given a limited sample size. Initially, we found that the addition of the full dataset resulted in a worse predictive value for MCI conversion, which we speculate was due to overfitting of the training samples. This is a common phenomenon when the number of predictor variables is high compared with the number of subjects in the training set.³⁵ Hence, we decided to limit our analysis to the most promising features provided in the NQ and NR volumetric reports.

In some cases, we were not able to process the MR images for the baseline sequence, which excluded some patients from the study, all of whom were chosen through the ADNI data base. In total, 89 of 281 patients were excluded due to inadequate NQ or NR morphometry data. This was most often due to patient age and sex missing from the anonymized header information, a requirement of the NQ processing pipeline but a situation unlikely to occur in clinical practice because these are standard DICOM data fields. In addition, some patients had inadequate MPRAGE sequences due to technical factors, including motion, which we were not able to segment by using the NQ or NR programs; in these cases, we substituted a repeat MPRAGE sequence. Despite occasional differences in our ability to process a particular sequence, we found a strong linear correlation (r > 0.60) between NQ and NR for all compatible regions tested. Of interest, for hippocampal volume, there was a small underestimation bias for NR with respect to NQ, which may be attributable to different segmentation algorithms. Two outliers were noted, showing differences of approximately ±3 mL between software packages, which were attributable to segmentation differences.

Another limitation is that we did not incorporate biomarkers, neuropsychological assessments, or longitudinal imaging measures in our study. Rather, we sought to limit our study to testing the prognostic efficacy of 2 commercially available brain volumetric software packages in their own right, rather than by using additional data, which might confound direct comparisons. Of note, the Mini-Mental State Examination and ADAS-13 AUC scores were as high as, or higher than, those of hippocampal volume alone, in agreement with previous studies looking at the prognostic efficacy of multiple biomarkers in MCI.³⁶ This result is not surprising given that a dichotomous end point measure, conversion, was used to assess cognitive decline, and cognitive measures at baseline would be expected to strongly predict how close a subject is to the point of conversion. Future effort in evaluating the prognostic efficacy of these software packages should involve combining volumetric data with a suite of biomarkers and neuropsychological assessments by using machine-based learning approaches and continuous, rather than categoric, outcomes of cognitive decline. Subject factors such as financial or socioeconomic ones also remained unadjusted in the analysis. These factors would unlikely affect the within-subject design and were omitted to keep the comparison straightforward between single-versus-multiple volumetric outputs directly available from the imaging software packages.

Despite these limitations, our study had several strengths, including a large sample size of 192 patients through the ADNI data base. Indeed, ADNI uses a wide variety of vendor platforms, field strengths, and harmonized pulse sequences that patients are likely to encounter in future clinical settings. We also selected patients with 3-year clinical follow-up, which provided a more accurate designation of future conversion status, compared with studies with shorter follow-up duration.