Would Our Treatment Decisions Be Better Justified in the Absence of Observational Data?

In the May 2011 issue of the American Journal of Neuroradiology, Raymond et al¹ scrutinized at length the methodologic weaknesses of 2 recent articles about the natural history and risk of hemorrhage in brain arteriovenous malformations (AVMs), one from our department² and the other from Toronto Western Hospital.³ While part of their criticism is undeniable—such as inevitable patient selection bias in any observational study concerning a life-threatening disease or the effect of the choice of variables and statistical methods on the observed outcome—the main conclusion of their denunciatory analysis leaves us perplexed. The authors state that studies like this should not be used to inform clinical decisions and that relying on prognostic estimates based on these studies can be dangerous.

The question that remains unanswered, then, is, “What should we base our treatment decisions on at present?” As of today, no results from a randomized clinical trial concerning the treatment of brain AVMs exist. A Randomized Trial of Unruptured Brain AVMs is a laudable effort to shed more light on the issue and is underway, but we will probably have to wait for another decade before having conclusive results from that at our disposal. It will also tell us nothing about the behavior of ruptured AVMs, and it is doubtful that a randomized trial on ruptured AVMs will ever be conceived. Moreover, while the value of randomized controlled trials (RCTs) as the criterion standard to prove or disprove the effectiveness of a treatment cannot be disputed, they are unfortunately not immune to many of the same pitfalls the authors blame on observational studies, especially in cases of complex, rare, dangerous, and invasively treated diseases. Considerable subject selection bias makes them poorly generalizable to all patients, the cohort sizes and follow-up times are not likely to be more extensive than those in observational studies, results vary from center to center, disease progression may lead to crossing over to another treatment arm or termination of the study for many patients, and the choice of variables and statistical methods is based on educated guesses just like in retrospective studies. We are not saying that RCTs are unnecessary; we are just wondering whether our treatment decisions would, at present, really be less “dangerous” if we did not have even the observational data from historical cohorts to inform us?

While the authors are entitled to their opinion, there were, however, some factual errors in their analysis that need to be rectified.

• 1) The authors state that the overall hemorrhage rates differ by a factor of 2 in these 2 reports.^2,3 A closer look reveals that the average hemorrhage rate in our study for the first 5 years of observation was 4.7% per year (Table 2²), very similar to 4.6% per year in the Toronto study³ with a mean follow-up period of 2.9 years. This comparison is more meaningful than using the annual hemorrhage rate of 2.4% derived from the whole follow-up period of our study (with a mean duration of 13.5 years), especially because both studies observed and reported a decline in the hemorrhage risk with time.

• 2) Concerning the beginning of the follow-up, the authors claim that “In both studies, it is unclear when the clock started (first admission, diagnosis, referral?).” In our report, it is stated explicitly in the “Methods” that “follow-up data were collected starting from the admission to a neurosurgical referral center.”²

• 3. The authors claim that “Neither article presented any confidence intervals around their estimates of risk [of hemorrhage].” We have reported 95% confidence intervals both for all cumulative hemorrhage risk estimates derived from Kaplan-Meier life-table analyses (Table 2) and for relative risk ratios based on Cox proportional hazards uni- and multivariate models (Table 3²).

• 4) The authors state that “Furthermore, extrapolation of risks observed during a relatively small number of years to lifetime risks by multiplying the observed rate by the number of years the patient is expected to live is, to say the least, uncertain.” While we neither suggested nor performed such an arithmetic exercise (the cumulative rupture rates in our report are real observational data subjected to life-table analyses), we would still like to remind the authors that the annual probability of a certain outcome (eg, hemorrhagic stroke) should never be multiplied by years at risk; the proper formula to estimate the cumulative probability of the outcome is 1-(1-p)^t, where p = the annual probability of the outcome and t = time at risk in years, given that the risk remains constant with time (which does not seem to be the case for AVMs).

Despite the obvious shortcomings of observational analyses of historical cohorts and the complexity of AVMs as a disease, we still honestly believe that the results from our study, as well as from similar ones performed by others, will lead to better informed treatment decisions than complete lack of knowledge. It is true that the data from AVM natural history studies are not unequivocal in terms of all hemorrhage risk factors or for annual risk rates; thus, independent replications in different cohorts are important. Moreover, perhaps depending on the eye of the beholder, the results from different cohorts for many risk factors are not that divergent, after all, and eventually common trends will emerge (eg, our recent review⁴).

• © 2011 by American Journal of Neuroradiology