Abstract
BACKGROUND AND PURPOSE: The role of flow diversion in the management of aneurysms remains unknown. We sought to evaluate the community agreement regarding indications for flow diversion.
MATERIALS AND METHODS: A portfolio of 35 difficult aneurysm cases was sent to 40 clinicians with varying backgrounds and experience. Responders were asked whether they considered flow diversion a treatment option, whether other options were possible, whether recruitment in a randomized trial would be considered, and to select their final choice. Agreement was studied by using κ statistics.
RESULTS: Decisions for flow diversion were more frequent (n = 300, 39%) than decisions to coil (n = 163, 21.2%), to observe (n = 121, 15.7%), to occlude the parent vessel (n = 102, 13.2%), or to clip (n = 66, 8.6%). Sidewall aneurysm morphology was associated with flow diversion as the final choice (P = .001). Interjudge agreement was fair at best (κ <0.3) for all cases and all judges, despite high certainty levels (range, 7.2–8.9 ± 2.0 on a 0–10 scale). Agreement was no better within specialties or with more experience. All patients were judged to have other treatment options. Judges were willing to offer trial participation in 417 of 741 (56.3%) scenarios, more frequently when the aneurysm was sidewall (P = .001) or in the anterior circulation (P = .028).
CONCLUSIONS: Individuals did not agree regarding the indications for flow diversion. There is sufficient uncertainty to justify trials designed to protect patients from the potential risks of premature adoption of an innovation.
ABBREVIATIONS:
- BST
- best standard treatment
- CL
- confidence level
- FD
- flow diversion
- FIAT
- Flow diversion In Aneurysm Treatment
- RCT
- randomized controlled trial
Flow diversion (FD) of intracranial aneurysms is an innovative therapeutic approach.1⇓–3 The introduction of surgical innovations involves a process of trial and error when entering unknown territory.4 The early adoption of innovations can result in spectacular advances or unexpected complications: FD has been shown to be capable of arterial reconstruction in patients with giant aneurysms who previously might have been treated only with parent vessel occlusion, but FD has also been associated with the rupture of unruptured aneurysms.5,6 Using flow diversion also means working with new devices, which may also introduce unexpected adverse events. For example, embolism of material used to coat endovascular delivery tools has been implicated in the occurrence of fatal parenchymal hemorrhages, occurring at a distance from the aneurysm.7
Although there have been enthusiastic proponents for early adoption of FD,8⇓⇓–11 others have urged restraint,12,13 at least until reliable information becomes available. To minimize the potential harm associated with innovations, progressive steps have recently been proposed, including the careful selection of well-informed participants, when interventions are still considered “experimental.”14 Important questions that immediately arise are the following: 1) In what context are these experimental interventions to be offered, care or research? and 2) On what basis will patients be selected?
The approach regulatory agencies used to approve the first flow diverters was a registry of consecutive but carefully selected patients.1 Alternatively, beginning with a randomized trial would have been a prudent option.12 Long ago, Chalmers15 suggested that we should “randomize the first patient” to give each patient a chance to receive the best therapy until the best therapy is identified. In these situations, we have proposed care trials.16 The Flow diversion In Aneurysm Treatment (FIAT) study was conceived and designed to guide the ethical introduction of FD into daily clinical management of patients with difficult aneurysms.17 FIAT is meant to be offered to any patient for whom the use of FD is contemplated.
Should patients receive FD in the context of a registry or of a trial? It still remains unclear to whom FD should be offered. By definition, no one really knows because a proper comparative evaluation of FD efficacy and safety has yet to be performed for any group of patients.
To help with this dilemma, we sought to explore the opinions of clinicians and study agreement within the neurointerventional community regarding the use of FD. Furthermore, a survey of opinions may inform trial design considerations.
Materials and Methods
This study, which involved anonymous colleagues questioned on images of anonymous patients, was considered by our institutional review board committee and approval was deemed to not be required.
Patients
A portfolio of 35 clinical cases of difficult aneurysms was prepared, along with a short clinical vignette, such as the following: a 53-year-old man, 43-mm unruptured cavernous ICA aneurysm, ophthalmoplegia, passed test balloon occlusion. Respondents were asked a series of 5 questions: 1) whether they considered FD (± coils) to be a treatment option; 2) whether there was another treatment choice they would conceivably offer the patient (respondents could select from the following: i) no, FD ± coils only, ii) coiling ± stent placement, iii) parent vessel occlusion ± bypass, iv) surgical clipping, or v) observation); 3) whether they would consider offering the patient a randomized trial given a 50% chance of treatment with FD or a 50% chance of “best standard treatment” (BST), 4) their best final choice (same options as question 2), and 5) their confidence level regarding their final choice on a scale of 0–10 (Fig 1).
Example. One of the 35 cases, corresponding vignette, and survey questions that were presented to judges is illustrated.
All cases were selected either from the FIAT trial (n = 14), comparing FD with BST, or were patients treated with FD outside the randomized controlled trial (RCT) (n = 21), but raters were not informed of the source of the cases. All cases had thus been judged by at least 1 clinician to be a candidate for FD. In reality, 28 of the 35 patients had been treated with FD. The 7 other patients had also been judged appropriate for FD but had been allocated to BST. Our aims were the following: 1) to ensure that we had included cases appropriate for flow diversion, 2) to assess whether other treatment options would be chosen as preferable or possible, and 3) to assess the community willingness to participate in a trial such as FIAT.
Raters
The e-mailed or paper portfolio (On-line Appendix) was addressed to 40 clinicians, selected because they had presented or published on flow diverters or were on a list of potential FIAT participants, and all received responses were included. We did not require participants to have performed a minimum number of FDs. In the end, the survey was independently evaluated by 22 practicing interventionists with the following demographic information: 16 radiologists and 6 cross-trained neurosurgeons, able to deliver surgical and endovascular care. Respondents were from academic centers in North America (Canada and the United States) (n = 16) or Europe (n = 6). Clinicians had been in practice for 0–5 years (n = 6), 5–10 years (n = 3), or >10 years (n = 13). Nine judges (41%) had previous experience in >15 patients with flow diverters. Repeatedly responding to the same questionnaire can be a painful experience. Four senior raters (with >20 years of experience), selected on the basis of previous mutual collaborations, were commissioned to respond to the questionnaire twice, >3 months apart, to generate the intraobserver evaluations.
Statistics
κ statistics were calculated by using SPSS software (Version 20; IBM, Armonk, New York). The κ values reflect agreement beyond chance, with a score of zero no better than flipping a coin, 1 representing perfect agreement, and −1 representing perfect disagreement. κ values were interpreted according to Landis and Koch18: ≤0.0, poor; 0–0.2, slight; 0.21–0.4, fair; 0.41–0.6, moderate; 0.61–0.8, substantial; and 0.81–1.0, almost perfect agreement. Confidence in decision-making along a scale of 0–10 was analyzed with ANOVAs. Multiple comparisons among confidence levels according to treatment choices were adjusted according to Bonferroni. A Mann-Whitney or Kruskal-Wallis test was done (according to the number of categories of the independent variable) to compare the distribution of FD choices among different groups of aneurysms or patients (extra-/intradural, anterior/posterior circulation; neck size; randomized in FIAT or not; and sidewall versus bifurcation versus fusiform aneurysms) or different groups of raters (background; FD experience or years of experience). Exploratory comparisons between proportions of treatment choices for patients in FIAT or not in FIAT were performed by using χ2 tests and among mean confidence levels by using paired Student t tests. χ2 tests were also used to compare willingness to recruit patients in a RCT, adjusted for multiplicity according to Bonferroni.
Results
Patient and aneurysm characteristics are presented in Table 1.
Patient and aneurysm characteristics
Final Management Decisions with Confidence Levels (Questions 4 and 5)
Of 752 responses (97.7% of 770 possible responses; 2.3% missing responses) to question 4, which asked clinicians to make a final choice, decisions to use FD were more frequent (n = 300, 39%) than decisions to coil (n = 163, 21.2%), to observe (n = 121, 15.7%), to occlude the parent vessel (n = 102, 13.2%), or to surgically clip (n = 66, 8.6%).
Clinicians were generally confident that they had selected the best option (mean confidence levels [CLs] between 7.2 and 8.9). CLs were not significantly different for radiologists or surgeons (P = .09), for more or less experienced clinicians (P = .22), or for those having more or less FD experience (P = .11). CLs varied according to treatment choices (P = .012); clinicians showed more confidence when they chose observation (8.9) than FD (7.7, P = .043). FD was considered the best option by at least 1 clinician (minimum = 1, maximum = 18) for each case, but the same treatment option was not chosen unanimously for any single case. The most common final choice was different overall for patients in FIAT compared with those patients not randomized to FIAT: Clinicians favored FD in 36.6% of patients randomized to FIAT, whereas 42.2% of those not randomized to FIAT were selected for FD. The main difference was that responders chose observation more frequently for patients randomized to FIAT (23.5%) than for the patients treated outside the trial (11.0%) (P = .0001). The only characteristic associated with a FD choice was sidewall aneurysm morphology (P = .001). For 11 (31%) of 35 cases, FD was selected most frequently as the best final treatment choice. Ten of 11 (91%) of these cases were proximal carotid aneurysms (perhaps corresponding to American on-label use); the other was a 38-mm basilar trunk sidewall aneurysm (Fig 2).
Responses to question 4 (best final management choice). FD was commonly selected as the best final treatment choice for anterior circulation (A, 9-mm asymptomatic paraophthalmic aneurysm; 10 votes for FD) or sidewall aneurysms (B, 38-mm asymptomatic sidewall basilar trunk aneurysm; 12 votes for FD).
Interrater agreement regarding the best treatment choice was “poor” to “slight” for all clinicians (in the range of κ = 0.0–0.2) and no better among neurosurgeons or among radiologists or when responders were experienced or had performed >15 FD treatments (Table 2). Agreement was no better when answers were dichotomized (FD versus any other option). Intrarater agreement at least 3 months apart was better (κ = 0.387–0.634) (Table 2).
Inter- and intraobserver agreement regarding question 4 (best final management choice)
Is FD an Option for This Case? (Question 1)
FD was always an option (mean, 16 ± 4 positive responses to question 1; minimum = 7, maximum = 21 of 22 responders). Anterior location (P = .025) and sidewall morphology (P = .001) were aneurysm characteristics associated with considering FD as a treatment option.
The 2 cases with the highest number of dissenting raters, who did not consider FD to be an option, were the following: an incidental 20-mm MCA aneurysm in a 56-year-old woman (case 20, n = 15 negative answers) and a 67-year-old woman with headaches and a 38-mm unruptured basilar tip aneurysm (case 10, n = 14). On the other hand, there were 6 cases for which a single “No, FD is not an option” response (of 22 raters) was obtained (Fig 3).
Responses to question 1 (Is flow diversion an option?). FD was considered an option by at least 7 judges in all cases. The cases in which flow diversion was most frequently judged not to be a viable option were a 20-mm MCA bifurcation aneurysm (A, 15/22 “no” votes) and a 38-mm basilar tip aneurysm (B, 14/22 “no” votes). The cases in which flow diversion was least frequently thought not to be an option, with only 1/22 “no” votes each, were a 22-mm ophthalmic segment aneurysm (C) and a recurrent previously coiled carotid bifurcation aneurysm (D).
The judgment of whether FD was a treatment option did not feature significant agreement for any group of raters or for any group of patients (Table 3).
Interrater agreement regarding question 1 (Is FD an option in this case?)
For This Patient's Aneurysm, Are There Treatment Alternatives? (Question 2)
Treatment options other than FD were considered viable by at least some clinicians for all 35 cases. The greatest number of clinicians who judged that FD was the only possible option was 5 of 22 judges for a blister aneurysm in a 45-year-old man. All other cases had ≤4 votes for “FD is the only option,” with an overall mean of 1.85 judges considering that FD was the only appropriate treatment for an aneurysm (Fig 4).
Responses to question 2 (Any other treatment option?). In all cases, judges were able to find a viable alternative to flow diversion. The cases with the largest number of votes for “No, flow diversion only” were a ruptured supraclinoid carotid blister aneurysm (A, 5 votes) and a giant basilar aneurysm with an associated AVM (B, 4 votes). All other aneurysm-patient combinations were thought to have >1 treatment option.
Are Clinicians Willing to Recruit This Particular Patient in a Trial? (Question 3)
Respondents were willing to offer trial participation in 417 of 741 (56.3%) responses to question 3. The percentage of positive responses was 60.7% from surgeons, 54.6% from radiologists (P = .146), 52.3% from senior clinicians, 61.2% from junior clinicians (P = .022), 52.9% from clinicians with the largest FD experience compared with 65.1% from clinicians with less experience (P = .016), 52.9% for patients in FIAT, and 58.3% for those not in FIAT (P = .022). Given the multiplicity of testing, none of these P values were significant. In 74% of patients with proposed participation in question 3, FD was selected as the best option in question 4. Clinicians not willing to offer participation were more confident in their treatment choice (8.7 ± 1.3) than clinicians willing to offer participation (7.5 ± 1.5, P = .005).
In 4 cases, more than half of the raters showed willingness to include that patient in a RCT, all of which featured aneurysms in ophthalmic or paraophthalmic locations (Fig 5). Interrater agreement regarding recruitment in an RCT was poor; intrarater agreement was also low (Table 4). Clinicians were more willing to recruit a patient in an RCT when the lesion was in the anterior circulation (P = .028) or when the aneurysm featured a sidewall morphology (P = .001).
Responses to question 2 (Would you recruit this patient in a RCT?). Aneurysm-patient combinations, which more than half of judges were willing to include in a randomized trial, were all ophthalmic or paraophthalmic: for example, a 22-mm ophthalmic segment in a 57-year-old patient (A, 18/22 votes for trial participation) and a recurrent giant ophthalmic artery aneurysm in a 37-year-old patient (B, 20/22 votes). The case with the fewest number of judges willing to randomize was a 16-mm asymptomatic cavernous aneurysm in a 79-year-old patient (C, 4/22 votes).
Inter- and intrarater agreement regarding question 3 (Would you recruit this patient in a RCT?)
Discussion
The main finding of this survey was that if FD is frequently chosen as the best option for patients with difficult aneurysms, there was poor agreement regarding indications. Choices of FD as the best therapy were more frequent for sidewall and anterior circulation aneurysms, but the same patients were uniformly considered to have other treatment options. Substantial variability existed in deciding how to best manage these patients. Discrepant recommendations were made at the individual level, and clinicians' decisions did not follow according to their number of years in practice, specialty backgrounds, or experience with FD. Wide discrepancies were present within all subgroups of decision-makers and for all sorts of patients. If the best management decision for an individual patient is the consensus of a panel of clinicians, it appears reasonable to conclude that for most patients, one decision cannot rationally be justified over another. Nonetheless, individuals had high confidence levels regarding their treatment choices. Clinicians were willing to recruit patients in a randomized trial in half of the cases, but agreement regarding who should be enrolled was poor.
When reliable evidence is absent, the selection of patients for an innovative therapy can be based on various considerations. The first and perhaps most intuitive approach is to restrict the use of the unproven device to patients for whom conventional options are (relatively) contraindicated because risks are considered prohibitive or because efficacy of other management choices is expected to be low.13,19 One drawback of this strategy is that it is unlikely to portray the device in a good light because difficult cases are also more likely to have poor outcomes, compared with “standard cases.” Furthermore, if access to the device under strict conditions is truly enforced, a proper comparative evaluation cannot be done when users claim there are no viable alternative treatment options with which to compare results.20 Here, we see that restricting the use of FD to patients with aneurysms because they have “no other choice” really applies to very few patients; the answers to question 2 suggests that FD is rarely the sole possible treatment option.
Another approach to introducing an innovation is to restrict early use of the novelty to cases with the best chance of success, as judged from a priori reasoning, clinical intuition, or animal studies. In a sense, this is what was done for FD.1 The concept of the device makes it ideal for sidewall aneurysms. The success of FD with sidewall aneurysms yet failing in bifurcation or giant fusiform aneurysms has been confirmed in animal models.21⇓–23 Early series and meta-analyses have suggested that FD is associated with good outcomes in proximal carotid aneurysms; results were not as good in fusiform or bifurcation aneurysms of the posterior circulation.1,12,24,25 Perhaps most unsurprising, our survey confirms that anterior circulation and sidewall dispositions are aneurysm characteristics associated with frequent FD choices. However, 2 major difficulties exist with using this approach to decide who should receive a new treatment or device. First, comparisons among different patients treated with the same method cannot provide the evidence necessary to support treatment decision-making for individual patients: The fact that FD works better in proximal carotid compared with basilar aneurysms cannot be used to suggest that FD should not be performed for basilar aneurysms or should be used for anterior circulation aneurysms.
To make rational treatment choices based on evidence, we need a valid comparison among similar patients treated by different treatments, showing better results with FD compared with an alternative. Second, if a registry of selected patients eligible for standard treatment options is suggested, it remains unclear why FD should be chosen over established alternatives. Although in some cases, authors claimed patients were “untreatable” by standard means, and even though in most early series patients and aneurysms were carefully selected, few patients with aneurysms truly have no other management option, as shown by this survey. An approach that restricts FD use to patients considered to have the best chance of success may not be in the best interest of current patients because with narrow selection, some patients with truly difficult aneurysms in need of innovative treatments are denied a chance to receive the novelty. Those with more favorable characteristics for FD, enthusiastically selected to be offered the new treatment, are exposed to unknown risks but under pressure to accept. These patients, treatable by other means, may have had as good an outcome with standard therapy. In our view, these patients should have been offered participation in a trial comparing the innovation with standard treatment until the best treatment is identified.
A survey of prevalent opinions on treatment choices can be instructive in the context of a conventional trial: It may help in the design of the trial, in determining selection criteria, or in fixing the comparative interventions. It may also instruct the notion of community equipoise26 and indicate whether a trial is likely to be successfully completed. For example, answers to question 1 (Is FD an option for this patient?) could have been used to determine exclusion criteria had we found patients for whom “no” was a unanimous answer. In these types of trials, the priority is to come up with an answer to a precise research question.
In the context of surgical innovations, we believe these notions need to be revised because they presuppose knowledge that has yet to be discovered. Looking for agreement before initiating the trial is putting the cart before the horse. In the absence of reliable knowledge, on what basis would agreement be constructed? Waiting for agreement regarding indications is at the root of the “it is always too early until it is too late” phenomenon, leading to decades of medical history in which interventions are practiced without evidence that they are beneficial.27
Care trials such as FIAT differ from conventional trials.16 In the presence of little agreement and scant knowledge, they provide a prudent, orderly way to introduce innovations, to assist in the care of patients in the presence of uncertainty. They prioritize the protection of patients being offered promising but unvalidated therapies because the unknown implies potential risks. The trial is constructed on a principle of prudence: Novel therapies should be offered only within the trial until they are either shown to be beneficial and adopted to replace standard options or are shown to be harmful and abandoned. With FIAT, a patient for whom the treating physician considers using FD is a potential study participant. The treating physician is asked to select another option (BST), and the patient is allocated to either treatment with FD or BST. Thus, the patient has a 50% chance of receiving the promising FD, hypothesized but not yet proved better, but also a balanced 50% chance of receiving BST and avoiding the potential morbidity that might arise from using barely tested treatments that may entail risks.15,28 FIAT also includes a registry for patients with no other therapeutic option, but as this survey showed, this opt-out should not need to be used frequently.
Our survey may also inform whether the condition of “clinical equipoise” exists regarding the use of FD.26 While many clinicians may believe that equipoise requires an absence of preference or think that there is no reason to lean toward one treatment or another in this particular patient, this understanding is both conceptually odd and ethically irrelevant.26 According to Freedman, 26 who introduced and popularized the concept, clinical equipoise occurs when “there is no consensus within the clinical expert community about the comparative merits of the alternatives to be tested.” If this survey shows that such a condition is met for FD, we believe that for a clinician to use FD or any surgical innovation, the correct approach is to replace the notion of clinical equipoise with the notion that in the absence of a good exemptive reason, a clinician should only use an unvalidated intervention within the context of a declared care trial.16
The widespread discrepancies in management decisions shown in this survey should provide the evidence for collective community uncertainty needed to proceed with a trial. The choices patients and clinicians face are not care versus research. The use of FD in the care of patients is far from being established therapy. Controversy also remains regarding coiling or stent-assisted coiling versus open surgery for the treatment of difficult aneurysms. Randomized clinical trials are required if physicians want to properly counsel their patients regarding the relative risks and benefits of potential management options in the future. In the meantime, a more immediate goal is to enable experimentation with a promising innovation in the care of patients with difficult aneurysms while attempting to minimize the risks to patients who are exposed to the innovation.
Limitations
There were several limitations of this study. The questionnaire was submitted to 40 experts. Only 22 answers (55%) were received. The responders were not a random sample of a population of experts, and there is no reason to believe that they are representative of such a population, if it could be defined. We can only conjecture that results reflect the opinions of a self-selected group of clinicians who took the time and energy to respond. Some participants had little experience with FD, and one may question the significance of their opinions. However, agreement among more experienced participants was equally poor. The portfolio was artificially constructed. Absolute numbers of choices are thus not meaningful, but we were careful to present cases that are currently at least considered for FD. Agreement regarding particular cases (reliability) does not mean accuracy (in other words everyone could be wrong). Finally, answering a questionnaire and caring for real patients represent different contexts for decision-making; we can only speculate about how seriously the responders imagined they were dealing with important clinical decisions. Clinical decision-making in practice is rarely performed by a single individual in isolation and typically takes into consideration opinions from collaborators and patient preferences.
Conclusions
Wide variability and poor agreement exist regarding the indications for FD. This uncertainty should provide ample room for participation in trials designed to safely introduce promising technology in the care of patients with difficult aneurysms.
Footnotes
Disclosures: Daniel Roy—UNRELATED: Grants/Grants Pending: Stryker,* MicroVention,* Covidien,* Codman,* Comments: fellowship support. Alain Weill—UNRELATED: Grants/Grants Pending: MicroVention,* Stryker,* Covidien,* Codman,* Comments: We receive support from these entities in our academic fund for fellowship support; Payment for Lectures (including service on Speakers Bureaus): MicroVention,* Comments: We received money from MicroVention in our academic fund for a lecture I gave at the Asian Australasian Federation of Interventional and Therapeutic Neuroradiology 2014 conference (“Bailout in Aneurysm Treatment”). Jean Raymond—UNRELATED: Grant: Canadian Institute of Health Research,* Johnson & Johnson,* Comments: Canadian Institute of Health Research: funding for the Collaborative Unruptured Endovascular versus Surgery trial; Johnson & Johnson: unrestricted research funding for the Does Embolization with Larger coils lead to better Treatment of Aneurysms trial (investigator-initiated study). *Money paid to the institution.
REFERENCES
- Received October 16, 2014.
- Accepted after revision November 6, 2014.
- © 2015 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- Flow Diversion in the Treatment of Intracranial Aneurysms: A Pragmatic Randomized Care Trial
- Factors Influencing Confidence in Diagnostic Ratings and Retreatment Recommendations in Coiled Aneurysms
- Flow diversion of bifurcation aneurysms is more effective when the jailed branch is occluded: an experimental study in a novel canine model