|
| HOME | SUBSCRIPTIONS | CURRENT ISSUE | PAST ISSUES | CARDIOSOURCE | SEARCH | HELP | FEEDBACK |
|
|
|
|||||||||
|
|||||||||||
|
J Am Coll Cardiol, 2006; 47:1796-1802, doi:10.1016/j.jacc.2005.12.053
(Published online 11 April 2006). © 2006 by the American College of Cardiology Foundation |
Emory Angiographic Simulation Training (EAST) Center, Department of Medicine, Division of Cardiology, Emory University School of Medicine, Atlanta, Georgia
Manuscript received July 30, 2005; revised manuscript received December 15, 2005, accepted December 20, 2005.
* Reprint requests and correspondence: Dr. Christopher U. Cates, Vascular Intervention, Emory University, 1364 Clifton Road NE, Suite C-430, Atlanta, Georgia 30322 (Email: Christopher_Cates{at}emoryhealthcare.org).
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionReferences |
|---|
BACKGROUND: The Food and Drug Administration requires completion of VR simulation training for physicians learning to perform carotid stenting.
METHODS: Interventional cardiologists (n = 20) participating in the Emory NeuroAnatomy Carotid Training program underwent an instructional course on carotid angiography and then performed five serial simulated carotid angiograms on the Vascular Interventional System Trainer (VIST) VR simulator (Mentice AB, Gothenburg, Sweden). Of the subjects, 90% completed the full assessment. Procedure time (PT), fluoroscopy time (FT), contrast volume, and composite catheter handling errors (CE) were recorded by the simulator.
RESULTS: An improvement was noted in PT, contrast volume, FT, and CE when comparing the subjects’ first and last simulations (all p < 0.05). The internal consistency of the VIST VR simulator as assessed with standardized coefficient alpha was high (range 0.81 to 0.93), except for FT (alpha = 0.36). Test-retest reliability was high for CE (r = 0.9, p = 0.0001).
CONCLUSIONS: A learning curve with improved performance was demonstrated on the VIST simulator. This study represents the largest collection of such data to date in carotid VR simulation and is the first report to establish the internal consistency of the VIST simulator and its test-retest reliability across several metrics. These metrics are fundamental benchmarks in the validation of any measurement device. Composite catheter handling errors represent measurable dynamic metrics with high test-retest reliability that are required for the high-stakes assessment of procedural skills.
| ||||||||
These challenges in physician training have become even more important since the approval of carotid artery stenting by the Food and Drug Administration (FDA), a time in which many previously inexperienced physicians from various specialties are wishing to learn and practice this new, complex, and high-risk procedure (3). In addition, physicians in clinical practice who have completed formal post-graduate residency/fellowship training before widespread use of new procedures or techniques have previously found it difficult to train and acquire new skills. Traditional training methods for new procedures includes live animal models, cadavers, mechanical models, or supervised performance of the procedure on patients. The majority of teaching in the U.S. relies on the latter method for training physicians; however, this tradition of training on patients has provoked apprehension on the part of the profession and public alike regarding the manner in which physicians will acquire the necessary skill to safely perform endovascular procedures such as carotid angiography and stenting (4). Carotid angiography and stenting are potentially high-risk procedures for patients because, compared with other organs, the brain is much less tolerant to plaque disruption and subsequent alterations in blood flow and therefore would be less forgiving for physician training errors. Carotid stenting has a definite learning curve (5), similar to other procedures; however, this procedure is unique in that the risks conferred to the patient as a result of the physicians’ learning curve are unacceptably high and immediately apparent. Historically, the number of procedures performed by a physician operator and the duration of training has been used as crude measures of operator proficiency, because no standardized mechanism has been available to assess these post-training skills. Now, because carotid stenting will be performed by physicians across several subspecialties with varied technical skill sets, the need for an objective assessment of proficiency before credentialing physicians is now even more necessary (3). Virtual reality (VR), simulation-based training might help achieve this goal.
Virtual reality simulation has been extensively used as a training method in other high-skill tasks such as aviation and laparoscopic surgery. The study frequently referred to as VR training for the operating room (VR-to-OR) is a benchmark study for methodology in training with medical VR simulation (6). This prospective, randomized, double-blinded study demonstrated that residents who were trained on VR simulators made significantly fewer objectively assessed intra-operative errors compared with the standard-trained group when performing laparoscopic cholecystectomy. A second trial showed that a significant portion of the operators’ learning curve could be acquired through VR training outside the OR (6,7). Study results such as these have prompted the FDA to accept the use of VR simulation as part of a tiered training approach for carotid stenting (3,8). Furthermore, a VR-to-catheterization laboratory, prospective, randomized blinded study is drawing to a conclusion at our institution.
The change of the procedural-based medicine training paradigm from a traditional mentored-based patient approach to a VR-based training methodology has raised the issue of operator performance assessment. One of the main goals in VR simulator training should be to improve and measure operator performance. Furthermore, two other equally important goals of a VR simulator are to increase the consistency in performance (i.e., reduce performance variability) and reduce the error rate in performing endovascular procedures, which should translate into better clinical outcomes. The aim of this study was to demonstrate the utility of the Procedicus Vascular Interventional System Trainer (VIST; Mentice AB, Gothenburg, Sweden) simulator as a measuring tool for improvement in performance and a reduction in procedural errors on repeat testing during simulated carotid angiography.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
Apparatus. The VIST was specifically designed as a VR simulator for training in endovascular interventional procedures. The VIST device simulates the procedure exactly as it is performed in a live patient with the full physics vascular anatomy created from patient-specific digital data and can therefore be assessed and measured accurately and reliably. The use of the VIST as a valid measure of performance has been established (9,10). Face and content validity studies have demonstrated that a simulated carotid angiogram procedure with the VIST simulator closely resembles the angiographic anatomic appearance (aortic arch and carotid vasculature) and the catheter and guidewire dynamics (push-pull maneuvers and catheter/wire interaction and feel) that would be observed in a live carotid angiogram. Furthermore, the VIST simulated carotid angiogram procedure experience received very high Likert scores in the overall procedural realism of instruments (mean 4.9) and the overall sequencing of guidewires and catheters during the procedure (mean 4.8) (9).
The Procedicus VIST simulator is based on a dual processor (2 x 2.8 GHz processor), Pentium IV computer running Windows Microsoft XP Professional with 1 GB RAM, a 40-GB hard disk drive, a GeForce FX5200 128MB graphics card, and two 17-inch flat-panel monitors (Fig. 1). The interface and the actual devices used in the real procedure (catheters, wires, stents, and so on) are linked to the VR simulator through a proprietary full physics software package that then generates the fluoroscopic display. The simulation interface device is designed to sense the simultaneous translation and rotation of three co-axial tools (clinical tools), the flow of air from a syringe that shows as a contrast injection on the display, pressure by fluid compressed with an indeflator, and a foot switch for fluoroscopy and cine-angiography. Output of the device to the user is the application of force and torque on each of the tools on the basis of the calculations of the simulator for the full physics vascular anatomy simulated.
|
The translational position is measured with an optical encoder that, in combination with the transmission system, gives a resolution of 0.11 mm. The rotational angle is measured with an optical encoder that, in combination with the gear ratio to the locking device, gives a resolution of 7.9 to 31.4 milliradians (depending on which cart). The tool diameters are measured with an infrared optical sensor that gives a resolution of 0.02 mm and has a precision of about ±15%. The algorithms that calculate the diameter calibrate the parameter settings in run-time, to avoid drifting. The measurement span is between 0.1 and 3.0 mm.
All testing was performed in a quiet room with a table height of approximately 100 cm and the monitor position at eye level, simulating the catheterization laboratory environment.
Procedure. All participants were shown two live carotid angiography demonstrations via teleconference, followed by didactic training in the procedural technique. The participants were then moved to the simulation suite where a proctoring physician demonstrated a right carotid angiogram on the simulator, followed by the participants then completing the necessary components of the right carotid angiogram. The same was then done for the left carotid angiogram. A mentored, supervised rehearsal of a complete carotid angiogram was then performed. Five supervised but unmentored bilateral carotid angiograms were consecutively performed on the VIST simulator with dynamic and static metric measurements recorded by the simulator for every operator on each test trial. All simulated carotid angiograms were performed with the same case.
During the simulated procedures, successful completion of a carotid angiogram required the operator to first select the appropriate wires and catheters necessary for the procedure. A simulated type I aortic arch configuration was used for all subjects. An arch aortogram was performed with a 6-F pigtail catheter in the left anterior oblique view at 30° with simulated digital subtraction angiography. The right carotid artery was then entered with the 6-F right Judkins catheter under digital roadmap and hydrophilic wire guidance. The digital subtraction angiographic views taken for the right common carotid artery were right anterior oblique at 30° and 90°. Additional intracranial, digital subtraction angiography images of the right carotid artery through the levophase venous cycle were performed at right anterior oblique 90° and an antero-posterior projection with cranial angulation (Towne’s view). Similarly, the left common carotid artery was entered with similar technique, and digital subtraction angiography images were taken with left anterior oblique views at 30° and 90°. The intracranial left carotid artery was imaged with digital subtraction angiography, including the venous levophase cycle at left anterior oblique 90° and an antero-posterior projection with cranial angulation (Towne’s view). Other than a 6-F pigtail catheter, the only catheter used was the 6-F right Judkins catheter.
Metrics.
There were seven measures of the participants’ operative performance. The static metrics included the procedure time (PT), fluoroscopy time (FT), contrast volume, number of cine-loop recordings, and the dynamic metrics included composite catheter handling errors (CE). Composite catheter handling errors were further sub-divided up into two groups. Catheter vessel errors were defined as a dragging of the tip of the catheter along the vessel wall 
3 mm. A catheter movement error was defined by and recorded when the right Judkins catheter was advanced into the carotid artery without a guidewire distal to the catheter tip. If, at any time during the procedure, a catheter handling error was made, the operator was immediately notified by a clearly visible on-screen flashing red triangle giving proximate feedback to the operator of that error. Participants were also given overall procedural feedback on their performance at the end of each simulated case by a computer-generated print-out that outlined the log of the procedure, including all the CE that were made during the VR simulation. The data were cataloged by unique operator identifier and saved to an archived file.
Statistical analysis.
All statistical analyses were performed with SPSS version 10 (SPSS Inc., Chicago, Illinois). Differences between performances on test trials were examined for significance with analysis of variance (ANOVA) for repeated measures. An ANOVA calculated a correction term for the multiple comparisons. An ANOVA for repeated measures simply showed a statistical difference between conditions but did not specify which conditions. Specific contrasts between mean scores for the different trials were compared for statistical significance with Scheffe F-tests for controlled multiple comparisons. Test-retest reliability between trials one and two was assessed with Pearson’s Product Moment Correlation Coefficient. Internal consistency was assessed with coefficient alpha. A coefficient alpha statistic was used to estimate the reliability of data under conditions when two or more comparable scores per person were present (i.e., repeated measures design). The greater the consistency between these scores on separate test trials, the more likely the measurement error on the simulator’s part was minimally present; and so, the scores were likely to be accurate reflections of true scores. The observed scores would be considered acceptably reliable if alpha > 0.8 (11). A p value 
0.05 was considered significant.
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
|
|
|
|
|
|
|
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
The inherent risks of performing carotid angiography and interventions are not insignificant, with a reported combined stroke and death rate of approximately 1% (12). Training endovascular physicians from multiple specialties in the necessary cognitive and technical skills for carotid angiography and intervention presents an additional challenge, because vascular surgeons, interventional radiologists, and interventional cardiologists all carry different skill sets into the training setting. The fund of knowledge necessary to successfully perform carotid angiography requires the understanding of the neurovascular anatomy and the pathogenesis and management of carotid artery disease as well as mastering the skills of the complex percutaneous endovascular technique. A significant obstacle that all groups have to overcome is the acquisition of appropriate percutaneous coaxial catheter and wire handling skills. During procedures such as diagnostic carotid angiography and intervention, the operating trainee has a most difficult job of attempting to perform complex eye-hand movements while paying attention to all aspects of the high-risk procedure itself. This requires a great deal of trainee operator cognitive attentional resources. This "attentional load" of dual task performance places a significant demand on trainees’ cognitive resources that can easily overwhelm their attentional capacity (13). Allowing this process to occur in a simulated environment is preferable to allowing this process to occur while practicing on a patient.
This study showed that technical skills assessment with dynamic skill-related measures such as catheter handling errors is a more reliable metric for use in high-stakes assessment of procedural performance over more general measures such as PT, FT, and contrast use, as evidenced by consistently high test-retest reliability measures. Performing a procedure more quickly and with less contrast volume provides, at best, a crude assessment of the technical performance of the operator. Performing a procedure faster does not necessarily translate to a better performance of the procedure; however, dynamic metrics such as catheter handling are a more reliable and consistently demonstrable tool that can provide a better assessment of the operator’s skill. In time, with continued practice (learning curve), endovascular trainees eventually are able to automate to the fulcrum effects of the body wall and the degraded image quality of X-ray angiography, and this improvement in skill can be objectively measured by the VIST simulator (10,14).
In our study, interventional cardiologists quickly adapted to the VIST simulator while performing carotid angiography. A significant and progressive decline in CE with repeat test-trials was clearly demonstrated. The relatively low number of initial catheter handling errors performed during Trial 1 and their gradual reduction on subsequent trials is likely directly proportional to the vast experience of these operators who routinely perform coronary angiography and interventions. The low number of initial catheter movement errors observed might be explained by the fact that similar techniques for catheter advancement and removal are used during coronary angiography, particularly with left internal mammary artery graft injection and the required cannulation of the left subclavian artery.
The improvement in the technical skills performance of these trainee subjects was also found to be consistent. Consistency, as observed by the measures of score variability, has become an important performance parameter to look at when evaluating skill acquisition. Although the main goal of training is to improve performance, is it also imperative that trainee physicians are trained to perform well, consistently. This decrease in operator variability in the setting of improved performance suggests acquisition and firm establishment of proper procedural technique was probably enhanced by the system of proximate feedback. The trainee subjects showed less variability in their overall procedural technical skills performance as they progressed in repeated test-trials. We believe that the significant improvement in performance and improved consistency observed in these objective dynamic skill-related measures of psychomotor performance reflects an enhancement of learning on repeat testing and establishment of metric-based VR simulation learning curves.
The advancements in technology have made high-fidelity VR simulators increasingly able to be considered in other areas of medicine as an initial method of training (15–18). Recently, multiple professional societies have included a section on simulator training and have specified industry minimum training standards, with simulation as a part of their joint competency and credentialing standards document (19). Now, the validation of any endovascular procedural assessment tool must be considered a priority for procedural-based medicine (20). In this study, we have reported the internal validity of the VIST simulator, the test-retest reliability, and for the first time, the ability of a simulator to measure a learning curve in carotid endovascular procedures. Although consistently high coefficient alphas were observed with all of the measures except those of FT, of particular importance was the high reliability coefficient seen with the dynamic metric of CE (alpha = 0.90), which demonstrated the simulator’s ability to actually measure operator technical performance. Not only was initial test reliability better, but the test-retest reliability of CE was also the highest. The high reliability values obtained on CE makes this individual metric increasingly important as a potentially useful tool in the high-stakes performance assessment in new high-risk procedures, such as carotid stenting. The ongoing question, however, still remains: will the adequate attainment of this VR simulator skill translate into successful procedural and clinical outcomes? The VR-to-OR study is currently our only available reported randomized prospective trial that shows the benefit of VR training (6); however, a VR-to-cardiac catheterization prospective randomized blinded trial, Simulator Training Randomized versus Interventional Vascular Experience, is nearing completion and will be reported soon (8).
Potential limitations introduced into this study include a relatively small number of subjects used for analysis. Even with this sample size of experienced interventional cardiologists, however, statistically significant differences were shown across test trials. Furthermore, the number of trials performed was sufficient to assess the reliability and internal consistency of the VIST VR simulator. Another possible limitation might be the use of the same simulated case for each trial, because the subject’s improvement in performance might have resulted from an increased familiarity with the simulated procedure over successive test trials. Although partially true, it must be noted that the learning curve for performing any endovascular procedure, whether in vivo or ex vivo, requires both a familiarity of the procedure and the acquisition of the necessary technical skills. To show an improvement in dynamic metric measures of performance (i.e., catheter handling skills), the trainee must improve on their technique and not their familiarity with the simulator. Static measures of performance, such as PT, FT, and contrast volume, are more likely to be influenced by the familiarity of the procedure, simulated or real.
This study has shown that the VIST simulator can be used to track the technical skills improvement in procedural performance of experienced endovascular specialists, resulting in the objective demonstration of an individual operator learning curve. Improvements in performance were also easily distinguishable at the level of mean performance. Participants performed the procedure faster, used less contrast, and, most importantly, made fewer catheter handling errors with repeat testing using proximate feedback by the simulator during procedural learning. Performance improvement was also demonstrated by an improvement in consistency and decreased variability in scores during subsequent repeat testing. The VIST simulator also demonstrated consistently high coefficient alpha scores required for high-stakes operator assessment and high test-retest reliability measures with dynamic metrics of catheter skill. This study represents the first description of the ability to objectively measure and assess dynamic metrics of catheter performance and operator technical skill in vascular medicine. This study also establishes the basis for the creation of operator performance norms for endovascular physicians training in carotid procedures and serves as the reference point in establishing a learning curve for training in carotid procedures. Further study is in progress to determine whether those improved training skills translate into improved clinical outcomes.
|
References |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  W. Dunn and J. G. Murphy Simulation: About Safety, Not Fantasy Chest, January 1, 2008; 133(1): 6 - 9. [Full Text] [PDF]
|
|
|
|
|
|
D. B. Wayne, A. Didwania, J. Feinglass, M. J. Fudala, J. H. Barsuk, and W. C. McGaghie Simulation-Based Education Improves Quality of Care During Cardiac Arrest Team Responses at an Academic Teaching Hospital: A Case-Control Study Chest, January 1, 2008; 133(1): 56 - 61. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. R. Bates, C. J. D. Babb, D. E. Casey, C. U. Cates, G. R. Duckwiler, T. E. Feldman, W. A. Gray, K. Ouriel, E. D. Peterson, K. Rosenfield, et al. ACCF/SCAI/SVMB/SIR/ASITN 2007 Clinical Expert Consensus Document on Carotid Stenting: A Report of the American College of Cardiology Foundation Task Force on Clinical Expert Consensus Documents (ACCF/SCAI/SVMB/SIR/ASITN Clinical Expert Consensus Document Committee on Carotid Stenting) Vascular Medicine, February 1, 2007; 12(1): 35 - 83. [PDF]
|
|
|
|
 |
|
 C. U. Cates, A. D. Patel, and W. J. Nicholson Use of Virtual Reality Simulation for Mission Rehearsal for Carotid Stenting JAMA, January 17, 2007; 297(3): 265 - 266. [Full Text] [PDF]
|
|
|
|
 |
|
 American Society of Interventional & Therapeutic N, Society for Cardiovascular Angiography and Interve, Society for Vascular Medicine and Biology, Society of Interventional Radiology, E. R. Bates, J. D. Babb, D. E. Casey Jr, C. U. Cates, G. R. Duckwiler, T. E. Feldman, et al. ACCF/SCAI/SVMB/SIR/ASITN 2007 Clinical Expert Consensus Document on Carotid Stenting: A Report of the American College of Cardiology Foundation Task Force on Clinical Expert Consensus Documents (ACCF/SCAI/SVMB/SIR/ASITN Clinical Expert Consensus Document Committee on Carotid Stenting) J. Am. Coll. Cardiol., January 2, 2007; 49(1): 126 - 170. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | SUBSCRIPTIONS | CURRENT ISSUE | PAST ISSUES | CARDIOSOURCE | SEARCH | HELP | FEEDBACK |
