Eric D. Peterson, MD, MPH, FACC, FAHA, Chair; Frederick A. Masoudi, MD, MSPH, FACC, FAHA^⁎; Elizabeth DeLong, PhD; John P. Erwin III, MD, FACC; Gregg C. Fonarow, MD, FACC, FAHA; David C. Goff, Jr, MD, PhD, FAHA; Kathleen Grady, PhD, RN, FAHA, FAAN; Lee A. Green, MD, MPH; Paul A. Heidenreich, MD, MS, FACC, FAHA; Kathy J. Jenkins, MD, MPH, FACC; Ann Loth, RN, MS, CNS; David M. Shahian, MD, FACC

There is an increasing demand for information on the quality, cost, and appropriate use of cardiovascular technologies, including those used for imaging and therapeutic procedures. In addition, there is a need for the healthcare profession to take the lead in designing strategies to optimize the use of technology and provide ways to report this information externally.

The American College of Cardiology Foundation (ACCF)/American Heart Association (AHA) Task Force on Performance Measures (Task Force) commissioned the ACCF/AHA Methodology for the Development of Quality Measures for Cardiovascular Technology Writing Committee, which was charged with developing a measurement methodology to meet this demand. Although this document will determine the sequential process for developing these quality measures, no recommendations for using any specific technologies, devices, or therapies will be made. The intended audience for this document includes organizations considering the future development of quality indicators for healthcare technology, practicing clinicians, and specialists in cardiovascular medicine.

Although much work has already been accomplished in performance measurement with respect to certain therapeutics and processes of care, there is a need to better define metrics for technologies, in which the linkage with outcomes is less well established. The focus of this document is the articulation of a formal methodology for developing quality measures for cardiovascular technology when evidence-based guidelines recommendations are absent, termed appropriate use measures (AUMs) and structure/safety measures (SSMs).

The members of the writing committee included experienced clinicians and specialists in adult cardiovascular medicine, electrophysiology, interventional cardiology, cardiovascular imaging, and performance measures methodology.

The work of the writing committee was supported exclusively by the ACCF and AHA. Writing committee members volunteered their time, and there was no commercial support for the development of this performance methodology. Writing committee members were required to disclose in writing all current financial relationships with industry and other entities, as well as those 12 months before initiation of the writing effort, and verbally acknowledge these relationships to the other members at each meeting (Appendix A), in accordance with standard ACCF and AHA reporting policies. In addition, peer reviewers of the document were asked to disclose relationships with industry and other entities (Appendix B). The parent Task Force makes every effort to avoid actual, potential, or perceived conflicts of interest that may arise as a result of industry relationships or personal interests among the writing committee. Because this is a methodology document, the recommendations will not benefit or harm any company. Consequently, the Task Force has determined that members' relationships with pharmaceutical companies and device manufacturers are not relevant to the topic of this document. Authors' comprehensive disclosure information, which includes information on relationships with industry and other entities not relevant to this document, is available online at http://content.onlinejacc.org/cgi/content/full/j.jacc.2011.07.007/DC1.

Between October 15, 2010, and November 1, 2010, this document underwent peer review, during which ACCF and AHA members and other healthcare professionals had the opportunity to review and comment on the text before its final approval and publication. Forty-nine peer reviewers provided 410 individual comments.

The ACCF/AHA Methodology for the Development of Quality Measures for Cardiovascular Technology was adopted by the respective boards of directors of the ACCF and AHA in May 2011. This methodology will be reviewed for currency once annually and updated as needed. It should be considered valid until either updated or rescinded by the Task Force.

Modern medicine has made great strides in improving both quality of life and longevity in cardiovascular disease. Many of the tools used for the diagnosis and therapy of heart disease involve advanced technologies. With each new technological advance comes pressure for immediate implementation and dissemination, often before evidence of efficacy has been published. The substantial advances in cardiovascular technologies, accompanied by a concomitant growth in healthcare expenditures, have resulted in the need for more critical assessment of use of technology.

Existing methodology documents provide a wide range of guidance in the development of performance measures for cardiovascular care based on ACCF/AHA practice guidelines, but substantial gaps persist for measuring performance in the use of cardiovascular technology, in which firm guidelines recommendations are often lacking. The purpose of this document is to define a process to address these gaps, building on the current Task Force methodologies (1- 2), and to define a methodology that will permit the development of quality measurement in areas of cardiovascular technology, such as diagnostic imaging. This methodology is fundamentally based on the Institute of Medicine healthcare quality aims of providing safe, effective, patient-centered, timely, efficient, and equitable care (3).

The primary goal of the present document is to provide direction in measuring and improving the use of cardiovascular technology and to describe how such measures might be developed for the purposes of quality improvement and public reporting to reduce not only underuse but also overuse and misuse while also addressing proper and safe application. In collaboration with other involved stakeholders, physicians who care for patients with suspected or established cardiovascular disease and persons expert in the acquisition and interpretation of diagnostic tests are well positioned to contribute to this process with the goal of achieving the best possible patient outcomes. This document is meant to advance the efforts of the profession, the AHA, and the ACCF to define and promote the highest-quality cardiovascular practice and to ensure access to technologies for all patients who may benefit from them. It is anticipated that this effort will help focus the national dialogue on the use of cardiovascular technology and away from the current concerns about volume and cost alone to a more holistic emphasis on value. The ultimate objective of the Task Force is to achieve the best possible patient outcomes, including the development of measures that will promote optimal use of cardiovascular technology.

A broad array of external stakeholders is focusing attention on the use of cardiovascular technology, including the Federal Coordinating Council for Comparative Effectiveness Research (4), the Centers for Medicare and Medicaid Services, the National Quality Forum (NQF) (5), the NQF National Priorities Partnership (6), the American Medical Association–Physician Consortium for Performance Improvement (AMA–PCPI), and the National Committee for Quality Assurance.

These efforts are complementary to many current ACCF/AHA quality efforts, many of which may also address technology:

ACCF/AHA clinical practice guidelines are comprehensive and literature based, often requiring years of preparation. Although in the past, guideline topics included procedures (e.g., stress testing) or devices (e.g., implantable cardioverter-defibrillators [ICDs]), the recent approach has been solely disease based, with topics including stable angina pectoris, acute myocardial infarction, and heart failure.

ACCF/AHA performance measures distill guidelines recommendations into measures that reflect clinical care patterns as recommended by guidelines and are suitable for use for accountability. These measures focus on areas in which little controversy exists related to the evidence supporting the measure, valid measurement is feasible, and implementation will improve care. By definition, ACCF/AHA clinical performance measures are based on recommendations in ACCF/AHA practice guidelines (1- 2).

ACCF appropriate use criteria (AUC) provide information about the best current use of tests and procedures in specific clinical scenarios encountered in clinical practice, such as imaging for risk assessment before noncardiac surgery. AUC are literature based whenever possible but also incorporate a rigorous Delphi process to create a formal expert consensus based on the principle that input from a structured group of experts is more accurate than consensus from unstructured groups (7- 8). Overall, AUC attempt to define what, when, and for whom a test or procedure is (or is not) appropriate, on the basis of a balance of risks and benefits.

ACCF competency and training documents define the necessary knowledge base and training volume for a practitioner, usually directed toward the performance of a specific procedure, such as transesophageal echocardiography or peripheral vascular intervention.

The ACCF and AHA have collaborated on a number of other projects, including the development of data standards and registries. Each organization also engages in additional quality initiatives, including publication of expert consensus documents and scientific position papers, support of laboratory accreditation, physician credentialing, and extensive medical education programs. Together these efforts all play a role in the optimal deployment of medical technology. However, they do not provide explicit metrics useful for quality assessment, quality-improvement initiatives, or accountability (e.g., public reporting). The methodology to close this gap is the subject of the present document.

The writing committee defines AUMs as those measures based on existing AUC documents related to use of technology for which no guidelines recommendations are available. SSMs are defined as those measures related to the structural and patient safety aspects of cardiovascular technology in which there are neither guidelines recommendations nor AUC but for which other documents describing best practice have been published or endorsed by the ACCF or the AHA or both, and expert consensus can be achieved using a formal process.

Through the activities of the Task Force, the ACCF and AHA have played a leadership role in the development of performance measures and quality metrics for cardiovascular disease for >10 years. The Task Force is guided by both a specific process for measure development, as well as established methodologies for measure specification, which are described in documents that provide guidance to performance measure writing committees. Writing committees are instructed to develop measures that conform to specific attributes to the extent possible (Appendix C) (1- 2). Detailed methodology has also been developed for measures of outcomes (Appendix D) (9) and efficiency (Appendix E) (10). Although some attributes of measures may not be known completely at the time of measure development (e.g., construct validity or feasibility of implementation), all should be assessed during implementation. Not infrequently, field testing identifies issues that result in significant revision of measures. Finally, specific methodology for combining various measures into composite measures has also been specified by the Task Force (11).

The lexicon of healthcare measurement can be confusing and limiting. An important dichotomy differentiates measurement for the purpose of “accountability,” for example, public reporting or pay-for-quality incentives, from those used solely for internal quality-improvement efforts. Accordingly, the Task Force has recommended that the term performance measures apply to those measures of healthcare quality that are appropriate for public reporting. These performance measures developed by the Task Force are those measures that have been developed using ACC/AHA methodology that includes a peer review and public comment process (12). This process can include collaboration with the AMA–PCPI, the Centers for Medicare and Medicaid Services, the Joint Commission, and/or NQF. Performance measures are then distinct from quality metrics, which are intended for internal quality improvement but not public reporting. The Task Force defines such metrics as measures that have been developed to support self-assessment and quality improvement at the provider, hospital, and/or healthcare system level. These metrics represent valuable tools to aid clinicians and hospitals in improving quality of care and enhancing patient outcomes (12). The ACCF/AHA position that some measures are well suited for internal quality improvement but less well suited for public reporting and accountability is supported by Chassin et al (13).

Cardiovascular performance measures have historically relied on the availability of Class I guideline recommendations, usually with Level of Evidence (LOE) A, that is, based on randomized controlled trials (RCTs), as illustrated in (Table 1). The methodology guiding the translation of guideline recommendations into performance measures has been explicitly delineated by the ACCF/AHA, and this methodology provides guidance to the writing committees (1- 2).

This methodological constraint is both a strength and a limitation. On the one hand, reliance on this LOE ensures that measurement is limited to clearly efficacious processes of care or proven outcomes. However, few trials provide evidence to evaluate important parameters such as safety, appropriateness, efficiency, patient satisfaction, and quality of life, or accepted best practices such as smoking cessation counseling. This is especially true for the many diagnostic or therapeutic procedures performed in patients with diagnosed or suspected cardiovascular conditions, which have not generally been investigated in an RCT format.

In the absence of such trials, specialists in cardiovascular medicine and surgery have made substantial investments in additional data systems that allow benchmarked assessment of quality of care that can be used as performance measures. National registries (14) and databases (15) established by professional organizations have enabled the capture of broad-based, patient-specific clinical data to foster these initiatives. The Society of Thoracic Surgeons National Database elements and data were the basis of the NQF-endorsed National Voluntary Consensus Standards for Cardiac Surgery, adopted in 2004. These 21 measures of quality underlie current performance measurement in cardiac surgery (16). However, they do not directly address appropriateness.

To date, the Task Force has commissioned 8 sets of performance measures that have been published or are in development (Table 2). Many have been developed in close collaboration with the AMA–PCPI, and some have been revised since first publication. Relatively few measures in these sets focus even tangentially on use of technology, and those that do typically focus on underuse (e.g., documentation of left ventricular systolic function assessments for heart failure and acute myocardial infarction) or timeliness of technology deployment (e.g., timeliness of primary percutaneous coronary intervention [PCI]). Only recently have measures of technology overuse been proposed, and such measures have not yet been approved for implementation. To a large extent, this gap is related to the challenges in adapting the existing ACCF/AHA methodologies to assess the quality of technology use.

Performance measures should be selected on the basis of strong evidence of meaningful patient benefit and should be directly linked to positive patient outcomes (28). For example, ACCF/AHA performance measures include processes of care that have the potential for patient harm when not followed, for example, the failure to prescribe beta blockers following myocardial infarction. Thus, reperfusion therapy for ST-elevation myocardial infarction fulfills the criteria for a technology-based performance measure. In some instances, technology measure development could follow the ACCF/AHA methodology for performance measures even though the process being measured lacks the requisite LOE A. One example is use of a technology when it is required to guide implementation of a Class I, LOE A recommendation. This logic was followed by the writing committee that developed the ACC/AHA/AMA–PCPI heart failure measure set when it created the left ventricular systolic function assessment measure (17). Measurement of ejection fraction is a Class I, LOE C recommendation in the “ACCF/AHA Guidelines for the Diagnosis and Management of Heart Failure in Adults” (29), but this measurement is essential to identify patients in whom Class I, LOE A recommendations apply for medical therapy with beta blockers and angiotensin-converting enzyme inhibitors or angiotensin receptor blockers. Similarly, the ACCF/AHA/AMA–PCPI chronic stable coronary artery disease measurement set includes the symptom and activity assessment measure (25- 26). Assessment of a patient's symptom status and level of activity is not supported by LOE A but is key to the implementation of diagnostic and therapeutic interventions that are Class I, LOE A recommendations. Although current performance measure documents have begun to incorporate such measures, an explicit methodology for their development has not been developed. The current document fills this gap and formalizes the process needed for development of performance measures and quality metrics for use of technology when LOE A evidence is lacking.

As noted above, the existing methodology for the development of performance measures requires the identification of a treatment or care process that has received a Class I, or more rarely, a Class III recommendation in ACCF/AHA practice guidelines (Figure 1, left side). Once it is determined that a gap in care is related to this recommendation, a measure is formulated and examined to ensure that it is attributable, measurable, feasible, and reliable and will improve outcomes (1- 2). This process usually, but not always, requires LOE A. This existing pathway provides a precedent for the creation of a select few technology performance measures, noted previously, that are either supported by LOE A or are necessary to implement other measures supported by LOE A. The latter existing technology measures based on LOE B or C (Table 1, right column) were created according to the current methodology for performance measures and can serve as a guide for future technology performance measures that meet this high standard. On the other hand, when no such linkages exist for a technology and where there is also no direct connection between the technology and the clinical outcomes that have been the basis for performance measures, technology metrics do not meet the existing definition of performance measures. In this case, development of a formal methodology for developing performance measures and quality-improvement metrics for cardiovascular technology is necessary and is the focus of the present document.

The evaluation of quality in the use of cardiovascular technologies requires consideration of multiple parameters that differ from other healthcare processes. Hence, in addition to the traditional performance measure pathway noted above, the Task Force proposes 2 new classes of quality measures for use in these situations and refers to these metrics as AUMs and SSMs (Figure 1, right side). These measures will be developed by writing committees commissioned by the Task Force that follow the methodology outlined in Section 5.4. AUMs and SSMs are conceptually intended to apply to uses of technology and procedures, including noninvasive diagnostic testing, minimally invasive diagnostic and therapeutic procedures, and surgery. These are based on specific indications, technologies, processes, or strategies of care for which data to support a Class I or Class III guideline recommendation are lacking but are supported by either AUC documents (in the case of AUMs) or other documents (in the case of SSMs) developed by ACCF or AHA writing groups or, alternatively, by specialty professional societies (e.g., the American Diabetes Association, American Society of Echocardiography, American Society of Nuclear Cardiology, Heart Failure Society of America, Heart Rhythm Society, Society for Cardiovascular Angiography and Interventions, and Society of Cardiovascular Computed Tomography), provided that they have been endorsed by either the ACCF or AHA and that they meet additional criteria noted in Section 6. Conceptually, the SSM pathway could also be adapted for future measure development in other processes of care beyond cardiovascular technology when no LOE A guideline recommendation or AUC are available. Importantly, the development of AUMs and SSMs must meet performance measure and quality metrics standards, including demonstrating a gap in care, formulation of metrics, and demonstration that these metrics have measurability, feasibility, reliability, impact, etc. Once developed, AUMs and SSMs would go through a similar path of validation through the Task Force, including a period of public comment and peer review, and then the formal endorsement of ACCF and AHA leadership. During this process it would be determined whether the proposed AUM or SSM is best suited as a quality metric used only for internal improvement or should be considered for public reporting and accountability, similar to performance measures.

The balance of benefit and harm in AUMs/SSMs can be more subtle than that found with usual performance measures. In many cases, an AUM/SSM is developed in recognition that traditional LOE A, although potentially feasible to obtain, is likely never to emerge for every indication or application of the technology. AUM/SSM development may also include measures of outcomes, including procedural complications, patient experience of care, costs, and, in the case of diagnostic tests, the impact of the technology on risk stratification, management decisions, and efficiency of care delivery. In these instances, the balance of benefit and harm is often less clear-cut than for the targets of the usual performance measures.

When properly constructed with an appropriate methodology, tested in implementation pilots, and supported by strong, formal consensus among multiple stakeholders, selected AUMs/SSMs have the potential to be useful for public reporting and accountability. An example of an imaging AUM that could be used for public reporting would be a measure that reports rates of inappropriate imaging within practices (10). Such a measure would be based on the AUC for imaging developed by the ACCF using a formal, prospectively defined consensus process (8).

In this respect, the imaging and revascularization AUC developed by the ACCF (8) can provide important guidance and be embedded in the AUM pathway for development of technology measures (Figure 1).

The challenges in applying the existing ACCF/AHA measurement methodologies for performance measures for cardiovascular technology are detailed in Appendix F. The remainder of this document describes the proposed methodology of the ACCF/AHA Task Force for developing AUMs and SSMs.

The writing committee has divided the quality measures for cardiovascular technology into 2 distinct domains to reflect the differing immediate goals of diagnostic and therapeutic technologies (information versus treatment) that will influence the associated measures of quality. Furthermore, the existing care structure and its associated accreditation or certification are often specific to diagnostic (e.g., echocardiography services) or therapeutic (e.g., surgical care) technologies.

Defining quality metrics for diagnostic tests is challenging for several reasons. Multiple components that involve different individuals and laboratory functions contribute to a “high-quality” test result (30). (Table 3) provides some examples of current metrics proposed by professional societies and other groups.

First, use of a particular test must be appropriate for the patient and condition, and the benefits of its performance must outweigh the possible harm. Although ACCF and AHA created practice guidelines on diagnostic testing in the past that recommended situations in which a diagnostic test might affect patient management (31- 32), these guidelines are no longer current and have been withdrawn from use as ACCF/AHA policy statements. They have been replaced recently with AUC that indicate whether a certain test or tests (e.g., echocardiography) are appropriate or inappropriate for a given clinical scenario. AUMs can be developed around these AUC documents.

If a test is to be considered high quality, it must be conducted according to the recommendations of the relevant professional society and in consideration of any specifications from the equipment manufacturer (if available). Those conducting the test must have an adequate level of skill, which is often demonstrated by fulfilling training requirements and/or certification. Similarly, the testing laboratory must show that it can adequately perform all the technical processes associated with testing and that its equipment meets all technical standards. This is usually established through accreditation. Although some diagnostic tests can be interpreted by those with standard medical training (e.g., hemoglobin value), other tests (e.g., magnetic resonance imaging) require advanced training for performance and interpretation. Certification of the interpreter is a common way of demonstrating such training and skill. SSMs could be developed around these metrics of training, accreditation, and credentialing. In addition, test results that depend heavily on interpretation (e.g., stress imaging) should be periodically examined at each laboratory to determine whether the test characteristics in actual practice match published standard values. Ideally, for example, the test performance (sensitivity and specificity; area under the receiver operator curve) of stress imaging for detecting coronary artery disease should be examined both at the level of the laboratory and the persons interpreting the test results.

Once the diagnostic test is complete (including interpretation if needed), the results need to be communicated to the referring provider in a timely manner. Such reporting should follow standards for clarity, completeness, and definitiveness (30), and this could be another topic for AUM or SSM development. A high-quality laboratory will also provide secure but easy access for appropriate persons to view actual test results. Finally, a high-quality test should contribute to improved outcome. Proof of such a contribution is challenging to obtain given the many factors that affect patient outcome and the relatively small contribution that would be expected from an individual diagnostic test. However, opportunities to examine the test-outcome link may arise as the adoption of electronic health records (EHRs) allows compilation of large longitudinal clinical data sets. An intermediate step would be to determine, within an accountable unit, whether a diagnostic test has a positive impact on downstream care by “reclassifying” patients who do or do not need specific therapy or procedures.

The design of quality measures for therapeutic/interventional procedures faces many of the same challenges as those for diagnostic procedures. Although certain aspects of procedural outcome are readily defined and captured, evaluation of procedural success is nonetheless complex. Different categories of interventions (e.g., vascular, valvular, and cardiac arrhythmia) have unique considerations for appropriateness, indications, measures of success, and complications. The writing committee recommends evaluating the performance of therapeutic/interventional procedures according to the components of such procedures. One such classification might include the following:

The following text expands on these 4 components.

Preprocedural evaluation to determine whether a procedure is justified requires several layers of investigation. First, individual patient characteristics must be assessed to determine whether the performance of any procedure (versus no procedure) is necessary. This must include discussion of the problem with the patient and family members, explanation of the benefits and risks of both performing and not performing the procedure in general and for this individual, discussion of possible alternative therapies, and careful inquiry about patient preferences. Once it is determined that therapy is justified, it must be decided whether a procedural intervention is best, as opposed to other treatment modalities, such as pharmacologic, nonpharmacologic, or behavioral therapy. In some cases, one intervention must be weighed against another (PCI versus coronary artery bypass surgery, or percutaneous atrial fibrillation ablation versus surgical maze procedure). Decisions guiding therapeutic interventions depend in many cases on the results of diagnostic technologies discussed earlier in this document. Furthermore, in many cases, determination of the most appropriate therapy requires consideration of the individual patient's values and preferences and adequate patient education. In addition, once a decision is made to pursue a procedural intervention, further diagnostic technologies may be needed to gather additional information necessary to maximize chances of procedural success.

Determination of procedural success should be tailored to the individual procedure under consideration and may be simple or complex. In most cases, a distinction must be made between acute (short-term) and long-term success of an intervention (Table 4). In general, short-term procedural success is related to technical aspects of the procedure, surrogate measures that are expected to correlate with long-term outcomes, and the absence of major procedure-related complications. Although developing such measures is not easy, determining clinical or long-term success can be more challenging to assess. Relief of symptoms, improved health-related quality of life, reduced need for medications or hospitalization, and improved survival can be reasonably straightforward to assess in the individual patient, but other outcomes are more difficult to assess directly and may require comparison with the natural history of disease, which can only be defined in populations.

Immediate technical or procedural success may not predict long-term success. For example, assessment of quality of the outcome following interventions for arrhythmias may be challenging, because arrhythmias are often intermittent and may occur at random and/or without precipitating factors. Thus, assessment of the success of catheter-based (or surgical) ablations requires knowledge of the number and frequency of the patient's arrhythmia episodes before the intervention to determine the duration of follow-up necessary to assess whether the intervention has cured or ameliorated the arrhythmia. The use of surrogate endpoints such as inducibility of tachycardia or termination of an ongoing tachycardia may serve as useful measures of immediate technical success but are of uncertain value in predicting long-term clinical success. Similarly, late complications, such as device infection, or disease progression may complicate assessment of long-term success.

Even within a specific procedure, such as implantation of an ICD to prevent cardiac arrest/sudden death, the specific indication may affect evaluation of success. In the case of ICD implantation after resuscitation from cardiac arrest, the risk of recurrent arrest has been fairly well defined. In the case of prophylactic implantation of an ICD in a patient thought to be at risk for sudden death, however, the actual risk is usually unclear. Further complicating assessment of success in prevention of sudden death is the fact that the common surrogate endpoint of “appropriate” ICD discharge is acknowledged to considerably overestimate actual reduction of sudden death. Finally, whatever the metrics used, the metrics must have meaning from a patient perspective.

Once metrics are established, a final challenge in evaluating the utility of interventions is assessment of adverse outcomes. Recognition of acute complications is usually obvious. However, recognition of late complications may be difficult due to inability to relate a given complication directly to a specific procedure. In addition, the person who performed the procedure or the laboratory in which the procedure was performed may not always learn of late complications. Without this information, however, the true success of the procedure, as well as evaluation of the quality of institutions and physicians, cannot be assessed.

It should be apparent from this discussion that assessment of both short-term or technical success and long-term or clinical success of therapeutic interventions is complex and challenging. As such, multiple factors and systems are required to evaluate optimal performance of therapeutic procedures. (Table 4) provides some examples of possible indicators for a variety of cardiovascular therapeutic procedures. However, crafting actual quality metrics for any of these would require careful consideration of a range of factors, including acuity of the illness, the skill of the operator, the experience of the support team, the health literacy of the patient, and the social determinants of health affecting quality of life, among other issues.

Accurate, timely, and clear communication of procedural results is critical to ensure optimal utilization and outcomes. Dissemination of results can be viewed at several levels. First, the patient and family members must be told what was accomplished, whether the expected technical goals of the procedure were met, the anticipated effect of the procedure on the patient's health, what restrictions may be imposed temporarily, and what changes in medications (including additions or subtractions) may be expected.

The second level of communication is to other physicians and providers involved in the patient's care. Verbal and written reports must clearly express positive and negative results of the procedure. If critical results of the procedure are not plainly expressed in written reports, alterations in aspects of the patient's care, such as medication adjustments, may not be carried out. For example, necessary changes in anticoagulant or antiplatelet therapy resulting from the procedure must be clearly stated, including the duration of these changes.

The third level of communication is dissemination of procedural results in the hospital and patient's medical record so that physicians caring for the patient in the future understand what procedure was performed, why it was performed, and the observed outcome. Failure to accurately and completely disseminate results could impact the patient's outcome as negatively as procedural failure.

Consistent development and testing of quality measures for cardiovascular technology will follow a guiding framework and ensure value and confidence in the end product. In an attempt to standardize and define the sequence of decisions in the construction of AUMs and SSMs, the writing committee has outlined a set of steps that reflect a common process.

There are 2 general approaches to the construction of quality measures. A proactive approach identifies a particular area in which it is believed a panel of metrics would improve patient care and outcomes. In this case, the developers begin with a “blank slate” and try to create a comprehensive yet manageable set of metrics that will reflect quality in the area. The outline below describes the sequence in such a process.

A reactive approach begins in response to a recognized problem. In this case, it might be expected that some evidence of a specific problem already exists, so the sequence of steps would begin at Step 4 below. Note that this approach may be less desirable than the proactive strategy, which is typically more comprehensive.

In many cases, AUMs and SSMs will not have the benefit of substantial real-world information about important aspects of feasibility, reliability, and validity. (This is also true of many performance measures derived from evidence-based guidelines.) These parameters should then be assessed in a testing phase. This testing should also attempt to identify any unintended consequences of measurement that might occur.

Several additional considerations are important components of the AUM development methodology, including oversight, evidence development, review, approval, implementation processes, and relationship to other ACCF/AHA documents.

The responsibility and authority for the creation, review, and promulgation of AUMs and SSMs is identical to that for performance measures and rests with the ACC/AHA Task Force. Specifically, only the Task Force can convene a writing committee, and that committee will work under the guidance and approval of the Task Force and in compliance with all other standard policies and procedures of the ACCF and AHA for development of such measures. This includes consideration of members' relationships with industry or other entities in constituting the writing committee and broad representation on the committee, including persons representing key stakeholder organizations and those with expertise in methodology. In considering whether to commission an AUM or SSM writing committee or add consideration of AUMs or SSMs to existing performance measure writing committees, the Task Force should require that there is an urgent clinical or quality need for a measure(s) and that the benefits would outweigh the harm of creating such measures.

Although conventional performance measures generally require LOE A, it is recognized that potential topics for AUMs or SSMs will rarely be supported by this level of evidence, yet are still important to quality of care. In the absence of high-quality evidence, the Task Force may use LOE C as generated by a rigorous prospective process as previously described. In practice, expert consensus as contained in documents developed by ACCF or AHA writing groups such as the AUC (in the case of AUMs) or other scientific or credentialing documents (in the case of SSMs) will be considered to meet this high standard and can be considered as the basis of quality measures. In addition, for technical topics not generally in the purview of ACCF or AHA documents, similar consideration for SSM development will be granted to documents of specialty professional societies as noted previously, provided that they have been endorsed by the ACCF and/or AHA and that they meet these organizations' established standards for document development, including an unbiased writing group, process transparency, peer review, and organizational approval.

AUMs and SSMs will be expected to undergo a review and approval process similar to that for any performance measure promulgated by the Task Force. This includes adherence to established ACCF/AHA methodology as well as Task Force policies and procedures, followed by formal peer review, public comment on the proposed measures, and approval by ACCF and AHA leadership. The writing committee also strongly recommends that any AUM or SSM deemed appropriate for public reporting or accountability purposes be submitted to a national consensus organization for endorsement consideration.

Although performance measures are derived from ACCF/AHA guidelines, the ability to construct AUMs and SSMs based on other endorsed or approved documents means that they may address issues not yet included in guidelines, as allowed by current ACCF and AHA policy for other statements and documents. However, as new evidence is developed (whether LOE A, B, or C) and new documents are written, it is expected that any relevant ACC and AHA documents, including AUMs and SSMs, would need to be revised to remain concordant. If there is an area where head-to-head nonconcordance occurs, writing committees, with the review/approval of ACCF and AHA leadership, must resolve the issue before approving it as policy, after which the non-concordant document will be updated and brought into concordance. In general, if evolving evidence leads to new guidelines recommendations that produce formal performance measures, these would take precedence over AUMs or SSMs.

The Task Force recognizes the need for performance measures and quality metrics for cardiovascular technology. This document proposes a basic methodology for development of such measures for optimal use of technology. Because LOE A is often lacking, the Task Force has identified 3 distinct pathways for technology measure development, all falling under the umbrella of quality measurements (Figure 1). First, existing performance measure methodology can be used when the therapeutic procedure or diagnostic test is an essential precursor or guide to implementation of a Class I recommendation or when there is an obvious patient safety concern. Existing performance measures have been developed in this manner. When this is lacking, a second pathway is proposed for development of such measures, termed AUMs, that includes reliance on existing AUC. When both guidelines recommendations and AUC are lacking, evidentiary standards based on formal expert consensus manifested by ACCF- or AHA-approved or endorsed standards can lead to development of SSMs. When properly constructed, tested in implementation pilots, and supported by strong consensus among multiple stakeholders, AUMs and SSMs have the potential to achieve the standards for purposes of public reporting and accountability. In other cases, measures created by these alternative pathways may be more suitable for use in internal quality improvement.

Unlike development of structural and process performance measures based on Class I guidelines recommendations, in which there is growing experience in clinical practice, development of AUMs and SSMs presents unique challenges. These are related to lack of precedents, guidelines, and a strong evidence base linking diagnostic testing or technical aspects of a procedure to clinical outcomes. These challenges are seemingly less daunting for those therapeutic procedures in which outcomes data are available or guideline indications are established. Nonetheless, measuring the quality of such procedures and their effectiveness at the level of the individual physician or procedural team will represent new hurdles.

The writing committee strongly recommends that AUMs and SSMs be developed under the aegis of the Task Force and follow standard policies and procedures of the ACCF and AHA for development, approval, and promulgation of such measures.

As such, the development of performance measures, AUMs, and SSMs for diagnostic and interventional procedures must adhere to the rigorous criteria established for such measures. A needs assessment should identify a gap in care or a quality issue. Ideally, these measures should be tested and evaluated prospectively to confirm that that they provide satisfactory feasibility, reliability, and validity before they are implemented broadly. As is the case for all quality measures, unintended consequences are likely to arise and need to be scrutinized. Evaluation should also be conducted in a variety of practice settings, including solo practices, medium-sized practices, and large practices, and also for inpatient procedures in hospitals serving communities of varying economic resources in urban, suburban, and rural locations. Such testing in real-world settings will be informative in future modifications of measurement design. In particular, because the concept and development of AUMs and SSMs represent a new paradigm, review of the early experience with quality measures related to cardiovascular technology is essential to ensure that they contribute meaningfully to quality of care.

Jensen S. Chiu, MHA, Specialist, Clinical Performance Measures

Several issues make development of performance measures for cardiovascular interventions, imaging procedures, and other technological advances challenging. These special challenges can be broadly classified as issues of 1) evidence development; 2) association with outcome; and 3) measurement feasibility, attribution, and application.

Many diagnostic and therapeutic procedures evolve into clinical practice without well-run and adequately powered RCTs and thus lack specific Class I, LOE A data for several reasons. First, many diagnostic and therapeutic procedures involve rapidly evolving technologies. Thus, it is often challenging to determine the optimal time to perform trials assessing these procedures. Ideally, a technology-based procedure would be mature enough that chances of proving its efficacy are maximized. Yet if a technology has already been widely adopted, its evaluation in trials is rendered more difficult due to acceptance into practice and a resulting lack of equipoise on the part of clinicians and others.

A second challenge for evidence development for diagnostic and therapeutic procedures relates to their potential dependence on operator skills. Unlike a drug whose dose and delivery can be standardized by protocols in an RCT evaluation, procedural safety and benefits are commonly related to the operator's experience and technical skill. RCTs, which typically include highly skilled operators, may overestimate the safety and effectiveness of a technology in the hands of a practitioner with a lower level of experience or skill.

A third concern is that RCTs must involve a large number of patients, often resulting in heterogeneous patient populations, and the relative benefit-risk of applying the procedure may vary greatly with different clinical indications. The result is that application of the trial results to what the clinician faces—an individual patient rather than the “average” patient—can only be achieved by inference or expert interpolation. This is true even of LOE A evidence incorporated into guidelines.

A fourth issue for evidence development in certain procedural areas is one of ethics and practicality. For example, there are no direct RCTs demonstrating that the timeliness of primary PCI for acute myocardial infarction is important or lifesaving. However, it would be unethical to randomize patients to slower reperfusion therapy just to prove this point. Similarly, delivery of diagnostic test reports in a timely and complete manner falls into a “common sense” category, and it would not be ethical to subject patients to an RCT evaluation.

These issues require that cardiovascular guidelines often use less robust evidence (LOE B and C) to support their recommendations (33). Such evidence is deemed acceptable to some experts (7) for use in the development of performance measures, provided it is used in conjunction with all other available evidence (if any) and is developed through a prospectively defined, formal consensus method. Informal or ad hoc expert consensus methods are not acceptable because they do not provide the necessary rigor to reliably mitigate the lack of RCT data, nor do they provide reliable, reproducible LOE C data.

Expert opinion is not considered to be empirical evidence except when acquired using an unbiased expert group following a formal, disciplined process such as the Delphi method, nominal group technique, “modified nominal group technique” or “modified Delphi,” or a consensus development conference (34). The specified process must be transparent, open to peer review, and publically reported. Other criteria for the use of expert opinion in the development of quality measures when no other evidence is available, as stated by the NQF, include a strong, explicit rationale for why the specific structure or process should be the focus of a quality measure (35). Finally, the experts must explicitly address the certainty or confidence that benefits to patients from the specific process or structure greatly outweigh potential harm.

Developers of quality measures for diagnostic procedures must recognize the indirect link between testing and outcomes, given the lack of direct therapeutic impact. Although diagnostic procedures can uncover information that may alter patient management and ultimately lead to improved outcomes, they are not directly therapeutic. Instead, any impact on outcomes depends on the clinician fulfilling the central role of receiving and acting appropriately on the diagnostic data, including institution of effective evidence-based therapy. Furthermore, patients must adhere to therapeutic recommendations to preserve the link from diagnostic testing to outcomes.

A parallel issue to the effectiveness or benefits side of diagnostic testing has to do with its safety or adverse consequences. Although some diagnostic procedures may have a low safety concern, others, such as procedures with ionizing radiation, may have adverse consequences for patient health. Furthermore, an inaccurate diagnostic study will have significant downstream consequences in terms of missed diagnosis of disease until an advanced stage on the one hand or unnecessary testing, procedures, and patient concern on the other.

Although drawing a direct association between technological procedures and clinical outcomes can be challenging, there is a clear, direct association between the volume and complexity of diagnostic and therapeutic procedures and cost. Measure development groups have traditionally not included economic considerations in their construction of performance measures. However, the performance of procedures lacking significant clinical benefit can markedly drive up healthcare resource utilization, either directly or through detection of “incidental findings.” In fact, diagnostic imaging has been the most rapidly increasing component of healthcare costs in patients covered by Medicare (36). Although economic considerations are usually not a key factor in the development of performance measures, cost provides a seemingly simple, measurable, and direct association with diagnostic and therapeutic procedures that may hold some appeal as a quality metric. However, the full understanding of the economics of diagnostic testing and therapeutic procedures requires complex modeling of the net costs. For example, the design of such models depends on the selection of the optimal design perspective (e.g., that of the patient, society, or payer) for which no consensus exists. Cost considerations may also take several different approaches, such as the cost (and value) of the technology, cost-effectiveness of the technology itself versus the comparative relative costs, and cost-effectiveness when implemented by different providers.

A feasible performance measure possesses the attribute that the required data can be obtained in a timely manner with reasonable effort and cost. Some proposed diagnostic and therapeutic AUMs/SSMs may lack this attribute. A tradeoff may exist between ease of data collection and quality of the measure. For example, claims data provide ease of data collection regarding procedural performance but lack fidelity to determine the clinical circumstances regarding testing appropriateness, accuracy, or association with subsequent outcomes. Registries of diagnostic or therapeutic procedures provide an alternative to claims data. Procedure-specific registries (e.g., the National Cardiovascular Data Registry for cardiac catheterization, PCI, carotid stenting, and ICDs) successfully demonstrate the feasibility of high-quality, multicenter data collection and its power to identify clinical quality issues (37). Regional or even hospital-level pilots provide potentially feasible alternatives with enhanced organizational relevance, although potentially limited in statistical power or external generalizability. These labor-intensive efforts generally have required either grant support or mandates from payers or regulatory bodies. An expanding future alternative to stand-alone clinical registries is the harvesting of information on diagnostic and therapeutic procedure quality directly from EHRs. However, such integrated EHR systems would need to be ubiquitous, collect detailed data using standardized definitions, and share data for aggregation. Within present federal mandates for expanded use of EHR systems, the opportunity to develop this mandate in an environment of standardized data definitions and uniform scope could enable data aggregation on regional and national levels.

Assigning the responsibility for ensuring optimal use of technology or procedure performance is often challenging and has ramifications for clinician autonomy, practice work flow, reimbursement, and interprovider relationships (2). Attribution for the selection of a procedure should involve both the ordering physician and the performing laboratory. Ideally, the primary accountability for the optimal use of a test or procedure should be with the ordering physician, who understands the clinical scenario best, while acknowledging that the performing laboratory may understand the strengths and weaknesses of a given procedure in a given patient better than the ordering physician. Additionally, processes such as decision support are most useful when provided directly to the referring clinician at the point of care, when the test/procedure is being ordered. Frequently, however, the concern about overuse is focused where the procedure is being performed, because reimbursement may be denied if documentation of clinical need cannot be provided. For example, the imaging laboratory is at financial risk when inappropriate procedures are performed. Yet to deny service may not take into account the clinician's judgment about a specific patient and may jeopardize the referring physician's future use of that service.

Clearly, joint attribution for the optimal use of diagnostic testing and other procedures is needed. The proceduralist/laboratory must provide education and feedback to clinicians about their referral patterns and offer suggestions in an objective and professional manner. Simple denial of performance of a procedure will not necessarily alter overuse or misuse, because this may lead to use of other laboratories rather than modification of procedure ordering. However, laboratory physicians must be engaged in the process, because only this group realizes financial gain from providing these services. Learning communities have supportive laboratories that understand how to improve utilization and communicate with referring physicians, and the ongoing use of clinical decision support can provide near-continuous feedback at the point of test/procedure ordering.

The current feasibility of performance measurement is limited for many diagnostic and therapeutic measures. Although claims data can often capture whether a procedure was done, they cannot be used to explore the clinical reasons for why the procedure was performed, the results of the test, the accuracy of these results, or their impact on subsequent clinical decision making. In this respect, EHRs may also aid in implementation of performance measures by facilitating collection of information on patient selection and appropriateness.

Finally, as with all measures used for accountability or public reporting, technology measures may be subject to unintended consequences. The most obvious of these are related to underuse of technology in an attempt to curb inappropriate use. The application of AUC to prevent overuse of diagnostic or interventional procedures could result in a reduction in their use, even among patients in whom these procedures are not only appropriate but clearly indicated. Thus, technological advances may be withheld from patients who are likely to benefit from them. Imaging examples include symptomatic patients not undergoing diagnostic evaluation for ischemic heart disease or asymptomatic patients with valvular regurgitation not receiving timely follow-up echocardiograms. Coronary artery bypass surgery or PCI report cards could result in higher-risk patients not receiving indicated myocardial revascularization, as has been speculated in the New York State experience (38- 39). Similarly, door-to-balloon measures could leave some patients without satisfactory reperfusion for acute myocardial infarction if emergency personnel are reluctant to provide PCI to patients outside the optimal 90-minute door-to-balloon time period.