With the recent publication of hospital-specific mortality outcomes of isolated coronary artery bypass grafting (CABG) and percutaneous coronary intervention (PCI), Massachusetts joined a growing number of states in publicly reporting risk-adjusted cardiac outcomes (1). Although there are a number of reasons to enthusiastically support public reporting, there are potential risks that should be considered when evaluating the overall benefits and costs of such programs.

The most compelling justification for the public reporting of clinical outcomes is the public's right to know about the care that they are likely to receive from hospitals and physicians they use. Such transparency of information should allow patients to make better informed decisions about their health care choices. Implicit in this reasoning is that the public be provided with accurate and meaningful data and that access to care remains unchanged. As will be described in the following text, these goals may be more elusive to achieve than hoped by the designers of public reporting programs.

Beyond the right to know, important potential additional benefits of public reporting include accelerating the adoption of “best practices” from successful medical centers, as well as leveraging the scrutiny of performance, which typically increases attention to process and quality improvement (the Hawthorne effect) (2). As a result of mandated reporting efforts, high quality datasets have been established and used for critical outcomes research (3- 5). Importantly, public reporting provides accountability and transparency in regard to quality assurance, and thereby enhances trust between patients, regulators, payors, and providers (6).

Alongside these clear benefits of public reporting, however, there are several limitations to current programs and concerns regarding unintended consequences of these efforts. Developing optimal data collection instruments and assuring adequate data quality from participating centers are significant challenges. In addition, it has been difficult to develop risk adjustment methods that adequately account for the severity of illness in extremely sick patients (7- 9). Although these risk-adjustment models have demonstrated excellent discrimination, calibration, and goodness-of-fit in the overall patient populations studied, there are concerns that these models do not adequately address the patients at highest risk (8). Finally, as demonstrated in recent surveys of interventional cardiologists, performing physicians may not fully accept the accuracy of risk adjustment (10), leading to avoidance of higher-risk patients, and providing perverse incentives to perform procedures in lowest-risk patient populations.

The first Massachusetts public report of PCI risk-adjusted mortality was released in 2005 for cases performed during 2003; it demonstrated comparable performance by all Massachusetts hospitals, as measured by the “Standardized Mortality Incidence Rate” (11). The statewide unadjusted in-hospital mortality after PCI was 1.71% (n = 12,657 total PCI cases for the last 9 months of 2003). Results from this report indicated that Brigham and Women's Hospital outcomes were within expectations given our case-mix, although higher-risk patients appeared to experience slightly worse outcomes compared with statewide averages than the lower-risk population (neither difference was statistically significant). Given the impact of the report on external assessment of the quality of care at our institution, we undertook a detailed exploration of the factors associated with mortality after PCI. Between January 2003 and December 2005, there were 85 in-hospital deaths among 5,050 patients receiving PCI performed at Brigham and Women's Hospital, with an unadjusted mortality rate of 1.68%. Although only 2.69% of patients had presented in cardiogenic shock, these patients represented more than 54% of the mortality after PCI, with such patients experiencing a 68-fold increase in the risk of death as compared with patients not presenting in shock.

As shown in (Table 1), 44.7% of patients who died before hospital discharge had at least 1 severe acute medical condition present before the index PCI procedure that was not accounted for in the data collection instrument used by the state mandated effort (the American College of Cardiology–National Cardiovascular Data Registry) (12). Typical examples of such severe acute comorbidities included advanced malignancy, active infection, acute stroke, perioperative myocardial infarction after major noncardiac surgery, and anoxic brain injury.

We sought to examine the mortalities further through detailed review of the clinical record and angiograms in an effort to classify the deaths into 1 of 3 categories: 1) no complication of the procedure thought to have contributed to death; 2) complication of procedure possibly related to patient's death; and 3) procedural complication materially contributing to the patient's death. Charts and films were reviewed independently by 2 board-certified interventional cardiologists blinded to both the identity of the patient and the performing interventional cardiologist. Determinations of causality were made based on major neurologic, vascular, or cardiac complications occurring during the procedure or during the hospitalization after the PCI. Examples include intracerebral hemorrhage, major vascular complications requiring surgery or leading to hemodynamic instability, coronary complications including dissection and loss of vessel, or acute or subacute stent thrombosis. Of the 85 deaths, 11 (13%) were categorized as being related to a complication of the PCI procedure. An additional 7 (8%) were determined to be possibly related to the PCI procedure. The majority of deaths (67 of 85, 79%), however, had no identifiable complication of the procedure that was plausibly related to the patient's death. Implicit in the rationale for using mortality as an end point of quality is that deaths are a reasonable surrogate for the overall quality of the care being provided. These data, however, illustrate that less than one-quarter of all deaths were possibly related to the PCI procedure itself.

Based on this analysis, we sought to improve the performance of the standard risk prediction model by adding available pre-procedural data elements including presentation with neurologic compromise after a presenting cardiac arrest, history of malignancy, in-hospital onset of acute coronary syndrome (e.g., after noncardiac surgery), and presentation to hospital with sepsis. The revised model was then tested using a backward selection algorithm on a boot-strap developed multivariate risk model using our single center PCI experience since 2005. This analysis of 4,921 consecutive PCI cases demonstrated that the addition of the 4 additional covariates modestly improved the discrimination of the model, with an improvement in the area under the receiver-operating characteristic curve from 0.919 to 0.937. However, this improvement was not statistically significant, with a pair-wise comparison for improvement in model discrimination having a p value of 0.171, despite adequate power (>80%) to detect a difference between the 2 models.

Although inconclusive, this initial analysis suggests that there may be additional value to expanding the existing risk prediction models to include high risk markers available at case presentation. In support of this hypothesis is the evidence from Massachusetts, which has recently begun to use a composite additional risk factor (“compassionate use PCI”) in the risk adjustment model for Massachusetts PCI outcomes for 2006. The “compassionate use” variable was developed to identify uniquely high-risk cases that were taken for PCI when the long-term prognosis of the patient was unclear to the operator, but when there was a class I indication for emergent revascularization. These included survivors of cardiac arrest with neurologic impairment in the setting of ST-segment elevation myocardial infarction (STEMI), use of percutaneous ventricular support systems to facilitate high risk PCI, and survivors of multiple cardiac arrests en route to the hospital.

Given the complexity and acuity of the patients treated with PCI, our analysis would suggest that overall mortality for a given provider is greatly influenced by the severity of illness of the patients the operator is willing to take to a procedure. We hypothesize that the impact of severity of illness may be underappreciated by current risk models and could significantly affect the estimation of quality of care by reporting agencies. However, the analysis above is based on a single center's experience, and may not be representative of centers that do not perform within the state's mortality predication expectation. Nonetheless, it would seem prudent to consider the addition of adjudicated outcomes of whether a death after PCI was either likely or possibly related to the procedure, as unrelated deaths in patients with clear indications for PCI procedures should not be counted “against” institutions or operators.

Data from the New York PCI registry revealed that the mortality after PCI has declined from 0.90% in 1997 (13) to 0.58% in 2003 (14), a reduction of 36% (p < 0.001). Although some point to this reduction in mortality as a success, it is important to realize that this improvement in outcomes can be attributed to many potential factors. It is possible that the reduction in mortality is attributable to improvements in the quality of care in the state. However, a simultaneous trend toward avoiding performance of PCI in higher-risk patients may also have contributed to the observed reduction in crude mortality in New York State. Evidence supporting this hypothesis includes an observed simultaneous decline in the proportion of patients presenting in cardiogenic shock treated with PCI in New York from 1.21% in 1997 to 0.85% in 2003—a 30% decrease (p < 0.001) (13- 14). During this same time period, there has been increasing recognition of the importance of emergent revascularization for these patients based on landmark clinical trials and supported by consensus guidelines for the treatment of STEMI complicated by cardiogenic shock (15- 16). In addition, data from the National Registry of Myocardial Infarction suggest increasing incidence of cardiogenic shock. The most provocative data come from a recently published retrospective study of the SHOCK (Should We Emergently Revascularize Occluded Coronaries for Cardiogenic Shock) registry comparing the outcomes of patients from New York and non-New York patients (17). New York State patients presenting with cardiogenic shock were less likely to receive angiography, PCI, or CABG. Although there was no significant difference in mortality for patients undergoing revascularization for shock between New York and non-New York patients, in-hospital mortality for New York State sites was 1.5-fold higher for patients not revascularized (p = 0.013 for interaction of New York site and revascularization). Further illustrating the difference in treatment patterns, for patients undergoing CABG in New York State, the mean time to CABG was 10-fold higher in New York patients (101.2 h vs. 10.3 h, p < 0.001). The authors' conclusion from these data was that public reporting has encouraged a risk-averse climate in New York State that has real public health implications.

Likewise, Massachusetts has released a clinical outcomes report covering 2003 to 2005 for PCI (18). As was observed in New York, the proportion of patients treated with cardiogenic shock has declined from 2.28% of all PCI cases in 2003 to 1.29% in 2005, a decline of 43% (Figure 1). This dramatic decline was coincident with the first 3 years of public reporting of PCI outcomes in the state. This decline in the proportion of patients treated with cardiogenic shock was associated with an observed decline in the overall crude mortality after PCI in Massachusetts from 1.71% to 1.56%.

This evidence is observational, and one can only infer a causal relationship between the public reporting of outcomes and the decline in treatment of the sickest patients in the state. It is important to note that for these comparisons, we used the broadest possible definition of shock from the New York registry; shock prevalence was taken as the sum of patients presenting as hemodynamically “unstable” or in “shock.” This definition is more inclusive than the American College of Cardiology–National Cardiovascular Data Registry definition used in Masssachusetts, and therefore would tend to underestimate the differences between the 2 states in the analysis that follows. It is also important to acknowledge that one cannot calculate the true rates of PCI treatment for cardiogenic shock because clinical hospital admission rates for this diagnosis by year are not publicly available at present. Therefore, we have chosen to use a surrogate measure, which is the proportion of patients treated with PCI who presented with cardiogenic shock.

To help understand how public reporting might influence clinicians to avoid the highest-risk cases, we propose a framework of relative risks and benefits and overall clinical acuity. In (Figure 2), the incremental health benefit for a patient undergoing PCI is plotted along the horizontal axis, and the risk of the procedure, here considered to be the likelihood of survival to discharge after the procedure, is plotted along the vertical axis. As shown in (Figure 2), the framework can be divided into 4 quadrants based on the procedural risk (likelihood of survival) and benefit (incremental health benefit to the patient). Shown in green in (Figure 2) is the low risk, high benefit (upper right) quadrant, in which, even in the face of public reporting, there should be minimal disincentive for physicians to perform the procedure. An example of such a case may be an otherwise healthy patient presenting with a non–STEMI, in which case there is substantial health benefit from PCI, and the risk of the procedure is quite low. Conversely, there are some patients in whom the risk is high and the benefit is low, shown in red as the lower left quadrant in (Figure 2). Such a case might include a patient presenting with sepsis complicated by a non–STEMI. The upper left and lower right quadrants have less certain tradeoffs in terms of risk and benefit and are represented in yellow.

(Figure 3) illustrates a variety of clinical scenarios with their putative locations plotted within this relative risk-benefit framework. The patient with sepsis and ST-wave changes is plotted in the lower left quadrant (in red). A series of patients with increasing severity of acute coronary syndromes is plotted in the mid-portion of the framework in gray, with the increasing risk (decreasing likelihood of survival) of the procedure noted along with the increasing incremental health benefits for the patient. At the extreme, the patients presenting in cardiogenic shock have the highest incremental health benefit gains from PCI, although their acute risk is also the highest, with observed post-procedural mortality rates of 30% to 50%. These cases are shown in green on the plot. At the other end of the clinical spectrum are patients with minimal or no coronary artery disease, who would be at extremely low risk after PCI, but who have no clear clinical benefit from the procedure either. Inappropriate use of PCI in these low-risk, but “no benefit,” patients has been observed, as in the series of patients in California who had minimal coronary disease but were nonetheless treated with PCI (19).

Also shown in (Figure 3) is the hypothetical migration of case-mix away from sickest patients toward lower-risk (and lower benefit) patients in the face of public reporting of risk-adjusted outcomes, shown as the dashed red arrow. Public reporting of outcomes can be expected to encourage physicians to favor procedures in which the patients are at lowest likelihood of death because it has been observed that even sophisticated risk adjustment does little to reduce the tendency of physicians to treat lower-risk patients (6). Exacerbating the probable migration of cases from high risk to low risk are the financial incentives to physicians and medical centers, which have been shown to drive increased performance of procedures when capacity for PCI exists (20), and would tend to favor performing an increasing number of procedures in lower-risk cases. The tendency to become more conservative in the face of public reporting of risk-adjusted outcomes is termed “risk avoidance creep,” and may partially explain the observed reduction in procedures for highest-risk patients, despite probable patient benefit. Risk avoidance creep is driven by the concern that public reporting of risk-adjusted mortality rates may result in either reduced patient referrals or ultimately, a loss of the ability to provide the services at all.

Given the importance of the public's right to understand the quality of care available, public reporting of risk-adjusted clinical outcomes should continue and expand beyond simply risk-adjusted mortality. However, recognizing the limitations of public reporting programs and their potential impact on physician behavior, the following principles are proposed in order to maximize the benefits of these programs while minimizing their potential harm.

Currently, outcomes report cards are focused almost exclusively on risk-adjusted mortality and procedural volumes. If physicians do not completely trust the risk-adjustment methods used, there is an incentive to reduce the proportion of the sickest patients treated, in order to have a more favorable report card. It is possible to reduce this incentive by providing additional measures of quality in the outcomes report, which would discourage “risk avoidance creep.” Reporting the center-specific proportion of high-risk cases treated, including all patients who presented with STEMI and, separately, patients in cardiogenic shock, should help balance the incentives. Moreover, additional clinical data elements that are likely to confer extreme risk to patients can be collected and incorporated into the risk adjustment models used. For PCI, these should start with the presence of incessant ventricular arrhythmias and the presence of coma in the setting of an acute coronary syndrome (in which neurologic prognosis is unclear at the time of urgent PCI). Strict and consistent definitions of all data elements as well as mandatory annual audits of participating centers are essential for such a system to be reliable. Consideration of separating out such cases coded (and rigorously audited) as “compassionate use” would allow physicians to continue to treat the sickest of patients without fear that such high-risk cases will adversely affect their report cards due to inadequate risk adjustment.

In addition, emphasis on center-specific public reporting, as opposed to physician-specific reports, could mitigate risk-avoidant behavior by shifting focus from a single person to what should be encouraged to be a team approach. Individual operator outcomes could still be collected and outliers referred for more thorough review by professional or state organizations.

Although states involved in public reporting have emphasized in-hospital mortality, 30-day or longer mortality may be more appropriate and may avoid the problems with accounting for interhospital transfers and discharge to nursing homes or hospice. In addition, there is rationale for nonmortality outcomes, including vascular or ischemic outcomes, which may be useful as indicators of technical proficiency and may have statistical advantages due to higher frequency. Finally, investigation of process measures may also help to illuminate differences in quality. National benchmarks for door to balloon times when treating acute myocardial infarction are an example of such a measure, but also could include discharge on appropriate medical therapy.

Collection of universal, granular, high-quality datasets is costly in terms of personnel and information system resources. It must be recognized that programs that diligently collect, review, and analyze detailed internal clinical outcomes data expend significant resources. For example, Brigham and Women's Hospital employs the equivalent of 2 full-time data coordinators as well as a systems administrator for reporting of outcomes of approximately 1,800 PCI cases per year. This level of infrastructure is required to assure adequate data quality despite comprehensive point-of-care clinical data collection systems and integration of the clinical outcomes database systems with the hospital information systems.

National consensus guidelines regarding the domains, scope, methods, and implementation of public reporting of clinical outcomes should be developed and implemented. Recent recommendations regarding cardiovascular outcomes assessment have been published (21), but broad adoption and implementation of such standards is needed. Strategies must be implemented for dataset development and evolution, with the goal of achieving data element definitions that are operationally specific, clinically meaningful, and as unambiguous as possible. In addition, standardized recommendations regarding optimal statistical methods for risk adjustment and reporting should be promulgated (21). In addition, guidelines as to how and when to investigate outlier performances, based on severity and timing of divergence from expectations should be developed. A recent report from the Institute of Medicine recommended creating a new board within the U.S. Department of Health and Human Services to coordinate development of standardized performance measures (22). It would be logical for such an organization to oversee standards development for public reporting of risk-adjusted clinical outcomes.

In addition to standard reporting of risk-adjusted outcomes and procedure volumes, significant attention should be devoted to studying procedure appropriateness and access to care. Assurance that “risk avoidance creep” does not occur, and that inappropriate low-risk patients are not being treated unnecessarily would likely require surveillance of a sample of representative angiograms from all practitioners. Where concerns are raised, a more thorough examination of practice patterns could be instituted. Equally important in the measures of appropriateness would be institution of measures to ascertain the proportion of high-risk patients who are not receiving generally indicated procedures. Specifically, collecting clinical data on all myocardial infarction patients treated at an institution, rather than only on those who undergo PCI, could provide insight into whether a center is evolving toward a more conservative approach, which may be at odds with its mission for comprehensive care of all patients presenting to that institution.

Our analysis of data from the Massachusetts experience with public reporting raises serious concerns about the unintended impact on clinical care. Clinical outcomes report cards serve a valuable public health purpose and should continue to expand as the public demands accountability from the health care system. However, this analysis raises concern over whether overall mortality is a reliable guide to quality. As practicing interventional cardiologists, we are concerned that we occasionally consider the impact of potential adverse events on our hospital's outcomes when evaluating the risks and benefits of a procedure for a particularly ill patient. However well-intentioned the practice of public reporting may be, the consequences appear to have had the opposite effect. With the thoughtful adoption of principles such as those outlined above, the significant benefits of public reporting of risk-adjusted outcomes may be achieved while minimizing the potentially deleterious effects of reducing physician willingness to perform procedures on patients who would significantly benefit from them.