This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Jollis, J. G. Search for Related Content PubMed PubMed Citation Articles by Jollis, J. G.

EDITORIAL COMMENT

Measuring the effectiveness of medical care delivery^*

^a Duke Clinical Research Institute, Duke University Medical Center, Durham, North Carolina, USA

Reprint requests and correspondence: Dr. James G. Jollis, PO Box 17969, Durham, North Carolina 27715
james.jollis{at}duke.edu

Lacking adequate resources to measure the effectiveness of medical practice, hospitals, health-care purchasers, governments and the media often turn to administrative data. In this issue of the Journal, Tu et al. (1) suggest a measure of medical effectiveness based on Ontario administrative data. Analyzing 52,616 patients hospitalized for acute myocardial infarction between 1994 and 1997, they identify 11

claims descriptors most associated with 30-day and 1-year mortality. The work goes on to externally validate the findings, demonstrating that these same descriptors and their corresponding regression model coefficients are similarly associated with mortality in administrative data from Manitoba and California.

In choosing from among 43 candidate variables for the final Ontario prediction rule, these researchers use reasonable selection criteria, including previously published risk factors, strength of association with mortality and clinical intuition. Two other considerations that may add to the suitability of claims diagnoses in mortality comparisons include the likelihood that a diagnosis represents a coexisting illness rather than a complication, and the potential for "gaming" of the variable. A significant concern regarding the use of claims data in mortality comparisons involves the difficulty in discerning diagnoses present on admission from those that accrue during hospitalization. In the latter case, the inclusion of complications in regression models gives hospitals credit for complications. For example, a patient with a large myocardial infarction who is not treated with reperfusion therapy may go on to develop shock or congestive heart failure, and these diagnoses would increase expected mortality in regression models, effectively "lowering the bar" for the involved hospital and masking less effective care. Claims data from Ontario may potentially avoid this dilemma, because hospitals in this province are required to add an additional level of detail to hospital discharge diagnoses that identifies them as either preexisting or occurring after hospital admission, and only diagnoses identified as present on admission are used for risk adjustment. For systems lacking the level of detail available in Ontario, diagnoses in the suggested prediction rules such as acute renal failure and cardiac dysrhythmias may represent complications for which hospitals "earn credit" as noted earlier.

Regarding gaming, in settings where hospital claims are used to rank health-care providers, public scrutiny may encourage providers to code high risk diagnoses whenever possible. Such a strategy would lead to an apparent improvement in outcomes due to manipulation of hospital discharge abstracts, rather than any real improvement in care. The reporting of diagnoses, such as shock, present among 2.5% of patients in the Ontario data, and approximately 7% of patients in clinical data, should be monitored at the hospital level to differentiate actual improvements in mortality from "coding creep," or increased specification of mortality model components (2).

These authors consider the ability of their models to account for mortality differences through a variety of measures, including calibration, or a comparison of actual to expected mortality, for 10 subgroups sorted by increasing risk. Most prominently featured is the area under the receiver operating characteristic (ROC) curve or c-statistic. This measure divides patients into two groups, alive or dead, and examines how often the model assigns higher mortality risk to the patient that died for all potential pairs of patients. A value of 1 represents discrimination 100% of the time or perfect discrimination, while a value of 0.5 implies discrimination 50% of the time or similar discrimination to that of a coin toss. The values herein of 0.77 to 0.79 are reasonably good for mortality models. In considering these measures, one should be aware that the inclusion of complications in mortality models improves model performance characteristics such as ROC areas and calibration despite obscuring the identification of most effective care. Thus, deciding if data are suitable for outcomes comparisons should also consider factors beyond these performance measures.

Most remarkable is the finding that the Charlson index (a group of 12 illnesses found to be associated with 1-year mortality among 559 New York University general medicine patients over a decade ago) had a similar association with mortality as the newly developed index, with ROC areas of 0.74 and 0.77 (3). This finding supports the notion that comorbid illness remains important across time, health systems and a broad array of medical conditions.

The present work of Tu et al. (1) provides us with prediction rules that have been developed and validated in three separate systems and can be readily applied to claims data. There are a number of settings where this approach may be adopted. First, the Ontario variables should be considered as candidate variables in the design of prospective data collection tools. Second, because the variables are based on the more broadly applied International Classification of Diseases, rather than the Clinical Modification version specific to the U.S. system, the prediction rule could be used in international comparisons of acute myocardial infarction care. These variables and their corresponding coefficients should also be useful in settings where relatively few patients are considered, such as the specification of mortality risk at the individual health-care provider level. In situations involving larger numbers of patients, such as regional or national comparisons, the Ontario conditions should be considered as individual variables without corresponding coefficients to obtain the strongest association with mortality.

Although the Ontario prediction rules may be applied to a number of situations, a fundamental question remains unanswered. Do mortality comparisons based on administrative data help identify effective medical care? In order to answer this question, the authors suggest as a next step the comparison of mortality-adjusted hospital rankings to hospital use of therapies supported by clinical trials. Studies in the past that have compared evidence-based care to mortality rankings have found better care among lower mortality providers. However, these correlations have not been uniform, as the occasional therapeutic practice at higher rated hospitals has implied less optimal care. For example, the Cooperative Cardiovascular Project identified lower use of reperfusion therapies among "America’s 60 best hospitals" despite better rankings for acute myocardial infarction mortality (4). Until mortality comparisons based on administrative data are found to reliably identify better medical practice, their findings should be considered provisional, providing opportunities in which processes and outcomes may be considered in more detail to verify or refute the level of medical effectiveness implied by mortality estimates.

Numerous studies of routine medical practice have documented substantial underuse of randomized trial-supported therapies. For example, only about one half of Medicare acute myocardial infarction patients who were "ideal candidates" for beta-blockers or reperfusion received such therapies (5). If these efficacious therapies were provided to all patients, their practical impact might be doubled. In bridging the considerable gap between clinical trial evidence and routine medical practice, what are the next steps? The refinement of claims data to identify effective medical therapy using the Ontario approach remains a practical and reasonable first step. This approach, however, should be considered an interim solution. In order to resolve fully the gulf between evidence from clinical trials and medical practice, information needs are much greater than current claims data can provide. Physicians and other health-care providers need to know if and when effective medical therapies are being employed, whether treatments found to be efficacious in clinical trials are similarly effective when applied to the general population and where resources may be best directed to improve care.

Recent experience during a European Society of Cardiology/American College of Cardiology exchange program suggested that we are at the threshold of a fundamental change in medical care. Despite differences in language and customs across countries, one feature that remained consistent across U.S. and European hospitals was identical computer technology. This new international electronic standard has placed information systems capable of measuring effectiveness within the reach of health systems in most of the developed world. With such innovation, the vast array of data generated during routine practice may actually be used to systematically improve practice. Rather than further refining claims data, our greatest efforts should involve redirecting the traditional medical practice of meticulously recording information in free text and paper-based systems toward the specification of such information in common electronic formats suitable for comparison across hospitals and health systems. This reconfiguring of the medical information process should be supported by the integration of information from other medical care activities, including scheduling, pharmaceutical administration and test results reporting. These efforts must also be supported by the development of sensible standards to protect confidentiality without overly obscuring information such that medical care cannot be evaluated. The featured Ontario work demonstrates that the tools exist to convert regional electronic medical data into information about medical effectiveness. Ultimately, relatively coarse tools such as administrative data comparisons will be replaced by systems that can directly monitor and guide medical effectiveness. Such an enhancement to the final critical step of therapeutic development will ensure that efficacious therapies proliferate beyond clinical trials so that all of society may benefit from medical discovery.

	Footnotes

 
^* Editorials published in the Journal of the American College of Cardiology reflect the views of the authors and do not necessarily represent the views of JACC or the American College of Cardiology.

	References

Top References

 
1. Tu JV, Austin PC, Walld R, Roos L, Agras J, McDonald KM. Development and validation of the Ontario acute myocardial infarction prediction rules. J Am Coll Cardiol. 2001;37:992–997[Abstract/Free Full Text]

2. SHOCK Registry InvestigatorsHochman JS, Boland J, Sleeper LA. Current spectrum of cardiogenic shock and effect of early revascularization on mortality. Circulation. 1995;91:873–881[Abstract/Free Full Text]

3. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies. J Chronic Dis. 1987;40:373–383[CrossRef][Medline]

4. Chen J, Radford MJ, Wang Y, Marciniak TA, Krumholz HM. Do ‘America’s Best Hospitals’ perform better for acute myocardial infarction? N Engl J Med. 1999;340:286–292[Abstract/Free Full Text]

5. Marciniak TA, Ellerbeck EF, Radford MJ, et al. Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project. JAMA. 1998;279:1351–1357[Abstract/Free Full Text]

This article has been cited by other articles:

				K. A A Fox, O. H Dabbous, R. J Goldberg, K. S Pieper, K. A Eagle, F. Van de Werf, A. Avezum, S. G Goodman, M. D Flather, F. A Anderson Jr, et al. Prediction of risk of death and myocardial infarction in the six months after presentation with acute coronary syndrome: prospective multinational observational study (GRACE) BMJ, November 25, 2006; 333(7578): 1091 - 1091. [Abstract] [Full Text] [PDF]

				C. B. Granger, R. J. Goldberg, O. Dabbous, K. S. Pieper, K. A. Eagle, C. P. Cannon, F. Van de Werf, A. Avezum, S. G. Goodman, M. D. Flather, et al. Predictors of Hospital Mortality in the Global Registry of Acute Coronary Events Arch Intern Med, October 27, 2003; 163(19): 2345 - 2353. [Abstract] [Full Text] [PDF]

				W. L. Holman, R. M. Allman, M. Sansom, C. I. Kiefe, E. D. Peterson, K. J. Anstrom, S. S. Sankey, S. G. Hubbard, R. G. Sherrill, and for the Alabama CABG Study Group Alabama Coronary Artery Bypass Grafting Project: Results of a Statewide Quality Improvement Initiative JAMA, June 20, 2001; 285(23): 3003 - 3010. [Abstract] [Full Text] [PDF]

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Jollis, J. G. Search for Related Content PubMed PubMed Citation Articles by Jollis, J. G.