CLINICAL STUDY: CORONARY ARTERY DISEASE: EDITORIAL COMMENT
The challenge of improving risk assessment in asymptomatic individuals: the additive prognostic value of electron beam tomography?*
Leslee J. Shaw, PhDa and
Robert A. O’Rourke, MD*
a Department of Health Policy and Management, Emory University, Atlanta, Georgia, USA
* University of Texas, San Antonio, Texas, USA
Reprint requests and correspondence: Dr. Leslee J. Shaw, Department of Health Policy and Management, Room 638, 1518 Clifton Road NE, Emory University, Atlanta, Georgia 30322
lshaw{at}cces.emory.edu
The detection of atherosclerosis has changed considerably over the past few decades, with an ever-increasing array of diagnostic tools available to the researcher and practicing clinician. For many decades, traditional risk factor assessment was the mainstay for prognostication, forming a basis for preventive strategies aimed at risk reduction. More recently, new laboratory markers (e.g., C-reactive protein), as well as cardiac imaging techniques (e.g., electron beam computed tomography [EBCT]), are being advocated as optimal tools for the detection of subclinical disease and as predictors of obstructive coronary artery disease (CAD) and worsening prognosis. Electron beam computed tomography is attractive due to the ease of measurement of coronary calcium as compared with other measures, such as carotid intimal medial thickness. Considerable previous evidence suggests that the extent of coronary calcification is strongly related to plaque burden, but is less reliable for detecting vulnerable plaque (1,2). Calcification of the coronary artery occurs as part of the development of atherosclerosis and is not present in a normal vessel wall. As an emerging technique, EBCT has merit but also disadvantages that temper our enthusiasm for its widespread clinical use (2).
Recently, the American College of Cardiology in association with the American Heart Association (AHA) presented an updated technology evaluation on the use of EBCT for the diagnosis of CAD in symptomatic patients, for risk assessment in asymptomatic individuals and for serial monitoring of drug therapy (2,3). Other reviews on EBCT have been published by the AHA Prevention V Conference and Blue Cross/Blue Shield Technology Evaluation Center (Chicago, Illinois). From either the payer perspective or that of medical societies, there is consensus that widespread use of EBCT for disease or risk assessment is not supported by current data (2,3). When examining the predictive accuracy of EBCT, the weight and quality of the evidence are dependent on the rigor with which the data are analyzed, as well as its comparative accuracy in relation to other noninvasive techniques (2). This editorial comment highlights the current state of evidence on EBCT, including examples from the Arad report (4) in this issue of the Journal.
In our cost-conscious era, new technology, such as EBCT or echocardiographic contrast agents, must now provide a sufficiently high level of scientific evidence before acceptance and assimilation into daily clinical practice. Health care payers and other group health organizations also require evidence of favorable outcomes and specific clinical pathways as part of the contractual process for new technology. European drug/device approval agencies and many western European countries are considering the requirement of long-term clinical outcome and cost-effectiveness data before approval and use of any new drug or device. Although these are new standards for diagnostic tests, therapeutic agents have long been held to the benchmark of well-designed, statistically powered, controlled clinical observations that are rigorously analyzed. Only recently have outcomes data on noninvasive testing of more than 1,000 patients been published (5,6).
 |
Challenges of event prediction with EBCT coronary calcium scores
|
|---|
A major benefit of EBCT would be the improved detection of subclinical disease in asymptomatic individuals (2–4). Detection of high risk subsets from asymptomatic cohorts was the focus of the current report by Arad et al. (4) and in a related series from the South Bay Heart Watch Program (4,7). Imaging of atherosclerotic plaque has been done by many techniques, including magnetic resonance imaging and positron emission tomographic imaging. For EBCT, calcification of the coronary arteries provides an additional method to assess plaque burden. From the St. Francis Medical Center and the South Bay Heart Watch Program (4,7), the prognostic value of EBCT was reported using an array of combined end points, including cardiac death (an infrequent event in low risk groups), acute coronary syndromes and coronary revascularization. Although combined end points are often used in small patient samples (e.g., n = 1,172) (4), factors contributing to cardiac mortality differ from those of nonfatal myocardial infarction and other coronary events. In general, the amount of coronary calcium estimates a small percentage of the total plaque burden, with most atherosclerotic lesions not being calcified (1,2,7). Rupture of vulnerable plaque is associated with the occurrence of acute coronary syndromes. Event prediction, including sudden cardiac death or acute coronary thrombosis, more often occurs in lesions with vulnerable plaque but less extensive coronary calcification (8,9). The low positive predictive value of coronary calcium in the current series is expected. The resulting usefulness of EBCT may be problematic in low risk groups in which the majority of events are acute myocardial infarction (i.e., in Arad et al. [4], 15 of 18 events were nonfatal infarction) (4).
Moreover, the use of coronary revascularization as an end point is suspect because of its subjective nature, as well as any influence calcium scores exert on post-test management. In the Arad study, only symptom-driven revascularization was considered. Despite this, incremental models are preferred, including an initial prognostic model estimating "hard" events (i.e., death and/or myocardial infarction), followed by models assessing combined events. This would also foster comparisons with other published predictive models of death or infarction. There should also be a disjointing between revascularization occurring early (i.e., within 90 days) and late after testing. Late revascularization reflects clinical worsening and is acceptable as a clinical outcome (10).
Although there may be some association between coronary calcium and many clinical outcomes, its major strength may be in predictive models that estimate the presence or extent of the underlying atherosclerotic disease burden. It is likely that the optimal outcome model using coronary calcium, as a surrogate for total plaque burden, may be the development of symptomatic coronary heart disease (excluding unstable anginal symptoms). These models, as yet untested, mirror the development of risk factor equations from the Framingham Heart Study (11–13).
In the Arad report, coronary calcium scores >80, >160 and >600 were independent predictors of combined events, with the adjusted odds ratio ranging from 14.3- to 20.2-fold higher (4). Comparison of the EBCT odds ratio with other published odds ratios (e.g., cholesterol levels) should be viewed with extreme caution, because variation is driven by sample size, outcome frequency and modeling strategies. For patients with an elevated calcium score, the odds ratio is defined as some X-fold increase in event risk over that of patients with a lower calcium score. However, this risk ratio is not independent of the underlying risk in the patient group. In an exceedingly low risk group, such as the Arad series, the odds ratio for a high calcium score may be elevated 14.3-fold, with the actual event rate remaining low risk at 0.8%; as such, odds ratios require insight into the observed event rate for interpretation. The current report lacks much of the prognostic detail that would provide enhanced insight into the value of EBCT.
Epidemiologic principles garnered over the last several decades on estimating prognosis may guide the evaluation of EBCT as a tool for risk stratifying groups (10,14). Historic, physical examination and laboratory variables form the basis for initial risk assessment and additional test selection (11). Previous research on population screening favors use of measured risk factors (2,3,11). In a diagnostic testing environment (e.g., exercise electrocardiography), a physician’s interview often supplants direct risk factor measurement. Generally, a physician’s interview would be biased toward underestimating the value of historic markers (in lieu of supplemental imaging data). Variation in total information content for risk prediction is expected between directly measured and historic risk factors (4,7). If EBCT is to be used as a screening tool for primary prevention, then its added value must exceed information obtained from low cost laboratory variables such as blood glucose or cholesterol levels. Recently, Taylor et al. (15) suggested that coronary calcium may provide complementary information in relation to Framingham risk profiles. If this is so, then quantifying the additive relation is necessary to focus clinical decision-making including EBCT.
The current series includes asymptomatic subjects who were self- or physician-referred for screening with EBCT (4). The challenges with screening asymptomatic individuals are that the life-time risk of CAD is appreciable and, despite having low event rates,  50% of all sudden cardiac deaths and acute myocardial infarctions occur in asymptomatic people (16). Provocative testing has demonstrated limitations when applied to screening of asymptomatic cohorts (17). Previous reports, such as the Lipid Research Clinic prevention trial (18), noted that a positive test was not associated with nonfatal myocardial infarction or that, such as in the Seattle Heart Watch study, an abnormal ST segment response during exercise was not associated with an increased risk of cardiac events (19). Limitations to testing asymptomatic subjects relate to a lower disease prevalence and apply to any diagnostic modality, including EBCT. The application of Bayes’ theorem may provide us with insight into the precision of risk assessment in low risk groups. Bayes’ theorem states that the probability of a positive test being truly positive (i.e., underlying disease is present) relates to the prevalence of disease in the screened population (17). When clinical risk is low, an abnormal test will not result in a substantial reclassification in risk, due to a minimal shift in post-test probabilities. These results have direct relevance to EBCT data derived from low risk cohorts where high misclassification rates (i.e., false negative/positive tests) are expected.
Recent clinical guidelines have identified risk cut-points that may guide the interpretation of the data of Arad et al. (4,20). The annualized cardiac death or myocardial infarction rate from their study was 0.4%—a decidedly low risk cohort (defined as annual risk <1%) (4,20). Annualized event rates were substantially higher in the report by Detrano et al. (7). A continuum of risk is expected through the spectrum of disease states, ranging from asymptomatic without and then with risk factors, and finally to even higher risk symptomatic outpatient groups (5). Variations in the predictive value of EBCT would be expected in groups with varying underlying risk profiles.
Risk-adjusted odds ratios for calcium scores  160 were elevated 19.7-fold. It is expected that this threshold would be associated with death or infarction rates of 4%—not unlike multivessel perfusion abnormalities or left ventricular dysfunction as demonstrated by other cardiac imaging procedures (4,20). Aggressive management of these high risk patients would be warranted (20). However, from the Arad report, the 75th percentile calcium score was 97—approaching their threshold score of 160 (4). The challenge of testing low risk groups is that, based on these data, 25% of the group (n = 293) would be identified as EBCT high risk (4). Thus, treatment would be recommended for a sizable proportion of individuals, with only one of five people receiving therapeutic benefit from the intervention. A further challenge to this cut-point is that the average score of patients without events was 135, with a wide standard deviation of 432. The goal of risk stratification should be to limit aggressive and costly care to a select group of high risk patients (5,6). From this series, the total cost to identify one event using EBCT would exceed $3,200 (21).
|
Evaluating current evidence: statistical and epidemiologic issues
|
|---|
Major challenges to the interpretation of EBCT data include a number of methodologic problems, including small sample size, failure to use appropriate survival analysis techniques, lack of standard covariate adjustment and other variations in data collection methods (10,12,14). Considering the recent introduction of EBCT, one would not expect a large amount of effectiveness data to be available. Standards for new technology include the development of published evidence where the estimation of patient outcomes must be established in sufficiently large patient samples, with the data rigorously collected and analyzed from a diverse array of patient subsets.
Simple risk assessment statistics include the positive (i.e., total number of events/number of patients with abnormal test results) and negative (i.e., total number of nonevents/number of patients with normal test results) predictive value of a risk marker. Standard measures to test diagnostic accuracy are sensitivity (defined as true positive test results/number of patients with disease) and specificity (defined as true negative test results/number of patients without disease). In the current report, the use of sensitivity and specificity measures varies from previous risk stratification reports (12). Despite this, the best predictive accuracy achieved in the receiver operating characteristics (ROC) curve of myocardial infarction or coronary death was achieved at a square root calcium score of 15, with sensitivity and specificity of 80% (4).
For prognostic analysis, standard assessments of risk include the use of a Cox proportional hazards model. As events occur over time, this analytic method calculates time-related variation in the occurrence of events based on a given predictive model. Two types of models are commonly used: unadjusted and adjusted equations. For an unadjusted model, EBCT coronary calcium scores would provide enhanced or equivalent outcome estimation as compared with cardiac risk factors or laboratory variables. A critical step in the assessment of prognosis is the use of risk-adjusted outcome models, where the prognostic significance of EBCT is assessed while controlling or adjusting for pretest information (including clinical history, physical examination and laboratory markers). The amount of added information could then be statistically quantified in relation to the estimation of cardiac events. This incremental value may be used to justify the added cost of this procedure. A 10% to 20% added value of stress testing (with and without imaging) in symptomatic patients for risk assessment purposes was reported (5,6). In the current report from the St. Francis Medical Center, unadjusted and adjusted logistic regression analyses were used (4).
The authors further explore the accuracy of EBCT coronary calcium scores using ROC curves (4). Traditionally, ROC curves are used for diagnostic models, whereas in the Cox model, a concordance (C-) index is used that reflects the ability of a model to correctly classify patients with and without events (10,14). A C-index near chance would be at 0.5, and perfect estimation would be 1.0. The ROC curves areas exceeded 0.80 in the Arad series (4). From the current ROC curves, the square root coronary calcium score proved to be a highly specific tool for the estimation of myocardial infarction or cardiac death, at a substantial loss in sensitivity, reflecting a high false negative rate (4). With false negative rates being a surrogate for cost waste, the induced downstream medical resource consumption from EBCT would be excessive. Optimization of ROC curves may be accomplished by partitioning the curve to optimize sensitivity and specificity measures. For EBCT, cost waste infers an inability to identify a truly low and high risk subset of the population. A more balanced predictive accuracy may be achieved in larger samples than that of the Arad series (n = 1,172). Furthermore, given the current sample, there is insufficient statistical power to detect outcome differences in this low risk group. Given a 1% difference in event rates, sample size estimates could approach 5,000 subjects to detect differences by calcium score subsets.
|
Building effectiveness data in a cost-conscious environment
|
|---|
One reason for the lack of support for the use of EBCT has been related to a new standard for medical evidence that is increasingly applied in a cost-conscious health care environment. Added value of a test to the clinician and consumer is determined by the amount of new prognostic or diagnostic data that are derived from each comparative test in relation to its production cost, as well as any induced costs. Historically, new medical technology is associated with an increase in medical service use and added resource intensity. Increasingly, health care payers have introduced new standards beyond initial efficacy data that provide clinical effectiveness data for patient management. Data by Arad et al. (4) provide us with insight into the current prognostic data that are available with EBCT, but call for the development of larger series established in heterogeneous patient subsets. In addition to the development of a large registry by the Society for Atherosclerotic Imaging, the National Institutes of Health is embarking on a multi-institution study (Multiethnic Study of Atherosclerosis) evaluating the ability of EBCT and other methods (e.g., ankle-brachial index and magnetic resonance imaging) to identify high risk subsets.
For low risk groups, many have argued that the cost to identify so few events far exceeds society’s willingness to pay, and, perhaps, our limited resources may be better allocated to areas of greater need. Evidence suggests that treating low risk patients is not cost effective as compared with treating higher risk groups (i.e., high risk cost-effectiveness model) (22). Screening for CAD in asymptomatic individuals is difficult owing to the infrequent event incidence and to the exaggerated costs of identifying rare outcomes. Using routine screening, Pilote et al. (23) recently explored the clinical yield of exercise testing in 4,334 asymptomatic subjects. The rate of positive tests was 15%, whereas the angiographic CAD rate was 0.4%. As a consequence, screening costs were excessive in relation to the amount of CAD detected and resulted in additional unnecessary testing (24).
Despite the interest on the part of the cardiology community for new diagnostic tools to detect preclinical CAD, current data are inadequate to support the use of EBCT. As a risk assessment test, EBCT is more costly than other types of testing, while currently providing less abundant evidence to justify its use. Because this test is being actively marketed to and used by the American public, the current weight of evidence suggests that it should be performed in selected groups, with the result being a more judicious use of this procedure. Moreover, a more rigorous approach to the evaluation of EBCT would require a randomized trial comparing this technology with comparative modalities for a definitive evaluation. As direct marketing to consumers has become a mainstay in our society, the medical community must increasingly set standards in the best interest of our patients and society, advocating clinically effective and cost-effective use of any medical procedure, including EBCT. We await a greater compendium of data to provide support for the additive prognostic value of EBCT in asymptomatic individuals.
|
Footnotes
|
|---|
* Editorials published in the Journal of the American College of Cardiology reflect the views of the authors and do not necessarily represent the views of JACC or the American College of Cardiology. 
|
References
|
|---|
1. Rumberger JA, Simons DB, Fitzpatrick LA, Sheedy PF, Schwartz RS. Coronary artery calcium area by electron-beam computed tomography and coronary atherosclerotic plaque area: a histopathologic correlative study. Circulation. 1995;92:2157–2162[Abstract/Free Full Text]
2. O’Rourke RA, Brundage BH, Greenland P, et al. ACC/AHA expert consensus document on computed tomography for the diagnosis of coronary artery disease. J Am Coll Cardiol. 2000;36:326–340[Free Full Text]
3. Smith SC Jr, Greenland P, Grundy SM. American Heart Association Conference Proceedings: Prevention Conference V: Beyond secondary prevention: identifying the high-risk patient for primary prevention: executive summary. Circulation. 2000;101:111–116[Free Full Text]
4. Arad Y, Spadaro LA, Goodman K, Newstein D, Guerci AD. Prediction of coronary events with electron beam computed tomography. J Am Coll Cardiol. 2000;36:1253–1260[Abstract/Free Full Text]
5. Shaw LJ, Berman DS, Hachamovitch R, et al. Noninvasive strategies for the estimation of cardiac risk. Am J Cardiol. 2000;86:1–2[Medline]
6. Alexander KP, Shaw LJ, Shaw LK, DeLong ER, Mark DB, Peterson ED. Diagnostic and prognostic value of the Duke treadmill score in women. J Am Coll Cardiol. 1998;32:1657–1664[Abstract/Free Full Text]
7. Detrano R, Wong ND, Doherty T, et al. Coronary calcium does not accurately predict near term future coronary events in high risk adults. Circulation. 1999;99:2633–2638[Abstract/Free Full Text]
8. Sangiorgi G, Rumberger JA, Severson A, et al. Arterial calcification and not lumen stenoses is highly correlated with atherosclerotic plaque burden in humans. J Am Coll Cardiol. 1998;31:126–133[Abstract/Free Full Text]
9. Farb A, Burke AP, Tang AL, et al. Coronary plaque erosion without rupture into a lipid core: a frequent cause of coronary thrombosis in sudden cardiac death. Circulation. 1996;93:1354–1363[Abstract/Free Full Text]
10. Shaw LJ, Eisenstein EL, Hachamovitch R, Heller GV, Miller DD. A primer of biostatistic and economic methods for diagnostic and prognostic modeling in nuclear cardiology: part II. J Nucl Cardiol. 1997;4(pt. 1):52–60[CrossRef][Medline]
11. Grundy SM, Pasternak R, Greenland P, Smith S Jr, Fuster V. Assessment of cardiovascular risk by use of multiple-risk-factor assessment equations. Circulation. 1999;100:1481–1492[Free Full Text]
12. Califf RM, Armstrong PW, Carver JR, D’Agostino RB, Strauss WE. Task Force 5: Stratification of patients into high, medium and low risk subgroups for purposes of risk factor management. J Am Coll Cardiol. 1996;27:1007–1019[CrossRef][Medline]
13. Anderson KM, Wilson PWF, Odell PM, Kannel WB. An updated coronary risk profile: a statement for health professionals. Circulation. 1991;83:356–362[Free Full Text]
14. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387[CrossRef][Medline]
15. Taylor AJ, Burke AP, O’Malley PG, et al. A comparison of the Framingham risk index, coronary artery calcification, and culprit plaque morphology in sudden cardiac death. Circulation. 2000;101:1243–1248[Abstract/Free Full Text]
16. Kannel WB, Feinleib M. Natural history of angina pectoris in the Framingham study: prognosis and survival. Am J Cardiol. 1972;29:154–163[CrossRef][Medline]
17. Diamond GA, Forrester JS. Analysis of probability as an aid in the clinical diagnosis or coronary artery disease. N Engl J Med. 1979;300:1350–1358[Abstract]
18. Ekelund LG, Suchindran CM, McMahon RP, et al. Coronary heart disease morbidity and mortality in hypercholesterolemic men predicted from an exercise test. J Am Coll Cardiol. 1989;14:556–563[Abstract]
19. Bruce RA, Fisher LD. Exercise-enhanced assessment of risk factors for coronary heart disease in healthy men. J Electrocardiogr. 1987;20:162
20. Gibbons RJ, Chatterjee K, Daley J, et al. ACC/AHA/ACP/ASIM guidelines for the management of patients with chronic stable angina. J Am Coll Cardiol. 1999;73:2182–2197
21. Rumberger JA, Behrenbeck T, Breen JF, Sheedy PF. Coronary calcification by electron beam computed tomography and obstructive coronary artery disease: a model for cost and effectiveness of diagnosis as compared with conventional cardiac testing methods. J Am Coll Cardiol. 1999;33:453–462[Abstract/Free Full Text]
22. Goldman L, Weinstein MC, Goldman PA, Williams LW. Cost-effectiveness of HMG-CoA reductase inhibition for primary and secondary prevention of coronary heart disease. JAMA. 1991;265:1145–1151[Abstract/Free Full Text]
23. Pilote L, Pashkow F, Thomas J, et al. Clinical yield and cost of exercise treadmill testing to screen for coronary artery disease in asymptomatic adults. Am J Cardiol. 1998;81:219–224[CrossRef][Medline]
24. Taylor AJ, O’Malley PG. Self-referral of patients for electron-beam computed tomography to screen for coronary artery disease. N Engl J Med. 1998;339:2018–2020[Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
A. J. Taylor, J. Bindeman, I. Feuerstein, F. Cao, M. Brazaitis, and P. G. O'Malley
Coronary Calcium Independently Predicts Incident Premature Coronary Heart Disease Over Measured Cardiovascular Risk Factors: Mean Three-Year Outcomes in the Prospective Army Coronary Calcium (PACC) Project
J. Am. Coll. Cardiol.,
September 6, 2005;
46(5):
807 - 814.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
T. M. Doherty, L. A. Fitzpatrick, D. Inoue, J.-H. Qiao, M. C. Fishbein, R. C. Detrano, P. K. Shah, and T. B. Rajavashisth
Molecular, Endocrine, and Genetic Mechanisms of Arterial Calcification
Endocr. Rev.,
August 1, 2004;
25(4):
629 - 672.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
L. J. Shaw, P. Raggi, E. Schisterman, D. S. Berman, and T. Q. Callister
Prognostic Value of Cardiac Risk Factors and Coronary Artery Calcium Screening for All-Cause Mortality
Radiology,
September 1, 2003;
228(3):
826 - 833.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
T. O. Cheng
Cost-effectiveness of Electron-Beam Computed Tomography in the Diagnosis of Coronary Artery Disease
Arch Intern Med,
November 26, 2001;
161(21):
2624 - 2624.
[Full Text]
[PDF]
|
|
|
|
|
|
|
A. Schmermund and R. Erbel
Unstable Coronary Plaque and Its Relation to Coronary Calcium
Circulation,
October 2, 2001;
104(14):
1682 - 1687.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
A. D. Guerci, Y. Arad, L. J. Shaw, R. A. O'Rourke, B. H. Brundage, V. F. Froelicher, P. Greenland, S. M. Grundy, R. Hachamovitch, G. M. Pohost, et al.
Electron Beam Computed Tomography for the Diagnosis and Prognosis of Coronary Artery Disease Response
Circulation,
April 24, 2001;
103
(16):
e87 - e87.
[Full Text]
[PDF]
|
|
|
|
Top
Challenges of event prediction...
Evaluating current evidence:...