Alice K. Jacobs, MD, FACC, FAHA, Immediate Past Chair

Jeffrey L. Anderson, MD, FACC, FAHA, Chair

Jonathan L. Halperin, MD, FACC, FAHA, Chair-Elect

Nancy M. Albert, PhD, CCNS, CCRN, FAHA

Mark A. Creager, MD, FACC, FAHA

David L. DeMets, PhD

Steven M. Ettinger, MD, FACC

Robert A. Guyton, MD, FACC

Judith S. Hochman, MD, FACC, FAHA

Frederick G. Kushner, MD, FACC, FAHA

E. Magnus Ohman, MD, FACC

William G. Stevenson, MD, FACC, FAHA

Clyde W. Yancy, MD, FACC, FAHA

• Executive Summary......214

• Full Report......218

• Introduction......218

• 1Workgroup 1: Clinical Practice Guidelines We Can Trust......219
  • 1.1Workgroup 1 Questions for Consideration......220
  • 1.2Workgroup 1 Recommendations......223
• 2Workgroup 2: Standards for Initiating a Systematic Review......223
  • 2.1Workgroup 2 Questions for Consideration......224
  • 2.2Workgroup 2 Recommendations......226
• 3Workgroup 3: Standards for Finding and Assessing Individual Studies......226
  • 3.1Workgroup 3 Recommendations......227
• 4Workgroup 4: Standards for Synthesizing the Body of Evidence......229
  • 4.1Workgroup 4 Questions for Consideration......229
  • 4.2Workgroup 4 Recommendations for Assessing and Addressing the Quality of a Body of Evidence......231
  • 4.3Workgroup 4 Recommendations for Standards for Quantitative Analysis Across a Body of Evidence......232
  • 4.4Workgroup 4 Recommendations Regarding ACCF/AHA Grading Methodology and Nomenclature......236
  • 4.5Workgroup 4 Additional Recommendations......238
• 5Workgroup 5: Standards for Reporting Systematic Reviews......238
  • 5.1Workgroup 5 Recommendations......241

• Appendix 1. Author Listing of Comprehensive Relationships With Industry and Others......243

• Appendix 2. Workgroup Comparison Tables......250

  • Workgroup 1 Comparison Table......250

  • Workgroup 2 Comparison Table......253

  • Workgroup 3 Comparison Table......256

  • Workgroup 4 Comparison Table......261

  • Workgroup 5 Comparison Table......263

• References......242

The American College of Cardiology Foundation (ACCF) and the American Heart Association (AHA) have jointly engaged in the development of clinical practice guidelines (CPGs) for nearly 3 decades, based on the shared vision of their responsibility to provide guidance to cardiovascular healthcare professionals and the patients they serve by translating the best available evidence into clinical practice.

The ACCF/AHA Task Force on Practice Guidelines (Task Force) oversees and directs the CPG development process and methodology that have been the foundation of our documents and responsible for their widespread recognition. In brief, once a topic is selected for a new, revised, or updated CPG, selected organizations and professional societies with similar interests and expertise are invited to participate as partners or collaborators, with the overall ACCF/AHA policy of being inclusive and collaborative. Next, a guideline writing committee (GWC) chair is selected by the Task Force, and together with the chair, potential GWC members are identified, based on a detailed and specific relationship with industry and other entities (RWI) policy stating that the chair and the majority of GWC members must have no relevant RWI. Once formed, the GWC outlines the content of the document, performs a detailed and specific evidence acquisition and review, drafts recommendations with limited preliminary text, and attends a consensus conference with GWC members of related CPGs where recommendations are vetted and reconciled across existing guidelines. After the conference, the document is finalized. Every recommendation is voted on by each GWC member with appropriate recusal based on relevant RWI. The finished document then undergoes extensive peer review and response, and any recommendation that is changed is voted on again by each GWC member. The final document is reviewed, potentially revised and approved by the Task Force, and sent to the ACCF Board of Trustees, the AHA Science Advisory and Coordinating Committee, and partnering or collaborating organizations for final approval. After this very thorough and robust process, the CPG has essentially been vetted throughout the academic and clinical community and is jointly published in the Journal of the American College of Cardiology and Circulation. Once published, it serves as official policy of both organizations, informing strategic initiatives, advocacy, programs, products, and services.

Of critical importance is the continual evolution of the development process and methodology that has characterized the ACCF/AHA CPGs and the ongoing challenge to timely respond to and integrate the continuous stream of new knowledge.

Commissioned by the U.S. Congress as part of the Medicare Improvement for Patients and Providers Act of 2008 to set standards for CPGs, the Institute of Medicine (IOM) issued 2 reports: “Clinical Practice Guidelines We Can Trust” (1) and “Finding What Works in Health Care: Standards for Systematic Reviews” (2) in March 2011. The IOM redefined CPGs as follows: “Clinical practice guidelines are statements that include recommendations intended to optimize patient care that are informed by a systematic review of the evidence and an assessment of the benefits and harms of alternative care options.” (1, p. 29)

The underpinnings of this definition are that a CPG should be based on a systematic review (SR) of existing evidence; be developed by a knowledgeable, multidisciplinary panel of experts and key stakeholders; consider important patient subgroups and preferences as appropriate; be based on a transparent process that minimizes conflicts of interest and biases; provide a clear explanation of the logical relationships between alternative care options and health outcomes; provide the ratings of both the strength of the recommendation and the quality of evidence; and be revised as appropriate based on new evidence ((1),2).

In view of IOM reports and the ACCF/AHA ongoing improvement processes, the Task Force commissioned 5 Workgroups to address the IOM recommendations and present their findings at a Methodology Summit.

Each Workgroup was charged with reviewing each IOM standard and supporting elements with respect to the Workgroup topic; reviewing the current ACCF/AHA CPG methodology that addresses the standard; understanding the gaps and potential barriers between the proposed standard and the current method; and proposing recommendations for the Task Force to incorporate and/or to maintain or change existing methodology.

The 5 Workgroups were commissioned in August 2011 and worked by teleconference to draft the preliminary report. Each Workgroup focused on a series of questions to address the specific standards proposed by the IOM. The IOM reports were vetted by using a table (see Appendix 2) that presented each of the proposed 21 standards and 82 supporting elements, the existing applicable ACCF/AHA CPG methodology, and the Workgroup's proposal related to the standard. Each Workgroup report consisted of a brief introduction with an overview of the section content and questions, the complete table, a discussion of the salient issues and challenges that the Workgroup addressed, and preliminary recommendations.

The Workgroups' reports were distributed before the Summit, which was held over 2 days in mid-December 2011. Each report was discussed by the conference attendees and revised by the Workgroups with revisions and final recommendations presented again to the full group. The near-final draft report of each Workgroup was completed by the close of the Summit.

After the Summit, the reports were finalized and then edited for consistency. The completed “ACCF/AHA Clinical Practice Guideline Methodology Summit Report” was reviewed by the Task Force at its semiannual meeting in June 2012. Priorities, resources, implementation, and operational issues were discussed. After this meeting, additional minor edits were made, and the Report was reviewed and accepted by the leadership of both the ACCF and the AHA in September 2012.

Fortified by the AHA's focus on the patient and the public it serves and its lay membership, and by the ACCF's current initiative on patient-centered care, the Task Force will invite patient representatives (defined as patients and former patients, patient advocates, or patient/consumer organization representatives) to participate as a member of the Task Force and a GWC. Methods to identify, select, train, and manage RWI and intellectual perspectives of patient representatives are proposed. Patient representatives will participate in topic selection, patient choices, values, preferences, and shared decision making.

The current CPG creation process will expand to include a separate evidence review committee (ERC), tasked with the SR process, in addition to a GWC, tasked with creating the scientific CPG. When appropriate, other stakeholders (e.g., policy makers and payers) will be invited to participate as members of the ERC. The ERC will be responsible for all phases of the SR process, including the identification, abstraction, and quality assessment of the evidence base.

The ACCF/AHA methodology will incorporate a formal SR of the evidence, initially, with a focused approach to a confined topic. Standardized protocols may serve to streamline and enhance the process by developing topic-specific questions that guide the direction of the SR (e.g., search for studies, data extraction, synthesis, and presentation of findings). The use of a PICO(TS) format (mnemonic: population, intervention, comparator, outcomes, timing, and setting) will be used to develop evidence question(s) for the CPG SR.

The current ACCF/AHA CPG process includes a detailed and specific RWI policy and overall mandate to ensure balance (race, ethnicity, sex, and intellectual expertise and experience) among the GWC. In addition, the concept of intellectual and practice perspective (the latter term is operative when an individual's income is enhanced by performing a specific test or procedure relevant to the guideline topic) will be defined, recognized, and managed. Similar to the ACCF/AHA choice of the term RWI rather than COI (conflict of interest) related to industry, the ACCF and AHA have chosen the terms intellectual perspective and clinical practice perspective (rather than bias) to denote intellectual and practice-related opinions and expertise based on evidence and/or experience.

An expanded group of external reviewers will be added to the extensive peer review process for the completed CPG before publication. The current review process includes scientific and clinical content experts as well as partnering and collaborating organizations and other related professional societies. External reviewers will comprise a full spectrum of relevant stakeholders, including public representatives and constituencies such as governmental agencies (e.g., the Agency for Healthcare Research and Quality [AHRQ] and the U.S. Food and Drug Administration [FDA]). Moreover, the SR protocol and completed evidence review will be opened to public comment. However, because of the ability to introduce potential interference and bias that cannot be adequately controlled or managed, the completed CPG will not be subject to public comment before publication.

• 1At least 1 patient representative should be a full voting member of the Task Force and of a GWC. Patient representatives could include patients, former patients, members of patients' families, caregivers, and laypeople with “health literacy,” including scientists, statisticians, engineers, and science writers.
• 2All nonprofessional members of a GWC should avail themselves of the U.S. Cochrane Center and Consumers United for Evidence-Based Medicine's “Understanding Evidence-Based Healthcare: A Foundation for Action” or an equivalent online learning module before accepting their position.
• 3Patient representatives' responsibilities should include the formulation of key clinical questions; topic selection; patient choices, values, and preferences; and issues surrounding quality of life. Patient representatives should be encouraged to provide input on the selection of diagnostic tests and treatment modalities.
• 4CPGs should be provided to an expanded group of external reviewers before publication. External reviewers would comprise a full spectrum of relevant stakeholders (in addition to the current group of scientific and clinical content experts, as well as to partnering, collaborating, and other relevant professional societies), such as healthcare specialty societies, agencies (e.g., federal government), and representatives of the public.
• 5The ACCF/AHA CPG development policy should be expanded to incorporate intellectual and clinical practice perspectives. This information is discussed by the Task Force during GWC formulation, and the expertise of the members is noted in the guideline section on writing committee composition. This review and selection process should be made transparent to the GWC during their orientation meeting/teleconference, and GWC members should be asked to verbally update the group if any changes occur. For specific CPGs, consideration should be given to capturing clinical practice perspectives on the author RWI disclosure table if the GWC member's income is enhanced by performing a specific test or procedure that is relevant to the CPG topic.

• 1An ERC composed of content experts, methodologists, statisticians, and other identified stakeholders (policy makers and payers) should participate in the creation of ACCF/AHA CPGs (initially, with a focused approach to a confined topic). The ERC will interact with members of the GWC; however, the responsibilities of the ERC will be separate, distinct, independent, and clearly delineated.
• 2In an effort to standardize the process by which ACCF/AHA CPGs are created, the PICO(TS) format should be applied to develop the SR questions. It will be the responsibility of the ERC to create the format of the SR.
• 3The current peer review process of ACCF/AHA CPGs should be expanded to include a formal peer assessment of the SR protocol. External stakeholders and patients' representatives should be included in this expanded process, which will serve to enhance the applicability of the document to real-world decision-making policies and clinical scenarios.

• 1The system and tool used for critical appraisal of each study, currently under development by the Task Force (ACCF/AHA Evidence Grading Tool), will be used in the pilot SR conducted in conjunction with CPGs. In this case, it is anticipated that the assessment of the quality of the study will be focused within the context of a PICO(TS) question.
• 2The pilot system for critical appraisal of each study will be composed of a quantitative assessment, a yes/no assessment of key features of the studies, and a scoring of each of the 3 domains identified by the IOM: risk of bias, relevance, and fidelity of implementation.
• 3The quantitative assessment will need some revision for application to observational studies and for focused assessment of the 3 domains identified by the IOM.
• 4A scoring of the quality of the study in each of the 3 domains will be qualitatively accomplished by a judgmental scoring by the members of the ERC. The domain, risk of bias, should be renamed “freedom from bias” (or “internal validity”) so that the judgment system is directionally similar for all 3 domains. For each of the 3 domains, which are 1) freedom from bias, 2) relevance, and 3) fidelity of implementation, one of following judgments will be assigned: fatally flawed, low (low freedom from bias, low relevance, or low fidelity), intermediate, or high. In addition, there will be a summary assessment of each study: fatally flawed study, low-quality study, intermediate-quality study, and high-quality study.
• 5For other studies being considered in other parts of the CPG, a similar evaluation of the quality of the individual studies may be done. It will be necessary, however, for each proposed recommendation, to clearly identify the population under consideration, intervention, comparator, outcome, and setting, at least in a broad and general way.

• 1Standardize the method of assessment and description of the quality of the body of evidence across studies and perform a qualitative assessment of individual studies and aggregated studies.
• 2Depict qualitative assessment across studies addressing the key elements for each PICO(TS) question in a summary table.
• 3Where appropriate, provide a risk-of-bias assessment table across studies. This table would be similar to the Cochrane format, which includes items such as random sequence generator, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selection report, or other bias.
• 4Generate standardized summary and evidence table templates specific to ACCF/AHA requirements.
• 5When available, high-quality SRs from reputable organizations (e.g., the Cochrane Collaboration and AHRQ evidence-based practice centers [EPCs]) should be used. If a new SR is needed, resources will need to be secured to pursue de novo analyses.
• 6Notwithstanding these efforts to standardize and improve the qualitative analysis of bodies of evidence, every effort should be made to minimize delays in time for guideline development/revision.

• 1The GWC, in conjunction with the Task Force, should determine when a specific meta-analysis is needed.
• 2With statistical consultation, preferred, acceptable methods for meta-analysis (including a Bayesian analysis when appropriate) should be defined in each instance.
• 3When available, high-quality meta-analyses from reputable organizations (e.g., the Cochrane Collaboration and AHRQ EPCs) should be used. If a new meta-analysis is needed, resources should be secured to pursue de novo analyses.
• 4Notwithstanding efforts to standardize and improve the approach to quantitative analyses across studies, every effort should be made to minimize delays in time for guideline development/revision.

• 1Retain the current basic ACCF/AHA Class of Recommendation (COR) and Level of Evidence (LOE) structure/nomenclature.
• 2Standardize how LOE: A, B, C are determined by using a validated ACCF/AHA Evidence Grading Tool (under development) that will incorporate features of existing tools.
• 3Change the wording of LOE to Quality of Evidence (QOE) (once the ACCF/AHA Evidence Grading Tool is operational).
• 4Add a separate category for QOE: E (expert opinion) and generate specific definitions and examples for QOE: E.
• 5Change “Treatment Effect” on the COR/LOE table to “Intervention Effect” and indicate that “Intervention” includes medications, devices, therapeutic strategies, procedures, diagnostic tests, and other.
• 6Add adjectives that “map” COR/LOE to the National Heart, Lung, and Blood Institute (NHLBI) (and Grading of Recommendations Assessment, Development and Evaluation [GRADE]) (i.e., I [strong]; IIa [moderate]; IIb [weak]; III [against]).

• 1An SR included in CPGs should be published as a separate peer-reviewed manuscript(s) when feasible.
• 2Recommendations supported by an SR should be identified in CPGs in addition to the appropriate COR and LOE; for example: (Class I; LOE: A)^SR.
• 3Within the CPG, salient tables and figures from the SR should be included to support recommendations. The remaining pieces of methodology will be hyperlinked to the original SR publication. By doing so, key elements of the SR will be available for public access after publication of the CPGs.
• 4CPG review tables should be incorporated into CPGs for non–SR-based recommendations and be available as online supplemental tables.
• 5CPG review tables should incorporate most study components that are provided in SR tables.

It is critically important that we continue to monitor the impact of the proposed IOM standards and ACCF/AHA Summit Workgroup recommendations on the overall timeliness and usefulness of our CPGs, in addition to the inherent resource consumption required. Most important will be the ongoing assessment of the changes in methodology in relation to improved patient care and outcomes. Yet to be determined is whether the inclusion of patients' representatives in CPG development and performance of a formal SR of the evidence will enhance the translation of recommendations as anticipated. What is clear, however, is that these changes have the potential to move us into the future as we prepare to incorporate cost and value into our CPG recommendations and to incorporate these recommendations into electronic clinical support systems.

The overarching goal in the evolution of the process and methods used to develop ACCF/AHA CPGs is to combine an ever-expanding evidence base with multidisciplinary expertise and experience while being mindful of patient values so as to improve clinical decision making and thereby improve the quality of care and outcomes for patients with cardiovascular disease. The ACCF and AHA remain steadfast in their commitment to this endeavor.

Alice K. Jacobs, MD, FACC, FAHA

Chair, ACCF/AHA Clinical Practice Guideline Methodology Summit

• COR = Class of Recommendation

• CPG = clinical practice guideline

• EPC = evidence-based practice center

• ERC = evidence review committee

• GRADE = Grading of Recommendations Assessment, Development and Evaluation

• GWC = guideline writing committee

• IOM = Institute of Medicine

• LOE = Level of Evidence

• PICO(TS) format = (mnemonic: population, intervention, comparator, outcomes, timing, and setting)

• QOE = Quality of Evidence

• RCT = randomized controlled trial

• RWI = relationship with industry and other entities

• SR = systematic review

The American College of Cardiology Foundation (ACCF) and American Heart Association (AHA) have jointly developed and published clinical practice guidelines (CPGs) since the early 1980s, when, in response to a governmental concern over the potential overutilization of pacemakers, the ACCF and AHA were asked to evaluate the evidence and provide recommendations for practice. In 1984, the “ACCF/AHA Guidelines for Permanent Cardiac Pacemaker Implantation” was published (3). Since then, the guideline effort has continued to expand, and currently, the ACCF/AHA Task Force on Practice Guidelines (Task Force) oversees and directs 17 CPGs, the majority of which are broadly disease based. In 2011, 2 revised, 2 new, and 3 focused updates to the CPGs were published.

The CPG development process and methodology continue to evolve. Over the past few years, several process improvement initiatives resulted in CPGs that have limited text and include evidence and summary tables. The writing process includes a consensus conference where members of multiple guideline writing committees (GWCs) meet to reach consensus and concordance on overlapping recommendations. Methodological enhancements include the development and testing of a scoring tool to consistently assess the quality of randomized controlled trials (RCTs) that inform recommendations and a thorough analysis of Bayesian approaches to evidence synthesis. This continued evolution is in response to the primary goal of providing evidence-based guidelines for healthcare professionals practicing cardiovascular medicine while maintaining relevancy and ease of use at the point of care. The recommendations are articulated in the time-honored and widely recognized ACCF/AHA Class of Recommendation (COR) and Level of Evidence (LOE) scheme. Currently, we are working to incorporate patient preference and shared decision making into the development and translation of our CPG recommendations.

In March 2011, the Institute of Medicine (IOM) published 2 reports, “Clinical Practice Guidelines We Can Trust” (1) and “Finding What Works in Health Care: Standards for Systematic Reviews” (2). There are 8 standards for developing trustworthy CPGs that include establishing transparency, managing conflict of interest, creating multidisciplinary development groups, basing the CPG recommendations on systematic reviews (SRs), establishing evidence foundations for rating the strength of recommendations, articulating recommendations, establishing methods for external review, and updating. It is noteworthy that the ACCF/AHA CPGs were recognized as being compliant with the majority of these standards (11.1). However, in view of these detailed reports and our ongoing challenge to respond to the continuous stream of new evidence in a timely manner while maintaining our robust processes for CPG generation and approval, the Task Force decided to hold a Methodology Summit that would focus on the standards for and performance of systematic evidence reviews and on the inclusion of patients and consumers in the CPG development process.

The purpose of this Methodology Summit was to compare and contrast the current ACCF/AHA CPG methodology with the standards proposed by the IOM and consider what, if any, changes or improvements should be implemented to enhance our development process and evidence review and evaluation. In August 2011, the Task Force commissioned 5 Workgroups:

• 1Clinical Practice Guidelines We Can Trust
• 2Standards for Initiating a Systematic Review
• 3Standards for Finding and Assessing Individual Studies
• 4Standards for Synthesizing the Body of Evidence
• 5Standards for Reporting Systematic Reviews

The invited members of each Workgroup, in addition to guests at the Summit (and including all Task Force members), brought a diversity of experience and expertise to this initiative. Specifically, the membership included those with special interest and experience in CPG development independent of the ACCF/AHA in addition to those with extensive experience in development of ACCF/AHA CPGs. Participants also included methodologists, biostatisticians, clinical and research cardiologists, epidemiologists, and nurses.

Each Workgroup was charged with 1) reviewing the IOM recommendations and sections of the current tools available for developing/conducting SR relevant to its topic; 2) comparing and contrasting the recommendations with current ACCF/AHA methodology, including an analysis/discussion of the gaps and barriers; and 3) drafting recommendations/considerations for changes and improvements to the evidence review process and the COR/LOE, including a discussion as to why changes may or may not be implemented. The report of the Workgroups follows.

Alice K. Jacobs, MD, FACC, FAHA

Immediate Past Chair, ACCF/AHA Task Force on Practice Guidelines

See (11.1).

(Authors: Dr. Frederick G. Kushner, Chair; Drs. Ralph G. Brindis, Mark A. Creager, Ralph L. Sacco, William A. Zoghbi, and Mr. William H. Roach, Jr.)

The Task Force has recommendations for patient-centered care within its methodology manual (4). GWC are encouraged to consider the role of patient preferences in decisions with substantial personal choice or values and to consider patient-specific modifiers, comorbidities, and issues of patient preference. The IOM report on “Clinical Practice Guidelines We Can Trust” (Section 3.2 and Section 3.3) (1) recommends the involvement of patients (or former patients) and the public (patient advocates, patient/consumer organizations) in the CPG process to formulate clinical questions and review draft CPGs. The IOM report also recommends that strategies for training participants should be adopted by guideline developers. Consumer involvement may provide transparency and a means of establishing relevancy and credibility for the application of evidence-based medicine to patients and other stakeholders and thereby dispel myths and fears through the mutual understanding of issues and values for both patients and providers. Consumers can be powerful allies as supporters of quality initiatives. They can also provide an important perspective from the patient's point of view in areas of uncertainty, where alternative options exist, or where there are substantial gaps in evidence. Finally, consumer involvement can provide valuable perspectives for enhancing shared decision making. Currently, there is a paucity of evidence surrounding the process and impact of integrating consumers into the CPG process. For the purposes of this document, “consumers” will be referred to as patient representatives.

• 1How can the Task Force and GWC identify the appropriate “patient representative”? What are the criteria to be met for selection of “patient representatives” who serve? Will they be volunteers or contracted for their work?

  As defined by the Cochrane Consumer Network, consumers (“patient representatives” herein) include all users or receivers of health care, including patients, members of the public, caregivers, family members, and members of consumer advocacy groups. Several organizations have identified or are in the process of identifying individuals to serve on GWC. The 4 largest of these are the Cochrane Collaboration through Cochrane Consumer Network and a U.S. subsidiary, Consumers United for Evidence Based Medicine, Agency for Healthcare Research and Quality (AHRQ), Consumers Union, and Guidelines International Network. Consumers Union is currently developing a list of “qualified” patient representatives to serve on evidence-based medical guideline committees under the auspices of a grant from the American Institutes for Research/AHRQ. They solicited interested parties from their subscriber base of 300,000 and received 2,000 expressions of interest. The AHA has a long history of inviting lay volunteers to serve on its boards and committees, in mission and fundraising activities, and in advocacy, and regularly recommends consumer representatives to private and governmental panels and committees. The American Society of Clinical Oncology has included consumers on their GWCs. Patient representatives with defined constituencies can be particularly valuable because of their institutional involvement. On the basis of the comments from Consumers Union and the American Society of Clinical Oncology and discussions among its members, Workgroup 1 recommends the following principles concerning the identification of appropriate patient representative candidates:

  • AA job description, including desired and necessary attributes and expectations, should be available to the Task Force and GWC chairs and to organizations that may recommend candidates.
  • BThe Task Force and GWC should seek nominations for patient representatives from trusted organizations with knowledge of the individual, such as the AHA, ACCF, Consumers Union, Cochrane Collaboration, and AHRQ.
  • CPatient representative candidates and/or their sponsoring organizations should submit their resumes, curriculum vitae, or personal statements.
  • DPatient representatives should be advised of the time commitment and compensated for travel expenses identical to physician and other Task Force and GWC members.
  • EPatient representative candidates, similar to all members, should complete the relationships with industry and other entities (RWI) disclosure.
  • FThe Task Force and/or GWC chair or designee should interview potential consumer candidates and identify other conflicts where they exist.
  • GThe development of a questionnaire similar to those used by major charitable organizations to elicit possible bias should be considered.
  • HProspective members should be required to confirm that they will give precedence to ultimate patient health, well-being, quality care, and value in their input, deliberation, and voting.
  • IPatient representatives, as is true for other members of the GWC, must be sensitive to their role as impartial members and not permit financial and nonfinancial conflicts, including personal, intellectual, or organizational relationships to influence their judgment.
  • JPatient representatives on GWCs should be engaged in the formulation of key clinical questions, topic selection, patient choices, values, preferences, and issues surrounding quality of life. They should be encouraged to provide input on the selection of diagnostic tests and treatment modalities.
  • KPatient representatives should be acknowledged as members of the GWC at publication, and a complete summary of their RWI should be provided. They should be subject to the same RWI and recusal rules as other members.

• 2Specifically, how is the patient representative to be integrated into the CPG process and/or peer review and/or external stakeholder review? What expectations are there for patient representative involvement?

As a member of the Task Force, patient representatives would assist in directing and overseeing CPG development and establishing policy with a focus on issues such as shared decision making, patient preference, value, translation, and implementation. Patient representatives may be best suited to participate in the initial work of the GWC, during the formulation of key clinical questions, topic selection, and outline development, and, particularly, to comment on the translation and communication of CPGs to the general public. They may not be as helpful or comfortable with SR or assignment of COR and LOE to specific recommendations. Patient representatives with certain technical skills such as science journalists and writers may aid in a patient-provider communication and summary section. Patient representatives should be expected to contribute key questions to be answered by the GWC. They should provide input about patient choices, values, preferences, and issues surrounding quality of life in selecting diagnostic modalities, therapies, medications, and follow-up.

• 3What form of training will be required for patient representative participants?

To integrate patient representatives effectively into the CPG creation process, substantial preparatory training is needed before they begin their service on the Task Force or GWC. One study observed that without preparatory training, patient representatives feel that they are “participating observers of technical language to which they could hardly offer input.” Patient representatives contributed infrequently to the discussions, had difficulty with the technical language, only contributed during discussions of patient education, and in general felt that their contributions were not subsequently acted on (5). Involvement of patient representatives requires their understanding of the evidence. Difficulty with medical terminology or other jargon is an important barrier to active or meaningful involvement. Well-informed and experienced patient representatives are more likely to have meaningful exchanges with the GWC than those less informed or less familiar with medical terminology (6). The capacity for active participation in GWCs presupposes foundations of access, knowledge, information, understanding, confidence, agency, engagement, and advocacy (7).

Although resource intensive, it is feasible to train patient representatives to understand the technical elements of CPG development. It is not expected that these members of GWCs understand most of the science related to specific medical issues. They may also have limitations in their ability to understand the details of SR or health economics. Nonetheless, focused instruction in the CPG development process will allow them to fulfill their role. Training is required for understanding elements of recommendation classifications and LOE, including treatment risks and benefits, comparative efficacy, biostatistics, and clinical trial design (i.e., the value of a single center case report or retrospective observational data versus prospective blinded RCT along with meta-analysis).

The Guideline International Network Patient and Public Involvement Working Group has been created to support the development, implementation, and evaluation of guideline-oriented patient and public involvement programs. They have found that patient representative training should cover the fundamentals of CPG development and approaches for reporting back to patient constituencies. Their participants concluded that training and support may facilitate understanding of the technical aspects of CPG development, address financial and organizational barriers to participation, and enhance mutual understanding. Guideline International Network Patient and Public Involvement Working Group collaboration priorities include the development of recruitment methods, training and support strategies, information material and tools, and glossaries of technical terms used in CPGs (8).

The U.S. Cochrane Center and Consumers United for Evidence-Based Medicine have created a web-based course, “Understanding Evidence-based Healthcare: A Foundation for Action,” through a grant from AHRQ (http://us.cochrane.org/understanding-evidence-based-healthcare-foundation-action) (9). This course is divided into 6 modules: INTRO (what is evidence-based health care and why is it important), ASK (importance of research questions in evidence-based health care), ALIGN (research design, bias, and LOE), ACQUIRE (assessing harms and benefits), APPRAISE (understanding healthcare statistics), and APPLY (critical appraisal and making better decisions for evidence-based care; determining causality) (9). The Cochrane Collaboration also has created the Cochrane Consumer Network to engage patient representatives in the development of SR, raise awareness among patient representatives, that is, serve as a clearinghouse of patient representatives for advisory groups, commission plain language summaries, and recruit coauthors for reviews. New ideas to promote patient representative involvement, such as videos, workshops, learning materials, evaluations, and use of social networks have been recently implemented.

• 4Should the Task Force support the standard to have CPGs publicly reviewed? How will that affect the final product? How much of a burden will that put on volunteers?

The IOM has recommended that public agencies, patients, and representatives of the public should be external reviewers. The IOM has also recommended that before publication, a draft of the CPG should be made available to the general public for comment and reasonable notice of impending publication should be provided to interested public stakeholders. The IOM believes that for transparency, fairness, completeness, and credibility, these recommendations are reasonable.

Currently, while the ACCF/AHA CPGs undergo an extensive peer review process that includes scientific and clinical content experts in addition to partnering, collaborating, and other relevant professional societies, they are not open to public review and comment. Fundamental to the CPG development process is the ability to develop the CPG without bias from commercial interests. The Workgroup carefully considered the value provided by opening the draft CPG to public opinion. It was recognized that a period of open public comment would introduce a window for potential interference in the process by industry and other external stakeholders that cannot be adequately controlled or managed. However, review of the CPG by public representatives and stakeholders such as governmental agencies, for example, the AHRQ and the U.S. Food and Drug Administration (FDA), as well as scientific and professional experts, is reasonable and recommended.

• 5How will conflicts of interest (organizational, intellectual, and practice based) be adjudicated by the GWC and Task Force? What is an “intellectual conflict” or “intellectual bias”?

The ACCF/AHA CPGs are in substantial compliance with 7 of the 8 standards for developing trustworthy CPGs proposed by the IOM, (11.1); the CPG-SR intersection is the subject of the remainder of this report. However, although the ACCF and AHA have a rigorous policy for defining, disclosing, and managing RWI (4), the policy is currently undergoing evaluation and updating for potential conflicts related to intellectual bias and practice (or employment) bias.

Intellectual or clinical practice bias (the latter term is operative when an individual‘s income is enhanced by performing a certain test or procedure relevant to the guideline topic) may result in a conflict of interest. An intellectual or clinical practice bias is a predisposition of an individual based on a reaction to a past or current event, treatment, relationship, or other circumstance, or an opinion, belief, or position so strongly held that it might prevent the individual from exercising objective judgment about a matter relevant to the work of the GWC. In the infrequent scenario where, in the opinion of the Task Force or GWC chair, a true bias (with its negative connotation) exists, that individual will not be invited to participate as a member of the GWC. In contrast, and similar to the ACCF/AHA choice of the term RWI rather than COI related to industry, the ACCF and AHA have chosen the terms intellectual perspective and clinical practice perspective to denote intellectual and practice-related opinions and expertise based on evidence and/or experience. Both science and patient representative GWC members with intellectual perspective and clinical practice perspective must be identified and included and their perspective disclosed and managed. It is understood that they have an open-minded approach to evidence and opinion that distinguishes them from individuals with actual intellectual or practice bias.

Identification of Potential Intellectual or Clinical Practice Perspectives of Candidates for Appointment to a GWC

Unlike potential conflicts arising from RWI, determination of the existence of an intellectual or clinical practice perspective may be subjective and less readily apparent to the Task Force. The key to mitigate effectively any conflict is early identification and management of the conflict by the Task Force initially and then by the GWC chair once the work begins. The Task Force chair should be responsible for implementing applicable intellectual and clinical practice perspective procedures. Following are ways to identify and adjudicate the perspectives:

• 1Nominations to GWCs should be sought from trusted organizations with a deep knowledge of the individuals nominated. For example, both the ACCF and the AHA, as well as many of their collaborating organizations, have a thorough understanding of CPG development, and each has a long relationship with numerous consumer and scientific volunteers who would have the credentials to serve on a GWC. From that relationship, each organization has knowledge of individual volunteers' life experiences, professional work, and personal and professional views. It has been the practice of the Task Force to maintain a balance of members with and without RWI so that a minimum of 50% of the members have no relevant RWI. In addition, the expertise of the GWC members is reviewed carefully to ensure that there is a balance of perspectives. The ACCF and AHA could request that other organizations meet these same requirements. In addition, it is the practice of the ACCF and AHA to provide additional scrutiny of nominations by officers and senior staff before the Task Force review. Seeking nominations from similar organizations would reduce the likelihood of nominees with unknown perspectives.
• 2Any candidate for GWC membership may be requested to submit a detailed curriculum vitae, which the Task Force and the GWC chair and staff can review for possible intellectual or clinical practice perspectives (or bias).
• 3The GWC chair, at his or her discretion, should be empowered to interview proposed GWC members who are not well known or who have possible or unknown intellectual or clinical practice perspectives.
• 4Candidates for GWC membership may be asked to complete a questionnaire through which they disclose circumstances that might give rise to a potential or actual perspective or bias related to the GWC assignment. The questionnaire could include inquiries found in questionnaires typically used by charitable health organizations with respect to director and officer perspectives or bias. As some individuals may have difficulty recognizing or disclosing their own intellectual or practice perspectives that may lead to a conflict, questionnaire inquiries should be broadly worded to obtain information from which the GWC chair, Task Force, or staff could identify actual or potential intellectual or practice perspectives.
• 5At each meeting of the GWC, remind the members of the policy whereby the group was formulated to incorporate a balance of RWI and varying intellectual and clinical practice perspectives that we continue to manage throughout the guideline writing effort.
• 6Prospective GWC members should be educated about being transparent with respect to their individual position and be required to confirm that they will give priority to ultimate patient health and well-being, quality care, and value in their input and deliberation and when voting on issues before the GWC.

Inclusion of Essential GWC Members With Known Intellectual and Clinical Practice Perspectives

To develop the most effective CPGs, a GWC may require the participation of an individual who is outspoken in support of or against a particular procedure, medication, or other matter relevant to the GWC's work in the absence of supportive data and therefore has a known intellectual or clinical practice perspective. In such cases, the Task Force should make certain that the GWC chair is informed of the individual's perspective and that the GWC membership is balanced in viewpoint. All GWCs should be composed of members determined to be able to apply fair judgment with respect to all issues and particularly those in contention, so that they will be able to make objective assessments of all the information relevant to the GWC task. In addition, the GWC chair should make certain that the individual follows the policy and procedures noted in the ACCF/AHA Methodology Manual and Policies (4) as it applies to recusal from discussion at the request of the chair.

Importance of Intellectual, in Comparison to Financial Conflict and RWI, to External Organizations and Government Regulators

Although conflicts arising from RWI have received most of the attention by external groups, the Task Force should effectively address all potential conflicts, whether arising from RWI or intellectual or clinical practice perspective. The Task Force should be prepared to demonstrate to external groups that its policies for managing any conflict are practical and effective.

Maintaining a Majority of GWC Members Without RWI and a Balance of GWC Members With Intellectual and Clinical Practice Perspectives Throughout the Duration of CPG Development

All GWC members should be assessed by the Task Force and GWC chair for potential relevant RWI at the outset of the process. A majority of members who have no relevant RWI must be assigned. They will be asked to commit to developing no new RWI or other relationships that may represent potential intellectual perspectives (e.g., serve as principal investigator of a new study relevant to the CPG topic) during the course of their service on the GWC. As RWI or intellectual perspectives may arise in connection with any one of the many recommendations a GWC may make in developing a CPG, the GWC chair and staff must be vigilant in identifying any relationships as they arise and adjudicating them in accordance with established policies.

• 1At least 1 patient representative should be a full voting member of the Task Force and each GWC. Patient representatives could include patients, former patients, members of patients' families, caregivers, and laypeople with “health literacy,” including scientists, statisticians, engineers, and science writers.
• 2All nonprofessional members of GWCs should avail themselves of the U.S. Cochrane Center and Consumers United for Evidence-Based Medicine's “Understanding Evidence-Based Healthcare: A Foundation for Action” (9) or an equivalent online learning module before accepting their position.
• 3Patient representatives' responsibilities should include the formulation of key clinical questions, topic selection, patient choices, values, and preferences, and issues surrounding quality of life. They should be encouraged to provide input about the selection of diagnostic tests and treatment modalities.
• 4CPGs should be provided to an expanded group of external reviewers before publication. External reviewers would comprise a full spectrum of relevant stakeholders (in addition to the current group of scientific and clinical content experts, as well as to partnering, collaborating, and other relevant professional societies), such as healthcare specialty societies, agencies (e.g., federal government), and representatives of the public.
• 5The ACCF/AHA CPG development policy should be expanded to incorporate intellectual and clinical practice perspectives. This information is discussed by the Task Force during GWC formulation and the expertise of the members is noted in the guideline section on writing committee composition. This review and selection process will be made transparent to the GWC during their orientation meeting/teleconference, and GWC members will be asked to verbally update the group if any changes occur. For specific CPGs, consideration will be given to capturing clinical practice perspectives on the author RWI disclosure table if the GWC member's income is enhanced by performing a specific test or procedure that is relevant to the guideline topic.

See (11.2).

(Authors: Dr. Steven M. Ettinger, Chair; Drs. Donna K. Arnett, Gregg C. Fonarow, Judith S. Hochman, Sharon-Lise T. Normand, and Gordon F. Tomaselli)

SRs and the resulting evidence-based CPGs serve as resources for healthcare decision-making policies. The ideal guideline provides comprehensive protocols and plans that are based on a thorough and extensive understanding of the medical literature, are scientifically valid, and are void of clinical bias. The IOM defines 8 standards (11.2), which guide the creation of a focused SR policy, that are essential for the creation of a scientifically valid CPG.

Workgroup 2 was charged with the task of reviewing current ACCF/AHA standards for an SR and identifying those elements that diverged from the recommendations outlined by the IOM. Where applicable, the Workgroup proposed alternative strategies relating to SR in an effort to enhance current ACCF/AHA policies. The Workgroup acknowledged that one of the challenges posed by modifications of existing ACCF/AHA evidence review policies is related to the rapid expansion of medical literature seen with publication of late-breaking clinical trials and RCTs. The Workgroup expressed concern that any modification in current ACCF/AHA evidence review policy would result in significant delays in the creation of evidence-based CPGs and the loss of clinical relevance of the document. Recognizing that constraints relating to staffing and financial issues would also affect the SR process, the Workgroup proposed 3 modifications of the current ACCF/AHA evidence review policy. These modifications were developed based on a review of the following key questions:

• 1Should other stakeholders/patient representatives be involved in the development of the SR protocol? If yes, to what level and what types of stakeholders/consumers should be included (e.g., payers, policy makers, patients, caregivers, family, and advocacy groups)? What type of training would this group need to be able to provide value to the current process? Would the ACCF/AHA provide the training or outsource it?

In an effort to expand the current ACCF/AHA process related to the creation of CPGs (initially, with a focused approach to a confined topic), 2 separate teams would be created: an evidence review committee (ERC), tasked with the SR process, and a GWC, tasked with creating the scientific CPG. When appropriate, other stakeholders (e.g., policy makers and payers) would be invited to participate as members of the ERC. The ERC will be responsible for all phases of the SR process. By expanding the professional and clinical expertise of the ERC, the “trustworthiness” of the final document as well as its applicability to real-world decision making may be enhanced (2). The Workgroup acknowledges the added benefit of reviewing and incorporating unique data sets provided by the “new” stakeholders that are potentially outside the scope of current published scientific literature. Although “unpublished data may not be used to support a recommendation” (4), information collected by policy makers and payers (e.g., economic and health plan data) may provide additional insight into the effectiveness of healthcare recommendations as it relates to various subpopulations (e.g., racial and ethnic minorities, women, and the aged). Stakeholders invited to participate in the ERC would undergo formal training (by ACCF/AHA staff) related to understanding current CPG writing policies of the ACCF/AHA. In addition, SRs require a rigorous scientific analysis of case reports, case series, cohort studies, clinical trials, RCTs, and blinded RCTs. An understanding of various point statistics (e.g., absolute risk difference, number needed to treat, number needed to harm) and relative treatment effects (e.g., odds ratio, relative risk, hazard ratio, and incidence rate ratio) are essential components of the process. Individuals invited to serve on the ERC may require additional training related to methodology and statistical analysis (or possess this expertise) with the expectation that this would be made available by the ACCF/AHA.

• 2Should we redefine how RWI and intellectual and clinical practice perspectives are presently managed for those involved in the development of the SR as it relates to intellectual, professional, and personal association? Would this make the process better?

The ACCF/AHA has an official RWI policy (4). The definition of relevant is published in the first Appendix of each CPG along with the authors' relevant RWI. The ACCF/AHA does not yet have an official policy regarding intellectual and clinical practice perspectives (see Workgroup 1), although a balance of perspectives and opinions is considered by the Task Force when selecting a GWC, each author's institution and title are provided in the document, and a general statement about the expertise of the members are noted under writing committee composition.

The following assumes that GWC structure will not change, but when appropriate, a separate ERC will be impaneled to draft and formalize a SR and include methodologists and clinical content experts. Although the GWC and ERC are separate committees, some members may be selected to serve as liaison to both. As such, the ERC will have the responsibility of developing the SR for use by the GWC in generating the scientific clinical document. The ERC will then review the developed CPG to confirm that the recommendations are consistent with the findings of the clinical trials and studies.

In general, we agree and are compliant with the IOM recommendations on many aspects of RWI and intellectual balance when constituting an SR team. The principle that disclosure is necessary but may not be sufficient is important. We also agree that individuals should be excluded if their participation would diminish the public perception of the independence and integrity of the newly created CPG.

The discussion around disclosure of conflict of interest/RWI in the IOM report (Section 2.2.1) (2) included the question of the structure of the ERC. We recommend that the ERC include members completely free of RWI and balanced with respect to intellectual and clinical practice perspectives (as defined by Workgroup 1) but that the GWCs adhere to the current ACCF/AHA policy of the chairs and >50% of the committee being free of relevant RWI. It is the consensus of the Workgroup that absence of experts (perhaps with relevant RWI) on the GWC would undermine the credibility of the review and CPG as much or more than the presence of committee members with RWI.

As stated by Workgroup 1, individuals thought to have true bias would not be eligible to serve on a GWC. The major issue pertains to the definition of intellectual and clinical practice perspective and how this standard would be adjudicated. It is unclear whether any method or protocol for assessing intellectual and professional perspective can be standardized and codified to allow for use across all document-generating initiatives. We attempted to address the various metrics that could be used to define intellectual or professional perspective considering a number of possibilities, including the percentage of one's income or whether income is enhanced if derived from a procedure, test, or topic relevant to the CPG topic (this could also be considered under financial perspective); the publication and speaking record of the candidate committee member on the topic; and whether or not the home institution or practice would benefit in a way that might promote a particular perspective, among others (see Workgroup 1). However, regardless of the metric(s) used, we recommend consideration of expanding disclosure to include relevant professional/intellectual or clinical practice perspectives in the published document for selected guidelines.

We recognize that some of the changes outlined would require an increase in the size and broadening of the expertise of the groups constituted to develop CPGs. Perhaps more importantly, the timeline for the entire process most certainly will be lengthened. In this context, the IOM report commented that an SR may take 1 year or more to produce, so RWI and intellectual and practice perspectives should be updated at regular intervals, but there is no guidance as to what those intervals should be. Currently, the ACCF/AHA process includes the provision that GWC members avoid accepting new relevant RWI throughout the duration of the development process.

• 3Should the SR topics be formulated using the PICO(TS) questions, AHRQ, or an alternative source? Should we consider using a governmental agency, and how would this affect the process (cost, time, and other considerations)? How would we train physicians and/or stakeholders/patient representatives?

In an effort to confirm the need for a focused update (or new or revised CPG), summaries of late-breaking clinical trials presented at major scientific conferences are compiled. The goal is to identify evidence that is important to patient-care issues (including but not limited to benefits relating to quality of life, morbidity and mortality, and economic outcomes). Members of the GWC and Task Force are balloted as to whether the findings of the new study may change a current recommendation. The Task Force reviews the results and a decision is made as to whether a focused update (or new or revised guideline) GWC should be convened.

Once the decision is reached to create a new, revised, or updated CPG, every effort is made to ensure the development of an objective, transparent, and scientifically valid document. The ACCF and AHA currently use a detailed standard format to generate an evidence review for the creation of CPGs (4). The ACCF, AHA, and IOM acknowledge that creation of a CPG and formulation of specific SR questions relating to the topic are challenging processes. Standardized protocols may serve to streamline and enhance the process by developing topic-specific questions that guide the direction of the SR (e.g., search for studies, data extraction, synthesis, and presentation of findings [2]). The IOM supports an SR that focuses on the development of questions that deal with the “uncertainties that underlie disagreement in practice and the outcomes and interventions that are of interest to patients and clinicians” (2). A CPG with newly developed recommendations created in such a manner may find improved real-world application and expand its value to healthcare decision makers. The use of a PICO(TS) format (2) is recommended by the Workgroup to develop evidence question(s) for the SR of a CPG (initially, with a focused approach to a confined topic). This effort will require additional resources for the education and training of committee members. Collaboration with other guideline developers and government organizations should be explored.

• 4Should we alter the current process of peer review as it relates to the assessment of the SR protocol—electronic registration, ACCF/AHA Web sites—or continue the current process of an internal review by the GWC?

As stated previously, SR and resulting evidence-based CPGs serve as a resource for healthcare decision-making policies. The IOM suggests that a formal review process be created to publicly vet the SR as early as possible in the process. The benefits of this recommendation include minimizing bias, providing an opportunity to identify ongoing SR by other groups, thereby avoiding unnecessary (duplicate) efforts, and encouraging collaboration among various organizations (2). Given these potential benefits, the Workgroup acknowledges that a formal peer review process of the SR may enhance the clinical value of the final document. It is essential that these SRs be rigorous, credible, and, to the extent possible, free of commercial, professional, and intellectual bias. Unlike the ACCF/AHA, the IOM does not distinguish between intellectual bias and perspective. Rather, the IOM defines certain standards, which are designed to minimize bias, that guide the creation of a focused SR policy. These include a defined protocol that ensures balanced and unbiased user and stakeholder input to the SR before the initiation of the process and a well-defined policy to manage bias and RWI for all individuals involved in the SR.

Having a mechanism for various stakeholders, including the public, payers, and governmental agencies, to be able to provide input on which topics would benefit from an SR, how the review is designed and conducted, and potential data sources is important. The Workgroup supports the recommendation that the SR questions be vetted by agencies in the public realm (e.g., FDA, Centers for Medicare and Medicaid Services, health plans such as Kaiser Permanente, and United Healthcare). ERC and GWC members would be responsible for determining the scientific validity of the comments, if there would be a need to modify the PICO(TS) question(s), and whether additional data/studies should be included in the SR. Concerns relating to the number of comments and potential bias of the public reviewers may be problematic, and therefore no direct response would be anticipated or required by the ERC and GWC. With the creation of an ERC and the inclusion of new stakeholders (e.g., policy makers and payers), the formal peer review and SR process would be enhanced. It is recommended that members of the ERC be responsible and make final decisions about the design, analysis, and reporting of the identified and selected data.

• 1An ERC composed of content experts, methodologists, statisticians, and other identified stakeholders (policy makers and payers) should participate in the creation of ACCF/AHA CPGs (initially, with a focused approach to a confined topic). The ERC should interact with the members of the GWC; however, the ERC's responsibilities will be separate, distinct, independent, and clearly delineated.
• 2In an effort to standardize the process by which ACCF/AHA CPGs are created, the PICO(TS) format should be applied to develop the SR questions. It will be the responsibility of the ERC to create the format of the SR.
• 3The current peer review process of ACCF/AHA CPGs should be expanded to include a formal peer assessment of the SR protocol. External stakeholders and patient representatives should be included in this expanded process that will serve to enhance the applicability of the document to real-world decision-making policies and clinical scenarios.

See (11.3).

(Authors: Dr. Robert A. Guyton, Chair; Drs. Jonathan L. Halperin, Sidney C. Smith, Jr., and Marnie Bertolet, and Ms. Marguerite A. Koster)

The 2 publications from the IOM released in 2011, “Clinical Practice Guidelines We Can Trust” (1) and “Standards for Systematic Reviews” (2), present a challenge to developers of CPGs. A compelling argument is made that CPGs are currently not at the level that we seek as we allocate scarce societal resources to fund an expanding repertoire of sophisticated preventive measures and therapies for human illness. In particular, the authors of “Clinical Practice Guidelines We Can Trust” stated that quality healthcare practice is guided by the combination of critically appraised and synthesized scientific evidence, clinical experiential knowledge and skill, and patient values and preferences (1). In this context, a new definition of CPGs has been created: CPGs are statements that include recommendations intended to optimize patient care that are informed by an SR of evidence and an assessment of the benefits and harms of alternative care options (1).

Relevant to the issue of finding and assessing individual studies, Standard 4.1 states that developers of CPGs should use SRs that meet the standards set by the IOM Committee on Standards for Systematic Reviews of Comparative Effectiveness Research (2). In addition, Standard 5.1 states explicitly that a summary of available evidence (and evidentiary gaps) and a description of the quality of evidence (including applicability), quantity (including completeness), and consistency of the aggregate available evidence should be a part of the CPG process, including a rating of the level of confidence in the evidence supporting the recommendation (1).

The charge to this Workgroup is also highlighted in the criteria for inclusion of CPGs in the National Guideline Clearinghouse. These criteria state that corroborating documentation must be available to prove that a systematic literature search of existing scientific evidence from peer-reviewed journals was performed during the CPG development process. It is notable that CPGs are not excluded from the National Guideline Clearinghouse if gaps in scientific evidence for some recommendations are present and there is documentation that a search has been made for this evidence and the gaps are specified in detail. The IOM report also states that (guidelines) that have not included a thorough SR of the relevant scientific evidence should be excluded from the National Guideline Clearinghouse (1).

Challenges in Finding and Assessing Individual Studies

The area of evidence appraisal is described in the IOM report as “besieged with problems.” However, there is a consensus that a standardized rating of evidence quality facilitates the application of individual studies to the risk-benefit analysis that is the foundation of a clinical recommendation.

There are major concerns about bias and relevance of published studies. A 2005 study of trials published in the New England Journal of Medicine, the Journal of the American Medical Association, and The Lancet found that approximately three fourths of the trials were funded by industry (1). Another study revealed that high-quality commercial trials in this study were found to be 5.3 times more likely to endorse the product of the funding industry than noncommercially funded studies of the same product (1). This is certainly understandable, as many of these trials constitute preapproval research conducted for approval or labeling of either devices or medications. The trials have been designed to prove that the drug or device is safe and effective. These trials often tend to utilize young healthy patients who differ from the actual target population of the drug or device. The trials may intentionally use as a comparator a placebo rather than a competitor drug, and often surrogate endpoints are used to allow a shorter period of time for follow-up.

Even when studies have minimal bias (i.e., they have high internal validity), they may not be relevant for the patient population being considered by the GWC. This is referred to as relevance, generalizability, or external validity. RCTs often have an underrepresentation of older patients, of patients with comorbidities, of ethnic minorities, and of low-income patients. The IOM report quoted a 2007 study of evidence quality in cardiovascular risk management and found that only 28% of 369 recommendations were based on high-quality evidence. The most common reason for downgrading the quality of evidence in RCTs was concern about generalizability, that is, extrapolating from the carefully selected trial enrollees to the general population in question (1).

The IOM report “Standards for Systematic Reviews” offered a set of standards for conducting the SR and elements of performance within these standards (2). In the area of finding and assessing individual studies, 6 standards were offered, composed of 30 elements that are addressed specifically in the (11.3). The Workgroup quickly recognized a number of challenges. First, ACCF/AHA CPGs have been focused on clinical topics rather than specific clinical questions. The standards for SR in general expect the SR to be focused around a limited number of clinical questions that may be optimally organized around the PICO(TS) format.

The second challenge the Workgroup faced was the labor-intensive and resource-intensive nature of SR. In the past, ACCF/AHA CPGs have used literature databases with search engines and relevant topics, key words, or medical subject headings. This has been augmented by the extensive knowledge of the literature derived from the experience of the GWC. The primary potential problem with the current system is the possibility of bias, in particular, the perpetuation of bias from year to year and from guideline to guideline.

The third challenge discussed by the Workgroup was the difficulty in transitioning from the current CPG process to a process focused around SR of relevant clinical questions. An effort was made to find an iterative transition from the current process to a process more fully and carefully informed by SR with implementation of multiple elements of performance from the IOM report.

IOM Standard 3.1: Conduct a comprehensive, systematic search for evidence

A formal SR of the evidence for the entire scope of topics in an ACCF/AHA CPG may not be feasible given existing resources. The Workgroup believes that a more formal review of the literature for ACCF/AHA CPGs, incorporating many of the performance elements proposed by the IOM committee, is an appropriate and necessary step for our guidelines to remain a trustworthy source for recommendations in cardiovascular clinical practice. If a comprehensive SR is not feasible, the processes and procedures used to review the evidence need to be predetermined and documented by an ERC that works with the GWC but is functionally separate.

The Workgroup believes that all evidence reviews should be conducted by staff research specialists who have had training in developing comprehensive search strategies and conducting searches.

The Workgroup believes that the ACCF/AHA CPG process needs to move in iterative manner toward a more robust SR of the literature. Acknowledging that a complete formal SR for all CPG topics is not possible at this time, given existing resources, the Workgroup recommends that 2 or 3 key clinical questions within selected upcoming CPGs be used to pilot an SR process. These key clinical questions could be part of a focused update or a focused approach to a confined topic. It should be noted, however, that if a focused update is to be the pilot of an SR, then the appropriate PICO(TS) question would need to be formulated for the focused update, and all relevant evidence, not just new evidence, would need to be reviewed. A “focused” update does not imply a limited SR. It does imply that the update question might be adequately framed in a single PICO(TS) question for efficient SR.

IOM Standard 3.2: Take action to address potentially biased reporting of research results

Gray literature databases should be searched and relevant findings included as appropriate as evidence in the SR. Gray literature sources would include clinical trial registries (e.g., http://www.clinicaltrials.gov), clinical databases such as the National Cardiovascular Data Registry or the Society of Thoracic Surgery database, industry reports to the FDA, and unpublished abstracts <2 years old. Lack of complete information and lack of peer review are issues that need to be considered with the gray literature. The gray literature should be searched especially for signals of unreported safety issues.

Communication with researchers about information to assess the risk of bias in a study is appropriate, as is communication about unpublished data that may supplement the published data of a study. Again, the absence of peer review is a particularly prominent issue, as is the potential perpetuation and exaggeration of the bias of the sponsor and/or researcher.

IOM Standard 3.3: Screen and select studies

For each pilot SR, criteria should be prespecified for inclusion or exclusion of studies. The use of clinical questions in a PICO(TS) format would greatly facilitate this process. Observational studies in addition to RCTs should be included. A separate ERC should conduct an SR of the literature. Members of the ERC should be without RWI relevant to the topic of the SR. The Workgroup recommends training a cadre of volunteer members to conduct the SR, including screening and selecting studies, abstracting data from the studies, and assessing the quality of the studies. ACCF/AHA members early in their academic careers may find benefit in becoming involved in this process with no financial remuneration. Academic credit might include letters to department chairs, listing as part of a group that conducts SR, and listing of ERC members as coauthors of the CPG along with the GWC. Individuals who screen and select studies would need to work from written procedures and forms. They would need training and verification of accuracy and consistency. The ACCF/AHA research staff should perform the literature searches. The titles and abstracts of all articles should be reviewed with dual screening by 2 ERC members, with a third member adjudicating conflicts. In addition to RCTs, observational studies should be included to address gaps in evidence from RCTs. Staff and screeners should also look for observational studies and unpublished studies seeking benefits and potential harm that would otherwise be missed if only RCTs were reviewed.

IOM Standard 3.4: Document research

The current ACCF/AHA CPG methodology is compliant with this IOM standard with the exception of documentation of the reason for exclusion of peer-reviewed RCTs and large observational trials published in major journals. This exclusion judgment should be documented and justified for each such study.

IOM Standard 3.5: Manage data collection

For each pilot SR, a standardized data extraction form should be developed that includes some standard elements and additional elements unique to the PICO(TS) question under consideration. The data extraction form should be piloted in a few studies and then revised. For the broader clinical questions in the more expansive CPGs (the portions of the guideline not subject to a formal SR), a similar data extraction form should be developed.

IOM Standard 3.6: Critically appraise each study

Documentation of the assessment of 1) the risk of bias, 2) the relevance of the study's populations, interventions, and outcomes measures, and 3) the fidelity of implementation of interventions is essential. The system used to assess the studies in these 3 domains is controversial. Multiple scoring tools exist, including the one under evaluation by the ACCF/AHA.

The Cochrane Risk of Bias tool uses a qualitative, domain-based evaluation and judges each study across 6 domains of bias: selection, performance, detection, attrition, reporting, and “other.” In each domain, bias is judged as low risk of bias, high risk of bias, or unclear risk of bias, with brief supporting comments for each judgment. There also is a summary judgment for each study summarizing the overall risk of bias incorporating information from all 6 domains. It is recommended that the summary judgment be outcome specific (10). The Cochrane tool is a nonquantitative tool of the type thought by many (including members of the Workgroup) to be preferable to quantitative tools or checklists.

For each study, the same 3 members of the ERC who performed data extraction could simultaneously perform a quality assessment (2 members perform a quality assessment with the third member to adjudicate differences). Whether a scoring tool is used or a nonquantitative tool such as the Cochrane tool is used, a separate tool is needed for observational studies and registries that is distinct from the tool used for RCTs.

Proposal for the Critical Assessment of Each Study

The critical appraisal of each study has 3 required elements of performance:

• 1Systematic assessment of the risk of bias, using predefined criteria
• 2Assessment of the relevance of the study's populations, interventions, and outcomes measures
• 3Assessment of the fidelity of the implementation of the interventions

The Workgroup proposes the following system for critical appraisal of each study:

• 1This system will be used in the pilot SR, conducted in conjunction with CPGs. In this case, it is anticipated that the assessment of the quality of the study will be focused within the context of a PICO(TS) question.
• 2It is expected that a 3-member team from the ERC associated with the GWC will conduct the critical assessment of each study. For efficiency, it is suggested that these 3 people be the same 3 people who conduct the data extraction from each selected study.
• 3The pilot system for critical appraisal of each study will be composed of a quantitative assessment, a yes/no assessment of key features of the studies, and a scoring of each of the 3 domains identified by the IOM: risk of bias, relevance, and fidelity of implementation.
• 4A quantitative assessment is currently undergoing evaluation and validation by the Task Force. This quantitative assessment will need some revision for application to observational studies and for focused assessment of the 3 domains identified by the IOM.
• 5The qualitative assessment of the key features of each study may be the yes/no answers on the quantitative assessment.
• 6A scoring of the quality of the study in each of the 3 domains will be qualitatively accomplished judgmental scoring by the 3 members of the ERC. The domain “risk of bias” should be renamed “freedom from bias” (or “internal validity”) so that the judgment system is directionally similar for all 3 domains. Otherwise, a high-quality assessment of the “risk of bias” domain would be “low,” while such an assessment of relevance or fidelity would be “high.” By renaming “risk of bias” as “freedom from bias,” all 3 domains can have similar scoring of high to low, indicating high to low quality in that domain. For each of the 3 domains—1) freedom from bias, 2) relevance, and 3) fidelity of implementation—one of the following judgments will be assigned: fatally flawed, low (low freedom from bias, low relevance, or low fidelity), intermediate, or high. In addition, there will be a summary assessment of each study: fatally flawed study, low-quality study, intermediate-quality study, and high-quality study.
• 7Although the freedom from bias of the study is somewhat independent of the PICO(TS) question being addressed, the other 2 domains, relevance and fidelity of implementation, are necessarily dependent on the population, intervention, comparator, outcome, and setting under consideration. Therefore, the assessment of the quality of the study must be made in the context of the PICO(TS) question under consideration.
• 8For other studies being considered in other parts of the CPG, a similar evaluation of the quality of the individual studies may be carried out. It will be necessary, however, to clearly identify, for each proposed recommendation, the population under consideration, intervention, comparator, outcome and setting, at least in a broad and general way.
• 9With this system, each study, when considered as part of the body of evidence for assessment of the LOE for a specific recommendation, will have an overall quality assessment. Each study will also have a qualitative assessment of quality in each of the 3 domains: freedom from bias, relevance, and fidelity of implementation. The scoring tool, if a quantitative tool is used, and the yes/no answers on key elements of the study will be used to inform the judgment on the 3 domains and the summary judgment of the quality of the study.
• 10It is suggested that the proposed system might combine the advantages of the quantitative tool, the qualitative yes/no assessment of key points in the study, and the need for judgment, which is study specific and context specific in the assessment of quality in each of the 3 domains and the summary study quality assessment.

See (11.4).

(Authors: Dr. Jeffrey L. Anderson, Chair; Drs. Nancy M. Albert, Elliott M. Antman, David L. DeMets, Eduardo Ortiz, Eric D. Peterson, and Clyde W. Yancy, and Ms. Marguerite A. Koster)

Workgroup 4 was assigned 2 overarching tasks: 1) to address setting standards for synthesizing a body of evidence, and 2) to review and make recommendations on grading of guidelines, that is, on the ACCF/AHA system of establishing COR/LOE.

The first task was subdivided into 4 sections, following the IOM's report on “Standards for Systematic Reviews.” These 4 sections comprised the following standards: 1) assessing the quality of a body of evidence, 2) describing the quality of a body of evidence, 3) qualitative synthesis of a body of evidence, and 4) quantitative assessment of a body of evidence (e.g., meta-analysis of the universe of studies to address a specific research question) and how to determine the need for qualitative and/or quantitative evaluations. It was suggested that the formulation of specific research questions be guided by the mnemonic PICO(TS).

Workgroup 4's scope of work and charge included the important task of comparing and contrasting the IOM report with ACCF/AHA CPG methodology for synthesizing the body of evidence, which is summarized in the (11.4). Where differences or deficiencies in current ACCF/AHA CPG processes were identified, the Workgroup was to propose recommendations for bringing these into compliance with the IOM recommendations or to defend the current processes where the Workgroup thought a change was not warranted.

Three key areas of discussion emerged with respect to the differences between the IOM recommendations and the ACCF/AHA processes: 1) what was the strength of evidence backing the advantage to patient care (superiority) of the IOM proposal? 2) what resources/cost implications would be associated with the proposed change (an overlapping concern)? and 3) how disruptive would a major change be to current ACCF/AHA programs (caused by a domino effect throughout the 2 organizations and the cardiovascular universe) from the current CPG COR/LOE grading system?

Overall, the Workgroup thought that the current ACCF/AHA CPG methodology has served the cardiovascular community well, as evidenced by improved outcomes associated with adherence to CPG recommendations ((11),12). Thus, the Workgroup endorsed a plan of incremental change occurring over time, that is, “evolution rather than revolution,” assessing the impact and value of change at each step along the way.

• 1Qualitative synthesis of a body of evidence: assessing and describing the quality of a body of evidence. Qualitative synthesis deals with analyzing the essential characteristics of a study, that is, risk of bias, consistency, precision, and directness; and, for observational studies, additionally dose-response association, confounding factors that would change the observed effect, and strength of association. Questions posed to the Workgroup included the following:

• •How does the current ACCF/AHA methodology look at the qualitative characteristics of a study and then look at the main variables across a body of evidence?
• •How do other evidence grading methodologies (i.e., Grading of Recommendations Assessment, Development and Evaluation [GRADE], NHLBI) address the synthesis and analysis of characteristics across a body of evidence? What is required for such an analysis, for example, data summary tables?
• •How does the qualitative synthesis/analysis change when some of the studies are observational?
• •What are some key qualitative characteristics that are important for RCTs as well as observational studies?
• •What is the optimal and most practical/efficient way to synthesize evidence looking across all studies and to focus on trends, themes, relevance, and observations?

Because Workgroup 3 addressed assessment of quality at the individual study level, Workgroup 4's recommendations apply to assessment of evidence quality across the body of evidence.

The term qualitative synthesis refers to evaluating and summarizing the findings of a body of evidence for each prespecified outcome of interest in a specific clinical question. The IOM further elaborates on the definition of qualitative synthesis as … an assessment of the body of evidence that goes beyond factual descriptions or tables that, for example, simply detail how many studies were assessed, the reasons for excluding other studies, the range of study sizes and treatments compared, or quality scores of each study as measured by a risk of bias tool. While an accurate description of the body of evidence is essential, it is not sufficient ((2), p. 196).

The IOM report recommends 4 elements of performance for conducting a qualitative synthesis of a body of evidence for an outcome of interest:

• 1Describe the clinical and methodological characteristics of the included studies, such as their size, inclusion or exclusion of important subgroups, timeliness, and other relevant factors.
• 2Describe the strengths and limitations of individual studies and patterns across studies in plain terms, how flaws in the design or execution of the study (or groups of studies) could bias the results, explaining the reasoning behind these judgments.
• 3Describe the relationships between the characteristics of the individual studies and their reported findings and patterns across studies.
• 4Discuss the relevance of individual studies to the populations, comparisons, cointerventions, settings, and outcomes or measures of interest.

The IOM report described 8 “basic characteristics of quality” that are essential for assessing the quality of a body of evidence for a particular outcome of interest, and a discussion of these characteristics is an important component of the qualitative synthesis. Characteristics are risk of bias (limitations in study design and conduct); consistency of results across studies; precision of the estimates of effect; directness of the evidence in terms of study design, populations, interventions, comparisons, or outcomes; and reporting bias. For observational studies, additional characteristics include dose-response association, plausible confounding factors that would change an observed effect, and strength of association (i.e., large magnitude of effect). Initially developed by the GRADE Working Group (13), these characteristics for assessment of a body of evidence have been endorsed by the Cochrane Collaboration, the AHRQ Effective Health Care Program, and many medical professional societies and healthcare organizations involved in evidence review efforts, especially for development of CPGs.

The GRADE system (13) designates 4 levels for assessing the overall quality of a body of evidence: “high,” “moderate,” “low,” and “very low.” These levels refer to the degree of confidence in the estimate of effect. A body of evidence consisting of RCTs begins at the highest level and can be downgraded on a point system based on methodological limitations of the studies (risk of bias), indirectness of evidence, inconsistency of reported results, imprecision of the effect estimates, and/or publication bias. A body of evidence consisting of observational studies begins at a “low” level and can be upgraded based on a large magnitude of effect, a dose-response association, or few confounding factors that would change the reported effect. The GRADE Working Group has developed a software program, GRADEprofiler (“GRADEpro”), for conducting the quality assessment of a body of evidence (14). The software generates 2 types of tables that are useful for summarizing the quality assessment: “evidence profile” tables and a “summary of findings” table.

The AHRQ Effective Health Care program (15) has essentially adopted the GRADE framework; however, it uses the term “strength” to assess what GRADE calls the “quality” of a body of evidence. Rather than use the term “very low” to categorize the lowest quality of evidence, the Effective Health Care program uses the term “insufficient.” Furthermore, the Effective Health Care program has an additional domain to assess “applicability” of the body of evidence to account for the degree to which the evidence is relevant to patient populations, settings, diseases or conditions, interventions, comparators, and outcomes of interest. At this time the Effective Health Care program does not mandate use of GRADEpro or other software and allows for use of other weighting systems or a qualitative approach.

Many medical professional societies (e.g., American College of Physicians, American College of Chest Physicians, and the American Academy of Neurology), government organizations (e.g., the NHLBI), and healthcare organizations (e.g., Kaiser and the Institute for Clinical Systems Improvement) have revised their grading systems and qualitative syntheses of a body of evidence to incorporate all or some of the characteristics or domains used by GRADE, AHRQ, the Cochrane Collaboration, and other leaders in SR methodology. Some organizations use a combination of the GRADEpro software evidence profiles and summary of findings tables and additional narrative description to characterize the body of evidence, whereas others rely on an entirely descriptive approach.

Most ACCF/AHA CPGs currently include narrative descriptions and evidence tables detailing the clinical characteristics of individual studies (e.g., patient populations, inclusion/exclusion criteria, endpoints, point estimates, and confidence intervals). Current ACCF/AHA CPGs do not, however, explicitly evaluate a body of evidence by outcome, nor do they explicitly take into consideration an evaluation of the basic characteristics of quality using a specific checklist or quality scale tool. Rather, the assessment of quality (and strengths and weaknesses of each study) is inherent to the overall evidence review and is based on the recognized expertise of the GWC.

• 1Accept the IOM-recommended basic characteristics of quality and elements for assessing quality across studies.
• 2Standardize the method of assessment and description of the quality of the body of evidence across studies and carry out a qualitative assessment of individual studies and aggregated studies. (This will require allocation of resources for additional methodological expertise, because this exercise is judged to be beyond the expertise and time commitment expected of GWC members alone; an ERC would likely assume this role.)
• 3Depict qualitative assessment across studies addressing the key elements for each PICO(TS) question in a summary table (e.g., similar to GRADE or other formats [Table 1]).
• 4Where appropriate, provide a risk-of-bias assessment table across studies (e.g., similar to the Cochrane format (10) that includes items such as random sequence generator, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selection report, or other bias.
• 5Generate standardized summary and evidence table templates specific to ACCF/AHA requirements.
• 6When available, high-quality SRs from reputable organizations (e.g., the Cochrane Collaboration and AHRQ evidence-based practice centers [EPC]) should be used. If a new SR is needed, resources will need to be secured to pursue de novo analyses.
• 7Notwithstanding these efforts to standardize and improve the qualitative analysis of bodies of evidence, every effort should be made to minimize delays in time for CPG development/revision.

• 2Standards for quantitative analysis (i.e., meta-analysis): Quantitative analysis refers to conducting a (multiple) meta-analysis and deals with the statistical manipulation of data along with the aggregation of data presented in individual studies. This analysis should address heterogeneity among study effects, determine the statistical uncertainty for all estimates, and assess the sensitivity of analysis based on the changes/differences in protocol, assumptions, and study selection.

Quantitative Analysis of Evidence

The standard evidence review to date has focused on qualitative attributes of relevant data to determine the LOE that might inform a CPG. This approach, although very pragmatic and reasonably successful in the past, involves subjective assessments and may be less appropriate for increasingly complex data in contemporary clinical trials and/or observational data.

Addressing Conventional Quantitative Analyses

Broadly speaking, a quantitative analysis would consider a more detailed assessment of the strength of evidence beyond the ordinary “positive” RCT. The notion of a positive trial as the ultimate arbitrator of an evidence base does not speak to the magnitude of the evidence or the potential impact of the evidence if applied to de novo patient cohorts. Specifically, a threshold p value of 0.05 is known to be an arbitrary threshold to determine significance and practically can be defined as the likelihood that the given finding is a random occurrence in 5%, or 1 chance in 20. This may or may not be compelling for certain interventions, and a stronger threshold, p<0.01, or the randomness is <1 chance in 100, may be more appropriate. This may be especially the case for more invasive or expensive interventions. An even more stringent threshold might be appropriate for large observational datasets that have a large number of observations, and a p value of 0.05 might be easily reached for nonsensical (nonclinically significant) observations. In these cases, a threshold of p<0.001, or <1 chance in 1,000 that the results are due to random accident, might be the preferred strategy. Thus, a more appropriate fit of statistical threshold with a data source and within the context of the given data might be a more reasonable approach and would add a helpful quantitative analysis to the current ACCF/AHA evidence review. It is likely that current GWCs exercise this kind of thoughtful review empirically, but as a policy statement, direction for these types of quantitative assessment have not previously existed.

Considering Newer Quantitative Analyses

Beyond the determination of p values, additional quantitative insight is determined from a review of confidence levels. Understanding the “splay” of data may put the top line findings of a given study in a different context and either strengthen the LOE despite a modest level of statistical significance or temper the LOE for a reasonable level of significance but with broad confidence levels. Similarly, evaluating relative risk reduction versus absolute risk reduction can be quite informative. A large multithousand-patient study with a relative risk reduction of 0.15 might reflect an actual risk reduction of only 1% or 2%. This is important information that generates a new perspective/interpretation of clinical trial results. The number needed to treat versus the number needed to harm represents a very informative quantitative representation of available data and further allows discrimination of the veracity of the findings. Importantly, the comparison of number needed to treat and number needed to harm captures the burden of risk associated with emerging new therapies for cardiovascular disease.

Incorporating More Rigorous Quantitative Methodology

Meta-analyses and Bayesian analyses are powerful data analytical tools that may further inform the significance of candidate data to support the generation of CPGs. However, care must be exercised in the use of both of these rigorous methodologies. Meta-analyses are subject to the quality of the studies that are aggregated to either identify a stronger signal of benefit or harm. Publication bias is a consistent concern, and a funnel plot analysis is necessary to mitigate this known source of bias. Moreover, many trials are sufficiently dissimilar so that the construct of a meta-analysis may be fundamentally flawed at the outset. However, a well-constituted meta-analysis may confirm findings from isolated trials and can be of sufficient quality to inform CPGs at the highest tier of evidence.

Bayesian analyses are similarly not perfect tools. The prior probability of an event determined from clinical judgment sets the threshold from which the posterior probability of a Bayesian analysis might be constructed. Bias may influence even a Bayesian analysis. However, a Bayesian approach is the closest statistical analytic model that approximates clinical judgment and decision making; in clinical practice, all practitioners begin with a “prior probability,” that is, an informed perspective of the expected clinical response to or outcomes of a test or intervention. Thus, the results of a well-performed Bayesian analysis may be very helpful in the construct of a CPG.

A more contemporary evidence review is needed for novel datasets that include not only traditional RCTs but also large observational datasets, adaptive clinical trial designs, genomic datasets, and nonrandomized data. As more quantitative data standards are adopted, it will be necessary to have statistical expertise embedded in the infrastructure of all CPG generation paradigms.

• 1GWCs, in conjunction with the Task Force, should determine when a specific meta-analysis is needed.
• 2With statistical consultation, preferred acceptable methods for meta-analysis (including a Bayesian analysis when appropriate) should be defined in each instance.
• 3When available, high-quality meta-analyses from reputable organizations (e.g., the Cochrane Collaboration and AHRQ EPC) should be used. If a new meta-analysis is needed, resources will need to be secured to pursue de novo analyses.
• 4Notwithstanding efforts to standardize and improve the approach to quantitative analyses across studies, every effort should be made to minimize delays in time for CPG development/revision.

• 3Review of ACCF/AHA COR/LOE versus other grading methodologies such as GRADE and NHLBI methodology for grading evidence and recommendations. Questions posed to the Workgroup on grading recommendations included the following:

• •What features of the COR/LOE allow for the synthesis and grading of evidence across multiple studies?
• •What features of GRADE, AHRQ EPC, and NHLBI methodology allow for the synthesis and grading of evidence across multiple studies?
• •What are some changes that may help the COR/LOE incorporate/improve on its current methods of synthesizing and grading evidence across multiple studies?

The objective of this question was to take a “10,000-foot” view of the ACCF/AHA COR/LOE approach and how this current methodology addresses and/or allows for the overall synthesis of evidence across multiple studies. Then, the objective was to compare the ACCF/AHA methodology with some other prominent methodologies in current use, such as GRADE, NHLBI, and AHRQ EPC (Tables (Table 2).(Table 3) to (Table 4).Tables 4a, 4b, 6, 8). The final goal then was to recommend whether and what practical changes/additions might be made to current ACCF/AHA COR/LOE methodology to improve its clarity and accuracy.

Review of COR/LOE Versus Other Methodologies

Professional societies that write CPGs typically provide the end user with a recommendation that has 2 dimensions. The first dimension is a statement of the strength of the recommendation, signifying the GWC's synthesized interpretation of the evidence with respect to how strongly the committee believes that the action/treatment may contribute to providing optimum care of a patient with the target condition discussed in the CPGs. The second dimension is a statement about the strength of the evidence on which the recommendation is based. The purpose of the strength of evidence component is to offer the end user insight into the range and type of evidence reviewed by the GWC when formulating the recommendation. The strength of recommendation and strength of evidence notations are provided in a hierarchical scheme that is ordinal and does not imply any mathematical relationship along the continuum of recommendations. The scheme by which the ordinal bins are depicted to end users varies by organization and may contain combinations of numbers, letters, or phrases that have face validity for conveying the rank order intended by the GWC.

Although the universally agreed-on purpose of CPGs is to offer an instruction set to clinicians caring for patients, it is also understood that individual clinical circumstances may necessitate deviation from the recommended path. Clinicians are advised to document their reasons for deviation from the recommendations. Beyond these straightforward statements, there is variation in how organizations view the intended use of their CPGs. Some organizations write their CPGs with anticipation that they will be intended for end users beyond clinicians, including policy makers responsible for reimbursement for medical care and even patients themselves. When such extended targets of the recommendations are contemplated by GWCs, the instructions include consideration of resources and perceived patient preferences.

Perspective on ACCF/AHA CPG Grading System Relative to Other Systems

With the background noted previously, it is useful to reflect on the ACCF/AHA CPG grading system. The intended audience is clinicians in North America caring for patients with cardiovascular disease. Although it is anticipated that the recommendations may be read with interest by policy makers and even patients, the instructions to GWCs for ACCF/AHA CPG development do not explicitly ask that the wording of recommendations take into account cost or patient preferences (although the GWC addresses the latter in each preamble, noting that patients should be informed of the risks, benefits, and alternatives to a particular treatment and should be involved in shared decision making whenever feasible, particularly for COR IIa and IIb, where the benefit-to-risk ratio may be lower). The GWC may comment in the accompanying text regarding resource implications, but such considerations are not intended to drive the wording of a recommendation. Underlying the ACCF/AHA methodology for CPG development is a set of considerations for focusing the ACCF/AHA recommendations specifically on the science of the data and are: 1) to reserve a place in the set of instructions for clinicians where science trumps other considerations because cost is dynamic and variable from system to system, 2) to allow those bodies in our society that determine how resources are to be allocated to have a reference point for the science review, and 3) in the absence of formal patient data using time tradeoffs and utility function testing, it is difficult to describe with certainty the distribution of patient preferences.

As with the other grading schemes, recommendations with the largest estimated size of intervention effect in clinical practice are those that are the strongest for (green rows in accompanying Table 3) or against (red rows in accompanying Table 3) providing an intervention because of lack of benefit or even some evidence of harm. All other recommendations are discretionary, and clinical responses to them are heavily influenced by the characteristics of the patient and the circumstances surrounding that patient's care.

There are no rigorously performed studies comparing the consequences of clinical actions that adhere to recommendations provided in one scheme versus another. Data are available, largely from registry reports, that adherence to the current ACCF/AHA system (Table 3), which is familiar to cardiovascular clinicians worldwide and drives ACCF/AHA performance measures, is associated with improved patient outcomes ((11),12). In the absence of evidence of superiority of one grading system over another, a decision was made to retain the current ACCF/AHA COR/LOE structure and nomenclature, with minor modifications to “map” to terminology in other well-known grading systems. Data from individual studies and multiple studies (pooled by either the random effects or fixed effects methods) can be accommodated by the current ACCF/AHA system.

• 1Retain the current basic ACCF/AHA COR and LOE structure/nomenclature.
• 2Standardize how LOE: A, B, C are determined using a validated ACCF/AHA Evidence Grading Tool (under development) that will incorporate features of existing tools.
• 3Change the wording of LOE to Quality of Evidence (QOE) (once the ACCF/AHA Evidence Grading Tool is operational).
• 4Add a separate category for QOE: E (expert opinion), and generate specific definitions and examples for QOE: E.
• 5Change “Treatment Effect” on the COR/LOE table to “Intervention Effect” and indicate that “Intervention” includes medications, devices, therapeutic strategies, procedures, diagnostic tests, and other.
• 6Add adjectives that “map” COR/LOE to the NHLBI (and GRADE) (i.e., I [strong]; IIa [moderate]; IIb [weak]; III [against]).
• 7Take every opportunity to align COR as closely as possible with LOE/QOE.
• 8Provide a chart to map ACCF/AHA COR/LOE to other major grading systems (e.g., Table 2).

Additional Comments:

• 1Plan for regular review (e.g., annually) of CPG recommendations and modify as appropriate.
• 2Develop a list of metrics that should be measured and assessed.

Workgroup 4 recommends that additional assessment of the methodological quality (risk of bias) in individual studies, as well as an evaluation of essential characteristics of quality across the body of evidence, by outcome, be incorporated into a qualitative synthesis of evidence in ACCF/AHA CPGs. To accomplish this, Workgroup 4 recommends the following:

• 1Consider use of a standardized tool, such as the Cochrane Risk of Bias Tool, for assessing risk of bias within and across studies (10). For observational studies, use a tool that specifically evaluates the methodological limitations of such studies, such as risk of selection bias, potential confounding factors that may affect the estimate of effect, and biases in the design and execution of nonrandomized studies. The Cochrane Collaboration is currently modifying its Risk of Bias tool for use in nonrandomized studies.
• 2For assessment of consistency, precision, directness, reporting bias, and, with observational studies, large magnitude of effect, a dose-response association, or confounding factors that would change the reported effect, follow the GRADE or AHRQ definitions for assessment of each characteristic across the body of evidence for each outcome.
• 3Consider depicting the summary evaluation of the quality characteristics for a body of evidence in a table format. Use the GRADE evidence profiles and summary of findings tables as models of table formats.
• 4In addition to summary quality evaluation tables, provide a high-level narrative synthesis of the quality of a body of evidence that includes the IOM's 4 elements of performance for conducting a qualitative synthesis, as described above.

See (11.5).

(Authors: Dr. E. Magnus Ohman, Chair; Drs. Deepak L. Bhatt, Kay Dickersin, Raymond J. Gibbons, and William G. Stevenson)

CPGs have become an important part of the practice of cardiology and other aspects of healthcare delivery. They represent some of the most downloaded and quoted papers published in the literature and therefore are of major importance to the practicing clinician. Furthermore, several outcomes studies have suggested that adherence to ACCF/AHA CPGs is associated with both better in-hospital as well as long-term outcomes ((11),(12),21). There are 2 crucial factors for dissemination and acceptance of CPGs. They need to be readable and not merely tables of synthesized information. They should also allow the reader to understand the logic and process of evidence synthesis to form coherent recommendations that inform the practice of medicine (22). Therefore, in producing CPGs, GWCs need to recognize that although the body of data for a recommendation ranges from minimal to extensive; the presentation must be comprehensive, logical, and readable to be a relevant resource for clinical practice.

As recommended throughout this report, our goal is for SR to underpin our CPG recommendations. Recognizing that it is not possible to complete SR for all CPGs over a short period of time, we recommend that our CPGs be transformed sequentially, with SR being sought or performed first for those topics deemed to be of greatest importance. When high-quality SRs are available and up to date, they should be summarized and used as a basis for our CPGs. In the event that “conflicting” meta-analyses have been published, thoughtful consideration should be given to developing a formal SR to provide clarity and therefore enable better recommendations for the CPGs. When high-quality SRs do not exist or an update is needed, the ACCF/AHA could commission a new review. In cases where an SR has been determined to be of lower importance to other topics at that time, an informal assessment of evidence combined with expert opinion is conducted.

On occasion, SR may be performed on topics for which there are few or no RCTs. With only 1 trial or a few small trials, one is not sure whether reporting biases may threaten the validity of the summary findings. Where no RCTs exist and the question of intervention effectiveness is important or RCTs are difficult to perform (e.g., during cardiac arrest), investigators may decide to seek data from observational studies such as analysis of administrative or registry data. SR of observational studies are potentially more complex than SR of RCTs, as searching methods for observational studies are not well established and important observational data could be missed. In addition, observational studies are subject to selection and other forms of bias that can make their findings difficult to interpret. Thus, SR of observational data should be used sparingly and will require careful review (see Workgroups 2, 3, and 4).

Incorporating Systematic Reviews as Part of the CPGs

CPGs are, by design, summaries of SRs that can be quite lengthy and detailed. In areas such as hypertension or prevention, for example, literally hundreds of studies are potentially eligible for SR of intervention effectiveness. With the use of methods for summarizing all relevant data, an SR fully encapsulates information in a condensed format that can be translated into CPGs.

In the early 1990s, the AHRQ supported the University of California San Francisco−Stanford EPC to partner with the ACCF/AHA team producing the “ACC/AHA/American College of Physicians−American Society of Internal Medicine Guidelines for the Management of Patients with Chronic Stable Angina,” published in 1999 ((23),24). An EPC representative of that group attended the meetings of the GWC, and the EPC made several contributions to the CPG. First, the EPC performed a number of small literature searches that led to studies being incorporated into the CPG, for example, a summary of RCTs and meta-analyses of garlic therapy for treatment of risk factors appeared in the tables and figures of the final CPG. The largest portion of the work done by the EPC was a careful SR of the evidence for medical therapy for chronic stable angina and, in particular, comparisons of the commonly used drugs (beta blockers and calcium antagonists) that were used in the treatment of chronic stable angina. This was published separately (24). An example of incorporating a salient summary of the EPC finding into the text of the CPG is shown in (Table 9) and (Table 10), and (Figure 25_gr1a) and (Figure 25_gr1b). This approach allowed the natural flow of the CPG to be maintained and allows the complete SR, representing original research on its own, to be published independently.

Independent review of SRs in addition to that of CPGs is an important check on scientific rigor and validity. We believe that the independent review performed in the course of consideration for publication achieves this aim. Previously completed SRs that have been published in a peer-reviewed journal already meet this standard. We recommend that a newly commissioned SR also be published separately and independently from the CPGs and subject to the journal's peer review process, which will ensure that they are of the highest standard. Independent publication would also allow the academic group that is charged with performing the SR to achieve academic recognition for its work. However, independent publication does create challenges. Ideally, the SR should be accepted and at least in press before it is incorporated into the CPGs so that any changes in response to peer review have been included. The publication of SR findings separately from evidence that is based on the SR and incorporated into the CPGs should not be considered duplicate publication. The ACCF/AHA CPGs focus on the summary information and any salient figures or tables that are included are modified and cited appropriately. Perhaps, a more difficult issue relates to the timing of publication and embargo. Although publication of the SR potentially provides a preview of information from the CPGs to interested parties, we do not feel that this would diminish the CPGs, as the SR is only a portion of the information that will be provided in the CPG and does not include recommendations and additional clinical synthesis. SRs do not make practice recommendations; they are simply a summary of relevant data addressing a clinical question. The GWC will provide COR/LOE on review of the SR and in conjunction with other recommendations applicable to a similar clinical setting. Thus, the SR and CPG can be developed simultaneously (with potential edits/revisions made in response to peer review) and the SR published before, without embargo, or nearly simultaneously with the CPG.

Discordance between the ERC and the GWC will be resolved through liaison committee members (see Workgroup 2). A key feature of the review will be transparency. The GWC should specifically cite the SR and indicate how it was incorporated into the CPG.

Guideline Review Tables in the Absence of a Systematic Review

Because SR will not initially be possible for all recommendations, it will be important to distinguish recommendations based on SR from those based on narrative reviews and/or expert opinion. For transparency, the evidence base for recommendations based on non-systematically collected information should be provided in summary tables called “CPG review tables” as well. Items presented would be similar to those presented for trials included in the SR evidence tables, for example, characteristics of the individual studies and factors that allow an assessment of potential bias (i.e., study “quality”), measures of effect size for each study, and list of primary and secondary outcomes reported. Particular attention will be paid to ensuring that all relevant studies are included and that the components noted above are included. CPG review tables from nonsystematic reviews should not be given more prominence than evidence tables from SRs and so can be referred to in the text but are better placed in online data supplements. It is important, though, that the printed or online text, tables, and figures summarize the information to enhance readability of the document and that the recommendations flow naturally from the text. The components of such tables and examples are listed in (Table 11).

Like the evidence tables from SRs, the CPG review tables will undergo peer review as part of the standard review process.

CPGs are strengthened when they are based on SR. Because it is not possible to complete SR immediately for all questions covered by ACCF/AHA CPGs, SR will be completed in a sequential fashion in order of importance of the question. Non-SRs and accompanying summary tables will be used for areas where SRs are not yet completed; however, SRs may be performed over time. It is believed that this combination will bring the current approach of the ACCF/AHA CPGs in compliance with the recommendations of the IOM, yet allow the CPGs to be readable, timely, and relevant.

• 1SRs included in CPGs should be published as separate peer-reviewed manuscript(s) when feasible. By doing so, all of the recommendations regarding methodology, peer review, and access will be compliant with the IOM recommendations (2).
• 2Recommendations supported by SR should be identified in CPGs in addition to the appropriate class and LOE. Example: (Class I; LOE: A)^SR.
• 3Within the CPG, salient tables and figures from SRs should be included to support recommendations. The remaining pieces of methodology will be hyperlinked to the original SR publication. By doing so, key elements of the SR will be available for public access once the CPG are published.
• 4CPG review tables should be incorporated into CPGs for non−SR-based recommendations and be available as online supplement tables.
• 5CPG review tables should incorporate the majority of study components that are provided in SR tables.

American College of Cardiology Foundation

William A. Zoghbi, MD, FACC, President

Thomas E. Arend, Jr., Esq., CAE, Interim Chief Staff Officer

William J. Oetgen, MD, MBA, FACC, Senior Vice President, Science and Quality

Charlene L. May, Senior Director, Science and Clinical Policy

Erin A. Barrett, MPS, Senior Specialist, Science and Clinical Policy

American College of Cardiology Foundation/American Heart Association

Lisa Bradfield, CAE, Director, Science and Clinical Policy

Debjani Mukherjee, MPH, Associate Director, Evidence-Based Medicine

Ezaldeen Ramadhan III, Specialist, Science and Clinical Policy

American Heart Association

Donna K. Arnett, PhD, MSPH, BSN, FAHA, President

Nancy Brown, Chief Executive Officer

Rose Marie Robertson, MD, FAHA, Chief Science Officer

Gayle R. Whitman, PhD, RN, FAHA, FAAN, Senior Vice President, Office of Science Operations

Judy Bezanson, DSN, RN, CNS-MS, FAHA, Science and Medicine Advisor, Office of Science Operations