When a topic is prioritized for review by the Task Force for a new or updated recommendation, the scope of the topic and approach to the review must be defined to guide the researchers undertaking the systematic review process. This section applies to systematic reviews undertaken for a new topic or to update an existing topic. Work plan development for topic reaffirmation updates is described in Section 4.7.
A topic team is appointed for each prioritized topic before topic scoping begins and consists of Task Force leads (including one of the Task Force Chairs), at least one AHRQ Medical Officer, and the EPC review team. EPCs are scientific research centers tasked with conducting systematic evidence reviews that serve as the foundation for Task Force recommendations. Based on expertise and interest, several Task Force members are assigned to serve as leads for each topic. An AHRQ Medical Officer is assigned to oversee the topic and may be joined by the Task Force Scientific Director and/or Associate Scientific Director in overseeing that topic. A lead investigator is assigned by the EPC to lead the evidence review team.
Two integrated documents are developed during this phase of the systematic review: a work plan and a research plan. Both of these documents are revised and finalized through discussions with the Task Force leads and the AHRQ Medical Officer in an ongoing process that includes public comment on the research plan. The work plan is drafted by the EPC review team and captures the history, previous Task Force recommendations, and proposed approach to the topic. The purpose of the work plan is to establish the review perspective for the upcoming review. The template for the work plan is described below and in Appendix V.
Based on the draft work plan, a draft research plan that contains the analytic framework, key questions, and inclusion/exclusion criteria is created for public comment. After approval by the Task Force leads, the draft research plan is posted on the Task Force Web site for 4 weeks to allow public comment. All comments received during the public comment period are provided verbatim to the topic team, and the EPC review team summarizes major themes and makes suggested revisions to the research plan based on these comments.
The topic team discusses any major suggestions for revisions, the EPC review team incorporates final revisions into the research plan, and the Task Force leads approve the final research plan. For new topics, the work plan may be peer reviewed and presented to the entire Task Force at one of its regular meetings. Development of a work plan generally takes from 6 to 7 months, including public comment.
Table of Contents
- 3.1 Determining Topic Scope and Review Approach
- 3.2 Methods Relevant to Work Plan Development
- 3.3 Previous Task Force Review and Recommendations
- 3.4 Search for New Synthesized Evidence/Pending Studies
- 3.5 Current Task Force Review Approach
- 3.6 Peer Review of Work Plan
- 3.7 Public Review of Research Plan
- 3.8 Task Force Approval of Final Research Plan
3.1 Determining Topic Scope and Review Approach
The Task Force has determined that using systematic reviews is the best method for organizing and evaluating the existing scientific evidence relevant to questions about a clinical preventive service. In order to answer the relevant questions about a clinical preventive service, the EPC review team usually undertakes a series of related systematic reviews to answer each of the key questions in the analytic framework.
3.1.1 Principles for Determining the Review Approach
During work plan development, the EPC review team considers the scope of the evidence needed for the Task Force to make its recommendation. For reviews undertaken to update existing Task Force recommendations, this process is based on:
- Examination of the previous Task Force recommendation(s), including the populations and clinical preventive services addressed, to determine their fit with current questions about the clinical preventive service
- Examination of the previous Task Force evidence review process for the topic and the review findings in order to identify established evidence, important review limitations, and evidence gaps
- Determination of current contextual information (e.g., changes in understanding of the nature of the disease process or changes in diagnosis, therapeutics, or practice; controversy over any of these elements)
In order to facilitate the consistent development of the review approach across topics, the Task Force has developed a template to guide the development of the final work plan (Appendix V).
The work plan can be considered generally analogous to a protocol, such as those developed for an AHRQ Effective Health Care Program review or a Cochrane review. It is also an articulation of the rationale for the scope decisions made in framing the topic.
3.1.1.1 Primary Care Interventions Addressed by the Task Force
The Task Force has adopted the Institute of Medicine's definition of primary care:
Primary care is the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and community. This definition acknowledges the importance of the patient clinician relationship as facilitated and augmented by teams and integrated delivery systems. 7
The Task Force considers interventions that are delivered in primary care settings or are judged to be feasible for delivery in or referable from primary care. To be feasible in primary care, the intervention should target patients seeking care in primary care settings. Additionally, clinicians and/or related staff in the primary care setting should have (or could have) the skills necessary to deliver the intervention, or the intervention could be one generally ordered or initiated by a primary care clinician.
Task Force recommendations address primary or secondary preventive services. Primary preventive measures are those provided to persons in a clinical setting to prevent the onset of a targeted condition (e.g., aspirin for the prevention of colorectal cancer, counseling for a healthful diet), whereas secondary preventive measures identify and treat asymptomatic persons who have already developed risk factors or preclinical disease but in whom the condition has not become clinically apparent (e.g., screening for colon cancer). Interventions that are part of the treatment and management of persons with clinical disease are usually considered tertiary prevention and are outside the scope of Task Force recommendations.
3.1.1.2 Incorporation of Subpopulation Considerations
The Task Force incorporates subpopulation-specific concerns when they may represent substantial heterogeneity in screening or preventive treatment effects. Data on the incidence/prevalence, complications, morbidity, and mortality of the condition of interest should be routinely summarized by age, race/ethnicity, sex, or other important topic specific clinical characteristics. Additional details on the process for incorporating subpopulation considerations into systematic reviews for the Task Force and its evidence deliberations are under development and will be detailed in a future version of the Procedure Manual.
3.2 Methods Relevant to Work Plan Development
The work plan template (Appendix V) stimulates thinking and guides the systematic consideration of the factors that experience has shown are important in planning a review to update or issue a new Task Force recommendation. Since most reviews conducted for the Task Force are for updating previous TF recommendations, the work plan template was developed with that purpose in mind. However, the same template can be used to plan and guide the systematic review for a new topic; sections addressing the previous Task Force recommendation and previous review findings would not be included.
3.2.1 Analytic Frameworks
The purpose of an analytic framework (Figure 4) is to clearly present in graphical format the specific questions that need to be answered by the literature review in order for the Task Force to evaluate the effectiveness and safety of the proposed preventive service. The specific questions are depicted graphically by linkages that relate interventions and outcomes. These linkages serve the dual purpose of identifying questions to help structure the literature review and of providing an "evidence map" after the review for the purpose of identifying gaps and weaknesses in the evidence. Further details about the design of analytic frameworks are provided in a 1994 paper by Woolf et al.
Figure 4 . Template of an Analytic Framework
3.2.1.1 Conventions for Graphics and Layout
The analytic framework diagram contains three types of items (population, actions, and outcomes). Below the diagram are annotated questions that correspond to specific items in the diagram. The annotated questions are designated in the diagram by superscript symbols. The conventions that follow are illustrated in the accompanying prototype (Figure 4).
The population appears at the left margin of the diagram and specifies the type(s) of patients to whom the evidence about the preventive service pertains. For example, the population description in an analytic framework for cervical cancer screening might be "women at risk for cervical cancer." Footnotes in this location refer to specific questions about the population that the evidence review must answer in order to evaluate the effectiveness of the preventive service. For example, it may be necessary to know the proportion of the population with a family history of colon cancer.
Actions, such as screening and treatment, appear as arrows linking the population to an outcome or linking one outcome to another. Curved arrows lead to adverse effects of the action (i.e., "harms"). The outcome to which the arrow points should result logically from the action (e.g., early detection of disease for screening, behavior change for counseling, reduced incidence of disease for immunizations or chemoprophylaxis). The name of the action (e.g., "screening with the prostate-specific antigen test") appears below the arrow. Each arrow is a linkage in the logical chain of evidence that connects the left side (population) and the right side (health outcomes) of the analytic framework. Overarching linkages directly connect these two sides. Overarching linkages generally represent studies in which the population is randomized to the clinical preventive service and health outcomes are measured. This is considered direct evidence. Each arrow is a "key question" that must be addressed by an evidence review. However, in the situation where there is robust direct evidence for the overarching linkage (such as multiple population-based screening trials), there may not be a need to address the multiple indirect evidence linkages through systematic review.
Outcomes are depicted using a rectangle; intermediate outcomes have rounded corners and health outcomes have squared corners. A health outcome that follows an intermediate outcome, which typically reflects the natural progression of disease (e.g., from "retinopathy" to "visual impairment"), is depicted by a dotted line (no arrowhead). Other important outcomes (e.g., societal/legal effects, non-disease benefits) can be included in an analytic framework if needed for the topic, and can be depicted as intermediate or health outcomes as defined below. Annotated footnotes are specific key questions that are associated with each linkage and that must be answered by the literature review. The key questions are written in detailed narrative below the analytic framework. Details required to interpret the key questions are further delineated in the inclusion/exclusion criteria for each review.
3.2.1.2 Analytic Frameworks Are Not Causal Pathways
Analytic frameworks as used by the Task Force are not intended to comprehensively depict all factors and variables that cause patients to receive the preventive service or are responsible for the intermediate outcomes and health outcomes associated with a preventive service. In short, they do not depict the "mechanism of action" for a preventive service. For example, an analytic framework for cervical cancer screening that is concerned mainly with two questions (whether the Papanicolaou test detects early disease and whether early detection reduces mortality) need not specify other covariables, such as the risk factors for cervical cancer, demographic characteristics of women who are more likely to be screened, etiological determinants of cervical cancer, or pathological progression of cervical cancer from the atypical cell stage to invasive disease.
Although the research plan is developed and established for an ongoing review, the components of an analytic framework are not static for a given topic, and may require revision for future reviews and recommendations as the scientific basis for the clinical preventive service advances and the current important clinical questions, populations, or outcomes change accordingly.
3.2.1.3 Analytic Frameworks Are Not Decision Trees, Clinical Algorithms, or Flowcharts
The use of arrows and boxes gives analytic frameworks the appearance of decision trees and flowcharts, but the purpose is quite different. Analytic frameworks are not intended to depict all possible outcomes of a particular event, as is expected of decision analysis models, or to calculate their probabilities. Similarly, analytic frameworks do not guide clinical decisionmaking for an individual patient, nor do they depict every action in the sequence of services for a clinical preventive service. Instead, the analytic framework is a logic model of the minimal, sequential clinical assumptions that must be verified using empirical evidence in order to determine the net benefit of a preventive service.
3.2.1.4 Actions Versus Outcomes
Analytic frameworks used by the Task Force distinguish between actions (e.g., obtaining a screening test, treatment with a drug) and outcomes (e.g., detection of a disease, reduced morbidity and mortality, change in patients' behavior, adverse effects). The performance characteristics (e.g., sensitivity, specificity) of a screening test is not itself an outcome. Actions are depicted by arrows, whereas outcomes are depicted by rectangles (Figure 4).
3.2.1.5 Intermediate Outcomes Versus Health Outcomes
Analytic frameworks used by the Task Force distinguish between intermediate outcomes and health outcomes, and consider both beneficial and harmful outcomes (e.g., adverse effects of screening and treatment).
Health outcomes
Health outcomes are symptoms, functional levels, and conditions that patients can feel or experience and are defined by measures of physical or psychological well-being. A clinical "sign" is not a health outcome that is not sensed by the patient; a clinical sign is analogous to an abnormality on a blood test or radiologic exam (and therefore an intermediate outcome). Examples of health outcomes include visual impairment, pain or dyspnea, functional status, quality of life, impotence after prostatectomy, child development, and death.
Intermediate outcomes
Intermediate outcomes are outcomes that may be influenced by a preventive service but are not health outcomes in and of themselves. They are pathologic, physiological, psychological, social, or behavioral measures and other study endpoints related to a preventive intervention. Examples include blood pressure, serum cholesterol, vitamin levels, asymptomatic ductal carcinoma in situ (DCIS), asymptomatic carotid artery stenosis, weight, dietary intake, car crashes, improved educational achievement, reduced rate of psychiatric hospitalizations, and physical activity. The USPSTF gives greater weight to evidence of an effect on health outcomes than evidence of an effect on risk factors or intermediate outcomes. The fact that a preventive service has a proven effect on an intermediate outcome does not necessarily establish that it can improve outcomes that are perceptible to patients.
At times the USPSTF may consider the evidence on societal (including caregiver) outcomes. The effect of an intervention may extend beyond the individual to society as a whole or to another individual. For example, reducing an individual's alcohol consumption decreases mortality related to car crashes not only for that person but also for others on the road. Screening for cognitive impairment may provide benefits to the caregiver beyond that to the individual. In addition, the USPSTF may consider outcomes that are not traditionally in the realm of health, such as educational attainment. These are not direct measures of health but are indicators of positive or negative effects on the larger society. When being considered in the context of an evidence review for the USPSTF, societal outcomes are represented in the analytic framework as an intermediate outcome or a health outcome, depending on the specific topic.
When data are available and relevant to decisions about the preventive service delivery, the Task Force considers data on both all-cause and cause-specific mortality in making its recommendations (go to Section 4.5 for discussion of these outcomes).
3.2.1.6 Revisions
Analytic frameworks can evolve with time and may appropriately differ when recommendations are updated because of changes in clinical questions or important uncertainties about the evidence. During the systematic review, it is sometimes necessary to revise an analytic framework to more clearly reflect the methods of the review. New key questions may be added when new interventions, outcomes, or logical arguments emerge during the course of the review. If these revisions only reflect improving the clear communication of the systematic review methods, they can be undertaken by the EPC review team. If there are any scientific ramifications to a potential analytic framework revision, review and approval by the AHRQ Medical Officer and Task Force leads is expected. Such changes are reflected in the final review and manuscript resulting from the systematic review.
3.2.2 Key Questions
Key questions are an integral part of the approach to conducting systematic reviews the Task Force uses in its recommendation process. Along with the analytic framework, these questions specify the logic and scope of the topic, and are critical to guiding the literature searches, data abstraction, and analysis processes.
Key questions, in association with the analytic framework, establish the necessary steps in the clinical logic that must be demonstrated to evaluate the effectiveness and harms of a clinical preventive service in primary care. Key questions articulate the key aspects of the relevant populations, interventions, and outcomes—aspects that are essential in order to focus the review on a manageable and clinically relevant topic and to clearly communicate to readers what the review will address. In constructing key questions, the topic team must balance specificity of detail and readability; the detailed inclusion/exclusion criteria provide additional necessary details to understand how the key questions will be interpreted in the systematic review.
Each question is clearly tied to a step in the analytic framework, although certain linkages that are already well established may not have a key question that is actively answered during the review for the Task Force. In addition, there may be reason to focus on an overarching linkage (and the associated key question) in an analytic framework rather than the indirect linkages (and their associated key questions). All key questions are reviewed and approved by the Task Force leads and AHRQ Medical Officer in the process of assessing and refining the topic before the detailed literature review is conducted. Input is also obtained through public comment and from the full Task Force (for complex or new topics). Key questions addressed in a systematic review are listed in the Methods section and used to organize the results in the final review. Key questions are addressed using up-to-date systematic review methods, under the current guidance and methods of the Task Force. Each key question is addressed through a distinct literature search, if necessary, and reported separately in the Results section of the review.
Contextual questions represent issues in a review for which the Task Force needs a valid but not necessarily systematic summary of current research in order to provide the context for its deliberation and recommendation statement. Contextual questions may address a range of different types of informational needs, including: 1) updated information for a key question that is not being systematically updated; 2) contextual information on natural history, current practice, prevalence and risk groups, or other aspects of the service which are part of the Task Force's considerations (e.g., screening interval, ages when screening should be stopped, newer technologies for screening and/or intervention); or 3) published modeling studies (when the Task Force has decided not to formally commission a modeling study). When formulating a work plan, issues in the background and introduction may emerge as candidates for formal contextual questions when the Task Force requires detailed and representative information to inform its consideration of the systematically reviewed evidence.
Although contextual questions are not necessarily addressed systematically, the approach taken may meet criteria for a systematic review. Comprehensive literature searches are not generally undertaken specifically to answer these questions. Information for contextual questions is gathered in a variety of ways: 1) through targeted literature searches, 2) from authoritative surveys or published reviews, 3) from expert input, and 4) opportunistically, while reviewing comprehensive literature searches for key questions. Contextual questions are not listed as separate questions in the Methods section of the report and are not reported in the Results section. The information resulting from the contextual questions is typically included as part of the Introduction or Discussion sections, and related as appropriate to the results of the systematic review.
3.3 Previous Task Force Review and Recommendations
To ensure that the current work plan builds coherently upon the Task Force's previous work on the topic, this part of the work plan succinctly summarizes the conceptual clinical framework and evidence foundation built by any previous USPSTF reviews and recommendation statements on the topic. The current Task Force recommendations are listed here verbatim, along with the analytic framework, key questions, summary of evidence table, main findings, and conclusions from the previous review. Methodological or scope limitations and evidence gaps identified in the previous review are also listed.
3.4 Search for New Synthesized Evidence/Pending Studies
At the work plan development stage, the EPC librarian works with the review team to develop a strategy for searching the literature to identify existing systematic reviews and other high-quality synthesized literature (such as meta-analyses). This is the first systematic search that will be incorporated into the overall searching done by the EPC for the topic. The purpose is to locate existing synthesized evidence that should be incorporated or built upon in the current systematic review, and the current methods emphasize finding all relevant synthesized evidence.
This synthesized evidence also provides background information that informs the approach to the topic and development of key questions. Background information that is typically collected from the synthesized evidence includes etiology and natural history, risk factors, screening strategies, interventions, current clinical practice, and prevalence and burden of disease/illness for the condition and for important subpopulations. Additional background information is the definition of "burden of suffering" of the condition in question. This burden is the ultimate target of implementing the preventive service. Evidence relevant to the burden of suffering, including the prevalence of the condition in various populations and the impact of the condition on the health of these populations (including societal or caregiver populations when relevant), is critical context for considering the potential population-level benefit of any clinical preventive service. The severity of the condition as measured by such metrics as prevalence and severity (e.g., number of life-years and quality-adjusted life-years lost in a population) is an important aspect of the burden of suffering. The burden of suffering of a condition defines the maximum possible benefit from prevention of that condition. The Task Force is also aware that implementation of various screening strategies can affect estimates of the burden of the disease, even in the absence of effective strategies, through lead, length, stage shift, and detection bias.
The following databases and Web sites are usually searched: Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Health Technology Assessment Database (United Kingdom), National Institute for Health and Care Excellence (United Kingdom), Institute of Medicine, PubMed® (using the systematic review search engine developed by the National Library of Medicine), and when appropriate, subject-specific databases (e.g., PsycINFO®). Searches are limited to literature published approximately 12 months prior to the last search of the previous review to the present.
In order to identify ongoing studies that could affect review scope and/or planning, the EPC librarian and/or topic team searches ClinicalTrials.gov, Current Controlled Trials (www.controlled-trials.com), Australian New Zealand Clinical Trials Registry (www.anzctr.org.au), and the World Health Organization's International Clinical Trials Registry (www.who.int/ictrp/).
The EPC review team also checks to determine whether there is a finished, in-process, or planned Community Preventive Services Task Force review for the clinical preventive service being reviewed. The timing of this search (work plan stage or later) is left to the discretion of the topic team.
3.5 Current Task Force Review Approach
3.5.1 Inclusion/Exclusion Criteria (Admissible Evidence)
The EPC review team, in consultation with the Task Force leads, clearly documents the criteria by which it will include evidence on a given key question. Such criteria might include study design (RCTs, cohort studies), setting, sample size, population studied, language(s) of publication, and year(s) of publication.
No generic criteria for admissible evidence have been established. Rather, the criteria are determined on a topic and key question basis, depending on the questions and the quality of the most applicable evidence anticipated being available. The goal is to identify the highest-quality evidence relevant to making an accurate determination of benefits and harms of delivering a preventive health service by primary care providers to persons living in the United States. All inclusion/exclusion criteria are posted for public comment, revised by the EPC, and approved by the Task Force leads.
One variable in the inclusion/exclusion criteria relates to the timeframe of the literature search. For a review to update a previous recommendation from the Task Force, the EPC review team establishes whether the key questions they are posing had been similarly addressed in the previous review. If they were addressed, the team may evaluate key studies previously reviewed, but would not systematically re-review the same literature. An exception to this would be if the Task Force decided to evaluate the validity of this evidence by a method different from that used in the previous review. If a key question has changed, or if the threshold for adequate evidence has changed, the team searches back in time for evidence available before the search period covered by the previous review. If the EPC does not systematically re-review the evidence from a previous USPSTF review, it will synthesize and incorporate the results of the previous review into the current review in order to allow a comprehensive consideration of the evidence for a topic.
In addition, the EPC review team searches for other systematic reviews on the topic. If another systematic review is found that is rigorous and addresses the same key question, the topic team may choose to incorporate that review as appropriate rather than redoing all of the work already represented in a good-quality, existing systematic review.
3.5.2 Use of Topic Experts
By design, EPC review teams consist of generalist clinicians, researchers, methodologists, and staff with various levels of content expertise. When appropriate, the EPC review team engages content experts and specialists as consultants or co-investigators to advise about work plan formulation and operational decisions made during the conduct of the review. To allow continuity with the previous Task Force review, the EPC review team may intentionally engage previous review team members as consultants or members of the current review team. Conflict of interest considerations are taken into account when engaging all content experts and specialists.
3.6 Peer Review of Work Plan
The work plan for full systematic reviews for new topics is usually peer reviewed before it is finalized. Four to six peer reviewers are chosen to provide content expertise, specialty perspective, topical research experience, and relevant methodological or policy expertise as appropriate to the topic. Peer reviewer lists are drafted by the EPC review team and amended and approved by the AHRQ Medical Officer. The EPC review team coordinates the peer review process (by telephone interview or through written communication) and incorporates peer reviewers' suggestions into the draft work plan. Peer reviewers' comments are not formally summarized. Instead, peer-reviewed work plans, the list of peer reviewers, and a synopsis of their comments and the resulting revisions are presented for final input and approval by the Task Force as a whole.
3.7 Public Review of Research Plan
Based on the full draft work plan, a draft research plan that contains the analytic framework, key questions, and inclusion/exclusion criteria is created for public comment (go to Section 9 for more detail on public comment processes). After approval by the Task Force leads, this document is posted on the Task Force Web site for 4 weeks to allow public comment and input on the research plan. All results from the comment period are provided verbatim to the topic team. The EPC review team summarizes major themes and makes suggested revisions based on these comments.
3.8 Task Force Approval of Final Research Plan
After the EPC review team incorporates revisions into the research plan, it is presented for final input and approval by the Task Force leads. The final approved research plan, including a section on Response to Public Comment, is posted on the Task Force Web site