Intended Audience
This guide is intended for program developers who are interested in submitting evidence-based cancer control programs for review and potential inclusion on the National Cancer Institute (NCI) Evidence-Based Cancer Control Programs (EBCCP) website.
Objectives
This guide provides a brief introduction to EBCCP, describes how to submit an evidence-based program to EBCCP, and explains the steps involved in an EBCCP review. This document will help you, as a program developer, to understand:
- The purpose of EBCCP
- The benefits of having your program included in EBCCP
- The criteria EBCCP will use to determine whether your program is eligible for review
- The steps involved in submitting your program for potential review
- The documentation you will need to submit for the review, if accepted
- Your responsibilities and contributions to the review process, as well as those of the EBCCP Development Team, the staff who will coordinate and conduct your review
- The end product of your review (posting on the EBCCP website)
About EBCCP
The EBCCP website is a searchable database of cancer control programs. It provides program planners and public health practitioners easy and immediate access to information about the programs and their materials that can be used to implement these programs in the populations and settings in which they were originally tested. EBCCP is a voluntary system in which program developers may elect to participate. It can also be accessed directly at https://ebccp.cancercontrol.cancer.gov/.
Benefits of Participating in an EBCCP Review
EBCCP reviews do require a commitment of time on the part of the program developer to prepare for submission and participate in the review process. However, the effort invested can return a number of benefits:
- Dissemination and expansion. Participating in a review offers you an opportunity to disseminate the products of your work through the EBCCP website while providing increased visibility and credibility for evidence-based approaches to cancer control. While your program materials may not have been developed with dissemination in mind, they can be disseminated directly from the EBCCP website in a downloadable, easy-to-print format or through CDs requested by interested implementers.
- Guidance for continued program development and evaluation. Ratings on the criteria assessed during the review process can provide direction for the ongoing development of program materials and future evaluations.
Determining Your Program's Submission Readiness
Before you decide to submit an evidence-based program to EBCCP, you will need to ensure that it addresses one of the EBCCP program areas and meets the criteria for inclusion. These two important components of submission readiness are described in the next section.
EBCCP Program Areas
EBCCP accepts interventions in 14 program areas:
- Breast Cancer Screening
- Cervical Cancer Screening
- Colorectal Cancer Screening
- Diet/Nutrition
- Follow-up to Screening
- HPV Vaccination
- Informed Decision Making
- Obesity
- Physical Activity
- Prostate Cancer Screening
- Public Health Genomics
- Sun Safety
- Survivorship / Supportive Care
- Tobacco Control
Eligibility Criteria for Inclusion
Programs must meet the following criteria to be eligible for an EBCCP review:
- Outcome finding(s) must be published in a peer-reviewed journal.
- The study must have produced one or more positive behavioral and/or psychosocial outcomes (p≤.05) among individuals, communities, or populations.
- Evidence of these outcomes must be demonstrated in at least one study using an experimental or quasi-experimental design. Experimental designs require random assignment, a control or comparison group, and pre- and post- assessments. Quasi-experimental designs do not require random assignment but do require a comparison or control group and pre- and post- assessments. Studies that are based on single-group, pre-/post-test designs do not meet this requirement.
- The program must have messages, materials, and/or other components in English that can be disseminated in a U.S. community or clinical setting.
- The program must have been evaluated within the past 10 years.
If your program meets the above criteria, the EBCCP Development Team welcomes you to submit your program for consideration, following the steps described in the next section. If you have questions about submission or are uncertain about whether your program meets the above criteria, please go to the Contact Us page of the EBCCP website and submit your question or concern.
Submitting Your Program to EBCCP
There are three steps involved in submitting an evidence-based program to EBCCP:
- Registering the program,
- Preparing a submission package of materials, and
- Participating in a kick-off call.
Step 1: Program Registration
If you are interested in pursuing inclusion in EBCCP, your first step is to complete a Program Registration Form on the EBCCP website at https://ebccp.cancercontrol.cancer.gov/register/index.do. This form asks you to confirm whether your program meets each of the five eligibility criteria noted above; you will also provide your contact information.
Step 2: Initial Submission
If your program meets the eligibility criteria based on the registration form, the EBCCP Development Team will send you an email inviting you to submit materials so that an official determination can be made about your program's eligibility. The following materials are requested at this stage:
- One electronic copy of the publication(s) relating to the outcomes of the research project (e.g., article reprints and preprints, final grant reports, chapters, monographs). You must indicate which primary outcomes publication includes the research findings.
- One complete set of all program materials in the format typically disseminated to the public (electronic, hard copy, or via a web address).
- Documents that describe the original plan for selecting outcome measure(s) and data analysis (e.g., grant application narratives) and how the study purpose was presented to participants (e.g., informed consent forms, protocols).
Step 3: Kick-Off Call
The third and final step of the submission process is to participate in a kick-off call with the EBCCP Development Team. The kick-off call provides an opportunity for both parties to ask questions, discuss the EBCCP review process, and identify additional materials, if necessary, that will be needed before the program is officially accepted for EBCCP review.
Once it has been determined that both parties wish to proceed with the review, you will be sent an Initial Approval Form. This form is used to finalize important details pertaining to your review (e.g., confirmation of the studies and outcomes that will be reviewed, confirmation of the products to be disseminated, and authorization to initiate program review).
The EBCCP Review Process
After your program has been formally accepted for review, it is assigned to one of the Review Coordinators (RCs) on the EBCCP Development Team. The RCs are Ph.D.-level evaluation and behavioral/social science specialists who serve as a technical liaison between developers and the pair of expert reviewers assigned to each review. The expert reviewers are Ph.D.-level professionals who have expertise in one or more of the EBCCP program areas. The identity of the reviewers assigned to your review will be kept confidential to ensure the integrity of the review process.
Both the RC and the reviewers have a role in rating the program. The reviewers rate the program's Research Integrity, while the RC rates Intervention Impact, Dissemination Capability, and RE-AIM criteria. More information about the EBCCP Rating System is provided in the next section. To facilitate the reviewers' work, the RC provides the reviewers with a review packet that includes all the study articles and other materials to be reviewed; detailed notes, including any supplemental information you may provide to explain how the study or studies were conducted; and information on the rating anchors. The RC is also responsible for drafting the EBCCP summary for your program, which will be posted on the EBCCP website at the conclusion of the review.
The EBCCP Rating System
The program and its materials are evaluated in four areas for the EBCCP review. These rating areas are described below.
Research Integrity
Research Integrity reflects the overall confidence reviewers can place in the findings of a program's evaluation based on its scientific rigor. The Research Integrity rating system comprises 16 criteria scored by independent experts. Scores on each criterion are given on a 5-point scale ranging from low quality to high quality. The overall integrity score is an average of the 16 criteria reflecting the merits of the science that went into the program evaluation.
Intervention Impact
Intervention Impact describes whether, and to what degree, a program is usable and appropriate for widespread application and dissemination. This rating is determined by the RC. Population Reach and Effect Sizes are separately rated on a 5-point scale; these ratings are then combined using the EBCCP Intervention Impact rating table to determine the impact score.
Dissemination Capability
Dissemination Capability refers to the readiness of program materials for use by others as well as a program's capability to offer services and resources to facilitate dissemination. The rating is given on a 5-point scale ranging from low quality (1.0) to high quality (5.0). Dissemination capability is measured through the assessment of three areas:
- Quality of implementation materials
- Training and technical assistance protocols
- Availability of quality assurance materials to determine whether implementation was done with high fidelity to the original model
RE-AIM
RE-AIM is a five-step framework designed to enhance the quality, speed, and public health impact of efforts to translate research into practice:
- Reach your intended target population
- Effectiveness or efficacy
- Adoption by target staff, settings, or institutions
- Implementation consistency, costs, and adaptations made during delivery
- Maintenance of intervention effects in individuals and settings over time
A RE-AIM scoring instrument has been developed to assess the first four steps above. The instrument consists of 22 items within 4 dimensions: Reach (5 items), Effectiveness (3 items), Adoption (6 items), and Implementation (8 items). The Maintenance step is not currently assessed by EBCCP. Additional information about RE-AIM can be found at http://re-aim.org.
Program Materials
As noted in the eligibility criteria, EBCCP programs are required to have messages, materials, and/or other components, in English, available for dissemination to potential implementers in U.S. community or clinical settings. Before your review begins, the EBCCP Development Team will work with you to determine how these materials will be provided to EBCCP users. There are two dissemination options:
- Materials will be made available to the public through the EBCCP website, at no charge to you. NCI will provide EBCCP users with a CD of these materials upon request.
- You disseminate the materials to EBCCP users. If you choose this option, a snapshot of all required program materials will be shown on the Program Materials page of the EBCCP program summary. This allows users to preview the materials but does not provide the full documents online. Users are provided with your contact information to obtain cost and ordering information. If your program is web-based and currently disseminated to the public, a direct link to your website will be posted instead.
EBCCP programs are also required to have written implementation guidance explaining the steps required to implement the program. If you do not have an existing implementation guide, one will be created for your program using an established NCI template. The Implementation Guide will be developed by the EBCCP Development Team from information you provide, or you may draft one using the NCI template.
Summary Approval and Publication
At the conclusion of your review, you will receive a draft summary containing descriptive information about the program; information about the study reviewed, including key findings; and the final EBCCP ratings. An example of a completed EBCCP summary can be found at https://ebccp.cancercontrol.cancer.gov/programDetails.do?programId=892493
You will have the opportunity to request clarification on the ratings and suggest edits to the summary before providing your final consent to post the summary via the Final Approval Forms. NCI considers all edits by developers and may confer with the EBCCP Development Team to resolve any technical questions or concerns.
Conclusion
EBCCP is a way for you to share advances in implementation science and improve practices in cancer control. Although participating in a review does require some time and effort, the benefits of having your program included on the EBCCP website should prove to be worth the investment. Below is a flowchart to help illustrate the process.
