NOAA B-WET Evaluation
The NOAA B-WET program strives for rigorous evaluation. Project-level evaluations have been supported as part of required grant activities for many years in order to help grantees document results and improve their activities. While project-level evaluation will continue to be supported, beginning in 2013 B-WET will begin rolling out an across-region, ongoing evaluation system to monitor program implementation and outcomes at the national level. Please see below for more information about these ongoing efforts.
B-WET has created a cross-region, internal evaluation system to monitor program implementation and outcomes on an ongoing basis. Results of this evaluation will be used to make adjustments to B-WET Federal Funding Opportunities (FFOs) and activities in order to improve the program. Once baseline data are available, B-WET staff will also be able to use these data to develop informed targets to serve as indicators of the program’s success. Ultimately, grantees will have access to reports aggregating their participating teachers’ responses to help inform their program development. This effort will complement the excellent project-level evaluation work conducted by B-WET grantees.
The proposed evaluation system is designed primarily to answer the following four overarching questions about B-WET’s grant program:
MWEE Implementation Questions
- 1. To what extent do regional B-WET programs support grantees in implementing Meaningful Watershed Educational Experiences (MWEEs)?
- 2. How are MWEEs implemented by grantees and teachers?
MWEE Outcome Questions
- 3. To what extent do B-WET funded projects increase teachers’ knowledge of watershed science concepts, their confidence in their ability to integrate MWEEs into their teaching practices, and the likelihood that they will implement high quality MWEEs?
- 4. To what extent do B-WET funded projects increase students’ knowledge of watershed concepts, attitudes toward watersheds, inquiry and stewardship skills, and aspirations towards protecting watersheds?
As part of this evaluation system, recipients of B-WET grants and teacher-participants in grantees’ professional development programs may be asked to voluntarily complete online questionnaires to provide evaluation data. One individual from each grantee organization will be asked to complete a questionnaire once per year of the award. For projects that work with teachers, the teacher-participants will be asked (using email addresses provided by the grantee organization) to complete one questionnaire at the close of their professional development and one after implementing Meaningful Watershed Educational Experiences with their students (before the end of the following school year). Grantees should be able to complete their questionnaire within 30-60 minutes (depending on the nature of their program) and teachers, within 30 minutes. B-WET grantees and teachers who respond to the questionnaires will remain anonymous to B-WET and NOAA.
This data collection will be conducted in a manner consistent with OMB guidelines (OMB Control No 0648-0658).
Please check back often for new resources to help participants respond to the national evaluation effort.
B-WET National Evaluation Plan (PDF) - All of the sub-documents below
- B-WET Watershed Education Literature Review (PDF) - (DOC)
- Evaluation Questionnaires:
- Presentation slides (PDF)
Student Data Collection (not collected as part of the national evaluation system)
The California B-WET Program has developed a comprehensive set of guidance documents to assist grantees in evaluating their projects in accordance with NOAA education evaluation standards. B-WET applicants and grantees are encouraged to use this information to develop evaluation strategies for their individual projects.
Education Project Evaluation from California B-WET
Other useful evaluation resources include:
MEERA-My Environmental Education Evaluation Resource Assistant
FAQs FOR ALL GRANTEES1.) Am I required to participate in the evaluation system?
No, participation is optional for all B-WET grantees. However we strongly encourage grantees to participate, if possible. This evaluation system is intended to collect data over the long term, so your participation this year will provide invaluable feedback for how it can be improved in the future. We anticipate that this evaluation system will have short-term and long-term benefits for programs, funders, and the field of environmental education broadly. The results will provide insight into how we can improve the NOAA B-WET program and tailor our services to you and the teachers and students you serve. Your input is valued and will contribute to a better B-WET program in the future, so please consider participating to the extent possible for your grant.
2.) What will I be asked to do?
If you have a Teacher Professional Development or Exemplary grant, we will ask you to fill out an online teacher contact information form to provide contact info for the teachers who participate in your professional development. If you will have multiple cohorts of teachers, you may complete this survey as many times as needed over the course of your grant. You will be asked for your contact information and participating teachers’ email addresses. No other teacher contact information is requested. In addition, all grantees (regardless of project type) will be asked to complete an online survey at the end of your grant period (or once per year if your grant is multiple years). The teachers will be asked to complete an online survey following their professional development, and another at the end of the following school year.
3.) Can I change or add other contacts from my organization to the evaluation system?
Due to the nature of the evaluation system, it is not possible to maintain multiple contacts for each organization. Grantees should identify one primary point of contact for the evaluation. That point of contact may always forward our email request to other partners so they can respond to the survey. If the person who has been receiving the evaluation system emails is not the appropriate point of contact, please provide updated contact information to the B-WET National Coordinator: Bronwen Rice, 202-482-6797, E-mail: Bronwen Rice. Please include your grant # in any information change requests.
4.) What type of questions are on the grantee survey and the two teacher surveys?
The survey grantees complete at the end of the grant year asks questions about grantee satisfaction with the B-WET program and grantee’s needs for additional support, the project implementation strategies used by grantees (e.g., length, location, goals), the type of content addressed in the grantee’s project, instructional practices used during the grantee’s project, the grantee’s perceptions of teacher and student outcomes, and the evaluation practices used for the grantee’s project.
The survey completed by teachers at the end of their B-WET-funded professional development asks questions that measure teachers’ perceptions of the way the professional development was implemented, changes in teachers’ knowledge of watershed concepts, changes in teachers’ confidence toward MWEE instruction, and teachers’ intentions to change their instructional practice.
Finally, the survey teachers will complete at the end of the school year following their professional development asks them about the strategies teachers used to implement MWEEs with their students (e.g., length, location, goals), the content addressed during the MWEEs, the instructional practices teachers used during MWEEs, and the teacher’s perceptions of student outcomes.
5.) I don’t provide formal professional development for teachers, only student activities. Can I still participate in this project?
Yes. The grantee survey is for all grantees working on MWEE implementation. However, we will not need you to complete the teacher contact information survey and will not collect data from teachers associated with your project. We are not currently collecting data at the student level as part of the national evaluation system, though those data are valuable and should be reported as part of your regular grant reports, as applicable. We do have a Student Item Bank with guidance that is completely optional and is available as a resource to you and your teacher participants. The Student Item Bank can be found on the resources tab of the B-WET Evaluation web page.
6.) Is my grant project being evaluated for performance as part of this evaluation system?
No. NOAA/B-WET will not review the data at the individual grant project level, but will only look at it regionally or nationally. The intent of the evaluation system is to review implementation practices and outcomes for the program as a whole, not performance of any single grant project, grantee, or teacher.
7.) Can I review the questionnaires ahead of time?
All of the questionnaires, as well as the entire evaluation plan, are available for your review at any time on the resources tab of the B-WET Evaluation web page
8.) Does this evaluation replace the project evaluation I proposed as part of my grant?
No, this request is separate and if you choose to participate in the evaluation system, you will still be responsible for completing the evaluation you proposed as part of your grant project. While you will have access to teachers’ results for your project, because this system is looking at common elements across the 7-region national B-WET program, it may not provide the project-specific information you need to answer all your questions about your project. In future years we hope to provide applicants with more information during the application period so that they may better incorporate these two levels of evaluation into their projects.
9.) I missed the overview webinar. Is there any place I can review this information?
The presentation slides and an archived version of the webinar are available for your review on the resources tab of the B-WET Evaluation web page.
10.) Whom do I contact for more information?
FOR PROJECTS WITH TEACHER AUDIENCES:1.) What information will you request about my grant project teachers?
We will request teachers’ email addresses, and no other contact information. We will not ask for their names. NOAA will not share the email addresses or use them for any other purpose. We will also ask you for the expected start and end dates of your grant project activities.
2.) What if I don’t have my teacher participants identified when I receive the request for their contact information?
The online teacher contact information form can be completed at any point, and can be submitted multiple times by the same person. Grantees will receive a request with the link to the form monthly, and are asked to complete it each time they have a new cohort of teachers identified. If you have no new information to provide you may disregard these monthly reminders.
3.) How do I define a cohort of teachers?
A cohort of teachers may be defined as appropriate for your project. A single cohort should all participate in activities associated with a single grant award, and those activities should be completed around the same time. For example, if you work with one group for a summer workshop only, and another for the summer workshop and follow-up school year training, it would make sense to define these as two cohorts. The first would receive their survey following the workshop, the second following completion of all of the follow-up activities. Note that the grantee is required to provide the anticipated end date for PD with a given cohort.
4.) How do I determine the "end date" of my professional development?
Grantees will be required to provide the anticipated end date for the entire professional development (PD) activity for each cohort of teachers. For the purposes of this evaluation system, we define the end date of the PD as when all of your interactions with that group of teachers are complete. For example, if you offer an intensive summer workshop, but then also provide in-classroom support throughout the school year, the end date should be the end of the school year, once all of your support activities are likely to be complete. It is OK to estimate this date, but your thoughtful consideration of this will help ensure that we are able to collect information about your entire program intervention.
Note that all of these are considered to be B-WET professional development that you might be providing to teachers. The "end date" is when all of your professional development ends for this cohort of teachers.
• One day workshops (usually less than 8 hours)
• Teacher institute (usually on consecutive days that cumulatively consist of 40 hours or more
• Multi-day workshops (events that last at least 6 hours, but are less than 40 hours, e.g., a three-day workshop on a specific topic or a series of five Saturday sessions)
• A college-level course
• Professional development provider training (training for individuals who provide teacher professional development)
• Individual teacher coaching and support (e.g., curriculum planning, shared teaching, demonstrations and/or other forms of in-school or in-field support)
• On-line professional development support (e.g., courses, webinars, discussion forums)
5.) What if I work with the same group of teachers over several years?
If you work with the same cohort of teachers over several years of a multi-year grant, please list the end date of the entire PD interaction, even if it is several years in the future. You will receive the grantee survey yearly, but teachers should only receive the PD survey at the end of their entire period of work with your organization.
The only exception to this guidance is if you are working on a project that, while it may be planned as a multiple year project, has only been awarded funding for a specific period. Grantees should only plan for the period of their project that has been awarded and set PD end dates accordingly. If a project continuation is granted and you plan to continue working with the same cohort(s) of teachers, please contact the B-WET national coordinator or your regional B-WET program manager to discuss how to update the PD end dates for any contacts you have already provided. New cohorts may always be submitted through the Teacher Contact Info Survey..
6.) May I contact my teacher participants myself to notify them about this evaluation system?
Yes, and you are encouraged to do so. There is sample notification text provided under the resources tab of the B-WET Evaluation web page. NOAA will also provide this information when we contact teachers directly, but if you want to contact them ahead of time, you may. This is also an opportunity for teachers to opt out prior to you submitting their contact information to us.
7.) What if my teacher participants don’t want to participate?
The will be the option to opt out of the project at any time. NOAA will not share their contact information (email address) or use it for any other purpose.
8.) How can I monitor my teachers’ participation if the survey is anonymous?
Teachers will receive a thank you email after they complete questionnaires. You may ask the teachers to send that thank you email to you as proof of their participation. If you are able to provide an incentive, such as credit or a stipend for participating in your grant project, then you could include their participation in this data collection as one of the incentive requirements. Another option would be to offer a thank you “gift” (small token, inclusion in a raffle, etc.) for proof of participation. These incentives will help B-WET achieve the response rates we need and we strongly encourage grantees to integrate participation in this evaluation system into their formal project requirements for teachers whenever possible.
9.) Some of my teacher participants have email addresses that include their names. How will their anonymity/identities be protected?
Only the evaluation system administrator will have access to the teachers’ email addresses when they are submitted, and individual teacher responses will not be associated with their email addresses. Our policy is to protect teachers identities, so those emails will never be used for any other purpose than this data collection.
10.) How will I know when my teachers participate, and how do I get access to the results for my project?
You will receive an individualized, password-protected link when you are contacted by NOAA to participate in the project and resent to you monthly in a reminder email. You may visit this link at any time to review a report of data associated with your project. This report will update automatically as teachers participate. You will not be able to identify teachers individually, but will only see the aggregate results for your project.
11.) Will I have access to the raw data for my project?
It may be possible to access the anonymous raw data for your grant project (beyond the report you will receive). If you need access to these data, please contact B-WET program staff to inquire about this.
12.) Do I need to go through my Institutional Review Board (IRB) approval process for this data collection?
Maybe. If you went through an IRB process for your grant project, we recommend contacting your IRB office to see if you need to get any additional clearance to participate in this evaluation system. If you would like to publish results in a peer-reviewed journal, then it is very likely that you will need to get formal IRB approval. Note that this data collection plan has gone through the federal Office of Management and Budget (OMB) Paperwork Reduction Act (PRA) process (OMB Control Number: 0648-0658 Expires: 02/29/2016). If you need to pursue IRB approval, B-WET program staff can provide more information about the project to help facilitate this.
13.) Can I link to your survey from my own? Is there a way I can include your survey questions in my own evaluation?
No. The evaluation system survey is set up so that teachers must receive individual links generated by the data collection system, so we are unable to provide a single link for you to distribute to all of your participants.
14.) How do I update teacher contacts or professional development dates that I’ve already submitted?
If the professional development end date or other information changes after submission, you should contact Bronwen Rice, B-WET National Coordinator (Bronwen.Rice@noaa.gov, 202.482.6797) with updates. Please include your grant # in any information change requests. The B-WET National Coordinator will need to work with the grantee to make the changes in the system. For this reason, we'd ask that we only make edits if there are significant changes. If the PD end date shifts by only a few days or even a week or two it isn't generally an issue, since we send teacher emails out on a monthly basis. If teacher participants drop they will always have the opportunity to opt-out of the emails, and to add new teachers you can just complete the survey again..
FOR PROJECTS WITH STUDENT AUDIENCES:1.) How should I use the Student Item Bank?
We are have developed guidance on the use of the student item bank. For the time being its use is completely optional and is available only as a resource to you and your teacher participants. These data will not be collected as part of the national evaluation system, though they are valuable and should be reported as part of your regular grant reports, as applicable. The Student Item Bank can be found on the resources tab of the B-WET Evaluation web page.