Program directors who use data to inform planning about their programs often make better decisions about what to change and about how to make systemic and sustainable changes. Programs committed to improvement must analyze existing data and collect and analyze additional data in order to understand program status in these areas:

• How well the current program is meeting the proposed program goals;
• How the program and community are changing;
• Root causes of problems or issues;
• The current and future needs of the program, students, staff, parents and the community;
• The impact of efforts, processes and progress; and
• Areas in need of improvement.

Of all the factors that affect student learning, program processes are the only measures that program directors, site coordinators and staff have direct control over. We do not control where students come from, their home life or why they think as they do. But we do have control over how we interact with students and families; how programs are implemented; and how we respond to changes, challenges and successes.

If an activity is primarily teacher directed, you may see students doing these things:

• Paying attention (tracking the teacher or teaching prop with their eyes);
• Taking notes;
• Listening as opposed to chatting
• Asking questions;
• Responding to questions; and/or
• Reacting with laughs, head shaking and audible words.

If an activity is primarily student directed, you may see students doing these things:

• Working independently or in small groups;
• Performing/presenting;
• Inquiring;
• Exploring;
• Explaining;
• Experimenting; and/or
• Moving and talking to each other.

Two strategies can help.

a. Be sure that everyone who will observe and rate observations has training on use of your observation tool and protocols. It is important that each observer understands your program implementation plan and knows what components they should see in the classroom. It is also important that you talk and agree on what would score a top rating and what would score a low rating. Require that raters provide documented illustrations for why they give a high or low rating.

b. Rather than trying to score activity components while in the classroom, the observer should just write down everything they hear and see (whether good or bad). They should later then use their notes to determine the extent to which components were implemented in fidelity with the plan and provide supporting evidence from their notes.

For example, the observer notes that every student is actively engaging in the activity, but the teacher is not moving around the room to check in with students.

The observer might determine that students are developing a sense of competence, as they appear to feel confident about working independently; this is a positive indicator. Yet, delivery might score lower because the teacher is not taking the opportunity to use guiding questions to move students to even higher levels of thinking, which is part of your fidelity plan.

Ensuring that you adhere to the same design as the creator planned is critical to achieving the same results that were achieved during the research process. If you change the intended plan for implementation, you can no longer estimate what the effects on students will be. Curriculum designers have “practiced” with components such as dosage and delivery, and provide recommendations based on their experience. For these reasons, it is important to design your implementation plan for research-based approaches based on what designers of that approach recommend.

Ensuring that you adhere to the same design as the creator planned is critical to achieving the same results that were achieved during the research process. If you change the intended plan for implementation, you can no longer estimate what the effects on students will be. Curriculum designers have “practiced” with components such as dosage and delivery, and provide recommendations based on their experience. For these reasons, it is important to design your implementation plan for research-based approaches based on what designers of that approach recommend.

There is no one specific tool that fits all programs. There are some Y4Y tools posted under “Tools” on the Click and Go 3 page that can be customized for your program needs. Also, several External Resource links on the Click and Go 3 page provide additional resources and guidance around fidelity. No predesigned tool will capture the unique nature of your program, so whether you are designing your own tool or using a predeveloped tool, be sure to add measurement criteria for the specific components in your program.

Each tool, observation checklist, student portfolio and rubric provides different perspectives for the various aspects of your program. Think about what you want to learn, who you want to focus on (students or staff), and how much time you have for scheduling, using the tool, and doing follow up. Different measures provide different information, therefore, multiple measures produce a more comprehensive overview of your program. Below you will find some pros and cons to each tool.

Observation Checklists (snapshot measure):

Pro: Provide the best opportunity to “see” what is happening in a program and to watch, listen and feel the interactions between facilitator(s) and students.

Con: Time consuming, may need to be conducted over multiple sessions to get the most comprehensive picture of what is happening.

Student Portfolio (measure over time):

Pro: Provides comprehensive documentation of an individual student’s progress over time.

Con: Can be inconsistent and takes time to plan and organize on the part of the facilitator. Often reflects only finished products and represents a student’s “best” work.

Rubric (inform process):

Pro: Clearly defines the benchmarks or skills that you want to measure with an explanation of what is expected in order to reach those benchmarks.

Con: Rubrics take time to develop, and everyone needs to agree to use them with fidelity for them to be effective.