Initial Publication Date: October 26, 2016

Measure Progress

How will you tell if you're reaching your goals? There are many frameworks for program evaluation. The model presented here is based on Marbach-Ad et. al. (2015). It includes the following five levels of program evaluation: Participation, Satisfaction, Learning, Application, and Impact (Adapted from: Colbeck, 2003; Guskey, 2000; Kirkpatrick, 1994, and Connolly, et al., 2006). In general, these levels progress from the easiest evaluation/most immediate feedback to more resource-intensive evaluation/longer-term feedback on how the center is achieving its vision and mission. For example, workshop participation could be garnered by collecting demographic data. Satisfaction could be assessed through a brief survey distributed at the end of the workshop. Learning could be measured by interview, survey, or focus group methods after the fact. Classroom observations could be conducted to assess how much attendees applied what they learned. To evaluate the impact on students, one could examine whether or not students of workshop participants showed more meaningful learning compared to students of instructors who did not attend the workshop. If time or resources aren't available for the full sequence, a subset of the measurements can still be valuable.

Evaluation and Assessment - Gili Marbach-Ad, University of Maryland-College Park
The CMNS Teaching and Learning Center lays out how they use this five-layer model for evaluating their professional development programs to improve teaching and learning.

Participation

What is it?

One of the most straightforward metrics to track is the number and demographics of the people who attend a given event.

How do you measure it?

Here are some ways participation data can be collected.

Track applications and attendance of events, including demographics and characteristics (e.g. department, professional title, etc.). You may use this to determine the characteristics of the audience that is attending, so that efforts can be made to recruit a more diverse audience for future events.
Create and track membership of an email discussion list.
Use surveys to gauge how participants learned about your program and reason for participation.

Data from these and other sources can be accumulated in a database for easier analysis.

Satisfaction

What is it?

This level refers to the extent to which attendees and facilitators were satisfied with their experience in the activity provided by the center. These data can be used to increase participation in future events.

How do you measure it?

Here are some ways satisfaction data can be collected and measured:

Conduct a survey among activity participants that asks about their program satisfaction. Ideally, distribute the survey as part of the program in order to maximize sample size, rather than waiting until a later date. Ask questions like:

Would you recommend this program to a colleague (Yes/No) Why?
Which components did you find most helpful/least helpful?

Conduct individual interviews or focus groups with participants after the event.
Facilitators can discuss among themselves whether the program met the desired goals.

Learning

What is it?

Beyond satisfaction with the experience, determine if participants achieved the goals and objectives of the program (e.g., "Participant will be able to identify evidence-based practices").

How can you measure it?

Learning can be measured by conducting pre/post-surveys, interviews, and focus groups regarding the content that was covered. You can probe to:

Participant beliefs about importance of material (e.g., "Do you think a flipped classroom is a valuable model?")
Knowledge, skills, habits of mind (e.g., Explain what you know about a flipped classroom).
Expertise, values, agency

Application

What is it?

Application refers to the degree to which participants apply the knowledge gained from the program in their classrooms or workshops.

How can you measure it?

Observe a class taught by program attendees. The persistence of application could be assessed through a series of longitudinal observations. Here are recommended protocols for classroom observation:

Classroom Observation Protocol for Undergraduate STEM (COPUS)
Reformed Teaching Observation Protocol (RTOP)

Impact

What is it?

Has the center delivered on its large-scale goals and objectives? For example, do students have better learning outcomes due to interventions that targeted faculty?

How can you measure it?

Impact can be measured in a variety of ways, including the following:

Student pre- and post-evaluation on:

Content knowledge (e.g., concept inventories)
Attitudes (e.g., CLASS survey).
Classroom climate (e.g., The Learning Climate Questionnaire (LCQ) developed at Purdue University)
Motivation to stay in the major

Student retention and attrition (e.g., D/F/W, persistence in the major, career aspirations, time to graduation).

One tool to track groups and shifts in student pathways is the Ribbon Tool from Tools for Evidence-based Action (TEA).

Track shifts in faculty or departmental culture (e.g., Are there shifts in faculty awareness, satisfaction and attitudes toward evidence-based teaching?)