Assessment

Jump Down Links: Flexible Modular Curricular Materials | Community Building and Pedagogical Orientation | References

Evaluation will occur throughout the project with formative and summative assessments that will measure immediate, medium-term and long-term changes. Internal and external evaluations as well as a research program will work together to build a picture of the project impact overall and address the project's three main goals.

1. Develop a suite of at least 30 flexible modular curricular materials, using large publicly available online datasets, that contribute to improved student quantitative reasoning.
2. Develop a community of faculty members engaged with materials and professional development designed to foster pedagogical orientation favoring open inquiry with large datasets.
3. Determine what mechanisms contribute to shifts in instructors' pedagogical orientation towards inquiry-based teaching.

Flexible Modular Curricular Materials

Community needs

In support of the project creating value by building a shared vision (Wenger et al., 2011), a qualitative study in the first year will investigate "How does the broader community characterize the needs and barriers to adopting curriculum that promotes inquiry with large data sets?" Assessment will begin with a series of interviews to test the protocol and to inform the "barriers and solution" workshop. Following the workshop, another round of interviews with select workshop participants and representatives from other data-rich curriculum groups will be conducted. The results will contribute to the body of knowledge about the needs, barriers, and solutions for faculty adoption of curriculum that promotes inquiry with large data sets. The findings will also ensure that the project aligns to the needs of the community it seeks to reach.

Student Learning Gains

This project aims to generate flexible modules and statistical vignettes to improve student quantitative reasoning. Each resource will be evaluated for their effectiveness in large general education classes with lab sections and student learning gains will be assessed when resources are used in combination or independently. An array of assessment approaches will be used including multivariate instruments that will be administered pre- and post-resource use. Furthermore, an initial pre-treatment baseline assessment during the first year will be conducted followed by post-use assessments.

Community Building and Pedagogical Orientation

Assessment programs will occur throughout the project to determine if 1) project goals are being achieved and 2) the extent in which professional development strategies promote the adoption of curricular materials. Evaluation will be both formative and summative, using mixed, methodology with a combination of surveys, focus groups, and interviews.

Overall Project

The overall project evaluation centers on the question:

To what extent do the various program components lead to achieving project goals?

Initial evaluation will address the extent to which project leadership establishes a robust project plan with clear roles and responsibilities, a plan to collaborate in planning and implementation of activities, and alignment to research-grounded components (Henderson et al., 2011). An external evaluation will also occur and center on overall project monitoring and will use a Causal Link Monitoring model (Britt et al., 2017). Internal evaluation will center at the relationship of the various program components to achieve the desired outcomes.

Professional Development

The professional development evaluation seeks to answer:

To what extent do professional development strategies promote the adoption of curricular materials that support open inquiry with large datasets?

The evaluation will investigate alignment of professional development programmatic activities to the goals of the project and to the needs and barriers identified in the first year. Then, the evaluation will examine the ways that participants attribute short-term and medium-term effects to the professional development experience.

References

Britt, H., R. Hummelbrunner, and J.Greene, 2017. Causal Link Monitoring. Retrieved from: http://www.betterevaluation.org/resources/overview/Causal_Link_Monitoring.

Henderson, C., A. Beach, and N. Finkelstein, 2011. Facilitating Change in Undergraduate STEM Instructional Practices: An Analytic Review of the Literature. Journal of Research in Science Teaching. 48(8): 952–984

Wenger, E., B. Trayner, and M. de Laat, 2011. Promoting and assessing value creation in communities and networks: a conceptual framework. Rapport 18, Ruud de Moor Centrum, Open University of the Netherlands. 56 pp.