Summary Results from our 4th Annual Report to the National Science Foundation

Evaluation of the fourth year of the SAGE 2YC project provided valuable insights into what's changing in the work of two cohorts of geoscience faculty Change Agents. Our theory of change is rooted in a professional development model that has two major goal domains of increasing evidence-based practices and building sustainable leadership. This model also stresses improving instructional practices, broadening participation of underrepresented students, and enhancing students' professional pathways. The model also purports that engaging faculty Change Agents in systematic reflection and inquiry on the changes they are making independently and in collaboration with others will promote continued improvement in 2YC geoscience education that leads to improved student outcomes. Faculty Change Agents are encouraged to share what they're learning with other faculty and staff in their colleges and in their regions. Through their engagement in a cycle of innovation, the Change Agents and their 2YCs learn about what changes in their own practice lead to more equitable student outcomes, and they network with other 2YC geoscience educators to scale these lessons to their colleges and other colleges and universities in their regions.

Overview: Major Findings

Efforts of the faculty Change Agents to date reveal a strong focus on course-level change in instructional practices in their own classrooms. These personal experiences lead to sharing about how to change instructional practices within their colleges as well as the regional workshops that they conduct and other professional development opportunities. In the Change Agents' work with college administrators formally engaged in the SAGE 2YC project, some Change Agents report support for making larger, systemic changes in instructional practices on their campuses.

Data gathered from various sources (e.g., instruments designed for the SAGE 2YC and personal interviews and focus groups with faculty Change Agents) show changes are being made in instructional practices that they believe will improve student success, and the Change Agents rely on these beliefs to guide their plans to make future changes. The faculty Change Agents understand that their continued learning and additional work is essential to continuing to improve the academic/learning outcomes of underrepresented students, reflecting increased appreciation for and understanding of disaggregated course-level outcomes data gathered since the beginning of the project. In their team action plans, many faculty Change Agents express an intent to continue integrating active learning, metacognition, and other student-centered instructional practices, stating that they see these strategies as important to improving success for all students and especially important to their efforts to improve student success for underrepresented student populations in the geosciences and STEM.

Student-Centered Classrooms

Still in process, data gathered by trained observers on the Reformed Teaching Observation Protocol (RTOP) so far has found 9 of 14 (64%) of cohort 1 faculty Change Agents observed in 2016 and again in 2018-19 showed higher RTOP scores at the second observation than the first. Notably, three faculty Change Agents moved from teacher-centered (RTOP scores of 0-29) to the transitional range (RTOP scores of 30-49), and two faculty Change Agents shifted two levels from the teacher-centered range to the student-centered range (RTOP scores of 50 or higher), skipping the transitional range entirely. These findings are important because they suggest advancements toward student-centered instructional practices in the majority of cohort 1 faculty Change agent classrooms.

The Educational Practices Inventory (EPI), designed by the project evaluation/research team and principal investigators specifically for SAGE 2YC, was administered to the faculty Change Agents near the beginning of their involvement in the project in either 2016 (cohort 1) or 2017 (cohort 2) and again in early 2019 for both cohorts. The instrument uses scaled items associated with strategies on success-for-all-students, teaching, classroom, information sharing, and student supports. By administering the EPI at two points in time as pre- and post-project assessment, we were able to gather rich program-level data to triangulate with other data points in the study. Two of these points follow:

• Over the course of the 4-year grant period, most faculty Change Agents have emphasized the two project goals of improving instructional practices and broadening participation more than facilitating students' professional pathways, and the EPI results corroborate this finding. For example, the EPI shows more limited implementation of strategies to inform students about geoscience careers, engage geoscience experts, and conduct other career-focused efforts than reformed instructional practices. This finding is consistent with the faculty Change Agent teams' action plans that consistently mention implementing instructional practices and less consistently mention enhancing students' professional pathways. Having said this, we find that some faculty Change Agent teams purposefully address SAGE 2YC's professional pathway goal with detailed plans that are central to current and future work (for more details, please see Table 3 in the 2019 Annual Report for SAGE 2YC (Acrobat (PDF) 4.6MB Sep24 19)).
• In 2019, the faculty Change Agents report that they continue to use lecture-led instructional practices but they also integrate lecture-led instruction with student-centered instructional strategies such as student-to-student interactions, inquiry, cooperative learning, active learning, as well as metacognition. These findings are corroborated by the above mentioned RTOP results revealing the majority of cohort 1 faculty Change Agents demonstrate positive change toward student-centered classroom instruction from the first observation near the start of the project and the second observation in year 4.

Leadership

Data gathered on the Bolman and Deal Leadership Frame instrument near the beginning and end of the project reveal changes in the ways in which the both cohorts of faculty Change Agents perceive their own leadership frames. For cohort 1, we see movement toward the identification of more than one frame in 2019 compared to 2016, and we also see some movement away from the structural frame in this shifting to other leadership frames, particularly to the human resources and symbolic frames. These results suggest that the faculty Change Agents are becoming more flexible in engaging in changing their own practice as well as in their work with others.

Similarly, we see some movement of cohort 2 faculty Change Agents from the structural frame to more multi-frame leadership, although as a group the faculty Change Agents in cohort 2 showed preference for a multi-frame leadership approach from the start. Results for cohort 2 faculty Change Agents are distributed more evenly across the four frames than the cohort 1 faculty Change Agents' results. These results may imply faculty Change Agents are shifting to multiple leadership frames and may also use these frames to motivate others to change. For example, adopting multiple leadership frames may facilitate the Change Agents' work with their college administrator to secure resources and execute action plans.

Spreading Change and Improvement in 2YC Geoscience Education

The faculty Change Agents' delivery of regional workshops has been an important component of the SAGE 2YC project from its inception. To date, 37 workshops, enrolling 484 participants, have occurred, with the most common focus being on the SAGE 2YC goal of enhancing success for all students. Numerous workshops have replicated aspects of the SAGE 2YC professional development model to encourage instructional practice in geoscience classrooms to be more student-centered as well as to be more inclusive and positively impactful for underrepresented students. Other aspects of the workshops focus on various aspects of improving geoscience programs, enhancing field experiences, and enhancing professional pathways in the geosciences.

Plans for the Future

Results from the annual workshops for the faculty Change Agents show that the Change Agents appreciate opportunities to convene and collaborate with each other, including sharing with other faculty how they are experiencing changes to their instructional practice, how they are improving the recruitment and engagement of underrepresented students, how they are encouraging other geoscience faculty in their colleges to change their instructional practice, and how they are engaging in other professional development activities to grow their networks on the regional and national levels. The faculty Change Agents also recognize and appreciate the involvement of campus administrators, and in turn, the administrators share insights into the value of the SAGE 2YC project from the 2YC institutional perspective. Many administrators acknowledge difficulties they experience in finding time to engage with faculty in as deep and meaningful way as the SAGE 2YC project provides. Without SAGE 2YC it is difficult to imagine how these productive relationships could have developed, but looking to the future, it is easy to see how important they will be to embedding and sustaining meaningful change.

All of the annual project evaluation reports for the SAGE 2YC project, including the 2019 report, are posted on the SAGE 2YC project evaluation page.