Developing and implementing institution data dashboards to drive course transformation
Shanna Shaked, UCLA
Program Description
We are working with institutional research staff, deans and other interested faculty to develop and implement institutional data dashboards that can be used to foster conversations and interventions to improve teaching.
Program Purpose
Many students are failing introductory STEM courses, with relatively large achievement gaps occurring in these courses.
We are using the institutional data dashboard to identify the courses with the largest fail rates and/or achievement gaps and then work with the faculty to identify ways to improve these outcomes.
Program Goals
Our general goal is to increase STEM major retention for all students, and particularly for students from traditionally underrepresented racial minority groups (URM).
Program Activities
1. Work with institutional research staff to identify the type of data available and develop preliminary dashboards. In our case, these displayed data on course grade distributions and comparisons among various demographic groups.
2. Use these dashboards with key leaders and faculty to identify ways to improve the dashboard and add elements that make it useful and motivating to faculty.
3. Identify courses of concern and start working with course faculty and departmental leaders to identify ways to improve outcomes and/or better integrate existing programs.
4. Implement interventions and then use more rigorous data analysis to identify effect of interventions.
5. Repeat as necessary.
We fostered a positive relationship with institutional research folks, so that they could create such a dashboard and respond to requests. They purchased a site license for Tableau based on this work.
Notes and Tips
Don't:
* Use the dashboard to attack individual faculty.
* Show any non-FERPA compliant data (i.e. remove all bars that involve small numbers of students)
* Draw large or confident conclusions from data, especially if not merited.
Do:
* Use the dashboard to motivate leaders and faculty positively by looking at the challenge posed in their courses. Offer support in tackling this challenge.
Evidence of Success
Using the dashboard, we identified introductory physics as one with relatively high differences in fail rates between different populations. So we were able to target consultations, active learning and learning assistant interventions on this series. This led to a series-wide revision in textbook, lab, and pedagogy, with preliminarily positive results in terms of conceptual normalized learning gains.
Future Work
We are not seeing a large decrease in achievement gaps associated with these interventions, so we are exploring ways to make the interventions more inclusive and fostering a greater sense of belonging in students.
We are also finding that these dashboard data are easily misinterpreted and misused, and we therefore need to create more controlled guidelines for use and distribution.