Institutional Change: Use Data »

Sustaining, Systematizing, Institutionalizing

Propelling inclusive excellence through thoughtful assessment

Ongoing, formative, and thoughtful assessment is critical to persistence efforts. Programs launched to broaden the pool of scientists and improve the quality of educational outcomes must incorporate evaluations that provide feedback about targeted outcomes.

Below we articulate metrics for measuring these outcomes as well as problems and future directions for even more effective assessments.

Common Metrics

There are a number of useful metrics used by Capstone Institutions to benchmark access and persistence efforts. Many of them focus on objective, well-accepted benchmarks, including:

• Performance in science classes (e.g., gateway science courses; overall GPA in a major)
• Participation in opportunities identified as hallmarks of high quality science education (e.g., participation in research opportunities)
• Persistence in science. For admitted students, persistence is typically defined as a match between stated interest in majoring in science at entry into higher education and final declared major. Using this operationalization, "success" occurs when students who intended to major in a STEM discipline ultimately end up doing so.
• Educational trajectories in the long-term (e.g., rates of graduation with STEM majors; rates of alumni graduate degree completion)

Other measures focus on student self-reported sense of scientific engagement and identity as well as intention to continue in science given the relationship between these variables and other measures of persistence as defined above.

• Common measures include use of the CURE and SURE surveys, developed by David Lopatto at Grinnell College.

Changes in science graduates between the 5-year periods, 1994–1998 and 2011–2015

×

Show caption

Hide

Changes in science graduates between the 5-year periods, 1994–1998 and 2011–2015 Data from IPEDS for periods indicated.Science majors includes standard STEM fields, including pre–health preparation and some professional fields, such as nursing, science teacher education, but excluding technical certifications like “science technicians.” Includes first and second majors in the 2011–2015 range, but they were not separately reported in 1994–1998. URM, underrepresented minority students including: black or African American, Hispanic or Latina(o), Native American or Alaska Native, Native Hawaiian or Pacific Islander (2011–2015), two or more races (2011–2015). Note that Morehouse, Spelman, and Xavier are HBCUs; nearly all their graduates for both periods fit into the URM category. The percent increase in total grads is based upon the number of baccalaureate graduates reported for the earlier and later periods. The increase in science grads overall is simply the difference in science majors granted for the two periods (combining changes for URM science grads with all other science grads). The percent increase in science grads is the percentage increase based upon the previous columns. The URM columns are parallel to the previous columns, except for URM only.

Reuse: This item is offered under a Creative Commons Attribution-NonCommercial-ShareAlike license http://creativecommons.org/licenses/by-nc-sa/3.0/ You may reuse this item for non-commercial purposes as long as you provide attribution and offer any derivative works under a similar license.

Caveats and Recommendations

Understand persistence using holistic assessment

Do not set your bar too low or too high

A priori benchmarks of progress that are set for persistence programs need to be chosen carefully. All of the Capstone Institutions shared examples of times during program development that data revealed expectations that started as either too low or too high. Comparisons of an individual institution's students to those from other institutions or to nationwide data needs to be done carefully, in the context of your institution's overall profile in higher education. If at the start of an institution's efforts, its STEM degree rate exceeds national norms but students from certain groups underperform on measures of academic achievement and access to opportunity, then there is still a problem that clearly needs addressing. We also agree that positive changes can take time. It is important not to get discouraged if a developed program does not move the needle sharply and quickly. Nonetheless, institutions must look for opportunities to assess changes formatively and regularly to know whether programs are moving outcomes steadily in the desired direction.

Chose your comparison groups intentionally

One important assessment decision involves the choice and definition of comparison groups to compare outcomes for program participants with the outcomes for students who have not participated. Create these groups intentionally and wisely based on who it is you are trying to serve and how you would define success. Often, institutions compare groups of students with various social identities (e.g., underrepresented minority students; first-generation college students; low-income students) to all other groups in order to determine whether the entirety of their student body evidences equivalent levels of success across meaningful metrics.

Determine who it is you are trying to serve and how you would define success. These are important decisions based on your own institution's goals and current outcomes across underrepresented groups. Institutions need to understand their own student populations' strengths and needs through dialogue and casual observation paired with more objective analysis of outcomes. It is especially important to know your students' academic/college level preparedness. With this knowledge, strategies developed to "level the playing field" can have maximum impact.

When possible, try to compare equivalent groups who do—versus do not—receive the program in order to create meaningful comparisons related to the impact of your programming.

Beware the most common measure of persistence in science; it is incomplete

One of the most popular ways to measure program success in higher education is to examine whether students' intention to major in science matches their eventual declared major. There are a number of problems in relying on the integrity of self-reported intended major report at the time of enrollment in order to assess persistence. This singular data point might hide important information. For example, a student who had taken significant coursework in STEM with a long-standing commitment to becoming a scientist would be considered the same as a student who, feeling pressured to pick an area of interest, decided in the moment to make the same choice. Ultimately, their persistence (or lack thereof) would be considered equivalent when they were beginning from rather different places. Our institutions agreed that persistence in science might not always be a desirable goal at our liberal arts institutions. When our students chose a non-STEM major, this might be viewed differently depending on a variety of factors, including the strength of the original stated intent and the reason for their choice.

We need better measures of persistence that go beyond examining whether there is a match between intended and declared major. We lose too much information given this operationalization. Better metrics of persistence would go beyond measuring intent through a singular time point and instead track a student's intended major from time of application through the declaration of the major. Another promising avenue for understanding persistence is to invest in efforts to understand both those students who declare a non-STEM major as well as students who are recruited to STEM. In that way, we will have a richer sense of the barriers and entry points to STEM, allowing more targeted interventions that obviate or capitalize on factors related to students' educational and career trajectories. Other promising complementary approaches for assessing persistence include:

Fostering institutional cultures that support inclusive excellence

When values that affirm the importance of access and diversity in science are shared broadly by a community, they work in concert with the goals and values of inclusive excellence.

Faculty Perspectives

Historically, some science faculty embraced a mentality that part of their job was to "weed out" students who were not likely to be successful as scientists. This attitude is antithetical to inclusive excellence. Faculty development needs to focus on how to help "weave in" all students, rather than to weed them out. This means that faculty must view their responsibility to teach every student who wants to learn science, rather than only focusing on the most well-prepared students in our classrooms and programs.

Faculty need to be educated about the power of student mindset on academic outcomes and trajectories (the work of Carol Dweck is particularly instructive here). We must cultivate growth rather than fixed mindsets in our students, emphasizing the idea that one must not come to science with natural talent or brilliance (fixed mindset) but rather that successful scientists are ones who view their learning through opportunities for practice, growth, and opportunity (growth mindset).

Following this, faculty, programs, and institutions need to watch the way that everyone in the community understands and discusses the purpose of any bridge or other programs that help get underprepared students ready for the rigors of college. Framing these programs as remedial or the students as needing remediation undermines their sense of belonging and is inconsistent with evidence finding that the right environments can foster success for all students. Instead, we should talk about these programs as providing opportunities for what Claude Steele calls "wise schooling" (pairing language around faculty/the institution's high expectations for students with the unquestioning belief that the students are capable of reaching them).

Institutional Perspectives

Campus climate is key to persistence and access. The more that the values above are collectively shared by stakeholders, the better off persistence efforts will be on your campus. Strategies that we have found effective at our Capstone Institutions include: broadening conversations about access to include large groups of faculty; sharing ideas and data with administrators; and connecting with strategic planning efforts that can incorporate the mission of fostering inclusive excellence for departments or institutions.

Related Resources

Cohen, Steele, & Ross (1999). The mentor's dilemma: Providing critical feedback across the racial divide. Personality and Social Psychology Bulletin, 25, 1302–1318.

Dweck, C.S. (2007). Mindset: The new psychology of success. Ballantine Books: New York.

Steele, C.S. (2010). Whistling Vivaldi: How stereotypes affect us and what we can do. W.W. Norton & Co: New York.