Initial Publication Date: December 3, 2021

Summative Evaluation Report Network of STEM Education Centers (NSEC)

Improving Undergraduate STEM Education (IUSE) program of the National Science Foundation (#1524832)

Nancy Shapiro, Associate Vice Chancellor at University System of Maryland and external evaluator for NSEC

 Summative Evaluation Report Network of STEM Education Centers (Acrobat (PDF) 225kB Dec6 21) 

December 4, 2021

Summative Evaluation:

This is the final year of the IUSE NSEC grant (2019-21). Over the life of the project, NSEC has built a community of STEM education centers and delivered a rich array of resources to support the community.  NSEC was not primarily in the business of creating new STEM Centers, but rather, it was created to amplify the work the existing centers were doing. In that process, NSEC has seen the growth of the STEM Center model across many more universities and states—an unintended, but not unanticipated outcome.   The project had five key goals:

  1. Build a learning, research, and implementation network for centers via conferences, workshops, communications, staff interactions, and an online platform.
  2. Showcase, celebrate, and understand the work of centers that are transforming undergraduate STEM education via case studies, research on center impacts, and center profiles.
  3. Serve as a resource and catalyst for centers, policymakers, funders, administrators, and the public on what works in STEM education via a national online platform of effective practices and programs, directory of experts in STEM education, and research on effective center and institutional practices, and center impacts.
  4. Create a coalition of actors that can address and engage in practices that are cross- and multi-institutional via seed grants for collaborative research and implementation proposals.
  5. Collectively work to improve institutional and national policies which strengthen undergraduate STEM education through guiding documents, participation in national dialogues, and policy statements.

The final report details the contributions and progress toward these ambitious goals and provides strong evidence that the work proceeded on multiple levels throughout the entire project.  The strength of the network and the resources that grew out of the structured collaborations were highly valued and valuable to a community of university educators who do not always have the luxury of adequate resources or attention from their own institutional leadership.

From the perspective of this project evaluator, the project delivered on its promises, in spite of the "Covid curveball" of the last 18 months and the distinctly longitudinal vision that drives any institutional transformation project.  The research that came out of this project (Carlisle and Weaver 2018, 2020) establishes a strong theoretical framework for projecting the "value added" of campus-based STEM Centers.  Similarly, the research being conducted by Bruce Goldstein (2021), seeks to lay out theoretical rationale for the value-add of "networks of networks," validating the NSEC model and placing it in a larger context of change models.

The third and most accessible product of the research from this project is Deborah Carlisle's tools for the self-assessment of Centers. The toolkit (based on the research mentioned above) goes beyond an evaluation rubric and includes a multi-module guidebook that captures an array of capacity building tools (current trends in center practices, resources to assist with strategic planning and evaluation, guiding questions to enhance organizational management, etc.).  Broad access to these tools will give institutions that are on the fence about establishing STEM Centers enough guidance to take the next step, knowing that considerable research has already been done to validate STEM Centers' contribution to undergraduate STEM education. The other benefit of the toolkit is that it is not prescriptive, but rather, derives organically from an institutional context.  That means that a STEM Center established at a large public university will not necessarily have the same benchmarks as a STEM Center launched at a smaller university.

According to the authors, the toolkit resources can be utilized for the following purposes: 

  • Benchmark assessment
  • Strategic planning for centers wishing to expand their role in undergraduate STEM education 
  • Five-year review
  • Administrative value (how to position and best utilize a center to contribute to institutional goals) 
  • Ways to support a center to achieve its mission
  • On-boarding a new center director and/or advisory board
  • Communication tool
  • Resources to assess needs and make the case for more resources
  • Educational value of center staff - where they fit in the bigger picture of center operations

The combination of identifying the mission and goals of STEM Centers, assessing them against validated benchmarks, and successfully communicating the value of the centers to maximize undergraduate student experiences with STEM education all speak to the importance of this toolkit as one of the most valuable and potentially transformative deliverables of this project.

The NSEC project attempted to do many things at once: create a network of STEM Centers, define what constitutes a STEM Center, establish a research base that grounds the theory of STEM Centers, build tools that support the sustainability of STEM Centers, evaluate the success and sustainability of the network, and create backup plans to continue to support STEM Centers when NSEC funding expires.   This is the classic illustration of "building the plane while flying it."

The PIs created an integrated leadership team with enough independence among the strands of work, that each research team, implementation team, and outreach team were given enough independence to explore, expand and implement their visions.  The PIs created a work-environment that balanced the necessary grounded research with the essential outreach to campuses, providing a secure lifeline to those individuals at the campus level so they were not cast adrift.  Annual meetings (in person and virtual) were the connective tissue of the project.  Researchers could ground their theoretical models in the lived reality of the campus STEM Center directors' experiences.  STEM Center directors and staff could draw on research, and question speculative conclusions based on their lived experiences. 

Looking ahead

Research Action Clusters (RAC): One of the more interesting and innovative elements of this project were the "seed grants" that went to faculty and junior staff members to experiment and implement projects within STEM Centers. The reports from these projects (detailed in earlier annual reports) confirm that seed money is a critical component of almost any center that wants to promote faculty-driven innovation (and one could argue that faculty-driven innovation is inherently more acceptable and sustainable than administrative driven innovation).  The documented successes of the RACs suggest that STEM Centers should try to incorporate seed money into their budgets, if possible.  Certainly, allowing NSEC to use NSF funding in this way paid off in a big way.

Dissemination and Communication (SERC, APLU, NSF):  The Association of Public Land Grant Universities (APLU) has hosted NSEC for the past five years.  As the organizational home of the Network, APLU has not only provided professional and administrative staff support; it has also modeled an important leadership role for higher education associations that strive to keep their membership connected and informed about student learning innovations.  APLU's support of NSEC is evidence of their commitment to serving students, as well as serving member institutions.  APLU can function as a role model for other higher education associations that represent different segments of the higher education community. NSEC is an investment in improving STEM teaching and learning across the broader landscape of post-secondary education; and organizational support, such as APLU's, can scale innovations in STEM teaching and learning, increasing STEM majors, STEM literacy, and STEM workforce more broadly.

The value of the NSEC work is at least three-fold: first, the work of the project generated more STEM Centers; second the project generated findings about establishing, evaluating, and promoting STEM Centers; and third, the project developed tools to build and improve STEM Centers.  All this new knowledge needs to be shared and communicated broadly. Currently, NSEC information lives on three websites (SERC, APLU and NSF), some of which are more user-friendly than others.  It would be especially valuable to the STEM education community if a way could be found to maximize the dissemination of the tools and research across different audiences.

Policy implications: Following on the dissemination discussion, STEM educators are not the only audience for the outcomes of this project. Undergraduate STEM educators will continue to fight an uphill battle:  not enough high school students are well-prepared to major in STEM fields when they get to college; workforce demands for math and science graduates will increase, not decrease; increasing the diversity of STEM faculty will become more important to expanding the pipeline for STEM majors; the K-12 STEM teacher shortages are increasing across the country with long term implications for our future citizenry and workforce.  Each of these challenges is a policy arena where STEM Centers could make positive contributions.  STEM Centers are not the only solution to these concerns, but they can and should be on the table when policy leaders are considering how to address the challenges they face when contemplating an unpredictable future of climate change, endemic pandemic, global economic interdependence, and cyber/political disruption.

The continuing challenge of sustainability of NSEC

The success of NSEC, overall, is a good place to start a discussion of sustainability beyond the grant-funded activity.  From the beginning, the NSEC leadership has included consideration of sustainability in all planning and reporting activity.  The PI's have acted on several of the options presented last year and are still waiting to hear about possible continued support from NSF through the IUSE program.

NSEC will continue beyond NSF funding. APLU has committed to maintenance of this community through factional PI support (Redd) and administrative support, continued listserv and communication mechanisms, maintenance of the website, occasional webinars/ gatherings online, collaboration with key other networks (AAU, NASEM, ASCN, BVA, etc.), and partnership for its biannual national conference. In its current approach, based on the success of the last two years, NSEC is partnering with the Accelerating Systemic Change Network (ASCN) to coordinate on regular national meetings (every other year).

NSEC PIs continues to explore the original sustainability options:

Option 1:   Aligning with POD and becoming a track or sub-group within that larger organization. 

  • This year NSEC developed a supplement that would create a partnership between POD, NSEC, and the NASEM Roundtable to support regional collaborations. However, this was deemed to be a new scope of work, so NSEC submitted this partnership proposal to IUSE in 2020. The grant proposal has not been decided yet at the time of this writing.

Option 2:  A second idea is a regional approach to organizing the STEM Centers into a network.

  • NSEC has engaged in the efforts to advance centers' roles in the national space.  They are intentional in partnering with other national networks and organizations that advance undergraduate STEM Education and have presented at or worked with over a dozen national organizations. This would most likely build on option 1. 
    • Connecting with other key networks: POD & ASCN in particular
    • Regional based work
    • Continuing the national meeting and other modes of connecting 
    • Stability in organizational structure
  • This larger-scale approach would potentially fit well with some of the national moves, such as creating a nationally distributed network of NSF-funded Technology Centers.

Option 3:   A third idea can be thought of as a "mid-scale infrastructure" opportunity.  Could NSEC become the seed of a new, useful space or hub that would serve as an umbrella for regional collaborations? Such a hub could be supported by fees, or other funding models. 

  • One theme that came out of the national conference was a recognition of a sense of saturation of communities working in this space. There is, therefore, a need to emphasize the key foci of NSEC which are (1) networking among STEM center leads to addressing  common kinds of challenges and problems which lead to improving undergraduate STEM education, (2) acting in collective or in aggregate to address challenges that go beyond a center as evidenced by the success of the  Research Action Cluster approach, and (3) upward communication to address policy and policy-makers (e.g. National Academies Roundtable). 

SWOT Analysis 

While it may seem incongruous to attempt a SWOT analysis at the end of a project, rather than the beginning or the middle, the traditional framework for strategic planning can provide a useful shorthand for assessing where NSEC is now, and what opportunities and challenges might lie ahead.

Strengths: 

  • A vibrant and active network that has grown to 213 STEM Centers at 168 institutions,
  • Seven Research Action Clusters (RAC), including a network report with an inventory of Statewide STEM Education Networks in the U.S and a STEM Tutoring Center, with tools and guidance for replicating the work; and work on improving undergraduate mathematics courses to embed active learning in mathematics courses which has potential to lead to greater student success,
  • An online platform where center directors can share successes, challenges, and failures, and can serve as a resource for policy makes and decision makers,
  • Expanded scholarship on what is known about creating effective STEM education networks,
  • Regular community-based events:  an annual conference, webinars, and related programs,
  • Connections to and among the variety of other networks and professional organizations invested in STEM education.

Weaknesses: 

  • Lack of senior faculty as part of the leadership of STEM Centers,
  • No source of sustainable funding (funding embedded in institutional budget with staff lines and operational funding),
  • Saturation of programs, networks, events in the STEM Education space, 
  • Associated lack of coordination among the programs, networks, and events in the STEM Education space.

Opportunities:

  • Disseminating the toolkit to establish quality criteria and processes for creating new programs at new universities,
  • Creating advocacy materials for existing STEM centers to advance their work and increase their budgets and their scale and scope,
  • Building out to the teacher education community with special STEM Center activities directed toward K-12 teachers,
  • Creating collaborations with K-12 teacher educators,
  • Following the U.S. Department of Education investment in undergraduate STEM education if/when major 2022 federal education funding bill is passed,
  • Using the new virtual/hybrid work environment to maximize connections between and among STEM Centers through NSEC,
  • Other infrastructure investment, whether from federal, state or private organizations,
  • Coordination among the myriad of actors in STEM education.

Threats:

  • Challenges of sustaining the network when the NSF funding ends,
  • Lack of cohesion among current STEM Centers that will result in weak centers failing,
  • Transitions of leadership at APLU, and at institutions with STEM Centers,
  • Competition within the STEM community for ownership of the model,
  • Dilution of leadership of campus-based STEM centers when senior faculty are replaced by non-tenured faculty and/or staff,
  • Shifts in higher education landscape – and local (campus-based) disinvestment in resources like STEM education centers.

Concluding thoughts and policy implications

The nation's STEM ecosystem is changing so rapidly it is almost impossible to keep up with the pace of change.  STEM workforce shortages are increasing, not decreasing. Solutions to climate change, endemic pandemic, global cybersecurity threats, engineering and infrastructure deterioration, AI, alternative energy sources, and STEM K-12 teacher shortages all put stress on the system and consume policy makers. 

And if the reality-based, real world STEM challenges, were not enough, today we are confronting an almost inconceivable and inexplicable war on science itself.  Social media is driving some people down dangerous rabbit holes that threaten our public health, our natural world, and our global security.   

Recently, Scientific American published an essay opinion about "anti-science:"

Antiscience has emerged as a dominant and highly lethal force, and one that threatens global security, as much as do terrorism and nuclear proliferation. We must mount a counteroffensive and build new infrastructure to combat antiscience, just as we have for these other more widely recognized and established threats.[1]

If we believe that education is the antidote to antiscience, we must invest in every effort to improve STEM education (P-20), and support ways to do that.  NSEC is building a knowledge network to improve STEM teaching and learning. The power of a network is that it simultaneously contributes new knowledge (intellectual merit) and has broad impact because it builds a network across institutions and states.

The P-20 implications are equally serious. Science literacy begins in P-12 schools in the early grades.  While STEM networks have undergraduate education at the heart of the work, in fact, the broadest possible impact will come through improving the STEM education of elementary and middle school teacher candidates.  Planting early seed of inquiry, curiosity, hypothesis testing, and scientific method all contribute to broader scientific literacy in the population at large. 

At the institution level, creating opportunities for STEM faculty to collaborate with education faculty to improve teaching and learning in the STEM fields across the P-20 spectrum might lead to productive partnerships and outcomes.

Similarly, collecting evidence from institutions where robust STEM centers exist could lead to stronger advocacy to support STEM Centers more broadly.

STEM Centers can be a resource for federal agencies looking to disseminate new knowledge about the cognitive science of learning STEM fields, and NSEC can be the dissemination network for new knowledge related to teaching and learning STEM.

It appears that the hard part of this work has been accomplished—the Network is established; it is credible and robust.  It should be seen as a resource moving forward.

 

 

Nancy S. Shapiro

nshapiro@usmd.edu

[1] Peter J. Hotez, "Escalating, Going Global and Killing Thousands:  Rejection of mainstream science and medicine has become a key feature of the political right in the U.S. and increasingly around the world," Scientific American, March 29, 2021 https://www.scientificamerican.com/article/the-antiscience-movement-is-escalating-going-global-and-killing-thousands/