February 2018 EvaluateUR Update

Thank you for your interest in EvaluateUR. This update highlights some of the progress made to date as part of our NSF WIDER project. The main goal of our WIDER project is to refine and broadly disseminate a successful undergraduate research assessment model that has been in place at SUNY Buffalo State since 2008. The model is based on a theory of how student outcomes could be improved by making evaluation an integral part of the undergraduate research experiences, leading to the design of a unique, evidence-based approach to teaching and learning. The project is a partnership among Buffalo State, CUR, Science Education Resource Center (SERC) at Carleton College, and Finger Lakes Community College.

A guiding principle of Evaluate UR is the desire to obtain reliable independent assessments of program impact without creating a measurement burden, while at the same time providing information to participating students that could help them gain new insights into their academic strengths and weaknesses and a new appreciation of the broad range of academic and personal skills for which they should assume responsibility.

In order to accomplish these objectives, 11 student outcome categories of interest have been identified: communication, creativity, autonomy, ability to deal with obstacles, practice and process of inquiry, nature of disciplinary knowledge, critical thinking and problem solving, understanding ethical conduct, intellectual development, culture of scholarship, and content knowledge skills/methodology. Each of these outcome categories is in turn defined by several discrete components designed to provide explanatory detail about outcome meanings. Student accomplishments on the components of each of these outcome categories are assessed using a five-point scale linked to an explanatory rubric. Faculty mentors and students each complete these assessment surveys three times – at the outset of the research, in the middle of the research, and at the end of the research experience. Faculty mentors rate students on each component and students evaluate their own progress using an identical instrument. Project procedures then revolve around repeated structured interactions between student and mentor: Following each of the three assessments, the mentor and student meet to discuss how each scored their assessments and the reasons for any assessment differences. This provides mentors with multiple opportunities to review and assess student work and provides students with objective assessments that can help them to achieve a deeper understanding of their strengths and weaknesses. There is also an option for the student-mentor pair to add additional outcomes and outcome components. This feature provides each student-mentor pair flexibility to assess discipline-specific outcomes or any other aspect of the research experience they are interested in assessing.

Accomplishments and updates include the following:

  1. We added a tagline to the name of the evaluation model to better reflect what it is. It now is: EvaluateUR: An Innovative Approach to Documenting Growth in Student Content Knowledge, Skills, and Self-Awareness
    This will appear on webpages and other materials produced for EvaluateUR.
  2. We successfully completed our Phase I pilot in the summer 2017. Six institutions participated in the pilot and include: Longwood University, SUNY New Paltz, SUNY Buffalo, University of Missouri Kansas City, and University of Washington Seattle, University of Wisconsin Eau Claire. Following the pilot, we met in Buffalo in October 2017 to share experiences and consider ways to add more features to EvaluateUR to support broad dissemination and adoption of the method.
  3. To facilitate implementation of EvaluateUR, we've developed a web-based project administrative tracking tool called a "Dashboard" to help site administrators track the progress of student/mentor pairs, and introduced a data analysis and reporting function called EZStats to prepare reports summarizing assessment data. We also added 'Assessment Score Reports' – comparisons of student and mentor assessment scores highlighting any significant differences -- to facilitate meaningful conversations between the student and mentor following the pre-, mid-, and end-of-research assessments.
  4. Additional institutions currently are preparing for our second pilot testing in summer 2018. In addition to the six institutions listed in #2 above, Berea College, California State University San Marcos, Capital University, College of the Holy Cross, Michigan Technological University, Southern Nazarene University, SUNY Brockport, University of Connecticut, University of Wisconsin-Whitewater, and Viterbo University.
  5. CUR has undertaken a variety of activities to disseminate the project. An upcoming session is scheduled for CUR's Biennial Conference 2018 being held in the D.C. area in July.

We anticipate recruiting for the 3rd round of pilot testing EvaluateUR as well as an upcoming CUR hosted webinar. Details will be shared as soon as additional details are available.

Thank you,

Jill Singer on behalf of the EvaluateUR leadership team