Getting Started

EvaluateUR Getting Started

Background

EvaluateUR is a strategy for obtaining an objective assessment of the value of undergraduate research (UR) by documenting student growth in critical academic and workforce-related knowledge and skills without placing a significant evaluation burden on students or faculty mentors. EvaluateUR provides reliable outcome data that can be used by program directors to demonstrate the value of UR experiences, while at the same time supporting student metacognitive awareness – the ability to identify one's own academic strengths and weaknesses. It does this by integrating assessments directly into the research process, using student-mentor conversations centered on an assessment protocol to provide students with detailed feedback and support their self-reflection.

EvaluateUR was developed at SUNY Buffalo State and is being disseminated nationally with funding from the National Science Foundation. The program and its utility are documented in two articles published in the CUR Quarterly (see the Publication page for pdf's of the articles).

Student Outcome Measures

EvaluateUR is built around 11 outcome categories that reflect a wide range of knowledge and skills. Each outcome category is defined by several components describing the student behaviors that define the outcome. The EvaluateUR assessment instrument is built around these outcome categories and components. Students score themselves on each component and mentors, using the same instrument, also independently score their student(s). On three occasions over the course of each research project, the students and mentors compare their respective assessments and discuss the reasons for any difference in scoring. These conversations are designed to help students gain greater academic self-awareness (metacognitive skills), and they provide opportunities for both the student and mentor to adjust their scores over time based on growth (or lack of growth) in student knowledge and skills. The 11 outcome categories are shown below.

Outcome Categories

  • Communication
  • Creativity
  • Autonomy
  • Ability to Deal with Obstacles
  • Intellectual Development
  • Critical Thinking and Problem Solving
  • Practice and Process of Inquiry
  • Nature of Disciplinary Knowledge
  • Content Knowledge and Methods
  • Ethical conduct
  • Career Goals

Learn more about Outcomes and Components »

To increase the flexibility of EvaluateUR, campus UR program directors have the authority to insert several additional program-wide outcome categories and/or components that would be added to the three assessment instruments for all student-mentor pairs. They may also add a limited number of open-ended questions designed to solicit student and mentor feedback about how the research project is going. These additions, if any, would be done when the program director registers and activates student and mentor pairs (see the Pre-Research page). Faculty mentors may also insert project-specific outcome categories and/or components.

Scoring Rubric

Using the assessment instrument, students score themselves and mentors independently score their students using a 5-point scale reflecting the frequency of the behavior described by each component of a given outcome. The scale indicates that a student Always, Usually, Often, Seldom or Never displays the action described in the respective outcome component. It is important that the site administrator explain the rubric during orientation (discussed in the Pre-Research and Resources sections) and prior to the mid-research and end-of-research assessments. It is essential to emphasize at these times that the assigned scores are less important than the conversation that follows the assessments, when the student and mentor share their rationales for assigning particular scores and discuss the reasons for differences, if any, in their perceptions.

Provision has also been made for both the student and mentor to indicate their confidence levels in assigning assessment scores. This is intended to reflect the degree of development of the student's metacognitive skills and the extent to which mentors have been able to observe their student researchers in a range of situations.

Benefits to Students, Mentors, and Administrators

EvaluateUR can contribute to making student research more productive by providing a framework and specific instruments for students and mentors to exchange views about how well students are doing across a wide range of variables bearing on research progress. The repeated student-mentor conversations provide opportunities for mid-course research corrections. But while the primary beneficiary of EvaluateUR should be the student, there are also benefits both for the faculty mentor and the director coordinating the program. A summary of benefits unique to EvaluateUR is provided below:

Students

  • Are introduced to a comprehensive list of competencies and skills that include but go beyond subject area knowledge that they will need in order to pursue graduate work and/or succeed in the workplace
  • Have productive conversations with their mentors about their research and their intellectual development
  • Obtain a realistic picture of their strengths and weaknesses across all of the competencies and skills they should strive to achieve
  • Gain confidence and greater self-awareness as they track their academic growth
  • Strengthen their applications to graduate programs or their resumes for entering the workforce

Mentors

  • Are able to make more consistent and dependable assessments
  • Gain new insights into the strengths and weaknesses of the student(s) they mentor
  • Are helped to focus their mentoring efforts on specific areas where students may need extra guidance in order to become more independent researchers
  • Are provided with examples of ways in which students in their classes may not be aware of all the competencies and skills they should strive to achieve and may benefit from more clarity about their academic strengths and weaknesses

UR/REU Directors

  • Acquire reliable data to support campus assessments of UR (and, often, related funding requests)
  • Obtain information that can support efforts to recruit and retain students
  • Have opportunities to present their program impact/outcomes at professional meetings and publish in professional journals

Continue to Pre-Research Step »