REU Program Assessment

Compiled by: David Mogk, Dept. Earth Sciences, Montana State University and Val Sloan, SOARS Center for Undergraduate Research, UCAR, Boulder, CO. Based on input from the GEO REU email list

What is assessment?

Data collection with the purpose of answering questions about: students' understanding, students' attitudes, students' skills, instructional design and implementation, and curricular reform (at multiple grain sizes)--from Diane Ebert-May. Assessment is "collecting data with a purpose" (NRC Integrating Research and Education: Biocomplexity Investigators Explore the Possibilities(NRC, 2003). What evidence is acceptable to demonstrate and measure progress towards goals, objectives and outcomes?

Assessment often addresses the "what questions about teaching and learning—what do students know and what can they do?" Assessment provides evidence that things are working or not (e.g. Do students actually learn better?). Evaluation may be defined as "the systematic investigation of the merit or worth of an object" (NSF 93-152) and is often used in the context of "what value has been added through this project, and how do you know?" Project evaluation is often used to demonstrate accountability (e.g. have the project goals been met?).

What's the difference between formative and summative assessment?

The goal of formative assessment is to gather feedback that can be used by the instructor and the students to guide improvements in the ongoing teaching and learning context. These are low stakes assessments for students and instructors.

The goal of summative assessment is to measure the level of success of proficiency that has been obtained at the end of an instructional unit (or REU program), by comparing it against some standard or benchmark. From Carnegie Mellon "Enhancing Teaching".

Why do assessment?

  • Improve student learning and development through improving the program.
  • Provides students and faculty substantive feedback about student understanding, and program strengths, weaknesses and impact.
  • Accountability: find out whether you have achieved defined goals- for student learning and overall programmatic effectiveness and include the results in annual reports to sponsors and grant proposals.
  • Gather data on your program with an eye to doing education research on REU programs.
  • Effective assessments, embedded into program design, can result in students who are happier and who perform better, and can help guide faculty to create more effective, efficient and gratifying programs that have greater impact.

Assessment of education and outreach programs may be done for many reasons, and on many scales. The results may be utilized by different interested groups-including top-to-bottom project reviews, evaluation of the effectiveness of specific materials or methods, indicators of student learning, long-term impacts of a project, or as confirmation that the goals of a project have been met. Assessment activities may also lead to more expansive research on learning projects. The scholarship of teaching and learning provides many exciting (and much needed) opportunities to form partnerships with the cognitive and social sciences.

Some of the high level goals that have been defined by NSF can be found in these documents: GeoVision Report (Acrobat (PDF) 3.3MB Sep20 13), AC-GEO (2009), and Empowering the Nation Through Innovation and Discovery (Acrobat (PDF) 2.7MB Sep20 13) the NSF Strategic Plan, and Strategic Framework for Education and Diversity, Facilities, International Activities, and Data and Informatics in the Geosciences (Acrobat (PDF) 11.2MB Sep20 13) .

The Right Tool for the Right Job

There are a few basic principles that will help you effectively develop your own assessment plans to best meet the needs of your project:

  • Clearly define project goals and expected outcomes at the start.
  • A list of IRIS REU Outcomes (Microsoft Word 2007 (.docx) 71kB Sep18 13) and the IRIS Logic Model (Acrobat (PDF) 120kB Sep19 13) have been provided by Michael Hubenthal.
  • What is the purpose of the assessment? Who will use the results and in what way?
  • Identify the baseline data you will need to document change.
  • There is an arsenal of assessment techniques that are available; pick the right tools and metrics that will provide the information required to meet your needs.
  • Assessment is done throughout the course of a project for varying reasons: formative assessment is done to provide feedback for ongoing activities, and to inform any needed mid-course corrections; summative assessment is done to measure a project's overall success; longitudinal assessment tracks impacts beyond the duration or initial scope of the project.
  • The assessment plans should be integral to the development and management of the project, not just added on as an after-thought.
  • Develop partnerships with colleagues who have knowledge and expertise in assessment.

Assessment of student learning outcomes require an entirely different set of instruments and metrics than overall program assessment. The following section provides some assessments developed by GEO-REU leaders to address both aspects of assessment.

Assessment Instruments Developed by REU PIs

The following is a compilation of assessment instruments submitted by REU PIs that have been used to document student progress and/or program progress submitted by PIs on the GEO-REU listserv:

  • Start of Internship questionnaire (Microsoft Word 2007 (.docx) 118kB Sep17 13)- submitted by Harmony Colella, Univ. of Miami (Ohio) for the IRIS RESESS internship program.
  • Research-Focusing Questions (Microsoft Word 35kB Sep17 13)- Dallas Abbott, LDEO Columbia University.

Related Resources on Mentoring

Mentoring is an important component of formative assessments of student performance. Many REU leaders are reporting that students are indicating that the single most important factor in their REU experience was the mentoring support.

"Sometimes the most valuable contribution a mentor can make is just time and attention. It is always surprising to talk to former mentees about their experiences and what they found valuable. Often, their comments focus on a few themes: (1) it helped to have someone believe in my potential, (2) it helped my confidence to know that I could talk or write to someone of your stature, (3) it helped to have you listen to some of my professional development plans and then hear your suggestions.

"When mentoring, don't forget that just your time and attention can have a very significant impact. The combination of the mentor's accessibility and approachability is critical and even small actions can be impactful. Examples may include having lunch with a student and establishing an open-door policy, or in a class setting learning students' names and making a point of requesting student feedback on course material during class time (Gall et al. 2003)."

Some Suggested Resources on Assessment