REU Program Assessment

Compiled by: David Mogk, Dept. Earth Sciences, Montana State University and Val Sloan, SOARS Center for Undergraduate Research, UCAR, Boulder, CO. Based on input from the GEO REU email list

Darrell Henry supervising a student in the field during the 2010 REU Yellowstone Project. Photo credit David Mogk

What is assessment?

Data collection with the purpose of answering questions about: students' understanding, students' attitudes, students' skills, instructional design and implementation, and curricular reform (at multiple grain sizes)--from Diane Ebert-May. Assessment is "collecting data with a purpose" (NRC Integrating Research and Education: Biocomplexity Investigators Explore the Possibilities(NRC, 2003). What evidence is acceptable to demonstrate and measure progress towards goals, objectives and outcomes?

Assessment often addresses the "what questions about teaching and learning—what do students know and what can they do?" Assessment provides evidence that things are working or not (e.g. Do students actually learn better?). Evaluation may be defined as "the systematic investigation of the merit or worth of an object" (NSF 93-152) and is often used in the context of "what value has been added through this project, and how do you know?" Project evaluation is often used to demonstrate accountability (e.g. have the project goals been met?).

What's the difference between formative and summative assessment?

The goal of formative assessment is to gather feedback that can be used by the instructor and the students to guide improvements in the ongoing teaching and learning context. These are low stakes assessments for students and instructors.

The goal of summative assessment is to measure the level of success of proficiency that has been obtained at the end of an instructional unit (or REU program), by comparing it against some standard or benchmark. From Carnegie Mellon "Enhancing Teaching".

Why do assessment?

  • Improve student learning and development through improving the program.
  • Provides students and faculty substantive feedback about student understanding, and program strengths, weaknesses and impact.
  • Accountability: find out whether you have achieved defined goals- for student learning and overall programmatic effectiveness and include the results in annual reports to sponsors and grant proposals.
  • Gather data on your program with an eye to doing education research on REU programs.
  • Effective assessments, embedded into program design, can result in students who are happier and who perform better, and can help guide faculty to create more effective, efficient and gratifying programs that have greater impact.

Assessment of education and outreach programs may be done for many reasons, and on many scales. The results may be utilized by different interested groups-including top-to-bottom project reviews, evaluation of the effectiveness of specific materials or methods, indicators of student learning, long-term impacts of a project, or as confirmation that the goals of a project have been met. Assessment activities may also lead to more expansive research on learning projects. The scholarship of teaching and learning provides many exciting (and much needed) opportunities to form partnerships with the cognitive and social sciences.

Some of the high level goals that have been defined by NSF can be found in these documents: GeoVision Report (Acrobat (PDF) 3.3MB Sep20 13), AC-GEO (2009), and Empowering the Nation Through Innovation and Discovery (Acrobat (PDF) 2.7MB Sep20 13) the NSF Strategic Plan, and Strategic Framework for Education and Diversity, Facilities, International Activities, and Data and Informatics in the Geosciences (Acrobat (PDF) 11.2MB Sep20 13) .

The Right Tool for the Right Job

There are a few basic principles that will help you effectively develop your own assessment plans to best meet the needs of your project:

  • Clearly define project goals and expected outcomes at the start.
  • A list of IRIS REU Outcomes (Microsoft Word 2007 (.docx) 71kB Sep18 13) and the IRIS Logic Model (Acrobat (PDF) 120kB Sep19 13) have been provided by Michael Hubenthal.
  • What is the purpose of the assessment? Who will use the results and in what way?
  • Identify the baseline data you will need to document change.
  • There is an arsenal of assessment techniques that are available; pick the right tools and metrics that will provide the information required to meet your needs.
  • Assessment is done throughout the course of a project for varying reasons: formative assessment is done to provide feedback for ongoing activities, and to inform any needed mid-course corrections; summative assessment is done to measure a project's overall success; longitudinal assessment tracks impacts beyond the duration or initial scope of the project.
  • The assessment plans should be integral to the development and management of the project, not just added on as an after-thought.
  • Develop partnerships with colleagues who have knowledge and expertise in assessment.

Assessment of student learning outcomes require an entirely different set of instruments and metrics than overall program assessment. The following section provides some assessments developed by GEO-REU leaders to address both aspects of assessment.

Assessment Instruments Developed by REU PIs

The following is a compilation of assessment instruments submitted by REU PIs that have been used to document student progress and/or program progress submitted by PIs on the GEO-REU listserv:

  • Start of Internship questionnaire (Microsoft Word 2007 (.docx) 118kB Sep17 13)- submitted by Harmony Colella, Univ. of Miami (Ohio) for the IRIS RESESS internship program.
  • Research-Focusing Questions (Microsoft Word 35kB Sep17 13)- Dallas Abbott, LDEO Columbia University.

    The process of mentoring students in research can be done many ways. We use something we call research focusing. The research focusing methodology was developed by Kim Kastens for students in a masters degree program in Earth & Environmental Sciences, and adapted to the undergraduate intern program by author Dallas Abbott. Three research focusing meetings are scheduled throughout the summer. The first meeting is in the third week of the program, after students have had time for orientation and in some cases, a week-long field trip. The meeting includes a PhD level facilitator and 2 intern peers. Each student has 20 minutes of time and stays for an hour. The peers are changed at each meeting so the students have the opportunity to learn about other research projects. Peers are also encouraged to ask questions. We also emphasize that we expect that the students will not know the answer to all the questions posed and that this is OK. Because undergraduates are used to tests where all the answers are supposed to be known, this is an essential feature of undergraduate research focusing. We added this feature when it became clear that many undergraduates were intimidated or discouraged when they did not know the answer to every question

    Because the peer group changes at each meeting, we ask each student for a 5 minute summary of their project and to write their question on the board at the start of both the second and third research focusing meeting. This requirement increases student facility with a simple explanation of their research that can be understood by anyone with a scientific background. It also requires students to think about the big picture of their research at each meeting. This is also important because students will be asked to give a 1 minute "commercial" about their research just before the start of the end-of-summer poster session.

    The second research-focusing meeting addresses the methods that each student is planning to use. This meeting is held in the fifth week of the intern program. Typically, they view any type of instrument as a black box, and the facilitator presses them to explain the underlying physics or chemistry by which they will be, for example, turning ground up rocks into numbers, and numbers into insight about the Earth. If sampling is involved, we require the student to articulate and defend the sampling scheme: why not more samples? Why not more closely-spaced? Why not cover a larger area? After a few students have been through this, they begin to see that there is an inherent trade-off between breadth and depth in almost any research project. Students articulate their assumptions and the limitations of their methods, and the facilitator tries to help them see that every method has limitations beyond which it is inappropriate to apply.

    The third research-focusing meeting is held at the beginning of the 8th week of the program, just before the students start to make their posters. At this time, most of the students already have some data. We found that it is too difficult for undergraduate students to address a theoretical interpretation of data without some preliminary data collection. Even then, we find that this is the most difficult research-focusing meeting. But with the guiding questions we provide, each student does, eventually, come up with something interesting and relevant to say for this meeting. This exercise gives them practice at thinking hypothetically. It makes it easier to start thinking about the meaning of their data while they are still in data acquisition mode, rather than waiting to begin interpretation after it's too late to get more or different data. It prepares them to recognize the unexpected while it is happening, because they have already thought through the expected outcomes. In addition, we also talk about displaying their data in poster format. This helps them to see where the data is going and to ask further questions that might be addressed with further data collection or in a senior thesis.

    The questions for each meeting are attached. I hope you find this useful.

    Best Wishes,

    Dallas Abbott

Related Resources on Mentoring

Mentoring is an important component of formative assessments of student performance. Many REU leaders are reporting that students are indicating that the single most important factor in their REU experience was the mentoring support.

"Sometimes the most valuable contribution a mentor can make is just time and attention. It is always surprising to talk to former mentees about their experiences and what they found valuable. Often, their comments focus on a few themes: (1) it helped to have someone believe in my potential, (2) it helped my confidence to know that I could talk or write to someone of your stature, (3) it helped to have you listen to some of my professional development plans and then hear your suggestions.

"When mentoring, don't forget that just your time and attention can have a very significant impact. The combination of the mentor's accessibility and approachability is critical and even small actions can be impactful. Examples may include having lunch with a student and establishing an open-door policy, or in a class setting learning students' names and making a point of requesting student feedback on course material during class time (Gall et al. 2003)."

Some Suggested Resources on Assessment