Preparing Future STEM Technicians by Using Assessment as a Teaching and Learning Tool in Course-based Research at Community Colleges

Summative Report

Prepared by: Bridget Zimmerman, Project Evaluator, and Daniel Weiler and Jill Singer, Co-Developers, EvaluateUR Method

NSF Award #1836033

Principal Investigator: Jill Singer

Note: A report on the outcomes of the five-year E-CURE project was submitted to the National Science Foundation in September 2023. The report has been edited for this web version but no project findings or conclusions have been omitted.

View or download the full report as a PDF (Acrobat (PDF) 368kB Oct19 23)

Background and Overview

The E-CURE project was designed to determine whether the methods employed successfully for more than 15 years in EvaluateUR for one-on-one student-mentor interactions could be modified to support course-based undergraduate research experiences (CUREs). After five years of project implementation, E-CURE has demonstrated that EvaluateUR can be adapted successfully for use in classrooms that vary widely in numbers of students enrolled, duration of student research projects, level of the course (e.g., lower level versus senior capstone), and student outcomes emphasized by the instructor. E-CURE has been demonstrated to be effective at both community colleges and 4-year institutions.                                                 

EvaluateUR measures student academic and metacognitive growth through several rounds of student self-assessments matched with their faculty mentor assessments, using identical assessment items and scoring metrics. EvaluateUR was developed for independent student research experiences, that is, individual students working on a one-to-one basis with faculty mentors. E-CURE modified EvaluateUR for the CURE classroom setting, where there is one instructor for many students. Its design replicates many key features of EvaluateUR including:

• Students are assessed in ten student E-CURE outcome categories, each defined by several outcome components, that include both content knowledge and outcomes that are critically important in the workplace, such as communication skills, creativity, autonomy, ability to overcome obstacles, critical thinking, problem-solving skills and teamwork.  Note: E-CURE initially did not include one of the EvaluateUR outcome categories (ethical conduct). Beginning in summer 2023, this outcome category has been added so that both EvaluateUR and E-CURE share the same list of outcome categories and components.
• Faculty may add other outcomes that reflect specific course knowledge and skill objectives. 
• To assess student progress, each outcome component is self-scored by the student researchers and scored independently by their instructors, using a five-point scale and accompanying scoring rubric, with the scores indicating how often a desired student outcome was observed (5-Always 4-Usually 3-Often 2-Seldom 1-Not yet).
• Students and their research mentors or classroom instructors share reviews of their respective assessment scores and discuss the reasons for their assessments. This feature is designed to help students understand their strengths and weaknesses in working to achieve these outcomes, and develop or enhance related metacognitive skills.
•  A data reporting feature allows undergraduate research mentors (EvaluateUR) and CURE instructors (E-CURE) to easily obtain pre- (EvalutateUR only), early/mid-, and end-of-research individual student score reports and, for E-CURE, class-wide summary assessment data for all course outcome components (see Guide to Data and Instructor Score Reports). These data enable EvaluateUR mentors and E-CURE instructors to document student learning gains and use the reports to help them design ways to improve their mentoring skills or their classroom instruction.

In addition, for E-CURE:

• It was recognized that instructors with larger enrollments in their CURE would not have the ability to observe each student such that they could confidently score all their students on all 10 outcome components. Thus, unlike EvaluateUR, the E-CURE design allows faculty to select a sub-set of outcome categories for assessment. This format enables faculty to focus on a 'core' set of  outcomes of particular relevance to their course research assignment(s).
• In EvaluateUR, mentors usually were already somewhat acquainted with the students they would be working with. Accordingly, that model was able to incorporate three assessment periods: Pre-research, early/mid-research and end-of-research. In E-CURE classrooms, however, most instructors knew little or nothing about many of their students, and did not think they could complete accurately a pre-research assessment. E-CURE therefore removed the pre-research assessment by instructors, thus providing time for instructors to observe and interact with students prior to a first assessment at the early/mid-research point. E-CURE instructors were also able to score "not yet able to observe" if they had not had a chance to observe a student's work relative to a specific outcome component.
•  E-CURE particularly emphasizes opportunities for instructors to provide timely online feedback to students to help them enhance their metacognitive skills.
• To help students further develop their metacognitive skills, twelve metacognition exercises were developed to complement E-CURE implementation.

Project Findings

E-CURE was developed and pilot-tested in 54 courses at 23 institutions over a five-year period between 2019 and 2023 in. Project findings show that:

1. E-CURE introduced student participants to a comprehensive list of competencies and skills they would need in order to succeed in graduate work and/or the workplace (see:  E-CURE outcome categories and Table 1, below).

2. Almost all E-CURE assessment components yielded statistically significant self-reported student gains (see Tables 2 and 4, below) and all but one E-CURE assessment component saw statistically significant instructor-reported student gains (see Tables 3 and 5, below).

The next three findings are based on data from student questionnaires, faculty surveys, and conversations between CURE instructors and PI Singer.

3. Student-instructor conversations and selected metacognitive exercises contributed to the development and enhancement of student metacognitive skills, characterized by learners becoming more aware of what learning strategies they are pursuing and why, and then using that awareness to make intentional adjustments to those strategies in order to learn more effectively. These conversations also helped most students confirm their plans for graduate school or employment in their major field.

4. E-CURE made it easier for instructors to identify the academic strengths and weaknesses of their students, enabling the instructors to focus their instruction more productively and  help students gain new insights into the relative efficacy of their learning strategies. 

5. E-CURE did not create an undue burden on students or instructors.

Detailed Findings

For the full report including detailed findings, data, and figures, view or download the PDF: Preparing Future STEM Technicians by Using Assessment as a Teaching and Learning Tool in Course-based Research at Community Colleges (Acrobat (PDF) 368kB Oct19 23)