Working with Your Assessment Consultant

Working with the Assessment Team

To assist the materials development teams in developing materials that reach the goals of the InTeGrate project, each team has an assigned consultant from the assessment team. One role of the assessment team member is to help development teams meet standards as the team is developing materials. Additionally, the assessment team member will ensure that embedded assessments will provide the data needed for evaluation and that the teaching materials are meeting their desired learning objectives. They are available to answer questions, consult on ideas, and make suggestions. Teams will work collaboratively with the assessment consultant through development, testing and implementation of materials. An early meeting between the materials development team and the assessment consultant is useful in establishing strategies for working together.

The assessment consultant will complete preliminary reviews of materials over several designated checkpoints indicated on the timeline. The assessment team has been trained in using the InTeGrate Materials Development and Refinement Rubric, which materials must pass prior to testing and publication. The assessment team must confirm that materials also meet their stated learning goals as well as overarching InTeGrate goals through analysis of data collected from classrooms.

Assessment Data

When piloting your InTeGrate materials, a variety of assessment data will be collected, which will be used by the InTeGrate assessment team and the evaluation team. These data will include:

• Course information
• Embedded assessments
• Pre- and post attitudinal surveys
• Pre- and post content assessments (GLE)
  
• Student preparedness and engagement data
• Student demographics

The assessment team will analyze these data to assess to what extent the module or course was successful in meeting its objectives of student learning. The assessment team will make recommendations as to whether the module can be made available to the public as part of the InTeGrate project. They will also assist the materials development team in determining what revisions, if any, are needed.

The assessment team will use the pre- and post-instruction student attitudinal survey (GLE+) and content responses (GLE) to assess to what extent the module addresses the goals of the InTeGrate project and to what extent all the materials being developed are meeting the project goals. They will also use the student responses to refine the GLE instrument.

External Evaluation

InTeGrate also has an external evaluation team who will use the data collected to measure the project's overall effectiveness at:

1. Expanding the number of students who enroll in geoscience courses. The evaluation team will look at patterns and trends in numbers of enrolled students, building on data from AGI. They will track this information by student demography.
2. Achieving enhanced learning. The evaluation team will use the same data sets as the assessment team but will take a cross-institutional, project-wide view of the data. They will examine whether learning gains are evenly distributed across demographic groups and institution types.
3. Achieving significant progress towards addressing the national challenge of environmental sustainability. The evaluation team will analyze the motivation and career interest surveys to see what connections students are making between the relevance of the materials to addressing these challenges and their engagement in participating toward a solution.

These data will be incorporated into our research results in aggregate. Presentation of the material will be confined to project work groups, professional publications, and conferences. When individual quotes are used to illustrate important points, names will not be identified.

Embedded Assessments

In addition to providing faculty using your finished curriculum materials a better understanding of what students are learning from the materials, the embedded assessments play a critical role for the InTeGrate project in evaluating if instructional materials and strategies are meeting the stated learning objectives. These assessments must be:

• Part of the flow of class,
  
• Graded by the faculty of the pilot testing classroom based upon a rubric,
• Summative, that is, measuring the level of understanding when learning related to that objective is completed, and
• Used in the same fashion in all materials pilot-testing classrooms.

Your team will choose three embedded assessment measures, for which will be provided to the InTeGrate project. These should provide more than one page of student work. The assigned assessment consultant can provide guidance on this item.

The format for the embedded assessments will vary depending on what aligns best with the given module. It could include quizzes, a subset of exam questions, or homework. For complete courses that have both lab or field components, the embedded assessments could be collected as part of the lab or field experience. However authors decide to implement the embedded assessments, all materials testers should employ the same method.

For details on the process for how you will submit embedded assessments to InTeGrate, please see Collecting Data in Your Classroom.

More detailed descriptions of various assessment tools are available from On the Cutting Edge. Copies of all the student work for the embedded assessments must be sent to the InTeGrate project from all pilot-testing classrooms.

Data Collection and Timing

Assessments will be collected both by module authors piloting their own material and by approved materials testers who were not involved in the authoring process. Module authors will need to get IRB approval prior to testing materials in the classroom (this should be initiated at checkpoint 2 in the materials development process). Read more about the details of the data collection process, including information about when each data collection step should be completed.