Working with Your Assessment Consultant

Working with the Assessment Team

To assist the materials development teams in developing materials that reach the goals of the InTeGrate project, each team has an assigned consultant from the assessment team. One role of the assessment team member is to help development teams meet project standards as the team is developing materials. Additionally, the assessment team member will advise teams to ensure that the teaching materials they develop are meeting their desired learning goals for the module/course.

The assessment consultant will complete preliminary reviews of materials over several designated checkpoints as indicated on the timeline. The assessment team has been trained in using the InTeGrate Materials Development and Refinement Rubric, which materials must pass prior to testing and publication. The assessment team must confirm that materials also meet their stated learning goals as well as overarching InTeGrate goals through analysis of data collected from classrooms.

Assessment Data

The assessment data collected during Phase 2 (classroom pilot) will help you individually gauge and validate the learning that is taking place in your classroom, plan informed revisions with your team, and collectively contribute to the InTeGrate learning goals. The project-level assessment data will be submitted to the InTeGrate project for use by the assessment team and the project evaluation team, while you will retain your formative assessments for the purposes of planning your module/course revisions with your team. The data that will be collected by the project include:

• Course information
• Summative assessment(s)
• Pre- and post attitudinal surveys
• Pre- and post content assessments (GLE) (multiple choice pre/post and post-only essay questions)
  
• Student preparedness and engagement data
• Student demographics

Pre- and post content (GLE) and attitudinal surveys

There is a common 8-question pre/post multiple choice test (GLE) that every InTeGrate student participant takes at the beginning and end of each course that uses InTeGrate materials (whether testing a module or an entire course). This provides baseline data used by the project for comparative purposes and as a measure of geosciences literacy. Similarly, there is a pre/post attitudinal survey as well as two common essay questions asked at the end of each course using InTeGrate materials. These data are used to gauge student progress toward meeting overall InTeGrate goals. For comparative purposes, these data must be collected at the beginning and end of the course in which the materials are used rather than immediately after InTeGrate instruction. Details about how and when to collect these data are described in data collection process. Since collecting student data requires institutional approval, teams need to start applying for permission prior to the start of the semester in which you plan to pilot the materials. Depending on timing, this may occur after Checkpoint 3 or 4. Your webteam liaison can help you determine when you should complete these steps. Read more about Collecting Data in your Classroom.

Module/Course Summative Assessments

Module/course summative assessments play a critical role for the InTeGrate project in evaluating if instructional materials and strategies are meeting the stated learning goals. In addition, they provide faculty using your finished curriculum materials a better understanding of what students are learning from them. These assessments must be:

• Graded by the faculty of the pilot testing classroom based upon a rubric,
• Summative and conclusive, that is, measuring the level of understanding when learning related to that module/course goal is completed, and
• Used in the same fashion in all materials pilot-testing classrooms.

Connections should be drawn between these assessments and the module/course learning goals as well as the overarching InTeGrate guiding principles. Just as the goals should be appropriate for the duration of the module, the summative assessments should be sufficient to demonstrate achievement of these learning goals for this same duration. The student work for these summative assessments will be provided to the InTeGrate project for each pilot-testing classroom. The assigned assessment consultant can provide guidance on this item.

The format for the assessments will vary depending on what aligns best with the given module/course. the assessments should demonstrate the achievement of the learning goals for the module/course. The team, the team leaders and the assessment consultant will work collaboratively to ensure that this is case. The assessment might be an essay question, lab report, writing assignment, class project or concept map designed to show holistic, higher order learning. All authors must use the same assessment during testing.

For details on the process for how you will submit summative assessments to InTeGrate, please see Collecting Data in Your Classroom.

More detailed descriptions of various assessment tools are available from On the Cutting Edge. Copies of all the student work for the summative assessments must be sent to the InTeGrate project from all pilot-testing classrooms.

Other Assessments

Your curriculum will include a variety of summative and formative assessments that collectively document various levels of learning achieved by your students. Student responses to these assessments should be archived when possible. After pilot testing is complete, these assessments will be particularly useful to help your team make revisions to your module or course. They will also be helpful when developing the "case study" that includes suggestions to other faculty interested in using your materials. Additionally, InTeGrate encourages you to publish the results of your curriculum development and testing efforts in a peer-reviewed journal. To do so you, will need to be able to document the learning process with the evidence provided by the formative and summative assessments of student work that you collect.

External Evaluation

InTeGrate also has an external evaluation team who will use the data collected to measure the project's overall effectiveness at a number of specific project goals (read more about the specific goals of the external evaluation). These data will be incorporated into project research results in aggregate. Presentation of the material will be confined to project work groups, professional publications, and conferences. When individual quotes are used to illustrate important points, names will not be identified.

Data Collection and Timing

Assessments will be collected both by module authors piloting their own material and by approved materials testers who were not involved in the authoring process. Module authors will need to get IRB approval prior to testing materials in the classroom (this should be initiated at checkpoint 2 in the materials development process). Read more about the details of the data collection process, including information about when each data collection step should be completed.