Assessment Data Feedback

Comments:

Reviewers of this question and rubric include: Emily Geraghty Ward,Barbara Bekken,Stuart Birnbaum

a. Feedback on the rubric: The rubric for the existing question worked reasonably because the three reviews showed acceptable inter-rater reliabilities, which is not common; however the role of the hydrologic cycle and tectonics was not made sufficiently explicit in the grading rubric. The raters found it more difficult and time-consuming to evaluate responses that were entirely text-based. It was easier to work with diagrams. The suggestions below for a revised question may account for this concern.

b. Feedback on the question: The question needs to clarify whether students should construct a rock cycle diagram (e.g., draw it), that incorporates the components of their rubric: above ground (including hydrologic), below ground, mass transfers, and energy transfers. Currently the question text on the InTeGrate site differs from the paper version, which also includes the rubric that students need to see. This needs to be added to the InTeGrate site. Currently, this question is very broad and incorporates material that students will likely know before engaging in this module. Therefore, the results from this question may not parse what students have learned in this module. To focus specifically on the challenging objectives in this module (Earth processes are the result of mass cycling and energy transfers from sources to sinks), we suggest that question be constrained to emphasize energy flows and mass cycling with less emphasis on the actual rock cycle. Perhaps give them a simplified 2D rock/hydrologic cycle and focus the question on identifying and describing mass and energy transfer throughout. This could be done by annotation, labeling, sketching and so on with specific emphasis on identifying both mass and energy transfers/transformations explicitly. Revising this question in this manner will also allow users to adapt it to an online platform, should they so desire.

c. Other comments: Most rubrics, by their nature, can constrain student thinking and may not account for creativity. They also can invite minimalist responses. When building a rubric, we encourage module designers to incorporate components such as "insightful," "logical", and so on. These correspond with the overarching goals of the InTeGrate project that focus on interdiciplinary problems, systems thinking, methods of inquiry, and habits of mind.

On a related note, please replace the key word "understand" in second bullet under goal #2 of the module objectives (e.g., explain or sketch or describe).

• The objectives (in purple) are aligned with the overall learning goals and the rubric is well-aligned with the objectives.
• Rubric problems: partial point values not described and this made it challenging for the assessment team to score
• Pt. 1 on rubric– we couldn't evaluate, so we just cut that out and reduced the total to 25 points.
• We noticed that few, if any, brochures did anything with ecosystem effects.
• We weren't sure what to do with the products that weren't really brochures. So we graded them in the same way as the others, not taking off points because of format.

When we compared our scores, one grader was systematically higher than the other. However, in general, the rubric and scoring separated well the excellent and terrible products; the greatest discrepancies were in the middle range of scores and would likely be remedied by more detailed information on the partial scores.