Peer Review of Teaching Activities

Based on input from participants at the 2017 Teaching Computation in the Sciences Using MATLAB workshop, the Teaching Computation community has begun conducting a peer review of MATLAB^® based teaching activities submitted via the Teaching Computation in the Sciences workshops and website, as well as those in the existing SERC collections. The activity peer review is modeled on the review process developed by the Teaching Mineralogy, Petrology, and Geochemistry in the 21st Century workshop and expanded upon through On the Cutting Edge and the Teach the Earth portal. The process involves members of the Teaching Computation in the Sciences community reviewing activities using a rubric and set of criteria to help calibrate reviewer scoring. Much like the On the Cutting Edge review, there are five elements of each activity that get reviewed during the process:

• computational, quantitative, and scientific accuracy
• alignment of goals, activity, and assessment
• pedagogic effectiveness
• activity robustness (usability and dependability of all activity components)
• completeness of the ActivitySheet web page for the activity

Each activity element receives scores from 1 (problematic) to 4 (exemplary) and constructive comments from two independent reviewers. Based on the results of the review, each activity receives one of the following four possible ratings:

• Exemplary: Activities in the Exemplary Collection have received Exemplary or Very Good scores in all five categories and must have been rated Exemplary in at least 3 of the 5. It is expected that no more than 10-20% of the activities in the collection would be awarded an Exemplary rating in this process. Aggregate scores of 18-20 are required.
• Passed Review: Activities that pass review have received positive reviews in all five categories, consisting of mostly of Very Good scores with possible Exemplary or Adequate scores in one or more areas. Authors with activities in this collection will receive recommendations from the reviewers and associate editors for ways of improving their activity such that it can be brought into the Exemplary Collection. Scores of 12-17 are required.
• Activity Idea: An activity with this rating contains the nucleus of a good teaching activity in the materials that were submitted, but in its current form does not contain sufficient information to be able to be widely used in science classes. Authors will be encouraged to invest energy in further developing the activity so that it become part of the Reviewed Collection. These are nucleus of good ideas that could be further developed.
• Deaccession: These activities contain serious deficiencies which would be difficult or impossible to remedy. They will be removed from the teaching activities collection entirely.

The review process is managed by an editor selected from the Teaching Computation in the Sciences community as well as Teaching Computation in the Sciences leadership. The editor manages activity assignments and reviewer communications, and is ultimately responsible for the final scoring of activities and communication of results to activity authors.

Rubric and Criteria

Computational, Quantitative, and Scientific Accuracy

Context: Does this activity cover a topic that is important to learning essential concepts, content or skills related to computation and quantitative thinking in the sciences; is the underlying content accurate, sufficiently developed to enhance learning, and unlikely to lead to misinterpretations?

• Exemplary—Excellent coverage of important computation topic(s) that will lead to enhanced student learning; this teaching activity includes concepts, content, or skills in a comprehensive manner, is well-referenced, avoids inaccuracies or misleading information, and is in accord with contemporary understanding of this topic.
• Very Good—this teaching activity covers concepts, content or skills with sufficient detail to impart understanding, but some background information is missing that could more completely develop the content and place it in contexts that are necessary for understanding.
• Adequate—this teaching activity introduces concepts and content in a rudimentary way, with only a minimal coverage of the underlying principles/methods; lack of detail may introduce inaccuracies or misinterpretations.
• Problematic—the underlying principles presented in this teaching activity are not in accord with contemporary understanding of this topic.

Alignment of Goals, Activity, and Assessment

Context: Are the learning goals, teaching activities, and assessments well-aligned such that they will a) enhance learning and b) provide evidence that learning has been achieved? (Note: learning goals may include content or concept mastery, skill development, affective aspects, or metacognition).

• Exemplary—learning goals are clearly stated, the activity itself has been developed to specifically address these goals, and learning outcomes assessments will reveal if the learning goals have been achieved.
• Very Good—teaching activities will likely lead to identified learning goals will likely lead to identified learning outcomes and assessments are identified, but could be further developed to more completely demonstrate learning outcomes by students.
• Adequate—Unclear if learning goals will be achieved; rudimentary mention of learning goals, assessments and learning outcomes are presented but few details are presented that would allow demonstration that student learning has been achieved.
• Problematic—learning goals and assessments are either absent or not aligned in ways that will enable learning.

Pedagogic Effectiveness

Context: This activity uses instructional practices appropriate for applying computational thinking to the subject matter (e.g., calculating average temperature from historical data and plotting the results, such as might be covered in a Climate Change, Atmospheric Sciences, or related course). It will enable and reinforce learning of relevant math, science, and programming principles

• Exemplary— A strong exercise that will promote student learning; is engaging and motivating for students; builds on prior competencies and knowledge; promotes higher-order thinking skills; encourages reflections on knowledge acquired and gaps (in science, math, and/or coding).
• Very Good—Students will likely be engaged; most components of good activity design are present (e.g., active learning, extends student competencies beyond rote memorization or ability to follow directions, reflections on learning and gaps).
• Adequate—May lead to some aspects of student learning; some elements of good activity design are present but does not constitute a comprehensive learning experience, e.g. this activity requires basic application of knowledge or skills, without challenging students to go beyond the instructions; does not require independent thought to reflect on connections with other course content or prior knowledge and skills.
• Problematic—rote application only; no evidence of requiring independent thought, or no awareness of how the exercise fits into the larger curriculum.

Robustness

Context: Ease of use; does this activity include all components needed to successfully complete the exercise; all components are stable and they work. (e.g., math works, code runs, results display)

• Exemplary—All activity components are present, all work well, and are in a stable form (e.g., activity assignment, data, images, code scripts or functions, URLs, . . .). Educators will have confidence that they can directly apply this in their class, can be used "as is". Good presentation of information preferred, such as instructive illustrations or a runnable Live Script format.
• Very Good—The activity works very well, with only minor glitches but users can figure it out with little effort; the required components of this exercise are available and can be used with little or no modifications.
• Adequate—the activity works to some degree, but users have to expend energy to make all the components work. The components of this exercise are available but may need to be modified, augmented, or updated to successfully complete the exercise or use beyond a toy demo.
• Problematic—Doesn't work, components work so poorly, are missing or not accessible that they impede use, preventing successful completion of the activity or producing and visualizing relevant results.

Completeness of the ActivitySheet web page for the activity

Context: The ActivitySheet includes sufficient information to allow an instructor/student to make an informed decision about whether or not this is an appropriate activity to use in their learning situation.

• Exemplary—the ActivitySheet provides all information needed for instructors to select this exercise for use in a class; and, provides guidance about how to optimally use this exercise; empowers instructors to effectively use this activity to enhance student learning. Information is provided to help instructors adopt or adapt to their own situation, including teaching tips.
• Very Good—most of the information is present in the ActivitySheet, but some of the fields could be further developed as noted in the text box below.
• Adequate—basic information about this activity is provided, but more detail is required to help an instructor know what the activity entails and how to best use this in their class.
• Problematic—Not useful critical information is missing.