InTeGrate Modules and Courses >Critical Zone Science > Module 7: Humans in the Critical Zone > Unit 7.3 - Panel Review
 Earth-focused Modules and Courses for the Undergraduate Classroom
showLearn More
hideHide
These materials are part of a collection of classroom-tested modules and courses developed by InTeGrate. The materials engage students in understanding the earth system as it intertwines with key societal issues. The materials are free and ready to be adapted by undergraduate educators across a range of courses including: general education or majors courses in Earth-focused disciplines such as geoscience or environmental science, social science, engineering, and other sciences, as well as courses for interdisciplinary programs.
Explore the Collection »
How to Use »

New to InTeGrate?

Learn how to incorporate these teaching materials into your class.

  • Find out what's included with each module
  • Learn how it can be adapted to work in your classroom
  • See how your peers at hundreds of colleges and university across the country have used these materials to engage their students

How To Use InTeGrate Materials »
show Download
hideHide
The instructor material for this module are available for offline viewing below. Downloadable versions of the student materials are available from this location on the student materials pages. Learn more about using the different versions of InTeGrate materials »

Download a PDF of all web pages for the instructor's materials

Download a zip file that includes all the web pages and downloadable files from the instructor's materials

Unit 7.3 - Panel Review

Ashlee Dere (University of Nebraska - Omaha) and Adam Hoffman (University of Dubuque)

Summary

Peer review is an important process in determining priorities for scientific research. Students will participate in a panel review of proposals for new CZOs and as a class decide on the proposal most worthy of funding. Students will read proposals, craft a detailed review of the merits and limitations of the proposal, and then discuss the proposals during an in-class panel review. Proposals will be evaluated on how well the proposed CZO would help address global challenges and advance Critical Zone science and require students to use knowledge gained in previous modules to assess and communicate which proposals meet these goals.

Share your modifications and improvements to this activity through the Community Contribution Tool »

Learning Goals

By the end of this unit students will be able to:

  • Evaluate scientific proposals from a Critical Zone perspective.
  • Articulate the strengths and weaknesses of a scientific proposal.
  • Determine which proposal has the strongest scientific and societal merit.
  • Communicate the significance of a proposed CZO to solving grand challenges.

Context for Use

The panel review activity should be completed the final day of the course, 1 - 2 weeks after the summative assessment proposals have been submitted to allow students time to read and respond to the proposals before this last class. This activity is intended to strengthen the summative assessment project by encouraging students to evaluate the work of their peers and assess which proposals most clearly articulate a plan for addressing grand challenges. Students will need to draw on the knowledge and skills acquired throughout the course to critically examine the proposals and communicate their reviews.

Description and Teaching Materials

Background

The National Science Foundation (NSF) in the United States, along with many other funding organizations, use a proposal and peer review process for determining which science projects should be funded. At NSF, proposals to conduct scientific research are submitted and then reviewed by a panel of experts in the field who provide advice and recommendations as to which proposals should be funded. The process is intended to determine which proposals have the potential to advance or transform our scientific knowledge while also contributing to achieving broader societal goals. Typically, a proposal is assigned a primary and secondary reviewer who are tasked with reviewing the proposal in detail and providing written comments on the merits of the proposed project. Other panelists will also read the proposal but the primary and secondary reviewers will provide the most feedback. The panelists read and evaluate the proposals prior to arriving at the panel review, which often lasts for 2-3 days. Once all panelists are together in the room, each proposal is discussed with the primary reviewer leading the initial discussion. After all proposals have been discussed, the proposals are ranked in order of funding priority. Proposals will then typically be funded in order of ranking until funds are exhausted, at which point unsuccessful proposals are returned to the authors with suggestions from the panel on how to strengthen the proposal.

For example, NSF uses the following merit review criteria (http://www.nsf.gov/pubs/policydocs/pappguide/nsf16001/gpg_3.jsp#IIIA ):

When evaluating NSF proposals, reviewers will be asked to consider what the proposers want to do, why they want to do it, how they plan to do it, how they will know if they succeed, and what benefits could accrue if the project is successful. These issues apply both to the technical aspects of the proposal and the way in which the project may make broader contributions. To that end, reviewers will be asked to evaluate all proposals against two criteria:

  • Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge.
  • Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.

The following elements should be considered in the review for both criteria:

1. What is the potential for the proposed activity to:

a. Advance knowledge and understanding within its own field or across different fields (Intellectual Merit)?

b. Benefit society or advance desired societal outcomes (Broader Impacts)?

2. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?

3. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?

4. How well qualified is the individual, team, or organization to conduct the proposed activities?

5. Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?

More details about the proposal merit review process can be found at the following links:

In completing the summative assessment activity, students have practiced crafting a research proposal for a new CZO that will help us address global challenges. Here, students will participate in a review panel that will discuss the merits of each proposal to determine which proposed CZO would be most likely to advance knowledge and benefit society.

Day 1 - (75 minutes)

Pre-class Homework:

  • Students will need to read review criteria, read at least 3 proposals and craft a detailed review of a proposal prior to class. Depending on class size, it may be advisable to organize students into groups to reduce the number of proposals each person will read. Ideally, each proposal would have one student assigned as the primary reviewer, and a different student as a secondary reviewer so that all proposals have at least two detailed reviews. Have students review the merit review criteria from NSF by browsing Section A Parts 1 and 2 at http://www.nsf.gov/pubs/policydocs/pappguide/nsf16001/gpg_3.jsp#IIIA or the other websites listed above. Students will also need to prepare a written response with their proposal review and bring it to class for use in the class discussion.
  • Proposal evaluations can be completed using the following worksheet: Proposal Review Worksheet (Microsoft Word 2007 (.docx) 17kB Dec26 16)

Overview - Review Panels (5 min)

  • Provide a brief review of the motivation and criteria for reviewing the proposals, and the need for the review process. Remind students that the discussion should be constructive and focus on the ideas outlined in the proposal, not individuals.

Activity 7.5 - Panel Review (60 min)

  • If possible, organize the classroom so students are facing each other for the panel discussion. Depending on the total number of proposals, set a time limit for discussion of each proposal, leaving at least 10 minutes at the end to discuss the ranking of top proposals. The primary student reviewer should initiate the discussion by restating the main ideas of the proposal and then highlight their assessment of how well the proposal would advance science and address grand challenges. Encourage students to summarize their main points rather than read directly from their written review. Then open up the discussion to the rest of the class, keeping in mind that the goal is to identify the strongest proposals. Once all proposals have been discussed, ask the students to work together to rank the proposals based on their scientific merit and potential to benefit society.

Wrap-up - Funded Proposal (10 min)

  • Ask students to provide feedback on the review process, both as an individual reviewer reading the proposals and their experience discussing the proposals on the review panel. Aim to identify what made a strong proposal and why the funded CZO would be best suited for addressing grand challenges.

Teaching Notes and Tips

Instructors will need to determine the number of proposals for each student to read and how to manage groups depending on the class size and level. The aim is for students to have a chance to review a proposal, provide comments and then discuss the proposal merits with their peers. Depending on class size, it may be advisable to organize students into smaller review groups to reduce the number of proposals each person will read. It may also be helpful to provide some time for students to compare their reviews with someone else who read the same proposal prior to the larger group discussion -- to help them feel more confident in contributing their ideas to the group. To allow more time for discussions, it may also be advisable to complete this activity during a scheduled final exam time rather than a regular 75 minute class time.

In addition, the activity requires that summative assessment proposals be submitted prior to the final week of courses to allow time for the instructor to assign reviewers and for the students to both read and respond to the proposals.


Assessment

Students will be assessed on the strength of their proposal reviews and on their participation in the panel review discussions.

  • Proposal Review
  • Panel Review Discussion

References and Resources

Already used some of these materials in a course?
Let us know and join the discussion »

Considering using these materials with your students?
Get advice for using GETSI modules in your courses »
Get pointers and learn about how it's working for your peers in their classrooms »

These materials are part of a collection of classroom-tested modules and courses developed by InTeGrate. The materials engage students in understanding the earth system as it intertwines with key societal issues. The collection is freely available and ready to be adapted by undergraduate educators across a range of courses including: general education or majors courses in Earth-focused disciplines such as geoscience or environmental science, social science, engineering, and other sciences, as well as courses for interdisciplinary programs.
Explore the Collection »