Understanding Uncertainty in Ecological Forecasts

×

Provenance: Tadhg Moore, Virginia Polytechnic Institute and State Univ
Reuse: This item is offered under a Creative Commons Attribution-NonCommercial-ShareAlike license http://creativecommons.org/licenses/by-nc-sa/3.0/ You may reuse this item for non-commercial purposes as long as you provide attribution and offer any derivative works under a similar license.

• Define ecological forecast uncertainty
• Explore the contributions of different sources of uncertainty (e.g., model parameters, model driver data) to total forecast uncertainty
• Understand how multiple sources of uncertainty are quantified
• Identify ways in which uncertainty can be reduced within an ecological forecast
• Describe how forecast horizon affects forecast uncertainty
• Explain the importance of specifying uncertainty in ecological forecasts for forecast users and decision support

This entire module can be completed in one 2 to 3-hour lab period, two 75-minute lecture periods, or three 1-hour lecture periods for introductory undergraduate students in Ecology, Environmental Science, Ecological Modelling, and Quantitative Ecology classes. This module can be coupled with other Macrosystems EDDIE ecological forecasting modules: Module 5 "Introduction to Ecological Forecasting"; Module 7 "Using Data to Improve Ecological Forecasts"; or Module 8 "Using Ecological Forecasts to Guide Decision-Making". There are two versions of this module, depending on whether instructors wish to incorporate R coding into their course curriculum. For course curricula that do not include computer coding, instructors can teach the R Shiny version of the module, which is a point-and-click web interface with interactive data visualization. For course curricula that include computer coding and R, instructors can teach the RMarkdown version of the module, which will ask student to read and revise R code to complete module activities. We found that teaching this module in one longer lab section with short breaks was more conducive for introductory students than multiple 1-hour lecture periods. Please see the instructor manual for detailed recommendations about module timing for different class schedule types.

Quick overview of the activities in this module

See the instructor manual, provided below, for a step-by-step guide for carrying out this module. A student handout describing Activities A, B, and C, and an instructor PowerPoint are also provided.

• Activity A: Students select a NEON site, visualize water temperature data at that site, build different models to simulate water temperature for their chosen NEON site, and generate forecasts without uncertainty.
• Activity B: Students generate multiple forecasts of water temperature with different sources of uncertainty and examine how uncertainty differs among models.
• Activity C: Students generate forecasts that include all sources of uncertainty and partition the contribution of different sources of uncertainty for their forecasts with different models. Students also complete a management scenario case study exploring how the representation of forecast uncertainty affects decision-making.

Why macrosystems ecology and ecological forecasting?

Macrosystems ecology is the study of ecological dynamics at multiple interacting spatial and temporal scales (e.g., Heffernan et al. 2014). For example, global climate change can interact with local land-use activities to control how an ecosystem changes over the next decades. Macrosystems ecology recently emerged as a new sub-discipline of ecology to study ecosystems and ecological communities around the globe that are changing at an unprecedented rate because of human activities (IPCC 2013). The responses of ecosystems and communities are complex, non-linear, and driven by feedbacks across local, regional, and global scales (Heffernan et al. 2014). These characteristics necessitate novel approaches for making predictions about how systems may change to improve both our understanding of ecological phenomena as well as inform resource management.

Forecasting is a tool that can be used for understanding and predicting macrosystems dynamics. To anticipate and prepare for increased variability in populations, communities, and ecosystems, there is a pressing need to know the future state of ecological systems across space and time (Dietze et al., 2018). Ecological forecasting is an emerging approach which provides an estimate of the future state of an ecological system with uncertainty, allowing society to prepare for changes in important ecosystem services. Ecological forecasts are a powerful test of the scientific method because ecologists make a hypothesis of how an ecological system works; embed their hypothesis in a model; use the model to make a forecast of future conditions; and then when observations become available, assess the accuracy of their forecast, which indicates if their hypothesis is supported or needs to be updated. Forecasts that specify uncertainty will be most useful for aiding decision-making. Consequently, macrosystems ecologists are increasingly using ecological forecasts to predict how ecosystems are changing over space and time.

In this module, students will generate an ecological forecast for a NEON site and quantify the different sources of uncertainty within their forecast. This module will introduce students to the concept of uncertainty within an ecological forecast; where uncertainty in a forecast comes from; how uncertainty can be quantified within a forecast; and how uncertainty can be managed.

Workflow for this module

Shiny app version, for either in-person or virtual instruction:

1. Instructor chooses method for accessing the R Shiny app (Regardless of which option you pick, all module activities are the same!):
   1. In any internet browser, go to: https://macrosystemseddie.shinyapps.io/module6/
      1. This option works well if there are not too many simultaneous users (<20)
      2. The app generally does not take a long time to load but requires consistent internet access
      3. It is important to remind students that they need to save their work as they go, because this webpage will time-out after 15 idle minutes. It is frustrating for students to lose their progress, so a good rule of thumb is to get them to save their progress after completing each objective
   2. The most stable option for large classes is downloading the app and running locally, see instructions at:https://github.com/MacrosystemsEDDIE/module6
      1. Once the app is downloaded and installed (which requires an internet connection), the app can be run offline locally on students' computers
      2. This step requires R and RStudio to be downloaded on a student's computer, which may be challenging if a student does not have much R experience (but this could be done prior to instruction by an instructor on a shared computer lab)
      3. If you are teaching the module to a large class and/or have unstable internet, this is the best option
2. Give students their handout ahead of time to read over prior to class or ask students to download the handout from the module Shiny app page when they arrive to class. There is an optional pre-class reading assignment and questions that students may complete prior to arriving to class. During class, the module is set up for students to complete discussion questions in the handout (Word document) as they navigate through the R Shiny app activities. As they navigate through the app, students will be prompted to answer questions in their handout, as well as download plots that they generate within the app and copy-paste them into their handout. The handout can be submitted to the instructor for potential grading.
3. Instructor gives a brief PowerPoint presentation that introduces ecological forecasting, sources of forecast uncertainty, and the different models they will be using (~30 mins).
4. After the presentation, the students divide into pairs. Each pair selects their own NEON site and visualizes their site's data (Activity A Objectives 1 and 2). The two students within a pair each build their own different models for predicting water temperature (Activity A Objective 3), and generate forecasts with no uncertainty using each of their models (Activity A Objective 4). For virtual instruction, we recommend putting two pairs together (n=4 students) in separate breakout rooms during this activity so the two pairs can compare results.
5. The instructor can ask students to wait until all students are finished Activity A and then they will all begin Activity B together. For virtual instruction, this would entail having the students come back to the main room for a short check-in.
6. In Activity B, the students work in their pairs to generate forecasts which include different sources of uncertainty (Activity B Objectives 5-8). Students may compare their forecasts with their partners and work together to answer questions embedded throughout this activity about why the different models are affected differently by the different sources of uncertainty.
7. In Activity C, student pairs generate forecasts including all sources of uncertainty and compare how the contributions of different sources of uncertainty to total forecast uncertainty varies among models (Activity C Objective 9). They then complete a management scenario individually and discuss with their partner how the uncertainty visualizations provided in the scenario affected their management decisions (Activity C Objective 10).

RMarkdown version, for either in-person or virtual instruction:

1. Prior to class, the instructor chooses a method for students to access and run the RMarkdown module (Regardless of which option you pick, all module activities are the same!)
   1. To access a version of the module which asks students to read and revise code to complete module activities, navigate to: https://github.com/MacrosystemsEDDIE/module6_R 
      1. This version is recommended for students and instructors with prior R coding experience
      2. To run the module, students will need R and RStudio downloaded on their computers
      3. Students can run the RMarkdown version of the module by either:
         1. downloading a zip file of the code from GitHub (easiest option) or 
         2. creating a GitHub account, forking the GitHub repository, and creating an RProject for the repository (advanced option)
2. In class, the instructor gives a brief PowerPoint presentation that introduces ecological forecasting, sources of forecast uncertainty, and the different models students will be using (~30 mins).
3. After the presentation, the students divide into pairs to work through the RMarkdown. For virtual instruction, we recommend putting two pairs together (n=4 students) in separate breakout rooms during this activity so the two pairs can compare their work.
4. We recommend regular check-ins with students as they work through the code, and no later than at the end of Activity A (the last task in Activity A is Objective 4. Generate a deterministic forecast (without uncertainty) in the RMarkdown). The instructor can ask students to wait until all students are finished with Objective 4 and then they will all begin Activity B, Objective 5 together. For virtual instruction, this would entail having the students come back to the main room for a short check-in.
5. In Activity B, the students work in their pairs to generate forecasts which include different sources of uncertainty (Activity B, Objectives 5-8). Students may compare their forecasts with their partners and work together to answer questions embedded throughout this activity about why the different models are affected differently by the different sources of uncertainty.
6. In Activity C, student pairs generate forecasts including all sources of uncertainty and evaluate how the contributions of different sources of uncertainty to total forecast uncertainty varies (Activity C, Objectives 9-10). They then build, fit, and calculate the uncertainty associated with a second forecast model of their own choosing and compare how the contributions of different sources of forecast uncertainty vary between the two models (Activity C, Objectives 11-13).

Teaching Materials:

• R shiny app: https://macrosystemseddie.shinyapps.io/module6/
  • To run the R Shiny app locally on your own computer, please see the instructions on the GitHub page: https://github.com/MacrosystemsEDDIE/module6
• RMarkdown GitHub repository: https://github.com/MacrosystemsEDDIE/module6_R 
  • This repository includes a README file with instructions about how to run the module
• Student_Handout_Module6.docx (Microsoft Word 2007 (.docx) 1.9MB Dec15 23) - Handout with questions for students to complete
• Instructor_Manual_Module6.docx (Microsoft Word 2007 (.docx) 3.8MB Dec15 23) - Instructor manual and troubleshooting for the module
• Instructor_PowerPoint_Module6.pptx (PowerPoint 2007 (.pptx) 5.2MB Mar5 24) - PowerPoint presentation to introduce core concepts & module activities 
  • Getting_Started_With_Shiny_Module6.pptx (PowerPoint 2007 (.pptx) 7MB Dec15 23) - Additional PowerPoint slides that provide a basic orientation to using an R Shiny app

Important Note to Instructors:

The R Shiny app and RMarkdown used in this module are regularly updated, so these module instructions will periodically change to account for changes in the code. If you have any questions or have other feedback about this module, please contact the module developers (see "We'd love your feedback" below).

We highly recommend that instructors familiarize themselves with the Shiny app or RMarkdown prior to the lesson. This will enable you to be more prepared to answer student questions.

Student understanding is assessed using interactive questions embedded throughout the R Shiny application or RMarkdown file. To answer these questions, students will need to manipulate and visualize data, fit models, generate forecasts, and create plots. For the Shiny app version of the module, student responses are recorded by downloading and copy-pasting plots and typing answers into a Word document report that they download from the application. For the RMarkdown version of the module, student responses are recorded directly into the RMarkdown, which can be knitted into a formatted html file.

• Activity A: Students identify patterns in air and water temperature data as well as relationships between these variables; describe the structures of the simple forecast models they fit and how well these models fit their data, and use deterministic forecasts (without uncertainty) to make predictions of future water temperature.
• Activity B: Students describe how different sources of forecast uncertainty differ among their four forecast models, interpret why the sources differ, and identify methods for reducing different types of forecast uncertainty.
• Activity C: Students compare the relative contributions of different sources forecast uncertainty among forecast models and identify how uncertainty quantification and communication affect management decision-making through a case study exercise.

Optional pre-class readings and videos:

Articles:

• Recommended for this module: Pielke, R. A. (1999). Who decides? Forecasts and responsibilities in the 1997 Red River flood. Applied Behavioral Science Review, 7(2), 83–101. (Optional pre-class discussion questions associated with this paper can be found in the student handout, which is downloaded from the Shiny app).
• General background reading on ecological forecasting:
  • Silver, N. (2012) Chapter 6: How to drown in three feet of water. Pages 176-203 in The Signal and the Noise: Why so many Predictions Fail – but some Don't. Penguin Books.
  • Dietze, M., & Lynch, H. (2019). Forecasting a bright future for ecology. Frontiers in Ecology and the Environment, 17(1), 3. https://doi.org/10.1002/fee.1994
  • Dietze, M. C., Fox, A., Beck-Johnson, L. M., Betancourt, J. L., Hooten, M. B., Jarnevich, C. S., Keitt, T. H., Kenney, M. A., Laney, C. M., Larsen, L. G., Loescher, H. W., Lunch, C. K., Pijanowski, B. C., Randerson, J. T., Read, E. K., Tredennick, A. T., Vargas, R., Weathers, K. C., & White, E. P. (2018). Iterative near-term ecological forecasting: Needs, opportunities, and challenges. Proceedings of the National Academy of Sciences, 115(7), 1424–1432. https://doi.org/10.1073/pnas.1710231115

Videos:

• NEON's Ecological Forecast: The Science of Predicting Ecosystems
• Fundamentals of Ecological Forecasting Series
  • Why Forecast?
  • Uncertainty Analysis

Module authorship contributions: CCC and RQT conceived the idea of this module and acquired funding for this project. TNM, CCC, and RQT developed the learning objectives and website text. MEL developed RMarkdown activities with feedback from CCC and RQT. TNM and MEL developed the module activities and code for the module with feedback from CCC and RQT. TNM, MEL, and CCC developed student assessment questions and led module testing and collection of student assessment data. MEL and TNM developed the student handout, instructor PowerPoint, and instructor manual with feedback from CCC and RQT. MEL, TNM, CCC, and RQT worked with instructors of the module and integrated feedback into improving the module.