Understanding Uncertainty in Ecological Forecasts

This module was developed by Moore, T. N., Lofton, M.E., Carey, C.C. and Thomas, R. Q. 24 July 2023. Macrosystems EDDIE: Understanding Uncertainty in Ecological Forecasts. Macrosystems EDDIE Module 6, Version 2. http://module6.macrosystemseddie.org. Module development was supported by NSF grants DEB-1926050 and DBI-1933016.

Summary

Ecological forecasting is a tool that can be used for understanding and predicting changes in populations, communities, and ecosystems. Ecological forecasting is an emerging approach which provides an estimate of the future state of an ecological system with uncertainty, allowing society to prepare for changes in important ecosystem services. Forecast uncertainty is derived from multiple sources, including model parameters and driver data, among others. Knowing the uncertainty associated with a forecast enables forecast users to evaluate the forecast and make more informed decisions. Ecological forecasters develop and update forecasts using the iterative forecasting cycle, in which they make a hypothesis of how an ecological system works; embed their hypothesis in a model; and use the model to make a forecast of future conditions and quantify forecast uncertainty. There are a number of approaches that forecasters can use to reduce uncertainty, which will be explored in this module.

Used this activity? Share your experiences and modifications

Learning Goals

By the end of this module, students will be able to:

  • Define ecological forecast uncertainty
  • Explore the contributions of different sources of uncertainty (e.g., model parameters, model driver data) to total forecast uncertainty
  • Understand how multiple sources of uncertainty are quantified
  • Identify ways in which uncertainty can be reduced within an ecological forecast
  • Describe how forecast horizon affects forecast uncertainty
  • Explain the importance of specifying uncertainty in ecological forecasts for forecast users and decision support

Context for Use

This entire module can be completed in one 2 to 3-hour lab period, two 75-minute lecture periods, or three 1-hour lecture periods for introductory undergraduate students in Ecology, Environmental Science, Ecological Modelling, and Quantitative Ecology classes. This module can be coupled with other Macrosystems EDDIE ecological forecasting modules: Module 5 "Introduction to Ecological Forecasting"; Module 7 "Using Data to Improve Ecological Forecasts"; or Module 8 "Using Ecological Forecasts to Guide Decision-Making". We found that teaching this module in one longer lab section with short breaks was more conducive for introductory students than multiple 1-hour lecture periods. Please see the instructor manual for detailed recommendations about module timing for different class schedule types.

Description and Teaching Materials

Quick overview of the activities in this module

See the instructor manual, provided below, for a step-by-step guide for carrying out this module. A student handout describing Activities A, B, and C, and an instructor PowerPoint are also provided.

  • Activity A: Students build different models to simulate water temperature for their chosen NEON site and generate forecasts without uncertainty.
  • Activity B: Students generate multiple forecasts of water temperature with different sources of uncertainty and examine how uncertainty differs among models.
  • Activity C: Students generate forecasts that include all sources of uncertainty and partition the contribution of different sources of uncertainty for their forecasts with different models. Students also complete a management scenario case study regarding how representation of forecast uncertainty affects decision-making.

Why macrosystems ecology and ecological forecasting?

Macrosystems ecology is the study of ecological dynamics at multiple interacting spatial and temporal scales (e.g., Heffernan et al. 2014). For example, global climate change can interact with local land-use activities to control how an ecosystem changes over the next decades. Macrosystems ecology recently emerged as a new sub-discipline of ecology to study ecosystems and ecological communities around the globe that are changing at an unprecedented rate because of human activities (IPCC 2013). The responses of ecosystems and communities are complex, non-linear, and driven by feedbacks across local, regional, and global scales (Heffernan et al. 2014). These characteristics necessitate novel approaches for making predictions about how systems may change to improve both our understanding of ecological phenomena as well as inform resource management.

Forecasting is a tool that can be used for understanding and predicting macrosystems dynamics. To anticipate and prepare for increased variability in populations, communities, and ecosystems, there is a pressing need to know the future state of ecological systems across space and time (Dietze et al., 2018). Ecological forecasting is an emerging approach which provides an estimate of the future state of an ecological system with uncertainty, allowing society to prepare for changes in important ecosystem services. Ecological forecasts are a powerful test of the scientific method because ecologists make a hypothesis of how an ecological system works; embed their hypothesis in a model; use the model to make a forecast of future conditions; and then when observations become available, assess the accuracy of their forecast, which indicates if their hypothesis is supported or needs to be updated. Forecasts that specify uncertainty will be most useful for aiding decision-making. Consequently, macrosystems ecologists are increasingly using ecological forecasts to predict how ecosystems are changing over space and time.

In this module, students will generate an ecological forecast for a NEON site and quantify the different sources of uncertainty within their forecast. This module will introduce students to the concept of uncertainty within an ecological forecast; where uncertainty in a forecast comes from; how uncertainty can be quantified within a forecast; and how uncertainty can be managed.

Workflow for this module (for either in-person or virtual instruction):

  1. Instructor chooses method for accessing the Shiny app (Regardless of which option you pick, all module activities are the same!):
    1. In any internet browser, go to:https://macrosystemseddie.shinyapps.io/module6/
      1. This option works well if there are not too many simultaneous users (<20) 
      2. The app generally does not take a long time to load but requires consistent internet access
      3. It is important to remind students that they need to save their work as they go, because this webpage will time-out after 15 idle minutes. It is frustrating for students to lose their progress, so a good rule of thumb is to get them to save their progress after completing each objective
    2. The most stable option for large classes is downloading the app and running locally, see instructions at: https://github.com/MacrosystemsEDDIE/module6
      1. Once the app is downloaded and installed (which requires an internet connection), the app can be run offline locally on students' computers
      2. This step requires R and RStudio to be downloaded on a student's computer, which may be challenging if a student does not have much R experience (but this could be done prior to instruction by an instructor on a shared computer lab)
      3. If you are teaching the module to a large class and/or have unstable internet, this is the best option
    3. As a last option, the app is accessible in any internet browser via Binder hosting: https://mybinder.org/v2/gh/MacrosystemsEDDIE/module6/main?urlpath=shiny/app/
      1. Prior to class, the instructor should go to the Binder webpage and make sure that it loads (because Binder is provided as a free platform, it occasionally needs to be loaded more than once the first time)
      2. It takes ~5-10 minutes to load, so it's helpful for students to start loading this webpage prior to instruction
      3. During instruction, this webpage occasionally times out if there are many users (even if on good internet), so this is a less stable alternative than options 1 or 2
  2. Give students their handout ahead of time to read over prior to class or ask students to download the handout from the module Shiny app page when they arrive to class. The module is set up for students to complete discussion questions in the handout (Word document) as they navigate through the R Shiny app activities. As they navigate through the app, students will be prompted to answer questions in their handout, as well as download plots that they generate within the app and copy-paste them into their handout. The handout can be submitted to the instructor for potential grading. 
  3. Instructor gives a brief PowerPoint presentation that introduces ecological forecasting, sources of forecast uncertainty, and the different models they will be using (~30 mins).
  4. After the presentation, the students divide into pairs. Each pair selects their own NEON site and visualizes their site's data (Activity A Objectives 1 and 2). The two students within a pair each build their own different models for predicting water temperature (Activity A Objective 3), and generate forecasts with no uncertainty using each of their models (Activity A Objective 4). For virtual instruction, we recommend putting two pairs together (n=4 students) in separate breakout rooms during this activity so the two pairs can compare results.
  5. The instructor can ask students to wait until all students are finished Activity A and then they will all begin Activity B together. For virtual instruction, this would entail having the students come back to the main room for a short check-in.
  6. In Activity B, the students work in their pairs to generate forecasts which include different sources of uncertainty (Activity B Objectives 5-8). Students may compare their forecasts with their partners and work together to answer questions embedded throughout this activity about why the different models are affected differently by the different sources of uncertainty.
  7. In Activity C, student pairs generate forecasts including all sources of uncertainty and compare how the contributions of different sources of uncertainty to total forecast uncertainty varies among models (Activity C Objective 9). They then complete a management scenario individually and discuss with their partner how the uncertainty visualizations provided in the scenario affected their management decisions (Activity C Objective 10).

Teaching Materials:

Teaching Notes and Tips

Important Note to Instructors:

The R Shiny app used in this module is regularly updated, so these module instructions will periodically change to account for changes in the code. If you have any questions or have other feedback about this module, please contact the module developers (see "We'd love your feedback" below).

We highly recommend that instructors familiarize themselves with the Shiny app prior to the lesson. This will enable you to be more prepared to answer questions related to certain areas of the app's functionalities.

Assessment

Student understanding is assessed using interactive questions embedded throughout the R Shiny application. To answer these questions, students will need to manipulate and visualize data, fit models, generate forecasts, and create plots in the application. Student responses are recorded by downloading and copy-pasting plots and typing answers into a Word document report that they download from the application.

  • Activity A: Students identify patterns in air and water temperature data as well as relationships between these variables; describe the structures of the simple forecast models they fit and how well these models fit their data, and use deterministic forecasts (without uncertainty) to make predictions of future water temperature.
  • Activity B: Students describe how different sources of forecast uncertainty differ among their four forecast models, interpret why the sources differ, and identify methods for reducing different types of forecast uncertainty.
  • Activity C: Students compare the relative contributions of different sources forecast uncertainty among forecast models and identify how uncertainty quantification and communication affect management decision-making through a case study exercise.

References and Resources

Optional pre-class readings and videos:

Articles:

  • Silver, N. (2012) Chapter 6: How to drown in three feet of water. Pages 176-203 in The Signal and the Noise: Why so many Predictions Fail – but some Don't. Penguin Books.
  • Dietze, M., & Lynch, H. (2019). Forecasting a bright future for ecology. Frontiers in Ecology and the Environment, 17(1), 3. https://doi.org/10.1002/fee.1994
  • Dietze, M. C., Fox, A., Beck-Johnson, L. M., Betancourt, J. L., Hooten, M. B., Jarnevich, C. S., Keitt, T. H., Kenney, M. A., Laney, C. M., Larsen, L. G., Loescher, H. W., Lunch, C. K., Pijanowski, B. C., Randerson, J. T., Read, E. K., Tredennick, A. T., Vargas, R., Weathers, K. C., & White, E. P. (2018). Iterative near-term ecological forecasting: Needs, opportunities, and challenges. Proceedings of the National Academy of Sciences, 115(7), 1424–1432. https://doi.org/10.1073/pnas.1710231115

Videos: