On the Cutting Edge Evaluation

The evaluation plan addresses two major effort domains of the project: the workshops and the website. All evaluations provide information to enable project staff to answer the following questions:"What are the results we are trying to achieve?" "Given these, who do we have to reach?" "What programs do we have with which to reach them?" "What evidence do we have that we did what we planned?"

×

On the Cutting Edge Theory of Change

Provenance: Rockman et al produced this image under contract to the Cutting Edge program. Cutting Edge holds copyright.
Reuse: This item is offered under a Creative Commons Attribution-NonCommercial-ShareAlike license http://creativecommons.org/licenses/by-nc-sa/3.0/ You may reuse this item for non-commercial purposes as long as you provide attribution and offer any derivative works under a similar license.

The intent of the evaluation is both formative and summative. From a formative perspective, we look to improve our overall program including the workshops, website collections, and leadership program.The current objectives of the summative evaluation plan are to assess the impact of the workshops and website on faculty teaching and student learning. A second aim is to evaluate how the program contributes to the research base on effective faculty development.

The evaluation plan uses a theory of change to examine and identify the strongest and most compelling evidence for the program impacts.

Jump to: Evaluation Reports

National Geoscience Faculty Survey

• 2016 National Geoscience Faculty Survey was conducted in November 2016 and is currently being analyzed
  • 2016 National Geoscience Faculty Survey Question Items (Acrobat (PDF) 915kB Jan21 19)
• 2012 Survey items
  • 2012 National Geoscience Faculty Survey Question Items (Acrobat (PDF) 855kB Jan21 19)
• 2009 Survey items
  • Screen 1: 2009 Geoscience Teaching Practices Survey (HTML File 7kB Sep3 15)
  • Screen 2: 2009 Geoscience Teaching Practices Survey (HTML File 46kB Sep3 15)
  • Screen 3: 2009 Geoscience Teaching Practices Survey (HTML File 48kB Sep3 15)
  • Screen 4: 2009 Geoscience Teaching Practices Survey (HTML File 28kB Sep3 15)
  • Screen 5: 2009 Geoscience Teaching Practices Survey (HTML File 2kB Sep3 15)
• 2004 Survey items
  • Question items for 2004 Geoscience Teaching Practices Survey (Acrobat (PDF) 775kB Sep4 15)

Publications and Presentations Related to Evaluation

This list includes publications related to the evaluation of the program.

Under Review

• Quardokus Fisher, K.. Under review. Characteristics and boundaries of the geoscience Education Community of practice. Journal for STEM Education Research.

2020

• Beane, R., Altermatt, E.R., Iverson, E.R., & Macdonald, R.H. (2020). Design and impact of the National Workshop for Early Career Geoscience Faculty. Journal of Geoscience Education. https://doi.org/10.1080/10899995.2020.1722787
• Gamage, K., McFadden, R. R., Macdonald, R. H. (2020). Development of Students' Skills in Introductory Geoscience Courses: A Comparison of Self-reported Teaching Practices at Two-Year and Four-Year Institutions. Journal of Geoscience Education, https://doi.org/10.1080/10899995.2020.1833655
• Viskupic, K., Egger, A. E., McFadden, R. R., Schmitz, M. D. (2020). Comparing desired workforce skills and reported teaching practices to model students' experiences in undergraduate geoscience programs. Journal of Geoscience Education, https://doi.org/10.1080/10899995.2020.1779568

2019

• Beane, R., McNeal, K. S., & Macdonald, R. H. (2019). Probing the National Geoscience Faculty Survey for reported use of practices that support inclusive learning environments in undergraduate courses. Journal of Geoscience Education, 67(4), 427-445. (Full article - open access)
• Egger, A. E. (2019). The Role of Introductory Geoscience Courses in Preparing Teachers"And All Students"For the Future: Are We Making the Grade?. GSA Today. (Abstract)
• Egger, A.E., Viskupic, K., and Iverson, E.R. (2019).Results of the National Geoscience Faculty Survey (2004-2016). (Acrobat (PDF) 5MB Dec4 19). National Association of Geoscience Teachers, Northfield, MN. 82 p.. Retrieved from https://serc.carleton.edu/NAGTWorkshops/CE_geo_survey/index.html.
• Lally, D., Forbes, C. T., McNeal, K. S., & Soltis, N. A. (2019). National Geoscience Faculty Survey 2016: Prevalence of systems thinking and scientific modeling learning opportunities. Journal of Geoscience Education, 67(2), 174-191. https://doi.org/10.1080/10899995.2019.1565286
• McFadden, R. R., Viskupic, K., Egger, A. E. (2019). Faculty self-reported use of data analysis and quantitative skills in undergraduate geoscience courses. Journal of Geoscience Education. https://doi.org/10.1080/10899995.2019.1700595
• Riihimaki, C. A., & Viskupic, K. (2019). Motivators and inhibitors to change: Why and how geoscience faculty modify their course content and teaching methods. Journal of Geoscience Education, 1-18. https://doi.org/10.1080/10899995.2019.1628590
• Soltis, N. A., McNeal, K. S., Forbes, C. T., & Lally, D. (2019). The relationship between active learning, course innovation and teaching Earth system thinking: A structural equation modeling approach. Geosphere. https://doi.org/10.1130/GES02071.1
• Viskupic, Karen and Ryker, Katherine and Teasdale, Rachel and Manduca, Cathryn and Iverson, Ellen and Farthing, Dori and Bruckner, Monica Z. and McFadden, Rory, (2019). Classroom Observations Indicate the Positive Impacts of Discipline-Based Professional Development. Journal for STEM Education Research, doi:10.1007/s41979-019-00015-

2017

• Manduca, C. A., Iverson, E. R., Luxenberg, M., Macdonald, R. H., McConnell, D. A., Mogk, D. W., & Tewksbury, B. J. (2017). Improving undergraduate STEM education: The efficacy of discipline-based professional development. Science Advances, 3(2). (Abstract)
• Teasdale, R., Viskupic, K., Bartley, J. K., McConnell, D., Manduca, C., Bruckner, M., Farthing, D., Iverson, E. (2017). A multi-dimensional assessment of reformed teaching practice in geoscience classrooms. Geosphere, 13(2).

2016

• Iverson, Ellen. (2016). Understanding the Factors that Support the Use of Active Learning Teaching in STEM Undergraduate Courses: Case Studies in the Field of Geoscience. Retrieved from the University of Minnesota Digital Conservancy, Abstract.
• Iverson, E.R. (2016). How Geoscience Faculty Learn about Improving Their Teaching Practice: Lessons from Ten Years of Interviews. Earth Educators' Rendezvous, Madison, WI. (Abstract)

2015

• Iverson, E., Bruckner, M., Larsen, K., Manduca, C., (2015) "Replicating the effects ongoing treatment through sustained networks: the Impact of the On the Cutting Edge Professional Development Program for Geoscience Faculty (CE) on faculty learning about teaching," Earth Educators' Rendezvous, Boulder, CO (Abstract)

2014

• Ryker, Katherine, Iverson, Ellen, Bruckner, Monica Z, Mcconnell, David, Manduca, Cathryn, and Larsen, Krista (2014). Connecting Beliefs To Action: How Geoscience Faculty Perceptions Of Their Professional Environment Impact Teaching, Geological Society of America, Abstracts with Programs, Vol. 46, No 6.
• Teasdale, Rachel, Cathryn Manduca, David Mcconnell, Julie Bartley, Monica Bruckner, Dori Farthing, Ellen Iverson, Karen Viskupic (2014). Observations of Undergraduate Geoscience Instruction in the US: Measuring Student Centered Teaching, AGU Fall Meeting, San Francisco, CA
• Viskupic, Karen, Steer, D., Kortz, K. M., Perkins, D., Wirth, K., Herbert, B., Singer, J., Manduca, C. A., (2014). Monitoring Undergraduate Growth Through The Major Using Embedded Assessment Questions, Abstracts with Programs, Vol. 46, No 6.

2013

• Bruckner, M.Z., Iverson, E., Manduca, C.McConnell, D., Bartley, J., Farthing, D., Teasdale, R., Viskupic, K. (2013). Using the on the Cutting Edge RTOP instrument to characterize how geoscience is taught. Geological Society of America Abstracts with Programs. Vol. 45, No. 7, p.313 abstract)

2011

• Bruckner, Monica Z., Rachel J. Beane, and Heather Macdonald (2011), On the Cutting Edge Webinar Series Supports Graduate Students and Post-Docs Pursuing an Academic Career (Acrobat (PDF) 1.8MB Dec18 11), Geological Society of America Abstracts with Programs, Vol. 43, No. 5, p. 255 (Abstract)
• Fox, Sean, Ellen A. Iverson, Cathryn A. Manduca, Karin B. Kirk, John R. McDaris, Carol J. Ormand, Monica Z. Bruckner (2011), Supporting Faculty Learning About Teaching: The On the Cutting Edge Website, Abstract ED23A-0612 Poster presented at 2011 Fall Meeting, AGU, San Francisco, Calif., 5-9 Dec. (Abstract)
• McConnell, David, Iverson, Ellen, Budd, David, Henry, Darrell, Kraft, Katrien J. van der Hoeven, McDaris, John R., MacDonald, Heather, Manduca, Cathryn, Srogi, LeeAnn, and Viskupic, Karen (2011). A Preliminary Snapshot of Geoscience Teaching at US Colleges and Universities. Geological Society of America Abstracts with Programs, Vol. 43, No. 5, p. 136. (Abstract)
• Teasdale, Rachel, David Budd, Cinzia Cervato, Ellen Iverson, Katrien J. van der Hoeven Kraft, Cathryn Manduca, David A. McConnell, John R. McDaris, Daniel P. Murray, and William Slattery (2011). Enhancing Student-Centered Teaching Practices: Approaches Developed On the New Cutting Edge Geosciences RTOP Website, Geological Society of America Abstracts with Programs, Vol. 43, No. 5, p. 255. (Abstract)

2010

• Manduca, Cathryn A., David W. Mogk, Barbara Tewksbury, R. Heather Macdonald, Sean P. Fox, Ellen R. Iverson, Karin Kirk, John McDaris, Carol Ormand, and Monica Bruckner (2010). SPORE: Science Prize for Online Resources in Education: On the Cutting Edge: Teaching Help For Geoscience Faculty: Science, v. 327, no. 5969, pp. 1095-1096.

2006

• Iverson, E.R., McLaughlin, J.A., Manduca, C.A. (2006) The Logic Model as Compass: Guiding Stakeholders Through the Consequences of Evaluation. Presented at American Evaluation Association 2006 Annual Conference.presentation (PowerPoint 1.5MB Nov29 06)
• Manduca, C.A., H. Macdonald, D. Mogk, and B. Tewksbury (2006). On the Cutting Edge: Evolving Themes, Enduring Impact (Acrobat (PDF) 335kB Apr25 06). Northfield, MN: Science Education Resource Center.

2005

• Iverson, E.R., C.A. Manduca, J.A. McLaughlin, R.H. Macdonald, (2005), Enhancing your Teaching and Developing New Leadership: Impact of the On the Cutting Edge Professional Development Program (PowerPoint 2.6MB Jan18 06), Eos Trans. AGU, 86(52), Fall Meet. Suppl., Abstract ED42A-02
• Macdonald, R.H., C.A. Manduca, D.W. Mogk, and B.J. Tewksbury (2005). "Teaching Methods in Undergraduate Geoscience Courses: Results of the 2004 On the Cutting Edge Survey of US Faculty." Journal of Geoscience Education, 53(3): 237.
• Manduca, C.A. and E.R. Iverson (2005). Understanding Faculty Work Habits as a Foundation for Professional Development (PowerPoint 1.8MB Jan18 06). Presented at POD Network 2005 Annual Conference.
• Fox, S., Manduca, C. A., Iverson, E., (2005). Building Educational Portals atop Digital Libraries D-Lib, 11(1).

2004

• Macdonald, R. H., C. A. Manduca, D. W. Mogk and B. J. Tewksbury (2004). "On the Cutting Edge: Improving Learning by Enhancing Teaching." In, Invention and Impact: Building Excellence in Undergraduate Science, Technology, Engineering, and Mathematics (STEM) Education. Washington, D.C., AAAS: 381.

Evaluation Reports

Overarching summary evaluation

The following comprehensive reports evaluate all components of the Cutting Edge project over long periods of time and utilize a variety of evaluation techniques (surveys, interviews, other analysis).

• 2013 External Evaluation Report (Acrobat (PDF) 855kB Aug10 15) Summative report of workshop and website evaluation from 2010-2013
• 2010 Cutting Edge Evaluation Report (Acrobat (PDF) 898kB Feb5 10) Summative report of workshop and website evaluation from 2002-2009
• 2005 Summary Evaluation Report (Microsoft Word 259kB Apr17 06) Summative report of workshop and website evaluation from 2003-2005
• 2004 summary of website evaluation activities (Microsoft Word 28kB Jan18 06) Report to determine how workshop participants between 2002 and 2004 use the website

Interview studies

The following reports present findings of interview studies related to either the Cutting Edge workshop program or website.

• 2007 Summary of Interviews at AGU (Microsoft Word 76kB Jan7 10) Interview study to determine impact of workshops on teaching beliefs and the role of the website for website-only users (non-workshop participants)
• 2005 External Heuristic Review (Acrobat (PDF) 979kB Jan18 06) Report outlining findings of website usability testing
• 2005 Summary report of phone interviews conducted with 54 workshop participants (Microsoft Word 79kB Jan18 06) Interviews of workshop participants from 2002-2004 to determine impacts of workshops
• 2005 Summary report of phone interviews conducted with 12 non-workshop website users (Microsoft Word 33kB Jan18 06) Interviews to determine how website users who had not attended a workshop make use of the website.
• 2004 AGU Focus Groups (Microsoft Word 38kB Jan18 06) Interviews to determine how both workshop participants and non-workshop participants make use of the website.

Survey studies

The following reports present findings of surveys related to the Cutting Edge workshop program or website.

• 2013 Results of Email Poll (Acrobat (PDF) 34kB Jan30 14), Large survey or geoscience faculty to determine awareness of workshops and website
• 2008 Bibliometric Survey of On the Cutting Edge Workshop Participants (Microsoft Word 2007 (.docx) 82kB Jan12 10), Luti Salisbury, study of publications by workshop attendees
• 2008 SERC Results of an Email Poll (Microsoft Word 42kB Dec12 08) email survey of geoscience faculty about both website and workshop awareness
• 2008 Participant Survey Summary (Microsoft Word 87kB Jan8 10)
• 2006 SERC Pop Up Survey Results (Microsoft Word 197kB Apr28 06) survey of random website users to determine how website is being used
• 2006 Summary report of awareness survey conducted (Microsoft Word 36kB May1 06) email survey of geoscience faculty about both website and workshop awareness
• 2005 Summary report of Cutting Edge users from SERC pop-up survey (Microsoft Word 44kB Jan18 06) popup survey of random website users in 2005 (data only)
• 2005 Summary report of 235 responses to impact survey (Microsoft Word 89kB Jan18 06) survey of workshop participants (prior to 2004) to determine workshop impact
• 2004 Summary report of 95 responses to follow up survey (Microsoft Word 127kB Jan18 06) follow up survey of workshop attendees (prior to 2004) to determine workshop impact