Gather Evidence

These pages were written by Anne Egger (Central Washington University) and Molly Kent (Science Education Resource Center, Carleton College), drawing on discussions and contributions from the 2014 Getting the Most Out of your Introductory Courses workshop.

One of the keys to being strategic in making changes that are effective and sustainable is gathering evidence that there are improvements that can be made, that the proposed changes have a chance of working, and that you've succeeded in achieving your goals. This evidence can come from many sources.

Explore the National Landscape

Your introductory courses and your institution are all part of the national landscape of undergraduate science education. There may be other features of that landscape that can help you, such as initiatives you can join or learn from, even in other disciplines. Listen to Jay Labov from the National Academy of Sciences discuss The Changing National Landscape of STEM Education. In this presentation, he describes the growing body of research around how people learn, the STEM pipeline, and the process the biology community went through to reform their teaching nationwide.

In January 2014, the NSF-funded Summit on the Future of Undergraduate Geoscience Education took place at the University of Texas, Austin. You can see the schedule, participants, a webcast of all presentations, and download the summary report of this summit.

Where to Look for Evidence that Improvements Can Be Made

Evidence that improvements can be made is likely the easiest to gather, and the kind you will be most familiar with. The data may already exist, or you may need to implement some simple assessments to gather it. Here are some suggestions:

• Student performance in the course. This does not necessarily need to be cumulative grades, which may vary by instructor or any number of other factors. But you might find that students consistently demonstrate a lack of understanding of a particular concept, or that attendance is low, or that they don't turn in assignments. Alternatively, you can look at grade breakdown by other demographic factors: major, number of science courses they've taken, math background.
• Course evaluations. Student evaluations of courses can be highly variable in their reliability and helpfulness, but you may be able to see trends over the long term. Do they consistently report that the textbook was useless? Maybe it's time to rethink whether or not you actually need to use the textbook.
• TA feedback. If you have graduate or undergraduate teaching assistants in your class, they likely have a lot more one-on-one interactions with the students and can provide their own perspective on what could be improved in the class. Design a survey for these students to fill out (anonymously, if necessary).

×
• Changes in enrollment.These may be increases or decreases. Dropping enrollment might be a symptom of lack of perceived relevance of the course; increasing enrollment could be the opposite or could be a symptom of a "rocks for jocks" attitude.
• Feedback from other departments. Do other majors require your course? How do those students perceive the relevance and utility of your course?
• Program-level assessment. If your introductory courses are part of your institution's general education program or are required by your own program, you might have access to program-level assessments where you can compare your course to others.
• Design your own assessment. If none of these options are producing the information you want, you can design your own assessment. You can find out more about how to design your own assessments effectively from the link /sp/library/assessment/index.html 'Assessment guide'] in Pedagogies in Action.

Where to Look for Evidence that Proposed Changes Might Work

Just as in your scientific research, you want to support your claims and hypotheses with evidence that similar changes have worked elsewhere. These data can come from traditional sources (the literature), but may also come from your colleagues down the hall or in the next building or in the next town.

• Explore the literature.The Pedagogy in Action project has assembled a Research on learning collection which may serve as a useful starting point. In addition, individual pedagogical practices also have literature collections associated with them.
• Additional collections that you may find helpful are:
• An annotated bibliography compiled by David McConnell's graduate students in geoscience education at North Carolina State University,
• Background reading suggested by the organizers of the Summit on the Future of Undergraduate Geoscience Education ,
• Key reports from the National Academies press (all of which can be downloaded for free), including
• How People Learn
• Discipline-Based Education Research
• Promising Practices in Undergraduate Science, Technology, Engineering, and Mathematics Education
• Transforming Undergraduate Education in Science, Mathematics, Engineering, and Technology.
• Look beyond the geosciences. Other science and engineering disciplines have undergone changes in their teaching methods and introductory courses and have much to share.

How to Collect Evidence to Evaluate Your Success

Evaluating the effects of your changes is important for a number of reasons. When you are successful at documenting improved students learning in your courses, you can share your work with your administrators and beyond. Documenting the change you are interested in can be challenging, however, so consider taking advantage of existing assessment instruments and resources within your institution. Consider partnering with faculty in your education programs, or colleagues in your teaching and learning center.

• Learn about assessment strategies for different types of learning environments in introductory courses.
• Learn about program assessment strategies.