A Curriculum by Design Part II: Student Learning Outcomes and Program Assessment

David Mogk, Dept. Earth Sciences, Montana State University

published Jan 20, 2014

In an earlier blogpost (A Curriculum by Design) I outlined the philosophy and process we used in the Department of Earth Sciences, Montana State University, to revise our undergraduate curriculum. This continuing contribution describes the process we used to identify student learning outcomes (SLOs) across the curriculum, and how these SLOs have been used to develop our department assessment plan (required in anticipation of our forthcoming institutional accreditation review).

Assessment

Assessment is collecting data with a purpose. The Department of Earth Sciences completed an extensive review of student learning outcomes at the programmatic level for all courses offered by the department. Our assessment plan is designed to: a) provide faculty with an opportunity to reflect on course goals, methods and expected student learning outcomes, b) aggregate these course learning goals into an overall departmental matrix of student learning outcomes, c) provide formative feedback to improve teaching and learning in Earth Sciences courses, and d) for accountability, to demonstrate that the departmental and institutional vision and mission are being addressed and that the curriculum is consistent with contemporary professional standards in the geosciences. The resulting SLO Matrix (Excel 2007 (.xlsx) 122kB Jan16 14) provides a rapid, visual map of the "landscape" of our curriculum; you can readily see areas of emphasis, and areas that might need more attention in our curricular development. This exercise also provided our faculty the opportunity to reflect deeply on the concepts and skills they emphasize in their own courses, gave them some incentives to revise courses to respond to the SLO goals, and opened the door for more extensive curricular discussions between faculty (who generally were not aware of content/skills being taught in courses related to their own).

Work on our portfolio of SLOs followed the design, development and implementation of a fully revised curriculum for degree options in the physical sciences offered by the department (Geology, Snow Science, Paleontology); a similar review process is in progress by our geographers for our Geography and GIS/Geographic Planning degree options. This revised curriculum is now in its second year of implementation, and we conducted a formative assessment designed to determine the degree to which departmental curricular goals are being met.

The Curricular Design Process

We undertook our curriculum design process following these principles:

We used a "backward design" process (Wiggins and McTighe, 2005) to define the profile of an ideal graduate from our program, identifying what a student should know (concepts and content knowledge, scientific "habits of mind") and be able to do (geoscience technical skills and other professional skills);
Design of learning progressions in key areas to ensure that students earn mastery of key concepts and skills through multiple exposures throughout the curriculum (the "Rule of 3's or 4's"—if something is worth learning students need at least three or four exposures to gain mastery: exposure, familiarization, competence, and ultimately mastery of the concept or skill); this also reflects the design of an integrated and coherent curriculum in which course content in a given course reflects on lessons learned in earlier courses and anticipates applications in more advanced courses;
Application of Bloom's Taxonomy of Cognitive Skills (Bloom 1956; updated Anderson et al., 2000) to promote development of higher-order thinking skills throughout the curriculum: a) starting with early exposure to Earth Science topics to stimulate interest and motivation to learn, and to recruit majors to the discipline, and b) to emphasize more advanced interpretation, application, analytical and synthetic reasoning skills about the Earth system for majors across the curriculum;
Adoption of an Earth system approach, emphasizing the connections between the solid earth, oceans, weather and climate, biota and humanity;
Preparation of students for the workforce of the 21^st Century (or graduate school), including disciplinary knowledge and skills, life-long professional skills, and scientific "habits of mind"; and
Alignment of the curriculum with the department vision and mission statements, with institutional curricular requirements (e.g. our "Core 2.0" course requirements), and with the MSU Strategic Plan.

The revised curriculum developed by the Department of Earth Sciences is described and represented graphically in Part I of A Curriculum by Design.

The Programmatic Assessment Process

Our programmatic assessment is the result of an intensive self-study by the faculty of the department to document the topics and skills that are emphasized in different courses across the curriculum and at different instructional levels.

We used the Matrix Approach to Curriculum Design module (Savina and Macdonald), from the National Association of Geoscience Teachers Building Strong Geoscience Departments project. The approach was further described by Dallas Rhodes, Georgia Southern University (2011), and adapted and expanded for use by the Department of Earth Sciences to more specifically represent departmental priorities, resources, and staffing. The major student learning outcomes fall into four main categories: Discipline Knowledge (concepts and content), Discipline-Specific Skills (technical skills), Earth Science Habits of Mind, and Professional Skills (communication, quantitative, information, and interpersonal).

Show See Description of These 4 Sets of Concepts and Skills

1. Discipline-Specific Knowledge: Graduates are expected to have in-depth knowledge of the following fundamental concepts in the Earth Sciences: the dynamics of the Earth system, including interactions among the solid Earth, Earth's surface, hydrosphere, biosphere, and with humanity. Students in the physical science degree options are expected to gain mastery of the concepts of geologic time, evolution life and history of the Earth system; the composition and architecture of Earth; and processes and phenomena observed on the surface of Earth (landscapes, weather and climate, biosphere). Students in the Geography degree option will gain mastery of the social, economic, cultural, and historical dimensions of humanity interacting with the Earth system.

2. Discipline-Specific Skills: All students in the Department of Earth Sciences will develop temporal (geologic and geographic history) and spatial reasoning skills (utilizing Geographic Information Science, and including other forms of 3- and 4-dimensional data representations). Students in the physical sciences will master skills required for professional development (e.g. field methods, computer modeling, experimental and analytical methods), and students in the social sciences will master the quantitative and qualitative methods typically used by professional geographers.

3. Earth Science "Habits of Mind": Students are expected to formulate questions; apply concepts, content knowledge, skills and tools; produce, critically evaluate, and appropriately represent data; interpret evidence and report results. Earth Science students must be able to draw inferences from observations or data that are incomplete, ambiguous and uncertain. Spatial reasoning, temporal reasoning, systems thinking and the ability to work in the field are all central to the professional development of Earth Scientists.

4. Professional Skills: a) Communication skills; Earth Science students are expected to present the results of their work in written, oral, and graphical formats; b) Quantitative skills; students are expected to apply mathematical formulations to represent physical and human dynamic systems; and c) Interpersonal skills; students are expected to work cooperatively and collaboratively to achieve common goals. Students should also be able to critically read the primary literature, and to have a fundamental knowledge of the theory, philosophy and intellectual history of the discipline.

These general topics are further sub-divided according to themes that are central to learning in the Earth Sciences. The identification of these sub-themes (represented in columns on the accompanying spreadsheet) was based on

An initial Gallery Walk done by faculty at a department retreat in which faculty identified key elements of their courses [e.g. strong field component, writing-intensive, etc.]
National initiatives that support excellence in Earth Sciences education including

The On the Cutting Edge program (NSF TUES Phase III),
The InTeGrate-Interdisciplinary Teaching of Geoscience for a Sustainable Future program (NSF-STEP)
The American Geosciences Institute's Geoscience Workforce Program,
The Association of American Geographers, and
The U.S. Department of Labor Geospatial Technology Competency Model.

The categories of SLOs used in this SLO Matrix Template (Excel 2007 (.xlsx) 122kB Jan16 14) address the learning goals that are specific to the Dept. of Earth Sciences, Montana State University (although guided by these other national programs). Other departments should feel free to modify this matrix to meet their own departmental vision, mission, faculty, student population and resources.

All faculty in the Department of Earth Sciences contributed to this self-study. The benefit of using self-assessment data is that the survey format allowed for easy and rapid responses from all faculty. Each faculty member was responsible for only a handful of courses, and individually, it did not take a major effort for faculty to complete their part of the matrix. However, there are known issues associated with self-reporting, particularly with respect to calibrating responses from different faculty that may result in some distortion of the reliability of these data (Teo, 2012; note that one faculty member's responses are clearly not valid). In some cases, the department head worked with individual faculty to reconsider the ratings they initially submitted as their perceived contributions were often reported as being too conservative compared to their faculty peers. Overall, most faculty made a good-faith effort in identifying areas of emphasis in their courses.

Faculty provided input about their contributions to student learning outcomes (represented in rows in the accompanying spread sheet) in the courses where they have primary instructional responsibility according to this rating scale: 3 (red)= this topic is central and essential to the course goals, and is strongly emphasized throughout the course; 2 (yellow)= this topic is considered in this course and supports course learning goals; 1 (blue) = this topic is introduced in the course but is not covered in depth; blank= topic not covered. The submitted course SLOs were then compiled by the department head into a single omnibus spreadsheet for the entire department. These color coded results in the matrix provide a rapid means to survey the curricular landscape of the Department of Earth Sciences.

This input was then re-sorted into a series of separate sheets for further programmatic analysis according to:

Rubrics within the department (we have course listings under three categories: Earth Science, ERTH, Geography, GPHY, and Geology, GEO);
Level of offering (100, 200, 300, 400)
Undergraduate degree options (we offer 5 degree options under our BS in Earth Sciences degree: Geography, Geographic Information Science/Planning, Geology, Paleontology and Snow Science),
Faculty contributions to the department's educational mission, and
The graduate program.

These sortings provide the evidence for deeper analysis of curricular strengths and points of emphasis, individual faculty contributions to student learning outcomes, representations of the overall student experiences in our degree programs, and faculty expectations for learning in our undergraduate and graduate degree programs.

In aggregate, the matrices attached to this report provide a very interesting assessment of the overall current curricular structure of the Department of Earth Sciences:

Areas of particular strength or emphasis are readily identified (red or yellow),
This visualization provides a road map to guide future curricular revision and development, and
This matrix can be used as a "gap analysis" to identify specific areas in need of further development. In some cases, we may simply ask that established courses change emphasis a bit to address missing components of the curriculum; in other cases we may need to develop new courses.

This matrix is not to be viewed as a representation of workload effort for each faculty member, as many courses are taught by multiple faculty in rotation, and other courses are offered in alternate years or on demand.

Analysis of Assessment Results

This assessment process has resulted in immediate benefits:

The course matrix has provided an important baseline of the current breadth and scope of our instructional efforts across the Earth Sciences curriculum.
The process has provided a forum for faculty to begin or continue conversations about the overall curriculum and how their courses contribute to the overall instructional mission.
Many faculty gained a better sense of what is being taught in related parts of the curriculum, and efforts are underway to better align our identified SLOs with learning progressions in our course sequences; and
Many faculty have independently commented on the personal value they derived from this assessment exercise, as they reflected deeply on their course goals, areas of emphasis, and overall contributions to the curriculum.
Overall, the faculty had never had the opportunity to see the totality of what we are covering our curriculum, and it has been a very positive thing for the department to see the depth and breadth of the scholarship that is being done in our department.
For full disclosure, I need to note that not all faculty are happy with the new curriculum. To accommodate new degree requirements of a full year of GIS and Weather and Climate for all Earth Science majors, we have had to a) discontinue some courses that were deemed to be redundant, b) compress some courses into new hybrid courses, c) change the level of instruction of some courses, and d) move some formerly required courses into the "free elective" (not required) courses that are now offered on alternate years. Time will tell how this all plays out.

The course matrix provides the evidence that describes the degree to which we have met university (MSU) and department learning goals. (I used this as the basis of my annual program assessment report that was submitted to my administration)

1. The Department of Earth Sciences contributes to the MSU Core 2.0 Curriculum (see tab Sorted by Level, 100 Level Courses)

Show See Evidence of contributions to the MSU Core Curriculum

The Department offers a wide array of Core 2.0 courses in the areas of Inquiry (ERTH 101, ERTH 201, GEO 105, GEO 111, GEO 140,GEO 208), Diversity (GPHY 121, GPHY 141), Contemporary Issues in Science (ERTH 102, GEO 103), and Research and Creative Experience (ERTH 212, and for majors GEO 429, GPHY 441, GPHY 484, ERTH 450). Inquiry classes require attention to the "...methods used to discover and create the factual and theoretical knowledge of the discipline." Earth Science inquiry courses have a demonstrated heavy emphasis on Earth Science "habits of mind" (e.g. systems thinking, spatial reasoning, and temporal reasoning), use of Earth data, and problem-solving skills. Contemporary Issues in Science courses "...examine the ways in which science contributes to the study of significant problems in the contemporary world to help individuals and society make informed decisions about these issues." Earth Sciences CIS courses focus on the connections between the Earth system and humanity, covering topics such as natural hazards, natural resources, the cultural, historical and economic impacts, and applications to public policy and planning. Diversity courses emphasize "...understanding of and sensitivity to other cultural perspectives prepares them to function in the global community...". Earth Sciences courses in this area have a primary focus on the cultural, historical and economic condition of humanity. Research courses "will incorporate a range of authentic experiences" and this is realized through embedded research projects within the Yellowstone Scientific Laboratory course (ERTH 212 for non-majors) and research experiences in the field and lab for majors in our degree programs. There is a strong emphasis on using Earth data in the classroom in Earth Sciences classes taught at all levels. This includes data that students collect themselves, and data ported from external databases (e.g. USGS, NOAA, EPA,...). In addition, Earth Sciences courses places a high value on communication skills (writing, oral and graphical presentations), quantitative skills, and applications of principles from sister disciplines (e.g. Biology, Physics, Chemistry), in accord with the goals of the Core 2.0 Curriculum.

2. Earth Sciences courses increasingly employ an Earth Systems approach.

Show See evidence that an Earth System approach is used

The Earth system approach seeks to demonstrate the connections between the solid earth, oceans, atmosphere, biota and humanity. Inspection of discipline knowledge (concepts and content; columns E through O) demonstrates the breadth of coverage of topics in each Earth Sciences course. Almost all courses taught under the ERTH and GEO rubrics are focused on physical aspects of the Earth system, but extend this coverage to impacts on and by humanity (e.g. natural hazards and resources). Courses under the GPHY rubric mostly focus on the human aspects of geography, but also include coverage of topics related to weather, climate, hydrosphere, biosphere, and land forms, all components of the "critical zone" that supports life on Earth. The overall breadth of courses that use an Earth System approach can best be observed in the sheet Sorted by Undergraduate Degree Option. In addition, the systems approach to instruction in the Earth Sciences is utilized throughout the curriculum with emphases on GIS (as an integrative tool for data representation), a specific focus on systems thinking, temporal and spatial reasoning. All students in the Earth Sciences degree programs share a common set of courses that contribute to an Earth System approach including Introduction to Earth System Science, Topics in Earth Science, a full year of GIS training, Weather and Climate, and Geomorphology.

3. The Department of Earth Sciences has developed an integrated curriculum with learning sequences that emphasize the development of higher order thinking skills.

Show See evidence that learning progressions lead to an integrated curriculum and to develop higher order learning skills

a. The sheet that represents the undergraduate degree options demonstrates the reinforcement of major themes throughout the four-year program of study. In the physical sciences, the themes of geologic time (history and evolution of Earth), composition and architecture of the solid earth, and processes that operate on the surface of the earth are addressed from multiple perspectives throughout the curriculum. This includes development of discipline specific skills (e.g. identification of rocks, minerals, fossils, structures, landforms; use of the petrographic microscope), and Earth science habits of mind. In the social sciences (Geography) degree options, human systems are revisited throughout the curriculum (social, economic, and historical perspectives), as well as technical skills (qualitative methods).

b. Higher order thinking skills are developed across the curriculum through use of case-based studies, problem-solving activities, acquisition and use of data, critical reading of the primary literature, and formulation and testing of hypotheses. Authentic questions and problems are commonly embedded into coursework at all levels as class projects, requiring students to acquire and integrate information from numerous sources to formulate a solution to the problem.

4. The Department of Earth Sciences prepares students a) to continue with graduate studies and/or b) to join the workforce in discipline.

Show See evidence that students are prepared for graduate school or the workforce

This is the profile of a student who successfully completes an Earth Sciences degree. Students who can (See: InTeGrate June 2012 workshop and webpage on: Geoscience Workforce

Understand geologic/geographic context, apply concepts and skills
Ask the next question
Know where to look for information
Formulate a plan to address the problem
Become critical producers and consumers of data
Integrate multiple lines of evidence
Communicate results; write a report, make a map, develop a GIS, and
Be life-long learners.

Overall Results of This Assessment of Programmatic SLO's:

The Department of Earth Sciences offers robust degree programs that prepare students for the next steps in their professional development. Course content and technical skills are developed in all Earth Sciences courses to conform to professional standards and competencies. There is a strong emphasis across the curriculum on development of communication skills, quantitative reasoning, information technology skills, and interpersonal skills through cooperative and collaborative learning. Problem-solving has been reported as a highly valued skill in the geosciences, and class activities are routinely developed to simulate or replicate professional practices, and in some cases, to engage authentic research activities. Results of course work commonly make direct contributions to the body of scientific knowledge, and with applications to issues of societal interest.

Resources that Can Help Get You Started Creating the Student Learning Outcomes Matrix for Your Own Department

Dept. of Earth Sciences, Montana State University Template (Excel 2007 (.xlsx) 122kB Jan16 14); feel free to start with the Blank Template (Excel 2007 (.xlsx) 38kB Jan16 14) of SLO topics, or refer to the matrix completed by our faculty to see what a matrix for an entire degree program might look like. You will have a different array of courses, and the SLOs may be a bit different depending on your department's vision, mission, staffing, profile of students, geographic setting, departmental resources, etc.

Mary Savina and Heather Macdonald had the first description of using a matrix approach to curriculum development, posted at Building Strong Geoscience Departments. Access the Matrix Approach to Curriculum Design module.

Dallas Rhodes, Georgia Southern University, expanded on the matrix approach in his 2011 GSA poster Curriculum and Program Learning Outcomes Mapping to Enhance Program Assessment

I expanded Dallas' matrix following input from other sources:

At a faculty retreat, I used a Gallery Walk (from the Starting Point collection of teaching strategies) to get faculty involved with defining the concepts and skills they thought were most important for their courses. I used large flip chart sheets each with a teaching emphasis (e.g. technical skills like rock and mineral identification, map reading; communication skills; quantitative skills; field instruction; modeling; etc.). These were posted on walls around the room, and the faculty migrated from sheet to sheet and wrote what they did under each heading and in which classes. I also had a number of blank sheets so faculty could post methods, activities, etc. that were otherwise important to their courses. So, my preliminary matrix featured skills and concepts that were identified by the faculty.

I wanted to address expectations of employers for the modern workforce, so I referred to the AAC&U Survey of Employers (Acrobat (PDF) 146kB Jan16 14). I also used the outcomes of the InTeGrate Workforce Geoscience and the 21st Century Workforce: Considering Undergraduate Programs in the Context of Changing Employment Opportunities Workshop, and information from the the American Geosciences Institute's Geoscience Workforce Program.

I was also interested in documenting where we were teaching "geoscience habits of the mind, and I relied on information from the Synthesis of Research on Thinking and Learning in the Geosciences project that emphasized spatial reasoning, temporal reasoning, systems thinking and field instruction. I also used information from the InTeGrate workshop on Teaching the Methods of Geoscience workshop.

For attributes of an Earth System Science approach, I used the Starting Point module on Earth System Science in a Nutshell or access the more extensive Site Guide for Earth System Science at SERC.

I also used information from the Using Date in the Classroom portal.

The first step in this process was to review the courses we offered, and to align these into our new curriculum requirements. Our overall course sequence matrix and the steps we took to realign our curriculum are posted at this blogpost (A Curriculum by Design).

Next Steps

Now that we have created this matrix based on a) geoscience faculty input, and b) national reports and events that largely reflect the experience and insights of academics, I plan to have the following groups fill out the matrix to see what aspects of the curriculum they value most:

Current students in our courses for majors: what is there perspective of what they value most in their pre-professional training?
Alumni: it will be interesting to have recent (and distant past) alumni reflect on their training in our department to see what concepts and skills have best served them in their own career trajectories; and
Recruiters from the industries that typically hire our students: it will be interesting to see if they value the same training (concepts, skills) that we emphasize in our curriculum. What will make our students more competitive in the workforce?
Faculty from R1 institutions: You receive our students into your graduate programs. Are you recruiting students to your graduate programs who have the knowledge, skills (and perhaps attitudes, personal attributes) such that they will be able to step into your programs and immediately contribute to your research mission? Many R1 institutions were represented at the Summit on the Future of Undergraduate Geoscience Education held in January 2014 at UT-Austin (thank you Sharon Mosher and the organizing committee). If you do this exercise with your faculty, please send me the your departmental matrix and I'll compile the results.

The results of these new surveys should provide good evidence of what we should all be doing in our undergraduate programs to make sure that students have earned mastery of the concepts and skills that will ensure their success in graduate school or in the workforce.

Thanks to all for your interest. I look forward to your comments about how the assessment process works at your institutions.

Comment? Start the discussion about A Curriculum by Design Part II: Student Learning Outcomes and Program Assessment

« Pitfalls of Metaphors: Does Warm Air Hold More Water? Educating for "Sapience" »