A Curriculum by Design Part II: Student Learning Outcomes and Program Assessmentpublished Jan 20, 2014
Assessment is collecting data with a purpose. The Department of Earth Sciences completed an extensive review of student learning outcomes at the programmatic level for all courses offered by the department. Our assessment plan is designed to: a) provide faculty with an opportunity to reflect on course goals, methods and expected student learning outcomes, b) aggregate these course learning goals into an overall departmental matrix of student learning outcomes, c) provide formative feedback to improve teaching and learning in Earth Sciences courses, and d) for accountability, to demonstrate that the departmental and institutional vision and mission are being addressed and that the curriculum is consistent with contemporary professional standards in the geosciences. The resulting SLO Matrix (Excel 2007 (.xlsx) 122kB Jan16 14) provides a rapid, visual map of the "landscape" of our curriculum; you can readily see areas of emphasis, and areas that might need more attention in our curricular development. This exercise also provided our faculty the opportunity to reflect deeply on the concepts and skills they emphasize in their own courses, gave them some incentives to revise courses to respond to the SLO goals, and opened the door for more extensive curricular discussions between faculty (who generally were not aware of content/skills being taught in courses related to their own).
Work on our portfolio of SLOs followed the design, development and implementation of a fully revised curriculum for degree options in the physical sciences offered by the department (Geology, Snow Science, Paleontology); a similar review process is in progress by our geographers for our Geography and GIS/Geographic Planning degree options. This revised curriculum is now in its second year of implementation, and we conducted a formative assessment designed to determine the degree to which departmental curricular goals are being met.
The Curricular Design Process
We undertook our curriculum design process following these principles:
- We used a "backward design" process (Wiggins and McTighe, 2005) to define the profile of an ideal graduate from our program, identifying what a student should know (concepts and content knowledge, scientific "habits of mind") and be able to do (geoscience technical skills and other professional skills);
- Design of learning progressions in key areas to ensure that students earn mastery of key concepts and skills through multiple exposures throughout the curriculum (the "Rule of 3's or 4's"—if something is worth learning students need at least three or four exposures to gain mastery: exposure, familiarization, competence, and ultimately mastery of the concept or skill); this also reflects the design of an integrated and coherent curriculum in which course content in a given course reflects on lessons learned in earlier courses and anticipates applications in more advanced courses;
- Application of Bloom's Taxonomy of Cognitive Skills (Bloom 1956; updated Anderson et al., 2000) to promote development of higher-order thinking skills throughout the curriculum: a) starting with early exposure to Earth Science topics to stimulate interest and motivation to learn, and to recruit majors to the discipline, and b) to emphasize more advanced interpretation, application, analytical and synthetic reasoning skills about the Earth system for majors across the curriculum;
- Adoption of an Earth system approach, emphasizing the connections between the solid earth, oceans, weather and climate, biota and humanity;
- Preparation of students for the workforce of the 21st Century (or graduate school), including disciplinary knowledge and skills, life-long professional skills, and scientific "habits of mind"; and
- Alignment of the curriculum with the department vision and mission statements, with institutional curricular requirements (e.g. our "Core 2.0" course requirements), and with the MSU Strategic Plan.
The revised curriculum developed by the Department of Earth Sciences is described and represented graphically in Part I of A Curriculum by Design.
The Programmatic Assessment Process
Our programmatic assessment is the result of an intensive self-study by the faculty of the department to document the topics and skills that are emphasized in different courses across the curriculum and at different instructional levels.
We used the Matrix Approach to Curriculum Design module (Savina and Macdonald), from the National Association of Geoscience Teachers Building Strong Geoscience Departments project. The approach was further described by Dallas Rhodes, Georgia Southern University (2011), and adapted and expanded for use by the Department of Earth Sciences to more specifically represent departmental priorities, resources, and staffing. The major student learning outcomes fall into four main categories: Discipline Knowledge (concepts and content), Discipline-Specific Skills (technical skills), Earth Science Habits of Mind, and Professional Skills (communication, quantitative, information, and interpersonal).
These general topics are further sub-divided according to themes that are central to learning in the Earth Sciences. The identification of these sub-themes (represented in columns on the accompanying spreadsheet) was based on
- An initial Gallery Walk done by faculty at a department retreat in which faculty identified key elements of their courses [e.g. strong field component, writing-intensive, etc.]
- National initiatives that support excellence in Earth Sciences education including
- The On the Cutting Edge program (NSF TUES Phase III),
- The InTeGrate-Interdisciplinary Teaching of Geoscience for a Sustainable Future program (NSF-STEP)
- The American Geosciences Institute's Geoscience Workforce Program,
- The Association of American Geographers, and
- The U.S. Department of Labor Geospatial Technology Competency Model.
The categories of SLOs used in this SLO Matrix Template (Excel 2007 (.xlsx) 122kB Jan16 14) address the learning goals that are specific to the Dept. of Earth Sciences, Montana State University (although guided by these other national programs). Other departments should feel free to modify this matrix to meet their own departmental vision, mission, faculty, student population and resources.
All faculty in the Department of Earth Sciences contributed to this self-study. The benefit of using self-assessment data is that the survey format allowed for easy and rapid responses from all faculty. Each faculty member was responsible for only a handful of courses, and individually, it did not take a major effort for faculty to complete their part of the matrix. However, there are known issues associated with self-reporting, particularly with respect to calibrating responses from different faculty that may result in some distortion of the reliability of these data (Teo, 2012; note that one faculty member's responses are clearly not valid). In some cases, the department head worked with individual faculty to reconsider the ratings they initially submitted as their perceived contributions were often reported as being too conservative compared to their faculty peers. Overall, most faculty made a good-faith effort in identifying areas of emphasis in their courses.
Faculty provided input about their contributions to student learning outcomes (represented in rows in the accompanying spread sheet) in the courses where they have primary instructional responsibility according to this rating scale: 3 (red)= this topic is central and essential to the course goals, and is strongly emphasized throughout the course; 2 (yellow)= this topic is considered in this course and supports course learning goals; 1 (blue) = this topic is introduced in the course but is not covered in depth; blank= topic not covered. The submitted course SLOs were then compiled by the department head into a single omnibus spreadsheet for the entire department. These color coded results in the matrix provide a rapid means to survey the curricular landscape of the Department of Earth Sciences.
This input was then re-sorted into a series of separate sheets for further programmatic analysis according to:
- Rubrics within the department (we have course listings under three categories: Earth Science, ERTH, Geography, GPHY, and Geology, GEO);
- Level of offering (100, 200, 300, 400)
- Undergraduate degree options (we offer 5 degree options under our BS in Earth Sciences degree: Geography, Geographic Information Science/Planning, Geology, Paleontology and Snow Science),
- Faculty contributions to the department's educational mission, and
- The graduate program.
These sortings provide the evidence for deeper analysis of curricular strengths and points of emphasis, individual faculty contributions to student learning outcomes, representations of the overall student experiences in our degree programs, and faculty expectations for learning in our undergraduate and graduate degree programs.
In aggregate, the matrices attached to this report provide a very interesting assessment of the overall current curricular structure of the Department of Earth Sciences:
- Areas of particular strength or emphasis are readily identified (red or yellow),
- This visualization provides a road map to guide future curricular revision and development, and
- This matrix can be used as a "gap analysis" to identify specific areas in need of further development. In some cases, we may simply ask that established courses change emphasis a bit to address missing components of the curriculum; in other cases we may need to develop new courses.
This matrix is not to be viewed as a representation of workload effort for each faculty member, as many courses are taught by multiple faculty in rotation, and other courses are offered in alternate years or on demand.
Analysis of Assessment Results
This assessment process has resulted in immediate benefits:
- The course matrix has provided an important baseline of the current breadth and scope of our instructional efforts across the Earth Sciences curriculum.
- The process has provided a forum for faculty to begin or continue conversations about the overall curriculum and how their courses contribute to the overall instructional mission.
- Many faculty gained a better sense of what is being taught in related parts of the curriculum, and efforts are underway to better align our identified SLOs with learning progressions in our course sequences; and
- Many faculty have independently commented on the personal value they derived from this assessment exercise, as they reflected deeply on their course goals, areas of emphasis, and overall contributions to the curriculum.
- Overall, the faculty had never had the opportunity to see the totality of what we are covering our curriculum, and it has been a very positive thing for the department to see the depth and breadth of the scholarship that is being done in our department.
- For full disclosure, I need to note that not all faculty are happy with the new curriculum. To accommodate new degree requirements of a full year of GIS and Weather and Climate for all Earth Science majors, we have had to a) discontinue some courses that were deemed to be redundant, b) compress some courses into new hybrid courses, c) change the level of instruction of some courses, and d) move some formerly required courses into the "free elective" (not required) courses that are now offered on alternate years. Time will tell how this all plays out.
The course matrix provides the evidence that describes the degree to which we have met university (MSU) and department learning goals. (I used this as the basis of my annual program assessment report that was submitted to my administration)
1. The Department of Earth Sciences contributes to the MSU Core 2.0 Curriculum (see tab Sorted by Level, 100 Level Courses)
2. Earth Sciences courses increasingly employ an Earth Systems approach.
3. The Department of Earth Sciences has developed an integrated curriculum with learning sequences that emphasize the development of higher order thinking skills.
4. The Department of Earth Sciences prepares students a) to continue with graduate studies and/or b) to join the workforce in discipline.
Overall Results of This Assessment of Programmatic SLO's:
The Department of Earth Sciences offers robust degree programs that prepare students for the next steps in their professional development. Course content and technical skills are developed in all Earth Sciences courses to conform to professional standards and competencies. There is a strong emphasis across the curriculum on development of communication skills, quantitative reasoning, information technology skills, and interpersonal skills through cooperative and collaborative learning. Problem-solving has been reported as a highly valued skill in the geosciences, and class activities are routinely developed to simulate or replicate professional practices, and in some cases, to engage authentic research activities. Results of course work commonly make direct contributions to the body of scientific knowledge, and with applications to issues of societal interest.
Resources that Can Help Get You Started Creating the Student Learning Outcomes Matrix for Your Own Department
Dept. of Earth Sciences, Montana State University Template (Excel 2007 (.xlsx) 122kB Jan16 14); feel free to start with the Blank Template (Excel 2007 (.xlsx) 38kB Jan16 14) of SLO topics, or refer to the matrix completed by our faculty to see what a matrix for an entire degree program might look like. You will have a different array of courses, and the SLOs may be a bit different depending on your department's vision, mission, staffing, profile of students, geographic setting, departmental resources, etc.
Mary Savina and Heather Macdonald had the first description of using a matrix approach to curriculum development, posted at Building Strong Geoscience Departments. Access the Matrix Approach to Curriculum Design module.
Dallas Rhodes, Georgia Southern University, expanded on the matrix approach in his 2011 GSA poster Curriculum and Program Learning Outcomes Mapping to Enhance Program Assessment
I expanded Dallas' matrix following input from other sources:
At a faculty retreat, I used a Gallery Walk (from the Starting Point collection of teaching strategies) to get faculty involved with defining the concepts and skills they thought were most important for their courses. I used large flip chart sheets each with a teaching emphasis (e.g. technical skills like rock and mineral identification, map reading; communication skills; quantitative skills; field instruction; modeling; etc.). These were posted on walls around the room, and the faculty migrated from sheet to sheet and wrote what they did under each heading and in which classes. I also had a number of blank sheets so faculty could post methods, activities, etc. that were otherwise important to their courses. So, my preliminary matrix featured skills and concepts that were identified by the faculty.
I wanted to address expectations of employers for the modern workforce, so I referred to the AAC&U Survey of Employers (Acrobat (PDF) 146kB Jan16 14). I also used the outcomes of the InTeGrate Workforce Geoscience and the 21st Century Workforce: Considering Undergraduate Programs in the Context of Changing Employment Opportunities Workshop, and information from the the American Geosciences Institute's Geoscience Workforce Program.
I was also interested in documenting where we were teaching "geoscience habits of the mind, and I relied on information from the Synthesis of Research on Thinking and Learning in the Geosciences project that emphasized spatial reasoning, temporal reasoning, systems thinking and field instruction. I also used information from the InTeGrate workshop on Teaching the Methods of Geoscience workshop.
I also used information from the Using Date in the Classroom portal.
The first step in this process was to review the courses we offered, and to align these into our new curriculum requirements. Our overall course sequence matrix and the steps we took to realign our curriculum are posted at this blogpost (A Curriculum by Design).
Now that we have created this matrix based on a) geoscience faculty input, and b) national reports and events that largely reflect the experience and insights of academics, I plan to have the following groups fill out the matrix to see what aspects of the curriculum they value most:
- Current students in our courses for majors: what is there perspective of what they value most in their pre-professional training?
- Alumni: it will be interesting to have recent (and distant past) alumni reflect on their training in our department to see what concepts and skills have best served them in their own career trajectories; and
- Recruiters from the industries that typically hire our students: it will be interesting to see if they value the same training (concepts, skills) that we emphasize in our curriculum. What will make our students more competitive in the workforce?
- Faculty from R1 institutions: You receive our students into your graduate programs. Are you recruiting students to your graduate programs who have the knowledge, skills (and perhaps attitudes, personal attributes) such that they will be able to step into your programs and immediately contribute to your research mission? Many R1 institutions were represented at the Summit on the Future of Undergraduate Geoscience Education held in January 2014 at UT-Austin (thank you Sharon Mosher and the organizing committee). If you do this exercise with your faculty, please send me the your departmental matrix and I'll compile the results.
The results of these new surveys should provide good evidence of what we should all be doing in our undergraduate programs to make sure that students have earned mastery of the concepts and skills that will ensure their success in graduate school or in the workforce.
Thanks to all for your interest. I look forward to your comments about how the assessment process works at your institutions.
Comment? Start the discussion about A Curriculum by Design Part II: Student Learning Outcomes and Program Assessment