Theme 2: Be strategic about policy content

Introduction

The content of teaching evaluation policies - meaning the exact words as well as the scope of what is included or not - can make it easier (or more difficult) to meaningfully evaluate teaching. Thus, careful and strategic consideration of policy content was a major focus of change work across LCC4 institutions.

First, institutions worked to identify and understand issues with their current teaching evaluation policies and practices, which served as motivation for making policy changes (theme 1). Then institutions strategized about which policy content changes would be productive in advancing teaching evaluation and maximizing teaching quality. People shepherding policy changes worked to strategically link the issues motivating policy changes with the content of the policy updates. This helped to narrow the focus of the policy change and motivate stakeholders to work on and ultimately approve or endorse the policy changes. For example, several institutions were motivated to change teaching evaluation policy because existing policy relied on a sole source of evidence to gauge teaching quality. This issue drove decisions to change policy to require multiple sources of evidence. Other institutions were motivated by concerns about the poor evidence that metrics used to evaluate teaching actually measured teaching effectiveness. This concern drove decisions to change policy to include more research-based assessments of teaching.

Changes to teaching evaluation policy across LCC4 reflected four types of content: 1) Including a definition of teaching excellence along with more specific criteria for evaluating with respect to the definition, 2) Requiring the use of multiple sources of evidence in order to generate more valid inferences about teaching quality, 3) Striving for feasible and manageable data collection, analysis, and use, and 4) Addressing equity and inclusion directly in policy. The specific content addressed at each institution related to what mattered for that institution (theme 1).

Define teaching excellence

A common challenge in teaching evaluation is lacking a shared understanding of what constitutes "good teaching." Therefore, multiple institutions identified the need to develop clearer definitions of what counts as excellent teaching. Some of these content revisions shifted language from vague terms such as "excellence" to terms such as "effectiveness," indicating that data have been collected to support claims about teaching quality or instructional improvement. Other content revisions offered criteria for determining teaching quality, both as a target against which teaching should be evaluated and as a foundation for developing teaching evaluation rubrics.

• The University of Portland initially had no formal definition for "excellence in teaching," which represented the sole criterion on which teaching effectiveness was judged. In embarking on policy change work, one of the first steps was to articulate a vision for teaching that identifies the facets that distinguish excellence from non-excellence in teaching. This vision will then be operationalized in content of teaching policy to be developed.
• Pomona College engaged in a collaborative process to establish a definition of inclusive teaching. Faculty were offered stipends to reflect on their teaching practices related to diversity, equity, and inclusion. An analysis of those reflections was used to establish the following definition of inclusive teaching: Inclusive teaching builds community, is student-centered, cares for the whole student, teaches critical content, and is accessible to all students.
• Georgia Southern University's College of Science & Mathematics established an ad hoc promotion and tenure committee to develop a teaching evaluation rubric to better define teaching quality. The committee reviewed literature to identify best practices for teaching and evaluation, solicited input from faculty, and worked with college leaders to draft a sample teaching evaluation rubric that describes teaching criteria and expectations for meeting the criteria. The college rubric is meant to stimulate thought and be a starting point for departments to develop their own guidelines.
• University of La Verne developed a new framework and rubric for teaching evaluation processes. The rubric articulates four domains of teaching practice that allow for more granular and accurate evaluation of teaching: course planning & preparation, learning environment, instruction, and seeking continuous improvement in teaching. For example, the learning environment domain includes the criterion of "Creates a culturally responsive environment: Value is demonstrated for students' backgrounds and experiences" and "The instructor makes meaningful connections between students' identities, background and experiences, and course content." An instructor proficient in this domain would make regular meaningful connections between content and students' identities, experiences, and backgrounds.

Advance sources of evidence

A common concern in policy change work among LCC4 institutions related to the validity of inferences made about teaching quality, especially related to the types of evidence used to evaluate teaching. The most common concerns were about relying solely on student end-of-course survey data to judge teaching quality, given that student ratings may not be related to student learning or success. Therefore, policy revisions articulated more robust and appropriate sources of evidence for teaching evaluation. At some institutions, policy changes stipulated the use of multiple sources of evidence from different perspectives related to teaching (e.g., student, peer, and instructor viewpoints). Other institutions sought to reframe evaluative tools as sources of evidence for improving teaching over time. Each of these examples strategically targets a local issue and leverages policy to shift toward gathering and using better evidence to evaluate teaching effectiveness.

• The University of Georgia had no campus-wide policy governing teaching evaluations and thus established a task force to make recommendations for such a policy. One of the primary recommendations of the task force was to ensure teaching evaluations used multiple sources of evidence given concerns about the limited usefulness of student ratings for improving teaching and the fact that class size (small vs. large), level (introductory vs. advanced), and enrollment (majors vs. non-majors) can influence student ratings in ways that are unrelated to teaching effectiveness. These initial recommendations served as the basis for the Office of Instruction to develop a policy statement for teaching evaluation.
• The University of Oklahoma Teaching Evaluation Working Group investigated promising practices in teaching evaluation that would minimize or mitigate documented issues with traditional teaching evaluation surveys. They recommended that colleges and departments incorporate multiple sources of evidence in evaluations of teaching quality, including at minimum information from both students and instructors.
• At the University of Oregon, teaching evaluation relied heavily on averaged numerical rankings from student surveys. Initial policy efforts aimed to revise the survey to emphasize student learning and reframe it as a source of evidence about students' experiences rather than the main tool for faculty evaluation. This reframe led to further policy updates to bolster the use of other sources of evidence in teaching evaluation. For instance, policy related to peer reviews was revised to provide evidence directly aligned with the institution's definition for teaching excellence. Furthermore, a centralized reflection survey system was established for instructors to provide their perspective at each stage of evaluation. Now the university-wide policy for teaching evaluation requires multi-source evaluations in which evaluators must use evidence from peers, students, and instructors themselves.

Make teaching evaluation feasible

A noteworthy concern in revising teaching evaluation policy is the time, effort, and direction needed to enable more meaningful evaluation of teaching. LCC4 institutions made concerted efforts to identify feasibility concerns for various stakeholders (e.g., students, faculty, evaluators), including concerns about efficiency (how much time and effort will this take?) and clarity of purpose (are the goals and processes sufficiently clear?). Institutions then used this information to inform policy development and refinement with the aim of mitigating as much as possible any increases in workload.

• In establishing its new teaching evaluation policy, the University of Georgia added explicit language to make it more feasible for faculty and units to follow the policy. First, units had three years to establish a peer evaluation process. Second, faculty and units were required to use multiple sources of evidence selected from a list of options, but not required to use any particular source of evidence. The hope with this flexibility was to make gathering and using the evidence more feasible. Third, exceptions were included in the policy to allow for early promotion or other special circumstances (e.g., credit for time at prior institution) that might affect the capacity of the faculty member or their unit to accumulate particular teaching-related evidence. Finally, much of the decision-making related to how to put policy into practice would be done at the unit level so that units could operationalize policy in ways that work for them, their faculty, and their discipline.
• The University of Oregon investigated issues with current peer review policy and practices and developed a set of recommendations and templates that aimed to make peer review more efficient and effective. Faculty noted that reviewing courses and course materials and writing a report took considerable time. The Teaching Engagement Program, along with a group of department chairs, drafted recommendations, including using a templated peer review form and report that align to Oregon's criteria for teaching evaluation. This shortened the time for peer reviewers by asking for specific examples in categories and organizing feedback from the reviewers to make it simpler for evaluators to use.
• Until 2023, the University of Oklahoma used an evaluation scale that was 1-5, out to 2 decimal places, for rating teaching, research, and service. This results in a lot of time and energy spent thinking about which good practices and actions counted more than others and placed talented faculty members in competition with each other over hundredths of decimal points. In 2023, the university opted to use the same 1-5 scale with integer values only. Although this was much more efficient and simplified reporting for faculty, it also required adjustment for those who are used to getting higher marks. The next step is to pilot a 3-level rating for teaching, research, and service during 2024: Improvement Needed (I), Valuable Contributions (V), and Extraordinary Impact(E). It is expected that nearly everyone will be in the V category unless there is a gap in performance or performing at a level that is far beyond the norm. The 3-tier model is accompanied by a rubric for how the tiers are combined for an overall assessment, which streamlines the process of evaluation for departmental leadership. 

Directly address equity and inclusion

Many current teaching evaluation policies do not explicitly address equity and inclusion in relation to pedagogy, which means these principles get lost in evaluating teaching and systemic inequities, and exclusions persist. Institutions in LCC4 are making policy changes to either create a standard for inclusive teaching or ensure that policies explicitly value equitable and inclusive teaching practices.

• In 2016, Pomona College faculty voted to add explicit language around creating inclusive classrooms to the section of the faculty handbook on faculty evaluation for tenure and promotion. The faculty are expected to be "attentive to diversity in the student body" and "foster an inclusive classroom where all students are encouraged to participate in discussions, studios, rehearsals, performances, activities and other course exercises." A new end-of-semester student feedback form, adopted by college faculty in 2020, also now includes open-ended questions asking how the instructor shaped inclusivity in the classroom and how the student respondent contributed to inclusivity in the classroom. Current work aims to create a framework that aligns behaviors – and sources of evidence of those behaviors – with the defining characteristics of inclusive teaching. The intention is to have this framework guide peer observation, as an additional process for formative and summative feedback during faculty evaluation.
• A revision to the University of La Verne framework for teaching evaluation integrated more specific language about considerations of diversity, equity, and inclusion across the four domains of teaching practice (i.e., course planning & preparation, learning environment, instruction, and seeking continuous improvement in teaching). The revisions included language that specifically called out efforts to create an equitable and inclusive environment in each domain. For example, the course planning & preparation domain now includes "Uses broadly applicable course materials appropriate to the discipline, course, or field that include diverse perspectives and representation." The learning environment domain includes several new criteria: cultivates safety and belonging for students, with awareness of and attention to differential life experiences; uses validating language; integrates individual and group learning approaches, as appropriate; and makes apparent usefulness or congruence of assignments or tasks to learners' educational goals. Faculty under review for annual performance, promotion, or tenure must provide tangible examples of these aspects of their teaching.
• The Faculty Senate at the University of Oregon was concerned about the potential for bias in teaching evaluation. This concern arose from an analysis of campus-wide data showing gender bias in end-of-course ratings by students, along with students' under-rating courses where they seemed to be learning more based on their grades in follow-on courses. Concern about these biases led to the establishment of a task force on changing student surveys that included fairness and transparency in teaching evaluation as one of its guiding principles.