University of Oregon

Institution Type: Highest Research Activity
MSI: no

Policy Level: Institution

Policy Status: Ratified

Keywords: criteria-based, multi-source, define teaching quality

Overview

The University of Oregon (UO) is undertaking a comprehensive revision of teaching evaluation to better link teaching evaluation to teaching quality. The changes began when UO faculty senators raised concerns from research about bias in teaching evaluation.In 2017, the Senate created a task force to investigate these concerns and findings of a report from our own campus indicating gender bias in student ratings and a disconnect between student learning and student ratings. The task force first was charged to experiment with student surveys but realized that a broader investigation into teaching evaluation models was necessary. It developed guiding principles that framed its work and the project going forward. That teaching evaluation should be:

• fair and transparent,
• conducted against clear criteria, and
• informed by data collected from peers, students & faculty themselves.

The group drew from research on bias in teaching evaluation (see our teaching and learning center's "2018 Statement of the Student Evaluation of Teaching" ) and closely tracked national calls for more holistic evaluation that was more consistent with evidence-based criteria and better aligned to valuing teaching as a university priority. The task force was motivated by early work from Teval (see Finkelstein et al. )and from AAU STEM Education meeting in 2017 discussing how to better align teaching evaluation policy and practice (see Dennin et. al.) Drawing from these ideas, we developed a paradigm to guide our process.

UO would seek alignment in how it "defines, develops, evaluates and rewards" teaching quality.

• Define: Good teaching should be transparently described to instructors. It shouldn't be an "I know it when I see it" proposition, but instead, focused and reflective practice over time.  At UO, there are four criteria that define quality teaching: Professional, Inclusive, Engaged, and Research-Informed.
• Develop: Professional development programs should support faculty growth as professional, inclusive, engaged and research-informed teachers.
• Evaluate: Teaching evaluation should be conducted using this same definition of teaching quality. Sources of evidence about teaching quality like peer review, student surveys, and instructor self-presentation should collect information about professional, inclusive, engaged, and research-informed teaching practices. Every faculty member should understand expectations and receive feedback related to a known definition of teaching quality.
• Reward: Teaching awards should be aligned to the same criteria. 

The sections below describe the steps UO has taken to develop and implement a multi-source, criteria-based teaching evaluation central policy. It is organized by policy change themes identified across multiple institutions.

Theme 1: Align policy change with what matters

Equity and Justice: One of the motivations for change at UO was a concern that teaching evaluation is unfair, with bias against women and faculty of color encoded into numerical rankings. UO faculty senators raised the issue of bias in teaching evaluations during the 2016-17 academic year, first in noting that faculty who uphold academic integrity standards might be scored down by students, then widening the discussion to consider national research on gender and racial bias.

This coincided with a research project at UO with local campus data about gendered rating differences in our course survey instrument. By the end of 2017, the senate created a task force to address the issues related to the problematic nature of numerical ratings.

Relevant Evidence. UO-specific evidence indicating that ratings from UO's student surveys were marred by the same problems identified in national research was an important early catalyst for action. An economics faculty member and then University Senate president, Bill Harbaugh, led a student project investigating the issue of bias in student survey ratings data from our own campus. The report found that there was a systematic pattern of lower ratings based on instructor gender, specifically for more open-ended survey questions. This matches other literature about the question-dependent nature of biased student ratings (Mengel et al.). While the overall ratings differences were relatively small compared to other published literature, they were large enough to impact the evaluations of faculty, especially when averaged ratings are used to rank faculty in a department. The Senate created a standing committee in May 2018. The committee reviewed these data indicating bias in UO's own surveys, collected examples of alternatives from vendors and other schools, and reviewed literature on problematic nature of student ratings, including meeting with some authors of major studies. The task force also convened stakeholders including deans, faculty, undergraduate and graduate student groups. Based on these conversations, the task force developed a new student survey tool and started piloting a Student Experience Survey with individual faculty members and departments. The survey continued to be piloted and revised based on feedback and internal analysis over the next year and a half. For example, the committee analyzed how well students understood and could respond to the specific questions. The committee also analyzed data from early pilots of the survey and found that for the same instructor, teaching the same course, the new survey elicited more qualitative feedback, and reduced the frequency of personal comments from 21% to 1.5% of comments.

UO's change process has had a persistent focus on refining and studying its evidence sources on teaching. Thanks to feedback from students and faculty, the teaching evaluation committee (the now-regularized task force) added a Midway Student Experience Survey in response to faculty wanting more formative feedback and students wanting the opportunity to provide feedback that could benefit their own experience, not just altruistically support course improvements for future students. Similarly, it added an Instructor Reflection survey each term using the same platform UO uses for its student surveys, responding to faculty concerns that student feedback was over-used without context specific from instructors and that many of the everyday acts of teaching innovation and change were invisible at moments of faculty review.

The University Senate valued the continued effort to assess and improve the evaluation system over time and turned that commitment into legislation requiring continued research and iteration.

Organizational Identity. These efforts were aligned with UO's identity as a research institution that values undergraduate teaching and aims to advance fairness in evaluations. As a campus that values teaching, our Teaching Engagement Program, moved into the Office of the Provost in 2018, and Provost's Teaching Academy had initiated conversations about what defines teaching quality, even before conversations about evaluation reform began. That early work to better define teaching quality was invaluable towards the work of creating policy that promoted criteria-based evaluation of teaching—inclusive, engaged, and research-informed was already familiar language to many faculty.

Between the University Senate and the Office of the Provost, UO was able to host a number of important events and conversations on our campus that spoke to UO's identity as a research university deeply invested in teaching. Our campus hosted speakers from peer institutions that were engaged in the same sorts of teaching evaluation reform efforts and invited representatives from the American Association of Universities to talk to academic leaders about better valuing undergraduate teaching.

Theme 2: Be strategic about policy content

Define teaching excellence. In the initial stages of UO's reform efforts, there was a realization that changing student surveys alone wasn't enough to address concerns about bias and a disconnect between teaching evaluation and teaching quality  – this first step would have to lead to larger changes in the overall framework and mental model for evaluating teaching. One of the Senate task force's guiding principles was that teaching evaluation should be conducted against clear criteria. Fortunately, before the teaching evaluation committee was formed, the Teaching Engagement Program had developed criteria for teaching excellence that were being used for instructor professional development on campus. These initial criteria were "inclusive, engaged, and research-informed". Through discussion and negotiation with the faculty union and University Senate, those criteria grew to include "professional" teaching, creating four criteria for teaching excellence that are used as the basis of faculty teaching evaluation at UO. The conditions defining these criteria are substantive while being broad enough for many faculty practices to align with them; this broadness makes them well-suited to customization by disciplinary colleagues. 

Advance sources of evidence: Another principle of the Senate task force was to ensure that evaluations were informed by data collected from peers, students and faculty themselves. Because we convened a faculty learning and leadership community about teaching evaluation, including peer review, we had heard from stakeholders that many evaluators felt that other sources of evidence about teaching besides the student ratings were often low quality. For example, peer reviews were sometimes perceived as "love letters," variable in content and form, and not aligned to a shared understanding of teaching quality, so of little value in either teaching improvement or teaching evaluation.  Rarely did they identify any areas for improvement or constructive critique of colleagues. Early Senate legislation called for the committee to expand its remit from revising student surveys to also bring forward motions on both policy for peer review and for an overall framework for teaching evaluation.

Make teaching evaluation feasible: By creating a teaching evaluation framework that is criteria-based, the sources of evidence collected are more directly aligned with the evaluation decisions. For example, in 2023, further senate legislated policies addressed issues with peer review practices. The senate committee and Provost's office investigated issues with current peer review policy and practices and developed a set of recommendations and templates that aimed to make peer review more efficient and effective. Faculty had noted that reviewing courses and course materials and writing a report took considerable time without additional structure to make the process more efficient. The Teaching Engagement Program, along with a group of department chairs, drafted recommendations, including using a templated peer review form and report that align to Oregon's criteria for teaching evaluation. This shortens the time for peer reviewers in that it asks for specific examples in categories and organizes feedback from the reviewers to make it simpler for evaluators to use.

Other sources of evidence, like results from Student Experience Surveys, are also summarized in alignment with the criteria for evaluation. The surveys no longer produce averaged numerical ratings, instead collecting primarily qualitative data about different teaching practices. For example, the surveys ask questions about "inclusiveness", "accessibility", and "relevance of course content", all of which are related to the definition of inclusive teaching, and therefore, results are summarized over time for that criterion. This is a change from previous reporting that organized student feedback by course section rather than by evaluation criteria, making it much more difficulty to identify patterns related to a specific criterion or to identify changes over time related to criteria for evaluation.

These changes to data collection and reporting are a crucial part of the overall evaluation changes, making it more feasible to carry out criteria-based evaluations mapping together evidence from multiple sources.

Directly address equity and inclusion. A guiding principle of the senate committee was to make teaching evaluation more fair and transparent. One of the first actions of the senate committee was to develop a disclaimer, or warning language to be included in faculty evaluations that used previous numerical rating data.

The committee also developed a policy that required continued efforts to refine the survey, as well as efforts to explicitly addresses bias by developing procedures for redacting discriminatory, obscene, or irrelevant student comments from faculty evaluation materials. The committee now oversees a process to flag and redact comments each term.

Our policy formally replaced student evaluations with "student experience surveys" which were required to prompt a student with specific prompts rather than the generic evaluative questions that elicited more gendered responses from students in the previous survey and in literature about course evaluations (Mengel et al.). This change was aimed at addressing bias in evaluation materials by mitigating a known sources of bias.

Theme 3: Make policy change someone's job

The policy change work was a collaborative effort involving multiple stakeholders. Initially driven by a task force formed by the faculty senate, the work later transitioned to a standing committee called Continuous Improvement and Evaluation of Teaching (CIET). The policy change process combined bottom-up and top-down approaches, with strong support initially from the faculty senate and the provost's office. The change frameworks used included aligning teaching evaluation with UO's "Define, Develop, Evaluate, Reward" principles and implementing criteria-based, multi-source evaluations inspired by frameworks from the TEval project and the Colorado Teaching Quality Framework.

Distribute leadership. The faculty senate, office of the provost, faculty union, and registrar's office all contributed to the policy change and implementation process. The senate CIET committee included membership from all of these stakeholder groups.  Not only did task force members share the workload of proposing and executing policy changes among themselves, they worked closely with the Provost's Office and Teaching Engagement Program to ensure policy changes could be implemented. Other faculty groups have played important roles throughout the change process so far including faculty leadership and learning communities, the Provost's Teaching Academy, Departmental Teaching Policy Leads and others.

Theme 4: Approach policy change as a process

The policy change process at UO started in 2017 and is still ongoing. It involved conducting internal research on our own campus, reviewing materials from exemplars across the country, leveraging the expertise and progress of other peer institution and organizations, collecting feedback from stakeholders including administrators, faculty and students, and working collaboratively with the faculty senate, faculty union and provost office.

Document and communicate. Early in the process the change team created a set of webpages that documented all the activities related to the change process. Those webpages include a timeline and links to other resources. These pages were a valuable way to document the process and be able to communicate as different stakeholder groups became more involved later in the process.

Support implementation and review adherence: The initial changes set an institution-wide policy for teaching evaluation that each academic department has to adopt into their own departmental policies and has the opportunity to make the policy match their departmental teaching values. This is a part of an overall process for departments to update their policies to match faculty union collective bargaining agreements. The Provost's office offers templates, workshops, stipended faculty positions to lead to policy revision work in departments, and implementation grants that support departmental efforts to align practices with new policies.

Revisiting policy over time. The faculty senate voted and approved six pieces of legislation between 2017 and 2024 related to improving teaching evaluation and the provost office negotiated multiple memorandums of understanding with the faculty senate and faculty union related to adopting new changes and the impact on faculty evaluations.

Starting in 2023, academic departments have a structured process to update more of their departmental policies to match the university-wide policy changes. As of 2024, we continue to support departments in policy change and implementation.

The senate policy emphasized the continuous nature of policy change. The committee was named the  "continuous improvement and evaluation of teaching", indicating the need to continue to investigate issues with evaluation of teaching, including revising the tools to collect evidence (like peer reviews and student experience surveys) and to investigate the issue of bias in faculty evaluation materials.

References

Dennin, M., Schultz, Z. D., Feig, A., Finkelstein, N., Greenhoot, A. F., Hildreth, M., Leibovich, A. K., Martin, J. D., Moldwin, M. B., O'Dowd, D. K., Posey, L. A., Smith, T. L., & Miller, E. R. (2017). Aligning Practice to Policies: Changing the Culture to Recognize and Reward Teaching at Research Universities. CBE—Life Sciences Education, 16(4), es5. https://doi.org/10.1187/cbe.17-02-0032‌

Finkelstein, N., Corbo, J., Reinholz, D., Gammon, M., & Keating, J. (n.d.). Evaluating teaching in a scholarly manner: A model and call for an evidence-based, departmentally-defined approach to enhance teaching evaluation for CU Boulder. Retrieved September 23, 2025, from https://www.colorado.edu/teaching-quality-framework/sites/default/files/attached-files/2017-11_tqf-white-paper_norecs.pdf

Mengel, F., Sauermann, J., & Zölitz, U. (2018). Gender Bias in Teaching Evaluations. Journal of the European Economic Association, 17(2), 535–566. https://doi.org/10.1093/jeea/jvx057