Limitations of the RTOP Instrument

Limitations of RTOP scoring for class development as a whole

RTOP scoring is useful for capturing many important elements of effective teaching. However, it has limitations that it does not always address. These include the following:

1. RTOP scoring deals with only a single class period. The variation in the class throughout the term is not evaluated. As such, all of the components in the course throughout a term cannot be determined.
2. RTOP scoring does not readily establish the interrelationship between classes.
3. Other components, such as labs, problem sets, homework, field trips, that are necessarily performed outside of the class period are not specifically evaluated.
4. RTOP scoring is not able to incorporate out-of-class student-student or student-instructor conversations.
5. RTOP scoring does not catch the out-of-class quality of grading or ungraded feedback of assignments by the instructor.

While not specifically evaluating these out-of-class components with RTOP scoring, activities such as graded readings and problem sets, extend the amount of learning a student can accomplish in a course. The students come to class prepared to learn allowing the instructor to teach at a higher level. They practice new skills and make use of new concepts in out of class assignments, preparing them for the introduction of new concepts in the next class. By making an explicit connection to these experiences the instructor activates this knowledge, a factor scored by RTOP, knits together in and out of class experiences, and enables students to construct higher level understanding.

All instructors understand that providing feedback through comments, office hours, and grading is one of the most time consuming aspects of teaching. This feedback is an integral to student learning (REF). RTOP does not capture this aspect of teaching directly.

Integrating Lecture and Lab

Although the student experience in the laboratory is not scored in the RTOP rubric, integrating explicitly the laboratory experience with lecture content will reinforce student learning, facilitate their ability to transfer their knowledge between course components, and allow them to enhance their metacognitive skills. For example, if an instructor introduced a pre-laboratory exercise in the lecture component, (s)he could allow for student-student discussions to theorize or predict (using graphs, diagrams, equations, etc. to represent phenomena) what the students might find. These practices could improve the Procedural Knowledge and the Propositional Knowledge portions of the RTOP score. Similarly, by having a post-laboratory discussion as part of a lecture, students would be able to assess the laboratory procedures, reflect on what they learned and how they learned it, activate prior knowledge, and address preconceived knowledge. These activities would increase the Lesson design and implementation and the Procedural Knowledge portions of the RTOP score. Furthermore, in large-class settings, by discussing laboratory results in lecture, students from different sections would be able to integrate knowledge enhancing the student-student interaction.

The Challenges of Collaborative Learning

While the RTOP score can miss important and good teaching practices, as described above, and result in relatively low scores, it can also over-represent activities that may not be considered good teaching and result in relatively high scores. For example, RTOP scores do not differentiate between good student collaboration and poor student collaboration. Such poor student collaboration can take several different modes. If students are not prepared for the class lesson (by not doing the appropriate level of pre-class reading, homework, etc.), the student conversation could be either: shallow, completely wrong, or even off-topic altogether. In large class settings it is difficult for the instructor to asses every group. Similarly, if the majority of group interactions are on task, a few groups may fail and may not achieve the level of understanding necessary to advance to new material. The dynamics of student groups and their ability to work together are also not captured by the RTOP. Groups may fail for a variety of reasons: one student may be over-confident or dominant, one student may be shy or unsure of her/his ability, one student may not want to work with a partner of lesser ability, or any other number of combinations. Lastly, poor design of collaborative activities could enforce student misconceptions and not provide the learning gain that is expected.

The RTOP Doesn't Measure Either What Students are Thinking or What They are Learning

Traditional lecture based instruction rests on the power of a well constructed logical sequence that supports students in learning to think about and use a set of concepts. Most faculty can recall an experience where they learned something important in this way, and most can distinguish between a strong lecture that makes a set of ideas clear and accessible, and a weak one that is confusing. These aspects of class design are scored in the RTOP scale on Procedural Knowledge, which includes items on whether the lesson focused on fundamental concepts, was coherent, used abstraction to support development of conceptual understanding, and build connections with other concepts.

One of the important lessons of the past fifteen years has been recognition that students do not always learn from these lectures in the intended way. What they 'hear' is not the same as what the professor 'says' because the meaning of language is dependent on the prior knowledge of the individual. Thus, individuals students thinking and learning from a lecture or any other learning experience will vary.

The RTOP does not directly measure students learning. It does however measure how effective the instructor is in assessing both students prior knowledge and their learning during the lesson. This assessment rests on the instructors ability to create opportunities in the class for students to share their thinking either with one another using discussion, with the whole class using strategies like concept tests, or through performance tasks involving writing, speaking, drawing or other forms of communication. By more effectively assessing students knowledge and understanding during the class, the instructor can then adjust instruction to maximize the learning of the class as a whole. A common intermediate step is to assess student learning through the answers of individual students to questions asked to the class as a whole. While this information is better than none, it typically represents the understanding of the most advanced students in the class and must be used with caution.

The Quality of Student-Instructor Interactions is Important

Infectious enthusiasm for a topic is often the mark of a well regarded instructor. In fact, studies show that this is important to learning, as is the ability of the professor to make students feel comfortable in a class. These aspects of teaching are not fully captured in the RTOP which looks at whether or not there is a climate of respect (item 20), whether or not the teacher is patient with the students (Item 23).

Similarly, an instructor's ability to draw students into the topic, either through force of personality, personal experience of the instructor, or by making the topic personally relevant to the students impacts learning.The strategies that the instructor uses to activate students prior knowledge and the connections built to other experiences are captured in items 1 and 10. However, an excellent instructor may build connections between the classroom experiences and the particular experiences of the students either as a group or as individuals. The effectiveness of the instructor in this regard is not measured.

Creating A Supportive Culture for All Students

While evidence shows that strong instruction that engages students is fundamental to broadening access to science learning, other aspects are important in working with diverse students. Within the classroom itself, the interactions with groups of students are important. Are members of minority populations in the class included appropriately in classroom activities? Are they called upon? Are their voices heard and acted upon with the same force as those from majority populations? Are steps taken to ensure their comfort in the classroom culture? Is care taken in the design of group activities to ensure both their comfort and participation? Ensuring that all students can participate fully in learning activities is a critical aspect of teaching, particularly in a discipline that does not have a strong track record for attracting students from groups underrepresented in the sciences. The RTOP measures the engagement of the students in the class as a group, it does not measure the effectiveness of the instructor in engaging specific populations within the class.

Similarly,the preparation of students in classrooms is becoming more variable. Thus developing teaching strategies that allow students to fill gaps in their preparation is important. To maximize learning for all students, those who can move more quickly or who have a stronger background also need to be challenged and supported. While the type of student-centered teaching captured by RTOP provides strong opportunities to address variable backgrounds and abilities. The RTOP does not measure the extent to which the instructor capitalizes on these opportunities.