How are RTOP Scores Determined?
To assess the inter-rater reliability of the research team after the initial round of observations, all team members scored a video of an introductory geology course taught by a community college instructor. Scores on the video ranged from 26-52 with a mean score of 35.8 with a standard deviation of 7.6 (Figure 1).
How the Researchers Scored their Observations
Considering the various subscales, the group consistently ranked propositional knowledge high. The scores were less consistent in lesson design and implementation and procedural knowledge (Figure 2). These results lead us to focus additional attention on those individual RTOP items that showed the greatest variation among observers on a series of shared class reviews.
The initial round of 34 class observations provided a range of scores from 14-79. The average score was 39.9 with a standard deviation of 16. For comparison, during the development of the RTOP instrument, observations were conducted in 40 science classrooms, yielding a mean score of 58.25 and a standard deviation of 21.3. Consequently, our initial analyses, using a specific rubric, yields results that are consistent with those of the original RTOP instrument.