Reflective Judgment and Course Performance

Colorado College, Tricia Waters

Summary

This project examined links between students' metacognition and other learning strategies (MSLQ), reflective judgment, and their overall course performance in two undergraduate courses: A First Year Experience, two block (two-month) course and a one block upper division course. Student e-journal entries were coded for reflective judgment (RJ) at the beginning and end of each course, and questionnaires examining metacognition and other learning strategies were administered at the beginning and end of each course. Student in both courses were taught the developmental sequence of reflective judgment midway through the course. Upper division students demonstrated a higher level of reflective judgment in e-journal entries compared to First Year Experience students and their questionnaire responses indicated a greater sense of self-efficacy and greater control over their learning. Students in both courses increased in critical thinking skills from the beginning to the end of the courses.

Context

The First Year Experience (FYE) course is the first course taken by students entering Colorado College. Seventeen students took this FYE entitled "Spirit and Nature" (and taught by Keith Kester, Chemistry Department at CC). Among the course goals were that students would be able to "understand the manner in which both science and religion seek meaning in the nature of things by ordering experience and imagination." and be able to "understand the manner in which scientific and religious theoretical paradigms change in time and cultural context." The course was two blocks (7 weeks) long. The upper division course was an advanced psychology seminar on Adolescence (taught by Tricia Waters, Psychology Department at CC). Among the course goals were that students would be able to "understand, compare and apply multiple theories of adolescent development" and that they would be able to "evaluate the empirical evidence for or against major theories of adolescent development." Instructors shared a common rubric (the Four Rs, see Appendix A) for guiding students' e-journal submissions and, in the Adolescence seminar, students were asked to rate their own entries for the four Rs and to code their last entry for Reflective Judgment. The Adolescence Seminar had 14 students and was one block (3 ½ weeks) long.

Teaching Practice

In both courses, students completed the Motivated Strategies for Learning Questionnaire (MSLQ) just prior to starting the course and at the end of the course (Pre/Post). The MSLQ has subscales tapping Cognitive and Metacognition (Rehearsal, Elaboration, Organization, Critical Thinking, Self-Regulation) and Motivation (Intrinsic Goal Orientation, Extrinsic Goal Orientation, Task Value).

As part of their course assignments, students wrote e-journals that helped them integrate course material. At the beginning of each class students received a description of the e-journal assignment detailing the Four Rs of e-journal writing. e-Journal entries could range from 1) Reporting what they had read or experienced, 2) Reacting to ideas and issues, 3) Reflecting on course materials and experiences, and 4) Relating course readings, films, discussions to their own aspirations as students and as citizens of the world. They were instructed that the first two types of entries were meant as a record of experiences, while the last two would reflect personal and intellectual engagement in course material.

Midway through each course, I taught students the reflective judgment developmental model. The Reflective Judgment (RJ) model describes a developmental progression in students' epistemology. The model was developed by King (1990) who observed that the evidence people use to answer the question "How do we know what we know?" changes with development. In the three earliest stages (Pre-Reflective 1-3) knowledge is based on personal beliefs. Students first trust the data of their senses (What you see is what you get.) and later trust authority to provide knowledge (What you get is what they say.). Pre-Reflective reasoners believe that knowledge is fixed and knowable.

In the middle stages (Quasi-Reflective 4-5) knowledge is regarded as uncertain and knowledge claims may be idiosyncratic. Students in this stage will use relativistic arguments or will fill in gaps in knowledge with situational variables or personal opinion. Beliefs are justified using evidence and reason, but evidence will be chosen to fit established beliefs (cherry picking evidence to suit the argument).

Reflective Thinking (6-7) involves the recognition that knowledge is constructed using information, evidence and reason from a variety of sources. In the lower stage, students compare evidence from different perspectives and across different contexts. Winning arguments are those that have the greatest utility. At the highest level, beliefs are justified probabilistically and are based on comparing and weighing evidence, examining alternative explanations and considering the best explanation given a variety of interpretations (preponderance of the evidence, strength of the argument).

After teaching about Reflective Judgment in Keith Kester's FYE at the beginning of the year, I realized that a good test of students' understanding of Reflective Judgment might be to ask them to apply their knowledge to their own writing. As a result I changed the final exam for the Advanced Seminar to include two metacognitive tasks: Asking students to rate their own e-journals for the Four Rs and having them score their last exam question for Reflective Judgment. This decision to change the format of the exam and place greater explicit emphasis on the development of metacognitive capacities in my students was a major change in the project from what I had originally intended.

Initially, my questions were more top down (assessment of students' metacognition with questionnaires, e-journals), but I actually believe the bottom up assessment –students' own appraisal of their metacognitive skills- proved to be the most effective way to help them critically scrutinize their own writing and consider their use of evidence to support contentions.

Conclusions and Evidence

In addition to the MSLQ and e-journal assignments using the four R's, the exam requirement in the Advanced Seminar to evaluate e-journal entries and their own exam response provided more evidence of metacognitive and reflective judgment processes.

Findings:

Research Question 1: Does Reflective Judgment change across the college years?

Students' e-journal entries were coded for Reflective Judgment (RJ) using two raters (Alpha reliability = .67). FYE students scored in the Pre-Reflective range on average (M= 2.85, SD=.37) compared to Advanced Seminar students who scored in the Quasi-Reflective range on average (M=5.05, SD=.63) suggesting that Reflective Judgment changes across the college years. There was no evidence that RJ changed within the short span of the individual courses.

Research Question 2: Are Metacognition, Motivation or Reflective Judgment related to overall performance in these courses?

Students in the FYE and Advanced Seminar differed slightly in the extent to which their scores on metacognitive and motivation questionnaire responses and their e-journal Reflective Judgments related to their overall course grade. Organization, Extrinsic Motivation and Persistence played a bigger role in FYE students' overall course grade, while, for Advanced Seminar students, e-journal Reflective Judgments, Intrinsic and Extrinsic Motivation, as well as Elaboration, and Persistence were correlated with overall course performance. (See Table 1 below):

Table 1: Correlates of Final Course Grades

Follow-up Multiple Regression Analysis revealed that the combination of Reflective Judgments and Metacognitive Processes accounted for 54% of the total variation in final grades:

Table 2: Reflective Judgment and Metacognitive Processes Predicting Course Performance

Including Motivational subscales in the Multiple Regression accounted for fully 71% of the variation in final grades.

Table 3: Reflective Judgment, Metacognitive and Motivational Processes Predicting Course Performance

Research Question 3: Did Metacognition change over the context of the courses?

Critical Thinking increased in both courses (Pre M = 4.88 vs. Post M = 5.33), F(1,24) =4.95, p < .05. There was a trend toward increase in Elaboration (Pre M = 5.17 vs. Post M = 5.38), p = .07. (See Figure 1).

Implications

I am convinced that careful, integrated discussions of critical thinking skills in the context of regular course assignments and discussions improves students' metacognitive abilities. The fact that the Advanced Seminar students showed higher mean levels of Reflective Judgment is consistent with previous research (King & Kitchener, 2004; Kitchener, et al., 1993) suggesting that Reflective Judgment increases across emerging adulthood. That students' critical thinking skills increased across these courses indicates that the interventions we used may be working. The change in critical thinking was greater in the Advanced Seminar and may have been a result of asking students to evaluate their own writing.

Looking Ahead

In future courses, I hope to provide explicit direction in metacognitive development in another of my undergraduate courses, and (potentially) follow students who are majors from the middle level of the psychology curriculum through a 400-level course.

Bibliography

King, P., & Kitchener, K. (2004). Reflective judgment: Theory and research on the development of epistemic assumptions through adulthood. Educational Psychologist, 39(1), 5-18.

Kitchener, K., Lynch, C., Fischer, K., Wood, P. (1993). Developmental Range of Reflective judgment: The effect of contextual support and practice on developmental stage. Developmental Psychology, 29(5), 893-906.

Pintrich, P.R., Smith, D., Garcia, T., & McKeachie, W. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). National Center for Research to Improve Postsecondary Teaching and Learning, Ann Arbor, MI.

Wirth, K., & Perkins, D. (2008). Learning to learn.
http://www.macalester.edu/geolog wirth/CourseMaterials.html.