Evaluating Learning

Course Evaluation | Student Responses to Course Evaluation Questionnaire | Evaluation and Assessment plan | Assessment of Students | Grading - Instrumental Analysis | Grading - Introduction to Urban Studies | Laboratory - Instrumental Analysis | Exams and Quizzes - Introduction to Urban Studies | Writing Assignments - Introduction to Urban Studies

Course Evaluation

At Vassar College, we have employed the course intersection method a total of five times in the past four years, and in two different class settings. Our primary use of the method has been with the Instrumental Analysis and Introduction to Urban Studies classes, as described in this SENCER
model. We also used the method to intersect two special summer courses in the Summer of 2003. These courses were part of a Vassar summer
program called Exploring Transfer / Exploring Research (ET/ER).

The Exploring Transfer program has been a Vassar program for over 20 years. It is a five week college-credit program that brings community college students who has the interest and potential to pursue a four year degree to campus, for an intense classroom experience. Students from the community colleges, mostly from campuses in the urban or suburban New York City area, apply for the program in a competitive process. The courses taught in the ET summer program are often taught by teams of faculty from Vassar and the community colleges, and the subjects covered range across the curriculum, but do not specifically include a laboratory science experience. For the past four years, Vassar has been able to offer a parallel program, Exploring Research, where the same student audience is introduced to laboratory science courses at the level and pace they would find at a typical four-year institution.

In the summer of 2003, one of the ER courses, taught by Stuart Belli and Christopher Smart (both of the Department of Chemistry) was called "The
Molecular Nature of Nature" and served as an introduction to the chemical study of natural phenomena. A second course, taught in the ET program by Pinar Batur (Sociology and Urban Studies) and Christopher Roellke (Education and Urban Studies) was called "Inequality in the City: Perspectives on Education and Urban Policy." These two courses were brought together for a two week period to study lead contamination in urban schools, in much the same way that the Instrumental Analysis and Introduction to Urban Studies classes were intersected as described in the preceding pages.

At the end of the 2003 ET/ER summer program, the students in both classes were asked to assess the intersection period specifically through a questionnaire that the four faculty members developed together for that purpose. The results are shown below.

Student Responses to Course Evaluation Questionnaire

Students rated the following on a scale of 1 (low effectiveness) to 5 (high effectiveness):

How effectively did the unit on lead poisoning expand your knowledge of how scientific research is conducted?

MEAN RESPONSESLectureReadingsGroup DiscussionAssignments
Policy Students (n=19)4.424.264.214.63
Science Students (n=14)3.863.864.074.0

How effectively did the unit on lead poisoning expand your knowledge of urban theory/policy?

MEAN RESPONSESLectureReadingsGroup DiscussionAssignments
Policy Students (n=19)4.474.424.374.79
Science Students (n=14)2.152.933.142.57

How effectively did the unit on lead poisoning broaden your understanding of how science and urban theory/policy interact?

MEAN RESPONSES
Policy Students (n=19)4.63
Science Students (n=14)3.64[end tr]

Students rated the following on a scale of 1 (decreased a lot) to 5 (increased a lot):

MEAN RESPONSESPolicy Students(n=19)Science Students (n=14)
As a result of the unit on lead poisoning, has
your interest in studying science increased or decreased?
3.954.50
As a result of the unit of lead poisoning has your interest in studying public policy increased or decreased?4.743.36
As a result of the unit on lead poisoning, has your interest in studying how science and public policy interact increased or decreased?4.423.71

Selected (and representative) Comments/ Suggestions (Policy Student Responses, n=19)

  • Need for more shared class time with science and policy students
  • Allow policy students to observe scientific data analysis more closely
  • More time for policy students to teach science students how policy works
  • Group activities and interactions highly effective; my favorite part of the program
  • Intellectually stimulating!

Selected (and representative) Comments/ Suggestions (Science Student Responses, n=14)

  • Scientific knowledge increased significantly by using ICP and XRF machines
  • Need for more and improved interaction and communication between science and policy students
  • Small group meetings were some of the best academic discussions at Vassar
  • Never thought chemistry could be fun!

When the course intersection method was used during the regular academic year at Vassar, in the Instrumental Analysis and Introduction to Urban Studies courses, the intersection period was not assessed separately from the course as a whole. Assessment of each course by the college's standard course evaluation questionnaires yielded a range of comments. But for the future, we are planning to introduce 5 level system of evaluation.

Evaluation and Assessment Plan

The evaluation efforts will be coordinated by Vassar's Department of Education  and will include external review. Our evaluation and assessment plan consists of five integrated and overlapping components, which are consistent with the types of mixed-method evaluations outlined in the NSF Evaluation Handbook (NSF97- 153, 1997) and the current literature (Atkin, 2001; Committee on Undergraduate Science Education, 1999; Pellegrino, 2001; Towne, 2001).

Faculty Focus Groups: Formative and Summative Assessment.

Monthly focus can be important tools prior to, during, during, and following the implementation of a pilot program (Krueger, 1994). In addition to soliciting formative and summative faculty assessment regarding strengths and weaknesses of course design and implementation, focus group facilitators will inquire about both short-term and long-term institutional support that may be required to sustain and strengthen the program. Periodically, focus groups will include additional faculty members from the Chemistry and Urban Studies departments (or other faculty involved in multidisciplinary programs) to discuss pedagogy, curricular alignment within majors.

Faculty Assessment of Student Learning.

The expected academic outcomes of these courses require a combination of quantitative and qualitative student assessment tools. In addition to more traditional essays and exams, students will be expected to demonstrate their scientific literacy and understanding of urban policy through performance assessments. These "authentic" measures of student learning will include the conducting of field experiments, data analysis, reporting of results, role-play simulations, and debating of "real-life" case study scenarios.

Student Formative Assessment: Quality Circles.

Quality circles will be conducted periodically throughout the semester. Quality circles will be conducted with small groups of students and will be facilitated by teaching interns in Urban Studies and Chemistry. The purposes of the quality circles are two-fold: 1) to provide students with an opportunity to reflect critically on the readings and instructional strategies utilized in the Chemistry 362 and Urban Studies 100 courses; 2) to provide the instructors with constructive formative feedback for course improvement. Student feedback on instruction and course design is typically solicited at the end of the semester in the form of standardized course evaluation questionnaires. These quality circles, on the other hand, give students the opportunity to initiate possible curricular adjustments during the course itself. We view students as partners in curriculum change and "shared course ownership" is a guiding principle of these quality circles. Another specific purpose of the quality circles is for students to provide a form of "peer assessment" of their colleagues across courses. That is, urban studies students can provide constructive feedback to the analytic chemistry students on their presentation of scientific findings. Similarly, analytic chemistry students can assess the quality of the policy proposals put forth by their colleagues in urban studies.

Student Summative Course Assessment: CEQ's, Exit Questionnaire,SALG.

Standard course evaluation questionnaires will be administered by Vassar's Office of the Registrar. Students will provide quantitative ratings on course organization, readings, assignments, exams and laboratory components. In addition, an exit questionnaire, designed and administered by our external reviewer (see below), will allow student feedback on course components that are idiosyncratic to Chemistry 362 and Urban Studies 100 (the cross course collaborations, cooperative learning strategies, performance assessments, role-play simulations, case study scenarios, etc.). SENCER's Student Assessment of Learning Gains (SALG), or some adaptation of it, would be a useful, externally designed assessment tool.

External Review.

We envision an external reviewer with expertise in SENCER approaches to undergraduate curriculum reform. The external reviewer will have access to focus group findings, student formative and summative assessments, and curricular materials associated with the project. We have identified a number of experienced SENCER faculty who would be excellent candidates for this type of external review.

Assessment of Students

In both of the two settings in which we used the course intersection method, the students were assessed by similar means. The science students were graded on the lab work involved in generating data for the lead studies that the students designed (i.e. determination and interpretation of lead levels in the samples collected). They were also assessed on the presentation that they made to the "policy makers," including their handling of questions and discussion. The students in the Urban Studies classes were assessed on policy presentations that they made based on the lead level results, and their handling of questions and discussion at those presentations. They were also assigned an opinion paper on the topic of lead contamination; an example of this assignment can be found in the Urban Studies course syllabus. Regardless of our efforts to assess student learning, we acknowledge that we do not know the long term effects of this experience on our students; we only hope that it helps them become more effective citizens.

Grading - Instrumental Analysis

Exams (3 plus final) –> 45%

Laboratory–> 45%
Write-up
Lab Notebook

Homework–> 10%

Grading - Introduction to Urban Studies

In Class Midterm–> 25%

Take-home Midterm (Observation Report) –> 20%

Short opinion papers–>20%

Final Exam–> 25%

Class Participation–>10%

Laboratory - Instrumental Analysis

The laboratory will have two components; short exercises designed to acquaint you with the operation of a specific instrument and longer experiments and/or investigations of much less structure intending to give you practice in adapting, designing, and applying analytical methods
to real problems. To avoid wasting your time waiting for instruments and equipment, there will be several experiments running simultaneously which you can rotate through. The exercises will have you collecting data and evaluating the effects of various instrument parameters. You will receive detailed procedures and the write-ups will also be closely scripted. For the investigations we will broaden our attention from the measurement itself to the complete analysis; everything from defining the question to devising a method and evaluating our results. I would like to give as much flexibility on the investigations as possible to allow each of you to explore your interests.

Exams and Quizzes - Introduction to Urban Studies

The first mid-term examination will be a closed book, in class exam, combining short answer and essay questions. It usually consists of 5 or 6 short answer questions and 2 essays.

The second mid-term examination (Observation Report) will be a challenge to encourage you to see and understand and to interpret the urban conundrum. For your Observation Report, you are expected to choose a location, such as downtown, or Main Street, or the neighboring K-Mart, or perhaps a soup kitchen. You are expected to report on what you see, and by giving examples, demonstrate how your readings and discussions in the class have improved the sharpness of your vision. In your take-home exam, you will be expected to write a theoretical discussion on what you observed. Even though it resembles torture, this exam is meant to be a learning experience. You will be allowed to discuss the assignment with other students and with me, and you may conduct additional research in the library in support of your essay. It is acceptable to ask for assistance in finding data or reference works, and even editorial advice. But the take-home exam cannot be written in collaboration with another person. In your essay you will be expected to cite sources you use, and to make proper use of references and quotations. The objective is to encourage you to utilize class material to analyze the "urban" in abstract and everyday life.

The comprehensive final examination will be given according to the college final exam schedule. It will consist of 5 or 6 short answer questions, and 2 essay questions.

Writing Assignments - Introduction to Urban Studies

A five page Opinion Paper is due after each section of the class, to tie each section's discussion to the others. The four short Opinion Papers (actually, there are five of them, but one of them will be given as an essay question at mid-term.) will become a guide to disciplinary diversity in Urban Studies.