Assessment Team Agenda

November 9 - 11, 2014

Authors' agenda »

Assessment team workspace »

Meeting Goals

• Understand the NSF expectation for the project to have quantitative data which shows learning has occurred and formulate a plan to ensure we are in a strong position. This encompasses:
  
• GLE pre/post data in classes that use multiple modules or whole courses. How can we enable detection of a signal?
  
• Summative assessments demonstrate that students can do things after an InTeGrate learning experience that they could not do otherwise
  
• Use a lens that reflects possible future publications in considering the role of summative assessments and essay and GLE questions.
  
• What questions what would people be interested in knowing about this data?
• What data do we have to support these questions
• Next steps
• Meet new module authors and support module author teams in their first steps toward checkpoint one
  

Sunday, November 9

*All workshop activities will take place at the Carleton College Weitz Center for Creativity, Larson Room 236 and adjacent areas. Assessment team will be in Weitz 231 on Sunday and Weitz 131 for the other assessment -only aspects of the meeting.

5:00 PM Welcome/social time with appetizers and cash-only bar (could participate in icebreaker activity with module author teams, sub-meetings for those with other items)

6:00 PM Catered dinner with cash-only bar

7:15 PM Opening Remarks-Cathy Manduca

7:30 PM Discussion of assessment team work items, goals for the meeting, and future goals ( Weitz 131) (skype in Emily)

Overview of items:

• Review of current assignments and workload (Stuart)
• Team members role with new module authors, module author agenda, and critical role of summative assessment (Stuart)
  
• Description of GLE Priorities: (Emily)
1. Systems thinking questions: update and next steps (Example responses)
   
2. Developing adoptable GLE question sets organized in order to be able to detect signal (based on David S. input and NSF feedback)
• Remove questions that are not working or have never been used
• Identify any gaps
  
• Update on unused GLE questions (Kristin)
• Topics coming up on Tuesday morning:
  
• Need for additional assessment consultants (2YC, interdisciplinary, IRT expertise) (Ellen/Stuart)
• Rendez Vous potential activities (Kristin and Ellen - Cathy?)
• AGU potential activities at NAGT booth (Ellen)
  
• IRT next steps (Josh gives us GSA talk)
• Items on the backburner (which may mean an additional proposal in the future):
  
• GLE and NGSS alignment
  
• Future of GLE essay questions
  
• GLE and energy literacy
  

Monday, November 10

8.30 AM (Weitz 131) Organize work on the systems thinking question analysis (Example responses)

9:00 AM (if possible) Listen to plenary with module author teams(Larson Room 236): Introduction to course-/module-level learning outcomes Learning Outcomes (PowerPoint 2007 (.pptx) 578kB Oct8 13)- Karen Viskupic

9:30 AM Assessment working time on systems thinking questions, we:

• used the set of questions that Emily sent https://serc.carleton.edu/integrate/info_team_members/assessment_team_work/new_questions.html
  
• scored using the rubric and posted our scores to a Google doc: https://docs.google.com/spreadsheets/d/1Q-aPJjIYrx-xTKaeSUo2R3400qnFdI_ODejb1GX47JI/edit#gid=0
• talked about the responses where we had the largest spread
• rewrote the rubric to the general systems thinking question
• Then used the rubric to significantly tweak the question ala backwards design - PDF of Google doc found on our workspace https://serc.carleton.edu/integrate/info_team_members/assessment_team_work/new_questions.html
• "None of us is as dumb as all of us"
  

11:30 AM Assessment sub-teams discuss systems thinking findings from their set of student work

12:00 PM Lunch at the Language/Dining Hall on campus

1:00 PM with module author teams(Larson Room 236): Learning objectives and assessment - Stuart

2:00 PM Work time in module/course groups to develop learning outcomes and assessments- Assessment Consultants work with groups

• View Team Assignments (private)

3:30 Break

3:45 PM (Weitz 131) Report outs from assessment groups on findings about systems thinking student responses

Once systems thinking question is resolved, develop plan for adoptable GLE question sets.

• We updated Emily on the process of revising the systems thinking question, the plan to collect data from new module authors and students, and then evaluate responses on Tuesday morning and make final decision.
• Emily described the need to develop sets of GLE multiple choice questions in sets of 20. This would be the existing 8 questions that all are piloting along with an additional set of 12. The rationale to creating these sets is two fold:
1. A larger set of questions is needed in order to have a hope of detecting a "signal" for the success of InTeGrate modules, particularly when a faculty member is using multiple modules or a full course.
2. Identify sets of questions and describe potential courses where they might best fit.
• Potential Spring 2015 pilots: Soils (Aida) Water and Cities (Aida - most have already piloted), Cli-Fi (Susan) Carbon/Climate/Energy (Leilani - doesn't think they'll be ready)?, potentially offshoot Enviro Justice en espanol, Potentially interdisciplinary water course (Gosselin) (Josh - doesn't think they will be ready), critical zone (Stuart - most have already piloted).
  
• First Task: The assessment consultant assigned to a spring pilot will pair with another consultant and make a first pass at identifying GLE multiple choice questions that might be applicable to the particular module.
• Second Task: Identify level 2 questions that do not work in that you can only get a 0 or a 2 not a "1." These questions need to be revised.
• At first glance, none of the GLE multiple choice seem to work for soils
• Cathy discussed Rendez Vous https://serc.carleton.edu/earth_rendezvous/2015/index.html
• Cathy would need to know this week if this group wants to add something specific. Research teams to address the questions we want to ask of the data is one possibility.
• Cathy indicated that InTeGrate leadership would be discussing the idea of research teams at the January virtual meeting.
  

If the systems thinking question is resolved before the teams break, consultant can rejoin their module author team.

5:30 PM Roadcheck

6:30 Dinner in Northfield - possible locations in folder but no organized reservations have been made.

Tuesday, November 11

8.30 AM - 9:00 AM Regroup on student data and module author teams(Weitz 131) (Ellen will be giving timeline talk for module authors)

9:00 AM - 10:00 AM Work with teams

10:00 AM (call Emily) Inform her of final systems thinking question and finish any work on it.

11:15 AM (Kristin and Ellen giving data collection talk for module authors)

11:00 AM Work with teams as needed

• Next steps with GLE question sets - Leilani, Aida, and Stuart will be working asynch. on this and will be in touch with Emily
  
• IRT next steps (Josh gives us GSA talk) - maybe schedule a webinar with assessment team on this
  
• Need for additional assessment consultants (2YC, interdisciplinary, IRT expertise) (Ellen/Stuart) - Mary sent ideas
  
• Rendez Vous potential activities (Kristin and Ellen - Cathy?) covered on Monday
  
• AGU potential activities at NAGT booth (Ellen) Emily didn't think we needed this at this point
  

12:00 PM Lunch at the Language/Dining Hall on campus

1:00 PM Plenary session: Developing and using rubrics - Josh

Work with module author teams as long as time permits