Initial Publication Date: January 9, 2015

Implementation Programs Testing InTeGrate Materials and Data Collection

jump to: Detailed Directions

Outline (click links or scroll down for detailed instructions)

Before your course begins:

  • Fill out the InTeGrate registration form which will capture demographic information we need and give us a record of your participation in the project.
  • Implementation Program Leads will handle obtaining IRB approval - double check with your leader if your institution requires signed or implied consent (More Details)

  • Decide how you will provide data to InTeGrate (More Details)
  • Decide on a Student ID, which will be used to label all student material submitted to InTeGrate (e.g. the student ID your institution assigns each student) (More Details)
  • Decide on a mechanism to measure student engagement, preferably class attendance (More Details)
  • Set up a SERC account (More Details)
  • Complete the Course Setup Form (to initiate the processing of your official contract and access to GLE questions) (More Details)

At the beginning of your course (first week of class)

  • Collect student signed consent forms, or inform your students about the project (signed vs implied consent) (More Details)
  • Submit initial roster of student ID's on your course status page (More Details)
  • Administer any pre-measures (this may include the InTeGrate attitudinal survey and/or GLE multiple choice questions)

At the end of your course

  • Administer any post-measures (this may include the InTeGrate post attitudinal survey and/or GLE content questions (multiple choice and essays)
  • Provide a final course roster, measure of student engagement, and ungraded student assessment responses, labeled clearly to indicate which assessment they pertain to and with student ID numbers only (no names) (More Details)
  • Complete the faculty reflection survey and provide any feedback you would like to provide to the material authors (More Details)

Before Your Course Begins

Complete the following initial data collection steps prior to the beginning of the term in which you will pilot/test materials. Thinking through the information below on how you'll provide data to InTeGrate, which student ID numbers to use, etc. will help you fill out the Course Setup Form.

Institutional Review Board & Student Consent Forms

Implementation Program Leads will handle obtaining IRB approval - double check with your leader if your institution requires signed or implied consent. IRB approval is required so that InTeGrate can use the data being collected from each institution.

Decide on the Mechanisms You'll Use to Provide Your Data to InTeGrate

Read through the general considerations below in detail and think through the mechanics of how you'll provide each piece of data to InTeGrate. The best answer will depend on the particulars of how you will be collecting the data (e.g. will the student be filling out tests on paper or online). Suggestions are given below. If you're not 100% confident of how this will work in your case, we encourage you to talk this through with your webteam liaison before filling out the course setup form (see below for more information on the course setup form).

Decide on a Student ID

It's critical that each piece of assessment data you collect be associated with a student ID (no names) that unambiguously and consistently identifies the particular student whose work it is. This will allow us to match work from a given student across all the data collected. While you can select any sort of ID you like, it's important that it be something that both you and the individual students have easy access to. You'll need to provide this ID whenever you share data with us and the student will need to know this ID when they fill out the student surveys at the beginning and end of the course. If your institution issues student ID cards that carry a student ID number and also provide this same ID number to you as the instructor, then that would be a good choice.

Decide on a Per-Student Measure of Student Engagement

In order to identify disengaged students from our data collection results, you will need to submit a proxy measure of students' engagement in the course. Attendance is by far the best measure to identify student engagement, so if it is possible to take attendance without significant disruption to your course, please use that measure. If attendance is not possible, another high-frequency measure, such as percentage of assignments completed is sufficient. Rates can be submitted on a spreadsheet with InTeGrate student ID and the single student engagement measure.

Set up a SERC Account

If you don't already have an account with SERC through InTeGrate or another project, you will need to create one.

Fill out the Course Setup Form

Filling out this form sets up a data collection space for you within our website and gives us the details we need to support you in the data collection process. We encourage you to work through your answers to these questions and consult your webteam liaison if you have any questions before you fill out the form.

After you fill out the course setup form, you will be assigned a random 8-digit number which is your SERC course ID. You will also receive a link to a course status page, unique to your course, which is accessible from your SERC account page (see image on right).

Your course status page is the place where you can securely upload files containing IRB forms, GLE and summative assessment responses, and provide/update your student ID roster. You can track student completion rate of the online pre- and post-instruction attitudinal surveys as well as determine which students have completed the survey. This page will serve as your data tracking page and you will want to check the page frequently to be sure students are completing the assessments as you intend, keeping in mind that InTeGrate expects an 80% completion rate for assessments.

There is no limit to the number of courses you may have. If your course has multiple sections taught by different instructors or taught in substantively different ways (e.g. one section online, another face-to-face), please treat each section as a different course; that is, fill out the course setup form for each section in order to keep the data separated.

At the Beginning of Your Course (first week of class)

The following list of data collection steps needs to be completed prior to any instruction in your class, typically during the first week of class. Please note that this data needs to be collected before any instruction in the class, not just prior to beginning to teach the InTeGrate materials.

Collect Student Consent Forms (if needed)

Distribute signed consent forms or implied consent forms, depending on what is required at your institution. Signed forms must be returned to the SERC office.

Submit an Initial Roster of Student IDs

Your course status page provides a place to copy and paste in a list of student IDs. You'll do this both at the beginning of the course (to track how the initial student survey is going) and at the end, so that we have a final roster of students who completed the course and from whom we'll expect to see all of the pieces of assessment data. If you keep an electronic student grade book or have access to a course roster that includes the student ID numbers, this should be just a simple copy and paste. List one student ID per line without delimiters such as commas. Copying a column from Excel or Google Docs works perfectly (just be careful the column header isn't included). If you manually compile the list, you may wish to save a copy of the list on your computer for future use. You can submit this list more than once if there's an error or change in enrollment, but be sure to include the full roster each time as it does not merge the changes; rather, it overwrites previous rosters.

Direct Your Students to Take the Initial Attitudinal Student Survey

After filling out the course setup form, you will be provided with a URL, which your students will use to fill out the initial attitudinal student survey to collect their demographic data and attitudinal viewpoints (you can also find this URL on your course status page). This can be done in class or independently outside of class. You'll want to emphasize the importance of completing the form since, without it, we won't be able to use subsequent data from that student. You'll also want to clearly communicate what student ID number they should use. If the ID number the student provides on their survey doesn't match the one you provided on the course status page, then we can't use the data. As students take the survey, it will be recorded in our system and the overall count of students will be visible on your course status page. You won't have access to the individual student responses. If you are giving credit for the survey (which we encourage), you can use the 'completed student surveys' section of your course status page to get a list of student IDs that have completed the survey. You will also see a listing of student IDs that have been entered on the student survey that don't match the roster of student IDs that you provided (e.g. from student(s) who didn't follow the instructions). You can (and should) correct these student IDs using the form on the course status page (make sure you are confident in these corrections, as once 'corrected' we cannot reverse this due to student ID encryption. This will help you achieve the expected 80% matched response rate.

Administer the Initial GLE Questions (multiple choice content questions)

You must have your students answer the 8 multiple choice GLE questions. These will not necessarily correlate to content covered in the modules or your course. They are benchmarks to allow us to compare students within and between institutions. The questions must be administered in class or in another closed environment that ensures students are answering the questions independently and without outside resources. To maintain control of these questions, any GLE questions must be administered in a secure environment. They must be given as a paper & pencil quiz or on a secure LMS system and proctored in either case. Questions cannot be returned to students. You are encouraged to use any additional GLE questions that pertain to your course, remembering to maintain control of the questions. If you are planning on extracting responses from an LMS system, you MUST do a trial download with a webteam member. Student responses to the GLE questions should be sent to SERC as soon as they are available.

At the End of Your Course

The following list of data collection steps needs to be completed at the very end of instruction in your class, typically during week before or during finals week. Please note that this data needs to be collected after all instruction in the class, not just after teaching the InTeGrate materials.

Administer the Post GLE Multiple Choice Questions

At the end of the term, you need to test students on the same 8 multiple choice questions that were given in the beginning of the semester. Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the responses to return to SERC, with student ID's and no names.

Administer Two Essay Questions

Students need to complete two common essay questions as part of a summative high-stakes assessment (e.g. final exam). Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the ungraded responses to return to SERC before you perform your own grading, again with student ID's and no names. A rubric is provided for common scoring, but you can assign whatever grade you think is appropriate for your course purposes. For instance, an introductory student who answers a Level 3 question earning a score of 1 out of 3 might get 100% credit on the exam. A senior answering the same question might be expected to earn 3/3 to earn full credit.

Direct Students to Fill Out the Final Attitudinal Student Survey

As with the initial student attitudinal survey, students can complete this survey in or outside of class and will need to provide their student ID. The survey is available at the URL provided at the top of your course status page. Please remember, InTeGrate expects an 80% matched response rate for all assessment data. You can also find the URL to the survey on your course status page.

Provide Us With a Final Course Roster, a Measure of Assignment Completeness and All Collected Student Assessment Answers

Use your course status page to upload:

    1. A final roster of students who completed the entire course (using the 'student roster' box in the lower right of the course status form).
    2. A document with your measure of assignment completion for each student, and
    3. Any student assessment data (GLE questions, summative assessments) you have not yet provided. Refer to the earlier instructions on how to provide data to InTeGrate.

Complete Reflection Survey

Complete the faculty reflection survey and provide any feedback you would like to provide to the material authors. This survey will help provide information to colleagues who are considering teaching with the InTeGrate materials that you have tested. Additional audiences interested in your feedback include the InTeGrate leadership and evaluation teams, and developers of the InTeGrate materials.