Collecting Data in Your Classroom
This page details the data collection process for both authors testing their module/courses as well as approved faculty independently testing materials for InTeGrate. The InTeGrate classroom testing process supports you in obtaining data about student learning in your class, suggests needed revisions in the materials, and helps the project understand its overall impact. (Learn more about the design of the InTeGrate assessment and evaluation).
Be sure you're familiar with the process and have thought through how you will implement it well before your course begins. If you have questions about the process, contact your webteam liaison at SERC. We expect to receive complete data (all surveys, GLE answers, summative assessments) for at least 80% of the students that complete your course. You'll want to consider how to integrate this data collection into your course in a way that ensures that level of completion.
Data Check A: Before Your Course Begins
Complete the following initial data collection steps prior to the beginning of the semester in which you will pilot/test materials. Some steps, like IRB approval, may take weeks to complete, so begin these steps as soon as you have enough information to do so. Thinking through the information below on how you'll provide data to InTeGrate, which student ID numbers to use, etc. will help you fill out the Course Setup Form.
Obtain Appropriate Institutional Review Board Approval
Every individual who will be collecting student data must gain IRB approval at their own institutions, including:
- Each materials development author who is implementing materials they wrote in their own classes. This step should happen at Checkpoint 2.
- Those who are implementing a module or course authored by another team.
- Those who are testing GLE questions.
IRB approval is required so that InTeGrate can use the data being collected from each institution. Every effort must be made to gain approval prior to testing in the classroom.
Decide on the Mechanisms You'll Use to Provide Your Data to InTeGrateRead through the general considerations below in detail and think through the mechanics of how you'll provide each piece of data to InTeGrate. The best answer will depend on the particulars of how you will be collecting the data (e.g. will the student be filling out tests on paper or online). Suggestions are given below. If you're not 100% confident of how this will work in your case, we encourage you to talk this through with your webteam liaison before filling out the course setup form (see below for more information on the course setup form).
Decide on a Student ID
It's critical that each piece of assessment data you collect be associated with a student ID (no names) that unambiguously and consistently identifies the particular student whose work it is. This will allow us to match work from a given student across all the data collected. While you can select any sort of ID you like, it's important that it be something that both you and the individual students have easy access to. You'll need to provide this ID whenever you share data with us and the student will need to know this ID when they fill out the student surveys at the beginning and end of the course. If your institution issues student ID cards that carry a student ID number and also provide this same ID number to you as the instructor, then that would be a good choice.
Decide on a Per-Student Measure of Student Engagement
In order to identify disengaged students from our data collection results, you will need to submit a proxy measure of students' engagement in the course. Attendance is by far the best measure to identify student engagement, so if it is possible to take attendance without significant disruption to your course, please use that measure. If attendance is not possible, another high-frequency measure, such as percentage of assignments completed is sufficient. Rates can be submitted on a spreadsheet with InTeGrate student ID and the single student engagement measure.
Set up a SERC Account
If you don't already have an account with SERC through InTeGrate or another project, you will need to create one.
Fill out the Course Setup Form
Filling out this form sets up a data collection space for you within our website and gives us the details we need to support you in the data collection process. We encourage you to work through your answers to these questions and consult your webteam liaison if you have any questions before you fill out the form.
After you fill out the course setup form, you will be assigned a random 8-digit number which is your SERC course ID. You will also receive a link to a course status page, unique to your course, which is accessible from your SERC account page (see image on right).×
Your course status page is the place where you can securely upload files containing IRB forms, GLE and summative assessment responses, and provide/update your student ID roster. You can track student completion rate of the online pre- and post-instruction attitudinal surveys (GLE+) as well as determine which students have completed the survey. This page will serve as your data tracking page and you will want to check the page frequently to be sure students are completing the assessments as you intend, keeping in mind that InTeGrate expects an 80% completion rate for assessments.There is no limit to the number of courses you may have. If your course has multiple sections taught by different instructors or taught in substantively different ways (e.g. one section online, another face-to-face), please treat each section as a different course; that is, fill out the course setup form for each section in order to keep the data separated.
Data Check B: At the Beginning of Your Course (first week of class)
The following list of data collection steps needs to be completed prior to any instruction in your class, typically during the first week of class. Please note that this data needs to be collected before any instruction in the class, not just prior to beginning to teach the InTeGrate materials.
Collect Student Consent Forms (if needed)
Distribute signed consent forms or implied consent forms, depending on what is required at your institution. Signed forms must be returned to the SERC office.
Submit an Initial Roster of Student IDs
Your course status page provides a place to copy and paste in a list of student IDs. You'll do this both at the beginning of the course (to track how the initial student survey is going) and at the end, so that we have a final roster of students who completed the course and from whom we'll expect to see all of the pieces of assessment data. If you keep an electronic student grade book or have access to a course roster that includes the student ID numbers, this should be just a simple copy and paste. List one student ID per line without delimiters such as commas. Copying a column from Excel or Google Docs works perfectly (just be careful the column header isn't included). If you manually compile the list, you may wish to save a copy of the list on your computer for future use. You can submit this list more than once if there's an error or change in enrollment, but be sure to include the full roster each time as it does not merge the changes; rather, it overwrites previous rosters.
Direct Your Students to Take the Initial Attitudinal Student Survey
After filling out the course setup form, you will be provided with a URL, which your students will use to fill out the initial attitudinal student survey to collect their demographic data and attitudinal viewpoints (you can also find this URL on your course status page). This can be done in class or independently outside of class. You'll want to emphasize the importance of completing the form since, without it, we won't be able to use subsequent data from that student. You'll also want to clearly communicate what student ID number they should use. If the ID number the student provides on their survey doesn't match the one you provided on the course status page, then we can't use the data. As students take the survey, it will be recorded in our system and the overall count of students will be visible on your course status page. You won't have access to the individual student responses. If you are giving credit for the survey (which we encourage), you can use the 'completed student surveys' section of your course status page to get a list of student IDs that have completed the survey. You will also see a listing of student IDs that have been entered on the student survey that don't match the roster of student IDs that you provided (e.g. from student(s) who didn't follow the instructions). You can (and should) correct these student IDs using the form on the course status page (make sure you are confident in these corrections, as once 'corrected' we cannot reverse this due to student ID encryption. This will help you achieve the expected 80% matched response rate.
Administer the Initial GLE Questions (multiple choice content questions)
You must have your students answer the 8 multiple choice GLE questions. These will not necessarily correlate to content covered in the modules. They are benchmarks to allow us to compare students within and between institutions. The questions must be administered in class or in another closed environment that ensures students are answering the questions independently and without outside resources. To maintain control of these questions, any GLE questions must be administered in a secure environment. They must be given as a paper & pencil quiz or on a secure LMS system and proctored in either case. Questions cannot be returned to students. You are encouraged to use any additional GLE questions that pertain to your course, remembering to maintain control of the questions. If you are planning on extracting responses from an LMS system, you MUST do a trial download with a webteam member. Student responses to the GLE questions should be sent to SERC as soon as they are available.
Data Check C: While You are Teaching the InTeGrate Module or CourseComplete the embedded and summative assessments as indicated in the module or course. Follow rubrics or other scoring guides provided by the authors for grading. Only the summative assessment responses will be provided to the InTeGrate project.
Administer and Collect the Summative Assessments
Your module/course will have summative assessment elements. As with other student work you are sending in, we require you to send in an ungraded copy of the student work that includes student ID's but no names. Prior to sending in the work, please make a copy of the papers. Then, you may grade your copy of the work and record the scores as you usually do. Before handing back student work, make a copy of it for yourself.
As your team begins work on revising your materials, your team may find it useful to have a full set of graded papers for each summative assessment. Your team may also want to compare how each team member used the rubric to ensure that the rubric accurately describes each item.
Data Check D: At the End of Your CourseThe following list of data collection steps needs to be completed at the very end of instruction in your class, typically during week before or during finals week. Please note that this data needs to be collected after all instruction in the class, not just after teaching the InTeGrate materials.
Administer the Post GLE Multiple Choice Questions
At the end of the term, you need to test students on the same 8 multiple choice questions that were given in the beginning of the semester. Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the responses to return to SERC, with student ID's and no names.
Administer Two Essay Questions
Students need to complete two common essay questions as part of a summative high-stakes assessment (e.g. final exam). You may choose to administer additional GLE level 3 essay questions related to your course content. Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the ungraded responses to return to SERC before you perform your own grading, again with student ID's and no names. A rubric is provided for common scoring, but you can assign whatever grade you think is appropriate for your course purposes. For instance, an introductory student who answers a Level 3 question earning a score of 1 out of 3 might get 100% credit on the exam. A senior answering the same question might be expected to earn 3/3 to earn full credit.
Direct Students to Fill Out the Final Attitudinal Student Survey
As with the initial student attitudinal survey, students can complete this survey in or outside of class and will need to provide their student ID. The survey is available at the URL provided at the top of your course status page. Please remember, InTeGrate expects an 80% matched response rate for all assessment data. You can also find the URL to the survey on your course status page.
Provide Us With a Final Course Roster, a Measure of Student Engagment and All Collected Student Assessment Answers
Use your course status page to upload:
- A final roster of students who completed the entire course (using the 'student roster' box in the lower right of the course status form).
- A document with your measure of student engagement for each student, and
- Any student assessment data (GLE questions, summative assessments) you have not yet provided. Refer to the earlier instructions on how to provide data to InTeGrate.
Complete Reflection Survey
For materials testers (non-authors): This survey will help provide information to colleagues who are considering teaching with the InTeGrate materials that you have tested. Additional audiences interested in your feedback include the InTeGrate leadership and evaluation teams, and developers of the InTeGrate materials.
For materials developers (authors): Complete the Post-pilot Reflection survey on your individual reporting page found in the reporting section of your module.