Collecting Data in Your Classroom

This page details the data collection process for both authors testing their module/courses as well as approved faculty independently testing materials for InTeGrate. The InTeGrate classroom testing process supports you in obtaining data about student learning in your class, suggests needed revisions in the materials, and helps the project understand its overall impact. (Learn more about the design of the InTeGrate assessment and evaluation).

Be sure you're familiar with the process and have thought through how you will implement it well before your course begins. If you have questions about the process, contact your webteam liaison at SERC. We expect to receive complete data (all surveys, GLE answers, summative assessments) for at least 80% of the students that complete your course. You'll want to consider how to integrate this data collection into your course in a way that ensures that level of completion.

Data Check A: Before Your Course Begins

Complete the following initial data collection steps prior to the beginning of the semester in which you will pilot/test materials. Some steps, like IRB approval, may take weeks to complete, so begin these steps as soon as you have enough information to do so. Thinking through the information below on how you'll provide data to InTeGrate, which student ID numbers to use, etc. will help you fill out the Course Setup Form.

Obtain Appropriate Institutional Review Board Approval

Every individual who will be collecting student data must gain IRB approval at their own institutions, including:

• Each materials development author who is implementing materials they wrote in their own classes. This step should happen at Checkpoint 2.
• Those who are implementing a module or course authored by another team.
  
• Those who are testing GLE questions.

IRB approval is required so that InTeGrate can use the data being collected from each institution. Every effort must be made to gain approval prior to testing in the classroom.

Read more about how to obtain IRB and why approval is necessary

Hide

Obtaining IRB Approval

Because the student assessment data is being used for research, we need to consider how human research guidelines apply. Many institutions have Institutional Review Boards that oversee this sort of data collection. Each institution implements this differently so you'll need to explore the situation at your institution. Carleton College has approved the InTeGrate project as EXEMPT (Acrobat (PDF) 106kB May22 12). This approval may be sufficient for you to provide the collected student data to the InTeGrate project. However, each person collecting student data needs to check with their institution as the institution may require a separate IRB application.

Although Carleton College approved the application as EXEMPT, you still need to inform your students about this research so that the students have the option to not have their student assessments included. This can be completed in two ways and you will need to select the option that aligns with your institution's interpretation of the EXEMPT status:

• Signed consent: In this case copies of the signed consent forms (with student names and InTeGrate ID's) would need to be sent and filed at the SERC office (Carleton College prefers signed student consent forms (Microsoft Word 31kB Jan15 13) as the means to ensure that students are informed and have the opt-out option).
• Implied consent: In this case, a list of students who have opted out (with their correlating InTeGrate ID's) should be provided to the SERC office (the University of Akron allows for the implied consent form (Microsoft Word 52kB May22 12)).

The completed IRB application for Carleton College can be found on the Forms and Documents page and can be used as reference if you are required to complete the process on your own at your institution.

If your institution does not have an Institution Review Board then the Carleton IRB approval is sufficient, as this would fall under Carleton's multi-site involvement of assessment research. You will still need to provide students a consent form or letter so they have an opportunity to opt out (see above) and to share the Carleton Exempt letter as well as the student consent form you will be using with your campus leader.

Decide on the Mechanisms You'll Use to Provide Your Data to InTeGrate

Read through the general considerations below in detail and think through the mechanics of how you'll provide each piece of data to InTeGrate. The best answer will depend on the particulars of how you will be collecting the data (e.g. will the student be filling out tests on paper or online). Suggestions are given below. If you're not 100% confident of how this will work in your case, we encourage you to talk this through with your webteam liaison before filling out the course setup form (see below for more information on the course setup form).

Show important considerations for submitting data to InTeGrate

Hide

Important considerations for submitting data to InTeGrate

1. Label all data with the corresponding student ID (no student names). We've found this is often easier to do by hand before scanning, but can be added electronically as well (blue painters tape works well to black out names, but can be removed without damaging the paper. This may not work with all scanners). (see below)
2. Provide data in a spreadsheet where feasible. For data that students have provided online (e.g. assessments done through an LMS), or for simple multiple choice questions (like the initial GLE questions), you can create a simple spreadsheet with "Student ID" in the first column followed by one question per column. The text of the question can be used as the header for each column, and student responses can be listed, one per row, down the spreadsheet. A single spreadsheet can contain multiple questions (e.g. from the GLE and a summative assessment) as long as the questions are clearly labeled. Be sure to provide the actual student answer and not simply the graded result (e.g. we need to know that the student selected "b" in a multiple choice test, not simply that they got it right or wrong).
3. If a spreadsheet isn't feasible, try to provide the data in an electronic format. For example, it might be impractical to transcribe hand-written student answers from an exam, or students may have handed in an assignment as a Word document. In these cases, you can provide the original electronic document (as long as it's in a widely used format) or scan the original student work. Many modern multi-function copiers, as are frequently found in departmental offices, are capable of scanning and outputting PDFs of documents in exactly the same way you might photocopy those documents. This is likely the simplest way for you to provide us with copies of hand-written student work: scan the stack with your copier and provide us with the resulting PDF file. Standard image formats are acceptable if scanning to PDF is unavailable. JPEG and PNG formats are the safest bet for image scans. Be sure the documents are clearly labeled with the student ID. And in cases where the data needed by the project is mixed with other material (e.g. a GLE question embedded in a larger exam), consider how you might scan only the part that is relevant.
4. Provide all electronic documents through the upload forms on the course status page (see below), not through email. This will help us track the data accurately while also ensuring our strategies for keeping student data anonymous are enforced. If you have more than one file to send as part of a collection or set (e.g. you have 10 scanned PDFs to send for summative assessment), you'll want to compress those ten files into a ZIP archive and upload only the ZIP archive instead of 10 individual files. Zipping files will be faster for you to send and for us to process. Details on how to create a ZIP file are at the bottom of the file upload section on your course status page. The InTeGrate assessment team will be notified when new files have been uploaded. Once the assessment team has processed the file, the files will be marked "accepted" on your course status page.
5. We need the ungraded student responses. If you're are going to mark up the student answer as part of your grading, please scan or copy the answer before grading and provide us with the unmarked copy.
6. If you need to provide the data as paper, do not send the only copy of the documents. Always photocopy the original documents before sending. Documents can be mailed to the SERC office and should include a cover sheet with a clear description of what they are (e.g. what course they relate to, what questions are included). Packages should be sent with a tracking number regardless of shipment method (USPS, FedEx, etc.). Please forward this tracking number to Krista Herbstrith so that we know when to expect the package (kherbstr (at) carleton (dot) edu).
   

   Science Education Resource Center
   Attn: MJ Davenport
   200 Division Street, Suite 210
   Northfield, MN 55057
   

Decide on a Student ID

It's critical that each piece of assessment data you collect be associated with a student ID (no names) that unambiguously and consistently identifies the particular student whose work it is. This will allow us to match work from a given student across all the data collected. While you can select any sort of ID you like, it's important that it be something that both you and the individual students have easy access to. You'll need to provide this ID whenever you share data with us and the student will need to know this ID when they fill out the student surveys at the beginning and end of the course. If your institution issues student ID cards that carry a student ID number and also provide this same ID number to you as the instructor, then that would be a good choice.

Decide on a Per-Student Measure of Student Engagement

In order to identify disengaged students from our data collection results, you will need to submit a proxy measure of students' engagement in the course. Attendance is by far the best measure to identify student engagement, so if it is possible to take attendance without significant disruption to your course, please use that measure. If attendance is not possible, another high-frequency measure, such as percentage of assignments completed is sufficient. Rates can be submitted on a spreadsheet with InTeGrate student ID and the single student engagement measure.

Set up a SERC Account

If you don't already have an account with SERC through InTeGrate or another project, you will need to create one.

Fill out the Course Setup Form

Filling out this form sets up a data collection space for you within our website and gives us the details we need to support you in the data collection process. We encourage you to work through your answers to these questions and consult your webteam liaison if you have any questions before you fill out the form.

×

The link to your course status page can be found on the right of your account page, as outlined in red in this screen shot.

Provenance: Matt Lauer, Carleton College
Reuse: This item is offered under a Creative Commons Attribution-NonCommercial-ShareAlike license http://creativecommons.org/licenses/by-nc-sa/3.0/ You may reuse this item for non-commercial purposes as long as you provide attribution and offer any derivative works under a similar license.

After you fill out the course setup form, you will be assigned a random 8-digit number which is your SERC course ID. You will also receive a link to a course status page, unique to your course, which is accessible from your SERC account page (see image on right).

Your course status page is the place where you can securely upload files containing IRB forms, GLE and summative assessment responses, and provide/update your student ID roster. You can track student completion rate of the online pre- and post-instruction attitudinal surveys (GLE+) as well as determine which students have completed the survey. This page will serve as your data tracking page and you will want to check the page frequently to be sure students are completing the assessments as you intend, keeping in mind that InTeGrate expects an 80% completion rate for assessments.

There is no limit to the number of courses you may have. If your course has multiple sections taught by different instructors or taught in substantively different ways (e.g. one section online, another face-to-face), please treat each section as a different course; that is, fill out the course setup form for each section in order to keep the data separated.

Data Check B: At the Beginning of Your Course (first week of class)

The following list of data collection steps needs to be completed prior to any instruction in your class, typically during the first week of class. Please note that this data needs to be collected before any instruction in the class, not just prior to beginning to teach the InTeGrate materials.

Collect Student Consent Forms (if needed)

Distribute signed consent forms or implied consent forms, depending on what is required at your institution. Signed forms must be returned to the SERC office.

Submit an Initial Roster of Student IDs

Your course status page provides a place to copy and paste in a list of student IDs. You'll do this both at the beginning of the course (to track how the initial student survey is going) and at the end, so that we have a final roster of students who completed the course and from whom we'll expect to see all of the pieces of assessment data. If you keep an electronic student grade book or have access to a course roster that includes the student ID numbers, this should be just a simple copy and paste. List one student ID per line without delimiters such as commas. Copying a column from Excel or Google Docs works perfectly (just be careful the column header isn't included). If you manually compile the list, you may wish to save a copy of the list on your computer for future use. You can submit this list more than once if there's an error or change in enrollment, but be sure to include the full roster each time as it does not merge the changes; rather, it overwrites previous rosters.

Show student ID encryption details

Hide

All student IDs stored on SERC's servers are encrypted using a one-way hash to protect the students' anonymity. This is the same way that passwords are stored, and the IDs cannot be unencrypted. To correlate records across many student responses and surveys, it is important the student and faculty always use the exact same ID (no names). The institutionally-assigned student ID that shows up on each student's school ID card and on class lists supplied to you by your registrar is likely the simplest solution because it is accessible to both you and your students.

Show Non-participating Students

Hide

If you have students who cannot participate in the study (if they are under 18 years old or opted out via IRB), they will be flagged by our system and their data will not be included in the final data set, even if you (or they) supply it. Surveys received by students under 18 will be automatically flagged and IRB exclusions will be entered manually into the system on our end.

Direct Your Students to Take the Initial Attitudinal Student Survey

After filling out the course setup form, you will be provided with a URL, which your students will use to fill out the initial attitudinal student survey to collect their demographic data and attitudinal viewpoints (you can also find this URL on your course status page). This can be done in class or independently outside of class. You'll want to emphasize the importance of completing the form since, without it, we won't be able to use subsequent data from that student. You'll also want to clearly communicate what student ID number they should use. If the ID number the student provides on their survey doesn't match the one you provided on the course status page, then we can't use the data. As students take the survey, it will be recorded in our system and the overall count of students will be visible on your course status page. You won't have access to the individual student responses. If you are giving credit for the survey (which we encourage), you can use the 'completed student surveys' section of your course status page to get a list of student IDs that have completed the survey. You will also see a listing of student IDs that have been entered on the student survey that don't match the roster of student IDs that you provided (e.g. from student(s) who didn't follow the instructions). You can (and should) correct these student IDs using the form on the course status page (make sure you are confident in these corrections, as once 'corrected' we cannot reverse this due to student ID encryption. This will help you achieve the expected 80% matched response rate.

Administer the Initial GLE Questions (multiple choice content questions)

You must have your students answer the 8 multiple choice GLE questions. These will not necessarily correlate to content covered in the modules. They are benchmarks to allow us to compare students within and between institutions. The questions must be administered in class or in another closed environment that ensures students are answering the questions independently and without outside resources. To maintain control of these questions, any GLE questions must be administered in a secure environment. They must be given as a paper & pencil quiz or on a secure LMS system and proctored in either case. Questions cannot be returned to students. You are encouraged to use any additional GLE questions that pertain to your course, remembering to maintain control of the questions. If you are planning on extracting responses from an LMS system, you MUST do a trial download with a webteam member. Student responses to the GLE questions should be sent to SERC as soon as they are available.

Data Check C: While You are Teaching the InTeGrate Module or Course

Complete the embedded and summative assessments as indicated in the module or course. Follow rubrics or other scoring guides provided by the authors for grading. Only the summative assessment responses will be provided to the InTeGrate project.

Administer and Collect the Summative Assessments

Your module/course will have summative assessment elements. As with other student work you are sending in, we require you to send in an ungraded copy of the student work that includes student ID's but no names. Prior to sending in the work, please make a copy of the papers. Then, you may grade your copy of the work and record the scores as you usually do. Before handing back student work, make a copy of it for yourself.

As your team begins work on revising your materials, your team may find it useful to have a full set of graded papers for each summative assessment. Your team may also want to compare how each team member used the rubric to ensure that the rubric accurately describes each item.

Data Check D: At the End of Your Course

The following list of data collection steps needs to be completed at the very end of instruction in your class, typically during week before or during finals week. Please note that this data needs to be collected after all instruction in the class, not just after teaching the InTeGrate materials.

Administer the Post GLE Multiple Choice Questions

At the end of the term, you need to test students on the same 8 multiple choice questions that were given in the beginning of the semester. Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the responses to return to SERC, with student ID's and no names.

Administer Two Essay Questions

Students need to complete two common essay questions as part of a summative high-stakes assessment (e.g. final exam). You may choose to administer additional GLE level 3 essay questions related to your course content. Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the ungraded responses to return to SERC before you perform your own grading, again with student ID's and no names. A rubric is provided for common scoring, but you can assign whatever grade you think is appropriate for your course purposes. For instance, an introductory student who answers a Level 3 question earning a score of 1 out of 3 might get 100% credit on the exam. A senior answering the same question might be expected to earn 3/3 to earn full credit.

Direct Students to Fill Out the Final Attitudinal Student Survey

As with the initial student attitudinal survey, students can complete this survey in or outside of class and will need to provide their student ID. The survey is available at the URL provided at the top of your course status page. Please remember, InTeGrate expects an 80% matched response rate for all assessment data. You can also find the URL to the survey on your course status page.

Provide Us With a Final Course Roster, a Measure of Student Engagment and All Collected Student Assessment Answers

Use your course status page to upload:

1. A final roster of students who completed the entire course (using the 'student roster' box in the lower right of the course status form).
2. A document with your measure of student engagement for each student, and
3. Any student assessment data (GLE questions, summative assessments) you have not yet provided. Refer to the earlier instructions on how to provide data to InTeGrate.

Complete Reflection Survey

For materials testers (non-authors): This survey will help provide information to colleagues who are considering teaching with the InTeGrate materials that you have tested. Additional audiences interested in your feedback include the InTeGrate leadership and evaluation teams, and developers of the InTeGrate materials.

For materials developers (authors): Complete the Post-pilot Reflection survey on your individual reporting page found in the reporting section of your module.