Implementation Programs Testing InTeGrate Materials and Data Collection

Outline (click links or scroll down for detailed instructions)

Before your course begins:

Fill out the InTeGrate registration form which will capture demographic information we need and give us a record of your participation in the project.
Implementation Program Leads will handle obtaining IRB approval - double check with your leader if your institution requires signed or implied consent (More Details)

Read more about how to obtain IRB and why approval is necessary

Obtaining IRB Approval

Because the student assessment data is being used for research, we need to consider how human research guidelines apply. Many institutions have Institutional Review Boards that oversee this sort of data collection. Each institution implements this differently so you'll need to explore the situation at your institution. Carleton College has approved the InTeGrate project as EXEMPT (Acrobat (PDF) 106kB May22 12). This approval may be sufficient for you to provide the collected student data to the InTeGrate project. However, each person collecting student data needs to check with their institution as the institution may require a separate IRB application.

Although Carleton College approved the application as EXEMPT, you still need to inform your students about this research so that the students have the option to not have their student assessments included. This can be completed in two ways and you will need to select the option that aligns with your institution's interpretation of the EXEMPT status:

Signed consent: In this case copies of the signed consent forms (with student names and InTeGrate ID's) would need to be sent and filed at the SERC office (Carleton College prefers signed student consent forms (Microsoft Word 31kB Jan15 13) as the means to ensure that students are informed and have the opt-out option).
Implied consent: In this case, a list of students who have opted out (with their correlating InTeGrate ID's) should be provided to the SERC office (the University of Akron allows for the implied consent form (Microsoft Word 52kB May22 12)).

The completed IRB application for Carleton College can be found on the Forms and Documents page and can be used as reference if you are required to complete the process on your own at your institution.

If your institution does not have an Institution Review Board then the Carleton IRB approval is sufficient, as this would fall under Carleton's multi-site involvement of assessment research. You will still need to provide students a consent form or letter so they have an opportunity to opt out (see above) and to share the Carleton Exempt letter as well as the student consent form you will be using with your campus leader.

Decide how you will provide data to InTeGrate (More Details)
Decide on a Student ID, which will be used to label all student material submitted to InTeGrate (e.g. the student ID your institution assigns each student) (More Details)
Decide on a mechanism to measure student engagement, preferably class attendance (More Details)
Set up a SERC account (More Details)
Complete the Course Setup Form (to initiate the processing of your official contract and access to GLE questions) (More Details)

At the beginning of your course (first week of class)

Collect student signed consent forms, or inform your students about the project (signed vs implied consent) (More Details)
Submit initial roster of student ID's on your course status page (More Details)
Administer any pre-measures (this may include the InTeGrate attitudinal survey and/or GLE multiple choice questions)

At the end of your course

Administer any post-measures (this may include the InTeGrate post attitudinal survey and/or GLE content questions (multiple choice and essays)
Provide a final course roster, measure of student engagement, and ungraded student assessment responses, labeled clearly to indicate which assessment they pertain to and with student ID numbers only (no names) (More Details)
Complete the faculty reflection survey and provide any feedback you would like to provide to the material authors (More Details)

Before Your Course Begins

Complete the following initial data collection steps prior to the beginning of the term in which you will pilot/test materials. Thinking through the information below on how you'll provide data to InTeGrate, which student ID numbers to use, etc. will help you fill out the Course Setup Form.

Institutional Review Board & Student Consent Forms

Implementation Program Leads will handle obtaining IRB approval - double check with your leader if your institution requires signed or implied consent. IRB approval is required so that InTeGrate can use the data being collected from each institution.

Read more about IRB & consent forms

IRB Approval

Because the student assessment data is being used for research, we need to consider how human research guidelines apply. Many institutions have Institutional Review Boards that oversee this sort of data collection. Each institution implements this differently so you'll need to explore the situation at your institution. Carleton College has approved the InTeGrate project as EXEMPT (Acrobat (PDF) 106kB May22 12). This approval may be sufficient for you to provide the collected student data to the InTeGrate project. However, each person collecting student data needs to check with their institution as the institution may require a separate IRB application.
Signed or Implied Consent

Although Carleton College approved the application as EXEMPT, you still need to inform your students about this research so that the students have the option to not have their student assessments included. This can be completed in two ways and you will need to select the option that aligns with your institution's interpretation of the EXEMPT status:

Signed consent: In this case copies of the signed consent forms (with student names, ID's you're using for InTeGrate, indication of opt in or out, and signature) would need to be sent and filed at the SERC office (Carleton College prefers signed student consent forms (Microsoft Word 31kB Jan15 13) as the means to ensure that students are informed and have the opt-out option).

Implied consent: In this case, a list of students who have opted out (with their correlating InTeGrate ID's) should be provided to the SERC office (the University of Akron allows for the implied consent form (Microsoft Word 52kB May22 12)).

The completed IRB application for Carleton College can be found on the Forms and Documents page and can be used as reference if you are required to complete the process on your own at your institution.
If your institution does not have an Institution Review Board then the Carleton IRB approval is sufficient, as this would fall under Carleton's multi-site involvement of assessment research. You will still need to provide students a consent form or letter so they have an opportunity to opt out (see above) and to share the Carleton Exempt letter as well as the student consent form you will be using with your campus leader.

Decide on the Mechanisms You'll Use to Provide Your Data to InTeGrate
Read through the general considerations below in detail and think through the mechanics of how you'll provide each piece of data to InTeGrate. The best answer will depend on the particulars of how you will be collecting the data (e.g. will the student be filling out tests on paper or online). Suggestions are given below. If you're not 100% confident of how this will work in your case, we encourage you to talk this through with your webteam liaison before filling out the course setup form (see below for more information on the course setup form).

Show important considerations for submitting data to InTeGrate

Important considerations for submitting data to InTeGrate

Label all data with the corresponding student ID (no student names). We've found this is often easier to do by hand before scanning, but can be added electronically as well (blue painters tape works well to black out names, but can be removed without damaging the paper. This may not work with all scanners). (see below)

Provide data in a spreadsheet where feasible. For data that students have provided online (e.g. assessments done through an LMS), or for simple multiple choice questions (like the initial GLE questions), you can create a simple spreadsheet with "Student ID" in the first column followed by one question per column. The text of the question can be used as the header for each column, and student responses can be listed, one per row, down the spreadsheet. A single spreadsheet can contain multiple questions (e.g. from the GLE and a summative assessment) as long as the questions are clearly labeled. Be sure to provide the actual student answer and not simply the graded result (e.g. we need to know that the student selected "b" in a multiple choice test, not simply that they got it right or wrong).

If a spreadsheet isn't feasible, try to provide the data in an electronic format. For example, it might be impractical to transcribe hand-written student answers from an exam, or students may have handed in an assignment as a Word document. In these cases, you can provide the original electronic document (as long as it's in a widely used format) or scan the original student work. Many modern multi-function copiers, as are frequently found in departmental offices, are capable of scanning and outputting PDFs of documents in exactly the same way you might photocopy those documents. This is likely the simplest way for you to provide us with copies of hand-written student work: scan the stack with your copier and provide us with the resulting PDF file. Standard image formats are acceptable if scanning to PDF is unavailable. JPEG and PNG formats are the safest bet for image scans. Be sure the documents are clearly labeled with the student ID. And in cases where the data needed by the project is mixed with other material (e.g. a GLE question embedded in a larger exam), consider how you might scan only the part that is relevant.

Provide all electronic documents through the upload forms on the course status page (see below), not through email. This will help us track the data accurately while also ensuring our strategies for keeping student data anonymous are enforced. If you have more than one file to send as part of a collection or set (e.g. you have 10 scanned PDFs to send for summative assessment), you'll want to compress those ten files into a ZIP archive and upload only the ZIP archive instead of 10 individual files. Zipping files will be faster for you to send and for us to process. Details on how to create a ZIP file are at the bottom of the file upload section on your course status page. The InTeGrate assessment team will be notified when new files have been uploaded. Once the assessment team has processed the file, the files will be marked "accepted" on your course status page.

We need the ungraded student responses. If you're are going to mark up the student answer as part of your grading, please scan or copy the answer before grading and provide us with the unmarked copy.

If you need to provide the data as paper, do not send the only copy of the documents. Always photocopy the original documents before sending. Documents can be mailed to the SERC office and should include a cover sheet with a clear description of what they are (e.g. what course they relate to, what questions are included). Packages should be sent with a tracking number regardless of shipment method (USPS, FedEx, etc.). Please forward this tracking number to Krista Herbstrith so that we know when to expect the package (kherbstr (at) carleton (dot) edu).

Science Education Resource Center
Attn: Krista Herbstrith
200 Division Street, Suite 210
Northfield, MN 55057

Decide on a Student ID

It's critical that each piece of assessment data you collect be associated with a student ID (no names) that unambiguously and consistently identifies the particular student whose work it is. This will allow us to match work from a given student across all the data collected. While you can select any sort of ID you like, it's important that it be something that both you and the individual students have easy access to. You'll need to provide this ID whenever you share data with us and the student will need to know this ID when they fill out the student surveys at the beginning and end of the course. If your institution issues student ID cards that carry a student ID number and also provide this same ID number to you as the instructor, then that would be a good choice.

Decide on a Per-Student Measure of Student Engagement

In order to identify disengaged students from our data collection results, you will need to submit a proxy measure of students' engagement in the course. Attendance is by far the best measure to identify student engagement, so if it is possible to take attendance without significant disruption to your course, please use that measure. If attendance is not possible, another high-frequency measure, such as percentage of assignments completed is sufficient. Rates can be submitted on a spreadsheet with InTeGrate student ID and the single student engagement measure.

Set up a SERC Account

If you don't already have an account with SERC through InTeGrate or another project, you will need to create one.

Fill out the Course Setup Form

Filling out this form sets up a data collection space for you within our website and gives us the details we need to support you in the data collection process. We encourage you to work through your answers to these questions and consult your webteam liaison if you have any questions before you fill out the form.

×

The link to your course status page can be found on the right of your account page, as outlined in red in this screen shot.
Provenance: Matt Lauer, Carleton College
Reuse: This item is offered under a Creative Commons Attribution-NonCommercial-ShareAlike license http://creativecommons.org/licenses/by-nc-sa/3.0/ You may reuse this item for non-commercial purposes as long as you provide attribution and offer any derivative works under a similar license.

After you fill out the course setup form, you will be assigned a random 8-digit number which is your SERC course ID. You will also receive a link to a course status page, unique to your course, which is accessible from your SERC account page (see image on right).

Your course status page is the place where you can securely upload files containing IRB forms, GLE and summative assessment responses, and provide/update your student ID roster. You can track student completion rate of the online pre- and post-instruction attitudinal surveys as well as determine which students have completed the survey. This page will serve as your data tracking page and you will want to check the page frequently to be sure students are completing the assessments as you intend, keeping in mind that InTeGrate expects an 80% completion rate for assessments.
There is no limit to the number of courses you may have. If your course has multiple sections taught by different instructors or taught in substantively different ways (e.g. one section online, another face-to-face), please treat each section as a different course; that is, fill out the course setup form for each section in order to keep the data separated.

At the Beginning of Your Course (first week of class)

The following list of data collection steps needs to be completed prior to any instruction in your class, typically during the first week of class. Please note that this data needs to be collected before any instruction in the class, not just prior to beginning to teach the InTeGrate materials.

Collect Student Consent Forms (if needed)

Distribute signed consent forms or implied consent forms, depending on what is required at your institution. Signed forms must be returned to the SERC office.

Submit an Initial Roster of Student IDs

Your course status page provides a place to copy and paste in a list of student IDs. You'll do this both at the beginning of the course (to track how the initial student survey is going) and at the end, so that we have a final roster of students who completed the course and from whom we'll expect to see all of the pieces of assessment data. If you keep an electronic student grade book or have access to a course roster that includes the student ID numbers, this should be just a simple copy and paste. List one student ID per line without delimiters such as commas. Copying a column from Excel or Google Docs works perfectly (just be careful the column header isn't included). If you manually compile the list, you may wish to save a copy of the list on your computer for future use. You can submit this list more than once if there's an error or change in enrollment, but be sure to include the full roster each time as it does not merge the changes; rather, it overwrites previous rosters.

Show student ID encryption details

All student IDs stored on SERC's servers are encrypted using a one-way hash to protect the students' anonymity. This is the same way that passwords are stored, and the IDs cannot be unencrypted. To correlate records across many student responses and surveys, it is important the student and faculty always use the exact same ID (no names). The institutionally-assigned student ID that shows up on each student's school ID card and on class lists supplied to you by your registrar is likely the simplest solution because it is accessible to both you and your students.

Show Non-participating Students

Direct Your Students to Take the Initial Attitudinal Student Survey

After filling out the course setup form, you will be provided with a URL, which your students will use to fill out the initial attitudinal student survey to collect their demographic data and attitudinal viewpoints (you can also find this URL on your course status page). This can be done in class or independently outside of class. You'll want to emphasize the importance of completing the form since, without it, we won't be able to use subsequent data from that student. You'll also want to clearly communicate what student ID number they should use. If the ID number the student provides on their survey doesn't match the one you provided on the course status page, then we can't use the data. As students take the survey, it will be recorded in our system and the overall count of students will be visible on your course status page. You won't have access to the individual student responses. If you are giving credit for the survey (which we encourage), you can use the 'completed student surveys' section of your course status page to get a list of student IDs that have completed the survey. You will also see a listing of student IDs that have been entered on the student survey that don't match the roster of student IDs that you provided (e.g. from student(s) who didn't follow the instructions). You can (and should) correct these student IDs using the form on the course status page (make sure you are confident in these corrections, as once 'corrected' we cannot reverse this due to student ID encryption. This will help you achieve the expected 80% matched response rate.

Administer the Initial GLE Questions (multiple choice content questions)

You must have your students answer the 8 multiple choice GLE questions. These will not necessarily correlate to content covered in the modules or your course. They are benchmarks to allow us to compare students within and between institutions. The questions must be administered in class or in another closed environment that ensures students are answering the questions independently and without outside resources. To maintain control of these questions, any GLE questions must be administered in a secure environment. They must be given as a paper & pencil quiz or on a secure LMS system and proctored in either case. Questions cannot be returned to students. You are encouraged to use any additional GLE questions that pertain to your course, remembering to maintain control of the questions. If you are planning on extracting responses from an LMS system, you MUST do a trial download with a webteam member. Student responses to the GLE questions should be sent to SERC as soon as they are available.

At the End of Your Course

The following list of data collection steps needs to be completed at the very end of instruction in your class, typically during week before or during finals week. Please note that this data needs to be collected after all instruction in the class, not just after teaching the InTeGrate materials.

Administer the Post GLE Multiple Choice Questions

At the end of the term, you need to test students on the same 8 multiple choice questions that were given in the beginning of the semester. Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the responses to return to SERC, with student ID's and no names.

Administer Two Essay Questions

Students need to complete two common essay questions as part of a summative high-stakes assessment (e.g. final exam). Questions must be securely proctored and NEVER returned to students to maintain control of the questions. Please copy or scan the ungraded responses to return to SERC before you perform your own grading, again with student ID's and no names. A rubric is provided for common scoring, but you can assign whatever grade you think is appropriate for your course purposes. For instance, an introductory student who answers a Level 3 question earning a score of 1 out of 3 might get 100% credit on the exam. A senior answering the same question might be expected to earn 3/3 to earn full credit.

Direct Students to Fill Out the Final Attitudinal Student Survey

As with the initial student attitudinal survey, students can complete this survey in or outside of class and will need to provide their student ID. The survey is available at the URL provided at the top of your course status page. Please remember, InTeGrate expects an 80% matched response rate for all assessment data. You can also find the URL to the survey on your course status page.

Provide Us With a Final Course Roster, a Measure of Assignment Completeness and All Collected Student Assessment Answers

Use your course status page to upload:

A final roster of students who completed the entire course (using the 'student roster' box in the lower right of the course status form).
A document with your measure of assignment completion for each student, and
Any student assessment data (GLE questions, summative assessments) you have not yet provided. Refer to the earlier instructions on how to provide data to InTeGrate.

Complete Reflection Survey

Complete the faculty reflection survey and provide any feedback you would like to provide to the material authors. This survey will help provide information to colleagues who are considering teaching with the InTeGrate materials that you have tested. Additional audiences interested in your feedback include the InTeGrate leadership and evaluation teams, and developers of the InTeGrate materials.

« Previous Page Next Page »