Initial Publication Date: February 18, 2014

Module Development Timeline

Jump down to: Contract Initiated | Phase 1: Materials In Development | IRB information | Phase 2: Classroom Pilot | Phase 3: Post-Pilot Materials Revision

Development Timeline

Overview of the Materials Development Timeline

Each module development team will follow a series of steps and checkpoints designed to 1) align the materials development process with the overall GETSI project goals and timeline and 2) allow for timely interaction among the materials development teams, assessment team, UNAVCO, SERC staff, and technical advisors. After your proposal has been accepted and your contract is initiated, there are three major phases of developing, testing, and revising materials. After these phases are complete your, materials will be published on the GETSI website and will be publicly available for use.

Phase 1: Materials in Development (as a team) - for GETSI Phase 2 this is expected to be ~8-12 months from February 2018
Each team's authors work together to create detailed descriptions of materials for faculty use and student versions of the materials when necessary. The GETSI Materials Development Rubric (Microsoft Word 2007 (.docx) 134kB Dec7 17) outlines the expectations for how the materials will be created. It is based on the InTeGrate Materials Development and Refinement Rubric with minor project-specific differences. The GETSI Project Manager, Beth Pratt-Sitaula, and the team's assessment consultant provide guidance to the team to help them understand and meet the elements of the rubric. At the end of Phase 1 the materials will need to pass through the rubric, demonstrating that they have all of the desired attributes. In Phase 2, the materials will be tested in each author's classroom and student data will be collected. In the past, authors had to obtain Institutional Review Board (IRB) approval from their own institution to collect this data. However, beginning January 2018, new IRB guidelines allow multi-site data collection to be covered by Carleton's IRB exempt status (click below to read more about the guidelines). For those locations with an IRB, you should inform your IRB that you are participating in this study and that Carleton is acting as the IRB of record.

Read more about new 2018 IRB guidelines

IRB guidelines
Because the student assessment data is being used for research, we need to consider how human research guidelines apply. Most institutions have Institutional Review Boards (IRB) that oversee this sort of data collection. Carleton College has approved the GETSI student data and faculty survey data collection project as EXEMPT Carleton-UNAVCO IRB exempt letter (Microsoft Word PRIVATE FILE 27kB Feb6 18), and student consent is not required to collect data for the project. As per the new 2018 guidelines on Federal Policy for the Protection of Human Subjects, Carleton is also acting as the IRB of record for all institutions collecting this data Carleton IRB of Record (Microsoft Word 27kB Feb6 18), meaning all institutions are covered by its EXEMPT status (if you have worked with SERC, InTeGrate or GETSI before, you will recognize this as a big change). It will not be necessary for each author to complete a Human Subjects application to his/her own institutional IRB. Instead, please inform your IRB that you are participating in this project, that it has been deemed EXEMPT by Carleton, and that Carleton is the IRB of record under the new guidelines. You can provide the Carleton-UNAVCO IRB exempt letter (Microsoft Word PRIVATE FILE 27kB Feb6 18) if they wish for proof.

As this process is new, please contact Ellen Iverson if your IRB has any questions: eiverson@carleton.edu.

While we are not required to inform students of the research due to the exempt status, we recommend including the following statement in your syllabus:This course is participating in an NSF-sponsored (National Science Foundation) GETSI project (GEodesy Tools for Societal Issues) to improve geoscience education. Anonymized responses to some course assessments will be used as part of GETSI research.
More documentation can be found on the Forms and Documents (opens in a new window) page.

Phase 2: Classroom Pilot (individually) - for GETSI Phase 2 this should be finished by April 2019 (or sooner!)
During the classroom pilot phase, module authors take their newly developed curriculum and try it out in their own classroom. Authors collect data that supports the evaluation of the team's materials and of the projects overall goals. Learn more about the design of the assessment and evaluation. Piloting allows authors to understand and document the learning that is taking place in the classroom. This helps ensure that the desired level of learning is being achieved and demonstrates pieces of the curriculum that may need revisions. Since each team member individually pilots the materials, there will be multiple sets of data to inform the material revisions. The classroom pilots also contribute to the evaluation of the overall GETSI project effort by gathering data that is common to all GETSI classroom pilots. Read more about the assessment data being collected and how it relates to your module and the overarching project on the Assessment Data page.

Phase 3: Post-Pilot Materials Revision (as a team) - for GETSI Phase 2 this should be completed by August 1, 2019 (or sooner!)
The materials development team uses data collected during their classroom pilot along with assessment team feedback from the rubric review to guide the team toward meaningful revisions. Authors create individual case studies based on experiences and reflections from implementing the materials, and finalize the information that supports other faculty in using the materials. By the end of the process, the teaching materials and case studies will become publicly-available web pages that other faculty may access and implement in their own classes.

The show/hide links in each section outline the complete steps involved in passing the different phases of the materials development process and are separated by project roles. We encourage you to familiarize yourself with your own responsibilities as well as those of others.

Materials Team Initiates Work and Contract Initiated

As part of the application process, teams confirmed the topic of the module, provided a short description of the module that defines its scope and its central pedagogic features, described how the materials the team develops address the guiding principles of GETSI/InTeGrate teaching materials, provided contact information for team members, and indicated the details about the course in which each team member will test the materials. This information will be used to establish the contract for each team member to ensure payment of the stipend. Each team member should understand the terms and conditions of the contract prior to the completion of this step.

Much, if not all, of this information will be completed at the face-to-face new author development meeting.

Show Materials Team Responsibilities

Materials Team Responsibilities:

Input the information critical to initiating your contract. It should appear on your team reporting and checkpoint 1 pages, including:

Summary of module (checkpoint 1 work page)

How team's module addresses the guiding principles of GETSI/InTeGrate teaching materials (checkpoint 1 work page)

Timeline for when the team estimates completing checkpoints 1-4 (reporting)

Contact information for team members (reporting)

Information about course in which each team member will be teaching GETSI material (reporting)

Familiarize yourself with the Content Management System (CMS). You may be tempted to work in Word or Google Docs (or other software with which you are familiar), but this will hamper your progress and your ability to communicate with your assessment team consultant.

Learn about Working Together as an InTeGrate Team.

Show GETSI Project Manager Responsibilities

GETSI Project Manager Responsibilities (Beth Pratt-Sitaula): Welcome and orient everyone to the details of materials authoring. Review that all required information is available on the team reporting pages and in line with the project expectations:

Summary of module (top of checkpoint 1 page)

How the team's module addresses the guiding principles (top of reporting page)

Timeline for when the team estimates completing checkpoints 1-4 (within each checkpoint area on reporting page)

Contact information for team members (bottom reporting page)

Course information in which each team member will be teaching GETSI material (bottom of reporting page)

If there are issues that need to be addressed prior to contract initiation, communicate with the team to assist them in making these changes. If the team's information is complete and passes your initial review, send an email via the team's email list that (a) signals the UNAVCO office that contracts can be sent to team members, and (b) signals the assessment person to familiarize themselves with the materials summary, timeline, and how the module addresses the guiding principles.

Show Assessment Consultant Responsibilities

Assessment Consultant Responsibilities: Once assigned to a materials development team, familiarize yourself with the materials summary, how the materials address the guiding principles, and timeline. If you have not already made initial contact with materials team, use the team's email list to do so now. Alert the GETSI leadership if you have any major concerns with the content thus far.

Show Webteam/SERC Responsibilities

Phase 1: Materials In Development

As the team develops materials, they will reach various checkpoints where feedback will be given from the overall GETSI Project Manager (Beth), the team's assessment consultant (Ellen or Stuart), and the GETSI webteam liaison (Monica). The purpose of the checkpoints is to ensure that the team has adequate input and support to develop successful materials as described by the GETSI Materials Development Rubric (Microsoft Word 2007 (.docx) 134kB Dec7 17) and is moving toward successfully passing rubric based review at Checkpoint 4. By checking the work against the rubric several times throughout development, there is a much smaller chance of having to make major revisions once the module is thought to be complete.

Checkpoint 1: Goals, Outline, Summative Assessment, and a Plan for Development

The purpose of this checkpoint is to review the initial design and scope of the materials, as well as the timeline and communication plan. This checkpoint insures that the team has a common vision and plan for creating the materials and that all support people understand and can meet the timeline.

The materials development team will have succinctly summarized the module content and how the module will address the guiding principles as part of the contract set up process. The next step in writing the module is for each team to develop module level goals, an overall outline of the materials, and a draft of their summative assessment (note: the summary and guiding principles were covered prior to this step, but may need to be modified as materials are further developed). This information should be recorded on the Checkpoint 1 Workpage in the team's module pages. This work will help the materials development team, GETSI Project Manager, assessment consultant, and webteam liaison all have a better understanding of the scope of the work being done and its alignment with the GETSI rubric.

At this point, the materials development team should also establish a timeline, ways to communicate with one another, and strategies for managing their collective progress. When the team's material is ready for review, the team should inform Beth and the assessment consultant via the team's email list (listed on the Materials in Progress page).

Timing for this checkpoint:

This checkpoint should be completed as much as possible during the in-person development workshop.

It will take your assessment consultant and Beth approximately 1-2 weeks to review your work.

Show Materials Team Responsibilities

Materials Team Responsibilities: By the end of Checkpoint 1, the following items should be recorded in your module webspace:

a list of module level learning goals (checkpoint 1 workpage)

a draft of summative assessment(s) for the module (checkpoint 1 workpage)

a module outline (checkpoint 1 workpage)

any edits to short description/summary and guiding principles (checkpoint 1 workpage)

a team communication plan (bottom of team reporting page)

a team geodesy data plan (bottom of team reporting page)

When your team feels you have reached Checkpoint 1 you will need to:

Contact Beth via the team's email list. She will review the new content on your webspace. At this point, we are checking that the scope and organization of the materials match critical aspects of the GETSI Materials Development Rubric (Microsoft Word 2007 (.docx) 134kB Dec7 17) prior to further development and testing. Beth will either make suggestions for revisions or pass your module to your assessment consultant for review.

Contact the GETSI webteam liaison and have a conversation about the structure of your module. This will allow her to create the necessary 'empty spaces' to hold your planned activity sheets (for instructions to the faculty) and student materials.

Inform your Institutional Review Board (IRB) approval process of your participation in the project.

Show GETSI Project Manager Responsibilities

GETSI Project Manager Responsibilities: Review the scope of the material on the checkpoint 1 Workpage for the module goals, outline, and summative assessment, as well as the team communication plan (on team reporting page) and approve that the material has met the requirements of Checkpoint 1. Once approved, inform the assessment consultant (via the team email list) that the description, goals, and timeline are ready for their review.

Show Assessment Consultant Responsibilities

Show Webteam Responsibilities

Checkpoint 2: Breadth, Depth, and Assessment

The purpose of this checkpoint is to provide comprehensive feedback on the structure of the materials, the pedagogic design of the materials and assessments, and the alignment of assessments, materials and goals. Checkpoint 2 is designed to occur at the earliest point in the materials development process, when it is possible to provide detailed and comprehensive feedback. The goal is to spot any developing problems early enough to prevent unnecessary work.

In order to be able to evaluate the team's progress towards meeting the design rubric, the GETSI Project Manager and assessment consultant will be looking for both breadth and depth in the module by checkpoint 2, including the assessments. Specifically, the team's web pages should include:

a complete outline that shows the breadth of the module (overview page), including activity descriptions, pedagogic strategies, planned assessment strategies, and descriptions of any materials that will be developed for students (detailed in unit pages).

at least one complete unit that includes faculty materials, student materials, and formative and summative assessments with rubrics to display the depth of your materials (work with the webteam liaison if you have questions, and refer to the Teaching Materials Format page)

complete example(s) of a summative assessment(s) that will address the module-level goals (linked from the assessment page)

Once these pieces are complete, the team should communicate with the GETSI Project Manager (Beth) via the team's email list to initiate the review of the materials. She will review and provide feedback for revisions and pass them to the assessment consultant when the material is ready for review. The assessment consultant will score the materials using the GETSI Materials Development Rubric (Microsoft Word 2007 (.docx) 134kB Dec7 17) and provide constructive feedback.

Timing for this checkpoint:

This checkpoint should occur about half-way through the materials development process, an absolute minimum of 3-4 weeks prior to checkpoint 3, 4-6 weeks prior to materials review, and 11-15 weeks prior to the earliest pilot test.

It will take your assessment team member and team leader approximately 1-2 weeks to score your work.

Show Materials Team Responsibilities

Materials Team Responsibilities: Once your module has achieved both breadth and depth (and everything is recorded on your webspace as listed above), communicate with the GETSI Project Manager that your materials are ready for Checkpoint 2 review via your team email list.
If you have not already done so, when you have reached checkpoint 2, inform your Institutional Review Board (IRB) of your participation.

Show GETSI Project Manager Responsibilities

Show Assessment Consultant Responsibilities

Show Webteam Responsibilities

Checkpoint 3: Materials Nearly Complete (75-90%)

The purpose of this checkpoint is to provide feedback on any final elements that are likely to prevent the materials from passing the rubric and finalize the timeline for the review process.

The materials development team continues working to create materials. At this checkpoint, some portions of the materials may be complete and others may be mostly complete, however, no individual pieces should be missing entirely. Your webspace should include full descriptions of activities, pedagogic approaches, and all assessments with needed rubrics.

Once materials are nearly complete, the team communicates with the team leader via the team's email list to initiate another review of the materials and to set the date when Checkpoint 4 (Materials Review) will occur.

Timing for this checkpoint:

This checkpoint should be completed 1 to 2 weeks prior to the materials review (Checkpoint 4) deadline, and a absolute minimum of 9-11 weeks prior to the earliest classroom pilot date.

It will take your assessment team member and team leader approximately 1-2 weeks to review your work.

Show Materials Team Responsibilities

Materials Team Responsibilities:

Alert the team lead via the team's email list that the module is ready for Checkpoint 3 review.

Confirm the final review date when materials will be measured against the rubric.

Provide a fully developed summative assessment (opens in a new window) on the assessment page in your module (if you have more than one, list them all). Provide details on how and when the summative assessment will be administered during your classroom pilot (e.g. after unit 3 as a graded exam question). (note: ignore InTeGrate IRB info on this link)

If your materials are being reviewed close to the time of your classroom pilot (e.g. within 9 weeks), remember to address the data collection steps required to Collect Data in your Classroom (opens in a new window). Please use the Course Set up Form.

Show GETSI Project Manager Responsibilities

GETSI Project Manager Responsibilities: Review and provide feedback on all the available content (in addition to any comments and discussions that have already occurred) as well as module length (in terms of realistic teaching time). Review the module and either make suggestions for revisions, or pass the module to the team's assessment consultant for review (via an email to the team's email list).

Show Assessment Consultant Responsibilities

Show Webteam Responsibilities

Webteam Responsibilities:

Continue to provide support for the CMS as needed.

Verify that all summative assessments are clearly labeled/linked on the assessment page of the module.

Prior to reaching Checkpoint 4, the webteam liaison also reads through the module and provides formatting and technical assistance as needed (checking links, adding pages to the navigation menu, editing page titles, removing stock text, ensuring that uploaded documents open, etc.)

Checkpoint 4: Materials Review and Revisions

This checkpoint is actually an interval of time over which the materials to be reviewed are finalized on the website, the review is conducted, and revisions necessary to pass the rubric are completed.

The materials development team enters the checkpoint when the materials for review are fully available in the module pages. Student worksheets are uploaded as needed, activity descriptions are completed, and the overview of the materials on the module overview page are sufficient for another instructor to understand how the parts of the module build upon one another to create a cohesive module. Once complete, the materials are reviewed by the GETSI Project Manager. The team may be asked to make revisions or additions. When the Project Manager has determined that the materials are ready for review, she initiates the review by contacting the assessment consultant. The materials are reviewed using the GETSI Materials Development Rubric (Microsoft Word 2007 (.docx) 134kB Dec7 17) by three members of the assessment team: the assessment consultant and two other InTeGrate assessment team members

Timing for this checkpoint:

This checkpoint should be completed a minimum of 7 to 9 weeks prior to testing- which allows time for reviews and revisions. This is only enough time if there are no major issues with the module.

It will take the team leader and assessment team at least 2 weeks to review your work (from the scheduled date).

Show Materials Team Responsibilities

Materials Team Responsibilities:

Ensure your team has responded to any assessment consultant feedback that was made during checkpoints 1-3; if you choose not to change materials based on feedback, make sure you clarify the reasoning for this decision and communicate this to the GETSI Project Manager.

Be sure that formatting is similar among all the module pages and will support easy use and review of all of your materials. Ask the webteam liaison for assistance as needed.

Email the GETSI Project Manager via the team's email list to let them know your module is ready for review.

After review is complete, revise materials as needed. You may need to create a plan with your assessment consultant and GETSI Project Manager to complete revisions.

Complete the Materials Development Reflection Survey (opens in a new window) after materials have passed the review and before teaching.

Prior to the beginning of the term during which you plan to pilot your materials, follow the initial data collection steps (opens in a new window). Note that you may need to wait to complete this step until closer to the start of the term in which you will individually pilot your materials.

Show GETSI Project Manager Responsibilities

Show Assessment Team Responsibilities

Materials Team Receives first payment upon completing Checkpoint 4

Phase 2: Classroom Pilot

During Phase 2 you will teach with your materials and collect data that your team will use to improve the materials. These data are also used to evaluate the project as a whole. Learn more about how these data will help you and the project.

Pilot testing steps occur for each individual author (not as a team). Please note that there are data collection steps that need to be addressed prior to the start of the semester in which you plan to pilot the module in your class (see the "Data Check A: Before your course begins" section below in the Outline of Piloting Data Checks). Depending on timing, this may occur after Checkpoint 3 or 4. Your webteam liaison can help you determine when you should complete these steps. Read more about Collecting Data in your Classroom.

Show Outline of Piloting Data Checks A - D

Data Check A: Before your course begins. Read more about Data Check A

Inform your Institutional Review Board (for those authors' whose schools have IRB's) of participation in the GETSI project
Decide how you will provide data to GETSI/SERC
Decide on a student ID method
Complete the GETSI Course Setup Form.

Data Check B: At the beginning of your course (first week of class) Read more about Data Check B

Tell GETSI Facilitator (Beth) the student IDs of any students who withdraw consent. She will inform SERC.
Submit initial roster of student ID's (using the course status page)
Direct students to initial attitudinal survey
Administer the initial GLE multiple choice questions

Data Check C: While you are teaching GETSI curriculum Read more about Data Check C

Administer and collect summative assessments
Turn summative assessments in to SERC as soon as possible thereafter

Option 1: via the course status page
Option 2: mail to Attn: Ellen Iverson (GETSI Project), Science Education Resource Center, Carleton College, One North College St, Northfield, MN 55057. Note tracking numbers and be sure to have additional paper copy in case there is a mail loss situation

Data Check D: At the end of your course Read more about Data Check D

Administer the final GLE multiple choice questions and send to SERC
Direct students to complete the final attitudinal survey
Provide a final course roster and/or check for unknown students on the tester's course status page
Complete the post-pilot reflection survey on your individual reporting forms
All student assessment data is submitted and recorded in the tester's course status page

You will receive the results of the assessment as soon after testing by both authors as feasible. The Materials Team, Assessment Consultant, and GETSI Project Manager will hold a virtual meeting to discuss results. These results will include

a summary of the GLE pre/post questions, two common essay questions (if used) and attitudinal data for your module situated in the results to date for the project as a whole.
an evaluation by assessment team of how well the summative assessment functioned for determining whether students met the module goals.

Phase 3: Post-Pilot Materials Revision and Review

During this phase of the materials development process, the team will revise the materials based on the information gathered during the pilot testing including the experiences of the team in teaching with the materials and the assessment results. Authors will complete the materials that support other faculty in using the materials, including the instructor testimonials which describe the adaptation of the materials for each author's class. Please be sure you have signed the faculty consent form for these public self-authored stories. At the end of phase 3 the materials will be published on the project website.

Checkpoint 5: Revision Plan

This work begins with the development of a revision plan by the team. This revision plan is based on all three classroom pilot tests, and is informed by the data collected, reflections, case studies, and an intimate knowledge of what worked/didn't work with the module. Comparing and contrasting challenges and successes with team members will help development teams create a strong coordinated revision plan. Each team will need to define changes as either:

Global: changes to be applied to the module as a whole.

Individual: changes that may help individual teaching scenarios.

Authors will work together to develop the plan, including a timeline and assignment of specific responsibilities.

Show Materials Team Responsibilities

Show GETSI Project Manager Responsibilities

Show Assessment Consultant Responsibilities

Checkpoint 6: Individual Instructor Stories and Coordinated Revisions Completed

The goal of this checkpoint is to revise the materials and prepare them for final review. As is the case with journal reviews, the authors will need to prepare a letter indicating how they addressed the concerns raised by the assessment team.

Authors revise materials in keeping with their plan. In addition, they attend to the final edits and formatting described below in the Materials Checklist. They obtain iterative feedback from their team lead as they work. The team leader oversees editing to ensure materials from different teams share a common voice and are fully completed.

Materials Checklist

All necessary student materials are available on-line and consistent with the GETSI/InTeGrate student materials format.

All necessary instructor materials are available on-line and consistent with the GETSI/InTeGrate teaching materials format.

Writing style and tone are appropriate for the material. Teaching materials should follow best practices for web writing and present an accessible and compelling case for their use. Materials directed at students should have a voice and presentation appropriate for use in a broad range of classroom.

Spelling and grammar are correct.

Navigation within the web pages follows GETSI/InTeGrate conventions and is complete and consistent.

All links are functional.

Appropriate attribution is given and sources cited throughout. Copyright information

All teacher-only materials are identified and communicated to the team webteam member so they can be placed in a protected space (only accessible to instructors).

Provide team's webteam member with three examples of the kind of image the team would like to use on their front page.

All downloadable materials:

are provided formats appropriate for their intended use.

appropriately cite their sources.

Figures and images have captions including attribution and references as appropriate.

All included materials (including photos), both online and downloadable, have accurate licensing information and can be legally redistributed by GETSI.

Individual instructor stories are finished and faculty consent signed (read more about IRB and consent). Submit your instructor stories with this form. If you want to see the components ahead of time, here is a Word version of the Instructor Story File (Microsoft Word 2007 (.docx) 113kB Nov1 16).

When the authors feel the materials are finished, they provide a letter to their team leader summarizing the changes they have made in response to the initial assessment team feedback. The letter should detail rationale for both changes made *and* areas where they didn't make the requested changes, similar to a letter in response to peer review of a scientific paper. Specifically the letter will address:

the editing suggestions from the initial Materials Development Rubric
assessment team feedback based on in-class testing data
issues and challenges that were called out in the piloting instructors' post-instruction reflection

The author team notifies Beth when this letter is ready. Beth initiates the 3 parallel elements of the final review: science review, technical review, and copy editing. These results in feedback to the team which they address to reach checkpoint 7.

Show Show detailed process between checkpoint 6 and 7

Beth notifies the webteam member that checkpoint 6 has been reached. The webteam member moves the module into final format including:

front page layout with image selected in coordination with SERC
instructor stories
check to make sure formatting, including navigation is correct
student materials cloned
science reviewer group given read access to instructor and student materials
materials tagged with controlled vocabularies

The webteam member also generates a technical review document which contains a list of items that need attention from the authors: broken links, major formatting problems, prov/reuse info that is incomplete or seems incorrect. They send this document to the team email list with a 2 week deadline and document the date that the technical review was completed and when the author response is expected on the reporting page. They notify Beth that step 1 has been completed.
Beth arranges copyediting and documents an expected completion date on the reporting page.
Beth arranges to have a science review conducted (if neither of the authors are content experts in the topic). Beth identifies reviewers and lets web member know their email address/es so they can be added to the security group. Once the module has been sent out for review Beth documents an expected review completion date on the reporting page. When science reviews are complete Beth sends a summary of suggestions to the author email list with a 2 week deadline and documents this on the reporting page.
When the 'authors should respond by' deadlines for steps 2 and 5 are known Beth will document an expected checkpoint 7 deadline on the reporting page.

Note that the webteam work in step 1 may largely (or even entirely) be completed before checkpoint 6 is reached. Steps 2-5 will occur in parallel.

Checkpoint 7: Final Assessment and Content Review and Final Revisions

The GETSI Project Manager (Beth) will manage the final three reviews (science review, technical review, copyediting review) and the Assessment Lead (Ellen) will manage the final assessment. They will work with the module team to ensure appropriate changes are made.