Part 1: Sensory data collection protocol development

Students will create data collection protocols that facilitate consistent collection of sensory data.

Students in the sciences are often provided with data to analyze but are not involved in the collection of the data or provided with the protocols under which it was collected or they collect data following an already established protocol. Students outside of the natural and physical sciences may have little or no exposure to the process of collecting data. This unit guides students through the process of developing protocols for the collection of data that might help characterize environmental problems.

Prior to beginning the unit, students should:

• be familiar with the difference between qualitative and quantitative data (Unit 1 of this module)
• be familiar with objective and subjective observations (Unit 1 and Unit 2 of this module)
• be familiar with collecting and recording data (Unit 2 of this module)

In previous units of this module, students will have completed the following:

• analyzed provided sets of different types of data, classified each set as qualitative or quantitative, and discussed appropriate use (Unit 1 of this module)
• maintained a log of personal sensory experiences (Unit 2 of this module)
• analyzed data collection logs to identify patterns in both their experiential perceptions and in their methods and interpretations of data collection (Unit 2 of this module)

In Part 2 of this unit, students will use the developed protocols to collect data, and in Unit 5 of this module they will report their data on a map they develop.

In the context of this module, sensory data is used as an indicator of environmental conditions and as a proxy for scientific data that might be collected with analytical instrumentation. If the environment has been degraded, sensory data may be the first indicator and, as such, carefully collected sensory data can guide researchers as they plan and conduct fieldwork that includes the collection of soil, water, and air samples for analyses using analytic equipment. A subsequent module could use the collected sensory data in such a manner.

If the course context allows, instructors may wish to build on this work after the completion of the module by using collected data and maps to assess the environmental impact of land use on a community, and to communicate their findings to a broad audience. If used in these ways, the protocols need to be explicit so that users of the data understand its context and limitations.

Materials

Students will need writing materials and their natural senses (nose and ears). It is possible that some students may have sensory limitations; such limitations should be incorporated into the discussion and the development of protocols.

Groups

This module is to be conducted by groups and includes fieldwork. Each group should have four to six students and may be assigned by the instructor or self-selected by the students. It is important that the students within the group are comfortable with each other and are able to coordinate their schedules in order to complete the fieldwork within a short (one-week) time period.

Introductory alternatives

Within this module:

During the first part of the class session, students discussed a log of immediate sensory experiences recorded prior to class (see Unit 2: Sensory Log & Holistic Reflection).

If this unit is not preceded by the Sensory Log Unit (15-20 min)

Students will be in groups of four to six. Within the groups, each member will have one minute to describe a recent sensory experience (something they smelled, tasted, heard, saw, or felt) as clearly as possible to the group. The purpose of the one-minute limit is to get students to focus on the most important aspects of their sensory experience; an observation should be both clear and concise. Other group members should write down the strengths and weakness of the descriptions provided, what is clear about the related experience, and any questions they have (they should not ask any questions of the speaker). When all members of the group have spoken, the group will have five minutes to discuss the strengths and weaknesses noticed and develop a list of what should be included and excluded to provide a clear description of a sensory experience that is as objective as possible. They should also note any significant similarities and variations between the senses. The students should be prompted to consider why they included or excluded various items and how those impacted them and their group members. Each group then provides a one- to two-minute summary of its key findings to the entire class.

Introduction to Data Collection (5-10 min)

The instructor will begin the protocol development process by reviewing the prior class discussion (Sensory Log Exercise or alternative activity described above) and consensus on components of good sensory data collection and reporting. The instructor may find it useful to review the sample protocols in the resources section of this page.

The instructor should discuss the importance of developing protocols:

• The researcher needs to consider what the data will be used for in order to ensure appropriate data is collected. Usually, data collection is driven by a guiding question and/or hypothesis. If a researcher wanted to document the impact of environmental contaminants on exposed populations, while data from chemical or mechanical analyses is necessary, such data is not adequate to characterize the lived experience. The protocols help to ensure a consistent data set that will meet the end goals of the students' projects.
• The researcher needs to have a plan of action that is based on known environmental conditions that will impact the data collected. Sounds and odors move through an open system that has a variety of inputs that can impact the data collected. The protocols should take into consideration the complex environmental factors by ensuring a record is made of environmental conditions that would impact the data and by incorporating techniques that will minimize those impacts when feasible. To be effective, the team developing the protocols needs to assess the system in which it will be working, identify the inputs (environmental conditions such as air temperature and wind speed and direction) that are likely to impact the sensory output collected as data, and ensure that those inputs are recorded.
• Environmental science professionals, such as geoscientists, develop and use protocols to ensure that the data collected allows an accurate and reproducible characterization of a field area.

The instructor should also introduce three key data collection concepts:

• Accuracy: How close your measurements are to the true measurement.
• Precision: The extent to which repeated measurements under the same condition match up.

Many instructors use the concept of an arrow and a bull's-eye to demonstrate accuracy and precision. For example, you will be accurate if you shoot your arrows near the bull's-eye. You will be precise if your arrows group together. Thus you can be precise but not accurate if your arrows group together in the outer ring of the bull's-eye.

• Reliability: The extent to which repeated measures by multiple individuals under the same condition yield the same result. This is similar to precision.

The instructor should encourage students to keep these concepts in mind as they develop a data collection protocol. The instructor should ensure that some groups develop protocols for measuring sounds, while others create protocols for collecting scents. This protocol will inform students' final mapping projects and associated field plans, but the students should expect their protocol to evolve as they develop a guiding question and site to investigate.

Draft protocol development and review (20–25 min)

Each group will develop a protocol that describes what to collect, how to collect it, the specific details to be recorded, and a measurement scale. While the sensory data would seem to be the primary focus of the group's work, the group will need to develop a working model of the system in which the odors or sounds exist, determine which inputs are likely to impact the movement and detection of the odors or sounds, and develop protocols to collect data on those inputs. The overarching goal is to be able to collect data that is reproducible, clear, and concise, and would have meaning to a wide audience. After ten minutes, each group is paired with another group and shares and discusses their draft data collection protocols for five to ten minutes. Each group then takes five to ten minutes to revise its protocol as necessary. Throughout the development process, the instructor should be listening in on the group discussions, examining the draft protocols, asking questions and making suggestions, and encouraging the groups in their efforts.

Protocol exchange and field test (5–10 min)

When the protocols are ready, each group will trade its protocol with another group and instruct the new partner group in its use. The members of each group will use the protocol they have been provided with to collect sensory data prior to the next class session as a "beta test." (Alternative: student groups leave the classroom and have 15–20 minutes to collect data using the protocol.) The instructor will explain that exchanging protocols will reduce confirmation bias (you get the results you expected) and will allow students to identify places where the protocols are not explicit enough. This is a good time for the instructor to remind students about the concepts of accuracy, precision, and reliability.

If class time is available, students may be released from class and allowed to perform their field test of the protocol immediately; otherwise, students will need to perform the field test as homework prior to the next class session. In the field, students (in pairs or their groups) will collect small data sets (from three to six collection points) and prepare a critique of the protocol that includes both strengths, weaknesses, and questions. The instructor will direct the students to prepare the critique based on the following questions:

1. What would you want to have changed if you were to use this again?
2. What worked very well and should not be changed?
3. What worked well but was difficult to understand how to do?
4. What would you want to know if you were the developer?
5. To what extent does the protocol address issues of precision, accuracy, and reliability?
6. To what extent does the protocol lead to data collection that informs the research question?

Protocol review, revision and group reflection (25–30 min)

At the beginning of the next class session, each group meets for five minutes to develop a consensus critique of the protocol it implemented. The partner groups will then meet and discuss the strengths and weaknesses of protocols and possible improvements (five minutes each). Each group will then meet for five to ten minutes to adjust their protocols based on the critiques. The instructor will circulate between the groups and provide question prompts to the groups if necessary. Each group then reports out the key features of its final protocol to the class. The instructor will summarize the key features on the board/screen at the front of the classroom and will facilitate any discussion or clarification from the class. The instructor should revisit precision, accuracy, and reliability and stress that although these are laudable goals, it is difficult to achieve them when collecting these types of data. If necessary, the instructor should revisit the concept of effective research or guiding questions. The instructor should also prompt students to think about the challenges of discerning between different smells or sounds and thinking about the relationship between various data points. Finally, the instructor should lead a brief discussion about the challenge of characterizing a rapidly changing, open system. Because this is an open system, smells and sounds do not stay within the boundaries of a study site. In addition, it may be difficult to reproduce the data collected on a given day and time because variations in the inputs to the system will result in changes to the sensory outputs. What causal factors might impact their perception of smells and sounds, i.e. what factors might perturb the system? (e.g. wind, temperature, humidity). How does the students' own perception impact data collection? For example, you may hear some nice music coming from a building and linger at that spot, or you might walk away from a loud car alarm.

Assessment of environmental degradation typically involves recording visual observations of a site and the related contamination as well as the collection of samples (soil, water, and air) for immediate and off-site physical and chemical analysis. The collection of all such data is subject to strict collection protocols to ensure uniformity of results and to minimize the likelihood of incorrect results. Sounds and odors may be recorded but rarely in a systematic way that would mimic how the physical samples collected. However, sounds, odors, and visual impacts are often the initial identifiers of the negative impacts of environmental contamination, and they have an immediate impact on the lived experiences of those who are subjected to the contamination. During protocol development, the instructor should emphasize:

• the importance of documenting all impacts of environmental degradation, not only those that are easily quantified through laboratory analyses;
• the consideration of the sensory impact of environmental contamination on the lived experience and its relation to the chemistry and the wind and water movement of the contaminants;
• the importance of the physical environmental conditions in the movement of contaminants, and the interaction between those conditions and level of contamination detected; and
• the importance of adhering to protocols in order to obtain a valid set of data that can be used to characterize an impacted location, direct future studies, and help communicate the impact of contamination to a broad audience.

It is important to get the students to treat their senses as detection instruments. As such, the instruments are highly variable in their ability to detect. They are also portable, inexpensive, and familiar to the user. The trick is to develop a protocol that allows the results of different operators to be compared in a meaningful way.

The senses are subject to sensory extinction, which can occur when a smell or sound is more difficult to detect after an extended exposure.

Protocols should include calibration (what rates as a strong versus weak scent or a loud versus quiet sound); collection techniques (i.e. one long sniff versus several short ones or facing a sound versus turning an ear toward it). Specific ideas should not be shared with students; the instructor should ask probing questions if they do not come up.

• How will you determine the relative strength (or weakness) of a sound or scent?
• Is there a difference between "pleasant" and "unpleasant"? Is there a standard you can apply?
• What, exactly, should the person collecting sounds (or scents) be doing? For how long?

The draft protocols provide an opportunity to refine data collection procedures by a process of testing and revision. Indeed, this iterative process is a key characteristic of rigorous scientific methodology and scientific thinking.

The protocols focus on scents and sounds because they travel more freely through the environment and can be used in subsequent activities to monitor the movement of "contaminants" though the environment. An environmental scientist would use (and possibly develop) similar protocols to guide the collection of soil, water, and air samples.

The protocol development process must include an analysis of system inputs that are likely to effect the sensory outputs. Those inputs (the environmental conditions such as temperature, humidity, wind, topography) will need to be recorded to allow for proper analysis of the data. The instructor can ask:

• What external factors might influence the data you collect?
• How should those factors be documented?

People subject to environmental contamination often sense unusual odors in the air or water. In 2011, residents of Wedron, Illinois, began reporting the odor of gasoline in their tap water; in October 2013, residents of Irwindale, California, began complaining about the smell emanating from a hot sauce factory; and, in January 2014, residents of Charleston, West Virginia, noticed the odor of licorice in the air and in their tap water. In each of these cases, the odor was the first indication of contamination moving through the environment, but in none of these cases was there an effort to systematically quantify the odor. In Wedron, residents brought water samples to the local health department to "share" the smell. In Irwindale and Charleston, where the smells were more widely experienced, local officials followed the odors back to their sources. In each case, data collection protocols would have helped to characterize the impact accurately, indicate the need for action, and possibly to identify the source more quickly.

Instructors from disciplines outside of field-based science might find the sample protocols in the References section below useful in guiding student through this process.

Formative assessments should take place throughout the unit. The unit is designed to have students assess each other's work and make suggestions for improvement. Key assessment points are: 1) when group members discuss strengths and weaknesses of initial sense descriptions; 2) when groups exchange discuss draft data collection protocols; 3) when groups report back after implementing protocols. The attached rubric may be used by the students at 2) and 3). (Assessment Rubric for Protocols (Microsoft Word 2007 (.docx) 53kB Feb12 16))

While groups are working and sharing, the instructor should monitor the groups to ensure that student discussions are substantive. The development of protocols is intended to be student-driven; students should be prompted to consider how well the developing protocols will work, especially the ease with which others would be able to implement them and the reproducibility of resulting data.

If the unit is used as part of a broader module that includes data collection, protocols should be included as part of the final project submission in support of the collected data.

If the unit is presented as a stand-alone project, the final protocols should be submitted along with a justification.

Examples of Data Collection Protocols

• Washington NatureMapping Program, Data Collection Protocol
• Virginia Department of Conservation and Recreation, Summary of Procedures for Collection and Analysis of Vegetation Data
• San Francisco Estuary Wetlands Regional Monitoring Program, Data Collection Protocol
• Idaho State University, GIS Training and Research Center, Field Sampling Protocols
• Golondrinas de las Americas, The Golodrinas Handbook (pdf)

Protocols and Pilot Studies: Taking Data Collection Projects Seriously ([link  https://www.tandfonline.com/doi/epdf/10.1080/10691898.1998.11910607?needAccess=true 'full pdf']), Journal of Statistics Education v.6, n.1 (1998) - published by the American Statistical Association

Potential Student Resource for Understanding Uncertainty, Error, and Confidence

Pedagogical Resources from Pedagogy in Action

• Cooperative Learning module
• Experience-Based Environmental Projects module
• Peer Review module
• Process of Science module