How to Write ConcepTests

Every instructor has created multiple choice questions but some of these may not be suitable for ConcepTests. A ConcepTest should not require students to simply pull up some fact from their long-term memory. Questions that ask students to recognize a fact (e.g., Which is an igneous rock? a. Basalt, b. Sandstone, c. Marble) are knowledge questions and do little to assess students true understanding of key concepts in a course.

In contrast, questions that seek to probe a student's comprehension or application (Bloom et al., 1956) of a concept often make ideal ConcepTests. Comprehension might involve making a prediction of what would happen in an experiment or how changing one variable would affect others. Application often involves applying rules or principles to new situations or using known procedures to solve problems. Questions that ask students to solve a problem using a known equation or to select a procedure to complete a new task would be considered application questions. It is useful to keep in mind that features that a geologist may think are obvious may not be at all clear to a beginning student. Consequently pictures of features (e.g., faults, rocks) that students have not seen before may make excellent questions for the interpretation or classification categories described below. For further discussion of how to design effective questions see Beatty et al., (2006). Also, check out the ConcepTest examples page for more examples.

Here are several categories of questions that make strong ConcepTests:

Interpreting

Questions that involve changing the form of presentation such as paraphrasing material from the text or creating an image to illustrate a concept in the text require students to interpret. Students should be able to convert text to an illustration or vice versa. Much geological information is presented in the form of diagrams, graphs, maps, or cross sections. These visual learning prompts hold little mystery for instructors but are often unfamiliar to students. They also encourage students to make simple observations, a key element in the scientific process. For example, students may be asked a question about the rock cycle that requires them to interpret data from a temperature vs. time graph. (See also the "How many plates?" question from the What are ConcepTests page.)

The graph on the right illustrates how temperature changed with time for part of the rock cycle. Which of the following is best represented by the graph? (N=180)

1. Sediment is lithified to form sedimentary rock (3% of students chose this answer)
2. Sedimentary rocks are metamorphosed to form metamorphic rocks (35% of students chose this answer)
3. Sedimentary rock is converted to magma* (44% of students chose this answer)
4. Metamorphic rocks are uplifted to Earth's surface (4% of students chose this answer)
5. Magma cools to form plutonic igneous rock (14% of students chose this answer)

Classifying

Classification requires that students recall features used to classify an object/phenomena, identify key features by observation, and finally place those features in the context of the general classification scheme. Knowledge would allow a student to remember that a rock sample seen in lab was granite. Comprehension involves taking a new rock sample and correctly classifying it as granite.

What igneous rocks are pictured below? (Images of granite and basalt were shown. N=50)

1. Gabbro and granite (6%)
2. Basalt and rhyolite (2%)
3. Granite and basalt* (90%)
4. Rhyolite and gabbro (2%)

Summarizing

A key element of understanding information is being able to summarize the principal concepts from a reading assignment or video. Exercises that ask students to read a newspaper article can present science in the context of everyday life. Such articles often provide opportunities for students to separate observations from hypotheses or to recognize the procedures associated with scientific analysis. Alternatively, students may struggle to fully understand ideas presented in their readings or lecture. Consequently, we can present a variety of definitions with varying degrees of accuracy and completeness and ask students which is the best option.

Analyze the following student descriptions of the relationship between CFCs and ozone. Which is the best? (N=163 on first attempt; 167 for peer instruction)

1. CFCs are manufactured gases that destroy the ozone layer and produce oxygen. (13% on 1st attempt; 0% after peer instruction)
2. Chlorine forms from the disintegration of manufactured gases. The chlorine reacts with ozone molecules to form two gases that can not block incoming UV radiation. (19% on 1st attempt; 14% after peer instruction)
3. Oxygen molecules are broken down by UV radiation to free oxygen atoms that combine with chlorine monoxide to form new oxygen molecules and free chlorine. Chlorine helps break down CFCs with UV radiation. (26% on 1st attempt; 9% after peer instruction)
4. UV radiation breaks apart complex manufactured chlorofluorocarbons into its constituent atoms, including chlorine. Chlorine destroys ozone molecules as it cycles through a pair of reactions that produce free oxygen molecules.* (42% on 1st attempt; 77% after peer instruction)

Inferring

Inferring involves finding a pattern in a series of examples. The correct use of inference reflects an understanding of how related features are linked together. Inference involves recognizing the correct pattern in a series of examples, identifying relationships between multiple features, or drawing a logical conclusion from presented information. Memorizing material as a series of discrete facts or isolated definitions (knowledge) is not sufficient to answer such questions correctly. Inference ConcepTests might make use of completion tasks (What term is missing from this ordered list: atom, element, _______, rock?), analogies (see below), or oddity questions (Which volcano doesn't belong on this list? Mount St. Helens, Mount Shasta, Mount Augustine, Mauna Loa.)

Which statement is the best analogy? (N=147 on first attempt; 157 after peer instruction)

1. An umbrella is to rain as the stratosphere is to solar radiation. (12% on 1st attempt; 4% after peer instruction)
2. An umbrella is to rain as the stratosphere is to ultraviolet radiation. (18% on 1st attempt; 5% after peer instruction)
3. An umbrella is to rain as the ozone layer is to solar radiation. (26% on1st attempt; 11% after peer instruction)
4. An umbrella is to rain as the ozone layer is to ultraviolet radiation.* (44% on 1st attempt; 80% after peer instruction)

 

Two wells (A and B) are drilled into rocks that have the same porosity, but the rocks around well A have a higher permeability than these around well B. Suppose both wells are pumped at the same rate. Which statement is true? (N=145 on first attempt; 133 after peer instruction)

1. Well A will have a larger cone of depression (39% on first attempt; 27% after peer instruction)
2. Well B will have a larger cone of depression* (53% on first attempt; 71% after peer instruction)
3. The cone of depression will be the same for both wells. (8% on first attempt; 2% after peer instruction)

 

Applying

Answers to application questions describe the differences between related phenomena and consider before and after situations. Application often involves applying rules or principles to new situations or using known procedures to solve problems. Key words used in application questions include demonstrate, relate, show, use, compare. Application ConcepTests may involve execution or implementation of procedures. Execution refers to using a proscribed series of steps to complete a task; for example, using an equation to solve a problem. Implementing involves selecting a principle or set of principles to use to solve a problem. For example, applying the principle of superposition to determine the relative ages of a series of rock units.

If radioactive decay began with 400,000 parent isotopes, how many parent isotopes would be left after 3 half lives? (N=200)

1. 200,000 (1%)
2. 100,000 (23%)
3. 50,000* (74%)
4. 25,000 (2%)