Writing Better Multiple Choice Questions

Wednesday 1:30pm-4:00pm E Building 104

Afternoon Mini Workshop

Conveners

Cody Kirkpatrick, Indiana University-Bloomington

Lauren Burns, University of North Carolina at Charlotte

Whether because of class size, grading time, or other constraints, sometimes we need to include multiple choice questions in our summative assessments. However, it is often difficult and time-consuming to write "good" questions that reliably assess the knowledge we want them to. In this workshop, attendees will review the Bloom's Taxonomy framework and the literature on multiple choice question design; evaluate (good and bad) example questions from other sources; and we will spend time writing, sharing, and revising our own questions for future use.

Goals

During this workshop, participants will:

Recall the levels of Bloom's Taxonomy
Recognize by example the fundamentals of good (and bad) question design
Work independently on writing your own questions (whether to assess content knowledge, procedural skill, etc.)
Share and revise questions with colleagues, leaving with new questions to ask on your own assessments

Program

1:30 Welcome; what makes a multiple choice question good or bad?

* Slides for today

* Activity 1: Examples of "bad" questions (Google Doc - we'll add comments as a whole group)

2:00 Bloom's Taxonomy and the cognitive level of questions

* Handout: Bloom's level descriptions from Anderson and Krathwohl

* Popular in K12: Costa's levels of questioning

* Activity 2: Identify the Bloom's level of these questions (Google Doc - we'll add comments as a whole group)

2:30 (10 min) Break + (10 min) identify a learning outcome from one of your courses you want to collaborate on at your table

2:50 Work on your own to write an Apply or Analyze question (we'll offer suggestions on where to start, if you need them)

* Some question stem examples and the "Item Shells" table by Haladyna (Google Doc)

* Activity 3: Working on your own questions (you'll work in one of these Google Jamboards)
Geology - Geomorphology - Atmosphere, Oceans, and Climate

3:10 Peer discussion: help your colleagues check their questions

3:30 Whole-group discussion: share your example questions

3:50 Workshop evaluation

4:00 Adjourn

Questions During our Workshop

"How many answer choices do we need?" Three is often sufficient! See, for example: "Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research" by Rodriguez (2005).
"Where do I come up with good distractors?" Ask open-ended questions and use student responses to those! See, for example: "Multiple-Choice Item Distractor Development Using Topic Modeling Approaches" by Shin et al. (2019).
"What about asking several questions from the same figure or image?" Yes, great idea... as long as each of them addresses a separate, clearly-identifiable learning outcome. These are often called "testlets."
"What about select-all-that-apply questions?" Yes, these work. Opinion: how you construct these might depend on the level of your students. If you have less-experienced students, you might want to separate some of the options out into separate, simpler questions (e.g., Question 1 is warm vs. cold, Question 2 is humid vs. dry). And beware, we do know that complex-type questions are more difficult than more conventional multiple choice questions (e.g., Tripp and Tollefson 1985).

Additional Resources

Bloom's Taxonomy References

Armstrong, P., 2010: Bloom's Taxonomy. Vanderbilt University Center for Teaching. Retrieved 3 July 2023. (The traditional "pyramid" graphic, and some action words for each level.)
Barton, M., 2018: Flipping Part 2. Indiana University Center for Teaching and Learning, Retrieved 5 July 2023. (Very helpful blog post with a reminder that Bloom's is a TWO-dimensional taxonomy. Graphic has representative tasks at each level.)
Center for Teaching and Learning, UNC Charlotte, Bloom's Taxonomy of Educational Objectives. Retrieved 5 July 2023. (Examples in Tables 1 and 2 are relevant to science classes, so are helpful for our attendees to think about.)
Cummins, K., 2019: A teacher's guide to Bloom's Taxonomy. Retrieved 5 July 2023. (Blog post with verb list for each level -- scroll halfway down to find them.)

Selected Publications on Question Writing

Brame, C., 2013: Writing good multiple choice test questions. Vanderbilt University Center for Teaching. Retrieved 3 July 2023.
Fuhrman, M., 1996: Developing Good Multiple-Choice Tests and Test Questions, Journal of Geoscience Education, 44:4, 379-384, DOI: 10.5408/1089-9995-44.4.379
Moreno, R., Martínez, R. J., & Muñiz, J., 2006: New guidelines for developing multiple-choice items. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 2(2), 65–72. https://doi.org/10.1027/1614-2241.2.2.65
Towns, M. H., 2014: Guide to developing high-quality, reliable, and valid multiple-choice assessments. Journal of Chemical Education, 91, 1426−1431. 10.1021/ed500076x