Writing Better Multiple Choice Questions

Wednesday 1:30pm-4:00pm E Building 104
Afternoon Mini Workshop

Conveners

Cody Kirkpatrick, Indiana University-Bloomington
Lauren Burns, University of North Carolina at Charlotte

Whether because of class size, grading time, or other constraints, sometimes we need to include multiple choice questions in our summative assessments. However, it is often difficult and time-consuming to write "good" questions that reliably assess the knowledge we want them to. In this workshop, attendees will review the Bloom's Taxonomy framework and the literature on multiple choice question design; evaluate (good and bad) example questions from other sources; and we will spend time writing, sharing, and revising our own questions for future use.

Goals

During this workshop, participants will:

  • Recall the levels of Bloom's Taxonomy
  • Recognize by example the fundamentals of good (and bad) question design
  • Work independently on writing your own questions (whether to assess content knowledge, procedural skill, etc.)
  • Share and revise questions with colleagues, leaving with new questions to ask on your own assessments

Program

1:30 Welcome; what makes a multiple choice question good or bad?

* Slides for today

* Activity 1: Examples of "bad" questions (Google Doc - we'll add comments as a whole group)

2:00 Bloom's Taxonomy and the cognitive level of questions

* Handout: Bloom's level descriptions from Anderson and Krathwohl

* Popular in K12: Costa's levels of questioning

* Activity 2: Identify the Bloom's level of these questions (Google Doc - we'll add comments as a whole group)

2:30 (10 min) Break + (10 min) identify a learning outcome from one of your courses you want to collaborate on at your table

2:50 Work on your own to write an Apply or Analyze question (we'll offer suggestions on where to start, if you need them)

* Some question stem examples and the "Item Shells" table by Haladyna (Google Doc)

* Activity 3: Working on your own questions (you'll work in one of these Google Jamboards)
Geology - Geomorphology - Atmosphere, Oceans, and Climate

3:10 Peer discussion: help your colleagues check their questions

3:30 Whole-group discussion: share your example questions

3:50 Workshop evaluation

4:00 Adjourn

Questions During our Workshop

Additional Resources

Bloom's Taxonomy References

  • Armstrong, P., 2010: Bloom's Taxonomy. Vanderbilt University Center for Teaching. Retrieved 3 July 2023. (The traditional "pyramid" graphic, and some action words for each level.)
  • Barton, M., 2018: Flipping Part 2. Indiana University Center for Teaching and Learning, Retrieved 5 July 2023. (Very helpful blog post with a reminder that Bloom's is a TWO-dimensional taxonomy. Graphic has representative tasks at each level.)
  • Center for Teaching and Learning, UNC Charlotte, Bloom's Taxonomy of Educational Objectives. Retrieved 5 July 2023. (Examples in Tables 1 and 2 are relevant to science classes, so are helpful for our attendees to think about.)
  • Cummins, K., 2019: A teacher's guide to Bloom's Taxonomy. Retrieved 5 July 2023.  (Blog post with verb list for each level -- scroll halfway down to find them.)

Selected Publications on Question Writing

  • Brame, C., 2013: Writing good multiple choice test questions. Vanderbilt University Center for Teaching. Retrieved 3 July 2023.
  • Fuhrman, M., 1996: Developing Good Multiple-Choice Tests and Test Questions, Journal of Geoscience Education, 44:4, 379-384, DOI: 10.5408/1089-9995-44.4.379
  • Moreno, R., Martínez, R. J., & Muñiz, J., 2006: New guidelines for developing multiple-choice items. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 2(2), 65–72. https://doi.org/10.1027/1614-2241.2.2.65
  • Towns, M. H., 2014: Guide to developing high-quality, reliable, and valid multiple-choice assessments. Journal of Chemical Education, 91, 1426−1431.  10.1021/ed500076x 

Further Reading

  • Recommended: Parkes and Zimmaro, 2016: Learning and Assessing with Multiple-Choice Questions in College Classrooms.  Routledge Press.  (Includes question writing, overall assessment design, and evaluation of assessment validity and reliability.  If you use lots of M/C assessments, it's a must-have.)
  • Recommended: Anderson and Krathwohl, 2001: A Taxonomy for Learning, Teaching, and Assessing.  Addison & Wesley.  (The current Bloom's Taxonomy reference guide.)
  • Also consider: Haladyna, 1997: Writing Test Items to Evaluate Higher Order Thinking. Allyn & Bacon.  (Useful guide for all types of higher-order question types.)
  • Also consider: Haladyna and Rodriguez, 2013: Developing and Validating Multiple Choice Test Items.  Routledge.  (Discusses all types of assessment items, including open-ended, measuring writing ability, and more.  Neat chapter on "automatic item generation.")
  • Also consider: Shank, 2021: Write Better Multiple Choice Questions to Assess Learning. Independently published, available on Amazon and elsewhere.  (Easy-to-read book that covers lots of the issues of multiple-choice assessment design.)
  • Also consider: McDonald, 2002: Systematic Assessment of Learning Outcomes: Developing Multiple-Choice Exams.  Jones & Bartlett.  (Written for nursing, I still find it a good reference when I need to be pushed to think outside of the geosciences.)