Statistics and Error Rates in Death Penalty Cases

Students learn that when data presentations are drawn from popular media accounts, as opposed to standard statistical reports or textbook presentations, additional critical reading and thinking skills are required.

In the course of the activity, students will see how to

• construct a simple tree diagram for computing proportions in two stages
• pay attention to the numerator and denominator of a rate in context
• estimate rate of change and average values on a linear time series plot
• understand some pitfalls of using a double y-axis plot

This is an interactive lecture activity, with in-class discussion. In its most basic form, it is appropriate for an algebra-based introductory statistics class or a quantitative reasoning class following the introduction of basic descriptive statistics, including proportions and time series plots. Followup questions can be developed for more advanced classes.

At a brisk pace, covering just the basic calculations, the activity can be covered in about half a class period (10-15 minutes for set up and group discussion, 10 minutes for wrap-up). At a more leisurely pace, leaving time for side discussions, it could easily expand to a full period.

I provide a handout summarizing the news sources, as described above, along with the following discussion questions. In small groups, the students respond to the questions. We then reassemble the whole class to clear up any misunderstandings and to pool insights.

Discussion Questions

1. The Los Angeles Times published a lengthy article on the Liebman study ("Death Penalty Overturned in Most Cases. Justice: Study finds courts void execution more than two-thirds of the time."June 12, 2000, p. A1). We read there that "state courts initially overturned 47% of the death sentences, having found serious legal flaws. Later federal review discovered 'serious error'...in 40% of the remaining cases, resulting in an overall 68% reversal rate nationwide." How was this overall rate computed?
2. The British journal The Economist published a series of articles on America's death penalty. "America's Death Penalty Lottery" (10 June 2000, p. 15), states: "Since 1976 (when the death penalty was reintroduced), America has executed 640 people (due to be 642 by the time you read this). So the 87 people exonerated after being sentenced to death amount to one reprieve for every seven killed. ...If an airline crashed once for every time it reached its destination, it would surely be suspended immediately." How do you reconcile the one-in-seven figure with the preceding Los Angeles Times summary?
3. "Dead Man Walking Out" (The Economist, 10 June 2000, p.21) reports that "Since 1973, 87 prisoners have been released from death row after new evidence suggested they were innocent...And late acquittals are increasing. Between 1973 and 1993, an average of 2.5 death row inmates a year were found to be innocent. In the six years after that, the rate nearly doubled to 4.6." This passage is clearly meant to suggest that the error rate is increasing. But is it? Does the following graphic, reproduced from from the Los Angeles Times article, shed any light on the matter?
   
   Show LA Times graphic
   Hide
   
4. In "Murder One" (17 June 2000, p.33) The Economist described the pervasiveness of the errors: "85% of the states which have the death penalty make serious mistakes in 60% or more of the cases. ...If there is any state that looks unusually bad it is Virginia: for its size it executes five times as many people as other states and reverses only a quarter the number of cases (see chart)." 
   
   Show Economist graphic
   Hide
   
    
   What do you find confusing about this chart? What recommendations do you have for improving the presentation?

To make the activity authentic, I like to provide newspaper clippings or printed excerpts. If you have access to Nexis/Lexis, you can find these online, but unforunately without charts and pictures. The graphics included here are my Excel reconstructions of the originals (I even tried to match the color scheme from the Economist). The articles referenced the Liebman report, and I found the raw data in the Appendices there:

• Overall Error Rates and Percent Death Sentences Carried Out (Microsoft Word 69kB May15 07) (used in The Economist)
• Death Row Population (Microsoft Word 74kB May15 07) (used in LA Times).

1. As Yogi Berra said (of baseball), "90% of this game is half mental." A simple tree diagram showing with two stages, one state and one federal, does the trick. Students can usually do this quickly.
2. Computing a proportion is straightforward, but the one-in-seven figure is midleading. Sensible ratios might be the number of innocent defendants sentenced to death divided by the total number sentenced, or, more dramatically, the number of innocent people executed divided by the total number of executions. The one-in-seven figure is a mismatch of numerator and denominator: it divides the number of people released from death row by the number of executions. These points were cogently argued by MIT Professor Arnold Barnett in an op/ed piece.
3. The time series from the LA TImes shows that the death row population grew approximately linearly over the study period. This makes it easy to estimate the average death row population from 1973 to 1993 (approx 1500) and from 1993 to 1999 (approx 3200). Since the population roughly doubled, with a constant error rate we would expect the number of errors to double. Students typically figure out that the total death row population matters. With some prodding, they understand how to estimate the value average of a linear function over different time periods.
4. Double-y axis plots are notoriously misleading, because there are two separate scales. Nevertheless, Excel's Chart Wizard touts them as a "Classic comparison chart. Lines plotted on two axes." Many people's first reaction to the Economist graphic is that more than 100% of the sentences are being carried out! With some discussion, students realize that by tweaking the scales, one can make two variables look either highly correlated or nearly independent!

I count activities like this as part of a classroom participation grade. Each group designates a recorder, who turns in a brief summary of the group's discussion at the end of class. I also expect active participation in discussion when the class reconvenes as a whole to pool insights.

For a homework exercise, I ask students to create their own versions of the graphs and summary statistics, in a way that avoids the points of confusion identified in class. In courses that use journaling, I ask for individual responses to the activity their journals.

In a writing-based first-year seminar course, one of my favorite assignments ask students to write critiques of three data graphics that they have culled from media reports (the only ground rule is that at most one can come from USA Today!).

• Liebman, James S., Jeffrey Fagan and Valerie West (2000). A Broken System, Part I: Error Rates in Capital Cases, 1973-1995. Columbia University School of Law. The original study to which the various news items refer.
• Barnett, Arnold (2000, August 30) It's a Crime What Some People Do With Statistics. From the Wall Street Journal, referenced at Junkscience.com. Interesting comments relevant to the second discussion question above.
• Death Penalty Fact Sheet, PDF document, updated May 1, 2007. From the Death Penalty Information Center, where you can also find links to current news stories on this issue.
• Death Penalty 'Curriculum Web Site from the Michigan State Communication Technology Laboratory. Lesson plans for possible two-week unit at the high school level.
• Lesser, Lawrence M. (2007) Critical Values and Transforming Data: Teaching Statistics with Social Justice, Journal of Statistics Education Volume 15, Number 1. See section 3.5 for specific examples using the death penalty.