Assessing the Measurement and Validity of Ambiguous Concepts in Ethnic Conflict Datasets

1. To get students to think critically about the sources of data and how data are produced
2. To sensitize students to validity and measurement challenges, especially when dealing with complex, under-theorized, or ambiguous concept
3. To familiarize students with commonly used datasets in ethnic conflict studies so that they can be more informed users of those datasets in their own research
4. To give students experience using and writing about simple descriptive statistics

These assignments were designed for a class of 25-30 students, mostly sophomores and juniors from a variety of disciplinary backgrounds (though social science majors comprise the largest group). Basic statistical knowledge is required, though students who lack such knowledge can still complete this assignment with one-on-one guidance or with an optional data analysis workshop held outside of class hours.

Prior to handing out this assignment, I schedule time in class to talk about variables, measurement, and validity and assign a reading on these issues. The two written assignments themselves are due in the third and fourth week of the term, which is when the students first encounter more empirical (as opposed to theoretical) readings.

This sequence would work in larger classes, so long as most students have the necessary grounding in statistics to carry out the first writing assignment; if many students lack this foundation, providing proper coaching could be burdensome. The sequence would also work in different subject areas so long as there are commonly used (and easily accessible) datasets associated with that topic.

Prior to handing out the assignment instructions, I assign an article on "Improving Data Quality: Actors, Incentives, and Capabilities" by Yoshiko Herrera and Devesh Kapur (full citation given below). This article outlines some of the challenges of working with large political science datasets and suggests ways that students can assess the quality of the data they find. I also schedule time in class to discuss this article in order to get students thinking about data quality, measurement, and validity.

The assignment consists of two parts, provided below. Part I: Comparing Two Datasets (Acrobat (PDF) 72kB Sep19 08) Part II: Datasets in Scholarly Writing (Acrobat (PDF) 71kB Sep19 08) I provide links on the course website to the various datasets that students can use for this assignment. The names and locations of these datasets are provided in the attachment below; all are free to use. (Acrobat (PDF) 68kB Sep19 08) I also provide them a list of sample articles they can use for part 2 of the assignment; these are also given in an attachment below. They are free to use other articles if they clear it with me ahead of time. (Acrobat (PDF) 88kB Sep19 08)

I find that students who have never used large datasets are often confused by how they are structured, how to use the codebooks, and how to take raw files and view them in Excel or a statistics program. To help de-mystify the process, I find it useful to schedule a class visit from a reference librarian or academic technologist who can talk about the practical details of finding, downloading, and navigating datasets.

I also offer an optional evening session for students who would like some coaching on basic descriptive statistics. This session lasts about an hour and is intended primarily for those students who might be less adept at performing and interpreting basic statistics. I also invite students who might know how to carry out the statistical analysis but might need a refresher on the software to attend as well. In a larger class, a teaching assistant could take on this role.

To make it easier for students to work with the datasets, I make the downloadable files available in three formats: Excel, SPSS, and Stata. This way, students are able to open and view the files in the format most comfortable for them.

I evaluate the two papers using the following criteria:

Content & Organization

• Does the student articulate a clear, specific, and interesting thesis?
• Does the student support this thesis using appropriate and logical reasoning and sound evidence?
• Does the student discuss the strengths and weaknesses of each dataset in an informed and knowledgeable way?
• Does the student use descriptive statistics correctly and effectively in his/her analysis of the datasets?
• Does the student use his/her exploration of the datasets to develop a nuanced critique of the scholarly article?
• Does the student acknowledge and deal with possible counterarguments to his/her argument?
• Is the paper organized logically? Does the student use effective transitions to connect one idea to the next?

Writing & Presentation

• Is the paper free from spelling, grammar, punctuation, and usage errors?
• Is the prose clear, direct, and free of jargon?
• Is the language precise and concrete?
• Does the student use charts and tables effectively when presenting statistical results? Does the student title and introduce those charts and tables in the body of the text?

Article assigned to students prior to the assignment:

Herrera, Yoshiko M. 2007. Improving data quality: actors, incentives, and capabilities. Political Analysis, vol. 15, pp. 365-386.