10 Foundational Quantitative Reasoning Questions

Neil Lutsky, Psychology

I. What do the numbers show?

  • What do the numbers mean?
  • Where are the numbers?
    • Is there numerical evidence to support a claim?
    • What were the exact figures?
    • How can seeking and analyzing numbers illuminate important phenomena?
  • How plausible is a possibility in light of back of the envelope calculations?

II. How representative is that?

  • What's the central tendency?
    • "For instance is no proof."
    • Mean, Mode, and Median.
  • Interrogating averages:
    • Are there extreme scores?
    • Are there meaningful subgroups?
    • Who's in the denominator?
    • What's the variability (standard deviation)?
  • What are the odds of that? What's the base rate?

III. Compared to what?

  • What's the implicit or explicit frame of reference?
  • What's the unit of measurement?
  • Per what?
  • What's the order of magnitude?
  • Interrogating a graph:
    • What's the Y-axis? Is it zero-based?
    • Does it K.I.S.S., or is it filled with ChartJunk?

IV. Is the outcome statistically significant?

  • Is the outcome unlikely to have come about by chance?
    • "Chance is lumpy."
    • Criterion of sufficient rarity due to chance: p < .05
  • What does statistical significance mean, and what doesn't it mean?

V. What's the effect size?

  • How can we take the measure of how substantial an outcome is?
  • How large is the mean difference? How large is the association?
  • Standardized mean difference (d): d = (μ1-μ2)/σ

VI. Are the results those of a single study or of a literature?

  • What's the source of the numbers: PFA, peer-reviewed, or what?
  • Who is sponsoring the research?
  • How can we take the measure of what a literature shows?
  • The importance of meta-analysis in the contemporary world of QR.

VII. What's the research design (correlational or experimental)?

  • Design matters: Experimental vs. correlational design.
  • How well does the design support a causal claim?
  • Experimental Design:
    • Randomized Controlled Trials (RCT): Research trials in which participants are randomly assigned to the conditions of the study.
    • Double blind trials: RCTs in which neither the researcher nor the patient know the treatment condition.
  • Correlational Design: Measuring existing variation and evaluating co-occurrences, possibly controlling for other variables.
    • Interrogating associations (correlations):
      • Are there extreme pairs of scores (outliers)?
      • Are there meaningful subgroups?
      • Is the range of scores in a variable restricted?
      • Is the relationship non-linear?

VIII. How was the variable operationalized?

  • What meaning and degree of precision does the measurement procedure justify?
  • What elements and procedures result in the assignment of a score to a variable?
  • What exactly was asked?
  • What's the scale of measurement?
  • How might we know if the measurement procedure is a good one?
    • Reliability = Repeated applications of the procedure result in consistent scores.
    • Validity = Evidence supports the use to which the measure is being put.
  • Is the measure being manipulated or "gamed"? The iatrogenic effects of measurement.

IX. Who's in the measurement sample?

  • What domain is being evaluated? Who's in? Who's not?
  • Is the sample from that domain representative, meaningful, and/or sufficient?
  • Is the sample random?
  • Are two or more samples that are being compared equivalent?

X. Controlling for what?

  • What other variables might be influencing the findings?
  • Were these assessed or otherwise controlled for in the research design?
  • What don't we know, and how can we acknowledge uncertainties?