10 Foundational Quantitative Reasoning Questions

Neil Lutsky, Psychology

I. What do the numbers show?

  • What do the numbers mean?
  • Where are the numbers?
    • Is there numerical evidence to support a claim?
    • What were the exact figures?
    • How can seeking and analyzing numbers illuminate important phenomena?
  • How plausible is a possibility in light of back of the envelope calculations?

II. How representative is that?

  • What's the central tendency?
    • "For instance is no proof."
    • Mean, Mode, and Median.
  • Interrogating averages:
    • Are there extreme scores?
    • Are there meaningful subgroups?
    • Who's in the denominator?
    • What's the variability (standard deviation)?
  • What are the odds of that? What's the base rate?

III. Compared to what?

  • What's the implicit or explicit frame of reference?
  • What's the unit of measurement?
  • Per what?
  • What's the order of magnitude?
  • Interrogating a graph:
    • What's the Y-axis? Is it zero-based?
    • Does it K.I.S.S., or is it filled with ChartJunk?

IV. Is the outcome statistically significant?

  • Is the outcome unlikely to have come about by chance?
    • "Chance is lumpy."
    • Criterion of sufficient rarity due to chance: p
    • What does statistical significance mean, and what doesn't it mean?

    V. What's the effect size?

    • How can we take the measure of how substantial an outcome is?
    • How large is the mean difference? How large is the association?
    • Standardized mean difference (d): d = (μ1-μ2)/σ

    VI. Are the results those of a single study or of a literature?

    • What's the source of the numbers: PFA, peer-reviewed, or what?
    • Who is sponsoring the research?
    • How can we take the measure of what a literature shows?
    • The importance of meta-analysis in the contemporary world of QR.

    VII. What's the research design (correlational or experimental)?

    • Design matters: Experimental vs. correlational design.
    • How well does the design support a causal claim?
    • Experimental Design:
      • Randomized Controlled Trials (RCT): Research trials in which participants are randomly assigned to the conditions of the study.
      • Double blind trials: RCTs in which neither the researcher nor the patient know the treatment condition.
    • Correlational Design: Measuring existing variation and evaluating co-occurrences, possibly controlling for other variables.
      • Interrogating associations (correlations):
        • Are there extreme pairs of scores (outliers)?
        • Are there meaningful subgroups?
        • Is the range of scores in a variable restricted?
        • Is the relationship non-linear?

    VIII. How was the variable operationalized?

    • What meaning and degree of precision does the measurement procedure justify?
    • What elements and procedures result in the assignment of a score to a variable?
    • What exactly was asked?
    • What's the scale of measurement?
    • How might we know if the measurement procedure is a good one?
      • Reliability = Repeated applications of the procedure result in consistent scores.
      • Validity = Evidence supports the use to which the measure is being put.
    • Is the measure being manipulated or "gamed"? The iatrogenic effects of measurement.

    IX. Who's in the measurement sample?

    • What domain is being evaluated? Who's in? Who's not?
    • Is the sample from that domain representative, meaningful, and/or sufficient?
    • Is the sample random?
    • Are two or more samples that are being compared equivalent?

    X. Controlling for what?

    • What other variables might be influencing the findings?
    • Were these assessed or otherwise controlled for in the research design?
    • What don't we know, and how can we acknowledge uncertainties?