docs: Tutorial 27 - When the Average Hides the Action (distributional DiD with CiC)

[igerber]

Summary

• New tutorial docs/tutorials/27_cic_distributional_effects.ipynb - a business-framed walkthrough of ChangesInChanges/QDiD on a seed-locked loyalty-program 2x2 (repeated cross-sections) where mean DiD reads $0.22 (p = 0.90) while the known-by-construction truth is a $3.01 mean effect concentrated below the median (CiC ATT: $3.05). Covers: reading the QTE profile; joint "which quantiles moved" claims via sup-t uniform bands (excluding zero for exactly tau = 0.05-0.50, with the bands-silent-not-exonerating framing above the median and the tau = 0.55 pointwise blip, p = 0.044, as the multiple-testing trap they prevent); a live interior-range guardrail demo on a short-support control sample (warnings render in the committed output; interior range (0.17, 0.996), NaN tail inference); the scale-equivariance centerpiece (levels-vs-logs flips mean DiD's verdict and shifts QDiD's profile by dollars while CiC's counterfactual quantiles agree to floating-point precision on unconditional fits - the concrete form of the p. 447 CiC-over-QDiD recommendation); covariate-composition confounding fixed with covariates=['tenure'] (9.38 [8.32, 10.45] truth-excluding -> 6.53 [5.35, 7.70] truth-covering); and a practitioner_next_steps() close.
• The notebook contains NO warnings filters and NO asserts - both locked structurally by the companion drift test (AST-based check); the machine-specific checkout path prefix is normalized out of committed stream text and /Users/ is drift-locked out of all committed outputs.
• Committed WITH executed outputs (nbsphinx renders committed outputs on RTD; the QTE figures are the payload). ~65s runtime under DIFF_DIFF_BACKEND=python, well within the 600s notebook-CI budget.
• Companion drift test tests/test_t27_cic_distributional_effects_drift.py (19 tests): re-derives every prose-quoted number from the public API, pins the uniform-band split/interior range/equivariance identity (rtol 1e-14, libm-safe)/QDiD scale gap/covariate CI endpoints, and cross-checks the rendered markdown + output surfaces. The ~1-minute covariate bootstrap re-derivation is @pytest.mark.slow (T24 precedent).
• Registrations: RTD toctree (Tutorials: Business Applications), docs/tutorials/README.md catalog (also backfilling the previously missing Tutorial 25 entry), docs/doc-deps.yaml, fresh CHANGELOG [Unreleased] section; the tutorial's completed TODO row is removed.

Methodology references (required if estimator / math changes)

• Method name(s): ChangesInChanges (CiC), QDiD, DifferenceInDifferences - tutorial/documentation only; no estimator implementation changes.
• Paper / source link(s): Athey & Imbens (2006), Econometrica 74(2), 431-497 (Theorem 3.1 construction, eq. 17 interior range, Assumption 3.4 support, p. 447 CiC-over-QDiD); Melly & Santangelo (2015) Assumption 4 (conditional support diagnostic); Baker, Callaway, Cunningham, Goodman-Bacon & Sant'Anna (2025) practitioner workflow; Callaway's R qte package (parity conventions: bootstrap-only inference, fixed-95% sup-t bands, xformla covariate branch).
• Any intentional deviations from the source (and why): None introduced - the tutorial restates documented behavior and documented deferrals (discrete-outcome bounds, analytical SEs, full Melly-Santangelo, staggered distributional DiD) as documented.

Validation

• Tests added/updated: tests/test_t27_cic_distributional_effects_drift.py (19 tests; DGP constant + AST sync guards, numeric pins for every quoted claim, warning-behavior locks, rendered-surface quote checks, no-filters/no-asserts/no-local-paths notebook contract). Executed end-to-end: pytest --nbmake green (57s), all 19 drift tests green including the slow covariate re-derivation.
• Backtest / simulation / notebook evidence (if applicable): The notebook itself is the evidence - seed-locked DGP with known truth; story robustness verified across seeds 27-38 (6/12 fully green on every criterion simultaneously, failures graceful).

Security / privacy

• Confirm no secrets/PII in this PR: Yes (committed outputs scrubbed of machine-specific paths; drift-locked).

---

Generated with Claude Code

[github-actions]

Overall Assessment

✅ Looks good — no unmitigated P0/P1 findings.

Executive Summary

• The PR is documentation/tutorial plus drift tests only; no estimator, weighting, variance/SE, or default behavior changes were introduced.
• Affected methods are ChangesInChanges, QDiD, and comparison-only DifferenceInDifferences.
• Methodology claims align with docs/methodology/REGISTRY.md, including documented qte-parity deviations and deferred scope.
• The new drift test covers the load-bearing tutorial claims: mean-vs-CiC contrast, uniform bands, interior-range NaN inference, scale equivariance, and covariate conditioning.
• I did not run the test suite; this review environment lacks pytest.

Methodology

Finding 1 — P3 informational: documented qte-parity deviations are used correctly

Location: docs/tutorials/27_cic_distributional_effects.ipynb:L449-L616, docs/tutorials/27_cic_distributional_effects.ipynb:L679-L750, docs/tutorials/27_cic_distributional_effects.ipynb:L829-L921
Impact: The tutorial relies on bootstrap-only inference, fixed-95% sup-t uniform bands, qte’s QDiD finite-sample form, and the qte xformla covariate branch. These are all documented in docs/methodology/REGISTRY.md:L3961-L4045, including the relevant Note / Deviation from R labels, so they are not methodology defects.
Concrete fix: No action required. Keep the tutorial wording tied to the registry’s qte-parity scope.

Finding 2 — P3 informational: deferred CiC scope is properly disclosed

Location: docs/tutorials/27_cic_distributional_effects.ipynb:L1074-L1077, TODO.md:L115-L119
Impact: Discrete-outcome bounds, analytical SEs, full Melly-Santangelo, staggered/multi-period distributional DiD, and treatment-on-controls remain deferred and are tracked. This is acceptable under the project’s deferred-work policy.
Concrete fix: No action required.

Code Quality

No findings. The PR does not modify production code.

Performance

No blocking findings. The slow covariate bootstrap drift test is marked @pytest.mark.slow at tests/test_t27_cic_distributional_effects_drift.py:L311-L334, which is consistent with the documented QR bootstrap cost.

Maintainability

No findings. The new tutorial is registered in the docs toctree and doc dependency map at docs/index.rst:L75-L88 and docs/doc-deps.yaml:L660-L675.

Tech Debt

No findings. The completed tutorial TODO row was removed, while remaining CiC/QDiD deferred work stays tracked in TODO.md:L115-L119.

Security

Finding 1 — P3 informational: local-path drift guard is narrow

Location: tests/test_t27_cic_distributional_effects_drift.py:L437-L444
Impact: The committed notebook output scan only rejects /Users/. I did not find actual secrets or local path leaks in the added notebook/test, but Linux or Windows absolute paths would not be caught by this guard.
Concrete fix: Optionally broaden the assertion to cover /home/, /tmp/, C:\Users\, and other absolute checkout prefixes.

Documentation/Tests

No blocking findings. The drift test re-derives the tutorial’s reader-facing numeric claims from the public API and checks rendered markdown/output surfaces at tests/test_t27_cic_distributional_effects_drift.py:L197-L435.

I did not run the full tests because pytest is not installed in this review environment.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 3d586f2520df33db37c830332301ce4d52d2570f

---

Overall Assessment

✅ Looks good — no unmitigated P0/P1 findings.

Executive Summary

• Re-review scope: prior review had no P0/P1 findings; the prior P3 local-path guard concern is addressed by the broader denylist in tests/test_t27_cic_distributional_effects_drift.py:L437-L446.
• This PR changes docs/tutorial/test registration only; no production estimator, weighting, variance/SE, or default behavior changes are introduced.
• Affected methods are ChangesInChanges/CiC and QDiD; DifferenceInDifferences is used only as a comparison baseline.
• Methodology claims align with docs/methodology/REGISTRY.md:L3961-L4037, including documented qte-parity deviations and deferred scope.
• I could not run tests because pytest is not installed in this environment.

Methodology

Finding 1 — P3 informational: documented CiC/QDiD conventions are used correctly

Severity: P3
Impact: The tutorial relies on bootstrap-only inference, fixed-95% sup-t bands, CiC interior-range NaN inference, qte finite-sample QDiD conventions, and the qte xformla covariate branch. These are documented in the registry with Note/Deviation labels, so they are not methodology defects. See docs/tutorials/27_cic_distributional_effects.ipynb:L49-L55, docs/tutorials/27_cic_distributional_effects.ipynb:L453-L456, docs/tutorials/27_cic_distributional_effects.ipynb:L610-L615, docs/tutorials/27_cic_distributional_effects.ipynb:L829-L924; registry support at docs/methodology/REGISTRY.md:L3967-L4000 and docs/methodology/REGISTRY.md:L4022-L4034.
Concrete fix: No action required.

Finding 2 — P3 informational: deferred CiC scope remains tracked

Severity: P3
Impact: The tutorial’s “not covered” items — discrete-outcome bounds, analytical SEs, full Melly-Santangelo, and staggered/multi-period distributional DiD — are disclosed and tracked. See docs/tutorials/27_cic_distributional_effects.ipynb:L1072-L1077, TODO.md:L115-L118, and docs/methodology/REGISTRY.md:L4011-L4016.
Concrete fix: No action required.

Code Quality

No findings. The PR does not modify production code, and the added drift test uses the public estimator APIs rather than duplicating estimator internals.

Performance

Finding 1 — P3 informational: slow covariate bootstrap test is marked slow

Severity: P3
Impact: The covariate branch re-derivation is intentionally expensive and is marked @pytest.mark.slow, with the runtime rationale documented in the test. See tests/test_t27_cic_distributional_effects_drift.py:L311-L334.
Concrete fix: No action required.

Maintainability

No findings. The tutorial is registered in the RTD toctree and dependency map at docs/index.rst:L75-L88 and docs/doc-deps.yaml:L660-L675.

Tech Debt

No findings. The completed tutorial backlog row was removed, while remaining CiC deferred work is still tracked in TODO.md:L115-L118.

Security

No findings. The prior local-path guard concern is resolved by checking common macOS/Linux/Windows/temp prefixes in committed notebook outputs at tests/test_t27_cic_distributional_effects_drift.py:L437-L446.

Documentation/Tests

No blocking findings. The new drift tests pin the tutorial’s load-bearing claims: mean-vs-CiC contrast, QTE profile, uniform bands, interior-range warning/NaN behavior, scale equivariance, covariate conditioning, rendered prose/output quotes, no warning filters/asserts, and local-path output hygiene at tests/test_t27_cic_distributional_effects_drift.py:L197-L454.

I did not run the suite because python -m pytest --version failed with No module named pytest.