Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,33 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- **New tutorial: `docs/tutorials/27_cic_distributional_effects.ipynb` - "When the
Average Hides the Action: Distributional DiD with Changes-in-Changes".** A
business-framed walkthrough of `ChangesInChanges`/`QDiD` on a seed-locked loyalty-
program 2x2 (repeated cross-sections) where mean DiD reads $0.22 (p = 0.90) while
the known truth is a $3.01 mean effect concentrated in the bottom half of the spend
distribution: reading the QTE profile, joint "which quantiles moved" claims via
sup-t uniform bands (excluding zero for exactly tau = 0.05-0.50), a live interior-
range guardrail demo on a short-support control sample (the Assumption-3.4 and
interior-range warnings render in the committed output - no warning filters
anywhere in the notebook), the scale-equivariance centerpiece (levels-vs-logs flips
mean DiD's verdict and shifts QDiD's profile by dollars while CiC's counterfactual
quantiles agree to floating-point precision on unconditional fits - the concrete
form of the Athey-Imbens p. 447 CiC-over-QDiD recommendation), covariate-
composition confounding fixed with `covariates=['tenure']` (confidently-wrong 9.38
-> truth-covering 6.53 on a design ported from the calibrated methodology-test
DGP), and a `practitioner_next_steps()` close. Committed WITH executed outputs
(nbsphinx renders them on RTD; figures are the payload). Companion drift test
`tests/test_t27_cic_distributional_effects_drift.py` (19 tests) re-derives every
prose-quoted number from the public API, locks the no-filters/no-asserts notebook
contract, cross-checks the rendered surface, and slow-marks the ~1-minute covariate
bootstrap re-derivation (T24 precedent). Registered in the RTD toctree (Business
Applications), the tutorials catalog (also backfilling the missing Tutorial 25
entry), and `docs/doc-deps.yaml`.

## [3.8.0] - 2026-07-18

### Added
Expand Down
1 change: 0 additions & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m

| Issue | Location | Origin | Effort | Priority |
|-------|----------|--------|--------|----------|
| ChangesInChanges/QDiD tutorial notebook (2x2 distributional walkthrough: QTE grid, interior range, uniform bands, CiC-vs-QDiD comparison) - deferred from the implementation PR as a documented decision. | `docs/tutorials/` | #682 | Mid | Low |
| Tighten the mypy suppressions that back the enforced-zero posture: burn down `prep_dgp`'s per-module `[index]` override (needs a None-vs-array restructure that preserves the seeded RNG stream), and evaluate re-enabling the globally disabled codes (`arg-type`, `return-value`, `var-annotated`, `assignment`) one at a time — `assignment` alone hid several real annotation drifts found during the 2026-07 triage. | `pyproject.toml` `[tool.mypy]`, `diff_diff/prep_dgp.py` | lint-CI | Mid | Low |
| Align the four legacy dataset loaders (`load_card_krueger`, `load_castle_doctrine`, `load_divorce_laws`, `load_mpdta`) with the loud-fallback pattern of `load_prop99`/`load_walmart`: `UserWarning` + `df.attrs["source"]` marker on synthetic fallback (currently silent), plus optional checksum pinning for the CSV downloads. **Upgraded to a live defect 2026-07-13: the `causaldata/causal_datasets` GitHub repo backing castle/card_krueger/divorce is dead (404), so those loaders silently serve synthetic data everywhere - needs loud fallback + replacement sources.** | `diff_diff/datasets.py` | LWDiD precursor | Quick | Medium |
| Real-data CI canary for dataset-backed replication tests: `test_methodology_lwdid.py`'s Prop 99 / Walmart goldens skip (visibly) when loaders fall back to synthetic; add a lane or canary asserting `df.attrs["source"] == "lwdid_ssc_ancillary"` in CI so network regressions cannot silently de-gate the replication tests. Pairs with the loader-fallback repair row above. | `tests/test_methodology_lwdid.py`, `.github/workflows/` | LWDiD validation suite | Quick | Low |
Expand Down
3 changes: 3 additions & 0 deletions docs/doc-deps.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,9 @@ sources:
type: methodology
- path: docs/api/changes_in_changes.rst
type: api_reference
- path: docs/tutorials/27_cic_distributional_effects.ipynb
type: tutorial
note: "committed outputs quote seed-locked numbers; companion drift test test_t27_cic_distributional_effects_drift.py re-derives them"
- path: README.md
section: "Estimators (one-line catalog entry)"
type: user_guide
Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ Quick Links
tutorials/22_had_survey_design
tutorials/23_spillover_tva
tutorials/26_composition_drift_calibration
tutorials/27_cic_distributional_effects

.. toctree::
:maxdepth: 1
Expand Down
1,113 changes: 1,113 additions & 0 deletions docs/tutorials/27_cic_distributional_effects.ipynb

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions docs/tutorials/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,13 @@ Power-analysis decision guide for geo experiments (framed on a 50-state staggere
- When a clean-tail 2×2 is unbiased, the small-holdout and few-clusters caveats, and a CS-vs-2×2 decision guide
- Fully self-contained: runs live (no committed data files)

### 25. Synthetic Control for a Policy Evaluation: Two Routes to Inference (`25_synthetic_control_policy.ipynb`)
A single state adopts a clean-energy standard and no other state is a clean control - `SyntheticControl` builds a weighted donor-blend counterfactual, and with one treated unit the analytical inference fields are NaN by design, so the tutorial walks both genuine inference routes:
- Philosophy A - compare across regions: the ADH (2010) in-space placebo permutation (`in_space_placebo()`, RMSPE-ratio rank p-value) and the Firpo-Possebom (2018) confidence set
- Philosophy B - compare across time: Chernozhukov-Wuthrich-Zhu (2021) conformal inference, inverting a permutation-over-time test into a p-value for a hypothesized effect path and per-post-period confidence intervals
- The two philosophies side by side: what each does and does not protect against, and how to report them together
- Fully self-contained: runs live (no committed data files)

### 26. Composition Drift & Survey Calibration with balance (`26_composition_drift_calibration.ipynb`)
The failure-mode companion to Meta balance's `balance_diff_diff_brfss` tutorial: when non-response drift correlates with treatment timing, the design-weight DiD itself is biased and calibration becomes essential for the *causal* estimand:
- BRFSS-style smoking-ban DGP with no systematic arm-specific trends (parallel trends hold in expectation; planted ATT -3.0pp, realized -2.98pp) and treatment-correlated non-response drift
Expand All @@ -144,6 +151,14 @@ The failure-mode companion to Meta balance's `balance_diff_diff_brfss` tutorial:
- Estimator sweep (CS / SunAbraham / ImputationDiD), `survey_metadata` DEFF diagnostics, and `as_balance_diagnostic` cross-package diagnostics
- Requires `pip install "balance>=0.21"` (this tutorial only); fully self-contained data

### 27. When the Average Hides the Action: Distributional DiD with Changes-in-Changes (`27_cic_distributional_effects.ipynb`)
A loyalty program looks dead on mean DiD ($0.22, p = 0.90) but in truth lifted the bottom half of the spend distribution - `ChangesInChanges` (Athey & Imbens 2006) recovers the full quantile-treatment-effect profile:
- Reading a QTE profile, and making joint "which quantiles moved" claims with sup-t uniform bands (they exclude zero for exactly tau = 0.05-0.50 here)
- The interior-range guardrail live: a short-support control sample triggers loud warnings and NaN tail inference instead of silent extrapolation
- The scale-equivariance centerpiece: levels-vs-logs flips mean DiD's verdict and shifts QDiD's profile by dollars, while CiC's counterfactual quantiles agree to floating-point precision (unconditional fits)
- Covariate-composition confounding fixed with `covariates=` (quantile-regression conditioning, qte `xformla` parity), and the `practitioner_next_steps()` close
- Companion drift-test file (`tests/test_t27_cic_distributional_effects_drift.py`); fully self-contained (runs live, no committed data files)

## Running the Notebooks

1. Install diff-diff with dependencies:
Expand Down
Loading
Loading