Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
purpose-phrasing, restoring the case as a genuinely clean negative control.

### Added
- **MMM calibration export (`diff_diff.mmm`)** - interop bridge converting DiD
experiment results into Marketing Mix Model calibration inputs, with no MMM
package dependency. `to_pymc_marketing_lift_test()` emits the lift-test
DataFrame consumed by PyMC-Marketing's `MMM.add_lift_test_measurements`
(columns `channel`/dims/`x`/`delta_x`/`delta_y`/`sigma`; `aggregate=` per-period
vs cumulative scaling, `n_units=` geo-to-national scaling, explicit
`on_wrong_sign=` policy mirroring PyMC-Marketing's monotonicity check).
`to_meridian_roi_prior()` emits Google Meridian lognormal ROI prior parameters
(`MeridianROIPrior` with `mu`/`sigma` via Google's closed form, spend-weighted
multi-experiment pooling, `se_widening=` transferability knob, and a
channel-scoped `.to_code()` snippet helper - Meridian's `roi_m` prior is
per-channel, so the snippet requires the experiment channel plus the model's
channel order, or an explicit `single_channel=True`). Deterministic
rescaling of a headline ATT is restricted to a verified count-weighted,
level-scale allowlist (DiD/SyntheticDiD/MultiPeriodDiD common-window
products; CallawaySantAnna via the exact `unit_periods=` scale - its simple
aggregation averages cells with unequal cohort durations); subclasses and
all other estimators fail closed and export only via the explicit
`incremental_outcome=` + `incremental_outcome_se=` pair on the Meridian
path. `DiDResults` gains a fit-stamped `estimator` provenance field
(`DifferenceInDifferences` vs `TwoWayFixedEffects`, which shares the class);
the exporters require DifferenceInDifferences provenance and reject TWFE
(negative-weight bias on staggered data) and hand-built results without
provenance. Treated counts are never inferred from result metadata.
- **Reviewer-eval harness: N-arm matrix, blinded grading, corpus grown 2 -> 11
(`tools/reviewer-eval/`, prep for the GPT-5.6 reviewer evaluation).**
`config/configs.json` moves from the two-arm `control`/`candidate` shape to an
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ Full guide: `diff_diff.get_llm_guide("practitioner")`.
- [Honest DiD](https://diff-diff.readthedocs.io/en/stable/api/honest_did.html) - Rambachan & Roth (2023) sensitivity analysis: robust CI under PT violations, breakdown values
- [Pre-Trends Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/pretrends.html) - Roth (2022) minimum detectable violation and power curves
- [Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/power.html) - analytical and simulation-based MDE, sample size, power curves for study design
- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html) - convert experiment results into MMM calibration inputs: PyMC-Marketing lift-test frames and Google Meridian lognormal ROI priors
- Conley spatial HAC SE (`vcov_type="conley"`) on cross-sectional `LinearRegression` / `compute_robust_vcov` plus panel `DifferenceInDifferences` / `MultiPeriodDiD` / `TwoWayFixedEffects` (with `conley_lag_cutoff` for within-unit Bartlett temporal HAC) - Conley (1999) spatial-correlation-aware SEs with parity vs R `conleyreg` on cross-sectional + panel fixtures, optional combined spatial + cluster product kernel via explicit `cluster=`, auto-activating sparse k-d-tree fast path for `n > 5_000`

## Survey Support
Expand Down
5 changes: 5 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ generic sparse-FE, QR+SVD rank-detection redundancy, `check_finite` bypass — m

| Issue | Location | Origin | Effort | Priority |
|-------|----------|--------|--------|----------|
| MMM interop follow-up: Meridian `roi_calibration_period` boolean-mask builder (align the ROI prior's definition window to the experiment dates; needs the user's Meridian time index + channel ordering as inputs) - deferred from v1 as a documented decision. | `diff_diff/mmm.py` | mmm-interop | Quick | Low |
| MMM interop follow-up: `practitioner_next_steps` handler step + `llms-practitioner.txt` Step 8 pointer recommending the MMM exporters as the reporting hand-off (tutorial PR-B scope). | `diff_diff/practitioner.py`, `diff_diff/guides/llms-practitioner.txt` | mmm-interop | Quick | Low |
| Tighten the mypy suppressions that back the enforced-zero posture: burn down `prep_dgp`'s per-module `[index]` override (needs a None-vs-array restructure that preserves the seeded RNG stream), and evaluate re-enabling the globally disabled codes (`arg-type`, `return-value`, `var-annotated`, `assignment`) one at a time — `assignment` alone hid several real annotation drifts found during the 2026-07 triage. | `pyproject.toml` `[tool.mypy]`, `diff_diff/prep_dgp.py` | lint-CI | Mid | Low |
| Align the four legacy dataset loaders (`load_card_krueger`, `load_castle_doctrine`, `load_divorce_laws`, `load_mpdta`) with the loud-fallback pattern of `load_prop99`/`load_walmart`: `UserWarning` + `df.attrs["source"]` marker on synthetic fallback (currently silent), plus optional checksum pinning for the CSV downloads. **Upgraded to a live defect 2026-07-13: the `causaldata/causal_datasets` GitHub repo backing castle/card_krueger/divorce is dead (404), so those loaders silently serve synthetic data everywhere - needs loud fallback + replacement sources.** | `diff_diff/datasets.py` | LWDiD precursor | Quick | Medium |
| Real-data CI canary for dataset-backed replication tests: `test_methodology_lwdid.py`'s Prop 99 / Walmart goldens skip (visibly) when loaders fall back to synthetic; add a lane or canary asserting `df.attrs["source"] == "lwdid_ssc_ancillary"` in CI so network regressions cannot silently de-gate the replication tests. Pairs with the loader-fallback repair row above. | `tests/test_methodology_lwdid.py`, `.github/workflows/` | LWDiD validation suite | Quick | Low |
Expand Down Expand Up @@ -117,6 +119,9 @@ Doable in principle, but no current caller and/or explicitly out of paper scope.
| ChangesInChanges analytical SEs (Athey-Imbens Theorems 5.1-5.3 influence functions, panel 5.5-5.7, Appendix B covariances; needs the footnote-31 boundary density estimator - note the review's suspected half-range/midpoint typo). Bootstrap is the shipped inference. | `diff_diff/changes_in_changes.py` | #682 | Low |
| Staggered/multi-period distributional DiD (Athey-Imbens Section 6 / Ciaccio arXiv:2408.01208v2; `ecic` is the staggered event-study CiC lineage - a distinct method from Ciaccio's copula approach, do not conflate). Reviewed: `docs/methodology/papers/ciaccio-2024-review.md`; ROADMAP row is reviewed-deferred pending demand. | `diff_diff/changes_in_changes.py` | #682 | Low |
| ChangesInChanges treatment-on-the-controls (Athey-Imbens Theorem 3.2: group-label exchange + negation; no qte equivalent to anchor parity). | `diff_diff/changes_in_changes.py` | #682 | Low |
| MMM interop: event-study temporal lift export (time-resolved effect path -> per-period calibration rows) - no consumer exists yet; PyMC-Marketing's temporal/adstock lift calibration is an open upstream feature request (pymc-marketing #2433) and the current `add_lift_test_measurements` API consumes only scalar lifts. Revisit when upstream ships. | `diff_diff/mmm.py` | mmm-interop | Low |
| MMM interop: Robyn `calibration_input` emitter - the only functional consumer is R Robyn (the official `robynpy` Python beta ships calibration data classes but its public API hardcodes `calibration_input=None` and feeds a constant-zero calibration objective; verified 2026-07-18). Revisit if Python Robyn revives. | `diff_diff/mmm.py` | mmm-interop | Low |
| MMM interop: Meridian GeoX emitter - GeoX ("converts experiment results into priors", SDiD on roadmap) is announced but unshipped (waitlist-only, no repo/spec as of 2026-07-18). Revisit at GeoX GA. | `diff_diff/mmm.py` | mmm-interop | Low |
| Rust-backend CR2 Bell-McCaffrey port (`return_dof` in the Rust vcov dispatch + CR2 algebra) — **premise re-scoped 2026-07-09**: the scores-based DOF + low-rank factored `A_g` changes made the NumPy CR2-BM path BLAS-bound (`O(n_g k²)` per cluster; 4.1s→38ms at n=100k/k=40), so a Rust port buys ~nothing and adds a parity surface. Revisit only if profiling shows CR2-BM hot again. | `rust/src/linalg.rs` | — | Low |
| Clustered-CR1 inference df **default flip to `"cluster"` (G−1) at v4** — the opt-in `df_convention=` knob landed 2026-07 (DiD/TWFE/MPD + LinearRegression; REGISTRY §TwoWayFixedEffects deviation note); the remaining work is the major-version default change (moves every clustered p-value/CI) + migration note + flipping `TestDfConvention`/`test_moderate_t_pins_residual_df_convention` expectations. Also evaluate extending the knob to standalone estimators with CR1-t inference at that time. Lifecycle tracked in docs/v4-deprecations.yaml (M-004..M-006). | `diff_diff/linalg.py::LinearRegression`, `diff_diff/estimators.py`, `diff_diff/twfe.py` | — | Medium |
| CallawaySantAnna **unbalanced-panel R parity — LANDED** via `allow_unbalanced_panel=True` (matches R `did::att_gt(allow_unbalanced_panel=TRUE)` / `DRDID::reg_did_rc`: ATT bit-exact on cells AND dynamic aggregation via fixed unit-cohort-mass `pg` + a per-unit WIF; SE up to the documented CR1 `sqrt(G/(G-1))` factor). The earlier "weighting" framing was a mis-diagnosis — on unbalanced panels the dominant divergence from R is the *estimator* (within-cell differencing vs RC-on-pooled-obs), not only the weighting; both are resolved by the flag. The DEFAULT path keeps within-cell differencing as a documented design choice and now emits a `UserWarning` on unbalanced input (no-silent-failures). **Remaining deferred:** `survey_design=` × `allow_unbalanced_panel=` (per-obs vs per-unit weight resolution — currently fail-closed `NotImplementedError`); and covariate / ipw / dr × the flag R-parity verification (the RC path supports them; the committed golden covers `reg` no-cov). | `staggered.py`, `staggered_aggregation.py` | SE-audit D3 | Low |
Expand Down
9 changes: 9 additions & 0 deletions diff_diff/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,11 @@
)
from diff_diff.lpdid import LPDiD
from diff_diff.lpdid_results import LPDiDResults
from diff_diff.mmm import (
MeridianROIPrior,
to_meridian_roi_prior,
to_pymc_marketing_lift_test,
)
from diff_diff.power import (
PowerAnalysis,
PowerResults,
Expand Down Expand Up @@ -566,6 +571,10 @@
"Alert",
"OutcomeShape",
"TreatmentDoseShape",
# MMM calibration export (interop)
"to_pymc_marketing_lift_test",
"to_meridian_roi_prior",
"MeridianROIPrior",
# LLM guide accessor
"get_llm_guide",
]
Expand Down
1 change: 1 addition & 0 deletions diff_diff/estimators.py
Original file line number Diff line number Diff line change
Expand Up @@ -816,6 +816,7 @@ def _refit_did_absorb(w_r):
conley_lag_cutoff=(self.conley_lag_cutoff if _fit_vcov_type == "conley" else None),
df_convention=self.df_convention,
inference_df=_inference_df_used,
estimator=type(self).__name__,
)

self._coefficients = coefficients
Expand Down
102 changes: 102 additions & 0 deletions diff_diff/guides/llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2731,3 +2731,105 @@ cross-period aggregations are enumerated in
current release; see that document for phrasing rules, the
no-traffic-light decision, unit-translation policy, and schema
stability policy.

## MMM Calibration Export

Interop exporters converting experiment results into Marketing Mix Model
(MMM) calibration inputs. Pure numpy/pandas — no MMM package is imported;
outputs are a plain DataFrame / frozen dataclass. DERIVED export
(rescaling the result's own att/se) is restricted to the verified
estimand allowlist below; EXPLICIT export (the Meridian
incremental_outcome/incremental_outcome_se pair) accepts any result
object. DiDResults additionally requires fit-stamped provenance
`estimator == "DifferenceInDifferences"` — the class is shared with
TwoWayFixedEffects, whose staggered-data coefficient carries
negative-weight bias, and hand-built results without provenance fail
closed. Experiment context an estimation result cannot know — channel,
spend levels, treated-unit count, period count — is always explicit user
input. Treated counts are NEVER inferred from result metadata:
`n_treated` on several result classes counts treated rows
(unit-periods), not units.

```python
from diff_diff import (
SyntheticDiD,
to_pymc_marketing_lift_test,
to_meridian_roi_prior,
)

result = SyntheticDiD().fit(panel, outcome="revenue", treatment="treated",
unit="geo", time="week")

# PyMC-Marketing (also prophetverse): lift-test DataFrame for
# MMM.add_lift_test_measurements — columns [channel, *dims, x, delta_x,
# delta_y, sigma], all in original data units.
df_lift = to_pymc_marketing_lift_test(
result,
channel="tv", # must match the MMM's channel_columns
x=50_000.0, # baseline per-period spend
delta_x=20_000.0, # spend change during the test (nonzero;
# negative for go-dark tests)
dims={"geo": "US-CA"}, # optional model-dim columns (geo-level MMM)
aggregate="mean", # "mean" per-period lift | "sum" cumulative
# (requires n_periods AND cumulative x/delta_x)
on_wrong_sign="raise", # "raise" | "drop" | "keep" — PyMC-Marketing
# hard-rejects sign(delta_y) != sign(delta_x)
)

# Google Meridian: lognormal ROI prior parameters for roi_m calibration.
prior = to_meridian_roi_prior(
result, # or a sequence — pooled with spend weights
spend=200_000.0, # incremental spend during the experiment
n_units=30, # treated geos (explicit, from your design)
n_periods=8, # post-treatment periods the ATT spans
se_widening=1.5, # >1 encodes experiment-to-MMM skepticism
)
prior.roi_mean, prior.roi_sd # pooled ROI moments
prior.mu, prior.sigma # LogNormal params (Google's closed form,
# matches lognormal_dist_from_mean_std)
prior.to_dict() # JSON-ready
print(prior.to_code( # PriorDistribution snippet (pinned to
channel="tv", # Meridian 1.7.0). roi_m is PER-CHANNEL and a
media_channels=["search", "tv"], # scalar broadcasts to every channel,
)) # so channel scope is required: vector prior in
# model channel order (others keep Meridian's
# default LogNormal(0.2, 0.9)), or
# single_channel=True for 1-channel models.
```

Key semantics:

- Lift frame `delta_y = att * (n_units or 1)` under `aggregate="mean"`
(leave `n_units=None` for geo-level MMM rows; pass the treated-geo count
when calibrating a national MMM from a geo experiment), additionally
`* n_periods` under `aggregate="sum"`; `sigma` scales identically.
UNIT CONTRACT: x/delta_x/delta_y/sigma must share one aggregation unit
(one MMM row's unit) - under "sum" the x/delta_x you pass must be
cumulative spends over the experiment window, valid only when one MMM
row spans that window. Scaling assumes additive level outcomes.
ESTIMAND ALLOWLIST (exact type-name match; subclasses/unknown classes
FAIL CLOSED): deterministic rescaling is only valid for count-weighted
level-scale headlines - DiDResults/SyntheticDiDResults/
MultiPeriodDiDResults (n_units/n_periods common-window products) and
CallawaySantAnnaResults (ONLY aggregate="sum" +
unit_periods=<total treated unit-periods of the aggregated cells>,
sum_g n_g * T_g; its overall ATT is always the simple aggregation and
cohort durations are unequal, so n_units x n_periods and "mean" are
rejected). Every other estimator (StackedDiD equal-horizon weights,
dose-normalized HAD WAS, etc.) is rejected on the lift path and exports
only via incremental_outcome= + incremental_outcome_se= on the Meridian
path (both required together; att/se are not read; the SE is never
inferred from a point-estimate ratio).
- Meridian per-experiment `roi = att * n_units * n_periods / spend` (or
an explicit `incremental_outcome=` override); multi-experiment pooling
is spend-weighted with `roi_sd = sqrt(sum((w_i * sd_i)^2))` (independence
assumed — widen via `se_widening` for correlated experiments).
- Guards: non-finite `att` / non-positive `se` raise (bootstrap NaN
contract upstream); wrong-signed lift rows raise by default with
`NonMonotonicError` guidance; `x + delta_x < 0` raises (post-test spend
cannot be negative); `n_units`/`n_periods` must be positive integers;
a non-positive pooled ROI mean raises
(lognormal positivity) pointing to pooling / wider priors / Meridian's
contribution parameterizations; `on_wrong_sign="drop"` raises rather
than emit an empty frame; array-valued `att` (event-study results) is
rejected — aggregate to a headline ATT first.
1 change: 1 addition & 0 deletions diff_diff/guides/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ Full practitioner guide: call `diff_diff.get_llm_guide("practitioner")`
- [Honest DiD](https://diff-diff.readthedocs.io/en/stable/api/honest_did.html): Rambachan & Roth (2023) sensitivity analysis — robust CI under parallel trends violations, breakdown values
- [Pre-Trends Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/pretrends.html): Roth (2022) Section II.A-B no-individually-significant (NIS) box-probability pretest power + minimum detectable violation; `pretest_form='nis'` (default) implements the paper's primary form, `pretest_form='wald'` retained as paper-supported alternative (Propositions 1+3+4 all apply); linear-violation MDV in Roth's γ units when relative-time labels are threaded through `fit()`; full Σ_22 routing on non-bootstrap CallawaySantAnna and SunAbraham adapters
- [Power Analysis](https://diff-diff.readthedocs.io/en/stable/api/power.html): Analytical and simulation-based power analysis — MDE, sample size, power curves for study design
- [MMM Calibration Export](https://diff-diff.readthedocs.io/en/stable/api/mmm.html): Convert experiment results into Marketing Mix Model calibration inputs — `to_pymc_marketing_lift_test()` emits the PyMC-Marketing/prophetverse lift-test DataFrame (`channel`/dims/`x`/`delta_x`/`delta_y`/`sigma` for `MMM.add_lift_test_measurements`), `to_meridian_roi_prior()` emits Google Meridian lognormal ROI prior parameters (spend-weighted multi-experiment pooling, `se_widening` transferability knob, channel-scoped `.to_code()` snippet - `roi_m` is per-channel, so it requires `channel=`+`media_channels=` or `single_channel=True`). Deterministic scaling is restricted to a verified count-weighted allowlist (DiD/SyntheticDiD/MultiPeriodDiD common-window products; CallawaySantAnna via exact `unit_periods=`; exact type-name match, subclasses fail closed; DiDResults needs fit-stamped `estimator == "DifferenceInDifferences"` provenance - TWFE returns the same class and is rejected); all other result types export only via the explicit `incremental_outcome=`+`incremental_outcome_se=` pair on the Meridian path. Experiment context is always explicit - treated counts are never inferred from result metadata
- Conley spatial HAC SE (`vcov_type="conley"`) on cross-sectional `LinearRegression` / `compute_robust_vcov` PLUS panel `DifferenceInDifferences` / `MultiPeriodDiD` / `TwoWayFixedEffects` (with `conley_lag_cutoff=<int>` for within-unit Bartlett temporal HAC) — Conley (1999) spatial-correlation-aware SEs with haversine/euclidean/callable distance metric and Bartlett/uniform spatial kernel; panel path uses the R `conleyreg`-form block-decomposed sandwich (within-period spatial + within-unit Bartlett serial, same-time excluded); parity vs R `conleyreg` (Düsterhöft 2021) on cross-sectional AND panel `lag_cutoff > 0` fixtures. Combining with explicit `cluster=<col>` applies the combined spatial + cluster product kernel `K_total[i,j] = K_space · 1{c_i = c_j}` (cluster must be constant within each unit across periods on the panel path; validator-enforced). DiD takes `unit=<col>` as a fit-time kwarg when `vcov_type="conley"` (not on `__init__`). Sparse k-d-tree fast path auto-activates for `n > 5_000` with bartlett kernel + haversine/euclidean metric

## Tutorials
Expand Down
Loading
Loading