feat: MMM calibration export (diff_diff.mmm) - PyMC-Marketing lift tests + Meridian ROI priors

[igerber]

Summary

• New interop module diff_diff/mmm.py converting DiD experiment results into MMM calibration inputs, with zero MMM-package dependency (pure numpy/pandas; emits plain DataFrames/dataclasses)
• to_pymc_marketing_lift_test(): emits the lift-test DataFrame consumed by PyMC-Marketing's MMM.add_lift_test_measurements (channel/dims/x/delta_x/delta_y/sigma), with an explicit four-column unit contract (aggregate="mean"|"sum"), n_units= geo-to-national scaling, on_wrong_sign={"raise","drop","keep"} policy mirroring the upstream monotonicity check, an x + delta_x >= 0 post-test-spend guard, and reserved-column dims validation; schema is prophetverse-compatible
• to_meridian_roi_prior(): emits Google Meridian lognormal ROI prior parameters (MeridianROIPrior) via Google's closed form, with spend-weighted multi-experiment pooling (per the Meridian FAQ recommendation, citing sec 3.4 of Google's MMM calibration whitepaper), an se_widening= transferability knob, and a channel-scoped .to_code() snippet helper (roi_m is per-channel in Meridian, so vector priors require the model channel order; scalar output needs explicit single_channel=True)
• Duck-typed against the flat att/se surface every headline results class exposes; treated counts are never inferred from result metadata (n_treated is a row count on several classes); n_units/n_periods must be positive integers
• Docs: new REGISTRY.md "MMM Calibration Export (interop)" section, docs/api/mmm.rst, llms.txt + llms-full.txt sections, README one-liner, references.rst entries, doc-deps.yaml entry, CHANGELOG, TODO.md follow-up rows for documented deferrals (Meridian roi_calibration_period mask, event-study temporal lift export, Robyn, GeoX)

Methodology references (required if estimator / math changes)

• Method name(s): MMM lift-test calibration export (PyMC-Marketing schema); Meridian lognormal ROI prior construction with spend-weighted experiment pooling. Interop only - no estimator or inference change.
• Paper / source link(s): PyMC-Marketing lift-test calibration docs (https://www.pymc-marketing.io/en/stable/notebooks/mmm/mmm_lift_test.html); Google Meridian "Set custom prior distributions using past experiments" (https://developers.google.com/meridian/docs/advanced-modeling/set-custom-priors-past-experiments) and meridian/model/prior_distribution.py at 1.7.0 (lognormal closed form + per-channel roi_m batch shape + default LogNormal(0.2, 0.9)); Meridian FAQ (spend-weighted average ROI for multiple experiments, citing sec 3.4 of Google's MMM calibration whitepaper); Zhou, Choe & Hetrakul (2023), Meta Marketing Science (calibration motivation)
• Any intentional deviations from the source (and why): Pooled roi_sd = sqrt(sum((w_i * sd_i)^2)) treats experiments as independent - this uncertainty propagation is this library's policy (the FAQ prescribes only the spend-weighted mean); documented with a **Note:** in REGISTRY.md, with se_widening= as the correlated-experiments mitigation.

Validation

• Tests added/updated: tests/test_mmm.py (new, 51 behavioral tests): exact schema emission, hand-computed scaling for mean/sum/n_units, wrong-sign policies incl. drop-all-raises, go-dark boundary (x + delta_x == 0 passes, below raises), NaN/zero-SE rejection, broadcasting/empty-sequence/reserved-dims errors, scipy lognormal moment round-trip (1e-12), spend-weighted pooling math, channel-scope regression for .to_code() (experiment prior lands only in its channel slot), units-not-rows pin (no silent n_treated fallback), and cross-estimator duck-typing contract (DifferenceInDifferences / SyntheticDiD / MultiPeriodDiD / CallawaySantAnna)
• Backtest / simulation / notebook evidence (if applicable): End-to-end smoke against real pymc-marketing 0.19.4 in an isolated venv - the emitted frame was accepted by MMM.add_lift_test_measurements into the model graph, and a wrong-signed row reproduced upstream NonMonotonicError; mu/sigma verified bit-identical to Meridian's lognormal_dist_from_mean_std over 200 random draws. Tutorial notebook follows in PR-B.

Security / privacy

• Confirm no secrets/PII in this PR: Yes

---

🤖 Generated with Claude Code

https://claude.ai/code/session_014wAAYSVo1Yf3XMcxNMrqsj

[github-actions]

Overall Assessment

⚠️ Needs changes — one unmitigated P1 anti-pattern in the new test helper. I did not find a methodology mismatch in the new MMM exporters.

Executive Summary

• Affected methods: to_pymc_marketing_lift_test() and to_meridian_roi_prior() in diff_diff/mmm.py.
• Methodology matches the new REGISTRY.md entry and the cited PyMC-Marketing / Meridian contracts: lift-test schema, sign monotonicity, lognormal ROI prior construction, and explicit unit/spend inputs are consistent.
• Documented deviations are correctly treated as non-blocking: independent-experiment pooled SD is labeled with a **Note:**, and deferred Meridian roi_calibration_period / temporal export work is tracked in TODO.md.
• Blocking issue: tests/test_mmm.py::make_result() computes t_stat, p_value, and CI inline instead of using safe_inference(), violating the project’s known anti-pattern rule.
• I could not run tests in this sandbox because pytest and numpy are not installed.

Methodology

P3 — Documented Meridian pooled-SD policy

Impact: to_meridian_roi_prior() pools experiment means with spend weights and uses sqrt(sum((w_i * sd_i)^2)) for pooled SD. Meridian’s FAQ documents spend-weighted average ROI, while the SD propagation is a library policy. This is explicitly documented in REGISTRY.md with a **Note:**, so it is not a defect. See diff_diff/mmm.py:L580-L596 and docs/methodology/REGISTRY.md:L5620-L5626.

Concrete fix: None required.

No unmitigated methodology findings

PyMC-Marketing’s lift-test API consumes channel, x, delta_x, delta_y, and sigma, evaluates saturation(x + delta_x) - saturation(x), and enforces monotonic sign consistency; the exporter mirrors that contract with reserved-column validation, sign policy, and x + delta_x >= 0 guard. Meridian documents ROI priors as lognormal distributions from point estimate / standard error and defines roi_m defaults / scalar broadcasting; the exporter’s lognormal conversion and channel-scoped .to_code() behavior match that source contract. (pymc-marketing.io)

Code Quality

P1 — Inline inference computation in new test helper

Location: tests/test_mmm.py:L29-L36

Impact: The new helper constructs DiDResults with t_stat=att / se, a hard-coded p_value, and a manually computed CI. The review rules explicitly classify new inline inference computation as P1 because it bypasses the project-wide safe_inference() all-or-nothing NaN contract. This is test code, but it still creates result objects with statistical fields that can become inconsistent as tests evolve.

Concrete fix: import safe_inference and build the helper fields from it:

from diff_diff.utils import safe_inference

t_stat, p_value, conf_int = safe_inference(att, se)
return DiDResults(
    att=att,
    se=se,
    t_stat=t_stat,
    p_value=p_value,
    conf_int=conf_int,
    ...
)

Performance

No findings. The exporters are small linear passes over experiment rows; no obvious performance issue is introduced.

Maintainability

No blocking findings. The new module is cohesive, has explicit validation helpers, and keeps MMM package dependencies out of runtime imports.

Tech Debt

P3 — Deferred MMM interop items are properly tracked

Impact: Missing Meridian roi_calibration_period mask building, temporal lift export, Robyn export, and GeoX export are documented in TODO.md, so they are not blockers under the deferred-work policy. See TODO.md:L52-L53 and TODO.md:L123-L125.

Concrete fix: None required for this PR.

Security

No findings. I did not see secrets, credentials, or unsafe external execution paths in the changed code.

Documentation/Tests

P3 — Meridian .to_code() snippet is TensorFlow-flavored only

Location: diff_diff/mmm.py:L59-L85

Impact: Meridian docs note that JAX users need tfp.substrates.jax imports. The generated snippet always imports tensorflow_probability as tfp, which is fine as a TensorFlow snippet but may be less helpful for JAX-backed Meridian users. (developers.google.com)

Concrete fix: Optional: add a backend="tensorflow"|"jax" argument to .to_code() or document that the snippet is TensorFlow-style and must be adapted for JAX.

Test execution

I attempted to run PYTHONDONTWRITEBYTECODE=1 pytest -q tests/test_mmm.py -p no:cacheprovider, but pytest is not installed. A simple import smoke also failed because numpy is not installed in this sandbox.

Path to Approval

1. Replace the inline inference fields in tests/test_mmm.py::make_result() with safe_inference(att, se).
2. Re-run tests/test_mmm.py in an environment with the project test dependencies installed.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: ad675abfab638760a6faeb674f1becd61ead2702

---

Overall Assessment

✅ Looks good — no unmitigated P0/P1 findings.

Executive Summary

• Re-review finding resolved: tests/test_mmm.py::make_result() now uses safe_inference() for t_stat, p_value, and CI.
• Affected methods: to_pymc_marketing_lift_test() and to_meridian_roi_prior() in diff_diff/mmm.py.
• Methodology matches the new Registry entry and the cited PyMC-Marketing / Meridian contracts for lift-test shape, saturation-difference semantics, lognormal ROI priors, and explicit unit/spend inputs. (pymc-marketing.io)
• The Meridian pooled-SD policy is a documented library deviation, so it is P3 informational, not a defect.
• I could not run tests here because pytest and numpy are not installed in the sandbox.

Methodology

P3 — Documented Meridian pooled-SD policy

Severity: P3 informational

Impact: to_meridian_roi_prior() uses spend-weighted ROI pooling and sqrt(sum((w_i * sd_i)^2)) for pooled SD. The Meridian FAQ recommends spend-weighted average ROI and notes further weighting may be warranted when SEs differ; the PR’s SD propagation is explicitly documented as a library policy in the Registry. See diff_diff/mmm.py:L588-L604, docs/methodology/REGISTRY.md:L5620-L5626, and TODO.md:L52-L53. (developers.google.com)

Concrete fix: None required.

No unmitigated methodology findings

PyMC-Marketing documents a lift-test frame with channel, x, delta_x, delta_y, and sigma, and scores the empirical lift against saturation(x + delta_x) - saturation(x). The exporter emits that schema and applies the Registry’s scaling/sign/empty-output guards. Meridian documents constructing ROI priors from experiment point estimates and standard errors via lognormal_dist_from_mean_std, and its default-prior docs confirm scalar prior broadcasting plus default roi_m = LogNormal(0.2, 0.9); .to_code() avoids accidental scalar broadcast by requiring explicit channel scope. (pymc-marketing.io)

Code Quality

No findings.

Re-review note: the previous P1 inline inference anti-pattern is resolved at tests/test_mmm.py:L27-L38.

Performance

No findings.

The exporters are linear over the result rows and do not introduce heavy runtime dependencies.

Maintainability

No findings.

The new module is cohesive and keeps PyMC-Marketing/Meridian as output contracts rather than runtime dependencies.

Tech Debt

P3 — Deferred MMM interop follow-ups are tracked

Severity: P3 informational

Impact: Meridian roi_calibration_period, event-study temporal lift export, Robyn export, and GeoX export are deferred but tracked in TODO.md, so they are not blockers under the review policy. See TODO.md:L52-L53 and TODO.md:L123-L125.

Concrete fix: None required for this PR.

Security

No findings.

I did not see secrets, credential handling, unsafe subprocess execution, or network calls in the changed code.

Documentation/Tests

No blocking findings.

The new tests cover schema emission, scaling, wrong-sign policy, empty/drop-all handling, NaN/SE rejection, count handling, lognormal moment round-trip, pooling, channel-scoped code generation, and cross-estimator duck typing.

Test execution attempted:
PYTHONDONTWRITEBYTECODE=1 pytest -q tests/test_mmm.py -p no:cacheprovider

Result: not run because pytest is not installed. A Python import smoke also failed because numpy is not installed.