From 24318faf3d56933eefee7f7a5d9b755ca70b5dd4 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 17 Aug 2026 11:43:37 +1000 Subject: [PATCH] =?UTF-8?q?Flip=20wave=20C1=20to=20repointed=20=E2=80=94?= =?UTF-8?q?=20the=20three=20advanced.myst=20datasets=20(Track=20C)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit lecture-python-advanced.myst#372 merged, so advanced.myst now reads all three files from this repo. This closes the red window that opens the moment a repoint lands: build_audit fails `--strict` on any dataset marked `landed` while a consumer already reads data-lectures, so main has been red since that merge. Measured in both directions rather than asserted, which is what makes this the wave's acceptance test rather than bookkeeping. Against post-merge main: `landed` gives exit 1 with three `migration_inconsistencies`, one per file; `repointed` gives exit 0 with all five problem buckets empty. The dry-run has to run against the consuming repo's post-merge main, because build_audit reads `origin/main` through git show and cannot see an unmerged branch. The three consumer notes come out. They were added in #92 to record that the lecture still read its own copies while the dataset sat at `landed` — that is now false, and a note claiming an unrepointed consumer would be exactly the kind of stale record #91 added them to prevent. CATALOG.md regenerates with those three warning markers gone; the three that remain are the unrelated ws#46 readers (caron.npy, nom_balances.npy, us_adult_heights.csv), which are genuinely still on local copies. Verified the edit touched nothing else: for each of the three manifests, the only top-level key differing from main is `consumers`, and `source.note`, `license.note` and `integrity.upstream.note` are byte-identical to main. That check is not decorative — the first attempt at this edit matched `integrity.upstream.note` instead, because both sit at the same indent, and silently deleted the provenance record. Anchor on the `consumers:` block. Ledger is now 36 datasets, all `repointed`, 0 `landed`. NOT done here, and the order matters: advanced.myst has not published since this repoint, so its published notebooks still carry the old URLs. The three local copies stay until it does. That is load-bearing for hansen_jagannathan_1991_data.json, which is fetched over the network at cell-execution time and therefore has no stale-serving grace period — its deletion breaks every published notebook and Colab reader the instant the blob leaves advanced.myst's main. Co-Authored-By: Claude Opus 5 (1M context) --- CATALOG.md | 6 +++--- lectures/bbh_macro_quarterly.csv.yml | 8 ------- lectures/bbh_michigan_monthly.csv.yml | 8 ------- .../hansen_jagannathan_1991_data.json.yml | 7 ------- migration.yml | 21 +++++++++++++------ 5 files changed, 18 insertions(+), 32 deletions(-) diff --git a/CATALOG.md b/CATALOG.md index a6cf2e3..7fa4da3 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -15,8 +15,8 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**SCF_plus_mini_no_weights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini_no_weights.csv)
SCF+ mini, weight-expanded — net wealth and income, 1950-2016 | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 72.4 MB | [lecture-python-intro · mle.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/mle.md)
[lecture-wasm · mle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/mle.md)
[lecture-intro.zh-cn · mle.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/mle.md)
[test-actions-lecture-intro · mle.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/mle.md) | | [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)
Ames, Iowa — residential house sales, 2006-2010 | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md)
[lecture-intro.zh-cn · fitting_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/fitting_distributions.md)
[lecture-intro.zh-cn · observed_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/observed_distributions.md) | | [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)
French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)
[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)
[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md) | -| [**bbh_macro_quarterly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_macro_quarterly.csv)
Bhandari-Borovička-Ho replication — quarterly US macro series for the belief-wedge VAR, 1955Q1-2019Q4 | constructed | [Replication package for "Survey data and subjective beliefs in business cycle models" (Bhandari, Borovička and Ho), file `data input/FRED/data_FRED.xlsx`](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 31.5 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md)
⚠️ Reads its own copy at `lectures/_static/lecture_specific/subjective_beliefs_business_cycles/`, not this file — the repoint PR in lecture-python-advanced.myst follows this one, and migration.yml records the dataset as `landed` until it merges | -| [**bbh_michigan_monthly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_michigan_monthly.csv)
Michigan Surveys of Consumers monthly aggregates and the US unemployment rate, 1978-01 to 2020-03 (BBH replication extract) | constructed | [Bhandari, Borovička and Ho replication package (Zenodo), carrying University of Michigan Surveys of Consumers published aggregates and a US Bureau of Labor Statistics series retrieved via FRED](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ⚠️ restricted | ✅ verified | ✅ committed | 11.9 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md)
⚠️ Reads its own copy at `lectures/_static/lecture_specific/subjective_beliefs_business_cycles/`, not this file — the repoint PR in lecture-python-advanced.myst follows this one, and migration.yml records the dataset as `landed` until it merges | +| [**bbh_macro_quarterly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_macro_quarterly.csv)
Bhandari-Borovička-Ho replication — quarterly US macro series for the belief-wedge VAR, 1955Q1-2019Q4 | constructed | [Replication package for "Survey data and subjective beliefs in business cycle models" (Bhandari, Borovička and Ho), file `data input/FRED/data_FRED.xlsx`](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ✅ permitted | ✅ verified | ✅ committed | 31.5 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) | +| [**bbh_michigan_monthly.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/bbh_michigan_monthly.csv)
Michigan Surveys of Consumers monthly aggregates and the US unemployment rate, 1978-01 to 2020-03 (BBH replication extract) | constructed | [Bhandari, Borovička and Ho replication package (Zenodo), carrying University of Michigan Surveys of Consumers published aggregates and a US Bureau of Labor Statistics series retrieved via FRED](https://doi.org/10.5281/zenodo.10194324) | CC-BY-4.0 | ⚠️ restricted | ✅ verified | ✅ committed | 11.9 KB | [lecture-python-advanced.myst · subjective_beliefs_business_cycles.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/subjective_beliefs_business_cycles.md) | | [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)
French Revolution — monthly specie value of the assignat, 1791-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)
[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)
[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md)
⚠️ Reads a local `datasets/` copy, not this file | | [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)
The Ends of Four Big Inflations — appendix tables, transcribed | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)
[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md)
[lecture-intro.zh-cn · inflation_history.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inflation_history.md)
[test-actions-lecture-intro · inflation_history.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inflation_history.md) | | [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)
World Population Review — Brazilian city populations, 2023 | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | @@ -29,7 +29,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)
Forbes Billionaires — individual net worth | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)
Forbes Global 2000 — firm size measures | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**fp.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fp.dta)
Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates | verbatim | [Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings)](https://www.aeaweb.org/articles?id=10.1257/aer.p20161068) | | ✅ permitted | ✅ verified | n/a (verbatim) | 1000.1 KB | [lecture-python.myst · mle.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/mle.md)
[lecture-python.zh-cn · mle.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/mle.md)
[lecture-stats · mle.md](https://github.com/QuantEcon/lecture-stats/blob/main/lectures/mle.md) | -| [**hansen_jagannathan_1991_data.json**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_jagannathan_1991_data.json)
Hansen-Jagannathan (1991) replication — US asset returns 1891-1986, with annual consumption (three-table bundle) | constructed | [Robert J. Shiller's public data workbooks (chapt26.xlsx, ie_data.xls) and FRED](http://www.econ.yale.edu/~shiller/data.htm) | | ✅ permitted | ✅ verified | ⚠️ unrecovered | 62.0 KB | [lecture-python-advanced.myst · hansen_jagannathan_1991.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/hansen_jagannathan_1991.md)
⚠️ Reads its own copy over an own-repo raw URL (`lecture-python-advanced.myst/.../hansen_jagannathan_1991/`), not this file — the repoint PR follows this one, and migration.yml records the dataset as `landed` until it merges | +| [**hansen_jagannathan_1991_data.json**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_jagannathan_1991_data.json)
Hansen-Jagannathan (1991) replication — US asset returns 1891-1986, with annual consumption (three-table bundle) | constructed | [Robert J. Shiller's public data workbooks (chapt26.xlsx, ie_data.xls) and FRED](http://www.econ.yale.edu/~shiller/data.htm) | | ✅ permitted | ✅ verified | ⚠️ unrecovered | 62.0 KB | [lecture-python-advanced.myst · hansen_jagannathan_1991.md](https://github.com/QuantEcon/lecture-python-advanced.myst/blob/main/lectures/hansen_jagannathan_1991.md) | | [**hansen_singleton_1982_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1982_data.csv)
Hansen-Singleton (1982) replication — monthly US gross real market return and consumption growth, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 11.4 KB | [lecture-python.myst · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1982.md)
[lecture-python.zh-cn · hansen_singleton_1982.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1982.md) | | [**hansen_singleton_1983_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1983_data.csv)
Hansen-Singleton (1983) replication — monthly US returns, consumption and inflation, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 25.5 KB | [lecture-python.myst · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/hansen_singleton_1983.md)
[lecture-python.zh-cn · hansen_singleton_1983.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/hansen_singleton_1983.md) | | [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)
Japan — deaths by single year of age, 2023 | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md)
[lecture-intro.zh-cn · fitting_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/fitting_distributions.md)
[lecture-intro.zh-cn · observed_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/observed_distributions.md) | diff --git a/lectures/bbh_macro_quarterly.csv.yml b/lectures/bbh_macro_quarterly.csv.yml index e478b4c..4e544d5 100644 --- a/lectures/bbh_macro_quarterly.csv.yml +++ b/lectures/bbh_macro_quarterly.csv.yml @@ -231,14 +231,6 @@ schema: consumers: - repo: QuantEcon/lecture-python-advanced.myst file: lectures/subjective_beliefs_business_cycles.md - note: > - Reads its own copy at - `lectures/_static/lecture_specific/subjective_beliefs_business_cycles/`, - not this file — the repoint PR in lecture-python-advanced.myst follows - this one, and migration.yml records the dataset as `landed` until it - merges. Listed now because a correction to these bytes must reach the - lecture regardless of where it currently reads them from - (QuantEcon/data-lectures#91). builder: builders/bbh_macro_quarterly.py builder_status: committed diff --git a/lectures/bbh_michigan_monthly.csv.yml b/lectures/bbh_michigan_monthly.csv.yml index 639774b..23b1663 100644 --- a/lectures/bbh_michigan_monthly.csv.yml +++ b/lectures/bbh_michigan_monthly.csv.yml @@ -243,14 +243,6 @@ schema: consumers: - repo: QuantEcon/lecture-python-advanced.myst file: lectures/subjective_beliefs_business_cycles.md - note: > - Reads its own copy at - `lectures/_static/lecture_specific/subjective_beliefs_business_cycles/`, - not this file — the repoint PR in lecture-python-advanced.myst follows - this one, and migration.yml records the dataset as `landed` until it - merges. Listed now because a correction to these bytes must reach the - lecture regardless of where it currently reads them from - (QuantEcon/data-lectures#91). builder: builders/bbh_michigan_monthly.py builder_status: committed diff --git a/lectures/hansen_jagannathan_1991_data.json.yml b/lectures/hansen_jagannathan_1991_data.json.yml index e205a26..1512dda 100644 --- a/lectures/hansen_jagannathan_1991_data.json.yml +++ b/lectures/hansen_jagannathan_1991_data.json.yml @@ -305,13 +305,6 @@ schema: consumers: - repo: QuantEcon/lecture-python-advanced.myst file: lectures/hansen_jagannathan_1991.md - note: > - Reads its own copy over an own-repo raw URL - (`lecture-python-advanced.myst/.../hansen_jagannathan_1991/`), not this - file — the repoint PR follows this one, and migration.yml records the - dataset as `landed` until it merges. Listed now because a correction to - these bytes must reach the lecture regardless of where it currently reads - them from (QuantEcon/data-lectures#91). # --------------------------------------------------------------------------- # Builder diff --git a/migration.yml b/migration.yml index 26b9e18..ec6fc3d 100644 --- a/migration.yml +++ b/migration.yml @@ -715,22 +715,28 @@ datasets: # whatever is on this repo's main at that moment — data first, always). bbh_macro_quarterly.csv: pilot: C1 - status: landed + status: repointed prior_pattern: local-path landed: pr: QuantEcon/data-lectures#92 date: 2026-08-17 - repoints: [] + repoints: + - repo: QuantEcon/lecture-python-advanced.myst + pr: QuantEcon/lecture-python-advanced.myst#372 + date: 2026-08-17 cutover: null bbh_michigan_monthly.csv: pilot: C1 - status: landed + status: repointed prior_pattern: local-path landed: pr: QuantEcon/data-lectures#92 date: 2026-08-17 - repoints: [] + repoints: + - repo: QuantEcon/lecture-python-advanced.myst + pr: QuantEcon/lecture-python-advanced.myst#372 + date: 2026-08-17 cutover: null # Already read over an own-repo raw URL rather than a local path, so deleting @@ -738,12 +744,15 @@ datasets: # period. Repoint and publish before any deletion. hansen_jagannathan_1991_data.json: pilot: C1 - status: landed + status: repointed prior_pattern: own-repo landed: pr: QuantEcon/data-lectures#92 date: 2026-08-17 - repoints: [] + repoints: + - repo: QuantEcon/lecture-python-advanced.myst + pr: QuantEcon/lecture-python-advanced.myst#372 + date: 2026-08-17 cutover: null # Planned waves that have not landed anything here yet. `datasets` names the