diff --git a/AGENTS.md b/AGENTS.md index aae1bdb..2dc751f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,13 +4,13 @@ Guidance for coding agents (and humans) making changes in this repository. Read ## What this repo is -The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). +The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. The published tree is **flat** (`lectures/`, since 2026-07-16) and live on GitHub Pages; the remaining transition is the custom domain — files are served at `quantecon.github.io/data-lectures/lectures/` today and will move to `https://data.quantecon.org/lectures/` once DNS is resolved (PLAN Phase 4, [#15](https://github.com/QuantEcon/data-lectures/issues/15)). The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). ## Rules ### Layout -- Do **not** add new consumer-keyed directories (no `lecture-xyz/` folders). New datasets go in the flat published tree; if the restructure (PLAN Phase 2) hasn't landed yet, put new files where the pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)) is landing them and note it in the PR. +- Do **not** add new consumer-keyed directories (no `lecture-xyz/` folders). New datasets go directly in the flat published tree, `lectures/`, with their sidecar manifest beside them. - No folder may imply ownership by a lecture series — any lecture can consume any file. ### Every dataset needs a class and a manifest @@ -36,6 +36,15 @@ The Feb 2025 migration left files that cannot fully satisfy the rules above. The - **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`). - **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. +### Repointing a lecture — two ordering traps + +Both are cheap to follow and expensive to discover. `PLAN.md` carries the reasoning and the current counts. + +- **Never delete a file a sibling repo reads.** `lecture-wasm` fetches `lecture-python-intro`'s *committed blobs* by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set. +- **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. + +Cross-repo repoints are worked from [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) — same branch name in each repo, one PR per repo, no aggregate PR. + ### Corrections vs vintages - **Corrections** (bad parse, wrong units, corrupt rows): fix **in place**, same filename — every consumer should get the fix. Use the manifest's `consumers` list to know which lectures to rebuild/review. @@ -56,11 +65,21 @@ When writing or reviewing URLs that fetch from this repo (in docs, tests, or lec - Final form once Pages is live: `https://data.quantecon.org/lectures/` - Never reference a non-default branch in a published URL. -### LFS +### LFS, and `sources/` vs `lectures/` + +**The published tree is plain git. Do not put an LFS object in `lectures/`** (settled 2026-08-06, PLAN Phase 3). Every published dataset fits comfortably in plain git — the largest, `SCF_plus_mini_no_weights.csv`, is 72.4 MiB against GitHub's 100 MiB limit. Keeping it that way means **no consumer can ever hit the raw-vs-media trap above**; the hazard is removed rather than managed. -- LFS is **per-path**, opt-in, large binaries only. Never add a blanket rule like `*.csv filter=lfs`. -- Do not LFS-track an **existing** file until you've confirmed no consumer fetches it via `raw.githubusercontent.com` — converting silently turns their download into a pointer file. -- The Pages deploy workflow must checkout with `lfs: true` or it publishes pointer files. +LFS exists here for one purpose: **upstream inputs that builders consume and no lecture reads**, which live in `sources/` and are never served. + +- `lectures/` — a published dataset. Plain git, sidecar manifest required, its filename is an API. +- `sources/` — a builder input. Per-path LFS, **no** manifest, not served, recorded instead in `sources/README.md` — the audit trail: origin, retrieval date, licence, upstream identifier (DOI where one exists), `sha256`, and the builder that consumes it. + +Rules that still apply: + +- LFS is **per-path**, opt-in, large binaries only. Never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta`. +- Do not LFS-track an **existing** file until you've confirmed no consumer fetches it via `raw.githubusercontent.com` — converting silently turns their download into pointer text. +- A builder must read its input from `sources/`, never over the network from another QuantEcon repo. That is how a retired repo becomes load-bearing again. +- The Pages deploy checks out with `lfs: false` while nothing published is an LFS object. If that ever changes, it must become `lfs: true` or Pages publishes pointer files. ### Dynamic builders @@ -76,7 +95,9 @@ Because this repo is a **stability cache, not a content-distribution host** (see For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a **known yes, recorded once per source** — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's `redistribution` field as a **cheap binary gate** (`permitted` / `restricted`): a fast `permitted` for public statistics agencies, `restricted` blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file. -One exception: a `restricted` file the lectures have already served publicly may be cached here if it is marked `redistribution: restricted` with a `note` and logged for licence review in the migration-licensing tracker ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) — resolve it (permission, an open replacement, or removal) before `data.quantecon.org` is promoted as a public open-data host. +**Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). A file the lectures have **already served publicly** migrates here with its licence recorded *as found* — including `redistribution: restricted` and a null licence name where that is the honest answer — and is logged in the inventory ([#35](https://github.com/QuantEcon/data-lectures/issues/35), feeding [workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) with a `note`. Resolve it (permission, an open replacement, or removal) before `data.quantecon.org` is promoted as a public open-data host: **that promotion is the gate, not the file's move.** Rehosting the same bytes with better provenance and an explicit licence field improves on the status quo, so a licence question is never a reason to stall a migration. + +This covers **inherited** data only. A genuinely new dataset — one with no prior life in a lecture repo — still has its licence established *before* it lands, as the P5 additions all did. ### The audit dashboard stays truthful @@ -96,8 +117,10 @@ The generated dashboard (`scripts/build_audit.py`, [#20](https://github.com/Quan ## Repo map ``` -lectures/ # the published tree — flat, served at data.quantecon.org/lectures/ - # 9 datasets + business_cycle's upstream metadata dumps +lectures/ # the published tree — flat, live on Pages; data.quantecon.org pending + # 19 datasets (10 with manifests; the 8 static intro files + # and business_cycle_data.csv still need theirs) plus + # business_cycle's two upstream metadata dumps (see #13) # manifests live here as sidecars: .yml scripts/ # builders + generators — NOT published business_cycle.py # writes business_cycle_data.csv into lectures/ diff --git a/PLAN.md b/PLAN.md index 1708f84..a9b6cbb 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,6 +1,8 @@ # PLAN — `data-lectures` (formerly `QuantEcon/data`) -**Status:** active roadmap (last updated 2026-07-17) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on +**Status:** active roadmap (last updated 2026-08-06) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on + +**Where the numbers stand (audit dashboard, 2026-08-06):** 10 of 41 static datasets migrated and repointed, 31 to go; 22 lectures still fetch live API data; 35 committed orphans; 0 legacy-repo references; 5 URL forms in use. This repository is being shaped into the **single canonical repository for data consumed by the QuantEcon lecture series**, referenced by stable URLs and documented in the manual. @@ -38,6 +40,50 @@ This repository is being shaped into the **single canonical repository for data - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary** - Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting +## Repoint rules + +Two rules learned the hard way. Both are about *ordering*, both are cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. + +### 1. Repoint a sibling reader before deleting the file it reads + +`lecture-wasm` mirrors `lecture-python-intro`'s sources and fetches intro's **committed blobs** by URL — e.g. `long_run_growth.md` reads `raw.githubusercontent.com/QuantEcon/lecture-python-intro/main/lectures/datasets/mpd2020.xlsx`. Deleting intro's copy in the repoint PR therefore 404s the wasm build immediately. + +So the general rule "delete the lecture repo's own copy in the same repoint PR" holds **only** when no sibling reads that copy. Where one does, the sibling's repoint must land **first or together**, and the deletion goes in the same set — never in an earlier PR with the sibling's fix scheduled later. + +This affects every `intro` + `wasm` dataset, which is all 16 of the multi-consumer files below. + +### 2. Repoint every consumer of a dataset together + +The strict audit has **no green state for a partially-repointed dataset**. `scripts/build_audit.py` fails a record marked `pending`/`landed` while any consumer already reads data-lectures, *and* fails one marked `repointed`/`final` while any consumer still does not. That is deliberate — it is what makes the tracker trustworthy — but it means a dataset with two consuming repos cannot be moved one repo at a time without the drift alarm firing in the gap. + +**16 of the 31 remaining datasets have two consuming repos, and every one of them is `lecture-python-intro` + `lecture-wasm`.** There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. + +Practically: one branch name across data-lectures + every consuming repo, PRs opened together, lecture repoints merged first, then the `migration.yml` flip to `repointed` — that last push is what re-runs the audit, and by then reality and the tracker agree. + +## Migration tracks + +The remaining work decomposes by **consuming series** rather than by hosting pattern, because — apart from the `intro`/`wasm` pairing — each series now owns its own data. This is the execution view; the phases below remain the machinery each track passes through. + +| Track | Datasets | Coupling | Blocked on | +| --- | --- | --- | --- | +| **A — `intro` + `wasm`** | 17: the 8 landed intro statics, the 6 `high_dim_data` files, `life-expectancy…`, `usa-gini…`, `graph.txt` | **paired — repoint together, always** | nothing to start; `usa-gini` needs the SCF files first | +| **B — `python.myst`** | 7: `maketable1/2/4.dta`, `fp.dta`, `hansen_singleton_1982/1983_data.csv`, `NEWQDATA.csv` | none | nothing | +| **C — `advanced.myst`** | 6: `dataBHS.mat`, `acs_data_summary.csv`, `bbh` ×2, `fred_data.csv`, `hansen_jagannathan_1991_data.json` | none | nothing (builder recovery is in-wave work, not a gate) | +| **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track | +| **E — dynamic / live-API** | the UNRATE twin, then the 15 incidental API lectures | wasm is the forcing customer | [#14](https://github.com/QuantEcon/data-lectures/issues/14) schema decisions, [#26](https://github.com/QuantEcon/data-lectures/issues/26) fetch layer | +| **X — orphan sweep** | 35 committed orphans across 6 repos | per repo | that repo's repoints landing first | +| **Y — infra / cutover** | DNS → custom domain → interim-to-final URL sweep → QEP | — | an external infra answer on `52.64.86.66` | + +`lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X. + +**Tracks A–D are independent of each other and can run in any order or in parallel.** The only hard dependencies in the whole programme are: `usa-gini-nwealth-tincome-lincome.csv` is built from `SCF_plus_mini.csv` (so it follows the SCF migration inside Track A); Track E's rollout needs its own template proven first; Track X follows its repo's repoints; and Track Y's cutover is last. + +Track Y is the one item with **external lead time** — it waits on whether `52.64.86.66` can be decommissioned, which is an infrastructure answer rather than a migration one. Worth starting that enquiry in parallel with the data work rather than at the end. + +### Where this work happens + +Repoints span data-lectures plus one or two lecture repos and must land together, which is exactly what [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) exists for: all the repos cloned side by side, `bin/foreach` for cross-repo greps and branch creation, and its stated pattern of *same branch in each repo → edit → commit per repo → one PR per repo*. `data-lectures` is already in its manifest. The artifacts still live here — this PLAN, the manifests, `migration.yml` — and every PR still lands in its own repo; the workspace is the bench, not the destination. + ## Phases Ordering note: phases 1–3 and 6 can proceed now; phase 4 needs the DNS question resolved; phase 5 follows layout, **except its go-live guardrails, which must precede the first repoint**; phase 7 needs the sources recorded in phase 6; phase 8 (the pilot) is the first end-to-end pass through phases 2–7's machinery and requires phase 7's byte-compare for the files it touches **plus phase 5's go-live guardrails** — the first repoint turns `raw/main` into a production URL, so the repo must not go live unprotected; phase 9 follows the pilot (interim URL form makes repoints churn-tolerant to start earlier). @@ -63,16 +109,28 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. ### Phase 3 — Storage -- [ ] Per-path LFS via `.gitattributes` — large binaries only, small teaching files plain git (data#1; avoid `high_dim_data`'s blanket `*.csv` rule) -- [ ] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints and the branch-only SCF file) +**Settled 2026-08-06 — the published tree stays 100% plain git.** The two files that drove the LFS requirement do not actually need it: `SCF_plus_mini.csv` is 31.3 MiB and `SCF_plus_mini_no_weights.csv` is 72.4 MiB, both comfortably under GitHub's 100 MiB limit. Keeping `lectures/` free of LFS entirely means **no consumer can ever meet the raw-vs-media URL trap** — it removes the hazard rather than managing it, and the sequencing constraint below stops applying to anything served. + +Only one file genuinely forces LFS, and it is not a dataset: + +| Path | Contents | Storage | Served? | +| --- | --- | --- | --- | +| `lectures/` | every published dataset, including both SCF minis and the 4 `cross_section` CSVs | **plain git** | yes | +| `sources/` | upstream inputs that builders consume but no lecture reads — `SCF_plus.dta` (99.1 MiB) | **per-path LFS** | **no** | -**Sequencing constraint:** enabling LFS breaks every `raw.githubusercontent.com` URL for the paths it covers (pointer files). Do not LFS-track existing files until consumers use a URL form that survives it (`github.com/{org}/{repo}/raw/{ref}/…` interim, or Pages final). +- [ ] Add `sources/` for builder inputs, with `sources/README.md` as the **audit trail**: one row per committed file recording where it came from, when, its licence, the upstream identifier (DOI where one exists), its `sha256`, and which builder consumes it. A file in `sources/` is not a published dataset and gets no sidecar manifest — this README is its provenance record +- [ ] Per-path LFS via `.gitattributes`, scoped to `sources/` only — never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta` (data#1) +- [ ] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints) +- [ ] **Repoint `generating_mini.md`'s input URL.** The SCF builder currently reads its source over the network from the repo being retired — `pd.read_stata('https://github.com/QuantEcon/high_dim_data/blob/main/SCF_plus/SCF_plus.dta?raw=true')`. Archiving `high_dim_data` while that line stands re-introduces exactly the legacy-repo dependency this project drove to zero. Point it at `sources/` before archiving +- [ ] Set the Pages job's checkout to `lfs: false` once the above holds — nothing under `lectures/` is an LFS object, so the 99 MiB `.dta` never needs downloading on a dashboard build (it runs on every push to `main` plus weekly) + +**Sequencing constraint** (still applies to anything that *does* enter LFS): enabling LFS breaks every `raw.githubusercontent.com` URL for the paths it covers — those URLs return pointer text, not data, so consumers fail with a confusing parse error rather than a 404. Do not LFS-track an existing file until its consumers use a form that survives it. Keeping the published tree plain-git means no consumer-facing path is ever affected. ### Phase 4 — Publishing - [x] GitHub Pages deploy of the published tree, **`lfs: true` at checkout** (else pointer files publish) — landed 2026-07-17 with the audit dashboard (`.github/workflows/audit-dashboard.yml`, [#20](https://github.com/QuantEcon/data-lectures/issues/20)): the default `quantecon.github.io/data-lectures/` site serves the dashboard at `/` and the published tree at `/lectures/`. The custom domain below stays open - [ ] `data.quantecon.org` DNS + custom domain (an old NestJS box on AWS Sydney currently answers this name — investigate before repointing) -- [ ] Verify `access-control-allow-origin: *` on served files (pyodide/JupyterLite, meta#143) +- [x] Verify `access-control-allow-origin: *` on served files (pyodide/JupyterLite, meta#143) — **verified 2026-08-06**: `quantecon.github.io/data-lectures/lectures/lingcod_msy_recovery.csv` returns `access-control-allow-origin: *`. The requirement is met on the default Pages domain today and does **not** wait on the custom domain; re-verify once DNS moves - [ ] Monitor Pages soft limits (~1 GB site, 100 GB/month) ### Phase 5 — Automation (`.github/`) @@ -97,6 +155,8 @@ Full automation: - [ ] Manifest per dataset for the **9** files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below - [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared - [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand + + **Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). Inherited data — anything the lecture repos already serve publicly — migrates with its licence recorded **as found**, including `redistribution: restricted` and `name: null` where that is the honest answer. Moving the same bytes to a canonical host with better provenance and an explicit licence field improves on the status quo, so the migration does not wait on review; what needs further thought is tracked in [#35](https://github.com/QuantEcon/data-lectures/issues/35) with alternatives, and resolved before `data.quantecon.org` is promoted as a public open-data host. That promotion is the gate, not each file's move. This generalises the exception AGENTS.md already carried for `countries.csv`, and applies to **inherited** data only — a genuinely new dataset still establishes its licence before it lands - [x] Keep-or-drop decision for the files with no consumer anywhere — **dropped 2026-07-16** in the Phase 2 restructure, rather than promoting them into the published namespace: - `GDP_per_capita_world_bank.csv` and `Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv` — an org-wide code search returns **zero** references to either, they are freely re-downloadable from the World Bank, and their licence was never established. Rehosting a stale snapshot nobody reads is the opposite of this repo's purpose - `fig_3.ods` — confirmed to carry no provenance the published `.xlsx` lacks: both parse to a single `Sheet1` of identical shape (34×6) and `DataFrame.equals` returns true, so it is a pure format twin @@ -125,8 +185,8 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde ### Phase 9 — Adoption (broad sweep — the step that stalled in Feb 2025) -- [ ] Repoint the remaining consuming lectures as datasets land here (data#4) — mechanical once the pilot proves the convention (~25 files beyond the pilot set) -- [ ] Remove lecture repos' duplicate copies as each repoint merges (tracked with the orphan sweep in meta#337) +- [ ] Repoint the remaining consuming lectures as datasets land here (data#4) — **31 datasets**, organised as tracks A–D above. Mechanical, but see "Repoint rules": repoint all consumers of a dataset together, and never delete a copy a sibling repo reads +- [ ] Remove lecture repos' duplicate copies as each repoint merges (tracked with the orphan sweep in meta#337) — 35 orphans today, Track X. Note the wasm mirror copies are only safe to delete **after** wasm reads data-lectures directly, not before - [ ] Intake rule for migrations: constructed datasets arrive **with their builders**; the 5 known constructed-but-unscripted files (`hansen_jagannathan_1991_data.json`, `fred_data.csv`, the two `bbh` extracts, `acs_data_summary.csv`) need their pipelines recovered or rewritten — recorded as QEP follow-ups per meta#338 - [ ] Graduate the convention to a QEP and merge manual#108, with the remaining sweep as its rollout checklist