From 61a8e6f1447935ba0f09ea154e940c870e9505b6 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 13:37:35 +1000 Subject: [PATCH 1/2] Reorganise the migration by consuming series, and write down two repoint rules The plan's wave structure grouped the remaining 31 datasets by hosting pattern. Grouping them by consuming series turns out to fit the work far better, because apart from one pairing each series now owns its own data: 16 of the 31 are multi-consumer and every one of them is intro + wasm. The last cross-series dataset was the P2 pandas_panel trio, already done. So PLAN.md gains a tracks table (A intro+wasm, B python.myst, C advanced, D programming, E dynamic, X orphans, Y infra) alongside the phases, which stay as the machinery each track passes through. Tracks A-D are mutually independent; the whole programme has only four real dependencies and they are named. Two ordering rules are recorded in both files, learned from reading what the repos actually do rather than from a failure: lecture-wasm fetches lecture-python-intro's committed blobs by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. The standing "delete the copy in the same repoint PR" rule holds only where no sibling reads it. The strict audit has no green state for a partially-repointed dataset -- pending/landed fails once any consumer reads data-lectures, repointed/final fails while any consumer still does not. That is deliberate, but it means all consumers of a dataset move together or the drift alarm fires in the gap. Also: licensing no longer gates migration. Inherited data migrates with its licence recorded as found and anything needing further thought is tracked in #35; the gate is the promotion of data.quantecon.org as a public open-data host, not each file's move. Phase 4's CORS box is ticked -- the Pages domain returns access-control-allow-origin: * today, so pyodide/JupyterLite does not wait on DNS. And AGENTS.md's stale "mid-transition to a flat tree" framing and 9-dataset repo map are corrected to what the tree actually holds. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 23 ++++++++++++++++++----- PLAN.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 70 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index aae1bdb..8fc4f75 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,13 +4,13 @@ Guidance for coding agents (and humans) making changes in this repository. Read ## What this repo is -The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). +The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. The published tree is **flat** (`lectures/`, since 2026-07-16) and live on GitHub Pages; the remaining transition is the custom domain — files are served at `quantecon.github.io/data-lectures/lectures/` today and will move to `https://data.quantecon.org/lectures/` once DNS is resolved (PLAN Phase 4, [#15](https://github.com/QuantEcon/data-lectures/issues/15)). The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). ## Rules ### Layout -- Do **not** add new consumer-keyed directories (no `lecture-xyz/` folders). New datasets go in the flat published tree; if the restructure (PLAN Phase 2) hasn't landed yet, put new files where the pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)) is landing them and note it in the PR. +- Do **not** add new consumer-keyed directories (no `lecture-xyz/` folders). New datasets go directly in the flat published tree, `lectures/`, with their sidecar manifest beside them. - No folder may imply ownership by a lecture series — any lecture can consume any file. ### Every dataset needs a class and a manifest @@ -36,6 +36,15 @@ The Feb 2025 migration left files that cannot fully satisfy the rules above. The - **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`). - **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. +### Repointing a lecture — two ordering traps + +Both are cheap to follow and expensive to discover. `PLAN.md` carries the reasoning and the current counts. + +- **Never delete a file a sibling repo reads.** `lecture-wasm` fetches `lecture-python-intro`'s *committed blobs* by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set. +- **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. + +Cross-repo repoints are worked from [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) — same branch name in each repo, one PR per repo, no aggregate PR. + ### Corrections vs vintages - **Corrections** (bad parse, wrong units, corrupt rows): fix **in place**, same filename — every consumer should get the fix. Use the manifest's `consumers` list to know which lectures to rebuild/review. @@ -76,7 +85,9 @@ Because this repo is a **stability cache, not a content-distribution host** (see For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a **known yes, recorded once per source** — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's `redistribution` field as a **cheap binary gate** (`permitted` / `restricted`): a fast `permitted` for public statistics agencies, `restricted` blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file. -One exception: a `restricted` file the lectures have already served publicly may be cached here if it is marked `redistribution: restricted` with a `note` and logged for licence review in the migration-licensing tracker ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) — resolve it (permission, an open replacement, or removal) before `data.quantecon.org` is promoted as a public open-data host. +**Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). A file the lectures have **already served publicly** migrates here with its licence recorded *as found* — including `redistribution: restricted` and a null licence name where that is the honest answer — and is logged in the inventory ([#35](https://github.com/QuantEcon/data-lectures/issues/35), feeding [workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) with a `note`. Resolve it (permission, an open replacement, or removal) before `data.quantecon.org` is promoted as a public open-data host: **that promotion is the gate, not the file's move.** Rehosting the same bytes with better provenance and an explicit licence field improves on the status quo, so a licence question is never a reason to stall a migration. + +This covers **inherited** data only. A genuinely new dataset — one with no prior life in a lecture repo — still has its licence established *before* it lands, as the P5 additions all did. ### The audit dashboard stays truthful @@ -96,8 +107,10 @@ The generated dashboard (`scripts/build_audit.py`, [#20](https://github.com/Quan ## Repo map ``` -lectures/ # the published tree — flat, served at data.quantecon.org/lectures/ - # 9 datasets + business_cycle's upstream metadata dumps +lectures/ # the published tree — flat, live on Pages; data.quantecon.org pending + # 19 datasets (10 with manifests; the 8 static intro files + # and business_cycle_data.csv still need theirs) plus + # business_cycle's two upstream metadata dumps (see #13) # manifests live here as sidecars: .yml scripts/ # builders + generators — NOT published business_cycle.py # writes business_cycle_data.csv into lectures/ diff --git a/PLAN.md b/PLAN.md index 1708f84..177502a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,6 +1,8 @@ # PLAN — `data-lectures` (formerly `QuantEcon/data`) -**Status:** active roadmap (last updated 2026-07-17) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on +**Status:** active roadmap (last updated 2026-08-06) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on + +**Where the numbers stand (audit dashboard, 2026-08-06):** 10 of 41 static datasets migrated and repointed, 31 to go; 22 lectures still fetch live API data; 35 committed orphans; 0 legacy-repo references; 5 URL forms in use. This repository is being shaped into the **single canonical repository for data consumed by the QuantEcon lecture series**, referenced by stable URLs and documented in the manual. @@ -38,6 +40,50 @@ This repository is being shaped into the **single canonical repository for data - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary** - Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting +## Repoint rules + +Two rules learned the hard way. Both are about *ordering*, both are cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. + +### 1. Repoint a sibling reader before deleting the file it reads + +`lecture-wasm` mirrors `lecture-python-intro`'s sources and fetches intro's **committed blobs** by URL — e.g. `long_run_growth.md` reads `raw.githubusercontent.com/QuantEcon/lecture-python-intro/main/lectures/datasets/mpd2020.xlsx`. Deleting intro's copy in the repoint PR therefore 404s the wasm build immediately. + +So the general rule "delete the lecture repo's own copy in the same repoint PR" holds **only** when no sibling reads that copy. Where one does, the sibling's repoint must land **first or together**, and the deletion goes in the same set — never in an earlier PR with the sibling's fix scheduled later. + +This affects every `intro` + `wasm` dataset, which is all 16 of the multi-consumer files below. + +### 2. Repoint every consumer of a dataset together + +The strict audit has **no green state for a partially-repointed dataset**. `scripts/build_audit.py` fails a record marked `pending`/`landed` while any consumer already reads data-lectures, *and* fails one marked `repointed`/`final` while any consumer still does not. That is deliberate — it is what makes the tracker trustworthy — but it means a dataset with two consuming repos cannot be moved one repo at a time without the drift alarm firing in the gap. + +**16 of the 31 remaining datasets have two consuming repos, and every one of them is `lecture-python-intro` + `lecture-wasm`.** There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. + +Practically: one branch name across data-lectures + every consuming repo, PRs opened together, lecture repoints merged first, then the `migration.yml` flip to `repointed` — that last push is what re-runs the audit, and by then reality and the tracker agree. + +## Migration tracks + +The remaining work decomposes by **consuming series** rather than by hosting pattern, because — apart from the `intro`/`wasm` pairing — each series now owns its own data. This is the execution view; the phases below remain the machinery each track passes through. + +| Track | Datasets | Coupling | Blocked on | +| --- | --- | --- | --- | +| **A — `intro` + `wasm`** | 17: the 8 landed intro statics, the 6 `high_dim_data` files, `life-expectancy…`, `usa-gini…`, `graph.txt` | **paired — repoint together, always** | nothing to start; `usa-gini` needs the SCF files first | +| **B — `python.myst`** | 7: `maketable1/2/4.dta`, `fp.dta`, `hansen_singleton_1982/1983_data.csv`, `NEWQDATA.csv` | none | nothing | +| **C — `advanced.myst`** | 6: `dataBHS.mat`, `acs_data_summary.csv`, `bbh` ×2, `fred_data.csv`, `hansen_jagannathan_1991_data.json` | none | nothing (builder recovery is in-wave work, not a gate) | +| **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track | +| **E — dynamic / live-API** | the UNRATE twin, then the 15 incidental API lectures | wasm is the forcing customer | [#14](https://github.com/QuantEcon/data-lectures/issues/14) schema decisions, [#26](https://github.com/QuantEcon/data-lectures/issues/26) fetch layer | +| **X — orphan sweep** | 35 committed orphans across 6 repos | per repo | that repo's repoints landing first | +| **Y — infra / cutover** | DNS → custom domain → interim-to-final URL sweep → QEP | — | an external infra answer on `52.64.86.66` | + +`lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X. + +**Tracks A–D are independent of each other and can run in any order or in parallel.** The only hard dependencies in the whole programme are: `usa-gini-nwealth-tincome-lincome.csv` is built from `SCF_plus_mini.csv` (so it follows the SCF migration inside Track A); Track E's rollout needs its own template proven first; Track X follows its repo's repoints; and Track Y's cutover is last. + +Track Y is the one item with **external lead time** — it waits on whether `52.64.86.66` can be decommissioned, which is an infrastructure answer rather than a migration one. Worth starting that enquiry in parallel with the data work rather than at the end. + +### Where this work happens + +Repoints span data-lectures plus one or two lecture repos and must land together, which is exactly what [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) exists for: all the repos cloned side by side, `bin/foreach` for cross-repo greps and branch creation, and its stated pattern of *same branch in each repo → edit → commit per repo → one PR per repo*. `data-lectures` is already in its manifest. The artifacts still live here — this PLAN, the manifests, `migration.yml` — and every PR still lands in its own repo; the workspace is the bench, not the destination. + ## Phases Ordering note: phases 1–3 and 6 can proceed now; phase 4 needs the DNS question resolved; phase 5 follows layout, **except its go-live guardrails, which must precede the first repoint**; phase 7 needs the sources recorded in phase 6; phase 8 (the pilot) is the first end-to-end pass through phases 2–7's machinery and requires phase 7's byte-compare for the files it touches **plus phase 5's go-live guardrails** — the first repoint turns `raw/main` into a production URL, so the repo must not go live unprotected; phase 9 follows the pilot (interim URL form makes repoints churn-tolerant to start earlier). @@ -72,7 +118,7 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. - [x] GitHub Pages deploy of the published tree, **`lfs: true` at checkout** (else pointer files publish) — landed 2026-07-17 with the audit dashboard (`.github/workflows/audit-dashboard.yml`, [#20](https://github.com/QuantEcon/data-lectures/issues/20)): the default `quantecon.github.io/data-lectures/` site serves the dashboard at `/` and the published tree at `/lectures/`. The custom domain below stays open - [ ] `data.quantecon.org` DNS + custom domain (an old NestJS box on AWS Sydney currently answers this name — investigate before repointing) -- [ ] Verify `access-control-allow-origin: *` on served files (pyodide/JupyterLite, meta#143) +- [x] Verify `access-control-allow-origin: *` on served files (pyodide/JupyterLite, meta#143) — **verified 2026-08-06**: `quantecon.github.io/data-lectures/lectures/lingcod_msy_recovery.csv` returns `access-control-allow-origin: *`. The requirement is met on the default Pages domain today and does **not** wait on the custom domain; re-verify once DNS moves - [ ] Monitor Pages soft limits (~1 GB site, 100 GB/month) ### Phase 5 — Automation (`.github/`) @@ -97,6 +143,8 @@ Full automation: - [ ] Manifest per dataset for the **9** files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below - [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared - [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand + + **Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). Inherited data — anything the lecture repos already serve publicly — migrates with its licence recorded **as found**, including `redistribution: restricted` and `name: null` where that is the honest answer. Moving the same bytes to a canonical host with better provenance and an explicit licence field improves on the status quo, so the migration does not wait on review; what needs further thought is tracked in [#35](https://github.com/QuantEcon/data-lectures/issues/35) with alternatives, and resolved before `data.quantecon.org` is promoted as a public open-data host. That promotion is the gate, not each file's move. This generalises the exception AGENTS.md already carried for `countries.csv`, and applies to **inherited** data only — a genuinely new dataset still establishes its licence before it lands - [x] Keep-or-drop decision for the files with no consumer anywhere — **dropped 2026-07-16** in the Phase 2 restructure, rather than promoting them into the published namespace: - `GDP_per_capita_world_bank.csv` and `Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv` — an org-wide code search returns **zero** references to either, they are freely re-downloadable from the World Bank, and their licence was never established. Rehosting a stale snapshot nobody reads is the opposite of this repo's purpose - `fig_3.ods` — confirmed to carry no provenance the published `.xlsx` lacks: both parse to a single `Sheet1` of identical shape (34×6) and `DataFrame.equals` returns true, so it is a pure format twin @@ -125,8 +173,8 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde ### Phase 9 — Adoption (broad sweep — the step that stalled in Feb 2025) -- [ ] Repoint the remaining consuming lectures as datasets land here (data#4) — mechanical once the pilot proves the convention (~25 files beyond the pilot set) -- [ ] Remove lecture repos' duplicate copies as each repoint merges (tracked with the orphan sweep in meta#337) +- [ ] Repoint the remaining consuming lectures as datasets land here (data#4) — **31 datasets**, organised as tracks A–D above. Mechanical, but see "Repoint rules": repoint all consumers of a dataset together, and never delete a copy a sibling repo reads +- [ ] Remove lecture repos' duplicate copies as each repoint merges (tracked with the orphan sweep in meta#337) — 35 orphans today, Track X. Note the wasm mirror copies are only safe to delete **after** wasm reads data-lectures directly, not before - [ ] Intake rule for migrations: constructed datasets arrive **with their builders**; the 5 known constructed-but-unscripted files (`hansen_jagannathan_1991_data.json`, `fred_data.csv`, the two `bbh` extracts, `acs_data_summary.csv`) need their pipelines recovered or rewritten — recorded as QEP follow-ups per meta#338 - [ ] Graduate the convention to a QEP and merge manual#108, with the remaining sweep as its rollout checklist From a9e04ee1d096777c4272a9757b42e3bfe0375143 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 13:53:19 +1000 Subject: [PATCH 2/2] Settle Phase 3 storage: plain-git published tree, sources/ for builder inputs The LFS requirement turns out to rest on a false premise. Both SCF minis fit plain git -- 31.3 MiB and 72.4 MiB against GitHub's 100 MiB limit -- so the published tree can stay entirely free of LFS objects. That removes the raw-vs-media URL trap for consumers rather than managing it, which is a better outcome than any sequencing rule. One file genuinely needs LFS, and it is not a dataset. SCF_plus.dta is 99.1 MiB, has no lecture consumer, and exists only as input to the SCF builder. It goes to a new unpublished sources/ tree under per-path LFS, with sources/README.md as its audit trail: origin, retrieval date, licence, upstream identifier, sha256, and the consuming builder. Files there get no sidecar manifest because they are not published datasets. Also recorded: generating_mini.md currently reads its input over the network from high_dim_data itself -- pd.read_stata('https://github.com/QuantEcon/high_dim_data/blob/main/SCF_plus/SCF_plus.dta?raw=true') Archiving that repo while the line stands would re-introduce the legacy-repo dependency this project drove to zero, so the builder is repointed at sources/ before archiving. And with nothing published under LFS, the Pages job can check out lfs:false and stop pulling 99 MiB on every dashboard build. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 18 ++++++++++++++---- PLAN.md | 18 +++++++++++++++--- 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8fc4f75..2dc751f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,11 +65,21 @@ When writing or reviewing URLs that fetch from this repo (in docs, tests, or lec - Final form once Pages is live: `https://data.quantecon.org/lectures/` - Never reference a non-default branch in a published URL. -### LFS +### LFS, and `sources/` vs `lectures/` -- LFS is **per-path**, opt-in, large binaries only. Never add a blanket rule like `*.csv filter=lfs`. -- Do not LFS-track an **existing** file until you've confirmed no consumer fetches it via `raw.githubusercontent.com` — converting silently turns their download into a pointer file. -- The Pages deploy workflow must checkout with `lfs: true` or it publishes pointer files. +**The published tree is plain git. Do not put an LFS object in `lectures/`** (settled 2026-08-06, PLAN Phase 3). Every published dataset fits comfortably in plain git — the largest, `SCF_plus_mini_no_weights.csv`, is 72.4 MiB against GitHub's 100 MiB limit. Keeping it that way means **no consumer can ever hit the raw-vs-media trap above**; the hazard is removed rather than managed. + +LFS exists here for one purpose: **upstream inputs that builders consume and no lecture reads**, which live in `sources/` and are never served. + +- `lectures/` — a published dataset. Plain git, sidecar manifest required, its filename is an API. +- `sources/` — a builder input. Per-path LFS, **no** manifest, not served, recorded instead in `sources/README.md` — the audit trail: origin, retrieval date, licence, upstream identifier (DOI where one exists), `sha256`, and the builder that consumes it. + +Rules that still apply: + +- LFS is **per-path**, opt-in, large binaries only. Never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta`. +- Do not LFS-track an **existing** file until you've confirmed no consumer fetches it via `raw.githubusercontent.com` — converting silently turns their download into pointer text. +- A builder must read its input from `sources/`, never over the network from another QuantEcon repo. That is how a retired repo becomes load-bearing again. +- The Pages deploy checks out with `lfs: false` while nothing published is an LFS object. If that ever changes, it must become `lfs: true` or Pages publishes pointer files. ### Dynamic builders diff --git a/PLAN.md b/PLAN.md index 177502a..a9b6cbb 100644 --- a/PLAN.md +++ b/PLAN.md @@ -109,10 +109,22 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. ### Phase 3 — Storage -- [ ] Per-path LFS via `.gitattributes` — large binaries only, small teaching files plain git (data#1; avoid `high_dim_data`'s blanket `*.csv` rule) -- [ ] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints and the branch-only SCF file) +**Settled 2026-08-06 — the published tree stays 100% plain git.** The two files that drove the LFS requirement do not actually need it: `SCF_plus_mini.csv` is 31.3 MiB and `SCF_plus_mini_no_weights.csv` is 72.4 MiB, both comfortably under GitHub's 100 MiB limit. Keeping `lectures/` free of LFS entirely means **no consumer can ever meet the raw-vs-media URL trap** — it removes the hazard rather than managing it, and the sequencing constraint below stops applying to anything served. -**Sequencing constraint:** enabling LFS breaks every `raw.githubusercontent.com` URL for the paths it covers (pointer files). Do not LFS-track existing files until consumers use a URL form that survives it (`github.com/{org}/{repo}/raw/{ref}/…` interim, or Pages final). +Only one file genuinely forces LFS, and it is not a dataset: + +| Path | Contents | Storage | Served? | +| --- | --- | --- | --- | +| `lectures/` | every published dataset, including both SCF minis and the 4 `cross_section` CSVs | **plain git** | yes | +| `sources/` | upstream inputs that builders consume but no lecture reads — `SCF_plus.dta` (99.1 MiB) | **per-path LFS** | **no** | + +- [ ] Add `sources/` for builder inputs, with `sources/README.md` as the **audit trail**: one row per committed file recording where it came from, when, its licence, the upstream identifier (DOI where one exists), its `sha256`, and which builder consumes it. A file in `sources/` is not a published dataset and gets no sidecar manifest — this README is its provenance record +- [ ] Per-path LFS via `.gitattributes`, scoped to `sources/` only — never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta` (data#1) +- [ ] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints) +- [ ] **Repoint `generating_mini.md`'s input URL.** The SCF builder currently reads its source over the network from the repo being retired — `pd.read_stata('https://github.com/QuantEcon/high_dim_data/blob/main/SCF_plus/SCF_plus.dta?raw=true')`. Archiving `high_dim_data` while that line stands re-introduces exactly the legacy-repo dependency this project drove to zero. Point it at `sources/` before archiving +- [ ] Set the Pages job's checkout to `lfs: false` once the above holds — nothing under `lectures/` is an LFS object, so the 99 MiB `.dta` never needs downloading on a dashboard build (it runs on every push to `main` plus weekly) + +**Sequencing constraint** (still applies to anything that *does* enter LFS): enabling LFS breaks every `raw.githubusercontent.com` URL for the paths it covers — those URLs return pointer text, not data, so consumers fail with a confusing parse error rather than a 404. Do not LFS-track an existing file until its consumers use a form that survives it. Keeping the published tree plain-git means no consumer-facing path is ever affected. ### Phase 4 — Publishing