diff --git a/README.md b/README.md index b437f14..38f52e4 100644 --- a/README.md +++ b/README.md @@ -20,14 +20,67 @@ A Python package for versioning and monitoring dataset changes throughout the ma pip install dstrack ``` -## Quickstart +Requires Python 3.11 or later. + +## Getting started + +Initialize a store in your project, then take your first snapshot. + +**1. Create a local store** - this adds a `.dstrack/` directory at the current location: + +```bash +dstrack init +``` + +```text +ℹ Generating local store structure at /path/to/.dstrack. +✔ Finished creating local store: /path/to/.dstrack +``` + +**2. Track a dataset** - point `dstrack` at a data file to snapshot it. Given a small +`data.csv`: + +```csv +id,name,value +1,alpha,10.5 +2,beta,20.0 +3,gamma,15.25 +``` ```bash -dstrack +dstrack track data.csv ``` +```text +ℹ Reading data.csv and computing snapshot... +✔ Snapshot written (new dataset, dataset ). +ℹ Stored at /path/to/.dstrack/datasets//snapshots/.json +``` + +**3. Re-track as the data changes** - running `track` again on the same path extends the +dataset's lineage instead of starting a new one: + +```bash +dstrack track data.csv +``` + +```text +✔ Snapshot written (continued lineage, dataset ). +``` + +Each snapshot captures the file's schema, a content fingerprint, and per-column +statistics. See the [Getting Started guide](https://leoyala.github.io/dstrack/getting_started/) +for options such as `--name`, `--reader`, and `--dataset-id`. + ## Features -- Dataset versioning across ML pipeline stages -- Change detection and drift monitoring -- Lightweight CLI interface +- **Dataset versioning** - snapshot a dataset and track its lineage across pipeline stages +- **Rich snapshots** - schema hash, content fingerprint, and per-column statistics (ranges, null rates, and more) +- **CSV out of the box** - pure standard-library reader, no heavy dependencies +- **Lightweight CLI** - a small, git-like local store you can commit alongside your code + +## Roadmap + +Change detection and drift monitoring are on the way - comparing snapshots, surfacing +schema and distribution shifts, and failing CI on drift. See the +[roadmap](docs/roadmap.md) for what's planned. diff --git a/docs/decisions/0003-local-snapshot-store-layout.md b/docs/decisions/0003-local-snapshot-store-layout.md index cf1dcdb..ad785d6 100644 --- a/docs/decisions/0003-local-snapshot-store-layout.md +++ b/docs/decisions/0003-local-snapshot-store-layout.md @@ -18,7 +18,7 @@ needs a concrete answer to four questions: renames and moved files? 3. How can a user quickly search across many tracked datasets, or view the history of one, without reading every snapshot's full payload? -4. How does `dstrack snapshot` know which dataset a given file belongs to, given that +4. How does `dstrack track` know which dataset a given file belongs to, given that the same logical dataset may live at different paths on different machines? A key property, established by ADR-0001, shapes the answer to all four: a snapshot @@ -78,7 +78,7 @@ Two different jobs need two different mechanisms: - **History of one dataset** (`dstrack log `) only ever reads that dataset's own `snapshots/`, so no index is needed to make it fast. `dstrack log ` is also accepted, resolved to a `dataset_id` via the same path-matching used by - `dstrack snapshot` (see [Path resolution](#path-resolution-relative-root-overridable-never-a-shared-pointer)); if + `dstrack track` (see [Path resolution](#path-resolution-relative-root-overridable-never-a-shared-pointer)); if the path doesn't exactly match any dataset's last recorded one, `--dataset-id` is required instead. - **Search across many datasets** (`dstrack search ...`, by name/tag/pipeline stage) @@ -86,7 +86,7 @@ Two different jobs need two different mechanisms: tags, files that may also contain large histograms and MinHash/HyperLogLog sketches (see ADR-0001). Search needs a cheap, lightweight path. -`log.jsonl` is the fix for both, and is committed to git. Every `dstrack snapshot` call +`log.jsonl` is the fix for both, and is committed to git. Every `dstrack track` call writes the full snapshot JSON, appends one line to `log.jsonl` with only the lightweight identity fields (`snapshot_id`, `parent_snapshot_id`, `created_at`, `created_by`, `dataset_name`, `dataset_path`, `description`, `tags`, `num_rows`, @@ -98,7 +98,7 @@ rebuild step. `cache/index.db` (SQLite, stdlib `sqlite3`, no new dependency) is a pure performance accelerator for cross-dataset search, built by reading the small `log.jsonl` files, never the heavy snapshot JSON. It is **not** a second source of truth: it is gitignored -and safe to delete at any time. Every `dstrack snapshot` call appends its new row +and safe to delete at any time. Every `dstrack track` call appends its new row straight to the index, and every `dstrack search` also stats each dataset's `log.jsonl` first and re-syncs any lines it's missing (size/mtime tracked per dataset), so first-use, deletion, and pulls from other machines are all covered too. Between the @@ -110,7 +110,7 @@ Every snapshot's `dataset_path` (ADR-0001) is stored relative to a *path root*, written with `/` separators regardless of OS, and interpreted back with `pathlib.Path` so it round-trips correctly cross-platform. The path root defaults to the store root (the directory `.dstrack/` was found in, by walking up from cwd); -`dstrack snapshot --root ` overrides it for a single invocation, for cases where +`dstrack track --root ` overrides it for a single invocation, for cases where the data doesn't live under the checkout at all (a separate mount, a CI temp directory, a per-environment bucket). @@ -122,7 +122,7 @@ layouts disagree. Instead, every snapshot honestly records the relative path use history to append to. Nothing needs to be declared ahead of time, and there is no default/override split to maintain. -This also answers how `dstrack snapshot ` knows which dataset a file belongs to: +This also answers how `dstrack track ` knows which dataset a file belongs to: it computes the candidate relative path and compares it against each dataset's most recent (`HEAD`) `log.jsonl` entry. An exact match continues that dataset's lineage. No match means either a genuinely new dataset, @@ -133,7 +133,7 @@ are resolved the same way: pass `--dataset-id ` explicitly (found via `dstrack log`/`dstrack search`) to say `"this is a continuation, not a new dataset."` ### Snapshot write path -Given `dstrack snapshot [--name NAME] [--root DIR] [--dataset-id ID]`: +Given `dstrack track [--name NAME] [--root DIR] [--dataset-id ID]`: 1. Resolve the store root by walking up from cwd to find `.dstrack/`. 2. Resolve the path root: `--root` if given, otherwise the store root. Compute @@ -156,7 +156,7 @@ Given `dstrack snapshot [--name NAME] [--root DIR] [--dataset-id ID]`: fully written. ### Readers are chosen per invocation, never persisted -A reader is resolved fresh on every `dstrack snapshot` call: inferred from the file's +A reader is resolved fresh on every `dstrack track` call: inferred from the file's extension, or named explicitly by `--reader`, which accepts either a registered reader name (`--reader csv`) or a `package.module:ClassName` spec that `dstrack` imports directly (`--reader mypackage.readers:ExcelReader`). @@ -169,7 +169,7 @@ execution. That is perfectly safe as an argument the invoking user typed themsel it is exactly how gunicorn, uvicorn and celery accept application objects. It stops being safe the moment the same string is read from a file, because `.dstrack/` is *committed to git* (see [Location](#location)) and therefore travels with the repo. A reader spec stored -in a snapshot would mean that cloning a repository and running `dstrack snapshot` executes +in a snapshot would mean that cloning a repository and running `dstrack track` executes an importable path chosen by whoever wrote that commit, turning a pull request into a code execution vector on every reviewer's and CI runner's machine. @@ -197,12 +197,12 @@ is the supported answer. ## Example walkthrough Snapshot two new datasets. There is no separate registration step, the first -`dstrack snapshot` call for a given file is what creates it: +`dstrack track` call for a given file is what creates it: ``` dstrack init -dstrack snapshot data/train.csv --name "Customer Churn" -dstrack snapshot data/catalog.parquet --name "Product Catalog" +dstrack track data/train.csv --name "Customer Churn" +dstrack track data/catalog.parquet --name "Product Catalog" ``` The first call mints a `dataset_id`, computes `dataset_path` relative to the store @@ -224,7 +224,7 @@ Later, `data/train.csv` is updated and re-snapshotted, no name needed, since the dataset already exists: ``` -dstrack snapshot data/train.csv +dstrack track data/train.csv ``` `dstrack` computes `dataset_path` (`data/train.csv`), finds it matches the @@ -246,7 +246,7 @@ and once, for their first snapshot on this machine, tells `dstrack` which datase it continues, since the path won't auto-match: ``` -dstrack snapshot /mnt/shared-data/exports/train_2026_07.csv \ +dstrack track /mnt/shared-data/exports/train_2026_07.csv \ --root /mnt/shared-data/exports \ --dataset-id 8f14e45f-ceea-4c6a-8f31-8b0e2e1a9c3f ``` diff --git a/docs/getting_started.md b/docs/getting_started.md new file mode 100644 index 0000000..a005b93 --- /dev/null +++ b/docs/getting_started.md @@ -0,0 +1,154 @@ +--- +icon: lucide/play +--- + +# Getting Started + +This guide walks through the full `dstrack` workflow: initializing a store, taking your +first snapshot, and understanding how snapshots and lineage work. + +## Prerequisites + +- **Python 3.11 or later** +- Install the package: + +```bash +pip install dstrack +``` + +- Verify the installation: + +```bash +dstrack version +# 0.1.0 +``` + +## 1. Initialize a store + +`dstrack` keeps its history in a local store - a `.dstrack/` directory, conceptually +similar to git's `.git/`. Create one at the root of your project: + +```bash +dstrack init +``` + +```text +ℹ Generating local store structure at /path/to/.dstrack. +✔ Finished creating local store: /path/to/.dstrack +``` + +This creates: + +```text +.dstrack/ +├── datasets/ # one directory per tracked dataset +└── .gitignore # ignores the local .cache/ directory +``` + +A few things worth knowing: + +- The store is discovered by walking **up** the directory tree from where you run a + command, so you can `track` datasets from any subdirectory of your project. +- Set the `DSTRACK_ROOT_PATH` environment variable to point at a store elsewhere. +- Re-running `init` where a store already exists fails by design. Pass `--allow-exists` + to turn that into a warning instead: + +```bash +dstrack init --allow-exists +``` + +```text +⚡ Local store path already exists: /path/to/.dstrack +``` + +## 2. Track a dataset + +Tracking reads a data file, computes a **snapshot**, and stores it. Create a small +`data.csv` to follow along: + +```csv +id,name,value +1,alpha,10.5 +2,beta,20.0 +3,gamma,15.25 +``` + +Then snapshot it: + +```bash +dstrack track data.csv +``` + +```text +ℹ Reading data.csv and computing snapshot... +✔ Snapshot written (new dataset, dataset ). +ℹ Stored at /path/to/.dstrack/datasets//snapshots/.json +``` + +A snapshot is an immutable record of the dataset's state at that moment. It captures: + +- **Schema** - column names and inferred types, plus an order-independent `schema_hash` +- **Content fingerprint** - a SHA-256 hash of the source file +- **Per-column statistics** - counts, ranges, null rates, and distribution summaries + +The dataset name defaults to the file stem (`data` above); override it with `--name`. + +## 3. Snapshots and lineage + +Run `track` again on the **same path** and `dstrack` recognizes the dataset, extending its +lineage rather than starting a new one. The new snapshot points back to the previous one +via its `parent_snapshot_id`: + +```bash +dstrack track data.csv +``` + +```text +✔ Snapshot written (continued lineage, dataset ). +``` + +If you rename or move a dataset file, the recorded path no longer matches, so `dstrack` +would start a new lineage. To keep the history connected, continue the existing dataset +explicitly with `--dataset-id`: + +```bash +dstrack track renamed.csv --dataset-id +``` +For more information on what `dstrack track` allows you to do, simply ask for the help: + +```bash +dstrack track --help +``` + +## Where snapshots live + +Each snapshot is a JSON file under its dataset's directory: + +```text +.dstrack/ +└── datasets/ + └── / + └── snapshots/ + └── .json +``` + +The store also keeps an append-only log and a `HEAD` pointer per dataset. `.dstrack/` is +plain text and safe to commit alongside your code, giving you a versioned audit trail of +how your datasets evolved. + +## Benchmarking (optional) + +`dstrack` ships a second command, `dstrack-benchmark`, that generates a synthetic CSV and +measures snapshot-creation performance - handy for understanding overhead on large files: + +```bash +dstrack-benchmark run --rows 100000 +``` + +This command creates a synthetic file in a temporary location to measure performance. + +## What's next + +Change detection and drift monitoring - comparing snapshots to surface schema and +distribution shifts, and failing CI when data drifts - are planned. See the +[roadmap](roadmap.md) for the full picture. diff --git a/docs/index.md b/docs/index.md index 9befcf9..a850214 100644 --- a/docs/index.md +++ b/docs/index.md @@ -11,14 +11,18 @@ icon: lucide/rocket **Dataset versioning and monitoring for the machine learning lifecycle.** -`dstrack` helps data scientists and ML engineers track how datasets evolve over time — catching schema drift, distribution shifts, and unexpected mutations before they silently break pipelines or degrade model performance. +`dstrack` helps data scientists and ML engineers track how datasets evolve over time - catching schema drift, distribution shifts, and unexpected mutations before they silently break pipelines or degrade model performance. ## Features -- **Dataset versioning** — snapshot and compare datasets across pipeline stages -- **Change detection** — identify schema and structural changes between versions -- **Drift monitoring** — detect distribution shifts that can affect model performance -- **Lightweight CLI** — simple command-line interface with no heavy dependencies +- **Dataset versioning** - snapshot a dataset and track its lineage across pipeline stages +- **Rich snapshots** - schema hash, content fingerprint, and per-column statistics +- **CSV out of the box** - pure standard-library reader, no heavy dependencies +- **Lightweight CLI** - a small, git-like local store you can commit alongside your code + +!!! note "On the roadmap" + Change detection and drift monitoring - comparing snapshots to surface schema and + distribution shifts - are planned. See the [roadmap](roadmap.md). ## Installation @@ -30,13 +34,25 @@ Requires Python 3.11 or later. ## Quickstart +Initialize a store, then snapshot a dataset: + ```bash -dstrack +dstrack init +dstrack track data.csv ``` +```text +ℹ Reading data.csv and computing snapshot... +✔ Snapshot written (new dataset, dataset ). +ℹ Stored at /path/to/.dstrack/datasets//snapshots/.json +``` + +New here? The [Getting Started guide](getting_started.md) walks through the whole flow +step by step. + ## Why dstrack? -Data pipelines break silently. A column gets renamed upstream, a vendor changes a file format, or a feature distribution shifts after a data refresh — and you only find out when model accuracy drops in production. +Data pipelines break silently. A column gets renamed upstream, a vendor changes a file format, or a feature distribution shifts after a data refresh - and you only find out when model accuracy drops in production. `dstrack` gives you an audit trail for your datasets so you can catch these problems early, understand what changed, and reproduce any past state of your data. diff --git a/docs/roadmap.md b/docs/roadmap.md index 0ee6c1d..4b9cfd5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -6,7 +6,7 @@ icon: lucide/route This page tracks the planned development of `dstrack`. Each milestone ships as a versioned release and builds directly on the previous one. -## v0.1 — Project Skeleton +## v0.1 - Project Skeleton Foundation tooling: packaging, linting, type checking, CI/CD, and release automation. @@ -19,15 +19,15 @@ Foundation tooling: packaging, linting, type checking, CI/CD, and release automa Foundation for all future capabilities: reading a dataset and capturing an immutable record of its state. -- [ ] Local snapshot store +- [x] Local snapshot store - [x] CSV support (no extra dependencies) -- [ ] Content fingerprinting (deterministic, format-agnostic) -- [ ] Schema extraction (column names and types) -- [ ] Per-column statistics (counts, ranges, null rates, etc.) -- [ ] `dstrack init` and `dstrack snapshot` commands +- [x] Content fingerprinting (deterministic, format-agnostic) +- [x] Schema extraction (column names and types) +- [x] Per-column statistics (counts, ranges, null rates, etc.) +- [x] `dstrack init` and `dstrack track` commands - [ ] `dstrack log` command -## v0.2 — Diff Engine +## v0.2 - Diff Engine Compare two snapshots and surface what changed between them. @@ -37,7 +37,7 @@ Compare two snapshots and surface what changed between them. - [ ] `dstrack diff` command - [ ] Structured JSON output for machine consumption -## v0.3 — CLI Polish +## v0.3 - CLI Polish A first-class command-line experience ready for daily use. @@ -47,7 +47,7 @@ A first-class command-line experience ready for daily use. - [ ] `--json` flag on all commands - [ ] Shell completion -## v0.4 — Format and Integration Expansion +## v0.4 - Format and Integration Expansion Support the data formats researchers and engineers actually use. @@ -56,7 +56,7 @@ Support the data formats researchers and engineers actually use. - [ ] NumPy array support (optional install extra) - [ ] `dstrack.pandas` convenience module for snapshotting DataFrames directly -## v0.5 — CI/CD Integration +## v0.5 - CI/CD Integration Make drift detection a native step in automated pipelines. @@ -64,7 +64,7 @@ Make drift detection a native step in automated pipelines. - [ ] Integration into other frameworks: polars, moflow, etc. - [ ] CI integration documentation and working examples -## v0.6 — Migration Engine & Documentation and Examples +## v0.6 - Migration Engine & Documentation and Examples Everything a new user needs to be productive in under 10 minutes. @@ -76,7 +76,7 @@ Everything a new user needs to be productive in under 10 minutes. - [ ] API reference (auto-generated from source) - [ ] Automated docs deployment on each release -## v1.0 — Stable Release +## v1.0 - Stable Release A production-ready release with documented compatibility guarantees. diff --git a/src/dstrack/_benchmark/_profiling.py b/src/dstrack/_benchmark/_profiling.py index 7ca9fcc..70fdc1a 100644 --- a/src/dstrack/_benchmark/_profiling.py +++ b/src/dstrack/_benchmark/_profiling.py @@ -12,7 +12,7 @@ from pathlib import Path from typing import TypeAlias -# (filename, lineno, funcname) — the key pstats uses to identify a profiled frame. +# (filename, lineno, funcname) - the key pstats uses to identify a profiled frame. FrameKey: TypeAlias = tuple[str, int, str] # (primitive_calls, total_calls, total_time, cumulative_time) _FrameTotals: TypeAlias = tuple[int, int, float, float] diff --git a/src/dstrack/readers/_csv.py b/src/dstrack/readers/_csv.py index 5a7d809..a587a06 100644 --- a/src/dstrack/readers/_csv.py +++ b/src/dstrack/readers/_csv.py @@ -4,7 +4,7 @@ from collections import Counter from collections.abc import Iterator from pathlib import Path -from typing import Any +from typing import Any, TextIO from ._protocol import Cell, ColumnInfo @@ -124,7 +124,7 @@ def _coerce(raw: str | None, dtype: str) -> Cell: if raw not in _BOOL_ALL: raise ValueError(f"Invalid boolean value for bool column: {raw!r}") return raw in _BOOL_TRUE - # string, datetime64, bytes — keep as str + # string, datetime64, bytes - keep as str return raw @@ -237,6 +237,37 @@ def columns(self) -> list[ColumnInfo]: self._columns = self._detect_columns() return list(self._columns) + def _stat_signature(self, fh: TextIO) -> tuple[int, int]: + """Return the change signature ``(mtime_ns, size)`` for an open file. + + Args: + fh: An open handle to the CSV file. + + Returns: + The file's ``mtime_ns`` and size, used to detect whether the file + was modified between schema inference and iteration. + """ + stat = os.fstat(fh.fileno()) + return (stat.st_mtime_ns, stat.st_size) + + def _raise_if_file_changed(self, fh: TextIO) -> None: + """Raise if the file changed since schema inference recorded it. + + Compares the current signature of ``fh`` against the one captured in + [_detect_columns()][dstrack.readers._csv.CsvReader._detect_columns]. + Does nothing if no signature has been recorded yet. + + Args: + fh: An open handle to the CSV file. + + Raises: + RuntimeError: If the file's signature differs from the recorded one. + """ + if self._file_stat is not None and self._stat_signature(fh) != self._file_stat: + raise RuntimeError( + f"File changed between schema inference and iteration: {self._path}" + ) + def _detect_columns(self) -> list[ColumnInfo]: """Read up to ``sample_rows`` rows and infer a ColumnInfo per field. @@ -253,8 +284,7 @@ def _detect_columns(self) -> list[ColumnInfo]: ) samples: dict[str, list[str]] = {} with self._path.open(newline="", encoding=self._encoding) as fh: - stat = os.fstat(fh.fileno()) - self._file_stat = (stat.st_mtime_ns, stat.st_size) + self._file_stat = self._stat_signature(fh) reader = csv.DictReader(fh, **self._csv_kwargs) fieldnames = reader.fieldnames if not fieldnames: @@ -313,12 +343,7 @@ def iter_batches(self, batch_size: int = 1000) -> Iterator[list[list[Cell]]]: batch: list[list[Cell]] = [] with self._path.open(newline="", encoding=self._encoding) as fh: - stat = os.fstat(fh.fileno()) - current = (stat.st_mtime_ns, stat.st_size) - if self._file_stat is not None and current != self._file_stat: - raise RuntimeError( - f"File changed between schema inference and iteration: {self._path}" - ) + self._raise_if_file_changed(fh) reader = csv.DictReader(fh, **self._csv_kwargs) _ = reader.fieldnames # consume header row reader.fieldnames = names