Introduce file change detection

README.md

@@ -20,14 +20,67 @@ A Python package for versioning and monitoring dataset changes throughout the ma
 pip install dstrack
 ```
 
-## Quickstart
+Requires Python 3.11 or later.
+
+## Getting started
+
+Initialize a store in your project, then take your first snapshot.
+
+**1. Create a local store** - this adds a `.dstrack/` directory at the current location:
+
+```bash
+dstrack init
+```
+
+```text
+ℹ Generating local store structure at /path/to/.dstrack.
+✔ Finished creating local store: /path/to/.dstrack
+```
+
+**2. Track a dataset** - point `dstrack` at a data file to snapshot it. Given a small
+`data.csv`:
+
+```csv
+id,name,value
+1,alpha,10.5
+2,beta,20.0
+3,gamma,15.25
+```
 
 ```bash
-dstrack
+dstrack track data.csv
 ```
 
+```text
+ℹ Reading data.csv and computing snapshot...
+✔ Snapshot <snapshot-uuid> written (new dataset, dataset <dataset-uuid>).
+ℹ Stored at /path/to/.dstrack/datasets/<dataset-uuid>/snapshots/<snapshot-uuid>.json
+```
+
+**3. Re-track as the data changes** - running `track` again on the same path extends the
+dataset's lineage instead of starting a new one:
+
+```bash
+dstrack track data.csv
+```
+
+```text
+✔ Snapshot <snapshot-uuid> written (continued lineage, dataset <dataset-uuid>).
+```
+
+Each snapshot captures the file's schema, a content fingerprint, and per-column
+statistics. See the [Getting Started guide](https://leoyala.github.io/dstrack/getting_started/)
+for options such as `--name`, `--reader`, and `--dataset-id`.
+
 ## Features
 
-- Dataset versioning across ML pipeline stages
-- Change detection and drift monitoring
-- Lightweight CLI interface
+- **Dataset versioning** - snapshot a dataset and track its lineage across pipeline stages
+- **Rich snapshots** - schema hash, content fingerprint, and per-column statistics (ranges, null rates, and more)
+- **CSV out of the box** - pure standard-library reader, no heavy dependencies
+- **Lightweight CLI** - a small, git-like local store you can commit alongside your code
+
+## Roadmap
+
+Change detection and drift monitoring are on the way - comparing snapshots, surfacing
+schema and distribution shifts, and failing CI on drift. See the
+[roadmap](docs/roadmap.md) for what's planned.

docs/decisions/0003-local-snapshot-store-layout.md

@@ -18,7 +18,7 @@ needs a concrete answer to four questions:
    renames and moved files?
 3. How can a user quickly search across many tracked datasets, or view the history of
    one, without reading every snapshot's full payload?
-4. How does `dstrack snapshot` know which dataset a given file belongs to, given that
+4. How does `dstrack track` know which dataset a given file belongs to, given that
    the same logical dataset may live at different paths on different machines?
 
 A key property, established by ADR-0001, shapes the answer to all four: a snapshot
@@ -78,15 +78,15 @@ Two different jobs need two different mechanisms:
 - **History of one dataset** (`dstrack log <dataset_id>`) only ever reads that
   dataset's own `snapshots/`, so no index is needed to make it fast. `dstrack log <path>`
   is also accepted, resolved to a `dataset_id` via the same path-matching used by
-  `dstrack snapshot` (see [Path resolution](#path-resolution-relative-root-overridable-never-a-shared-pointer)); if
+  `dstrack track` (see [Path resolution](#path-resolution-relative-root-overridable-never-a-shared-pointer)); if
   the path doesn't exactly match any dataset's last recorded one, `--dataset-id` is
   required instead.
 - **Search across many datasets** (`dstrack search ...`, by name/tag/pipeline stage)
   would otherwise mean opening every dataset's snapshot JSON just to read its name or
   tags, files that may also contain large histograms and MinHash/HyperLogLog sketches
   (see ADR-0001). Search needs a cheap, lightweight path.
 
-`log.jsonl` is the fix for both, and is committed to git. Every `dstrack snapshot` call
+`log.jsonl` is the fix for both, and is committed to git. Every `dstrack track` call
 writes the full snapshot JSON, appends one line to `log.jsonl` with only the lightweight
 identity fields (`snapshot_id`, `parent_snapshot_id`, `created_at`, `created_by`,
 `dataset_name`, `dataset_path`, `description`, `tags`, `num_rows`,
@@ -98,7 +98,7 @@ rebuild step.
 `cache/index.db` (SQLite, stdlib `sqlite3`, no new dependency) is a pure performance
 accelerator for cross-dataset search, built by reading the small `log.jsonl` files,
 never the heavy snapshot JSON. It is **not** a second source of truth: it is gitignored
-and safe to delete at any time. Every `dstrack snapshot` call appends its new row
+and safe to delete at any time. Every `dstrack track` call appends its new row
 straight to the index, and every `dstrack search` also stats each dataset's `log.jsonl`
 first and re-syncs any lines it's missing (size/mtime tracked per dataset), so
 first-use, deletion, and pulls from other machines are all covered too. Between the
@@ -110,7 +110,7 @@ Every snapshot's `dataset_path` (ADR-0001) is stored relative to a *path root*,
 written with `/` separators regardless of OS, and interpreted back with
 `pathlib.Path` so it round-trips correctly cross-platform. The path root defaults to
 the store root (the directory `.dstrack/` was found in, by walking up from cwd);
-`dstrack snapshot --root <dir>` overrides it for a single invocation, for cases where
+`dstrack track --root <dir>` overrides it for a single invocation, for cases where
 the data doesn't live under the checkout at all (a separate mount, a CI temp
 directory, a per-environment bucket).
 
@@ -122,7 +122,7 @@ layouts disagree. Instead, every snapshot honestly records the relative path use
 history to append to. Nothing needs to be declared ahead of time, and there is no
 default/override split to maintain.
 
-This also answers how `dstrack snapshot <path>` knows which dataset a file belongs to:
+This also answers how `dstrack track <path>` knows which dataset a file belongs to:
 it computes the candidate relative path and compares it against
 each dataset's most recent (`HEAD`) `log.jsonl` entry. An exact match continues that
 dataset's lineage. No match means either a genuinely new dataset,
@@ -133,7 +133,7 @@ are resolved the same way: pass `--dataset-id <uuid>` explicitly (found via
 `dstrack log`/`dstrack search`) to say `"this is a continuation, not a new dataset."`
 
 ### Snapshot write path
-Given `dstrack snapshot <path> [--name NAME] [--root DIR] [--dataset-id ID]`:
+Given `dstrack track <path> [--name NAME] [--root DIR] [--dataset-id ID]`:
 
 1. Resolve the store root by walking up from cwd to find `.dstrack/`.
 2. Resolve the path root: `--root` if given, otherwise the store root. Compute
@@ -156,7 +156,7 @@ Given `dstrack snapshot <path> [--name NAME] [--root DIR] [--dataset-id ID]`:
    fully written.
 
 ### Readers are chosen per invocation, never persisted
-A reader is resolved fresh on every `dstrack snapshot` call: inferred from the file's
+A reader is resolved fresh on every `dstrack track` call: inferred from the file's
 extension, or named explicitly by `--reader`, which accepts either a registered reader
 name (`--reader csv`) or a `package.module:ClassName` spec that `dstrack` imports
 directly (`--reader mypackage.readers:ExcelReader`).
@@ -169,7 +169,7 @@ execution. That is perfectly safe as an argument the invoking user typed themsel
 it is exactly how gunicorn, uvicorn and celery accept application objects. It stops being
 safe the moment the same string is read from a file, because `.dstrack/` is *committed to
 git* (see [Location](#location)) and therefore travels with the repo. A reader spec stored
-in a snapshot would mean that cloning a repository and running `dstrack snapshot` executes
+in a snapshot would mean that cloning a repository and running `dstrack track` executes
 an importable path chosen by whoever wrote that commit, turning a pull request into a code
 execution vector on every reviewer's and CI runner's machine.
 
@@ -197,12 +197,12 @@ is the supported answer.
 ## Example walkthrough
 
 Snapshot two new datasets. There is no separate registration step, the first
-`dstrack snapshot` call for a given file is what creates it:
+`dstrack track` call for a given file is what creates it:
 
 ```
 dstrack init
-dstrack snapshot data/train.csv --name "Customer Churn"
-dstrack snapshot data/catalog.parquet --name "Product Catalog"
+dstrack track data/train.csv --name "Customer Churn"
+dstrack track data/catalog.parquet --name "Product Catalog"
 ```
 
 The first call mints a `dataset_id`, computes `dataset_path` relative to the store
@@ -224,7 +224,7 @@ Later, `data/train.csv` is updated and re-snapshotted, no name needed, since the
 dataset already exists:
 
 ```
-dstrack snapshot data/train.csv
+dstrack track data/train.csv
 ```
 
 `dstrack` computes `dataset_path` (`data/train.csv`), finds it matches the
@@ -246,7 +246,7 @@ and once, for their first snapshot on this machine, tells `dstrack` which datase
 it continues, since the path won't auto-match:
 
 ```
-dstrack snapshot /mnt/shared-data/exports/train_2026_07.csv \
+dstrack track /mnt/shared-data/exports/train_2026_07.csv \
     --root /mnt/shared-data/exports \
     --dataset-id 8f14e45f-ceea-4c6a-8f31-8b0e2e1a9c3f
 ```

docs/getting_started.md

@@ -0,0 +1,154 @@
+---
+icon: lucide/play
+---
+
+# Getting Started
+
+This guide walks through the full `dstrack` workflow: initializing a store, taking your
+first snapshot, and understanding how snapshots and lineage work.
+
+## Prerequisites
+
+- **Python 3.11 or later**
+- Install the package:
+
+```bash
+pip install dstrack
+```
+
+- Verify the installation:
+
+```bash
+dstrack version
+# 0.1.0
+```
+
+## 1. Initialize a store
+
+`dstrack` keeps its history in a local store - a `.dstrack/` directory, conceptually
+similar to git's `.git/`. Create one at the root of your project:
+
+```bash
+dstrack init
+```
+
+```text
+ℹ Generating local store structure at /path/to/.dstrack.
+✔ Finished creating local store: /path/to/.dstrack
+```
+
+This creates:
+
+```text
+.dstrack/
+├── datasets/     # one directory per tracked dataset
+└── .gitignore    # ignores the local .cache/ directory
+```
+
+A few things worth knowing:
+
+- The store is discovered by walking **up** the directory tree from where you run a
+  command, so you can `track` datasets from any subdirectory of your project.
+- Set the `DSTRACK_ROOT_PATH` environment variable to point at a store elsewhere.
+- Re-running `init` where a store already exists fails by design. Pass `--allow-exists`
+  to turn that into a warning instead:
+
+```bash
+dstrack init --allow-exists
+```
+
+```text
+⚡ Local store path already exists: /path/to/.dstrack
+```
+
+## 2. Track a dataset
+
+Tracking reads a data file, computes a **snapshot**, and stores it. Create a small
+`data.csv` to follow along:
+
+```csv
+id,name,value
+1,alpha,10.5
+2,beta,20.0
+3,gamma,15.25
+```
+
+Then snapshot it:
+
+```bash
+dstrack track data.csv
+```
+
+```text
+ℹ Reading data.csv and computing snapshot...
+✔ Snapshot <snapshot-uuid> written (new dataset, dataset <dataset-uuid>).
+ℹ Stored at /path/to/.dstrack/datasets/<dataset-uuid>/snapshots/<snapshot-uuid>.json
+```
+
+A snapshot is an immutable record of the dataset's state at that moment. It captures:
+
+- **Schema** - column names and inferred types, plus an order-independent `schema_hash`
+- **Content fingerprint** - a SHA-256 hash of the source file
+- **Per-column statistics** - counts, ranges, null rates, and distribution summaries
+
+The dataset name defaults to the file stem (`data` above); override it with `--name`.
+
+## 3. Snapshots and lineage
+
+Run `track` again on the **same path** and `dstrack` recognizes the dataset, extending its
+lineage rather than starting a new one. The new snapshot points back to the previous one
+via its `parent_snapshot_id`:
+
+```bash
+dstrack track data.csv
+```
+
+```text
+✔ Snapshot <snapshot-uuid> written (continued lineage, dataset <dataset-uuid>).
+```
+
+If you rename or move a dataset file, the recorded path no longer matches, so `dstrack`
+would start a new lineage. To keep the history connected, continue the existing dataset
+explicitly with `--dataset-id`:
+
+```bash
+dstrack track renamed.csv --dataset-id <dataset-uuid>
+```
+For more information on what `dstrack track` allows you to do, simply ask for the help:
+
+```bash
+dstrack track --help
+```
+
+## Where snapshots live
+
+Each snapshot is a JSON file under its dataset's directory:
+
+```text
+.dstrack/
+└── datasets/
+    └── <dataset-uuid>/
+        └── snapshots/
+            └── <snapshot-uuid>.json
+```
+
+The store also keeps an append-only log and a `HEAD` pointer per dataset. `.dstrack/` is
+plain text and safe to commit alongside your code, giving you a versioned audit trail of
+how your datasets evolved.
+
+## Benchmarking (optional)
+
+`dstrack` ships a second command, `dstrack-benchmark`, that generates a synthetic CSV and
+measures snapshot-creation performance - handy for understanding overhead on large files:
+
+```bash
+dstrack-benchmark run --rows 100000
+```
+
+This command creates a synthetic file in a temporary location to measure performance.
+
+## What's next
+
+Change detection and drift monitoring - comparing snapshots to surface schema and
+distribution shifts, and failing CI when data drifts - are planned. See the
+[roadmap](roadmap.md) for the full picture.

docs/index.md

@@ -11,14 +11,18 @@ icon: lucide/rocket
 
 **Dataset versioning and monitoring for the machine learning lifecycle.**
 
-`dstrack` helps data scientists and ML engineers track how datasets evolve over time — catching schema drift, distribution shifts, and unexpected mutations before they silently break pipelines or degrade model performance.
+`dstrack` helps data scientists and ML engineers track how datasets evolve over time - catching schema drift, distribution shifts, and unexpected mutations before they silently break pipelines or degrade model performance.
 
 ## Features
 
-- **Dataset versioning** — snapshot and compare datasets across pipeline stages
-- **Change detection** — identify schema and structural changes between versions
-- **Drift monitoring** — detect distribution shifts that can affect model performance
-- **Lightweight CLI** — simple command-line interface with no heavy dependencies
+- **Dataset versioning** - snapshot a dataset and track its lineage across pipeline stages
+- **Rich snapshots** - schema hash, content fingerprint, and per-column statistics
+- **CSV out of the box** - pure standard-library reader, no heavy dependencies
+- **Lightweight CLI** - a small, git-like local store you can commit alongside your code
+
+!!! note "On the roadmap"
+    Change detection and drift monitoring - comparing snapshots to surface schema and
+    distribution shifts - are planned. See the [roadmap](roadmap.md).
 
 ## Installation
 
@@ -30,13 +34,25 @@ Requires Python 3.11 or later.
 
 ## Quickstart
 
+Initialize a store, then snapshot a dataset:
+
 ```bash
-dstrack
+dstrack init
+dstrack track data.csv
 ```
 
+```text
+ℹ Reading data.csv and computing snapshot...
+✔ Snapshot <snapshot-uuid> written (new dataset, dataset <dataset-uuid>).
+ℹ Stored at /path/to/.dstrack/datasets/<dataset-uuid>/snapshots/<snapshot-uuid>.json
+```
+
+New here? The [Getting Started guide](getting_started.md) walks through the whole flow
+step by step.
+
 ## Why dstrack?
 
-Data pipelines break silently. A column gets renamed upstream, a vendor changes a file format, or a feature distribution shifts after a data refresh — and you only find out when model accuracy drops in production.
+Data pipelines break silently. A column gets renamed upstream, a vendor changes a file format, or a feature distribution shifts after a data refresh - and you only find out when model accuracy drops in production.
 
 `dstrack` gives you an audit trail for your datasets so you can catch these problems early, understand what changed, and reproduce any past state of your data.