Add snapshot creation mechanism

codecov.yaml

@@ -2,9 +2,8 @@ coverage:
   status:
     project:
       default:
-        target: auto
+        target: 95%
         threshold: 0.5%
-        base: auto
 comment:
   layout: "diff, flags, files"
   behavior: default

docs/API/snapshot.md

@@ -0,0 +1,5 @@
+---
+icon: lucide/camera
+---
+
+::: dstrack.snapshot

docs/decisions/0001-dataset-snapshot-content.md

@@ -105,3 +105,4 @@ These compact structures enable cheap diff and similarity queries across snapsho
 - Content fingerprints (MinHash, HyperLogLog) enable cheap cross-snapshot diff and similarity queries without loading raw data, at the cost of being approximate.
 - Scope is intentionally limited to tabular data; non-tabular modalities (images, audio, video, graphs) are not supported and would require a new ADR to extend the schema.
 - `format_version` is required on every snapshot to allow the schema to evolve without breaking existing readers.
+- Statistics are computed in a single **in-memory** pass, so worst-case memory grows with the row count (per-column values, the distinct-row set, and per-string value counts are retained until the pass completes). A configurable `max_rows` limit bounds this: datasets larger than the limit are rejected with a clear error rather than silently exhausting memory. The limit is exposed on `StatsComputer` and via the `dstrack track --max-rows` option, and may be disabled from Python when enough memory is available.

docs/decisions/0003-local-snapshot-store-layout.md

@@ -146,13 +146,45 @@ Given `dstrack snapshot <path> [--name NAME] [--root DIR] [--dataset-id ID]`:
      new dataset, `--name` is required, and a new `dataset_id` (UUID4) is minted.
 4. If continuing an existing lineage, read `HEAD` and use it as `parent_snapshot_id`;
    otherwise `parent_snapshot_id` is `null`.
-5. Read the source with the reader inferred from the file extension, compute
+5. Read the source with the reader inferred from the file extension, or with the one
+   named by `--reader` (see [Readers are chosen per invocation, never
+   persisted](#readers-are-chosen-per-invocation-never-persisted) below), compute
    schema/stats/hashes per ADR-0001, and mint a new `snapshot_id` (UUID4).
 6. Write `snapshots/<snapshot_id>.json`, append the corresponding line to `log.jsonl`
    (including `dataset_path`), then atomically replace `HEAD` (write-temp + rename),
    in that order, so a crash never leaves `HEAD` pointing at a snapshot that wasn't
    fully written.
 
+### Readers are chosen per invocation, never persisted
+A reader is resolved fresh on every `dstrack snapshot` call: inferred from the file's
+extension, or named explicitly by `--reader`, which accepts either a registered reader
+name (`--reader csv`) or a `package.module:ClassName` spec that `dstrack` imports
+directly (`--reader mypackage.readers:ExcelReader`).
+
+**The resolved reader is deliberately not part of a snapshot.** No `reader` field is
+written to `snapshots/*.json` or to `log.jsonl`, and nothing in the store is ever read
+back to decide which reader to use. This is a security boundary, not an oversight:
+a `package.module:ClassName` spec is arbitrary import-by-name, which is arbitrary code
+execution. That is perfectly safe as an argument the invoking user typed themselves, and
+it is exactly how gunicorn, uvicorn and celery accept application objects. It stops being
+safe the moment the same string is read from a file, because `.dstrack/` is *committed to
+git* (see [Location](#location)) and therefore travels with the repo. A reader spec stored
+in a snapshot would mean that cloning a repository and running `dstrack snapshot` executes
+an importable path chosen by whoever wrote that commit, turning a pull request into a code
+execution vector on every reviewer's and CI runner's machine.
+
+The rule is therefore: **a reader spec may travel from the user to `dstrack`, never from
+the store to `dstrack`.** Anything that would need a per-dataset default reader must be
+solved another way, by an installed plugin claiming the extension via the
+`dstrack.readers` entry-point group (code the user chose to `pip install`, on the same
+footing as any other dependency), never by resurrecting an import path out of committed
+history.
+
+The cost is real and accepted: a dataset whose extension is not claimed by any installed
+reader needs `--reader` on every invocation, and cannot be made to "just work" for a
+teammate by committing that fact. Publishing or vendoring the reader as a plugin package
+is the supported answer.
+
 ### What's committed vs. local-only
 
 | Path | Committed? | Why |
@@ -249,3 +281,8 @@ this machine) return the same answer regardless of which machine ran them.
   dataset history; this is an implementation invariant, not just a convention.
 - `cache/index.db` can be deleted at any time with no data loss, only a slower next
   search until it's rebuilt.
+- No reader information is ever written to the store, and no future field may reintroduce
+  it: because `.dstrack/` is committed, an importable path read back out of it would
+  execute code chosen by whoever authored the commit. Readers reach `dstrack` from the
+  invoking user (`--reader`) or from installed plugins, never from committed history. Any
+  proposal for a per-dataset default reader supersedes this ADR rather than extending it.

pyproject.toml

@@ -33,6 +33,7 @@ GitHub = "https://github.com/leoyala/dstrack"
 
 [project.scripts]
 dstrack = "dstrack._cli:app"
+dstrack-benchmark = "dstrack._benchmark:app"
 
 [build-system]
 requires = ["uv_build>=0.11.26,<0.12.0"]

src/dstrack/_benchmark/__init__.py

@@ -0,0 +1,54 @@
+"""Benchmark for dstrack's snapshot creation pipeline.
+
+Generates a synthetic CSV in a temporary directory, then times how long it
+takes to build snapshot metadata and statistics for it, alongside a summary of
+the environment the benchmark ran in and a call tree of dstrack's own methods.
+
+Layers
+------
+[dstrack._benchmark._synthetic][]
+    Describes and writes the synthetic dataset.
+
+[dstrack._benchmark._runner][]
+    Runs the snapshot pipeline against it and times each phase.
+
+[dstrack._benchmark._profiling][]
+    Reduces the ``cProfile`` run to a call tree of dstrack's own methods.
+
+[dstrack._benchmark._environment][]
+    Describes the machine the benchmark ran on.
+
+[dstrack._benchmark._report][], [dstrack._benchmark._cli][]
+    Render runs to a console, and wire the whole thing to ``dstrack-benchmark``.
+"""
+
+from ._cli import app
+from ._environment import EnvironmentInfo, collect_environment_info
+from ._profiling import CallGraph, CallNode, CallScope
+from ._report import ConsoleReporter
+from ._runner import (
+    BenchmarkObserver,
+    BenchmarkResult,
+    BenchmarkRun,
+    BenchmarkRunner,
+    SilentObserver,
+)
+from ._synthetic import SyntheticColumn, SyntheticCsvSpec, write_synthetic_csv
+
+__all__ = [
+    "BenchmarkObserver",
+    "BenchmarkResult",
+    "BenchmarkRun",
+    "BenchmarkRunner",
+    "CallGraph",
+    "CallNode",
+    "CallScope",
+    "ConsoleReporter",
+    "EnvironmentInfo",
+    "SilentObserver",
+    "SyntheticColumn",
+    "SyntheticCsvSpec",
+    "app",
+    "collect_environment_info",
+    "write_synthetic_csv",
+]

src/dstrack/_benchmark/_cli.py

@@ -0,0 +1,114 @@
+"""Command-line entry point for the snapshot benchmark."""
+
+import shutil
+import tempfile
+from collections.abc import Iterator
+from contextlib import contextmanager
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from dstrack._benchmark._report import ConsoleReporter
+from dstrack._benchmark._runner import BenchmarkRunner
+from dstrack._benchmark._synthetic import SyntheticCsvSpec
+
+app = typer.Typer(
+    no_args_is_help=True,
+    pretty_exceptions_show_locals=False,
+    help="Benchmark dstrack's snapshot creation pipeline against a synthetic CSV.",
+)
+
+_DEFAULTS = SyntheticCsvSpec()
+
+
+@app.callback()
+def _main() -> None:
+    """Keep ``run`` a named subcommand so a bare invocation shows help.
+
+    Typer promotes a lone command to the top level, which quietly discards
+    ``no_args_is_help``; an explicit callback keeps the command group intact so
+    running ``dstrack-benchmark`` with no arguments prints help instead of
+    silently kicking off a full-size benchmark.
+    """
+
+
+@contextmanager
+def _workspace(reporter: ConsoleReporter, *, keep_csv: bool) -> Iterator[Path]:
+    """Yield a path for the synthetic CSV, discarding it unless ``keep_csv``."""
+    tmp_dir = Path(tempfile.mkdtemp(prefix="dstrack-benchmark-"))
+    csv_path = tmp_dir / "synthetic_dataset.csv"
+    try:
+        yield csv_path
+    finally:
+        if keep_csv:
+            reporter.csv_kept(csv_path)
+        else:
+            shutil.rmtree(tmp_dir, ignore_errors=True)
+
+
+@app.command(help="Generate a synthetic CSV and benchmark snapshot creation on it.")
+def run(
+    rows: Annotated[
+        int, typer.Option(min=0, help="Number of data rows to generate.")
+    ] = _DEFAULTS.num_rows,
+    numeric_cols: Annotated[
+        int, typer.Option(min=0, help="Number of numeric (float) columns.")
+    ] = _DEFAULTS.num_numeric_cols,
+    string_cols: Annotated[
+        int, typer.Option(min=0, help="Number of string columns.")
+    ] = _DEFAULTS.num_string_cols,
+    datetime_cols: Annotated[
+        int, typer.Option(min=0, help="Number of datetime columns.")
+    ] = _DEFAULTS.num_datetime_cols,
+    bool_cols: Annotated[
+        int, typer.Option(min=0, help="Number of boolean columns.")
+    ] = _DEFAULTS.num_bool_cols,
+    null_rate: Annotated[
+        float,
+        typer.Option(min=0.0, max=1.0, help="Fraction of non-id cells left null."),
+    ] = _DEFAULTS.null_rate,
+    seed: Annotated[
+        int, typer.Option(help="Random seed for reproducible synthetic data.")
+    ] = _DEFAULTS.seed,
+    keep_csv: Annotated[
+        bool,
+        typer.Option(
+            help="Keep the generated CSV instead of deleting it after the run."
+        ),
+    ] = False,
+    profile: Annotated[
+        bool,
+        typer.Option(
+            help="Profile dstrack's own methods while building the snapshot. "
+            "Adds cProfile overhead to the reported metadata/stats timings."
+        ),
+    ] = True,
+    profile_limit: Annotated[
+        int,
+        typer.Option(
+            min=0,
+            help="Max number of child methods shown under each node of the "
+            "profile call tree, ranked by cumulative time.",
+        ),
+    ] = 20,
+) -> None:
+    spec = SyntheticCsvSpec(
+        num_rows=rows,
+        num_numeric_cols=numeric_cols,
+        num_string_cols=string_cols,
+        num_datetime_cols=datetime_cols,
+        num_bool_cols=bool_cols,
+        null_rate=null_rate,
+        seed=seed,
+    )
+    reporter = ConsoleReporter()
+    runner = BenchmarkRunner(spec, observer=reporter, profile=profile)
+
+    with _workspace(reporter, keep_csv=keep_csv) as csv_path:
+        benchmark_run = runner.run(csv_path)
+        reporter.report(benchmark_run, profile_limit=profile_limit)
+
+
+if __name__ == "__main__":
+    app()