From d1ec1448b67088b87fa1e95d6cd464b5d65485f3 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Thu, 9 Jul 2026 19:08:13 +1000 Subject: [PATCH] docs(drizzle): refresh the stash-drizzle skill; correct the eq/ne index contract `skills/stash-drizzle/SKILL.md` ships inside the `stash` tarball and `stash init` installs it for every Drizzle project. It was last substantively updated 2026-05-14; the Drizzle adapter moved 2026-07-08. Four defects, two of which produce silently wrong behaviour. **`eq`/`ne` require `equality`.** Both the skill and the operator's own TSDoc said "either `equality` or `orderAndRange`". The runtime disagrees: `createComparisonOperator` gates `orderAndRange` on `['gt','gte','lt','lte']` and takes the encrypting path only when `config.equality` is set. On an `orderAndRange`-only encrypted column, `eq` falls through to the plain Drizzle operator, which binds the raw plaintext through the column's `toDriver` and compares it against ciphertext. Verified: eq on equality:true -> lazy operator (encrypts) eq on orderAndRange -> plain SQL, bound param "x" (plaintext) No throw, no match. Contrast the `jsonb*` operators, which throw `EncryptionOperatorError` when `searchableJson` is absent. Corrected in the skill and in the TSDoc; documented as a hazard. The only Drizzle test file covers jsonb, which is why nothing caught it. **`npx generate-eql-migration` cannot work as documented.** The skill says to install `@cipherstash/stack` and then run that bin, but it belongs to the separate `@cipherstash/drizzle` package, which stack does not depend on. npm then resolves a registry package by that name and returns 404. Replaced with `stash eql install --drizzle`, which runs `drizzle-kit generate --custom` with the bundled EQL SQL (the older bin pins its own EQL version and downloads from GitHub). **An additive `db push` is promoted by `stash db activate`.** The skill told Proxy users cutover would promote the rollout's pending config. It does not -- cutover promotes only the rename pending -- and the cutover-time `db push` calls `discardPendingConfig()` before writing its own, so the un-activated rollout pending is discarded. Proxy keeps serving an active config that knows nothing about `_encrypted` for the whole dual-write window. `db activate` appeared nowhere in the skill. **Encrypted columns must be nullable.** The storage DDL declared an encrypted `jsonb NOT NULL` column, which invariant 1 of the doctrine shipped alongside it forbids. Also corrected the "EQL extension" wording: EQL is a schema plus a composite type, installed by `stash eql install`. Also: documents the Drizzle rename migration `encrypt cutover` scaffolds to keep drizzle-kit's snapshot in sync; adds the missing `timestamp` and `text` dataType values (CastAs has 8 members, the skill listed 6); repoints closed #447 at open #585; states that Drizzle is EQL v2 only and that `@cipherstash/stack/drizzle` is self-contained, not a re-export of `@cipherstash/drizzle`. --- .changeset/stash-drizzle-skill-refresh.md | 43 +++++++++++++++++ packages/stack/src/drizzle/operators.ts | 10 +++- skills/stash-drizzle/SKILL.md | 59 +++++++++++++++++------ 3 files changed, 94 insertions(+), 18 deletions(-) create mode 100644 .changeset/stash-drizzle-skill-refresh.md diff --git a/.changeset/stash-drizzle-skill-refresh.md b/.changeset/stash-drizzle-skill-refresh.md new file mode 100644 index 000000000..bad1e0715 --- /dev/null +++ b/.changeset/stash-drizzle-skill-refresh.md @@ -0,0 +1,43 @@ +--- +"@cipherstash/stack": patch +"stash": patch +--- + +Refresh the bundled `stash-drizzle` skill and correct the `eq`/`ne` operator docs. + +The skill ships inside the `stash` tarball and `stash init` installs it for every +Drizzle project, so its errors land in customer repos. It was last substantively +updated before `packages/drizzle` and the Drizzle adapter moved. + +- **`eq` / `ne` require `equality`, not "`equality` or `orderAndRange`."** The + skill and the operator's own TSDoc both claimed either index worked. On an + `orderAndRange`-only encrypted column, `eq` falls through to the plain Drizzle + operator and compares your **plaintext** value against the ciphertext column. + It does not throw. The query simply cannot match. Corrected in the skill and in + `packages/stack/src/drizzle/operators.ts`'s TSDoc, and documented as a hazard + alongside the contrast that the `jsonb*` operators *do* throw when + `searchableJson` is missing. +- **`npx generate-eql-migration` does not work for the documented install.** The + skill tells you to `npm install @cipherstash/stack drizzle-orm`, then run that + bin — but it belongs to the separate `@cipherstash/drizzle` package, which + `@cipherstash/stack` does not depend on. `npx` then looks for a package of that + name on the registry and gets `404 Not Found`. Replaced with + `stash eql install --drizzle`, which runs `drizzle-kit generate --custom` with + the bundled EQL SQL. +- **An additive `db push` is promoted by `stash db activate`, not by + `stash encrypt cutover`.** The skill told Proxy users the rollout's pending + config would be promoted at cutover. It won't: cutover promotes only the + *rename* pending, and the cutover-time `db push` calls `discardPendingConfig()` + first — so an un-activated rollout pending is discarded and Proxy serves the + old active config for the entire dual-write window. `db activate` was not + mentioned anywhere in the skill. +- **Encrypted columns must be nullable.** The storage DDL declared an encrypted + `jsonb NOT NULL` column, which the agent doctrine shipped alongside it forbids + because it breaks inserts during a rollout. Also corrected "EQL extension" — + EQL is a schema plus a composite type, installed by `stash eql install`. +- Documents the Drizzle-specific rename migration that `stash encrypt cutover` + scaffolds to keep drizzle-kit's snapshot in sync; adds the missing `timestamp` + and `text` `dataType` values (the `CastAs` union has 8 members, the skill listed + 6); repoints closed issue #447 at open #585; notes that the Drizzle path is EQL + v2 only and that `@cipherstash/stack/drizzle` is self-contained (not a re-export + of `@cipherstash/drizzle`). diff --git a/packages/stack/src/drizzle/operators.ts b/packages/stack/src/drizzle/operators.ts index 0644d7de3..09925255f 100644 --- a/packages/stack/src/drizzle/operators.ts +++ b/packages/stack/src/drizzle/operators.ts @@ -977,7 +977,10 @@ export function createEncryptionOperators(encryptionClient: EncryptionClient): { // Comparison operators /** * Equality operator - encrypts value for encrypted columns. - * Requires either `equality` or `orderAndRange` to be set on {@link EncryptedColumnConfig}. + * Requires `equality` on {@link EncryptedColumnConfig}. `orderAndRange` does not + * enable it: without `equality` this falls back to the plain Drizzle operator, + * comparing the plaintext value against the encrypted column, which matches + * nothing and does not throw. * * @example * Select users with a specific email address. @@ -990,7 +993,10 @@ export function createEncryptionOperators(encryptionClient: EncryptionClient): { /** * Not equal operator - encrypts value for encrypted columns. - * Requires either `equality` or `orderAndRange` to be set on {@link EncryptedColumnConfig}. + * Requires `equality` on {@link EncryptedColumnConfig}. `orderAndRange` does not + * enable it: without `equality` this falls back to the plain Drizzle operator, + * comparing the plaintext value against the encrypted column, which matches + * nothing and does not throw. * * @example * Select users whose email address is not a specific value. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 15012b410..665fc7ab8 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -22,40 +22,47 @@ Guide for integrating CipherStash field-level encryption with Drizzle ORM using npm install @cipherstash/stack drizzle-orm ``` -The Drizzle integration is included in `@cipherstash/stack` and imports from `@cipherstash/stack/drizzle`. +The Drizzle integration is included in `@cipherstash/stack` and imports from `@cipherstash/stack/drizzle`. It is self-contained — you do not need the separate `@cipherstash/drizzle` package, which is an older parallel implementation with different symbol names (`createProtectOperators`, `extractProtectSchema`). + +The Drizzle path targets **EQL v2**. `encryptedType` emits the `eql_v2_encrypted` column type and the operators emit `eql_v2.*` SQL. EQL v3 has no Drizzle surface yet: `stash eql install --eql-version 3` skips the Drizzle migration path and installs directly. ## Database Setup -### Install EQL Extension +### Install EQL + +EQL (Encrypt Query Language) provides the server-side functions for searchable encryption. It is **not** a PostgreSQL extension — do not `CREATE EXTENSION eql_v2`. It is a schema (`eql_v2`) plus a composite type (`public.eql_v2_encrypted`), installed by the CLI. -The EQL (Encrypt Query Language) PostgreSQL extension enables searchable encryption functions. Generate a migration: +For Drizzle, generate a migration containing the EQL SQL: ```bash -npx generate-eql-migration -# Options: -# -n, --name Migration name (default: "install-eql") -# -o, --out Output directory (default: "drizzle") +npx stash eql install --drizzle +# --name migration name +# --out output directory (default: "drizzle") ``` -Then apply it: +That runs `drizzle-kit generate --custom` and writes the bundled EQL SQL into the generated migration. Apply it as usual: ```bash npx drizzle-kit migrate ``` +> The older `npx generate-eql-migration` is a bin of the separate `@cipherstash/drizzle` package. If you only installed `@cipherstash/stack` (as above), that command fails with `404 Not Found` — npm looks for a package by that name and there isn't one. It also pins its own EQL version and downloads the SQL from GitHub, rather than using the version bundled with your CLI. Prefer `stash eql install --drizzle`. + ### Column Storage -Encrypted columns use the `eql_v2_encrypted` PostgreSQL type (installed by EQL). If not using EQL directly, use JSONB: +Encrypted columns use the `eql_v2_encrypted` composite type (created by `stash eql install`). Without EQL installed, an encrypted column is plain `jsonb` — but searchable encryption needs EQL; storage alone does not. ```sql CREATE TABLE users ( id SERIAL PRIMARY KEY, - email eql_v2_encrypted, -- with EQL extension - name jsonb NOT NULL, -- or use jsonb + email eql_v2_encrypted, -- encrypted, EQL installed + name jsonb, -- encrypted, no EQL age INTEGER -- non-encrypted columns are normal types ); ``` +> **Encrypted columns are nullable.** Never add `NOT NULL` at creation. The application writes ciphertext *after* the column exists, so a `NOT NULL` constraint breaks inserts during a rollout. Never declare them `text`, `varchar`, or `bytea`. + ## Schema Definition Use `encryptedType()` to define encrypted columns in Drizzle table schemas: @@ -97,7 +104,7 @@ const usersTable = pgTable("users", { | Config Option | Type | Description | |---|---|---| -| `dataType` | `"string"` \| `"number"` \| `"json"` \| `"boolean"` \| `"bigint"` \| `"date"` | Plaintext data type (default: `"string"`) | +| `dataType` | `"string"` \| `"text"` \| `"number"` \| `"bigint"` \| `"boolean"` \| `"date"` \| `"timestamp"` \| `"json"` | Plaintext data type (default: `"string"`) | | `equality` | `boolean` \| `TokenFilter[]` | Enable equality index | | `freeTextSearch` | `boolean` \| `MatchIndexOpts` | Enable free-text search index | | `orderAndRange` | `boolean` | Enable ORE index for sorting and range queries | @@ -401,7 +408,13 @@ Generate the migration with `drizzle-kit generate`. The generated SQL should be > stash db push > ``` > -> If this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected. The pending row will be promoted to active by `stash encrypt cutover` in the cutover step. +> If this is the project's first encrypted column, `db push` writes directly to the active EQL config and you're done. If an active config already exists, `db push` writes the new config as `pending` — **promote it now with `stash db activate`.** +> +> ```bash +> npx stash db activate +> ``` +> +> This step is easy to skip and the failure is silent. `stash encrypt cutover` promotes only the *rename* pending, later in the cutover step — it will not promote this additive one. Worse, the cutover-time `db push` calls `discardPendingConfig()` before writing its own pending, so an un-activated rollout pending is thrown away. Proxy would keep serving the old active config, which knows nothing about `email_encrypted`, for the whole dual-write window. > > SDK-only users can skip this step. @@ -458,7 +471,7 @@ Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination ord If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. -> **SDK-only note:** `stash encrypt cutover` currently requires a pending EQL configuration set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. [Issue #447](https://github.com/cipherstash/stack/issues/447) tracks decoupling this requirement. +> **SDK-only note:** `stash encrypt cutover` currently requires a pending EQL configuration set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. Decoupling this is tracked in [issue #585](https://github.com/cipherstash/stack/issues/585) — under EQL v3 there is no configuration table at all, so the precondition disappears. #### Cutover: rename swap and activate @@ -482,6 +495,8 @@ stash encrypt cutover --table users --column email Inside one transaction it: (1) renames `email` → `email_plaintext` and `email_encrypted` → `email`, (2) promotes the pending EQL config to `active` (and the prior active to `inactive`), (3) records a `cut_over` event in `cs_migrations`. +**Then, for Drizzle projects specifically, it scaffolds a migration for the rename.** The swap above ran outside drizzle-kit's authority, so its snapshot at `/meta/_snapshot.json` still describes the pre-rename shape — and the next `drizzle-kit generate` would emit a confused diff trying to recreate the old layout. The scaffolded migration carries idempotent rename SQL: a no-op against the database you just cut over (the old column names are gone), but correct when replaying migrations onto a fresh database. Commit it. If the scaffold fails, the CLI warns and you resync with `drizzle-kit pull`. + The Drizzle schema you just edited now matches the physical DB shape — `email` is the encrypted column. Keep the temporary `email_plaintext: text('email_plaintext')` declaration in the schema file until the drop step: ```typescript @@ -553,8 +568,8 @@ All three are read-only. | Operator | Usage | Required Index | |---|---|---| -| `eq(col, value)` | Equality | `equality: true` or `orderAndRange: true` | -| `ne(col, value)` | Not equal | `equality: true` or `orderAndRange: true` | +| `eq(col, value)` | Equality | `equality: true` | +| `ne(col, value)` | Not equal | `equality: true` | | `gt(col, value)` | Greater than | `orderAndRange: true` | | `gte(col, value)` | Greater than or equal | `orderAndRange: true` | | `lt(col, value)` | Less than | `orderAndRange: true` | @@ -570,6 +585,18 @@ All three are read-only. | `jsonbGet(col, selector)` | Get value using JSONB `->` operator | `searchableJson: true` | | `jsonbPathExists(col, selector)` | Check if JSONB path exists | `searchableJson: true` | +> **A missing index degrades silently — declare the index the operator needs.** +> +> If an encrypted column lacks the required index, most operators fall back to the plain Drizzle operator and compare your **plaintext** value against the ciphertext column. Nothing throws; the query simply cannot match. `eq` on an `equality`-less column is the easy mistake — `orderAndRange` does *not* enable it. +> +> ```typescript +> // email: encryptedType('email', { orderAndRange: true }) // no equality! +> await db.select().from(users).where(await ops.eq(users.email, 'alice@example.com')) +> // -> falls back to a plaintext comparison. Returns nothing. No error. +> ``` +> +> The `jsonb*` operators are the exception: they **throw** `EncryptionOperatorError` when `searchableJson` is missing, rather than degrading. + ### Sort Operators (sync) | Operator | Usage | Required Index |