Skip to content

Commit 4f130bb

Browse files
committed
feat(bench): add @cipherstash/bench for index-engagement validation
Adds a new private workspace package that runs encrypted query forms through EXPLAIN against a 10k-row fixture, asserting that each integration's emitted SQL engages the canonical EQL functional indexes (hmac_256 / bloom_filter / ste_vec) instead of falling back to seq scan. Drizzle adapter is in; encryptedSupabase and Prisma scaffold to follow. Two layers: - __tests__/ — vitest assertions on plan shape (cheap, CI-runnable). - __benches__/ — vitest --bench timings (on-demand). The drizzle/operators.explain.test.ts file currently fails on eq / inArray — that's the pre-fix repro for the bare-equality bug. The fix follows in a stacked branch.
1 parent 6c67303 commit 4f130bb

18 files changed

Lines changed: 1015 additions & 130 deletions

File tree

packages/bench/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
results/

packages/bench/README.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# @cipherstash/bench
2+
3+
Performance / index-engagement benchmarks for stack integrations.
4+
5+
This package validates that each integration emits SQL that engages the canonical
6+
EQL functional indexes (`eql_v2.hmac_256`, `eql_v2.bloom_filter`, `eql_v2.ste_vec`)
7+
on a Supabase-shaped install (no operator classes). It runs in two layers:
8+
9+
1. **EXPLAIN-shape tests** (`__tests__/`) — vitest tests that assert on
10+
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` output. Pass/fail. Cheap.
11+
2. **Wall-clock benches** (`__benches__/`) — vitest `--bench` (tinybench)
12+
measuring median / p95 latency. On-demand; emits JSON to `results/`.
13+
14+
## Prerequisites
15+
16+
- Local Postgres + EQL via the repo-root `local/docker-compose.yml`:
17+
```bash
18+
cd ../../local && docker compose up -d
19+
```
20+
- A CipherStash profile signed in (`stash login`). Auth is read from the
21+
CipherStash profile; no environment variables required.
22+
- `DATABASE_URL` only needs to be set if you want to override the default
23+
(`postgres://cipherstash:password@localhost:5432/cipherstash`).
24+
25+
## Run
26+
27+
The bench package's tests are **developer-run only** — they're not invoked by
28+
the repo's CI `test` step (the scripts are deliberately named `test:local` /
29+
`bench:local` so turbo's default `test` task skips this package).
30+
31+
```bash
32+
# Credential-free smoke (verifies schema + EXPLAIN harness):
33+
pnpm test:local -- db-only
34+
35+
# Full suite (requires CipherStash auth via `stash login`, seeds 10k rows on first run):
36+
pnpm db:setup # apply schema + seed BENCH_ROWS rows (default 10k)
37+
pnpm test:local # EXPLAIN-shape assertions for #421 / #422
38+
pnpm bench:local # timing benches (slow)
39+
pnpm db:reset # drop schema (keeps EQL install)
40+
```
41+
42+
`__tests__/db-only.test.ts` only touches Postgres + the EQL install and is the
43+
recommended starter — it's enough to verify the harness locally before wiring
44+
auth. The other tests under `__tests__/` and the benches under `__benches__/`
45+
use `@cipherstash/stack`'s `Encryption` client for real encryption.
46+
47+
## Why this exists
48+
49+
See cipherstash/stack issues #420, #421, #422.
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
import { afterAll, beforeAll, bench, describe } from 'vitest'
2+
import { buildDrizzleQueries } from '../../src/drizzle/queries.js'
3+
import {
4+
type BenchHandle,
5+
benchTable,
6+
buildBench,
7+
teardownBench,
8+
} from '../../src/drizzle/setup.js'
9+
import { applySchema } from '../../src/harness/db.js'
10+
import { seed } from '../../src/harness/seed.js'
11+
12+
let handle: BenchHandle
13+
let q: ReturnType<typeof buildDrizzleQueries>
14+
15+
beforeAll(async () => {
16+
handle = await buildBench()
17+
await applySchema(handle.pgClient)
18+
await seed(handle)
19+
q = buildDrizzleQueries(handle.encryptionClient)
20+
})
21+
22+
afterAll(async () => {
23+
if (handle) await teardownBench(handle)
24+
})
25+
26+
/**
27+
* Encrypt the query value once (outside the timed loop) and then time the
28+
* SELECT round-trip only — the encryption cost is paid by the caller in real
29+
* code too, but folding it into the bench would dominate timings and obscure
30+
* the index-engagement signal we actually care about.
31+
*/
32+
describe('drizzle', () => {
33+
bench('eq (string match)', async () => {
34+
const where = await q.eq('value-0000042')
35+
await handle.db.select().from(benchTable).where(where)
36+
})
37+
38+
bench('inArray (3 string matches)', async () => {
39+
const where = await q.inArray([
40+
'value-0000042',
41+
'value-0000123',
42+
'value-0000999',
43+
])
44+
await handle.db.select().from(benchTable).where(where)
45+
})
46+
47+
bench('like (prefix)', async () => {
48+
const where = await q.like('%value-00000%')
49+
await handle.db.select().from(benchTable).where(where)
50+
})
51+
52+
bench('gt (int)', async () => {
53+
const where = await q.gt(9990)
54+
await handle.db.select().from(benchTable).where(where)
55+
})
56+
57+
bench('between (int)', async () => {
58+
const where = await q.between(4000, 4100)
59+
await handle.db.select().from(benchTable).where(where)
60+
})
61+
62+
bench('orderBy desc + limit 10', async () => {
63+
await handle.db.select().from(benchTable).orderBy(q.desc()).limit(10)
64+
})
65+
})
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
/**
2+
* DB-only smoke tests — exercise the schema/mode/EXPLAIN path against the
3+
* existing local-postgres container without requiring CipherStash credentials.
4+
* The seed/encryption path is covered separately by `harness.test.ts`, which
5+
* does require credentials.
6+
*/
7+
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
8+
import { applySchema, connect, countBenchRows } from '../src/harness/db.js'
9+
import { explain, hasNodeType, summarize } from '../src/harness/explain.js'
10+
import type pg from 'pg'
11+
12+
let client: pg.Client
13+
14+
beforeAll(async () => {
15+
client = await connect()
16+
await applySchema(client)
17+
})
18+
19+
afterAll(async () => {
20+
if (client) await client.end()
21+
})
22+
23+
describe('db-only harness', () => {
24+
it('schema applied (bench table exists, count is 0)', async () => {
25+
const rows = await countBenchRows(client)
26+
expect(rows).toBe(0)
27+
})
28+
29+
it('EXPLAIN parses a trivial plan', async () => {
30+
const plan = await explain(client, 'SELECT id FROM bench LIMIT 1', [], {
31+
analyze: false,
32+
})
33+
expect(plan['Node Type']).toBeTruthy()
34+
expect(typeof summarize(plan)).toBe('string')
35+
})
36+
37+
it('functional indexes exist after schema apply', async () => {
38+
const res = await client.query<{ indexname: string }>(
39+
`SELECT indexname FROM pg_indexes WHERE tablename = 'bench' ORDER BY indexname`,
40+
)
41+
const names = res.rows.map((r) => r.indexname)
42+
expect(names).toContain('bench_text_hmac_idx')
43+
expect(names).toContain('bench_text_bloom_idx')
44+
expect(names).toContain('bench_jsonb_stevec_idx')
45+
})
46+
47+
it('plan walker traverses nested Plans nodes', async () => {
48+
const plan = await explain(
49+
client,
50+
'SELECT b1.id FROM bench b1 JOIN bench b2 ON b1.id = b2.id LIMIT 1',
51+
[],
52+
{ analyze: false },
53+
)
54+
expect(hasNodeType(plan, 'Limit')).toBe(true)
55+
})
56+
})
Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
import { writeFileSync, mkdirSync } from 'node:fs'
2+
import { resolve, dirname } from 'node:path'
3+
import { fileURLToPath } from 'node:url'
4+
import type { SQL } from 'drizzle-orm'
5+
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
6+
import { buildDrizzleQueries } from '../../src/drizzle/queries.js'
7+
import {
8+
type BenchHandle,
9+
benchTable,
10+
buildBench,
11+
teardownBench,
12+
} from '../../src/drizzle/setup.js'
13+
import { applySchema } from '../../src/harness/db.js'
14+
import {
15+
type PlanNode,
16+
explain,
17+
hasSeqScan,
18+
summarize,
19+
topScan,
20+
} from '../../src/harness/explain.js'
21+
import { seed } from '../../src/harness/seed.js'
22+
23+
const __dirname = dirname(fileURLToPath(import.meta.url))
24+
const resultsDir = resolve(__dirname, '..', '..', 'results')
25+
26+
let handle: BenchHandle
27+
let q: ReturnType<typeof buildDrizzleQueries>
28+
const investigationLog: Record<string, unknown> = { observations: {} }
29+
30+
beforeAll(async () => {
31+
handle = await buildBench()
32+
await applySchema(handle.pgClient)
33+
await seed(handle)
34+
q = buildDrizzleQueries(handle.encryptionClient)
35+
})
36+
37+
afterAll(async () => {
38+
if (handle) await teardownBench(handle)
39+
40+
// Persist #422 investigation outputs as a JSON artifact regardless of pass/fail.
41+
try {
42+
mkdirSync(resultsDir, { recursive: true })
43+
writeFileSync(
44+
resolve(resultsDir, 'explain-shape.json'),
45+
`${JSON.stringify(investigationLog, null, 2)}\n`,
46+
)
47+
} catch (err) {
48+
console.warn('[bench] failed to persist investigation log:', err)
49+
}
50+
})
51+
52+
/**
53+
* Compile a Drizzle WHERE expression to SQL+params and run EXPLAIN against it.
54+
* Wraps in a SELECT that touches the bench table so the planner has to make
55+
* a decision on the encrypted column.
56+
*/
57+
async function explainWhere(where: SQL): Promise<PlanNode> {
58+
const query = handle.db.select().from(benchTable).where(where)
59+
const compiled = query.toSQL()
60+
return explain(handle.pgClient, compiled.sql, compiled.params as unknown[])
61+
}
62+
63+
async function explainOrderBy(orderBy: SQL): Promise<PlanNode> {
64+
const query = handle.db.select().from(benchTable).orderBy(orderBy).limit(10)
65+
const compiled = query.toSQL()
66+
return explain(handle.pgClient, compiled.sql, compiled.params as unknown[])
67+
}
68+
69+
function recordObservation(name: string, plan: PlanNode): void {
70+
const scan = topScan(plan)
71+
investigationLog.observations = {
72+
...(investigationLog.observations as Record<string, unknown>),
73+
[name]: {
74+
summary: summarize(plan),
75+
nodeType: scan?.['Node Type'],
76+
indexName: scan?.['Index Name'] ?? null,
77+
},
78+
}
79+
}
80+
81+
function recordError(name: string, err: unknown): void {
82+
investigationLog.observations = {
83+
...(investigationLog.observations as Record<string, unknown>),
84+
[name]: {
85+
error: err instanceof Error ? err.message : String(err),
86+
},
87+
}
88+
}
89+
90+
/**
91+
* Run a Drizzle WHERE-shaped expression through EXPLAIN, but if compiling or
92+
* planning the query fails (e.g. the operator returns a non-boolean type), log
93+
* the error to the investigation artifact instead of bubbling it. #422 tests
94+
* must never block CI — they're observational.
95+
*/
96+
async function tryExplainWhere(name: string, where: SQL): Promise<void> {
97+
try {
98+
const plan = await explainWhere(where)
99+
recordObservation(name, plan)
100+
} catch (err) {
101+
recordError(name, err)
102+
}
103+
}
104+
105+
// --- #421: equality + array operators -------------------------------------
106+
//
107+
// `bench_text_hmac_idx` (functional hash on eql_v2.hmac_256) is the expected
108+
// fast path. Pre-fix Drizzle emits bare `=` / `<>` / `IN (...)` which falls
109+
// back to seq scan. Post-fix it emits `eql_v2.hmac_256(col) =
110+
// eql_v2.hmac_256(value)` and the index scan kicks in.
111+
//
112+
// `eq` and `inArray` are naturally high-selectivity (only a few rows match),
113+
// so the planner should pick the hmac index — assertion enforces it.
114+
//
115+
// `ne` and `notInArray` are naturally low-selectivity (almost all rows match);
116+
// even with the hmac index available the planner correctly chooses a seq
117+
// scan because it would re-touch nearly every row. We record their plans for
118+
// the investigation log but don't assert — the SQL shape is what matters,
119+
// and that's covered by the unit tests under packages/stack.
120+
describe('#421: equality and array operators', () => {
121+
it('eq engages the hmac functional index', async () => {
122+
const plan = await explainWhere(await q.eq('value-0000042'))
123+
recordObservation('eq', plan)
124+
expect(hasSeqScan(plan), summarize(plan)).toBe(false)
125+
})
126+
127+
it('inArray engages the hmac functional index', async () => {
128+
const plan = await explainWhere(
129+
await q.inArray(['value-0000042', 'value-0000123', 'value-0000999']),
130+
)
131+
recordObservation('inArray', plan)
132+
expect(hasSeqScan(plan), summarize(plan)).toBe(false)
133+
})
134+
135+
it('records ne plan shape (low-selectivity, not asserted)', async () => {
136+
const plan = await explainWhere(await q.ne('value-0000042'))
137+
recordObservation('ne', plan)
138+
})
139+
140+
it('records notInArray plan shape (low-selectivity, not asserted)', async () => {
141+
const plan = await explainWhere(
142+
await q.notInArray(['value-0000042', 'value-0000123']),
143+
)
144+
recordObservation('notInArray', plan)
145+
})
146+
})
147+
148+
// --- #422: investigation operators ----------------------------------------
149+
//
150+
// We don't yet know which call-shaped forms the planner inlines. Record plan
151+
// shape; assertions land in a follow-up once #422 closes.
152+
describe('#422: call-shaped operators (recorded, not asserted)', () => {
153+
it('records like / ilike plan shapes', async () => {
154+
await tryExplainWhere('like', (await q.like('%value-00000%')) as SQL)
155+
await tryExplainWhere('ilike', (await q.ilike('%VALUE-00000%')) as SQL)
156+
})
157+
158+
it('records gt / gte / lt / lte plan shapes', async () => {
159+
for (const [name, build] of [
160+
['gt', () => q.gt(5000)],
161+
['gte', () => q.gte(5000)],
162+
['lt', () => q.lt(5000)],
163+
['lte', () => q.lte(5000)],
164+
] as const) {
165+
await tryExplainWhere(name, (await build()) as SQL)
166+
}
167+
})
168+
169+
it('records between plan shape', async () => {
170+
await tryExplainWhere('between', (await q.between(2500, 7500)) as SQL)
171+
})
172+
173+
it('records jsonb operator plan shapes', async () => {
174+
for (const [name, build] of [
175+
['jsonbPathQueryFirst', () => q.jsonbPathQueryFirst('$.idx')],
176+
['jsonbGet', () => q.jsonbGet('$.idx')],
177+
['jsonbPathExists', () => q.jsonbPathExists('$.idx')],
178+
] as const) {
179+
await tryExplainWhere(name, await build())
180+
}
181+
})
182+
183+
it('records ORDER BY plan shape (asc / desc)', async () => {
184+
for (const [name, sql] of [
185+
['asc', q.asc()],
186+
['desc', q.desc()],
187+
] as const) {
188+
try {
189+
const plan = await explainOrderBy(sql)
190+
recordObservation(`orderBy_${name}`, plan)
191+
} catch (err) {
192+
recordError(`orderBy_${name}`, err)
193+
}
194+
}
195+
})
196+
})

0 commit comments

Comments
 (0)