Skip to content

Harden pgvector 10M production evidence#22

Draft
CaspianG wants to merge 9 commits into
mainfrom
benchmark/pgvector-10m-strict
Draft

Harden pgvector 10M production evidence#22
CaspianG wants to merge 9 commits into
mainfrom
benchmark/pgvector-10m-strict

Conversation

@CaspianG

Copy link
Copy Markdown
Owner

Summary

  • add bounded COPY ingest and halfvec storage for large pgvector runs
  • validate remote row counts, index presence, checkpoint resume, and explicit prewarming
  • add IVFFlat and binary-HNSW candidate profiles with original-vector reranking
  • decouple durable ingest checkpoints from rebuildable index tuning

Verification

  • 658 passed, 5 skipped before the final tuning additions
  • 32 passed, 2 skipped on the focused streaming/workflow suite
  • real pgvector 0.8.1 service smokes pass for halfvec, IVFFlat, binary HNSW, COPY, prewarm, and complete resume

Current evidence boundary

The 10M corpus and candidate indexes were exercised on a real Docker/WSL2 PostgreSQL service. The strict 10M result is intentionally not checked in yet: the monolithic profiles do not simultaneously meet recall >= 0.95 and p99 <= 100 ms. The next iteration is sharded pgvector fanout over smaller per-shard graphs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant