Skip to content

feat(snapshot): parallel PK-range chunks with per-chunk resume - #66

Merged
lekhrocks merged 5 commits into
mainfrom
fix/cdc-completion
Aug 21, 2026
Merged

feat(snapshot): parallel PK-range chunks with per-chunk resume#66
lekhrocks merged 5 commits into
mainfrom
fix/cdc-completion

Conversation

@lekhrocks

Copy link
Copy Markdown
Owner

What

Reworks the full-table snapshot path into parallel PK-range chunk workers and extracts the JPA persistence layer into a syncflow-persistence module.

Parallel snapshot (F15)

  • SnapshotExecutor splits each mapped table into PK ranges and runs up to parallelism worker virtual threads over a shared work queue. Each worker owns one exclusive JDBC connection clone for its lifetime, so no two in-flight tasks ever share a java.sql.Connection (the prior round-robin clone assignment could run two tasks on the same connection).
  • Per-chunk resume checkpoints (snapshot_checkpoints.chunk_index, migration V15) — resume continues each chunk from its own keyed cursor. cursorWithinRange guards a legacy whole-table checkpoint from being resumed out-of-range.
  • Chunk bounds computed in BigDecimal (AbstractJdbcSnapshotConnector.rangeChunks) to eliminate long overflow on near-full BIGINT domains (max + 1 wrapping, negative rangeSpan collapsing to step = 1). Empirically verified across Long.MIN_VALUE..Long.MAX_VALUE.
  • Offset-based sources (Mongo, PK-less JDBC tables) paginate from a per-chunk batch counter instead of the shared global counter, so parallel collections no longer slip or re-read rows.
  • Snapshot writes upsert when a destination PK is mapped, so a resume that re-reads already-committed rows is idempotent. Added MySqlWriter.upsertSql (ON DUPLICATE KEY UPDATE) — the base emitted Postgres ON CONFLICT, which a MySQL destination rejected.
  • Writer rollback() now discards the residual buffer, so a failed run can't leak buffered rows into the next pipeline's destination.
  • Terminal states serialized under one lock: cancel/complete/fail cannot race, and the cancellation flag is not released early (a completing worker can't override a cancelled snapshot).

Multi-tenancy / reliability

  • CheckpointStore takes the tenant id explicitly instead of reading the worker-thread ThreadLocal (never set on pool/virtual threads), which keyed every chunk's checkpoints to TenantId.DEFAULT — a second tenant could resume off the first tenant's cursor.
  • DistributedLockService guards snapshot start (S2) so two pods can't start the same pipeline concurrently; a stale RUNNING row left by a crashed pod is treated as startable.

Module split

JPA entities, Spring Data repositories, and Flyway migrations (V1–V15) move from syncflow-api into a new syncflow-persistence module (@EntityScan/@EnableJpaRepositories via PersistenceConfig, scanBasePackages = "com.syncflow" picks it up).

Validation

  • ./gradlew test — 758 unit tests green.
  • ./gradlew test -Dtests.integration=true — 759 tests green incl. SnapshotIntegrationTest (5), ChunkedSnapshotCorrectnessTest (2, real Postgres), DatabaseMigrationValidationTest (11, V1–V15), CdcIntegrationTest (11), KafkaIntegrationTest (4).

Extract JPA entities/repositories/migrations into a syncflow-persistence
module and rework full-table snapshot into parallel PK-range chunk workers.

Parallel snapshot (F15):
- SnapshotExecutor splits each mapped table into PK ranges, runs up to
  `parallelism` worker virtual threads over a shared work queue, each owning
  one exclusive JDBC connection clone (no cross-thread Connection sharing).
- Per-chunk checkpoint rows (chunk_index column, V15 migration) so resume
  continues each chunk from its own keyed cursor.
- Chunk bounds computed in BigDecimal to avoid long overflow on near-full
  BIGINT domains; offset-based sources paginate from a per-chunk counter so
  parallel collections no longer slip or re-read rows.
- Snapshot writes upsert on a mapped destination PK so a resume over already
  committed rows is idempotent; MySqlWriter emits MySQL ON DUPLICATE KEY
  UPDATE (Postgres ON CONFLICT was being emitted for MySQL destinations).
- Writer rollback discards the residual buffer so a failed run cannot leak
  rows into the next pipeline's destination.
- Cancel/complete/fail terminal states serialized under one lock so a cancel
  cannot be overridden by a completing worker; cancellation flag is not
  released early.

Multi-tenancy / reliability fixes:
- CheckpointStore takes the tenant id explicitly instead of reading the
  unset worker-thread ThreadLocal, which keyed every chunk's checkpoints to
  TenantId.DEFAULT.
- DistributedLockService guards snapshot start (S2) so two pods cannot start
  the same pipeline concurrently.
…ort review

Parallel-snapshot correctness:
- cancel() is now a no-op on a terminal job (COMPLETED/FAILED/CANCELLED) so a
  late cancel can no longer flip a fully-committed snapshot to CANCELLED.
- Progress publish re-checks cancellation under progressLock, so a cancel that
  lands mid-loop can no longer re-persist a RUNNING-status row over CANCELLED
  and strand a zombie RUNNING job.
- Offset-driven resume (Mongo, PK-less JDBC): the per-chunk batch counter now
  starts one below the first read's ordinal, so the next page advances the
  offset instead of re-reading (duplicating) the first resumed batch.
- Chunk workers return before the first read if already cancelled, closing the
  window where a cancel landing before the loop-top check still wrote and
  auto-committed the chunk's first page.

Connector / writer:
- coerceCursor only coerces when the chunk bounds are real Numbers. A whole
  chunk is also used for text/uuid/date PKs, so coercing an all-digit cursor
  to Long there broke TEXT keyseek ('00123' -> 123); removed isNumericCursor.
- coerceCursor's BigDecimal branch now guards NumberFormatException like the
  Long branch.
- writeBatch flushes buffered rows that were staged as an UPSERT, so an
  upsert batch no longer silently degrades to a plain INSERT when a plain
  insert follows on the same table+columns.
- MySQL upsert uses VALUES(col) instead of the 8.0.19+ row-alias form, which
  fails on MariaDB and MySQL < 8.0.19; corrects the portability doc comment.

Config / tests:
- maxChunks capped at 1024; a huge value could allocate ~1GB of chunk ranges.
- JdbcBatchWriterTest updated for the VALUES(col) MySQL shape.
@lekhrocks
lekhrocks merged commit 6c4e5f5 into main Aug 21, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant