feat(snapshot): parallel PK-range chunks with per-chunk resume - #66
Merged
Conversation
Extract JPA entities/repositories/migrations into a syncflow-persistence module and rework full-table snapshot into parallel PK-range chunk workers. Parallel snapshot (F15): - SnapshotExecutor splits each mapped table into PK ranges, runs up to `parallelism` worker virtual threads over a shared work queue, each owning one exclusive JDBC connection clone (no cross-thread Connection sharing). - Per-chunk checkpoint rows (chunk_index column, V15 migration) so resume continues each chunk from its own keyed cursor. - Chunk bounds computed in BigDecimal to avoid long overflow on near-full BIGINT domains; offset-based sources paginate from a per-chunk counter so parallel collections no longer slip or re-read rows. - Snapshot writes upsert on a mapped destination PK so a resume over already committed rows is idempotent; MySqlWriter emits MySQL ON DUPLICATE KEY UPDATE (Postgres ON CONFLICT was being emitted for MySQL destinations). - Writer rollback discards the residual buffer so a failed run cannot leak rows into the next pipeline's destination. - Cancel/complete/fail terminal states serialized under one lock so a cancel cannot be overridden by a completing worker; cancellation flag is not released early. Multi-tenancy / reliability fixes: - CheckpointStore takes the tenant id explicitly instead of reading the unset worker-thread ThreadLocal, which keyed every chunk's checkpoints to TenantId.DEFAULT. - DistributedLockService guards snapshot start (S2) so two pods cannot start the same pipeline concurrently.
…ort review
Parallel-snapshot correctness:
- cancel() is now a no-op on a terminal job (COMPLETED/FAILED/CANCELLED) so a
late cancel can no longer flip a fully-committed snapshot to CANCELLED.
- Progress publish re-checks cancellation under progressLock, so a cancel that
lands mid-loop can no longer re-persist a RUNNING-status row over CANCELLED
and strand a zombie RUNNING job.
- Offset-driven resume (Mongo, PK-less JDBC): the per-chunk batch counter now
starts one below the first read's ordinal, so the next page advances the
offset instead of re-reading (duplicating) the first resumed batch.
- Chunk workers return before the first read if already cancelled, closing the
window where a cancel landing before the loop-top check still wrote and
auto-committed the chunk's first page.
Connector / writer:
- coerceCursor only coerces when the chunk bounds are real Numbers. A whole
chunk is also used for text/uuid/date PKs, so coercing an all-digit cursor
to Long there broke TEXT keyseek ('00123' -> 123); removed isNumericCursor.
- coerceCursor's BigDecimal branch now guards NumberFormatException like the
Long branch.
- writeBatch flushes buffered rows that were staged as an UPSERT, so an
upsert batch no longer silently degrades to a plain INSERT when a plain
insert follows on the same table+columns.
- MySQL upsert uses VALUES(col) instead of the 8.0.19+ row-alias form, which
fails on MariaDB and MySQL < 8.0.19; corrects the portability doc comment.
Config / tests:
- maxChunks capped at 1024; a huge value could allocate ~1GB of chunk ranges.
- JdbcBatchWriterTest updated for the VALUES(col) MySQL shape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reworks the full-table snapshot path into parallel PK-range chunk workers and extracts the JPA persistence layer into a
syncflow-persistencemodule.Parallel snapshot (F15)
parallelismworker virtual threads over a shared work queue. Each worker owns one exclusive JDBC connection clone for its lifetime, so no two in-flight tasks ever share ajava.sql.Connection(the prior round-robin clone assignment could run two tasks on the same connection).snapshot_checkpoints.chunk_index, migrationV15) — resume continues each chunk from its own keyed cursor.cursorWithinRangeguards a legacy whole-table checkpoint from being resumed out-of-range.AbstractJdbcSnapshotConnector.rangeChunks) to eliminatelongoverflow on near-fullBIGINTdomains (max + 1wrapping, negativerangeSpancollapsing tostep = 1). Empirically verified acrossLong.MIN_VALUE..Long.MAX_VALUE.MySqlWriter.upsertSql(ON DUPLICATE KEY UPDATE) — the base emitted PostgresON CONFLICT, which a MySQL destination rejected.rollback()now discards the residual buffer, so a failed run can't leak buffered rows into the next pipeline's destination.Multi-tenancy / reliability
CheckpointStoretakes the tenant id explicitly instead of reading the worker-threadThreadLocal(never set on pool/virtual threads), which keyed every chunk's checkpoints toTenantId.DEFAULT— a second tenant could resume off the first tenant's cursor.DistributedLockServiceguards snapshotstart(S2) so two pods can't start the same pipeline concurrently; a staleRUNNINGrow left by a crashed pod is treated as startable.Module split
JPA entities, Spring Data repositories, and Flyway migrations (V1–V15) move from
syncflow-apiinto a newsyncflow-persistencemodule (@EntityScan/@EnableJpaRepositoriesviaPersistenceConfig,scanBasePackages = "com.syncflow"picks it up).Validation
./gradlew test— 758 unit tests green../gradlew test -Dtests.integration=true— 759 tests green incl.SnapshotIntegrationTest(5),ChunkedSnapshotCorrectnessTest(2, real Postgres),DatabaseMigrationValidationTest(11, V1–V15),CdcIntegrationTest(11),KafkaIntegrationTest(4).