[fix](streaming) reset offset provider state when ALTER JOB switches offset to initial - #66563
[fix](streaming) reset offset provider state when ALTER JOB switches offset to initial#66563maks3201 wants to merge 1 commit into
Conversation
…offset to initial When an operator runs PAUSE JOB → ALTER JOB ... offset="initial" → RESUME JOB to force a fresh CDC snapshot, the JdbcSourceOffsetProvider still holds the old binlog position and split progress in memory. On the next scheduler tick, replayIfNeed() restores the stale binlog offset from offsetProviderPersist, noMoreSplits() returns true (believing the snapshot phase is already complete), and getNextOffset() returns the expired binlog position. The job dispatches a task from the stale offset instead of starting the expected fresh snapshot. The operator believes they reset the job; they did not. Fix: add SourceOffsetProvider.resetToInitialState() (default no-op) and implement it in JdbcSourceOffsetProvider to clear ALL cached state: currentOffset, binlogOffsetPersist, endBinlogOffset, tableSchemas, chunkHighWatermarkMap, remainingSplits, finishedSplits, committedSplitProgress, cdcSplitProgress, hasMoreData, and boundBackendId. Call it from alterJob() when the ALTER explicitly sets offset to "initial" or "snapshot", and null out offsetProviderPersist so that replayIfNeed() takes the fresh-start branch. Note: resetToInitialState() is distinct from the existing clearSnapshotState(). clearSnapshotState() is called during the NORMAL snapshot-to-binlog transition (inside updateOffset) and deliberately preserves currentOffset, binlogOffsetPersist, and endBinlogOffset — fields the incoming binlog phase depends on. resetToInitialState() is a strict superset that clears everything, because its purpose is a complete fresh start, not a phase transition.
|
This bug is present in released 4.1.x (confirmed on 4.1.2). If the fix is accepted, the |
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
CDC jobs can only modify the position in the PAUSED state, and the offset only accepts the exact position specified in the JSON above; initial, snapshot, earliest, or latest offsets are not accepted. I remember there being this limitation. Are you sure you can modify it successfully? https://github.com/apache/doris/blob/master/regression-test/suites/job_p0/streaming_job/cdc/test_streaming_postgres_job_special_offset.groovy#L216-L220 |
|
@JNSimba you're right, thank you for catching this. I re-checked the code and confirmed I mixed up two different callers from my local branch. The one I upstreamed here (on ALTER) is dead code on master. The real trigger is internal, it's a fallback that runs when a streaming task fails with a specific error and the job needs to re snapshot automatically, that path is reachable and it's where this reset logic actually belongs. I'm going to close this PR and move the reset logic into a follow up that includes the real trigger, so the fix is tied to code that actually executes. Thanks again for reviewing, sorry for the noise. |
|
Closing per discussion above - code path unreachable on master, will resubmit tied to the actual trigger. |
Proposed changes
Related PR: #66559 (independent fix in the same subsystem)
Problem Summary
When an operator runs
PAUSE JOB→ALTER JOB ... offset='initial'→RESUME JOBto force a fresh CDC snapshot,JdbcSourceOffsetProviderstill holds the old binlog position and split progress in memory. On the next scheduler tick the job dispatches a task from the stale offset instead of starting a fresh snapshot. The operator believes they reset the job; they did not.Why this is not just
clearSnapshotState(): The existingclearSnapshotState()method is invoked during the normal snapshot-to-binlog transition insideupdateOffset(). It deliberately preservescurrentOffset,binlogOffsetPersist, andendBinlogOffsetbecause the incoming binlog phase depends on them. Our newresetToInitialState()is a strict superset: it callsclearSnapshotState()for the split/progress subset, then also clears the binlog-phase fields (currentOffset,binlogOffsetPersist,endBinlogOffset,tableSchemas,hasMoreData,boundBackendId), because the intent is a complete fresh start — not a phase transition within an active job.Fix
SourceOffsetProvider.resetToInitialState()as adefaultno-op method on the interface (S3 and other providers are unaffected).JdbcSourceOffsetProvider: clear all cached split and binlog state (11 fields — see exhaustive list in the Javadoc).StreamingInsertJob.alterJob(): when the ALTER explicitly setsoffsettoinitialorsnapshot, callresetToInitialState()and null outoffsetProviderPersistso thatreplayIfNeed()takes the fresh-start branch.After the reset, the next
handlePendingState()tick sees empty state and initiates a full snapshot — identical to a brand-new job.Field-by-field reset audit
remainingSplitsfinishedSplitschunkHighWatermarkMapcommittedSplitProgresscdcSplitProgressnoMoreSplits()would return wrong answercurrentOffsetgetNextOffsetfrom returning stale binlogbinlogOffsetPersistreplayIfNeedfrom restoring old offsetendBinlogOffsettableSchemashasMoreDataboundBackendIdjobIdsourceTypesourcePropertiessnapshotParallelismcloudClustercachedSyncTablesIssue Number
Closes #66562
Checklist
resetToInitialStatecovering binlog-phase reset, snapshot-phase reset, idempotency,noMoreSplitscorrectness, and configuration preservation).cc @JNSimba — you maintain the streaming job subsystem; this touches the offset provider interface and the
alterJob()flow.Please also consider applying
dev/4.1.xlabel for backport — this bug affects any 4.1.x user attempting manual CDC recovery via ALTER JOB.