Skip to content

fix: handle missing checkpoint token from service#571

Draft
ayushiahjolia wants to merge 1 commit into
mainfrom
fix/checkpoint-token-keep-last
Draft

fix: handle missing checkpoint token from service#571
ayushiahjolia wants to merge 1 commit into
mainfrom
fix/checkpoint-token-keep-last

Conversation

@ayushiahjolia

@ayushiahjolia ayushiahjolia commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Issue #, if available:
#43
#565

Description of changes:
Standardizes how the SDK handles a missing/empty CheckpointToken from the Durable Functions API, and makes the checkpoint loop treat an absent response token as a first-class terminal signal. Previously the token was defaulted to an empty string and could be forwarded to checkpoint_durable_execution, where boto3 rejects a zero-length value with a ParamValidationError — crashing the background checkpoint thread.

Background -
The service omits CheckpointToken from a checkpoint response in exactly one case: the terminal checkpoint that ends the execution (an EXECUTION op with action SUCCEED/FAIL). Every non-terminal checkpoint — including empty/no-op ones — returns a fresh token, and the invocation event always carries a seed token. So an absent token legitimately means "the execution is over"; anywhere else it is an invalid response, and an empty token must never be sent back.

Situation Behavior
Non-terminal checkpoint, token present rotate to it (unchanged)
Execution end, small result SDK checkpoints no terminal op; backend finalizes from the invocation response — no null token seen (unchanged)
Execution end, large result / failure (SDK checkpoints the terminal op) recognized as completion: loop stops, queued orphan work settled, no further request issued
Non-terminal checkpoint returns no token invalid response → retryable CheckpointError → re-drive
Empty/None token about to be sent (e.g. malformed seed) retryable CheckpointError / GetExecutionStateError; no boto crash
parallel/map early completion (min_successful, failure tolerance) with branches still running racing branch checkpoints settled as orphaned; later ones rejected fast; nothing sent post-completion
Nested concurrency + timed waits, orphaned after completion resubmitter stops cleanly, no timer-thread crash

Every token anomaly resolves to a retryable error that re-drives with a fresh token; the one legitimate absent token (execution completion) is handled silently, and an empty token is never transmitted to the service.

Testing - Added unit tests

NOTE: Intentionally diverges from the JS/Java SDKs; they should be updated to match (guard empty-token sends, fail on non-terminal null).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@ayushiahjolia
ayushiahjolia force-pushed the fix/checkpoint-token-keep-last branch 2 times, most recently from 4467579 to 5281407 Compare July 22, 2026 23:51
@ayushiahjolia ayushiahjolia changed the title fix: keep last checkpoint token when omitted fix: validate checkpoint token and reject invalid responses Jul 22, 2026
@ayushiahjolia
ayushiahjolia force-pushed the fix/checkpoint-token-keep-last branch from 5281407 to f0793df Compare July 22, 2026 23:53
@ayushiahjolia ayushiahjolia changed the title fix: validate checkpoint token and reject invalid responses fix: handle missing checkpoint token from service Jul 22, 2026
@ayushiahjolia
ayushiahjolia force-pushed the fix/checkpoint-token-keep-last branch from f0793df to 7e09c90 Compare July 23, 2026 00:11
@ayushiahjolia
ayushiahjolia force-pushed the fix/checkpoint-token-keep-last branch from 7e09c90 to f3f1efb Compare July 23, 2026 17:31
@ayushiahjolia
ayushiahjolia force-pushed the fix/checkpoint-token-keep-last branch from f3f1efb to f77354e Compare July 23, 2026 20:22
@ayushiahjolia
ayushiahjolia marked this pull request as ready for review July 23, 2026 21:12
queued_op = QueuedOperation(operation_update, completion_event)

# Enqueue the wrapper object (operation_update can be None for empty checkpoints)
self._checkpoint_queue.put(queued_op)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we call _checkpoint_queue.put here, the condition self._execution_completed.is_set() might have been changed by another thread since previously checked.

@ayushiahjolia
ayushiahjolia marked this pull request as draft July 24, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants