Skip to content

feat(connectors): add source batch acknowledgments - #3855

Open
rohankumardubey wants to merge 2 commits into
apache:masterfrom
rohankumardubey:fix/connectors-source-ack
Open

feat(connectors): add source batch acknowledgments#3855
rohankumardubey wants to merge 2 commits into
apache:masterfrom
rohankumardubey:fix/connectors-source-ack

Conversation

@rohankumardubey

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Relates to #3635

Rationale

Establishes the generic source acknowledgment contract before migrating PostgreSQL source behavior and adding the kill-server regression test in a follow-up PR.

What changed?

Source plugins could poll another batch without knowing whether the previous batch was delivered, allowing cursors and persisted state to advance after failed sends.

The source FFI now carries batch IDs and supports ACK/NACK results. The SDK permits one in-flight batch, while the runtime ACKs only after both the Iggy send and state persistence succeed. Send failures, timeouts, and state-save failures produce a NACK.

Local Execution

  • Passed cargo fmt --all
  • Passed clippy with warnings denied for the connector SDK, runtime, and all source plugins
  • Passed all connector SDK, runtime, and source-plugin tests
  • Built the runtime and all source dynamic libraries successfully
  • Confirmed the new iggy_source_batch_result symbol is exported
  • cargo-sort, markdownlint, and taplo were not available locally
  • Pre-commit hooks did not run because no hook was installed

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 10, 2026
@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.80%. Comparing base (d35000d) to head (38e7853).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
core/connectors/sdk/src/source.rs 88.88% 27 Missing and 4 partials ⚠️
core/connectors/runtime/src/source.rs 95.83% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3855       +/-   ##
=============================================
- Coverage     76.95%   49.80%   -27.15%     
- Complexity     1021     1316      +295     
=============================================
  Files          1366     1377       +11     
  Lines        176130   153298    -22832     
  Branches     146173   122536    -23637     
=============================================
- Hits         135543    76352    -59191     
- Misses        36714    73785    +37071     
+ Partials       3873     3161      -712     
Components Coverage Δ
Rust Core 42.43% <90.90%> (-33.65%) ⬇️
Java SDK 66.40% <ø> (+3.18%) ⬆️
C# SDK 60.38% <ø> (-15.64%) ⬇️
Python SDK 89.98% <ø> (ø)
PHP SDK 82.97% <ø> (ø)
Node SDK 96.33% <ø> (ø)
Go SDK 69.13% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/runtime/src/main.rs 85.81% <100.00%> (+0.10%) ⬆️
core/connectors/runtime/src/manager/source.rs 93.50% <100.00%> (+0.03%) ⬆️
core/connectors/sdk/src/lib.rs 57.14% <100.00%> (+0.96%) ⬆️
core/connectors/runtime/src/source.rs 83.73% <95.83%> (+3.06%) ⬆️
core/connectors/sdk/src/source.rs 86.87% <88.88%> (+3.20%) ⬆️

... and 477 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant