Skip to content

feat(connectors): ship connector plugins in the iggy-connect docker image - #3658

Open
kriti-sc wants to merge 14 commits into
apache:masterfrom
kriti-sc:#3614-add-connectors-to-docker-image
Open

feat(connectors): ship connector plugins in the iggy-connect docker image#3658
kriti-sc wants to merge 14 commits into
apache:masterfrom
kriti-sc:#3614-add-connectors-to-docker-image

Conversation

@kriti-sc

@kriti-sc kriti-sc commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Closes #3614

Rationale

Dynamically inject plugins to the apache/iggy-connect image to reduce friction for testing out connectors.

What changed?

This change publishes the image in two flavors:

Fat (edge, x.y.z, latest): the runtime plus every connector plugin baked into /usr/local/lib, which is already on the runtime's plugin search path. A config can load a bundled plugin by file name with no path. This is the new default.
Slim (edge-slim, x.y.z-slim, latest-slim): the runtime binary only, i.e. today's image, for deployments that bring their own plugins.

To this end:

  1. Single source of truth for the plugin set. scripts/ci/connector-plugins.sh derives the plugin list from cargo metadata (every cdylib under core/connectors/). The edge tarball build, the Docker fat stage, the license gate, and the release notes all consume it, so a newly added connector is picked up everywhere with no hand-edited list.
  2. Dockerfile. The planner derives the plugin set; cook and build compile the plugin cdylibs alongside the runtime; a new runtime-fat stage copies the .so files in. The slim runtime stage is unchanged.
  3. Publish pipeline. The docker-buildx action gained a target input, and components can declare flavors in publish.yml. Each flavor keeps its own per-arch digest artifacts so the manifest step never merges flavors into one image, and only the default flavor creates the git release tag.
  4. Licensing. The third-party license gate now validates the runtime plus every plugin dependency closure (derived list), so a new connector is license-validated on the PR that adds it. The fat image ships a LICENSE-binary-fat covering the plugins; slim keeps the runtime-only manifest.
  5. Edge gate. The connectors image now refreshes :edge on any change under core/connectors/.
  6. Docs. Connectors README documents the flavors and zero-config plugin loading with a docker run quickstart; edge release notes derive the plugin list.

Local Execution

  • Built images locally. The fat image is 357 MB, and the build took 20 mins on a 4 CPU/12 GB Docker VM. Slim image is 163 MB. This validates that image will build comfortably on Github CI runners which are 4 CPU/16 GB.
  • Pre-commit hooks ran.

AI Usage

  1. Which tools? (e.g., GitHub Copilot, Claude, ChatGPT) Claude
  2. Scope of usage? (e.g., autocomplete, generated functions, entire implementation) First developed the design and pass criteria. Then, generated functions and debugging with Claude.
  3. How did you verify the generated code works correctly? Built the image locally and checked if all plugins present in the image.
  4. Can you explain every line of the code if asked? Yes.

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.28%. Comparing base (c59ca5a) to head (803b4ca).

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3658       +/-   ##
=============================================
- Coverage     76.95%   18.28%   -58.67%     
- Complexity     1021     1316      +295     
=============================================
  Files          1366     1376       +10     
  Lines        176130   150597    -25533     
  Branches     146173   119836    -26337     
=============================================
- Hits         135536    27544   -107992     
- Misses        36718   122424    +85706     
+ Partials       3876      629     -3247     
Components Coverage Δ
Rust Core 1.82% <ø> (-74.25%) ⬇️
Java SDK 66.40% <ø> (+3.18%) ⬆️
C# SDK 63.70% <ø> (-12.32%) ⬇️
Python SDK 89.98% <ø> (ø)
PHP SDK 82.97% <ø> (ø)
Node SDK 96.33% <ø> (ø)
Go SDK 69.13% <ø> (ø)
see 883 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kriti-sc kriti-sc changed the title #3614 add connectors to docker image feat(connectors): ship connector plugins in the iggy-connect docker image Jul 12, 2026
@kriti-sc
kriti-sc marked this pull request as ready for review July 19, 2026 18:18
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 19, 2026
@kriti-sc

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added the S-stale Inactive issue or pull request label Jul 30, 2026
@github-actions github-actions Bot removed the S-stale Inactive issue or pull request label Jul 30, 2026
Comment thread core/connectors/runtime/Dockerfile
Comment thread .github/actions/utils/docker-buildx/action.yml
Comment thread .github/workflows/_build_rust_artifacts.yml
@hubcio hubcio added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 3, 2026
A stage resolves to its final filesystem, so the slim runtime's
COPY --from=builder forced the whole builder stage: every -slim build
compiled the plugin cdylibs and ran cargo-about over their dependency
closures, then discarded all of it.

Split the builder into two siblings on a shared toolchain stage, one per
flavor, each cooking only its own package set and generating only the
license manifest its image ships. Keeping them siblings rather than
chaining the plugin build on top of the runtime build is what lets cargo
chef cook the plugin dependency closure into a pre-source layer, which
is the only part of it the registry cache can hold on CI.

The two final stages share a runtime-base for the same reason in the
other direction: with runtime-fat FROM runtime, a fat build pulls the
slim branch in and compiles the runtime binary twice.
…plit-fat-builder

fix(connectors): stop the slim image build from compiling plugins
@kriti-sc

Copy link
Copy Markdown
Contributor Author

/ready

@kriti-sc
kriti-sc requested a review from numinnex August 10, 2026 17:50
@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: ship connector plugins in the apache/iggy-connect docker image

3 participants