ci: publish CI images to internal registry first#4002
Draft
realFlowControl wants to merge 8 commits into
Draft
Conversation
|
realFlowControl
changed the title
- Update .gitlab/ci-images.yml to change the default CI_REGISTRY to registry.ddbuild.io and target the ddbuild registry path registry.ddbuild.io/ci/dd-trace-php/dd-trace-ci. - Make docker logins dynamic to support local builds, Docker Hub logins, and AWS ECR logins depending on the target registry server. - Bypass runner credential helper issues in Linux container environments by resetting ~/.docker/config.json. - Make registry and base image names fully configurable in docker-compose.yml and Dockerfiles, allowing parent base images to be dynamically resolved from ddbuild during child compilation steps.
- Update all GitLab CI generator scripts (.gitlab/generate-*.php) to use internal CI images from registry.ddbuild.io/ci/dd-trace-php/dd-trace-ci instead of pulling from Docker Hub via the mirror path. - This ensures test jobs use the newly compiled images directly from our project's ECR registry namespace.
- Add a new 'ci-publish' stage to .gitlab-ci.yml. - Implement 4 parallel matrix trigger jobs in .gitlab/ci-images.yml (Publish CentOS, Publish Bookworm, Publish Alpine, and Publish Windows) to run automatically after their respective build jobs succeed. - Each trigger calls the DataDog/public-images pipeline, passing the corresponding internal ddbuild ECR image as source and targeting public Docker Hub as destination under the exact same tag.
- Update all occurrences of bookworm-8 and shared-ext-8 to bookworm-9 and shared-ext-9 globally across .gitlab CI test generators, .gitlab/ci-images.yml, and .github workflows. - Update BOOKWORM_VERSION from 8 to 9 in tooling/bin/build-debug-artifact to ensure local debug builds pull and compile with the new version.
- Export MAKEFLAGS=-j at the top of build-extensions.sh. - This forces all underlying make invocations triggered by pecl install (including the heavy single-threaded gRPC, MongoDB, and parallel builds) to compile in parallel, drastically reducing build times on multi-core runner environments.
- Remove obsolete CI_REGISTRY, CI_REGISTRY_USER, and CI_REGISTRY_TOKEN from .gitlab/ci-images.yml. - Remove all complex, dynamic ECR/Docker Hub login shell blocks and AWS CLI installations from CentOS, Alpine, Bookworm, and Windows build jobs. - Rely entirely on the runner's native, pre-configured credentials for registry.ddbuild.io, significantly simplifying the pipeline configuration.
- Clean up dockerfiles/ci/README.md to document the new automated, secure internal ECR build flow. - Clarify that project collaborators no longer need to configure Personal Access Tokens (PATs) or credentials when building CI images. - Document how to trigger the manual sync to the public Docker Hub registry via downstream triggers in the 'ci-publish' stage.
realFlowControl
force-pushed
the
florian/ci-images
branch
10 times, most recently
from
0464d2c to
4840f7c
Compare
The image list (PHP versions and tags) is derived from the docker-compose.yml
+ .env files in each dockerfiles/ci/<os>/ dir (single source of truth).
.gitlab/generate-ci-images.php renders .gitlab/ci-images.yml.tpl, emitting per
Linux OS:
- <OS> build : one matrix job over PHP version; 'docker buildx bake
--no-cache --pull --push' builds both arches (x-bake
platforms from compose) on the amd64 runner's managed ci
builder and pushes a multi-arch manifest to
registry.ddbuild.io
- <OS> publish:<v>: manual mirror to Docker Hub via DataDog/public-images,
dependency-free (just syncs whatever is in the internal
registry)
Static preamble + Windows jobs live in .gitlab/ci-images.static.yml (Windows
is single-arch). The generator runs in generate-templates and is triggered as
a child pipeline via the manual 'ci-images' job; the old .gitlab/ci-images.yml
local include is removed.
realFlowControl
force-pushed
the
florian/ci-images
branch
2 times, most recently
from
433f153 to
228828a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When building CI docker images, this PR changes the process to:
registry.ddbuild.io(Datadog internal container registry)public-imagesdownstream job to magically sync those images to Docker Hub for usage with GitHub CI and external contributorsWins
Reviewer checklist