Gitlab Image Build to ECIR

docs/services/gitlab/cicd.md

@@ -0,0 +1,66 @@
+# Continuous Integration and Continuous Deployment (CI/CD) with EIDF GitLab
+
+A repository containing some common GitLab CI/CD configurations and relevant examples is maintained at [GitLab CI/CD Examples](https://gitlab.eidf.ac.uk/Liz/cicd-examples)
+
+## Harbor Integration and the ECIR
+
+We recommend that users setup Harbor Integration to automate the use of the EIDF Container Image Registry (ECIR) as the pull and push location for container images. The ECIR is recommended as it is part of EIDF infrastructure and hence provides speed over external registries as well as a number of built in Security features such as SBOM and security scanning via Trivy. More documentation around the ECIR can be found in the [ECIR documentation](../registry/overview.md) and the [ECIR working with documentation](../registry/working-with.md).
+
+We assume users have setup an ECIR project and Push Robot which are detailed in ECIR Working with sections [Creating a Project Repository](../registry/working-with.md#creating-a-project-repository) and [Creating Robot Accounts for the Registry](../registry/working-with.md#creating-robot-accounts-for-the-registry) respectively.
+
+Users should follow the [GitLab documentation on Harbor Integration](https://docs.gitlab.com/user/project/integrations/harbor/), making note of the following values to be input:
+
+- Harbor URL: `https://registry.eidf.ac.uk`
+- Harbor Project Name: Typically the same as the EIDF project you are working in e.g. `eidf123`
+- Username: The username of the Push Robot for your ECIR project, typically has a name like `eidf123-push-robot`
+- Password: The secret key of the push robot
+
+## Using an ECIR image in GitLab CI/CD
+
+The ECIR is a private registry and so the use of images stored in the ECIR requires authentication. A number of methods for this are documented in the [GitLab documentation](https://docs.gitlab.com/ci/docker/using_docker_images/#access-an-image-from-a-private-container-registry). We recommend setting up a variable defining the DOCKER_AUTH_CONFIG as described in the GitLab documentation above. When making use of the Harbor Integration this variable should be set as follows:
+
+```.gitlab-ci.yml
+variables:
+  DOCKER_AUTH_CONFIG: "{\"auths\":{\"$HARBOR_HOST\":{\"username\":\"$HARBOR_USERNAME\",\"password\":\"$HARBOR_PASSWORD\"}}}"
+```
+
+This will then authorise you to make use of the ECIR in your CI/CD pipelines, for example by pulling an image from the ECIR as the image for a job:
+
+```.gitlab-ci.yml
+
+variables:
+  DOCKER_AUTH_CONFIG: "{\"auths\":{\"$HARBOR_HOST\":{\"username\":\"$HARBOR_USERNAME\",\"password\":\"$HARBOR_PASSWORD\"}}}"
+
+my-job:
+  image: $HARBOR_HOST/$HARBOR_PROJECT/hello-world:latest
+  script:
+    - echo "This job uses an image from the ECIR"
+```
+
+## Building a Container Image with GitLab CI/CD
+
+Automation of building of container images is possible using GitLab CI/CD in the default GitLab Instance runner.
+
+Images can be built from a Dockerfile using [BuildKit](https://docs.docker.com/build/buildkit/). This requires users to include a stage in the .gitlab-ci.yml using the following template, taking special note of replacing parts in `< >` and if not using Harbor integration ensuring the Environment Variables are set correctly:
+
+```.gitlab-ci.yml
+build-rootless:
+  image:
+    name: moby/buildkit:rootless
+  stage: build
+  variables:
+    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
+    REG_IMAGE: "$HARBOR_HOST/$HARBOR_PROJECT/<image-name>:$CI_COMMIT_SHA"
+  before_script:
+    - mkdir -p ~/.docker
+    - echo "{\"auths\":{\"$HARBOR_HOST\":{\"username\":\"$HARBOR_USERNAME\",\"password\":\"$HARBOR_PASSWORD\"}}}" > ~/.docker/config.json
+  script:
+    - |
+      buildctl-daemonless.sh build \
+        --frontend dockerfile.v0 \
+        --local context=. \
+        --local dockerfile=<. or Dockerfile path to use> \
+        --output type=image,name=$REG_IMAGE,push=true
+```
+
+This will build an image using the specified Dockerfile and push it to the specified registry. The template is lightly adapted from the GitLab documentation on [building and pushing Docker images with BuildKit](https://docs.gitlab.com/ee/ci/docker/using_docker_buildx.html#build-and-push-docker-images-with-buildkit) where more examples and details can be found.

docs/services/gitlab/quickstart.md

@@ -2,7 +2,7 @@
 
 ## Accessing
 
-Access the EIDF gitlab in your browser by opening [https://gitlab.eidf.ac.uk/](https://gitlab.eidf.ac.uk/). You must be a member of an active EIDF project and have a user account to use the EIDF gitlab Service.
+Access the EIDF GitLab in your browser by opening [https://gitlab.eidf.ac.uk/](https://gitlab.eidf.ac.uk/). You must be a member of an active EIDF project and have a user account to use the EIDF GitLab Service.
 
 ![GitLab Login Page](../../images/access/gitlab-login.png)
 
@@ -34,8 +34,8 @@ From the command line, you can clone this repository by adding your username and
 
 To clone a repository using git over SSH, click on the "Code" button to get the git+ssh URL. You will need to have added an SSH key to your account for this method to work.
 
-For a more complete set of documentation relating to adding and using SSH keys with Gitlab, see the upstream [Gitlab Documentation](https://docs.gitlab.com/user/ssh/)
+For a more complete set of documentation relating to adding and using SSH keys with GitLab, see the upstream [GitLab Documentation](https://docs.gitlab.com/user/ssh/)
 
-## CI/CD Examples
+## Where to go next?
 
-A repository containing some common Gitlab CI/CD configurations and relevant examples is maintained at at [Gitlab CI/Cd Examples](https://gitlab.eidf.ac.uk/Liz/cicd-examples)
+CI/CD pipelines for automation of workflows are documented in the [GitLab CI/CD documentation](cicd.md).

docs/services/registry/working-with.md

@@ -13,6 +13,20 @@ User tokens can be accessed from the User Profile from the dropdown under your U
 
 See the FAQ section ['Unauthorised' error when logging into the registry from Docker](./faq.md#unauthorised-error-when-logging-into-the-registry-from-docker) for help with token expiry and authorisation issues.
 
+## Creating a Project Repository
+
+Each EIDF project can have a private space on the registry, this is called an Edinburgh Container Image Registry (ECIR) project. An ECIR project can be used to store images and artifacts that are private to the project.
+
+To create an ECIR project:
+
+1. Navigate to your EIDF project in the EIDF portal.
+2. Find the **Container Registry Project** section.
+3. Click the **Create Project** button.
+
+This will generate the ECIR project, which will be accessible from the registry interface within a few minutes. The project will be named with the same name as your EIDF project.
+![CreateECIRProject](../../images/registry/CreateECIRProject.png){: class="border-img"}
+   *Example Create ECIR Project*
+
 ## Push Commands
 
 In your project, there is a PUSH Command option which will give you the command templates for pushing to the Project repositories from different clients.
@@ -26,6 +40,20 @@ Each repository in a project has a COPY PULL button option once an image/artifac
 
 Clicking on a tag in a repository will open up the information on the artifact, this can include an overview of the image, vulnerability summary, SBOM and build history.
 
+## Creating Robot Accounts for the Registry
+
+If you are regularly using a repository from a project where you are sharing resources and need automated, read-only access (for example, pulling images into compute jobs), it is recommended to create a robot account with limited pull-only privileges.
+
+If you also need to publish images (for example, as part of an automated build or CI/CD pipeline), you should instead create a robot account with pull and push (read and write) permissions for the project.
+
+Robot accounts can be added by a project administrator as follows:
+
+1. In the ECIR project, click the **+ pull robot** or **+ push robot** button within the project.
+2. Wait a few minutes for the robot account to be created.
+3. Go to the **Robot Account** section of the project to access the credentials, which include a username and a CLI Secret for logging into the registry from Docker and other container services.
+
+Robot accounts with pull and push permissions have a default validity period of 30 days, after which they will expire and need to be renewed. This is to ensure that access is regularly reviewed and maintained.
+
 ## Using from the Command Line with Docker
 
 Important: Run these commands on a system that has Docker installed and has access to the ECIR.
@@ -61,20 +89,12 @@ https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 
 Login Succeeded
 ```
-
+To pull images from the registry, from private or authenticated projects, you will need to add a secret to the namespace you are using and reference it in your job definition. Note that user tokens have a limited validity period and therefore, robot accounts are recommended for long-term use. See the section on [Creating a Robot Account](#creating-robot-accounts-for-the-registry) for more details.
 From your command line, you can now push and pull images to the registry.
 
 ## Kubernetes/GPU Service Access
 
-To pull images from the registry, from private or authenticated projects, you will need to add a secret to the namespace you are using and reference it in your job definition. Note that user tokens have a limited validity period.
-
-If you are regularly using a repository from a project where you are sharing resources, it is recommended to create a robot account with limited read only privileges, this can be requested via a Helpdesk Request for your project.
-
-!!! important "Portal Management"
-
-    There will be new functionality soon added to the EIDF Portal to allow for project users to create read only robot accounts and for PI/Managers to create read/write robot accounts for use in CI/CD pipelines for image building.
-
-This is then treated like a normal user secret when you have the robot credentials.
+To pull images from the registry, from private or authenticated projects, you will need to add a secret to the namespace you are using and reference it in your job definition. Note that user tokens have a limited validity period and hence robots are recommended for long term use. See the section on [Creating a Robot Account](#creating-robot-accounts-for-the-registry) for more details.
 
 Secrets can be created in one of two ways, as detailed below, either directly via kubectl from your Docker config.json file, or by creating a YAML file.
 

mkdocs.yml

@@ -96,8 +96,8 @@ nav:
     #    - "Quickstart": services/mft/quickstart.md
     #    - "Using the MFT": services/mft/using-the-mft.md
     - "Code Collaboration":
-      - "Gitlab Overview": services/gitlab/index.md
-      - "Gitlab Quickstart": services/gitlab/quickstart.md
+      - "GitLab Overview": services/gitlab/index.md
+      - "GitLab Quickstart": services/gitlab/quickstart.md
       - "Container Image Registry Overview": services/registry/index.md
       - "Container Image Registry Projects and Caches": services/registry/projects.md
       - "Container Image Registry Use": services/registry/working-with.md