From 8d38a2dcb0bd73be8aba3043bac362f90e1f84ba Mon Sep 17 00:00:00 2001 From: Mike Jackson Date: Tue, 19 May 2026 17:47:46 +0100 Subject: [PATCH 1/3] Removed references to EIDF CES development environemnt eidf147-ces-dev02 An EIDF CES development environment is currently unsupported. SHS Container Guide documentation has been updated accordingly. Also did general formatting changes, for consistency. --- .../development-workflow.md | 298 ++++++++++-------- .../shs-container-user-guide/introduction.md | 9 +- .../workflow-examples.md | 68 ++-- 3 files changed, 184 insertions(+), 191 deletions(-) diff --git a/docs/safe-haven-services/shs-container-user-guide/development-workflow.md b/docs/safe-haven-services/shs-container-user-guide/development-workflow.md index fc74c77d5..69bfc9579 100644 --- a/docs/safe-haven-services/shs-container-user-guide/development-workflow.md +++ b/docs/safe-haven-services/shs-container-user-guide/development-workflow.md @@ -6,124 +6,142 @@ This document describes in detail the steps introduced in [The Safe Haven Contai ### 1.1 SHS-specific advice -- Declare SHS specific directories using the line `RUN mkdir /safe_data /safe_outputs /scratch` in your Dockerfile. While the CES tools automatically generate these directories within the container, explicitly creating them - enhances transparency and helps others more easily understand the container’s structure and operation. - Note that files should not be added to these directories through the Dockerfile prior to mounting, as they would be overwritten during the mounting process. The directories will be fully accessible within the container during run time. +**Declare SHS specific directories using the line `RUN mkdir /safe_data /safe_outputs /scratch` in your `Dockerfile`**. While the CES tools automatically generate these directories within the container, explicitly creating them enhances transparency and helps others more easily understand the container’s structure and operation. -- Start with a well-known application base container on a public registry. - Projects should add a minimum of additional project software and packages - so that the container is clearly built for a specific purpose. Avoid - patching and preserve the original container setup wherever possible. - Complex containers, particularly interactive ones, have been carefully - designed by competent developers to meet high security and usability - standards. Modifying these without the required knowledge might introduce - unwanted risks and side effects. +!!! note -- Do not copy data files into the image. As a general rule, images should only contain software and configuration files. Any data files required will be presented to the container at runtime (e.g. via the `/safe_data` mount) and should not be copied into the container during the build. + Files should not be added to these directories through the `Dockerfile` prior to mounting, as they would be overwritten during the mounting process. The directories will be fully accessible within the container during run time. -- Add all the additional content (code files, libraries, packages, data, and licences) needed for your analysis work to your Dockerfile. Since the SHS VMs do not have internet access, all necessary code, dependencies, and resources must be pre-packaged within the container to ensure it runs successfully. +**Start with a well-known application base container on a public registry**. Projects should add a minimum of additional project software and packages so that the container is clearly built for a specific purpose. Avoid patching and preserve the original container setup wherever possible. -- Apply the principle of least privilege, that is select a non-privileged user inside the container whenever possible. +!!! warning -- Both public and private containers can be pulled from GitHub Container Registry (GHCR) into the SHS. However, users within the SHS need to provide both a username and an access token to do so. This is a requirement of the 'ces-tools' used to pull containers into the SHS. + Complex containers, particularly interactive ones, have been carefully designed by competent developers to meet high security and usability standards. Modifying these without the required knowledge might introduce unwanted risks and side effects. - +**Do not copy data files into the image**. As a general rule, images should only contain software and configuration files. Any data files required will be presented to the container at runtime (e.g. via the `/safe_data` mount) and should not be copied into the container during the build. + +**Add all the additional content (code files, libraries, packages, data, and licences) needed for your analysis work to your `Dockerfile`**. Since the SHS VMs do not have internet access, all necessary code, dependencies, and resources must be pre-packaged within the container to ensure it runs successfully. + +**Apply the principle of least privilege**, that is select a non-privileged user inside the container whenever possible. + + ### 1.2 General recommendations -It is highly recommended that users follow these Dockerfile best practice guidelines: +It is highly recommended that users follow these `Dockerfile` best practice guidelines. -- Use fully-qualified and pinned images in all `FROM` statements. Images can be hosted on multiple repositories, and image tags are mutable. The only way to ensure reproducible builds is by pinning images by their full signature. The signature of an image can be viewed with `docker inspect --format='{{index .RepoDigests 0}}' `. The image repository is usually `docker.io` or `ghcr.io`. +**Use fully-qualified and pinned images in all `FROM` statements**. Images can be hosted on multiple repositories, and image tags are mutable. The only way to ensure reproducible builds is by pinning images by their full signature, and, where possible, citing a repository such as `docker.io` or `ghcr.io`. - **Example**: +!!! tip - ```dockerfile - # Incorrect - FROM nvidia/cuda:latest - FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 - FROM docker.io/nvidia/cuda:latest + The signature of an image can be viewed with the command: - # Correct - FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04@sha256:622e78a1d02c0f90ed900e3985d6c975d8e2dc9ee5e61643aed587dcf9129f42 - ``` + ```console + docker inspect --format='{{index .RepoDigests 0}}' + ``` -- Group consecutive and related commands into a single `RUN` statement. Each `RUN` statement causes a new layer to be created in the image. By grouping `RUN` statements together, and deleting temporary files, the final image image size can be greatly reduced. +Incorrect examples: - **Example**: +```dockerfile +FROM nvidia/cuda:latest +``` - ```dockerfile - # Incorrect - RUN apt-get -y update - RUN apt-get -y install curl - RUN apt-get -y install git +```dockerfile +FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 +``` + +```dockerfile +FROM docker.io/nvidia/cuda:latest +``` + +Correct example: + +```dockerfile +FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04@sha256:622e78a1d02c0f90ed900e3985d6c975d8e2dc9ee5e61643aed587dcf9129f42 +``` - # Correct - # - Single RUN statement with commands broken over multiple lines - # - Temporary apt files are deleted and not stored in the final image - RUN : \ - && apt-get update -qq \ - && DEBIAN_FRONTEND=noninteractive apt-get install \ - -qq -y --no-install-recommends \ - curl \ - git \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* \ - && : - ``` +**Group consecutive and related commands into a single `RUN` statement**. Each `RUN` statement causes a new layer to be created in the image. By grouping `RUN` statements together, and deleting temporary files, the final image image size can be greatly reduced. -- Use `COPY` instead of `ADD`. Compared to the `COPY` command, `ADD` supports much more functionality such as unpacking archives and downloading from URLs. While this may seem convenient, using `ADD` may result in much larger images with layers which are unnecessary. In the following example, using COPY after unpacking locally makes the image more efficient: +Incorrect example: - **Example**: +```dockerfile +RUN apt-get -y update +RUN apt-get -y install curl +RUN apt-get -y install git +``` - ```dockerfile - # Incorrect - FROM python:3.9-slim - # Unpack a tar archive - ADD my_project.tar.gz /target-dir/ - CMD ["python", "/target-dir/main.py"] +Correct example with a single `RUN` statement with commands broken over multiple lines. Temporary apt files are deleted and not stored in the final image. - # Correct - FROM python:3.9-slim - # Unpack my_project.tar.gz locally first, then copy - COPY my_project/ /target-dir/ - CMD ["python", "/target-dir/main.py"] - ``` +```dockerfile +RUN : \ + && apt-get update -qq \ + && DEBIAN_FRONTEND=noninteractive apt-get install \ + -qq -y --no-install-recommends \ + curl \ + git \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* \ + && : +``` -- Use multi-stage builds where appropriate. Separating build/compilation steps into a separate stage helps to minimise the content of the final image and reduce the overall size. +**Use `COPY` instead of `ADD`**. Compared to the `COPY` command, `ADD` supports much more functionality such as unpacking archives and downloading from URLs. While this may seem convenient, using `ADD` may result in much larger images with layers which are unnecessary. In the following example, using COPY after unpacking locally makes the image more efficient: - **Example**: +Incorrect example: - ```dockerfile - FROM some-image AS builder - RUN apt update && apt -y install build-dependencies - COPY . . - RUN ./configure --prefix=/opt/app && make && make install +```dockerfile +FROM python:3.9-slim +# Unpack a tar archive +ADD my_project.tar.gz /target-dir/ +CMD ["python", "/target-dir/main.py"] +``` - FROM some-minimal-image - RUN apt update && apt -y install runtime-dependencies - COPY --from=builder /opt/app /opt/app - ``` +Correct example: -- Use a minimal image in the last stage to reduce the final image size. When using multi-stage builds, the final `FROM` image should use a minimal image such as from [Distroless](https://github.com/GoogleContainerTools/distroless/) or [Chainguard](https://images.chainguard.dev/) to minimise image content and size. +```dockerfile +FROM python:3.9-slim +# Unpack my_project.tar.gz locally first, then copy +COPY my_project/ /target-dir/ +CMD ["python", "/target-dir/main.py"] +``` - **Example**: +**Use multi-stage builds where appropriate**. Separating build/compilation steps into a separate stage helps to minimise the content of the final image and reduce the overall size. - ```dockerfile - FROM some-image AS builder - # ... - FROM gcr.io/distroless/base-debian12 - # ... - ``` +Example: -- Use a linter such as [Hadolint](https://github.com/hadolint/hadolint) to verify the content of the `Dockerfile`. - Automated code linting tools can be very useful in detecting common mistakes and pitfalls when developing software. Some configuration tweaks may be required however, as shown in the example below. +```dockerfile +FROM some-image AS builder +RUN apt update && apt -y install build-dependencies +COPY . . +RUN ./configure --prefix=/opt/app && make && make install + +FROM some-minimal-image +RUN apt update && apt -y install runtime-dependencies +COPY --from=builder /opt/app /opt/app +``` - **Example**: +**Use a minimal image in the last stage to reduce the final image size**. When using multi-stage builds, the final `FROM` image should use a minimal image such as from [Distroless](https://github.com/GoogleContainerTools/distroless/) or [Chainguard](https://images.chainguard.dev/) to minimise image content and size. - ```console - # Ignore DL3008 (Pin versions in apt get install) - docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3008 - < Dockerfile - ``` +Example: + +```dockerfile +FROM some-image AS builder +# ... +FROM gcr.io/distroless/base-debian12 +# ... +``` + +**Use a linter to verify the content of the `Dockerfile`**. An example is [Hadolint](https://github.com/hadolint/hadolint). Automated code linting tools can be very useful in detecting common mistakes and pitfalls when developing software. Some configuration tweaks may be required however, as shown in the example below. + +Example: + +```console +# Ignore DL3008 (Pin versions in apt get install) +docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3008 - < Dockerfile +``` + +### 1.3 Other recommendations See the following resources for additional recommendations: @@ -135,7 +153,7 @@ See the following resources for additional recommendations: ### 2.1 Build and test the container locally -Docker, Podman, Kubernetes, and Apptainer container images can be created from a single Dockerfile, as all of them support the OCI container image format either natively or indirectly. +Docker, Podman, Kubernetes, and Apptainer container images can be created from a single `Dockerfile`, as all of them support the OCI container image format either natively or indirectly. Containers can be built using the following command: @@ -149,30 +167,32 @@ Run your image locally, so that minor errors can be immediately fixed before pro docker run : ``` +We recommend that you test containers without a network connection to best mimick their functionality inside the SHS, where the container will not be able to access the internet. With Docker and Podman, this can be achieved using the `--network none` command-line parameter. + ### Example -A container with the following Dockerfile: +A container with the following `Dockerfile`: ```dockerfile - FROM python:3.13.3@sha256:a4b2b11a9faf847c52ad07f5e0d4f34da59bad9d8589b8f2c476165d94c6b377 +FROM python:3.13.3@sha256:a4b2b11a9faf847c52ad07f5e0d4f34da59bad9d8589b8f2c476165d94c6b377 - # Create SHS directories - RUN mkdir /safe_data /safe_outputs /scratch +# Create SHS directories +RUN mkdir /safe_data /safe_outputs /scratch - # Add app files - COPY . /app - WORKDIR /app +# Add app files +COPY . /app +WORKDIR /app - # Install Python dependencies - RUN pip install --no-cache-dir -r requirements.txt +# Install Python dependencies +RUN pip install --no-cache-dir -r requirements.txt - # Default command - CMD ["python", "app.py"] +# Default command +CMD ["python", "app.py"] ``` where this is the app structure: -```console +```text . ├── app.py ├── requirements.txt @@ -195,7 +215,7 @@ docker run --rm myapp:v1.1 Podman can be used equivalently to Docker in the commands above. -### 2.2 Automating Dockerfile validation +### 2.2 Automating `Dockerfile` validation During development, we recommend that tools like GitHub or GitLab are used for version control and recording of the image content. Using the [`pre-commit`](https://pre-commit.com/) tool, it is possible to configure your local repository so that Hadolint (and similar tools) are run automatically each time `git commit` is run. This is recommended to ensure linting and auto-formatting tools are always run before code is pushed to GitHub. @@ -213,16 +233,16 @@ repos: To prepare the image before pushing to GHCR, we recommend to save the image with a unique, descriptive tag. While it is useful to define a `latest` tag, each production image should also be tagged with a label such as the version or build date. For non-local images, the registry and repository should also be included. Images can also be tagged multiple times. - **Example**: +For example: - ```console - docker build \ - --tag ghcr.io/my/image:v1.2.3 \ - --tag ghcr.io/my/image:latest \ - ... - ``` +```console +docker build \ + --tag ghcr.io/my/image:v1.2.3 \ + --tag ghcr.io/my/image:latest \ + ... +``` -The following commands can be used to upload the image to a GHCR repository. +The following commands can be used to upload the image to a GHCR repository, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. ```console echo "${GHCR_TOKEN}" | docker login ghcr.io -u $GHCR_NAMESPACE --password-stdin @@ -232,17 +252,15 @@ docker push "ghcr.io/$GHCR_NAMESPACE/$IMAGE:$TAG" docker logout ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - ### 2.5 Optional - Build and upload automation using GitHub Actions #### Building in CI -Below is a sample GHA configuration which runs Hadolint, builds a container named `ghcr.io/my/repo`, then runs the [Trivy](https://aquasecurity.github.io/trivy) container scanning tool. The Trivy [SBOM](https://www.cisa.gov/sbom) report is then uploaded as a job artifact. +Below is a sample GitHub Actions configuration which runs Hadolint, builds a container named `ghcr.io/my/repo`, then runs the [Trivy](https://aquasecurity.github.io/trivy) container scanning tool. The Trivy [SBOM](https://www.cisa.gov/sbom) report is then uploaded as a job artifact. This assumes: -- The repo contains a `Dockerfile` in the top-level directory, +- The repository contains a `Dockerfile` in the top-level directory, - The `Dockerfile` contains an `ARG` or `ENV` variable which defines the version of the packaged software. ```yaml @@ -291,18 +309,23 @@ jobs: path: 'dependency-results.sbom.json' ``` -Note: `secrets.GITHUB_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. +where `secrets.GITHUB_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. + +!!! note -Note that manually running Hadolint via pre-commit can be skipped if you are using pre-commit and the [pre-commit.ci](https://pre-commit.ci/) service. + Manually running Hadolint via pre-commit can be skipped if you are using pre-commit and the [pre-commit.ci](https://pre-commit.ci/) service. #### Publishing in CI -__Note__ Images can also be built and pushed from your local environment as normal. +!!! note -Once the stage has been reached where your software package is ready for distribution, the GHA example above can be extended to automatically publish new image versions to the GHCR. An introduction to GHCR can be found in the GitHub docs [here](https://docs.github.com/en/packages/quickstart). + Images can also be built and pushed from your local environment as normal. + +Once the stage has been reached where your software package is ready for distribution, the GHA example above can be extended to automatically publish new image versions to the GHCR. An introduction to GHCR can be found in the GitHub documentation, [Quickstart for GitHub Packages](https://docs.github.com/en/packages/quickstart). + +After the image has been built and scanned, the image can be pushed as follows: ```yaml -# After the image has been built and scanned - name: push image run: | set -euxo pipefail @@ -310,35 +333,40 @@ Once the stage has been reached where your software package is ready for distrib docker push "${image}" ``` -## Step 3. Test in CES test environment +## Step 3. Pull and run container inside the SHS + +To use the containers inside the SHS, log into the desired VM following [Safe Haven Services Access](..//safe-haven-access.md). -EPCC provides a test environment that allows users to test their containers in a SHS-like environment without having to be logged into the SHS itself. Containers that run successfully in such environment can then be expected to perform in the same way in the SHS. +!!! warning "Do not use container commands directly" -### 3.1 Accessing test environment + The CES tools **must** be used to pull and run containers within the SHS. This is to ensure that the correct data directories are automatically made available. -The test environment is located in the eidf147 project. Please ask your research coordinator to contact EPCC and request for you to be added to the test environment. + You **must not** use commands such as `podman run ...` or `docker run ...` directly. -### 3.2 Pull and run +### 3.1 Pull container -!!! warning "Do not use container CLIs directly" - The CES wrapper scripts **must** be used to run containers in the SHS. This is to ensure that the correct data directories are automatically made available. +Containers can only be used on the SHS desktop hosts using shell commands. Containers can only be pulled from the GHCR into the SHS using the CES tools `ces-pull` command. Hence containers must be pushed to GHCR for them to be used in the SHS. - You **must not** use commands such as `podman run ...` or `docker run ...` directly. +!!! note -Containers can only be used on the SHS desktop hosts using shell commands. Containers can only be pulled from the GHCR into the SHS using a `ces-pull` script. Hence containers must be pushed to GHCR for them to be used in the SHS. Although alternative methods can be used in the test environment, we encourage users to follow the exact same procedure as they would in the SHS. + Both public and private containers can be pulled from GitHub Container Registry (GHCR) into the SHS. However, users within the SHS need to provide both a username and an access token to do so. This is a requirement of the CES tools used to pull containers into the SHS. -Once access has been granted to the test environment in the eidf147 project, the user can pull a container from their GHCR repository using the `ces-pull` command: +You can pull a container from your GHCR repository using the `ces-pull` command: ```sh ces-pull [] ghcr.io//: ``` -Note: `` is a GitHub user name, `` is a GitHub access token with 'read:packages' scope that allows access to the image 'ghcr.io//:'. When pulling containers into the test environment or the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. +`` is a GitHub user name, `` is a GitHub access token with 'read:packages' scope that allows access to the image 'ghcr.io//:'. Both these arguments are mandatory. + +!!! tip -Note: The `` and `` arguments to `ces-pull` are mandatory. + When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. If a runtime is not specified then podman is used as the default. +### 3.2 Run container + The container can then be run with `ces-run`: ```sh @@ -352,13 +380,3 @@ ces-run [] ghcr.io//: - Use `--env-file=` to specify a file containing environment variables which will be set inside the container Containers that require a GPU can be run by adding the `--gpu` option. See `ces-run [] --help` for all available options. - -We recommend to test containers without network connection to best mimick their functionality inside the SHS, where the container will not be able to access the internet. With Podman, for example, this can be achieved by passing the option `--network=none` through the `opt-file`. - -Once the container runs successfully in the test environment, it is ready to be used inside the SHS. - -## Step 4. Pull and run container inside the SHS - -To use the containers inside the SHS, log into the desired VM using the steps described in the EIDF User Documentation: - -The same steps described in [Section 3.2](#32-pull-and-run) of this document can then be used to pull and run the container. diff --git a/docs/safe-haven-services/shs-container-user-guide/introduction.md b/docs/safe-haven-services/shs-container-user-guide/introduction.md index 51e7587b4..e5881931b 100644 --- a/docs/safe-haven-services/shs-container-user-guide/introduction.md +++ b/docs/safe-haven-services/shs-container-user-guide/introduction.md @@ -3,8 +3,10 @@ ## What is the CES? The Container Execution Service (CES) has been introduced to allow project code developed and tested by researchers outside the Safe Haven Services (SHS) in personal development environments to be imported and run on the project data inside the SHS using a well-documented, transparent, secure workflow. + The primary role of the SHS is to store and share data securely; it is not intended to be a software development and testing environment. The CES helps researchers perform software development tasks in their chosen environment, rather than the restricted one offered in the SHS. -This guide describes the process of building and testing SHS-ready containers outside the SHS, specifically within a test environment created in the EIDF, so that they are ready to be pulled and used inside the SHS. + +This guide describes the process of building and testing SHS-ready containers outside the SHS, so that they are ready to be pulled and used inside the SHS. ## Accessing the CES @@ -23,8 +25,7 @@ It is also assumed that software development best practices are followed, such a The development process includes the following steps: 1. Create a Dockerfile following [general recommendations](./development-workflow.md#12-general-recommendations) and [SHS-specific advice](./development-workflow.md#11-shs-specific-advice). -1. Build the image and upload to the GitHub Container Registry (GHCR), either locally or using a CI/CD pipeline. -1. Test the container in the [CES test environment](./development-workflow.md#31-accessing-test-environment) to ensure it functions correctly and has no external runtime dependencies. +1. Build and test the image and upload to the GitHub Container Registry (GHCR), either locally or using a CI/CD pipeline. 1. Login to a SHS desktop enabled for container execution to pull and run the container. This guide contains a number of [workflow examples](./workflow-examples.md) designed to assist users in building SHS-ready containers. Other [container examples](./container-examples.md) are also available in our [SHS Container Samples](https://github.com/EPCCed/shs-container-samples/) repository. @@ -32,7 +33,9 @@ This guide contains a number of [workflow examples](./workflow-examples.md) desi ## SHS directories Before continuing with this guide, it is important that the users are aware of the SHS file system directories so that they can be used correctly within their containers. Project data inside the SHS can be found in the `/safe_data/` directory. + The CES tools automatically map this to a directory called `/safe_data` inside the container. Additionally, two directories are created and mapped to the user's home directory. The first one is `/scratch`, a temporary directory that is removed after container termination on the host system. + The second one is a unique container job output directory mapped to the `/safe_outputs` directory in the container, where output files that the users wishes to preserve should be placed. Note that containers that have been pulled into the SHS are destroyed after they have been run. Only the files written to the container outputs directory are guaranteed to be retained. | Directory on host system | Directory in container | Intended use diff --git a/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md b/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md index 86aedf735..72d98441b 100644 --- a/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md +++ b/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md @@ -58,15 +58,13 @@ As mentioned in our guidelines, it is good practice to check the Dockerfile with docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3013 - < Dockerfile ``` -We can then define our GHCR variables: +We can then define our GHCR variables, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope: ```sh export GHCR_NAMESPACE=mynamespace export GHCR_TOKEN=mytoken ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - We can now build our container: ```sh @@ -88,26 +86,18 @@ docker push "ghcr.io/$GHCR_NAMESPACE/pytorch-test:latest" docker logout ``` -### E1 - Step 3. Test in CES test environment - -Log into the 'ces-dev02' VM of the project EIDF147, which is the designated test environment for the CES. +### E1 - Step 3. Pull and run in the SHS -Then, pull and run the container using the commands: +Pull and run the container inside the SHS using the following commands, where the namespace and token arguments to `ces-pull` are mandatory. ```sh ces-pull podman $GHCR_NAMESPACE $GHCR_TOKEN ghcr.io/$GHCR_NAMESPACE/pytorch-test:v1.1 ces-run podman --gpu ghcr.io/$GHCR_NAMESPACE/pytorch-test:v1.1 ``` -Note: The namespace and token arguments to `ces-pull` are mandatory. - -Note: When pulling containers into the test environment, instead of using the GitHub access token you used to push the container, it is **recommended** that you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. - -### E1 - Step 4. Pull and run in the SHS +!!! tip -The container can be pulled and run inside the SHS using the same commands as the [previous step](#e1-step-3-test-in-ces-test-environment). - -Note: When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. + When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. ## Example 2 - Python ML @@ -228,17 +218,15 @@ As mentioned in our guidelines, before building our container we first want to c docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3013 --ignore DL3042 --ignore DL3045 - < Dockerfile ``` -Note that we can deliberately ignore hadolint flags if the selected features are required by the container to run successfully. +Here we deliberately ignore hadolint flags if the selected features are required by the container to run successfully. -We then define our GHCR variables: +We can then define our GHCR variables, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope: ```sh export GHCR_NAMESPACE=mynamespace export GHCR_TOKEN=mytoken ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - We can now build our container: ```sh @@ -271,24 +259,18 @@ docker push "ghcr.io/$GHCR_NAMESPACE/python-ml-test:latest" docker logout ``` -### E2 - Step 3. Test in CES test environment +### E2 - Step 3. Pull and run in the SHS -As with the previous example, we now want to test our container in the designated test environment for the CES. Log into the 'ces-dev02' VM of the project EIDF147, then pull and run the container using the commands: +Pull and run the container inside the SHS using the commands, where the namespace and token arguments to `ces-pull` are mandatory: ```sh ces-pull podman $GHCR_NAMESPACE $GHCR_TOKEN ghcr.io/$GHCR_NAMESPACE/python-ml-test:v1.1 ces-run podman --gpu ghcr.io/$GHCR_NAMESPACE/python-ml-test:v1.1 ``` -Note: The namespace and token arguments to `ces-pull` are mandatory. - -Note: When pulling containers into the test environment, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. - -### E2 - Step 4. Pull and run in the SHS +!!! tip -We can import and run the container in the SHS using the commands from the [earlier step](#e2-step-3-test-in-ces-test-environment). - -Note: When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. + When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. ## Example 3 - Interactive RStudio Rocker container @@ -400,15 +382,13 @@ Before building our container, we want to check the Dockerfile with a linting to docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3008 - < Dockerfile ``` -We can then define our GHCR variables: +We can then define our GHCR variables, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope: ```sh export GHCR_NAMESPACE=mynamespace export GHCR_TOKEN=mytoken ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - We can now build our container: ```sh @@ -432,19 +412,17 @@ docker push "ghcr.io/$GHCR_NAMESPACE/rocker-test:latest" docker logout ``` -### E3 - Step 3. Test in CES test environment - -Log into the 'ces-dev02' VM of the project EIDF147, which is the designated test environment for the CES. +### E3 - Step 4. Pull and run in the SHS -Rocker is one of those containers that requires to be started by the 'root' user. As such, it should be run inside the SHS - and our test environment - using podman. To pull the container using podman as our container engine, we use the command: +Rocker is a container that requires to be started by the 'root' user. As such, it should be pulled intn the SHS and run using podman. To pull the container using podman as our container engine, we use the following command, where the namespace and token arguments are mandatory: ```sh ces-pull podman $GHCR_NAMESPACE $GHCR_TOKEN ghcr.io/$GHCR_NAMESPACE/rocker-test:v1.1 ``` -Note: The namespace and token arguments to `ces-pull` are mandatory. +!!! tip -Note: When pulling containers into the test environment, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. + When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. The Rocker container was designed to be run using docker. In order for it to run successfully with podman, the container directories `/var/lib/rstudio-server` and `/run` need to be mounted to a tmpfs. As such, the following options are required: @@ -453,7 +431,7 @@ The Rocker container was designed to be run using docker. In order for it to run '--mount type=tmpfs,destination=/run' ``` -Our full `opt-file.txt` then looks like this: +A `ces-run` `opt-file.txt` can include these and the other options required: ```text -p 8787:8787 @@ -462,7 +440,7 @@ Our full `opt-file.txt` then looks like this: --mount type=tmpfs,destination=/run ``` -If we want to set our password, we can add it to the `env-file.txt` as follows: +If we want to set our password, we can add it to a `ces-run` `env-file.txt` as follows: ```text PASSWORD=test @@ -474,7 +452,7 @@ We can then run our container: ces-run podman --opt-file opt-file.txt --env-file env-file.txt ghcr.io/$GHCR_NAMESPACE/rocker-test:v1.1 ``` -To streamline the process, we can place all the necessary commands into an executable script `run.sh` as follows: +To streamline the process, we can place all the necessary commands into an executable script, `run.sh`, as follows: ```sh opt_file="./opt-file.txt" @@ -497,16 +475,10 @@ echo -e "PASSWORD=test" >> ${env_file} ces-run podman --opt-file $opt_file --env-file $env_file ghcr.io/$GHCR_NAMESPACE/rocker-test:v1.1 ``` -After executing the `run.sh` script, we can open a browser tab and access RStudio at `localhost:8787`. As done previously during the local test, we can log in using the credentials `root` and `test` for username and password respectively. From within RStudio, we can then run our script, which can be found in `/src`, and then copy our output to `/safe_output` so that it can be preserved after we exit the container. +After executing the `run.sh` script, we can open a browser tab and access RStudio at `localhost:8787`. We can log in using the credentials `root` and `test` for username and password respectively. From within RStudio, we can then run our script, which can be found in `/src`, and then copy our output to `/safe_output` so that it can be preserved after we exit the container. The container is running successfully if: - The log-in is successful. - The rstudio user has full access to SHS directories `/safe_data`, `/safe_outputs` and `/scratch`. - The files saved in `/safe_outputs` and `/safe_data` (when writing permission is granted by IG) have correct permission on the host, that is they belong to the logged-in user. - -### E3 - Step 4. Pull and run in the SHS - -The container can then be imported inside the SHS using the same commands as [Step 3](#e3-step-3-test-in-ces-test-environment). - -Note: When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. From 02d5cd4d91f8f2f3d264060cbba44022199d2293 Mon Sep 17 00:00:00 2001 From: Mike Jackson Date: Tue, 19 May 2026 17:51:17 +0100 Subject: [PATCH 2/3] Fixed link in introduction.md --- .../shs-container-user-guide/introduction.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/safe-haven-services/shs-container-user-guide/introduction.md b/docs/safe-haven-services/shs-container-user-guide/introduction.md index e5881931b..c63d56fd6 100644 --- a/docs/safe-haven-services/shs-container-user-guide/introduction.md +++ b/docs/safe-haven-services/shs-container-user-guide/introduction.md @@ -28,7 +28,7 @@ The development process includes the following steps: 1. Build and test the image and upload to the GitHub Container Registry (GHCR), either locally or using a CI/CD pipeline. 1. Login to a SHS desktop enabled for container execution to pull and run the container. -This guide contains a number of [workflow examples](./workflow-examples.md) designed to assist users in building SHS-ready containers. Other [container examples](./container-examples.md) are also available in our [SHS Container Samples](https://github.com/EPCCed/shs-container-samples/) repository. +This guide contains a number of [workflow examples](./workflow-examples.md) designed to assist users in building SHS-ready containers. Other [container examples](./container-examples.md) are also available in our [SHS Container Samples](https://github.com/EPCCed/tre-container-samples/) repository. ## SHS directories From e21c30a5e2ffe2e0011749996f163aaca6c61b20 Mon Sep 17 00:00:00 2001 From: Mike Jackson Date: Tue, 19 May 2026 17:54:59 +0100 Subject: [PATCH 3/3] Fixed header in workflow-examples.md --- .../shs-container-user-guide/workflow-examples.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md b/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md index 72d98441b..620dc0326 100644 --- a/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md +++ b/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md @@ -412,7 +412,7 @@ docker push "ghcr.io/$GHCR_NAMESPACE/rocker-test:latest" docker logout ``` -### E3 - Step 4. Pull and run in the SHS +### E3 - Step 3. Pull and run in the SHS Rocker is a container that requires to be started by the 'root' user. As such, it should be pulled intn the SHS and run using podman. To pull the container using podman as our container engine, we use the following command, where the namespace and token arguments are mandatory: