diff --git a/docs/safe-haven-services/shs-container-user-guide/container-examples.md b/docs/safe-haven-services/shs-container-user-guide/container-examples.md index 387ab2ab8..95daf17e4 100644 --- a/docs/safe-haven-services/shs-container-user-guide/container-examples.md +++ b/docs/safe-haven-services/shs-container-user-guide/container-examples.md @@ -1,6 +1,6 @@ # Container Examples -To help with writing your own Dockerfiles to run within the SHS via the CES, a set of examples for commonly used software stacks can be found in our [SHS Container Samples](https://github.com/EPCCed/tre-container-samples/) repository. +To help with writing your own Dockerfiles to run within your Safe Haven via the CES, a set of examples for commonly used software stacks can be found in our [SHS Container Samples](https://github.com/EPCCed/tre-container-samples/) repository. Please contact your Service Manager if you need further support with the use of containers. diff --git a/docs/safe-haven-services/shs-container-user-guide/development-workflow.md b/docs/safe-haven-services/shs-container-user-guide/development-workflow.md index fc74c77d5..15f0cd071 100644 --- a/docs/safe-haven-services/shs-container-user-guide/development-workflow.md +++ b/docs/safe-haven-services/shs-container-user-guide/development-workflow.md @@ -1,137 +1,158 @@ # Development Workflow -This document describes in detail the steps introduced in [The Safe Haven Container Execution Service - CES](./introduction.md) for the development of containers for SHS use. +This document describes in detail the steps introduced in [The Safe Haven Container Execution Service - CES](./introduction.md) for the development of containers for use within Safe Havens. -## Step 1. Writing a `Dockerfile` +## Step 1. Writing a Dockerfile ### 1.1 SHS-specific advice -- Declare SHS specific directories using the line `RUN mkdir /safe_data /safe_outputs /scratch` in your Dockerfile. While the CES tools automatically generate these directories within the container, explicitly creating them - enhances transparency and helps others more easily understand the container’s structure and operation. - Note that files should not be added to these directories through the Dockerfile prior to mounting, as they would be overwritten during the mounting process. The directories will be fully accessible within the container during run time. +**Use GitHub or GitLab** for version control and recording of the container definition files. -- Start with a well-known application base container on a public registry. - Projects should add a minimum of additional project software and packages - so that the container is clearly built for a specific purpose. Avoid - patching and preserve the original container setup wherever possible. - Complex containers, particularly interactive ones, have been carefully - designed by competent developers to meet high security and usability - standards. Modifying these without the required knowledge might introduce - unwanted risks and side effects. +**Declare SHS-specific directories using the line `RUN mkdir /safe_data /safe_outputs /scratch` in your Dockerfile**. While the CES tools automatically generate these directories within the container, explicitly creating them enhances transparency and helps others more easily understand the container’s structure and operation. -- Do not copy data files into the image. As a general rule, images should only contain software and configuration files. Any data files required will be presented to the container at runtime (e.g. via the `/safe_data` mount) and should not be copied into the container during the build. +!!! note -- Add all the additional content (code files, libraries, packages, data, and licences) needed for your analysis work to your Dockerfile. Since the SHS VMs do not have internet access, all necessary code, dependencies, and resources must be pre-packaged within the container to ensure it runs successfully. + Files should not be added to these directories through the Dockerfile prior to mounting, as they would be overwritten during the mounting process. The directories will be fully accessible within the container during run time. -- Apply the principle of least privilege, that is select a non-privileged user inside the container whenever possible. +**Start with a well-known application base container on a public registry**. Projects should add a minimum of additional project software and packages so that the container is clearly built for a specific purpose. Avoid patching and preserve the original container setup wherever possible. -- Both public and private containers can be pulled from GitHub Container Registry (GHCR) into the SHS. However, users within the SHS need to provide both a username and an access token to do so. This is a requirement of the 'ces-tools' used to pull containers into the SHS. +!!! warning - + Complex containers, particularly interactive ones, have been carefully designed by competent developers to meet high security and usability standards. Modifying these without the required knowledge might introduce unwanted risks and side effects. + +**Do not copy data files into the image**. As a general rule, images should only contain software and configuration files. Any data files required will be presented to the container at runtime (e.g. via the `/safe_data` mount) and should not be copied into the container during the build. + +**Add all the additional content (code files, libraries, packages, data, and licences) needed for your analysis work to your Dockerfile**. Since Safe Haven VMs do not have internet access, all necessary code, dependencies, and resources must be pre-packaged within the container to ensure it runs successfully. + +**Apply the principle of least privilege**, that is select a non-privileged user inside the container whenever possible. + + ### 1.2 General recommendations -It is highly recommended that users follow these Dockerfile best practice guidelines: +It is highly recommended that you follow these Dockerfile best practice guidelines. + +**Use fully-qualified and pinned images in all `FROM` statements**. Images can be hosted on multiple repositories, and image tags are mutable. The only way to ensure reproducible builds is by pinning images by their full signature, and, where possible, citing a repository such as `docker.io` or `ghcr.io`. + +!!! tip + + The signature of an image can be viewed with the command: + + ```console + docker inspect --format='{{index .RepoDigests 0}}' + ``` + +Incorrect examples: + +```dockerfile +FROM nvidia/cuda:latest +``` + +```dockerfile +FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 +``` -- Use fully-qualified and pinned images in all `FROM` statements. Images can be hosted on multiple repositories, and image tags are mutable. The only way to ensure reproducible builds is by pinning images by their full signature. The signature of an image can be viewed with `docker inspect --format='{{index .RepoDigests 0}}' `. The image repository is usually `docker.io` or `ghcr.io`. +```dockerfile +FROM docker.io/nvidia/cuda:latest +``` - **Example**: +Correct example: - ```dockerfile - # Incorrect - FROM nvidia/cuda:latest - FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 - FROM docker.io/nvidia/cuda:latest +```dockerfile +FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04@sha256:622e78a1d02c0f90ed900e3985d6c975d8e2dc9ee5e61643aed587dcf9129f42 +``` - # Correct - FROM docker.io/nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04@sha256:622e78a1d02c0f90ed900e3985d6c975d8e2dc9ee5e61643aed587dcf9129f42 - ``` +**Group consecutive and related commands into a single `RUN` statement**. Each `RUN` statement causes a new layer to be created in the image. By grouping `RUN` statements together, and deleting temporary files, the final image image size can be greatly reduced. -- Group consecutive and related commands into a single `RUN` statement. Each `RUN` statement causes a new layer to be created in the image. By grouping `RUN` statements together, and deleting temporary files, the final image image size can be greatly reduced. +Incorrect example: - **Example**: +```dockerfile +RUN apt-get -y update +RUN apt-get -y install curl +RUN apt-get -y install git +``` + +Correct example with a single `RUN` statement with commands broken over multiple lines. Temporary apt files are deleted and not stored in the final image. - ```dockerfile - # Incorrect - RUN apt-get -y update - RUN apt-get -y install curl - RUN apt-get -y install git +```dockerfile +RUN : \ + && apt-get update -qq \ + && DEBIAN_FRONTEND=noninteractive apt-get install \ + -qq -y --no-install-recommends \ + curl \ + git \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* \ + && : +``` - # Correct - # - Single RUN statement with commands broken over multiple lines - # - Temporary apt files are deleted and not stored in the final image - RUN : \ - && apt-get update -qq \ - && DEBIAN_FRONTEND=noninteractive apt-get install \ - -qq -y --no-install-recommends \ - curl \ - git \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* \ - && : - ``` +**Use `COPY` instead of `ADD`**. Compared to the `COPY` command, `ADD` supports much more functionality such as unpacking archives and downloading from URLs. While this may seem convenient, using `ADD` may result in much larger images with layers which are unnecessary. In the following example, using COPY after unpacking locally makes the image more efficient: -- Use `COPY` instead of `ADD`. Compared to the `COPY` command, `ADD` supports much more functionality such as unpacking archives and downloading from URLs. While this may seem convenient, using `ADD` may result in much larger images with layers which are unnecessary. In the following example, using COPY after unpacking locally makes the image more efficient: +Incorrect example: - **Example**: +```dockerfile +FROM python:3.9-slim +# Unpack a tar archive +ADD my_project.tar.gz /target-dir/ +CMD ["python", "/target-dir/main.py"] +``` - ```dockerfile - # Incorrect - FROM python:3.9-slim - # Unpack a tar archive - ADD my_project.tar.gz /target-dir/ - CMD ["python", "/target-dir/main.py"] +Correct example: - # Correct - FROM python:3.9-slim - # Unpack my_project.tar.gz locally first, then copy - COPY my_project/ /target-dir/ - CMD ["python", "/target-dir/main.py"] - ``` +```dockerfile +FROM python:3.9-slim +# Unpack my_project.tar.gz locally first, then copy +COPY my_project/ /target-dir/ +CMD ["python", "/target-dir/main.py"] +``` -- Use multi-stage builds where appropriate. Separating build/compilation steps into a separate stage helps to minimise the content of the final image and reduce the overall size. +**Use multi-stage builds where appropriate**. Separating build/compilation steps into a separate stage helps to minimise the content of the final image and reduce the overall size. - **Example**: +Example: - ```dockerfile - FROM some-image AS builder - RUN apt update && apt -y install build-dependencies - COPY . . - RUN ./configure --prefix=/opt/app && make && make install +```dockerfile +FROM some-image AS builder +RUN apt update && apt -y install build-dependencies +COPY . . +RUN ./configure --prefix=/opt/app && make && make install + +FROM some-minimal-image +RUN apt update && apt -y install runtime-dependencies +COPY --from=builder /opt/app /opt/app +``` - FROM some-minimal-image - RUN apt update && apt -y install runtime-dependencies - COPY --from=builder /opt/app /opt/app - ``` +**Use a minimal image in the last stage to reduce the final image size**. When using multi-stage builds, the final `FROM` image should use a minimal image such as from [Distroless](https://github.com/GoogleContainerTools/distroless/) or [Chainguard](https://images.chainguard.dev/) to minimise image content and size. -- Use a minimal image in the last stage to reduce the final image size. When using multi-stage builds, the final `FROM` image should use a minimal image such as from [Distroless](https://github.com/GoogleContainerTools/distroless/) or [Chainguard](https://images.chainguard.dev/) to minimise image content and size. +Example: - **Example**: +```dockerfile +FROM some-image AS builder +# ... +FROM gcr.io/distroless/base-debian12 +# ... +``` - ```dockerfile - FROM some-image AS builder - # ... - FROM gcr.io/distroless/base-debian12 - # ... - ``` +**Use a linter to verify the content of the Dockerfile**. An example is [Hadolint](https://github.com/hadolint/hadolint). Automated code linting tools can be very useful in detecting common mistakes and pitfalls when developing software. Some configuration tweaks may be required however, as shown in the example below. -- Use a linter such as [Hadolint](https://github.com/hadolint/hadolint) to verify the content of the `Dockerfile`. - Automated code linting tools can be very useful in detecting common mistakes and pitfalls when developing software. Some configuration tweaks may be required however, as shown in the example below. +Example: - **Example**: +```console +# Ignore DL3008 (Pin versions in apt get install) +docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3008 - < Dockerfile +``` - ```console - # Ignore DL3008 (Pin versions in apt get install) - docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3008 - < Dockerfile - ``` +### 1.3 Other recommendations See the following resources for additional recommendations: - - - +- -## Step 2. Build, test locally and push to registry +## Step 2. Build, test locally and push to container registry ### 2.1 Build and test the container locally @@ -149,30 +170,32 @@ Run your image locally, so that minor errors can be immediately fixed before pro docker run : ``` -### Example +We recommend that you test containers without a network connection to best mimick their functionality inside your Safe Haven, where the container will not be able to access the internet. With Docker and Podman, this can be achieved using the `--network none` command-line parameter. + +#### Example A container with the following Dockerfile: ```dockerfile - FROM python:3.13.3@sha256:a4b2b11a9faf847c52ad07f5e0d4f34da59bad9d8589b8f2c476165d94c6b377 +FROM python:3.13.3@sha256:a4b2b11a9faf847c52ad07f5e0d4f34da59bad9d8589b8f2c476165d94c6b377 - # Create SHS directories - RUN mkdir /safe_data /safe_outputs /scratch +# Create SHS directories +RUN mkdir /safe_data /safe_outputs /scratch - # Add app files - COPY . /app - WORKDIR /app +# Add app files +COPY . /app +WORKDIR /app - # Install Python dependencies - RUN pip install --no-cache-dir -r requirements.txt +# Install Python dependencies +RUN pip install --no-cache-dir -r requirements.txt - # Default command - CMD ["python", "app.py"] +# Default command +CMD ["python", "app.py"] ``` where this is the app structure: -```console +```text . ├── app.py ├── requirements.txt @@ -187,7 +210,7 @@ docker build -t myapp:v1.1 . --platform linux/amd64 where `--platform linux/amd64` is added to ensure image compatibility with the SHS environment in case the image is being built on a different platform. -The container can then be tested with: +The container can then be tested by running: ```console docker run --rm myapp:v1.1 @@ -195,11 +218,11 @@ docker run --rm myapp:v1.1 Podman can be used equivalently to Docker in the commands above. -### 2.2 Automating Dockerfile validation +### 2.2 Automate Dockerfile validation (optional) -During development, we recommend that tools like GitHub or GitLab are used for version control and recording of the image content. Using the [`pre-commit`](https://pre-commit.com/) tool, it is possible to configure your local repository so that Hadolint (and similar tools) are run automatically each time `git commit` is run. This is recommended to ensure linting and auto-formatting tools are always run before code is pushed to GitHub. +Using the [`pre-commit`](https://pre-commit.com/) tool, it is possible to configure your local repository so that Hadolint (and similar tools) are run automatically each time `git commit` is run. This is recommended to ensure linting and auto-formatting tools are always run before code is pushed to GitHub. -To run Hadolint, include the hook in your `.pre-commit-config.yaml` file: +To run Hadolint, include the hook in a `.pre-commit-config.yaml` file: ```yaml repos: @@ -209,20 +232,20 @@ repos: - id: hadolint-docker ``` -### 2.3 Push to GHCR +### 2.3 Push to GitHub Container Registry -To prepare the image before pushing to GHCR, we recommend to save the image with a unique, descriptive tag. While it is useful to define a `latest` tag, each production image should also be tagged with a label such as the version or build date. For non-local images, the registry and repository should also be included. Images can also be tagged multiple times. +To prepare the image before pushing to the GitHub Container Registry (GHCR), we recommend saving the image with a unique, descriptive tag. While it is useful to define a `latest` tag, each production image should also be tagged with a label such as the version or build date. For non-local images, the registry and repository should also be included. Images can also be tagged multiple times. - **Example**: +For example: - ```console - docker build \ - --tag ghcr.io/my/image:v1.2.3 \ - --tag ghcr.io/my/image:latest \ - ... - ``` +```console +docker build \ + --tag ghcr.io/my/image:v1.2.3 \ + --tag ghcr.io/my/image:latest \ + ... +``` -The following commands can be used to upload the image to a GHCR repository. +The following commands can be used to upload the image to a GHCR repository, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. ```console echo "${GHCR_TOKEN}" | docker login ghcr.io -u $GHCR_NAMESPACE --password-stdin @@ -232,21 +255,19 @@ docker push "ghcr.io/$GHCR_NAMESPACE/$IMAGE:$TAG" docker logout ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - -### 2.5 Optional - Build and upload automation using GitHub Actions +### 2.5 Automate build and upload using GitHub Actions (optional) -#### Building in CI +#### Build container -Below is a sample GHA configuration which runs Hadolint, builds a container named `ghcr.io/my/repo`, then runs the [Trivy](https://aquasecurity.github.io/trivy) container scanning tool. The Trivy [SBOM](https://www.cisa.gov/sbom) report is then uploaded as a job artifact. +Below is a sample [GitHub Actions](https://github.com/features/actions) configuration, `.github/workflows/main.yaml`, which runs Hadolint, builds a container named `ghcr.io/my/repo`, then runs the [Trivy](https://aquasecurity.github.io/trivy) container scanning tool. The Trivy [SBOM](https://www.cisa.gov/sbom) report is then uploaded as a job artifact. -This assumes: +This configuration assumes that: -- The repo contains a `Dockerfile` in the top-level directory, -- The `Dockerfile` contains an `ARG` or `ENV` variable which defines the version of the packaged software. +- The repository contains a Dockerfile in the top-level directory, +- The Dockerfile contains an `ARG` or `ENV` variable which defines the version of the packaged software. +- `secrets.GITHUB_TOKEN` is a GitHub access token with 'repo' and 'write:packages' scope. ```yaml -# File .github/workflows/main.yaml name: main on: push: @@ -291,18 +312,21 @@ jobs: path: 'dependency-results.sbom.json' ``` -Note: `secrets.GITHUB_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. +!!! note -Note that manually running Hadolint via pre-commit can be skipped if you are using pre-commit and the [pre-commit.ci](https://pre-commit.ci/) service. + Manually running Hadolint via pre-commit can be skipped if you are using pre-commit and the [pre-commit.ci](https://pre-commit.ci/) service. -#### Publishing in CI +#### Publish container -__Note__ Images can also be built and pushed from your local environment as normal. +!!! note -Once the stage has been reached where your software package is ready for distribution, the GHA example above can be extended to automatically publish new image versions to the GHCR. An introduction to GHCR can be found in the GitHub docs [here](https://docs.github.com/en/packages/quickstart). + Images can also be built and pushed from your local environment as normal. + +Once the stage has been reached where your software package is ready for distribution, the GitHub Actions example above can be extended to automatically publish new image versions to the GHCR. + +After the image has been built and scanned, the image can be pushed as follows: ```yaml -# After the image has been built and scanned - name: push image run: | set -euxo pipefail @@ -310,55 +334,68 @@ Once the stage has been reached where your software package is ready for distrib docker push "${image}" ``` -## Step 3. Test in CES test environment - -EPCC provides a test environment that allows users to test their containers in a SHS-like environment without having to be logged into the SHS itself. Containers that run successfully in such environment can then be expected to perform in the same way in the SHS. - -### 3.1 Accessing test environment +## Step 3. Pull and run container inside the CES test VM -The test environment is located in the eidf147 project. Please ask your research coordinator to contact EPCC and request for you to be added to the test environment. +To test the container inside the CES test VM, first log into the CES test VM following [Accessing EIDF](../../../access). -### 3.2 Pull and run +!!! note -!!! warning "Do not use container CLIs directly" - The CES wrapper scripts **must** be used to run containers in the SHS. This is to ensure that the correct data directories are automatically made available. + While use of the CES test VM is optional, it is strongly recommended, to give you confidence that your container will successfully run within your Safe Haven. - You **must not** use commands such as `podman run ...` or `docker run ...` directly. +### 3.1 Pull container -Containers can only be used on the SHS desktop hosts using shell commands. Containers can only be pulled from the GHCR into the SHS using a `ces-pull` script. Hence containers must be pushed to GHCR for them to be used in the SHS. Although alternative methods can be used in the test environment, we encourage users to follow the exact same procedure as they would in the SHS. +Containers can only be used using shell commands. Containers can only be pulled from the GHCR into your Safe Haven using the CES tools `ces-pull` command so this command should be used within the CES test VM too. The `ces-pull` command accesses a SHS container pull proxy service through which containers are pulled. The container pull proxy service only allows containers to be pulled from authorised container registries - this is why your containers must be pushed to GHCR, as described above. -Once access has been granted to the test environment in the eidf147 project, the user can pull a container from their GHCR repository using the `ces-pull` command: +You can pull a container from your GHCR repository using the `ces-pull` command: ```sh ces-pull [] ghcr.io//: ``` -Note: `` is a GitHub user name, `` is a GitHub access token with 'read:packages' scope that allows access to the image 'ghcr.io//:'. When pulling containers into the test environment or the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. +`` can be one of `podman`, `apptainer`, or `k8s`, depending on what container runners are available in your Safe Haven. If a runtime is not specified, then `podman` is used as the default. + +`` is a GitHub user name, `` is a GitHub access token with 'read:packages' scope that allows access to the image `ghcr.io//:`. Both these arguments are mandatory. + +!!! warning "Do not use container pull commands directly" + + You **must not** use commands such as `podman pull` or `apptainer pull`. Using these commands within your Safe Haven will fail as these cannot use the SHS container pull proxy service through which containers are pulled. The EIDF CES test VM has access to a local deployment of the container pull proxy service. Using `ces-pull` within the CES test VM will give you confidence that you will be able to pull your container into your Safe Haven. -Note: The `` and `` arguments to `ces-pull` are mandatory. +!!! note -If a runtime is not specified then podman is used as the default. + Both public and private containers can be pulled from GitHub Container Registry (GHCR). However, you will need to provide both a username and an access token to do so. This is a requirement of the CES tools used to pull containers into your Safe Haven. -The container can then be run with `ces-run`: +!!! tip + + When pulling containers, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. + +### 3.2 Run container + +Once pulled, the container can be run with `ces-run`: ```sh ces-run [] ghcr.io//: ``` +`` should match that you provided to `ces-pull` i.e., one of `podman`, `apptainer`, or `k8s`, depending on what container runners are available in your Safe Haven. If a runtime is not specified, then `podman` is used as the default. + `ces-run` supports the following optional arguments under most runtimes: -- Use `--opt-file=` to specify a file containing additional options to be passed to the runtime -- Use `--arg-file=` to specify a file containing arguments to pass to the container command or entrypoint -- Use `--env-file=` to specify a file containing environment variables which will be set inside the container +- `--opt-file=` specifies a file containing additional options to be passed to the runtime. +- `--arg-file=` specifies a file containing arguments to pass to the container command or entrypoint. +- `--env-file=` specifies a file containing environment variables which will be set inside the container. + +Containers that require a GPU can be run by adding the `--gpu` option. + +!!! warning "Do not use container run commands directly" -Containers that require a GPU can be run by adding the `--gpu` option. See `ces-run [] --help` for all available options. + You **must not** use commands such as `podman run` or `apptainer run` directly as these will not mount your data directories. -We recommend to test containers without network connection to best mimick their functionality inside the SHS, where the container will not be able to access the internet. With Podman, for example, this can be achieved by passing the option `--network=none` through the `opt-file`. +!!! tip -Once the container runs successfully in the test environment, it is ready to be used inside the SHS. + See `ces-run [] --help` for all available options. -## Step 4. Pull and run container inside the SHS +## Step 4. Pull and run container inside your Safe Haven -To use the containers inside the SHS, log into the desired VM using the steps described in the EIDF User Documentation: +To use the containers inside the Safe Haven, log into your Safe Haven VM following [Safe Haven Services Access](..//safe-haven-access.md). -The same steps described in [Section 3.2](#32-pull-and-run) of this document can then be used to pull and run the container. +Now, follow the steps of [Step 3. Pull and run container inside the CES test VM](#step-3-pull-and-run-container-inside-the-ces-test-vm) to pull and run your container. diff --git a/docs/safe-haven-services/shs-container-user-guide/introduction.md b/docs/safe-haven-services/shs-container-user-guide/introduction.md index 51e7587b4..76e2aef30 100644 --- a/docs/safe-haven-services/shs-container-user-guide/introduction.md +++ b/docs/safe-haven-services/shs-container-user-guide/introduction.md @@ -3,46 +3,74 @@ ## What is the CES? The Container Execution Service (CES) has been introduced to allow project code developed and tested by researchers outside the Safe Haven Services (SHS) in personal development environments to be imported and run on the project data inside the SHS using a well-documented, transparent, secure workflow. + The primary role of the SHS is to store and share data securely; it is not intended to be a software development and testing environment. The CES helps researchers perform software development tasks in their chosen environment, rather than the restricted one offered in the SHS. -This guide describes the process of building and testing SHS-ready containers outside the SHS, specifically within a test environment created in the EIDF, so that they are ready to be pulled and used inside the SHS. + +This guide describes the process of building and testing SHS-ready containers outside the SHS, and testing them within an EIDF CES test VM, so that they are ready to be pulled and used inside the SHS. ## Accessing the CES -In order for a researcher user account to get access to CES, Research Coordinators from the corresponding Safe Haven need to submit a query to CES-enable the VM of interest (in not already enabled), then, per project-account, submit a query to configure the account. +In order for you to get access to CES, ask your Research Coordinator for your Safe Haven to submit a query to CES-enable the VM of interest (if not already enabled), then, per project-account, to submit a query to configure your account(s) to use the CES. + +## Accessing the EIDF CES test VM + +A CES test VM is available within EIDF that allows for containers to be tested in an SHS-like environment without having to be logged into the SHS itself. Containers that run successfully within this VM can then be expected to run in the same way in the SHS. + +To get access to this EIDF CES test VM, ask your Research Coordinator for your Safe Haven to submit a query that states that you wish access to the EIDF CES test VM. + +Then, if you do not already have an EPCC SAFE account, apply for an EPCC SAFE account following the instructions in [SAFE for users](https://epcced.github.io/safe-docs/safe-for-users/). + +Once you have a safe account, follow [How to request to join a project](../../access/project.md#how-to-request-to-join-a-project), and request to join the EIDF project 'eidf147: CES Development'. + +The EPCC CES team will check that your request from your Research Coordinator has been received. Once checked, an account on the CES test VM will be created and you will be notified of both your username and the CES test VM host name. + +You can then log into the CES test VM following [Accessing EIDF](../../../access). ## Getting started -We assume that users are already familiar with container concepts and have some experience in building their own images. If a user is new to container tools, the following resources would be a good starting point: +We assume that you are already familiar with container concepts and have some experience in building their own images. If you are new to container tools, the following resources would be a good starting point: - - - -It is also assumed that software development best practices are followed, such as version control using Git(Hub) and Continuous Integration (CI). This can help create container build audit trails to satisfy any IG (Information Governance) concerns about the provenance of the project code. +It is also assumed that software development best practices are followed, such as version control using Git and GitHub or GitLab, and, ideally, Continuous Integration (CI). This can help create container build audit trails to satisfy any Information Governance (IG) concerns about the provenance of the container's content. + +The [development workflow](./development-workflow.md) includes the following steps: + +1. Write a Dockerfile following both our SHS-specific advice and general recommendations. +1. Build, test the image locally and push to a container registry, for example, the GitHub Container Registry (GHCR), either locally or using a CI/CD pipeline. +1. Pull and run the container inside the CES test VM +1. Pull and run the container inside your Safe Haven on a VM enabled for container execution. + +This guide also contains a number of [workflow examples](./workflow-examples.md) designed to assist you in building SHS-ready containers. Other [container examples](./container-examples.md) are also available in our [SHS Container Samples](https://github.com/EPCCed/tre-container-samples/) repository. -The development process includes the following steps: +## SHS directories and containers -1. Create a Dockerfile following [general recommendations](./development-workflow.md#12-general-recommendations) and [SHS-specific advice](./development-workflow.md#11-shs-specific-advice). -1. Build the image and upload to the GitHub Container Registry (GHCR), either locally or using a CI/CD pipeline. -1. Test the container in the [CES test environment](./development-workflow.md#31-accessing-test-environment) to ensure it functions correctly and has no external runtime dependencies. -1. Login to a SHS desktop enabled for container execution to pull and run the container. +Before continuing with this guide, it is important that you are aware of the SHS file system directories so that they can be used correctly within their containers. -This guide contains a number of [workflow examples](./workflow-examples.md) designed to assist users in building SHS-ready containers. Other [container examples](./container-examples.md) are also available in our [SHS Container Samples](https://github.com/EPCCed/shs-container-samples/) repository. +Project data inside the SHS can be found in your `/safe_data//` directory. The CES tools automatically map, or mount, this to a directory called `/safe_data/` inside the container. -## SHS directories +Additionally, two directories are created and mapped to directories in your home directory. The first is a temporary directory that is mapped to the `/scratch/` directory within the container and which is removed after the container stops running on the host. -Before continuing with this guide, it is important that the users are aware of the SHS file system directories so that they can be used correctly within their containers. Project data inside the SHS can be found in the `/safe_data/` directory. -The CES tools automatically map this to a directory called `/safe_data` inside the container. Additionally, two directories are created and mapped to the user's home directory. The first one is `/scratch`, a temporary directory that is removed after container termination on the host system. -The second one is a unique container job output directory mapped to the `/safe_outputs` directory in the container, where output files that the users wishes to preserve should be placed. Note that containers that have been pulled into the SHS are destroyed after they have been run. Only the files written to the container outputs directory are guaranteed to be retained. +The second is a unique container job output directory mapped to the `/safe_outputs/` directory in the container, where output files that you wish to preserve can be placed. You are also free to write output files to `/safe_data/` within the container if you have been granted read-write access to your `/safe_data//` directory. -| Directory on host system | Directory in container | Intended use -| -------- | ------- | ------- | -| `/safe_data//`|/`safe_data`|Read-only access if required by IG, or read-write access, to data and other project files.| -|`~/outputs_` |`/safe_outputs` |Will be created at container startup as an empty directory. Intended for any outputs: logs, data, models. Content saved in this directory will be retained after the container exits.| -|`~/scratch_`|`/scratch`|Temporary directory that is removed after container termination on the host system. Any temporary files should be placed here.| +| Directory on host system | Directory in container | Intended use | +| ------------------------ | ---------------------- | ------------ | +| `/safe_data//` | `/safe_data/`| Read-only access if required by IG, or read-write access, to data and other project files. Can be used for outputs, depending on your Safe Haven's IG and usage conventions. | +| `~/outputs_/` | `/safe_outputs/` | Created at container startup as an empty directory. Can be used for any outputs e.g., logs, data, models, depending on your Safe Haven's IG and usage conventions. Content saved in this directory will be retained after the container exits. | +| `~/scratch_/` | `/scratch/`| Temporary directory that is removed when the container exits. Any temporary files should be placed here. | -Temporary files can also be written into any directory in the container’s internal file system. However, the use of `/scratch` can be more efficient if the service is able to mount it on high-performing storage devices. +!!! warning + + Only files created in `/safe_data/` or `/safe_outputs/` will continue to be available once a container exits. + +!!! note + + Temporary files can also be written into any directory in the container’s internal file system. However, the use of `/scratch/` can be more efficient if the service is able to mount it on high-performing storage devices. ## Advising Information Governance of required software stack -Projects must establish that the software stack they intend to import in the container is acceptable for the project’s IG approvals. Projects should only seek to use container-based software if the standard SHS desktop environment is not sufficient for the research scope. However, it is broadly understood that the standard desktop software, whilst useful in most cases, is inadequate for many purposes and specifically ML, and software import using containers is intended to address this. +Projects **must** establish that the software stack they intend to import in the container is acceptable for the project’s IG approvals. + +Projects should only seek to use container-based software if the standard SHS desktop environment is not sufficient for the research scope. However, it is broadly understood that the standard SHS desktop software, whilst useful in most cases, is inadequate for many purposes, for example machine learning, and software import using containers is intended to address this. diff --git a/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md b/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md index 86aedf735..805726d21 100644 --- a/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md +++ b/docs/safe-haven-services/shs-container-user-guide/workflow-examples.md @@ -1,6 +1,6 @@ # Workflow Examples -The following sections will guide the user in the process of creating different types of containers. +These examples will help guide you in the process of creating different types of containers. For a complete list of examples, please see our SHS Container Samples repository, at [EPCCed/shs-container-samples](https://github.com/EPCCed/tre-container-samples). @@ -21,13 +21,13 @@ numpy torch ``` -and `torch_gpy_test.py` is any script that performs a test task using PyTorch. +and `torch_gpy_test.py` is any script that performs a simple task using PyTorch. -### E1 - Step 1. Writing the Dockerfile +### Example 1 - Step 1. Writing a Dockerfile In our Dockerfile, we want to start by using an officially supported image that already contains as much of the software required to run the script as possible. -In our case, we want a ready-made python 3 image to start with, which we can find in [DockerHub](https://hub.docker.com/search?q=python). The latest stable version at the time of writing is [3.13.3](https://hub.docker.com/layers/library/python/3.13.3/images/sha256-981c77781aa563fc22ee5936fdd37e16679e3b28d32351430a6aede491f6e8b1), so we will use this and include the digest in our Dockerfile. +In our case, we want a ready-made Python 3 image to start with, which we can find in [DockerHub](https://hub.docker.com/search?q=python). The latest stable version at the time of writing is [3.13.3](https://hub.docker.com/layers/library/python/3.13.3/images/sha256-981c77781aa563fc22ee5936fdd37e16679e3b28d32351430a6aede491f6e8b1), so we will use this and include the digest in our Dockerfile. Next, we need to set up the SHS directories, copy our files into the container, install the necessary packages and finally execute the script. This process can be accomplished with the following Dockerfile: @@ -49,24 +49,24 @@ COPY torch_gpu_test.py . CMD ["python", "./torch_gpu_test.py"] ``` -### E1 - Step 2. Build and push to GHCR +### Example 1 - Step 2. Build, test locally and push to container registry -As mentioned in our guidelines, it is good practice to check the Dockerfile with a linting tool to detect common mistakes before building our container: +As mentioned in our container development workflow, it is good practice to check the Dockerfile with a linting tool to detect common mistakes before building our container: ```sh # Ignore DL3013 to allow --upgrade docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3013 - < Dockerfile ``` -We can then define our GHCR variables: +Here we deliberately ignore hadolint flags if the selected features are required by the container to run successfully. + +We can then define our GitLab Container Registry (GHCR) variables, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope: ```sh export GHCR_NAMESPACE=mynamespace export GHCR_TOKEN=mytoken ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - We can now build our container: ```sh @@ -88,30 +88,35 @@ docker push "ghcr.io/$GHCR_NAMESPACE/pytorch-test:latest" docker logout ``` -### E1 - Step 3. Test in CES test environment +### Example 1 - Step 3. Pull and run container inside the CES test VM -Log into the 'ces-dev02' VM of the project EIDF147, which is the designated test environment for the CES. +To test the container inside the CES test VM, first log into the CES test VM following [Accessing EIDF](../../../access). -Then, pull and run the container using the commands: +Pull the container using the following command, where both the `$GHCR_NAMESPACE` and `$GHCR_TOKEN` arguments to `ces-pull` are mandatory: ```sh ces-pull podman $GHCR_NAMESPACE $GHCR_TOKEN ghcr.io/$GHCR_NAMESPACE/pytorch-test:v1.1 -ces-run podman --gpu ghcr.io/$GHCR_NAMESPACE/pytorch-test:v1.1 ``` -Note: The namespace and token arguments to `ces-pull` are mandatory. +!!! tip + + When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. -Note: When pulling containers into the test environment, instead of using the GitHub access token you used to push the container, it is **recommended** that you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. +Now run the container using the following command: -### E1 - Step 4. Pull and run in the SHS +```sh +ces-run podman --gpu ghcr.io/$GHCR_NAMESPACE/pytorch-test:v1.1 +``` -The container can be pulled and run inside the SHS using the same commands as the [previous step](#e1-step-3-test-in-ces-test-environment). +### Example 1 - Step 4. Pull and run container inside your Safe Haven -Note: When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. +The container can be pulled and run inside the SHS using the same commands as in the [previous step](#example-1-step-3-pull-and-run-container-inside-the-ces-test-vm). ## Example 2 - Python ML -This example demonstrates how to build a container which requires a ML model that would normally be downloaded from the internet when first run. Such models are typically cached in a hidden directory so it can be difficult to understand how to manually download the model, where to put it, and how to load it. +This example demonstrates how to build a container which requires a machine learning (ML) model that would normally be downloaded from the internet when first run. + +Such models are typically cached in a hidden directory so it can be difficult to understand how to manually download the model, where to put it, and how to load it. The approach we take is to run a sample piece of code during the container build phase, which downloads the model to the hidden cache directory. This then becomes part of the container. @@ -133,7 +138,7 @@ import easyocr reader = easyocr.Reader(['en']) ``` -`test_easyocr.py`: +`test_easyocr.py` is defined as: ```py #!/usr/bin/env python3 @@ -166,9 +171,9 @@ for filename in sys.argv[1:]: and `doc1.png` is an image that contains text. -### E2 - Step 1. Writing the Dockerfile +### Example 2 - Step 1. Writing a Dockerfile -Similarly to the PyTorch example, we will start our Dockerfile using the `python:3` image: +As for the PyTorch example, we will base our Dockerfile on the `python:3` image: ```dockerfile FROM python:3.13.3@sha256:a4b2b11a9faf847c52ad07f5e0d4f34da59bad9d8589b8f2c476165d94c6b377 @@ -182,7 +187,7 @@ COPY cache_easyocr.py . RUN python ./cache_easyocr.py ``` -As there is no internet in the SHS, we need to make sure that HuggingFace does not attempt hub access: +As there is no internet access from within the SHS, we need to make sure that HuggingFace does not attempt to access its hub: ```dockerfile ENV HF_HUB_OFFLINE=1 @@ -219,26 +224,24 @@ COPY test_easyocr.py doc1.png /src/ ENTRYPOINT [ "python3", "/src/test_easyocr.py", "/src/doc1.png"] ``` -### E2 - Step 2. Build and push to GHCR +### Example 2 - Step 2. Build, test locally and push to container registry -As mentioned in our guidelines, before building our container we first want to check the Dockerfile with a linting tool to detect common mistakes: +As mentioned in our container development workflow, it is good practice to check the Dockerfile with a linting tool to detect common mistakes before building our container: ```sh # Ignore DL3013, DL3042 and DL3045 docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3013 --ignore DL3042 --ignore DL3045 - < Dockerfile ``` -Note that we can deliberately ignore hadolint flags if the selected features are required by the container to run successfully. +Here we deliberately ignore hadolint flags if the selected features are required by the container to run successfully. -We then define our GHCR variables: +We can then define our GHCR variables, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope: ```sh export GHCR_NAMESPACE=mynamespace export GHCR_TOKEN=mytoken ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - We can now build our container: ```sh @@ -271,28 +274,33 @@ docker push "ghcr.io/$GHCR_NAMESPACE/python-ml-test:latest" docker logout ``` -### E2 - Step 3. Test in CES test environment +### Example 2 - Step 3. Pull and run container inside the CES test VM + +To test the container inside the CES test VM, first log into the CES test VM following [Accessing EIDF](../../../access). -As with the previous example, we now want to test our container in the designated test environment for the CES. Log into the 'ces-dev02' VM of the project EIDF147, then pull and run the container using the commands: +Pull the container using the following command, where both the `$GHCR_NAMESPACE` and `$GHCR_TOKEN` arguments to `ces-pull` are mandatory: ```sh ces-pull podman $GHCR_NAMESPACE $GHCR_TOKEN ghcr.io/$GHCR_NAMESPACE/python-ml-test:v1.1 -ces-run podman --gpu ghcr.io/$GHCR_NAMESPACE/python-ml-test:v1.1 ``` -Note: The namespace and token arguments to `ces-pull` are mandatory. +!!! tip -Note: When pulling containers into the test environment, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. + When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. -### E2 - Step 4. Pull and run in the SHS +Now run the container using the following command: -We can import and run the container in the SHS using the commands from the [earlier step](#e2-step-3-test-in-ces-test-environment). +```sh +ces-run podman --gpu ghcr.io/$GHCR_NAMESPACE/python-ml-test:v1.1 +``` + +### Example 2 - Step 4. Pull and run container inside your Safe Haven -Note: When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. +The container can be pulled and run inside the SHS using the same commands as in the [previous step](#example-2-step-3-pull-and-run-container-inside-the-ces-test-vm). ## Example 3 - Interactive RStudio Rocker container -This section guides users through the process of creating a Rocker RStudio container that can be imported into the SHS and used for data analysis. In the example, a script is copied into the container, and the necessary packages are installed to ensure it runs correctly. Finally, RStudio is accessed from the host, allowing users to interact with the application as if it were running natively. +This section guides you through the process of creating a Rocker RStudio container that can be imported into the SHS and used for data analysis. In the example, a script is copied into the container, and the necessary packages are installed to ensure it runs correctly. Finally, RStudio is accessed from the host, allowing you to interact with the application as if it were running natively. We will assume that this is the directory structure of our files: @@ -335,9 +343,9 @@ p + geom_text(data=annotation, aes( x=x, y=y, label=label), , ggsave("test_figure.png") ``` -### E3 - Step 1. Writing the Dockerfile +### Example 3 - Step 1. Writing a Dockerfile -We start the process by selecting the parent image. Searching for Rocker in [DockerHub](https://hub.docker.com/search?q=rocker), we are presented with a number of options. Only some of these originate from official sources. We want to select the most appropriate image published from a reputable source, in our case the [Rocker Project](https://rocker-project.org/). +Again, we start by selecting a base image. Searching for Rocker in [DockerHub](https://hub.docker.com/search?q=rocker), presents us with a number of options. Only some of these originate from official sources. We want to select the most appropriate image published from a reputable source, in our case the [Rocker Project](https://rocker-project.org/). For this example, we choose the latest version of Rocker RStudio, [rocker/rstudio](https://hub.docker.com/r/rocker/rstudio) from [Docker Hub](https://hub.docker.com/). @@ -353,7 +361,7 @@ The next step is to include the SHS directories: RUN mkdir /safe_data /safe_outputs /scratch ``` -We then want to copy our script inside the container. As we do not mean to preserve the script, only its output, we will copy it in a new `/src` directory. Note that we cannot save the files to `/scratch` as they would otherwise be overwritten during the mounting process to the SHS directories. To avoid repeating the `RUN` command, we will simply add our new directory to it: +We then want to copy our script inside the container. As we do not mean to preserve the script, only its output, we will copy it in a new `/src` directory. Note that we cannot save the files to `/scratch` as they would otherwise be overwritten when running the container using the CES tools. To avoid repeating the `RUN` command, we will add our new directory to it: ```dockerfile RUN mkdir /safe_data /safe_outputs /scratch /src @@ -391,24 +399,24 @@ WORKDIR /src RUN r install_packages.R ``` -### E3 - Step 2. Build and push to GHCR +### Example 3 - Step 2. Build, test locally and push to container registry -Before building our container, we want to check the Dockerfile with a linting tool to detect common mistakes: +As mentioned in our container development workflow, it is good practice to check the Dockerfile with a linting tool to detect common mistakes before building our container: ```sh # Ignore DL3008 (Pin versions in apt get install) docker run --pull always --rm -i docker.io/hadolint/hadolint:latest hadolint --ignore DL3008 - < Dockerfile ``` -We can then define our GHCR variables: +Here we deliberately ignore hadolint flags if the selected features are required by the container to run successfully. + +We can then define our GHCR variables, where `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope: ```sh export GHCR_NAMESPACE=mynamespace export GHCR_TOKEN=mytoken ``` -Note: `GHCR_TOKEN` needs to be a GitHub access token with 'repo' and 'write:packages' scope. - We can now build our container: ```sh @@ -421,9 +429,13 @@ and run it locally: docker run --rm -e "PASSWORD=test" -p 8787:8787 ghcr.io/$GHCR_NAMESPACE/rocker-test:v1.1 ``` -We can then access RStudio by navigating to 'localhost:8787' in a browser. At the login page, type 'root' and 'test' for username and password respectively. Note that you will only be 'root' within the context of the container and not outside of it. The same applies in the test environment and SHS. +We can now open a browser tab and access RStudio at `localhost:8787`. We can log in using the credentials `root` for the username and `test` for the password. + +!!! note + + You are only 'root' within the context of the container, not outside of it. -Once we made sure the container runs, we can push our image to GHCR using our namespace and token: +If the container runs without errors, we can push our image to GHCR using our namespace and token: ```sh echo "${GHCR_TOKEN}" | docker login ghcr.io -u $GHCR_NAMESPACE --password-stdin @@ -432,28 +444,28 @@ docker push "ghcr.io/$GHCR_NAMESPACE/rocker-test:latest" docker logout ``` -### E3 - Step 3. Test in CES test environment +### Example 3 - Step 3. Pull and run container inside the CES test VM -Log into the 'ces-dev02' VM of the project EIDF147, which is the designated test environment for the CES. +To test the container inside the CES test VM, first log into the CES test VM following [Accessing EIDF](../../../access). -Rocker is one of those containers that requires to be started by the 'root' user. As such, it should be run inside the SHS - and our test environment - using podman. To pull the container using podman as our container engine, we use the command: +Pull the container using the following command, where both the `$GHCR_NAMESPACE` and `$GHCR_TOKEN` arguments to `ces-pull` are mandatory: ```sh ces-pull podman $GHCR_NAMESPACE $GHCR_TOKEN ghcr.io/$GHCR_NAMESPACE/rocker-test:v1.1 ``` -Note: The namespace and token arguments to `ces-pull` are mandatory. +!!! tip -Note: When pulling containers into the test environment, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. + When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. -The Rocker container was designed to be run using docker. In order for it to run successfully with podman, the container directories `/var/lib/rstudio-server` and `/run` need to be mounted to a tmpfs. As such, the following options are required: +The Rocker container was designed to be run using Docker. In order for it to run successfully with podman, the container directories `/var/lib/rstudio-server` and `/run` need to be mounted to a tmpfs. As such, the following options are required: ```sh '--mount type=tmpfs,destination=/var/lib/rstudio-server' '--mount type=tmpfs,destination=/run' ``` -Our full `opt-file.txt` then looks like this: +A `ces-run` `opt-file.txt` can include these and other options required: ```text -p 8787:8787 @@ -462,19 +474,19 @@ Our full `opt-file.txt` then looks like this: --mount type=tmpfs,destination=/run ``` -If we want to set our password, we can add it to the `env-file.txt` as follows: +If we want to set our password, we can add it to a `ces-run` `env-file.txt` as follows, replacing `` with a strong password: ```text -PASSWORD=test +PASSWORD= ``` -We can then run our container: +Now run the container using the following command: ```sh ces-run podman --opt-file opt-file.txt --env-file env-file.txt ghcr.io/$GHCR_NAMESPACE/rocker-test:v1.1 ``` -To streamline the process, we can place all the necessary commands into an executable script `run.sh` as follows: +To streamline the process, we can place all the necessary commands into an executable script, `run.sh`, as follows, replacing `` with a strong password: ```sh opt_file="./opt-file.txt" @@ -492,21 +504,29 @@ env_file="./env-file.txt" if [ -f "$env_file" ] ; then rm "$env_file" fi -echo -e "PASSWORD=test" >> ${env_file} +echo -e "PASSWORD=" >> ${env_file} ces-run podman --opt-file $opt_file --env-file $env_file ghcr.io/$GHCR_NAMESPACE/rocker-test:v1.1 ``` -After executing the `run.sh` script, we can open a browser tab and access RStudio at `localhost:8787`. As done previously during the local test, we can log in using the credentials `root` and `test` for username and password respectively. From within RStudio, we can then run our script, which can be found in `/src`, and then copy our output to `/safe_output` so that it can be preserved after we exit the container. +The script can be run as follows: + +```sh +bash run.sh +``` + +After starting the container, either via `ces-run podman` or `bash run.sh`, we can open a browser tab and access RStudio at `localhost:8787`. We can log in using the credentials `root` for the username and our password selected above. From within RStudio, we can then run our script, which can be found in `/src`, and then copy our output to `/safe_output` so that it can be preserved after we exit the container. The container is running successfully if: -- The log-in is successful. -- The rstudio user has full access to SHS directories `/safe_data`, `/safe_outputs` and `/scratch`. -- The files saved in `/safe_outputs` and `/safe_data` (when writing permission is granted by IG) have correct permission on the host, that is they belong to the logged-in user. +- Your log-in is successful. +- The 'root' user - your user within the container - has full access to SHS directories `/safe_data`, `/safe_outputs` and `/scratch`. +- The files saved in `/safe_outputs` and `/safe_data` (when writing permission is granted by IG) have correct permission on the host, that is, you. + +!!! note -### E3 - Step 4. Pull and run in the SHS + You are only 'root' within the context of the container, not outside of it. -The container can then be imported inside the SHS using the same commands as [Step 3](#e3-step-3-test-in-ces-test-environment). +### Example 3 - Step 4. Pull and run container inside your Safe Haven -Note: When pulling containers into the SHS, instead of using the GitHub access token you used to push the container, it is **recommended** you use a GitHub access token with 'read:packages' scope only. Restricting where you use your read-write token can keep your GHCR secure. +The container can be pulled and run inside the SHS using the same commands as in the [previous step](#example-3-step-3-pull-and-run-container-inside-the-ces-test-vm).