[skill] evaluation: add GDPVal (NeMo Gym Stirrup agent) support #2039
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # gdpval-sif.sh — ensure the GDPVal Stirrup Apptainer SIF exists on THIS cluster. | ||
| # | ||
| # Build-if-absent, reuse-if-present. Self-contained: the SIF is built and reused | ||
| # on the TARGET cluster's own filesystem — this NEVER copies a SIF from another | ||
| # cluster. Idempotent, so it's safe to run before every `nel run`; a subsequent | ||
| # run reuses the built SIF instantly. | ||
| # | ||
| # Usage: | ||
| # .agents/scripts/gdpval-sif.sh [<sif-dir-or-file>] [--commit <sha>] [--force|--check] | ||
| # <sif-dir-or-file> Persistent path on the target cluster's shared FS. | ||
| # DEFAULTS to $GDPVAL_SIF_DIR (from .env) when omitted. A | ||
| # directory -> <dir>/$GDPVAL_SIF_NAME (default python-3.13.gdpval.sif, | ||
| # matching the example config); a *.sif path | ||
| # is used verbatim. Bind-mount this SAME dir into the eval | ||
| # container at /gdpval/sif (see recipes/examples/gym_gdpval/). | ||
| # --commit <sha> NeMo Gym commit whose gdpval.def to build. Keep in sync | ||
| # with the config's install_on_the_fly.commit. | ||
| # --force Rebuild even if the SIF already exists. | ||
| # --check Verify-only preflight: exit 0 if the expected SIF exists, | ||
| # nonzero (listing what IS there) if not. Never builds. | ||
| # Use before `nel run` — NEL's mount validation is `test -d` | ||
| # and cannot see a missing/misnamed SIF file. | ||
| # | ||
| # Requires `apptainer` (or `singularity`) on PATH with unprivileged/fakeroot | ||
| # build support, plus network egress to GitHub/base image. Run on a node that has | ||
| # it — a login node, or (preferred for the ~30-min build) the CPU partition: | ||
| # srun -p cpu -t 01:00:00 --pty \ | ||
| # .agents/scripts/gdpval-sif.sh /lustre/<...>/gdpval/sif | ||
| # | ||
| # Env overrides: GDPVAL_GYM_COMMIT, GDPVAL_SIF_NAME, APPTAINER_BIN. | ||
| set -euo pipefail | ||
|
|
||
| # Keep GDPVAL_GYM_COMMIT in sync with install_on_the_fly.commit in the config. | ||
| GDPVAL_GYM_COMMIT="${GDPVAL_GYM_COMMIT:-dd41196f620f2af99947d776cbe5da9439d2a08d}" # pragma: allowlist secret | ||
| GDPVAL_SIF_NAME="${GDPVAL_SIF_NAME:-python-3.13.gdpval.sif}" | ||
| APPTAINER_BIN="${APPTAINER_BIN:-}" | ||
|
|
||
| _log() { printf '\033[2m %s\033[0m\n' "$*" >&2; } | ||
| _die() { printf '\033[31mgdpval-sif: %s\033[0m\n' "$*" >&2; exit 1; } | ||
| _usage() { sed -n '/^# gdpval-sif\.sh/,/^set -euo/p' "$0" | sed 's/^# \{0,1\}//; /^set -euo/d'; } | ||
|
|
||
| # --- parse args --- | ||
| target=""; force=0; check=0 | ||
| while [[ $# -gt 0 ]]; do | ||
| case "$1" in | ||
| --commit) GDPVAL_GYM_COMMIT="${2:?--commit needs a value}"; shift 2 ;; | ||
| --force) force=1; shift ;; | ||
| --check) check=1; shift ;; | ||
| -h|--help) _usage; exit 0 ;; | ||
| -*) _die "unknown flag: $1 (see --help)" ;; | ||
| *) [[ -z "$target" ]] || _die "unexpected extra arg: $1"; target="$1"; shift ;; | ||
| esac | ||
| done | ||
| # Default to $GDPVAL_SIF_DIR (.env) when no path is given, so agents run it hands-free. | ||
| target="${target:-${GDPVAL_SIF_DIR:-}}" | ||
| [[ -n "$target" ]] || { _usage; _die "no path given and GDPVAL_SIF_DIR is unset — pass a dir or set GDPVAL_SIF_DIR (see recipes/env.example)"; } | ||
|
|
||
| # --- resolve dir vs *.sif --- | ||
| if [[ "$target" == *.sif ]]; then | ||
| sif="$target"; sif_dir="$(dirname "$target")" | ||
| else | ||
| sif_dir="$target"; sif="$sif_dir/$GDPVAL_SIF_NAME" | ||
| fi | ||
| # --- verify-only mode (preflight) --- | ||
| # NEL's submit-time mount validation runs `test -d`, so it only proves the SIF *dir* | ||
| # exists — a dir holding the WRONG sif name (e.g. python-3.12 after a gym bump to a | ||
| # 3.13 def) passes validation, and the Stirrup agent then SILENTLY falls back to | ||
| # non-sandboxed exec. Run this before submitting to fail loudly instead. | ||
| if [[ "$check" -eq 1 ]]; then | ||
| if [[ -f "$sif" ]]; then | ||
| _log "SIF present: $sif ($(du -h "$sif" 2>/dev/null | cut -f1))" | ||
| echo "$sif"; exit 0 | ||
| fi | ||
| printf '\033[31mgdpval-sif: MISSING expected SIF: %s\033[0m\n' "$sif" >&2 | ||
| [[ -d "$sif_dir" ]] && { echo " dir exists but does not contain it; found:" >&2 | ||
| ls -1 "$sif_dir"/*.sif 2>/dev/null | sed 's/^/ /' >&2 || echo " (no .sif files)" >&2; } | ||
| echo " Build it with: $0 ${sif_dir} (or --commit <gym-sha> for a different def)" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| mkdir -p "$sif_dir" || _die "cannot create SIF dir: $sif_dir" | ||
|
|
||
| # --- reuse if present --- | ||
| if [[ -f "$sif" && "$force" -eq 0 ]]; then | ||
| _log "reusing existing SIF (no rebuild): $sif" | ||
| echo "$sif"; exit 0 | ||
| fi | ||
|
|
||
| # --- locate apptainer/singularity --- | ||
| if [[ -z "$APPTAINER_BIN" ]]; then | ||
| APPTAINER_BIN="$(command -v apptainer || command -v singularity || true)" | ||
| fi | ||
| [[ -n "$APPTAINER_BIN" ]] || _die "apptainer/singularity not found on PATH. Run on a node that has it \ | ||
| (e.g. 'module load apptainer', or inside the eval image). This script does NOT copy a SIF from another cluster." | ||
|
|
||
| def_url="https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/${GDPVAL_GYM_COMMIT}/responses_api_agents/stirrup_agent/containers/gdpval.def" | ||
| tmp="${sif_dir}/.build.$$.${GDPVAL_SIF_NAME}" | ||
| def_local="${sif_dir}/.gdpval.$$.def" | ||
| lock="${sif_dir}/.gdpval-sif.lock" | ||
|
|
||
| # --- build under a flock (double-checked) so concurrent runs don't double-build --- | ||
| exec 9>"$lock" || _die "cannot open lock file: $lock" | ||
| _log "acquiring build lock ($lock) ..." | ||
| flock 9 | ||
| # Re-check inside the lock: another builder may have finished while we waited. | ||
| if [[ -f "$sif" && "$force" -eq 0 ]]; then | ||
| _log "another builder produced it: $sif" | ||
| echo "$sif"; exit 0 | ||
| fi | ||
|
|
||
| _log "building GDPVal SIF (this can take ~20-40 min)" | ||
| _log " gym commit: ${GDPVAL_GYM_COMMIT}" | ||
| _log " def: ${def_url}" | ||
| _log " dest: ${sif}" | ||
| rm -f "$tmp" "$def_local" | ||
| # apptainer build cannot take a remote def URL as its source — fetch the def to a | ||
| # local file first, then build from it. | ||
| if command -v curl >/dev/null 2>&1; then curl -fsSL "$def_url" -o "$def_local" | ||
| else wget -qO "$def_local" "$def_url"; fi | ||
| [ -s "$def_local" ] || { rm -f "$def_local"; _die "failed to download def from $def_url"; } | ||
| # Prefer --fakeroot (needs an /etc/subuid entry for the build user); fall back to an | ||
| # unprivileged build where fakeroot is unavailable. | ||
| if "$APPTAINER_BIN" build --fakeroot "$tmp" "$def_local"; then | ||
| : | ||
| elif "$APPTAINER_BIN" build "$tmp" "$def_local"; then | ||
| _log "built without --fakeroot (unprivileged mode)" | ||
| else | ||
| rm -f "$tmp" "$def_local" | ||
| _die "apptainer build failed (see output above)." | ||
| fi | ||
| rm -f "$def_local" | ||
|
|
||
| # Atomic publish: a partial build never looks complete. | ||
| mv -f "$tmp" "$sif" || { rm -f "$tmp"; _die "failed to move built SIF into place: $sif"; } | ||
| _log "done: $sif" | ||
| echo "$sif" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Shared prepare/teardown snippets — include via: defaults: [_gym_prepare, _self_] | ||
| # | ||
| # Compensates for the eval image's deployment-oriented packaging when run | ||
| # standalone. Interpolate into each benchmark's command: | ||
| # ${gym_prepare.prepare} at the start (activate venv, checkout pin, repair venvs) | ||
| # ${gym_prepare.run} as the last line (data prep + rollout collection run in | ||
| # its own session, then process-group reaped so the run | ||
| # finalizes — see the run: comment for why) | ||
| # Remove once the eval image ships complete, ray-consistent venvs and Gym's | ||
| # shutdown reaps server process groups. | ||
|
|
||
| gym_prepare: | ||
| # Common command preamble shared by every benchmark: activate the baked Gym | ||
| # venv, optionally checkout the install_on_the_fly pin (only when /opt/Gym is a | ||
| # git repo; baked images aren't), then repair the image's incomplete per-server | ||
| # venvs. The repair: (1) rewrites each component requirements.txt — drops the | ||
| # editable "-e nemo-gym[dev]" line (it forces a ray>=2.55.1 re-resolve vs the | ||
| # image's pinned ray 2.49.2, breaking venv-less servers like arena_judge) and | ||
| # ensures ray==2.49.2 + tqdm; (2) installs the fixed requirements into each | ||
| # baked (skeleton) sub-venv; (3) fronts the main venv on PYTHONPATH so nemo_gym | ||
| # + framework deps resolve for server processes. | ||
| # NOTE: avoid bash ${VAR} here — OmegaConf parses ${...}; $(...) / $r / $v are fine. | ||
| # NOTE: every step is guarded (|| true / -q grep) so it is safe under set -e. | ||
| prepare: |- | ||
| set -ex | ||
| cd /opt/Gym | ||
| export UV_CACHE_DIR=/opt/cache/uv | ||
| source .venv/bin/activate | ||
| # install_on_the_fly: checkout the pin when /opt/Gym is a git repo; some | ||
| # images bake Gym at a fixed version (not a git repo) — use it as-is. | ||
| if [ -d .git ]; then | ||
| git remote add oss_pin "{{config.params.extra.nemo_gym.install_on_the_fly.url}}" 2>/dev/null || true | ||
| git fetch oss_pin | ||
| git checkout "{{config.params.extra.nemo_gym.install_on_the_fly.commit}}" | ||
| echo "=== NeMo Gym commit ===" && git rev-parse HEAD | ||
| else | ||
| echo "=== /opt/Gym is not a git repo; using baked-in Gym version ===" | ||
| fi | ||
| for r in /opt/Gym/responses_api_models/*/requirements.txt \ | ||
| /opt/Gym/responses_api_agents/*/requirements.txt \ | ||
| /opt/Gym/resources_servers/*/requirements.txt; do | ||
| [ -f "$r" ] || continue | ||
| grep -vE '^[[:space:]]*-e ' "$r" > "$r.fixed" || true | ||
| grep -qiE '^ray([<>=[]|$)' "$r.fixed" 2>/dev/null || echo 'ray==2.49.2' >> "$r.fixed" | ||
| grep -qiE '^tqdm' "$r.fixed" 2>/dev/null || echo 'tqdm' >> "$r.fixed" | ||
| mv "$r.fixed" "$r" 2>/dev/null || true | ||
| done | ||
| for v in /opt/Gym/responses_api_models/*/.venv \ | ||
| /opt/Gym/responses_api_agents/*/.venv \ | ||
| /opt/Gym/resources_servers/*/.venv; do | ||
| [ -d "$v" ] || continue | ||
| d="$(dirname "$v")" | ||
| [ -f "$d/requirements.txt" ] && uv pip install --python "$v/bin/python" -q -r "$d/requirements.txt" || true | ||
| done | ||
| export PYTHONPATH="/opt/Gym:$(/opt/Gym/.venv/bin/python -c 'import site; print(site.getsitepackages()[0])')" | ||
|
Comment on lines
+27
to
+70
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
file=".agents/skills/evaluation/recipes/examples/gym_gdpval/_gym_prepare.yaml"
printf '%s\n' '--- target file ---'
cat -n "$file" | sed -n '1,120p'
printf '%s\n' '--- PYTHONPATH references ---'
rg -n -C 3 'PYTHONPATH|gym_prepare|install_on_the_fly' .agents/skills/evaluation/recipes .agents/skills/evaluation 2>/dev/null | sed -n '1,240p'
printf '%s\n' '--- repository status ---'
git status --shortRepository: NVIDIA/Model-Optimizer Length of output: 34741 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- all repository PYTHONPATH references ---'
rg -n -i 'PYTHONPATH' . | sed -n '1,240p'
printf '%s\n' '--- relevant Gym evaluation files ---'
fd -t f . .agents/skills/evaluation | rg '(^|/)(gdpval|gym|slurm|execution|container|def|env)' | sed -n '1,240p'
printf '%s\n' '--- environment/container command context ---'
rg -n -C 5 'GDPVAL_CONTAINER_PATH|apptainer|singularity|--env|env ' .agents/skills/evaluation .agents/scripts 2>/dev/null | sed -n '1,320p'Repository: NVIDIA/Model-Optimizer Length of output: 33926 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- Gym GDPVal execution and environment configuration ---'
cat -n .agents/skills/evaluation/recipes/examples/gym_gdpval/example_gym_gdpval.yaml | sed -n '45,100p;185,270p'
printf '%s\n' '--- GDPVal reference container details ---'
cat -n .agents/skills/evaluation/references/gym-gdpval.md | sed -n '1,180p'
printf '%s\n' '--- SLURM environment guidance ---'
cat -n .agents/skills/evaluation/references/slurm.md | sed -n '1,180p'Repository: NVIDIA/Model-Optimizer Length of output: 22790 🏁 Script executed: #!/bin/bash
set -eu
cat -n .agents/skills/evaluation/recipes/examples/gym_gdpval/example_gym_gdpval.yaml | sed -n '95,190p'
printf '%s\n' '--- exact PYTHONPATH assignment semantics ---'
python3 - <<'PY'
import os
old = os.environ.get("PYTHONPATH")
gym = "/opt/Gym:/opt/Gym/.venv/lib/python3.13/site-packages"
os.environ["PYTHONPATH"] = gym
print({"inherited": old, "assigned": os.environ["PYTHONPATH"], "inherited_preserved": old is None or old in os.environ["PYTHONPATH"]})
PYRepository: NVIDIA/Model-Optimizer Length of output: 5672 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
import subprocess
assignment = 'export PYTHONPATH="/opt/Gym:$(printf %s /opt/Gym/.venv/lib/python3.13/site-packages)"'
for inherited in ["/host/component", "/host/one:/host/two", None]:
env = {} if inherited is None else {"PYTHONPATH": inherited}
result = subprocess.run(
["bash", "-c", assignment + '; printf "%s" "$PYTHONPATH"'],
env=env,
text=True,
capture_output=True,
check=True,
)
print(f"inherited={inherited!r} -> assigned={result.stdout!r}")
proposed = 'export PYTHONPATH="/opt/Gym:$(printf %s /opt/Gym/.venv/lib/python3.13/site-packages)${PYTHONPATH:+:$PYTHONPATH}"'
for inherited in ["/host/component", None]:
env = {} if inherited is None else {"PYTHONPATH": inherited}
result = subprocess.run(
["bash", "-c", proposed + '; printf "%s" "$PYTHONPATH"'],
env=env,
text=True,
capture_output=True,
check=True,
)
print(f"proposed inherited={inherited!r} -> assigned={result.stdout!r}")
PY
printf '%s\n' '--- YAML interpolation warning near the assignment ---'
sed -n '31,40p;68,71p' .agents/skills/evaluation/recipes/examples/gym_gdpval/_gym_prepare.yamlRepository: NVIDIA/Model-Optimizer Length of output: 1583 Preserve inherited Line 70 discards any 🤖 Prompt for AI Agents |
||
|
|
||
| # Data prep + rollout collection. Interpolate as the LAST line of each | ||
| # benchmark's command via ${gym_prepare.run} (after ${gym_prepare.prepare} and | ||
| # any benchmark-specific prep). | ||
| # | ||
| # Rollout collection is run in its OWN session (setsid) so its entire process | ||
| # tree — servers + their multiprocessing pools + Ray workers — can be reaped by | ||
| # process group. Why: ng_e2e's own cleanup (cli.py shutdown) only SIGINTs/SIGKILLs | ||
| # the tracked server PIDs after a 1s grace; it does NOT reap each server's child | ||
| # tree, so pools/Ray workers orphan and re-parent to the launcher (which is | ||
| # blocked reading our stdout) — they hold stdout open and the run never finalizes. | ||
| # (Invisible in the deployment flow because the whole node is discarded; only bites | ||
| # the deployment-free inline path here.) The inner shell records the session PGID | ||
| # (its own $$, before exec) so we can target it; real exit code is captured from | ||
| # wait and propagated via exit. | ||
| # | ||
| # Ray ALSO daemonizes gcs_server/raylet into their own session, so they escape the | ||
| # process-group reap; if left running they keep the launcher's stdout open and the | ||
| # run hangs in post-eval. So after the group reap we additionally stop Ray and kill | ||
| # its daemons by name (safe: matches only Ray daemons, not the launcher's python). | ||
| # Remove once Gym's shutdown reaps server process groups + Ray. | ||
| run: |- | ||
| ng_prepare_benchmark {{config.params.extra.nemo_gym.data_prep_params}} {{config.params.extra.nemo_gym.common_params}} | ||
| # The rollout command is written to a script and run from it, rather than passed | ||
| # to `bash -c '...'`. Gym params legitimately CONTAIN single quotes — comparison | ||
| # mode's ++multistage.stages='[{num_tasks: 45}, ...]' and ++...judge_panel='[{...}]' | ||
| # — and those would close a single-quoted `bash -c` wrapper early, so Hydra then | ||
| # receives the value split on spaces and dies with | ||
| # "no viable alternative at input '[{num_tasks:'". | ||
| # The heredoc delimiter is QUOTED ('GYM_RUN_EOF') so nothing expands while writing: | ||
| # $$ and $TAVILY_API_KEY / $INFERENCE_API_KEY land in the script literally and are | ||
| # expanded at run time by the shell that executes it, which is what we want. | ||
| cat > /tmp/gym_run.sh <<'GYM_RUN_EOF' | ||
| echo $$ > /tmp/gym_eval_pgid | ||
| exec ng_e2e_collect_rollouts {{config.params.extra.nemo_gym.collect_rollout_params}} {{config.params.extra.nemo_gym.common_params}} | ||
| GYM_RUN_EOF | ||
| setsid --wait bash /tmp/gym_run.sh & | ||
| __ev=$! | ||
| __rc=0; wait "$__ev" || __rc=$? | ||
| echo "Evaluator Gym finished!" | ||
| __pg=$(cat /tmp/gym_eval_pgid 2>/dev/null || echo) | ||
| [ -n "$__pg" ] && kill -9 -"$__pg" 2>/dev/null || true | ||
| timeout 60 ray stop --force >/dev/null 2>&1 || true | ||
| pkill -9 -f raylet >/dev/null 2>&1 || true | ||
| pkill -9 -f gcs_server >/dev/null 2>&1 || true | ||
| pkill -9 -f plasma_store >/dev/null 2>&1 || true | ||
| exit $__rc | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: NVIDIA/Model-Optimizer
Length of output: 10198
🌐 Web query:
Apptainer 1.4.2 build existing output file --force overwrite behavior documentation source💡 Result:
In Apptainer 1.4.2, the --force (or -F) flag is used with the build command to overwrite an existing container image file if it already exists [1]. When you provide the --force option, Apptainer will automatically delete and overwrite the existing image file without displaying the standard interactive confirmation prompt that would otherwise appear [2][3]. Note that this behavior differs from the --update flag, which is specifically used when building into existing sandbox containers; the --update option allows you to skip the definition file's header and add or modify sections within that existing sandbox container rather than overwriting the entire image [4][5].
Citations:
🌐 Web query:
SingularityCE build existing destination file --force overwrite behavior documentation source💡 Result:
In SingularityCE, the --force (or -F) flag is used with the singularity build command to overwrite an existing container image file if it already exists [1][2]. When this flag is provided, SingularityCE will delete and replace the existing image without prompting the user for interactive confirmation [3][4]. If you attempt to build an image to a destination where a file already exists without using the --force flag, the command will typically fail or prompt for confirmation, depending on the context and version. For sandbox (directory-based) containers, there is also an --update option [5][6]. While --force is used for overwriting files entirely, --update allows you to build into an existing sandbox container by skipping the definition file's header and only building the sections specified in the definition file [5][6]. Note that --update is only valid for sandbox containers [5][6].
Citations:
🌐 Web query:
site:github.com/apptainer/apptainer build failed cleanup output image partial file build.go --force💡 Result:
There is no native Apptainer or Singularity build command flag
--forcethat is documented to specifically "cleanup output image" or resolve "partial file" errors in build.go [1][2][3]. Errors related to failed cleanups or partial files during an Apptainer/Singularity build are typically environmental or related to how the build process interacts with the host filesystem. If you are experiencing a build failure, consider the following standard troubleshooting steps: 1. Use a local filesystem for temporary directories: Ensure your APPTAINER_TMPDIR (or TMPDIR) is not set to an NFS-mounted location, as this frequently causes stale file handles and cleanup failures [4][5]. 2. Check directory permissions and flags: Ensure the temporary directory is fully writable and does not have restrictive sticky bits or group permission settings that might interfere with container build processes [5]. 3. Debug the failure: Use the --debug flag with your build command (e.g., apptainer --debug build...) to identify the specific stage where the cleanup or file operation fails [6][7][8]. 4. Consider clearing the build cache: If you suspect a corrupted cache, you can often clear it or disable it temporarily using the --disable-cache flag during your build [7]. If you encountered a specific error message involving "cleanup output image" or "partial file" in your logs, it may be a known issue linked to specific versions of Apptainer or the underlying Go container libraries [6][8][9]. Searching the Apptainer GitHub repository for the specific error string from your debug output is the most reliable way to find patches or workarounds relevant to your version [2][6][9].Citations:
🌐 Web query:
site:github.com/sylabs/singularity source build failed cleanup destination file force💡 Result:
The "source build failed" error related to cleanup and destination files typically arises in Singularity (now Apptainer) builds when the tool detects an existing file or directory at the target path and requires explicit permission to overwrite it [1][2]. The software prevents accidental data loss by checking if the destination already exists [1]. If it is a non-empty directory or an existing file, the build will fail or pause to request confirmation unless the --force (or -F) flag is used [1][3]. Key points regarding this behavior: 1. Flag usage: If you encounter an error stating that the target exists or cannot be overwritten, add the --force flag to your command (e.g., singularity build --force [options] [destination] [definition]) [1][3]. This tells the build process to proceed with deleting or overwriting the existing destination [1]. 2. Interactive prompts: In non-automated environments, the tool may prompt you to confirm the deletion of an existing build target [1]. In automated/non-interactive environments (where stdin is not a terminal), the build will fail immediately with a message instructing you to use --force if you intend to overwrite the target [1]. 3. Sandbox permission issues: If the build fails because it cannot clean up a previous sandbox (often due to root ownership), ensure you have the appropriate permissions (e.g., using sudo or --fakeroot) [4]. Simply using --force may not resolve errors caused by lack of filesystem-level write permissions [4]. 4. Build cleanup: Apptainer/Singularity is designed to clean up temporary build bundles automatically upon completion or failure [5][6]. If an error occurs during the build, the internal cleanup routine attempts to remove temporary root filesystems and directories unless the --no-cleanup flag was used [5][6]. If you are seeing a specific error like failed to create symlink or permission denied, this often indicates the build engine is struggling to modify an existing directory structure that was likely created during a previous run with different permissions [4][7]. In such cases, manually removing the destination directory with appropriate permissions (e.g., sudo rm -rf ) before re-running the build is the most reliable solution [4].
Citations:
🏁 Script executed:
Repository: NVIDIA/Model-Optimizer
Length of output: 683
🏁 Script executed:
Repository: NVIDIA/Model-Optimizer
Length of output: 20119
🏁 Script executed:
Repository: NVIDIA/Model-Optimizer
Length of output: 353
🏁 Script executed:
Repository: NVIDIA/Model-Optimizer
Length of output: 50379
🏁 Script executed:
Repository: NVIDIA/Model-Optimizer
Length of output: 9327
Pass
--forceto both build attemptsIf the
--fakerootbuild leaves$tmp, Apptainer 1.4.2 and SingularityCE reject the fallback in non-interactive mode or prompt in interactive mode. Pass--forceto both commands.🤖 Prompt for AI Agents