[Bug]: nvidia-fs:2.27.3-rhel9.6 image ships only source code, no nvidia-gds-driver binary - nvidia-fs-ctr fails with exit 127

[friedrice99]

Describe the bug
The published nvcr.io/nvidia/cloud-native/nvidia-fs:2.27.3-rhel9.6 image contains only a raw git checkout of the gds-nvidia-fs source code and does not contain the nvidia-gds-driver
orchestrator script that the GPU Operator's daemonset expects.

The Ubuntu variants (2.27.3-ubuntu22.04) ship the binary at /usr/local/bin/nvidia-gds-driver (6485 bytes, executable). The RHEL variants do not.

This causes nvidia-fs-ctr in the GPU Operator's nvidia-driver-daemonset to fail in CrashLoopBackOff:

The PATH workaround documented in #1849 does not apply because the binary is not present anywhere in the RHEL image not just outside $PATH.

To Reproduce

1. Enable GDS in GPU Operator on a RHEL 9.x host with gds.version: 2.27.3-rhel9.6 — nvidia-fs-ctr enters CrashLoopBackOff with exit 127.

2. Ubuntu variant — has the orchestrator binary:

kubectl run check-ubuntu --rm -it \
   --image=nvcr.io/nvidia/cloud-native/nvidia-fs:2.27.3-ubuntu22.04 \
   --restart=Never --command -- which nvidia-gds-driver

3. RHEL variant — only source code, no binary:

kubectl run check-rhel --rm -it \
 --image=nvcr.io/nvidia/cloud-native/nvidia-fs:2.27.3-rhel9.6 \
 --restart=Never --command -- bash -c 'find / -name "nvidia-gds-driver*" 2>/dev/null; echo ===; ls /usr/local/gds-nvidia-fs/'

Expected behavior
nvcr.io/nvidia/cloud-native/nvidia-fs:2.27.3-rhel9.6 should contain a working orchestrator at the location the GPU Operator daemonset invokes (exec nvidia-gds-driver install), matching what the ubuntu22.04
variant ships.

Environment (please provide the following information):

• GPU Operator Version: v26.3.0
• Host OS: RHEL 9.7 (kernel 5.14.0-611.47.1.el9_7.x86_64)
• Container Runtime: cri-o 1.32
• Driver: 580.126.09 (open kernel module)
• GPU: NVIDIA A100 80GB PCIe

Workaround currently in use

Switched gds.version to 2.27.3-ubuntu22.04. The Ubuntu image's nvidia-gds-driver script does apt-get install linux-headers-${KERNEL_VERSION}, which fails on RHEL hosts because 5.14.0-611.47.1.el9_7.x86_64 is
not an Ubuntu apt package.

Patched the script to symlink kernel headers from the driver container's mount instead:

RUN sed -i '/apt-get -qq install --no-install-recommends linux-headers/c\
      mkdir -p /lib/modules/${KERNEL_VERSION} \&\& ln -sf /run/nvidia/driver/usr/src/kernels/${KERNEL_VERSION} /lib/modules/${KERNEL_VERSION}/build' \
      /usr/local/bin/nvidia-gds-driver

After this patch, make builds nvidia_fs.ko against the host RHEL 9.7 kernel headers (mounted from driver container) and insmod succeeds. gdscheck -p reports Platform verification succeeded. Module loads cleanly.

Cross-reference
#1849 — related but different scenario (Ubuntu host + Ubuntu nvidia-fs PATH workaround for gcc-12).

[rahulait]

Hi @friedrice99, can you please provide more details about your cluster? It looks like your host here is rhel9.7 and the pulled nvidia-fs image 9.6. We haven't released image for rhel9.7 yet. Hence wondering how it automatically picked 9.6 for your setup. We build and install on the systems and ubuntu and rhel uses different scripts. What you are trying is reasonable thing to do, but we don't guarantee it will always work. We would like to get logs for the nvidia-fs container run, your setup details so that we can understand whats going on there and debug further.

You can attach must-gather output or email it to us at operator_feedback@nvidia.com, it will contain all the logs and info we need.

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh

[friedrice99]

Hi @rahulait, thanks for the response.

On the 9.6 image for rhel9.7: The image was not auto-selected since no rhel9.7 image exists yet, I explicitly pinned gds.version: 2.27.3-rhel9.6.

On different scripts for RHEL vs Ubuntu: Understood. Could you clarify what the expected entrypoint/orchestrator for the RHEL image is? The GPU Operator daemonset calls exec nvidia-gds-driver install, but that binary is not present anywhere in the RHEL image. If RHEL uses a different script name or path, that would explain the mismatch but then the GPU Operator itself may need to be aware of it.

On logs: Happy to share must-gather output. That said, the kubectl run commands in the description already confirm the binary is absent from the RHEL image so if logs would still help narrow things down on your end, I'll attach them.

[rahulait]

@friedrice99 you can inspect the image and find different entrypoints for them. For example:

root@dev1:~# docker inspect --format='{{.Config.Entrypoint}}' nvcr.io/nvidia/cloud-native/nvidia-fs:2.27.3-ubuntu24.04
[nvidia-gds-driver install]
root@dev1:~# docker inspect --format='{{.Config.Entrypoint}}' nvcr.io/nvidia/cloud-native/nvidia-fs:2.27.3-rhel9.6
[ocp_dtk_entrypoint nv-fs-ctr-run-with-dtk]
root@dev1:~#

Within the rhel image, you can find the script using:

root@dev1:~# docker run -it --entrypoint /bin/bash nvcr.io/nvidia/cloud-native/nvidia-fs:2.27.3-rhel9.6 -c 'find . -name ocp_dtk_entrypoint'
./usr/local/bin/ocp_dtk_entrypoint
root@dev1:~#

[friedrice99]

Thanks @rahulait , that clarifies it. RHEL image being OCP/DTK-specific explains the source-only layout and different entrypoint

[rahulait]

I am going to close this issue for now. If you have further questions, please feel free to re-open or open a separate ticket so that it can be addressed. Thanks.

[jele84]

Hi everyone! I encountered the same problem in my k8s cluster.
My question is, why does gpu-operator specify the startup argument for gdrdrv:v2.5.2-rhel9.6 and nvidia-fs:2.27.3-rhel9.6
until [ -d /run/nvidia/driver/usr/src ] && lsmod | grep nvidia; do echo Waiting for nvidia-driver to be installed...; sleep 10; done; exec nvidia-gdrcopy-driver install, if Config.Entrypoint should be [ocp_dtk_entrypoint gdrcopy-run-with-dtk]?

• GPU Operator Version: v25.10.1
• Host OS: RHEL 9.7 (kernel 5.14.0-611.30.1.el9_7^1.x86_64)
• Container Runtime: cri-o 1.28.11
• Driver: 580.126.20-rhel9.7 (auto kernel module)