Users not found when password database is in a non-standard path

[ocaisa]

Saw this on the compute nodes of Lumi:

{EESSI/2025.06} alanocai@nid005265:~> id -un
id: cannot find name for user ID 327004947
327004947
{EESSI/2025.06} alanocai@nid005265:~> export LD_PRELOAD=/lib64/libnss_db.so.2
{EESSI/2025.06} alanocai@nid005265:~> id -un
alanocai

So looks like we need another symlink?

[boegel]

@bedroge Thoughts on this?

[ocaisa]

Do we just add another symlink under https://github.com/EESSI/compatibility-layer/blob/main/scripts/prefix-symlink-host-paths.sh#L31 ?

[ocaisa]

Adding another symlink is not really an option, while the LD_PRELOAD worked for 2025.06, it gives a glibc error for 2023.06. I wonder if we can build our own libnss_db.so.2

[boegel]

This library is available in EESSI compat layers:

/cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib64/libnss_db.so.2

and

/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/libnss_db.so.2

[bedroge]

@ocaisa So this happens with both 2023.06 and 2025.06, right?

Could you do an strace id -un and grep for var/lib/sss? I'm suspecting that we may have to change https://github.com/EESSI/gentoo-overlay/blob/main/sys-auth/sssd/sssd-2.8.2.ebuild#L152 and point it to the host location if the daemon is not built, as we do below for e.g. --with-pipe-path.

[ocaisa]

Almost nothing for /var at all:

{EESSI/2025.06} alanocai@nid005779:~> strace id -un |& grep '/var'
openat(AT_FDCWD, "/var/db/passwd.db", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

but this is a bit more interesting:

{EESSI/2025.06} alanocai@nid005779:~> strace id -un |& grep passwd
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/var/db/passwd.db", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
{EESSI/2025.06} alanocai@nid005779:~> LD_PRELOAD=/lib64/libnss_db.so.2 strace id -un |& grep passwd
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/var/lib/misc/passwd.db", O_RDONLY|O_CLOEXEC) = 3

[bedroge]

This may be relevant here: NixOS/nixpkgs#229486

[ocaisa]

So it seems like the database is just not where it is expected. Is it possible to add additional options for that?

[ocaisa]

Nope, seems not. Only option would be a symlink at their level, or a variant symlink at ours (and also patching glibc to look there?)

[bedroge]

Couldn't find such an option either, looks like it's baked into glibc. It's defined in $EPREFIX/usr/include/paths.h:

#define _PATH_VARDB     "/var/db/"

and then used here: https://github.com/bminor/glibc/blob/04e750e75b73957cf1c791535a3f4319534a52fc/nss/nss_db/db-init.c#L27.

I suppose we could indeed change that variable and point it to a variant symlink on CVMFS.

[bedroge]

That variable was initially set to $EPREFIX/var/db by the way, but we deliberately deprefixify these paths with a patch provided by @bartoldeman (https://github.com/EESSI/gentoo-overlay/blob/main/etc/portage/env/sys-libs/glibc), with the idea to use the host files for such services. But if distros decide to use non-default locations, we may have to rethink this.

[bartoldeman]

yes, although we're using sssd (which uses pipe or socket) so I haven't seen libnss_db in action yet.
https://linux.die.net/man/5/nsscache.conf says:

nssdb CACHE OPTIONS

These options configure the behaviour of the nssdb cache.

nssdb_dir
    Directory to store the Berkeley DB databases. Defaults to the current directory. Note that nss_db hardcodes the path to /var/lib/misc on Debian systems, and /var/db on Red Hat systems. 

one could go via variant symlinks, or patching libnss_db to check both locations.

[ocaisa]

So one way to do it would be to restore $EPREFIX/var/db as the search location and then make $EPREFIX/var/db/passwd.db a variant symlink which by default points to /var/db/passwd.db...and if we are doing that we should probably evaluate our other choices to see if we should prefer a similar approach for those too