XRootD MultiUser seems like failing

[shakertanim]

I have Xrootd server installed with multiuser plugin installed.

Seems like multiuser is denying system user and showing the user does not exists.

XRootD Log :

251022 16:30:50 210548 cryptossl_X509::CertType: certificate has 2 extensions
251022 16:30:50 210548 cryptossl_X509::CertType: Found RFC 382{0,1}compliant proxyCertInfo extension
251022 16:30:50 210548 cryptossl_X509::CertType: certificate has 10 extensions
251022 16:30:50 210548 XrootdBridge: almalinu.1:33@nat-243-cc-130-179-243-136.umanitoba.ca login as almalinux
251022 16:30:50 210548 multiuser_UserSentry: Multiuser denying access: XRootD mapped request to username that does not exist: almalinux
251022 16:30:50 210548 multiuser_UserSentry: Multiuser denying access: XRootD mapped request to username that does not exist: almalinux
251022 16:30:50 210548 ofs_stat: almalinu.1:33@nat-243-cc-130-179-243-136.umanitoba.ca Unable to locate /rucio/test/53/47/xrootd-new4; permission denied
251022 16:30:50 210548 XrootdXeq: almalinu.1:33@nat-243-cc-130-179-243-136.umanitoba.ca disc 0:00:01 (send failure)
251022 16:30:50 210548 multiuser_UserSentry: Anonymous client; no user set, cannot change FS UIDs
251022 16:30:50 210565 cryptossl_X509::CertType: certificate has 2 extensions
251022 16:30:50 210565 cryptossl_X509::CertType: Found RFC 382{0,1}compliant proxyCertInfo extension
251022 16:30:50 210565 cryptossl_X509::CertType: certificate has 10 extensions
251022 16:30:50 210565 XrootdBridge: almalinu.2:36@nat-243-cc-130-179-243-136.umanitoba.ca login as almalinux
251022 16:30:50 210565 multiuser_UserSentry: Multiuser denying access: XRootD mapped request to username that does not exist: almalinux
251022 16:30:50 210565 multiuser_UserSentry: Multiuser denying access: XRootD mapped request to username that does not exist: almalinux
251022 16:30:50 210565 ofs_stat: almalinu.2:36@nat-243-cc-130-179-243-136.umanitoba.ca Unable to locate /rucio/test/53/47/xrootd-new4; permission denied
251022 16:30:50 210565 XrootdXeq: almalinu.2:36@nat-243-cc-130-179-243-136.umanitoba.ca disc 0:00:00 (send failure)
251022 16:30:50 210565 multiuser_UserSentry: Anonymous client; no user set, cannot change FS UIDs

My Mapfile :

[root@grigory-alma9 etc]# cat /etc/grid-security/grid-mapfile
"/C=CA/O=Grid/CN=rucio-test.hpc.umanitoba.ca" almalinux

Authfile :

[root@grigory-alma9 etc]# cat xrootd/Authfile

Allow full access to /rucio for the mapped xrootd user
u anon.* /rucio a
u almalinux /rucio a

I am not sure why multiuser plugin is denying local user.

[almalinux@grigory-alma9 ~]$ id -u
1000

How is the UID resolved? Is it using the traditional /etc/password setup or something more complex (e.g., sssd)?

Its just simple /etc/passwd user.
Is xrootd running in a container that may not be able to resolve the UID?

Xrootd running as service not container.

Service name is : xrootd-privileged@.service

[root@grigory-alma9 build]# cat /usr/lib/systemd/system/xrootd-privileged@.service
[Unit]
Description=XRootD xrootd daemon instance %I
Documentation=man:xrootd(8)
Documentation=https://xrootd.org/docs.html
Requires=network-online.target
After=network-online.target

[Service]

Note "-R xrootd" here instructs xrootd to drop privileges to the xrootd Unix user.
ExecStart=/usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-%i.cfg -k fifo -s /var/run/xrootd/xrootd-%i.pid -n %i -R xrootd
Environment=XC_ENABLE_MULTIUSER=1
Type=simple
Restart=on-abort
RestartSec=0
KillMode=control-group
LimitNOFILE=65536
WorkingDirectory=/var/spool/xrootd

These provide xrootd with the ability to switch UIDs/GIDs for reading/writing files.
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_DAC_OVERRIDE
Capabilities=CAP_SETGID+p CAP_SETUID+p

[Install]
RequiredBy=multi-user.target
[root@grigory-alma9 build]#

Please help on the issue.

Best Regards
Kamrul Shaker

[djw8605]

Interesting, the relevant code is here:

xrootd-multiuser/src/UserSentry.hh

Line 151 in c9df3ec

retval = getpwnam_r(username.c_str(), &pwd, &buf[0], buflen, &result);

I'm not sure why it's failing. Is it possible that selinux is enforcing and intercepting the username lookups?

Is xrootd chrooting?

$ grep -i chroot /etc/xrootd/xrootd-*.cfg

Or could you check that the user is able to get the user info?

# What user is the daemon running as
ps -o user,pid,cmd -C xrootd

# Try the lookup as the daemon user (likely xrootd due to -R xrootd)
sudo -u xrootd getent passwd almalinux || echo "not found"

# Sanity check file perms
ls -l /etc/passwd /etc/nsswitch.conf
# Expected: /etc/passwd -> -rw-r--r-- root:root (0644)
#           /etc/nsswitch.conf -> -rw-r--r-- root:root (0644)

[shakertanim]

Please find the output :

[root@grigory-alma9 ~]# grep -i chroot /etc/xrootd/xrootd-.cfg
[root@grigory-alma9 ~]# grep -i -r chroot /etc/xrootd/
[root@grigory-alma9 ~]# grep -i -r chroot /etc/xrootd/
[root@grigory-alma9 ~]# su - xrootd
Last login: Thu Oct 23 17:51:51 UTC 2025 on pts/0
[xrootd@grigory-alma9 ~]$ # What user is the daemon running as
[xrootd@grigory-alma9 ~]$ ps -o user,pid,cmd -C xrootd
USER PID CMD
setroub+ 3699 xrootd -R xrootd -n rucio -c /etc/xrootd/xrdrucio.cfg
xrootd 35755 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-standalone.cfg -k fifo -s /var/run/xrootd/xrootd-standalone.pid -n standalone -R xrootd
[xrootd@grigory-alma9 ~]$
[xrootd@grigory-alma9 ~]$ # Try the lookup as the daemon user (likely xrootd due to -R xrootd)
[xrootd@grigory-alma9 ~]$ sudo -u xrootd getent passwd almalinux || echo "not found"
almalinux:x:1000:1000:almalinux Cloud User:/home/almalinux:/bin/bash
[xrootd@grigory-alma9 ~]$
[xrootd@grigory-alma9 ~]$ # Sanity check file perms
[xrootd@grigory-alma9 ~]$ ls -l /etc/passwd /etc/nsswitch.conf
-rw-r--r--. 1 root root 2120 May 30 2024 /etc/nsswitch.conf
-rw-r--r-- 1 root root 1908 Oct 22 17:15 /etc/passwd
[xrootd@grigory-alma9 ~]$ # Expected: /etc/passwd -> -rw-r--r-- root:root (0644)
[xrootd@grigory-alma9 ~]$ # /etc/nsswitch.conf -> -rw-r--r-- root:root (0644)
[xrootd@grigory-alma9 ~]$

FYI, Selinux is disabled.

[djw8605]

Interesting, so you have xrootd instances running? From the output of your ps, one is running as user setroub+ (very likely setroubleshoot):

setroub+ 3699 xrootd -R xrootd -n rucio -c /etc/xrootd/xrdrucio.cfg

How is that instance running? It certainly isn't running as privledged user from the systemd service. Could that instance be the one that is outputting the errors?

Try some of the commands with that user?

sudo -u setroubleshoot getent passwd almalinux || echo "not found"

But I think the key finding is you have 2 xrootd isntances running, and I suspect it's the rucio instance that is not running privileged and is failing the user lookup.

[shakertanim]

Ohhh ignore this process.

It was running in docker. I stopped it now.

Please find the new set below :

[root@grigory-alma9 xrootd]# ps -o user,pid,cmd -C xrootd
USER PID CMD
xrootd 35755 /usr/bin/xrootd -l /var/log/xrootd/xrootd.log -c /etc/xrootd/xrootd-standalone.cfg -k fifo -s /var/run/xrootd/xrootd-standalone.pid -n standalone -R xrootd

[djw8605]

And just to confirm, with that separate instance gone, you are still seeing the multiuser error?

[shakertanim]

Yes multiuser error is still there.

Just to check, if its possible to share grid-mapfile and Authfile template for multiuser case?

I need to check, if I am missing something or not.

[shakertanim]

@djw8605 Any update on this from your side?

Is it a bug, which is stopping me to use the service?

I am using both Almalinux 8 and Almalinux 9 for this. I am getting the same error for both of the OS version.

Any specific information should i share so you can debug the issue?

[bbockelm]

I'm really stumped on this, particularly as everything looks fairly simple / out-of-the-box. @shakertanim / @djw8605 - could you guys get on a Zoom call for a live debugging session perhaps?

My suggested next steps would be a bit more complex, including:

• Use strace -s 256 -f -p $PID to look at the system calls that occur around the attempted access and see if there's a clear denial going on.
• Install debug symbols for xrootd and set a breakpoint around getpwnam_r (I think that's relatively unused so there shouldn't be too much noise) to see if there's anything "surprising" being fed from the plugin to libc.

@shakertanim if you're not comfortable doing that, I think it's something Derek could guide over Zoom.

The real head-scratcher here is that, based on the code, libc is stating that there is no lookup errors and, definitively, the user doesn't exist. If it was a bizarre permissions problem, I would have expected an error.

Oh! One last thing - what version of the xrootd-multiuser RPM is being used?

[shakertanim]

xrootd-multiuser-2.2.0-1.2.osg25.el8.x86_64

I am using this version.

[shakertanim]

@bbockelm Is this the correct version for the multiuser plugin? xrootd-multiuser-2.2.0-1.2.osg25.el8.x86_64