Page MenuHomeVyOS Platform
Create Task
Maniphest T7234

Local User Commits Get Stuck When TACACS Servers Are Unavailable
Open, Urgent!PublicBUG
Actions

Description

VyOS configuration:

set interfaces bonding bond0 address '10.55.8.91/24'
set interfaces bonding bond0 description 'vyos-shared-if'
set interfaces bonding bond0 member interface 'eth0'
set interfaces bonding bond0 member interface 'eth2'
set interfaces bonding bond0 mode '802.3ad'
set interfaces bonding bond0 mtu '9000'
set protocols static route 0.0.0.0/0 next-hop 10.55.8.254
set service ssh port '22'
set system host-name 'vyos-test'

set system login tacacs server 192.168.1.50 key 'test123'
set system login tacacs server 192.168.1.50 port '49'
set system login tacacs server 192.168.1.51 key 'test123'
set system login tacacs server 192.168.1.51 port '49'
set system login tacacs source-address '10.55.8.91'
set system login tacacs timeout '10'

In this scenario, both TACACS servers are unavailable. I accessed the VyOS instance via SSH with my local user account (vyos).
Trying to delete system login via TACACS and commit gets stuck:

vyos@vyos-test#
[edit]
vyos@vyos-test# delete system login tacacs
[edit]
vyos@vyos-test# commit

Journal logs:

vyos@vyos-test# sudo journalctl -f
Mar 11 02:34:07 vyos-test sshd[2543]: pam_tacplus(sshd:auth): connection to srv[0] 192.168.1.50:49 failed: Operation now in progress
Mar 11 02:34:17 vyos-test sshd[2543]: pam_tacplus(sshd:auth): connection to srv[1] 192.168.1.51:49 failed: Operation now in progress
Mar 11 02:34:25 vyos-test sshd[2543]: Accepted password for vyos from 10.55.8.1 port 52196 ssh2
Mar 11 02:34:25 vyos-test sshd[2543]: pam_unix(sshd:session): session opened for user vyos(uid=1002) by (uid=0)
Mar 11 02:34:25 vyos-test systemd-logind[842]: New session 4 of user vyos.
Mar 11 02:34:25 vyos-test systemd[1]: Started session-4.scope - Session 4 of User vyos.
Mar 11 02:34:26 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_2881.mount: Deactivated successfully.
Mar 11 02:34:26 vyos-test sshd[2543]: pam_env(sshd:session): deprecated reading of user environment enabled
Mar 11 02:37:20 vyos-test sudo[3243]:     vyos : TTY=ttyS0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/journalctl -f
Mar 11 02:37:20 vyos-test sudo[3243]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Mar 11 02:37:42 vyos-test sudo[3288]:     vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/system_login.py'
Mar 11 02:37:42 vyos-test sudo[3288]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Mar 11 02:37:42 vyos-test vyos-configd[692]: Received message: {"type": "init"}
Mar 11 02:37:42 vyos-test vyos-configd[692]: config session pid is 3164
Mar 11 02:37:42 vyos-test vyos-configd[692]: commit_scripts: ['system_login']
Mar 11 02:37:42 vyos-test vyos-configd[692]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/system_login.py"}
Mar 11 02:37:42 vyos-test vyos-configd[692]: Sending response 8
Mar 11 02:37:42 vyos-test vyos-configd[692]: scripts_called: ['system_login']
Mar 11 02:40:01 vyos-test CRON[3304]: nss_tacplus: Configuration file(s) have changed, re-initializing

Terminating process via Ctrl+C
Trying another command and commit gets stuck as well:

vyos@vyos-test#
[edit]
vyos@vyos-test# delete system login tacacs
[edit]
vyos@vyos-test# commit

[edit]
vyos@vyos-test# set system host-name R-01
[edit]
vyos@vyos-test# commit

Journal logs:

vyos@vyos-test# sudo journalctl -f
Mar 11 02:34:07 vyos-test sshd[2543]: pam_tacplus(sshd:auth): connection to srv[0] 192.168.1.50:49 failed: Operation now in progress
Mar 11 02:34:17 vyos-test sshd[2543]: pam_tacplus(sshd:auth): connection to srv[1] 192.168.1.51:49 failed: Operation now in progress
Mar 11 02:34:25 vyos-test sshd[2543]: Accepted password for vyos from 10.55.8.1 port 52196 ssh2
Mar 11 02:34:25 vyos-test sshd[2543]: pam_unix(sshd:session): session opened for user vyos(uid=1002) by (uid=0)
Mar 11 02:34:25 vyos-test systemd-logind[842]: New session 4 of user vyos.
Mar 11 02:34:25 vyos-test systemd[1]: Started session-4.scope - Session 4 of User vyos.
Mar 11 02:34:26 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_2881.mount: Deactivated successfully.
Mar 11 02:34:26 vyos-test sshd[2543]: pam_env(sshd:session): deprecated reading of user environment enabled
Mar 11 02:37:20 vyos-test sudo[3243]:     vyos : TTY=ttyS0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/journalctl -f
Mar 11 02:37:20 vyos-test sudo[3243]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Mar 11 02:37:42 vyos-test sudo[3288]:     vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/system_login.py'
Mar 11 02:37:42 vyos-test sudo[3288]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Mar 11 02:37:42 vyos-test vyos-configd[692]: Received message: {"type": "init"}
Mar 11 02:37:42 vyos-test vyos-configd[692]: config session pid is 3164
Mar 11 02:37:42 vyos-test vyos-configd[692]: commit_scripts: ['system_login']
Mar 11 02:37:42 vyos-test vyos-configd[692]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/system_login.py"}
Mar 11 02:37:42 vyos-test vyos-configd[692]: Sending response 8
Mar 11 02:37:42 vyos-test vyos-configd[692]: scripts_called: ['system_login']
Mar 11 02:40:01 vyos-test CRON[3304]: nss_tacplus: Configuration file(s) have changed, re-initializing
Mar 11 02:42:21 vyos-test CRON[3304]: pam_unix(cron:session): session opened for user smmsp(uid=116) by (uid=0)
Mar 11 02:42:42 vyos-test CRON[3307]: (smmsp) CMD (test -x /etc/init.d/sendmail && test -x /usr/share/sendmail/sendmail && test -x /usr/libexec/sendmail/sendmail && /usr/share/sendmail/sendmail cron-msp)
Mar 11 02:43:22 vyos-test CRON[3304]: pam_unix(cron:session): session closed for user smmsp
Mar 11 02:45:48 vyos-test python3[3295]: tac_connect_single: connection failed with 192.168.1.50:49: Interrupted system call
Mar 11 02:45:48 vyos-test sudo[3288]: pam_unix(sudo:session): session closed for user root
Mar 11 02:45:59 vyos-test python3[3295]: Exception ignored in:
Mar 11 02:45:59 vyos-test python3[3295]: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
Mar 11 02:45:59 vyos-test python3[3295]: Exception ignored in sys.unraisablehook
Mar 11 02:45:59 vyos-test [3295]:
Mar 11 02:45:59 vyos-test python3[3295]: <built-in function unraisablehook>
Mar 11 02:45:59 vyos-test systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Mar 11 02:46:16 vyos-test sudo[3351]:     vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/system_host-name.py'
Mar 11 02:46:16 vyos-test sudo[3351]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Mar 11 02:46:16 vyos-test vyos-configd[692]: Received message: {"type": "init"}
Mar 11 02:46:16 vyos-test vyos-configd[692]: config session pid is 3164
Mar 11 02:46:16 vyos-test vyos-configd[692]: commit_scripts: ['system_host-name', 'system_login']
Mar 11 02:46:16 vyos-test vyos-configd[692]: Received message: {"type": "node", "last": false, "data": "/usr/libexec/vyos/conf_mode/system_host-name.py"}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"type": "host_name", "op": "set", "data": {"host_name": "R-01", "domain_name": ""}}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': None}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"type": "search_domains", "op": "delete", "data": ["system"]}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': None}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"type": "name_servers", "op": "delete", "data": ["system"]}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': None}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"type": "name_server_tags_system", "op": "get"}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': ['system']}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"type": "name_server_tags_system", "op": "delete", "data": ["system"]}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': None}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"type": "name_server_tags_system", "op": "add", "data": ["system"]}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': None}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"type": "hosts", "op": "delete", "data": ["system"]}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': None}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Request data: {"op": "apply"}
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Applying 6 changes
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Writing /etc/resolv.conf
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Writing /etc/hosts
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Writing /run/pdns-recursor/recursor.vyos-hostsd.conf.lua
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Writing /run/pdns-recursor/recursor.forward-zones.conf
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: pdns_recursor not running, not sending "reload-lua-config"
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: pdns_recursor not running, not sending "reload-zones"
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Success
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Mar 11 02:46:16 vyos-test vyos-hostsd[693]: Sent response: {'data': {'message': 'Applied 6 changes'}}
Mar 11 02:46:16 vyos-test dbus-daemon[836]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.13' (uid=0 pid=3360 comm="hostnamectl --static")
Mar 11 02:46:16 vyos-test systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Mar 11 02:46:16 vyos-test dbus-daemon[836]: [system] Successfully activated service 'org.freedesktop.hostname1'
Mar 11 02:46:16 vyos-test systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Mar 11 02:46:16 R-01 systemd-hostnamed[3361]: Hostname set to <R-01> (static)
Mar 11 02:46:16 R-01 systemd[1]: Stopping rsyslog.service - System Logging Service...
Mar 11 02:46:16 R-01 rsyslogd[2288]: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="2288" x-info="https://www.rsyslog.com"] exiting on signal 15.
Mar 11 02:46:16 R-01 systemd[1]: rsyslog.service: Deactivated successfully.
Mar 11 02:46:16 R-01 systemd[1]: Stopped rsyslog.service - System Logging Service.
Mar 11 02:46:16 R-01 systemd[1]: Starting rsyslog.service - System Logging Service...
Mar 11 02:46:16 R-01 rsyslogd[3364]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2302.0]
Mar 11 02:46:16 R-01 rsyslogd[3364]: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="3364" x-info="https://www.rsyslog.com"] start
Mar 11 02:46:16 R-01 systemd[1]: Started rsyslog.service - System Logging Service.
Mar 11 02:46:16 R-01 vyos-configd[692]: Sending response 1
Mar 11 02:46:16 R-01 sudo[3351]: pam_unix(sudo:session): session closed for user root
Mar 11 02:46:16 R-01 sudo[3369]:     vyos : TTY=pts/0 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/system_login.py'
Mar 11 02:46:16 R-01 sudo[3369]: pam_unix(sudo:session): session opened for user root(uid=0) by vyos(uid=1002)
Mar 11 02:46:16 R-01 vyos-configd[692]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/system_login.py"}
Mar 11 02:46:16 R-01 vyos-configd[692]: Sending response 8
Mar 11 02:46:16 R-01 vyos-configd[692]: scripts_called: ['system_host-name', 'system_login']
Mar 11 02:46:46 R-01 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Mar 11 02:49:19 R-01 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Mar 11 02:49:19 R-01 systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Mar 11 02:49:19 R-01 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.

Details

Version
1.4
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

a.hajiyev created this task.Tue, Mar 11, 2:52 AM
Viacheslav triaged this task as Urgent! priority.Tue, Mar 11, 7:13 AM
a.hajiyev renamed this task from Local User Commits Get Stuck When Takacs Servers Are Unavailable to Local User Commits Get Stuck When TACACS Servers Are Unavailable.Tue, Mar 11, 7:14 AM
pasik subscribed.Tue, Mar 11, 7:26 AM
a.hajiyev removed a project: VyOS 1.5 Circinus.Wed, Mar 12, 1:53 AM
a.hajiyev added a comment.EditedWed, Mar 12, 2:01 AM

I tested the same scenario in VyOS 2025.03.09-0613-rolling and did not observe the same issue.
VyOS works as expected and commit did not get stuck.

VyOS configuration:

set interfaces bonding bond0 address '10.55.8.93/24'
set interfaces bonding bond0 description 'vyos-shared-if'
set interfaces bonding bond0 member interface 'eth0'
set interfaces bonding bond0 member interface 'eth2'
set interfaces bonding bond0 mode '802.3ad'
set interfaces bonding bond0 mtu '9000'
set protocols static route 0.0.0.0/0 next-hop 10.55.8.254
set service ssh port '22'
set system host-name 'vyos-test'
set system login tacacs server 192.168.1.50 key 'test123'
set system login tacacs server 192.168.1.50 port '49'
set system login tacacs server 192.168.1.51 key 'test123'
set system login tacacs server 192.168.1.51 port '49'
set system login tacacs source-address '10.55.8.91'
set system login tacacs timeout '10'

Both TACACS servers are unavailable. I accessed the VyOS instance via SSH with my local user account (vyos).
Trying to delete system login via TACACS and commit did not get stuck:

vyos@vyos-test# delete system login tacacs
[edit]
vyos@vyos-test# commit
[edit]
vyos@vyos-test# set interfaces bonding bond0 description LAN-02
[edit]
vyos@vyos-test# commit
[edit]

Journal logs:

vyos@vyos-test# sudo journalctl -f
Mar 12 01:50:28 vyos-test vyos-configd[694]: scripts_called: ['system_login']
Mar 12 01:50:28 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_4070.mount: Deactivated successfully.
Mar 12 01:51:06 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_4070.mount: Deactivated successfully.
Mar 12 01:51:09 vyos-test systemd[1]: session-3.scope: Deactivated successfully.
Mar 12 01:51:09 vyos-test systemd[1]: session-3.scope: Consumed 15.385s CPU time.
Mar 12 01:51:09 vyos-test systemd-logind[1127]: Session 3 logged out. Waiting for processes to exit.
Mar 12 01:51:09 vyos-test systemd[1]: session-4.scope: Deactivated successfully.
Mar 12 01:51:09 vyos-test systemd-logind[1127]: Session 4 logged out. Waiting for processes to exit.
Mar 12 01:51:09 vyos-test systemd-logind[1127]: Removed session 3.
Mar 12 01:51:09 vyos-test systemd-logind[1127]: Removed session 4.
Mar 12 01:51:28 vyos-test systemd-logind[1127]: New session 8 of user vyos.
Mar 12 01:51:28 vyos-test systemd[1]: Started session-8.scope - Session 8 of User vyos.
Mar 12 01:51:29 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_5906.mount: Deactivated successfully.
Mar 12 01:51:30 vyos-test systemd-logind[1127]: New session 9 of user vyos.
Mar 12 01:51:30 vyos-test systemd[1]: Started session-9.scope - Session 9 of User vyos.
Mar 12 01:51:31 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_6075.mount: Deactivated successfully.
Mar 12 01:51:59 vyos-test vyos-configd[694]: Received message: {"type": "init"}
Mar 12 01:51:59 vyos-test vyos-configd[694]: config session pid is 6213
Mar 12 01:51:59 vyos-test vyos-configd[694]: config session sudo_user is vyos
Mar 12 01:51:59 vyos-test vyos-configd[694]: commit_scripts: ['system_login']
Mar 12 01:51:59 vyos-test vyos-configd[694]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/system_login.py"}
Mar 12 01:52:00 vyos-test vyos-configd[694]: Sending reply: SUCCESS with output
Mar 12 01:52:00 vyos-test vyos-configd[694]: scripts_called: ['system_login']
Mar 12 01:52:00 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_6213.mount: Deactivated successfully.
Mar 12 01:52:29 vyos-test vyos-configd[694]: Received message: {"type": "init"}
Mar 12 01:52:29 vyos-test vyos-configd[694]: config session pid is 6213
Mar 12 01:52:29 vyos-test vyos-configd[694]: config session sudo_user is vyos
Mar 12 01:52:29 vyos-test vyos-configd[694]: commit_scripts: ['interfaces_bonding_bond0']
Mar 12 01:52:29 vyos-test vyos-configd[694]: Received message: {"type": "node", "last": true, "data": "VYOS_TAGNODE_VALUE=bond0/usr/libexec/vyos/conf_mode/interfaces_bonding.py"}
Mar 12 01:52:30 vyos-test vyos-configd[694]: Sending reply: SUCCESS with output
Mar 12 01:52:30 vyos-test vyos-configd[694]: scripts_called: ['interfaces_bonding_bond0']
Mar 12 01:52:30 vyos-test systemd[1]: opt-vyatta-config-tmp-new_config_6213.mount: Deactivated successfully.
Mar 12 01:56:15 vyos-test systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Mar 12 01:56:15 vyos-test systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Mar 12 01:56:15 vyos-test systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Mar 12 01:56:15 vyos-test systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.