Page MenuHomeVyOS Platform
Create Task
Maniphest T6671

Confid dependency works incorrectly conntrack conntrack-sync
Closed, ResolvedPublicBUG
Actions

Description

Note: further details added in first comment, below

Confid dependency works incorrectly conntrack conntrack-sync
To reproduce:

set interfaces ethernet eth0 address '192.168.122.15/24'
set interfaces ethernet eth0 address '192.168.77.15/24'
set interfaces ethernet eth2 address '198.51.100.2/24'

set interfaces bonding bond0 address '192.0.2.1/24'
set interfaces bonding bond0 member interface 'eth3'

set high-availability vrrp group ETH2 address 198.51.100.200/24
set high-availability vrrp group ETH2 hello-source-address '198.51.100.2'
set high-availability vrrp group ETH2 interface 'eth2'
set high-availability vrrp group ETH2 priority '200'
set high-availability vrrp group ETH2 vrid '22'
set high-availability vrrp sync-group GROUP member 'ETH2'


set service conntrack-sync failover-mechanism vrrp sync-group 'GROUP'
set service conntrack-sync interface bond0 peer '192.0.2.77'

Reboot the router and get vyos-configd configuration error

vyos@r15# compare 
[system]
+ conntrack {
+     modules {
+         ftp
+         h323
+         nfs
+         pptp
+         sip
+         sqlnet
+         tftp
+     }
+ }

[edit]
vyos@r15#

log:

Aug 21 11:37:21 r15 sudo[1874]: pam_unix(sudo:session): session closed for user root
Aug 21 11:37:21 r15 sudo[1942]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim /usr/libexec/vyos/conf_mode/system_conntrack.py'
Aug 21 11:37:21 r15 sudo[1942]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Aug 21 11:37:21 r15 vyos-configd[753]: Received message: {"type": "node", "last": false, "data": "/usr/libexec/vyos/conf_mode/system_conntrack.py"}
Aug 21 11:37:21 r15 vyos-configd[753]: 'Interface' object has no attribute 'iftype'
Aug 21 11:37:21 r15 vyos-configd[753]: Sending response 4
Aug 21 11:37:21 r15 python3[1953]: Report time:      2024-08-21 11:37:21
Aug 21 11:37:21 r15 python3[1953]: Image version:    VyOS 1.5-rolling-202408210022
Aug 21 11:37:21 r15 python3[1953]: Release train:    current
Aug 21 11:37:21 r15 python3[1953]: Built by:         [email protected]
Aug 21 11:37:21 r15 python3[1953]: Built on:         Wed 21 Aug 2024 00:22 UTC
Aug 21 11:37:21 r15 python3[1953]: Build UUID:       3ab1141d-9c4c-4fd3-abb5-d562241a0ac3
Aug 21 11:37:21 r15 python3[1953]: Build commit ID:  c75174bb4a2230
Aug 21 11:37:21 r15 Architecture[1953]:     x86_64
Aug 21 11:37:21 r15 python3[1953]: Boot via:         installed image
Aug 21 11:37:21 r15 python3[1953]: System type:      KVM guest
Aug 21 11:37:21 r15 python3[1953]: Hardware vendor:  QEMU
Aug 21 11:37:21 r15 python3[1953]: Hardware model:   Standard PC (Q35 + ICH9, 2009)
Aug 21 11:37:21 r15 python3[1953]: Hardware S/N:
Aug 21 11:37:21 r15 python3[1953]: Hardware UUID:    eb9e740f-6e5c-4444-a3df-f4c8dd4b6546
Aug 21 11:37:21 r15 python3[1953]: Traceback (most recent call last):
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/libexec/vyos/conf_mode/system_conntrack.py", line 270, in <module>
Aug 21 11:37:21 r15 python3[1953]:     apply(c)
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/libexec/vyos/conf_mode/system_conntrack.py", line 249, in apply
Aug 21 11:37:21 r15 python3[1953]:     call_dependents()
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/lib/python3/dist-packages/vyos/configdep.py", line 147, in call_dependents
Aug 21 11:37:21 r15 python3[1953]:     f()
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/lib/python3/dist-packages/vyos/configdep.py", line 118, in func_impl
Aug 21 11:37:21 r15 python3[1953]:     run_config_mode_script(script, config)
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/lib/python3/dist-packages/vyos/configdep.py", line 106, in run_config_mode_script
Aug 21 11:37:21 r15 python3[1953]:     mod.verify(c)
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/libexec/vyos//conf_mode/service_conntrack-sync.py", line 72, in verify
Aug 21 11:37:21 r15 python3[1953]:     if len(get_ipv4(interface)) < 1:
Aug 21 11:37:21 r15 python3[1953]:            ^^^^^^^^^^^^^^^^^^^
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/lib/python3/dist-packages/vyos/template.py", line 458, in get_ipv4
Aug 21 11:37:21 r15 python3[1953]:     return Interface(interface).get_addr_v4()
Aug 21 11:37:21 r15 python3[1953]:            ^^^^^^^^^^^^^^^^^^^^
Aug 21 11:37:21 r15 python3[1953]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 334, in __init__
Aug 21 11:37:21 r15 python3[1953]:     if not self.iftype:
Aug 21 11:37:21 r15 python3[1953]:            ^^^^^^^^^^^
Aug 21 11:37:21 r15 AttributeError[1953]: 'Interface' object has no attribute 'iftype'
Aug 21 11:37:21 r15 noteworthy[1953]: 
Aug 21 11:37:21 r15 python3[1953]: cmd 'ip link show dev bond0'
Aug 21 11:37:21 r15 python3[1953]: returned (out):
Aug 21 11:37:21 r15 python3[1953]: returned (err):
Aug 21 11:37:21 r15 python3[1953]: Device "bond0" does not exist.
Aug 21 11:37:21 r15 sudo[1942]: pam_unix(sudo:session): session closed for user root
Aug 21 11:37:21 r15 sudo[1965]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim VYOS_TAGNODE_VALUE=\'lo\' /usr/libexec/vyos/conf_mode/interfaces_loopback.py'
Aug 21 11:37:21 r15 sudo[1965]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Aug 21 11:37:21 r15 vyos-configd[753]: Received message: {"type": "node", "last": false, "data": "VYOS_TAGNODE_VALUE=lo/usr/libexec/vyos/conf_mode/interfaces_loopback.py"}
Aug 21 11:37:22 r15 vyos-configd[753]: Sending response 1
Aug 21 11:37:22 r15 sudo[1965]: pam_unix(sudo:session): session closed for user root
Aug 21 11:37:22 r15 sudo[2000]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim VYOS_TAGNODE_VALUE=\'eth2\' /usr/libexec/vyos/conf_mode/interfaces_ethernet.py'

prio:

vyos@r15# /usr/libexec/vyos/priority.py | match "bond|conntrack"
       218  system_conntrack.py                 ['system', 'conntrack']
       320  interfaces_bonding.py               ['interfaces', 'bonding']
       799  service_conntrack-sync.py           ['service', 'conntrack-sync']
[edit]
vyos@r15#

it seems 218 system_conntrack.py calls 799 service_conntrack-sync.py but interface bonding does not exist on this step prio 320
https://github.com/vyos/vyos-1x/blob/2277371fe18577502ce318c23789f86d1ec97be7/src/conf_mode/system_conntrack.py#L109-L110

1.4.0 is not affected as bug started in this commit

vyos-1x

# fail
65d4ee332 configdep: T6559: add smoketest of dependency script error
08f524a62 configdep: T6559: use single dependency list with reset under configd

# first fail commit
655ad2e9a configdep: T6559: drop global redundancy removal to fix error reporting

# good
b81fa2e6c Merge pull request #3814 from vyos/mergify/bp/sagitta/pr-3810

Details

Difficulty level
Normal (likely a few hours)
Version
VyOS 1.5-rolling-202408210022, VyOS 1.4-stable-202408210309, VyOS 1.4-stable-202407310309-iso-amd64.iso
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Unspecified (please specify)

Related Objects
Search...

Event Timeline

Viacheslav created this task.Wed, Aug 21, 11:44 AM
Viacheslav triaged this task as High priority.
Viacheslav added a project: VyOS 1.4 Sagitta (1.4.1).
Viacheslav changed the task status from Open to Confirmed.Wed, Aug 21, 11:46 AM
Viacheslav updated the task description. (Show Details)
jestabro claimed this task.Wed, Aug 21, 1:02 PM
jestabro updated the task description. (Show Details)Wed, Aug 21, 4:29 PM
jestabro added a comment.EditedWed, Aug 21, 5:15 PM

A few immediate notes, before preparing the solution:
(1) this is independent of whether one is running under configd or not
(2) this is more easily triggered under 1.4/1.5, which has default ['system', 'conntrack', 'modules'] entries (fixed in current), though can be reproduced in current with the above and 'set ... conntrack modules ..'
(3) this was avoided in 1.4.0 by the global dependency pruning; that raised other serious issues, however (T6559), due to constraints of the legacy commit algorithm.

So, the general problem occurs as follows:

         dep
     A   -->   B
prio \        / prio
            C

in the case that B requires C for success, since dependencies are called by the originating script (A). Given the generality of the term 'dependency', let us recall the terms here have the specific meanings:

prio: If config scripts E and F are required by the configuration changes, prio E < F means that E will run first
dependency: If config script E is required by the configuration changes, and F is a dependency, then F will be manually called after running E

Under the legacy commit algorithm, config scripts know nothing about each other, and are run according to the priority queue.
Under configd, we are still in the same position due to issues with global pruning present in 1.4.0 (T6559), mentioned above.

We must instead take advantage of our knowledge of the legacy priority queue (T5492) and the tools developed for the xml cache (notably, T6319) and predictively drop dependencies that will later be called by priority.

pasik subscribed.Wed, Aug 21, 8:31 PM
Viacheslav renamed this task from Confid dependency works incorrectly conntrack connntrack-sync to Confid dependency works incorrectly conntrack conntrack-sync.Thu, Aug 22, 6:47 AM
Viacheslav updated the task description. (Show Details)
jestabro added a comment.Fri, Aug 23, 3:23 AM

PR to follow smoketests and cosmetic changes:
https://github.com/vyos/vyos-1x/compare/current...jestabro:configdep-prio

jestabro added a comment.Fri, Aug 23, 7:14 PM

The following simple commit will need to be backported to sagitta so that a PR for the above will backport cleanly:
https://github.com/vyos/vyos-1x/pull/4013

jestabro added a subtask: T5666: Provide list of config-mode scripts scheduled for proposed commit.Fri, Aug 23, 7:32 PM
jestabro changed the status of subtask T5666: Provide list of config-mode scripts scheduled for proposed commit from On hold to In progress.Fri, Aug 23, 7:35 PM
jestabro mentioned this in T5666: Provide list of config-mode scripts scheduled for proposed commit.
jestabro added a comment.Sun, Aug 25, 2:36 PM

PR:
https://github.com/vyos/vyos-1x/pull/4015

jestabro changed the task status from Confirmed to Backport candidate.Mon, Aug 26, 1:10 PM
jestabro moved this task from Need Triage to Finished on the VyOS 1.5 Circinus board.
jestabro moved this task from Need Triage to Backport Candidates on the VyOS 1.4 Sagitta (1.4.1) board.
jestabro closed this task as Resolved.Mon, Aug 26, 4:46 PM
jestabro moved this task from Backport Candidates to Finished on the VyOS 1.4 Sagitta (1.4.1) board.
jestabro closed subtask T5666: Provide list of config-mode scripts scheduled for proposed commit as Resolved.