Page MenuHomeVyOS Platform
Create Task
Maniphest T3609

BGP Peer Group Changes Slow
Resolved (N/A)PublicBUG
Actions

Description

It appears that peer-group changes are extremely slow; a single change to a peer group takes in excess of 1 minute. This server has no RADIUS configured, and it is a bare metal with 2x Xeon Silver 4210, so it's not a resource issue.
Config:

vyos@cr01a-vyos# run show conf com
set address-family ipv4-unicast redistribute connected route-map 'BGP-REDISTRIBUTE'
set address-family ipv4-unicast redistribute static route-map 'BGP-REDISTRIBUTE'
set address-family ipv6-unicast redistribute connected route-map 'BGP-REDISTRIBUTE'
set neighbor 192.168.253.6 peer-group 'BACKBONE'
set neighbor 192.168.253.7 peer-group 'BACKBONE'
set neighbor 192.168.253.15 peer-group 'WDC07'
set neighbor fd52:d62e:8011:fffe:192:168:253:6 address-family ipv6-unicast peer-group 'BACKBONEv6'
set neighbor fd52:d62e:8011:fffe:192:168:253:7 address-family ipv6-unicast peer-group 'BACKBONEv6'
set neighbor fd52:d62e:8011:fffe:192:168:253:15 address-family ipv6-unicast peer-group 'WDC07v6'
set parameters confederation identifier '4242420696'
set parameters confederation peers '4242420668'
set parameters default no-ipv4-unicast
set parameters graceful-restart
set peer-group BACKBONE address-family ipv4-unicast nexthop-self
set peer-group BACKBONE address-family ipv4-unicast route-map import 'BGP-BACKBONE-IN'
set peer-group BACKBONE address-family ipv4-unicast soft-reconfiguration inbound
set peer-group BACKBONE bfd
set peer-group BACKBONE ebgp-multihop '2'
set peer-group BACKBONE remote-as 'external'
set peer-group BACKBONE update-source 'dum0'
set peer-group BACKBONEv6 address-family ipv6-unicast nexthop-self
set peer-group BACKBONEv6 address-family ipv6-unicast route-map import 'BGP-BACKBONE-IN'
set peer-group BACKBONEv6 address-family ipv6-unicast soft-reconfiguration inbound
set peer-group BACKBONEv6 bfd
set peer-group BACKBONEv6 ebgp-multihop '2'
set peer-group BACKBONEv6 remote-as 'external'
set peer-group BACKBONEv6 update-source 'dum0'
set peer-group WDC07 address-family ipv4-unicast nexthop-self
set peer-group WDC07 address-family ipv4-unicast soft-reconfiguration inbound
set peer-group WDC07 bfd
set peer-group WDC07 remote-as '4242420670'
set peer-group WDC07 update-source 'dum0'
set peer-group WDC07v6 address-family ipv6-unicast nexthop-self
set peer-group WDC07v6 address-family ipv6-unicast soft-reconfiguration inbound
set peer-group WDC07v6 bfd
set peer-group WDC07v6 remote-as '4242420670'
set peer-group WDC07v6 update-source 'dum0'
[edit protocols bgp 4242420670]

It's difficult to show how long it's taking, but the following two commands took about a minute and a half each:

vyos@cr01a-vyos# set peer-group BACKBONE address-family ipv4-unicast route-map export BGP-BACKBONE-OUT
[edit protocols bgp 4242420670]
vyos@cr01a-vyos# set peer-group BACKBONEv6 address-family ipv6-unicast route-map export BGP-BACKBONE-OUT
[edit protocols bgp 4242420670]

Here's me timing it on one of the pair:

vyos@cr01b-vyos# time set peer-group BACKBONEv6 address-family ipv6-unicast route-map export BGP-BACKBONE-OUT

real    1m37.344s
user    0m0.192s
sys     0m0.051s
[edit protocols bgp 4242420670]

Details

Difficulty level
Unknown (require assessment)
Version
1.3-beta-202106071927
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible
Issue type
Unspecified (please specify)

Related Objects

Event Timeline

trae32566 created this task.Jun 9 2021, 9:39 AM
trae32566 updated the task description. (Show Details)Jun 9 2021, 9:51 AM
Viacheslav added a subscriber: Viacheslav.EditedJun 9 2021, 5:59 PM

Try to check the same directly in the FRR.

I can't reproduce it with your configuration. VyOS 1.3-beta-202106081558

[email protected]# edit protocols bgp 65003 
[edit protocols bgp 65003]
[email protected]# 
[edit protocols bgp 65003]
[email protected]# set peer-group BACKBONE address-family ipv4-unicast route-map export BGP-BACKBONE-OUT
[edit protocols bgp 65003]
[email protected]# 
[edit protocols bgp 65003]
[email protected]# time commit

real	0m1.719s
user	0m1.069s
sys	0m0.264s
[edit protocols bgp 65003]
[email protected]#
trae32566 added a comment.Jun 9 2021, 6:02 PM

I'm wondering if perhaps it's my prefix lists or route-maps maybe? I can upload those in a bit once I test how fast / slow it is in frr.

Viacheslav added a comment.Jun 9 2021, 6:11 PM

It may be problem with large prefix-lists T2425

pasik added a subscriber: pasik.Jun 9 2021, 6:44 PM
tsantiago.work added a subscriber: tsantiago.work.Jun 9 2021, 7:27 PM

Here's the complete BGP policy config, since the route-maps include prefix-lists, AS paths, and large communities:

set policy prefix-list BGP-REDISTRIBUTE rule 10 action 'deny'
set policy prefix-list BGP-REDISTRIBUTE rule 10 description 'Block WDC07 peering'
set policy prefix-list BGP-REDISTRIBUTE rule 10 prefix '192.168.63.0/28'
set policy prefix-list BGP-REDISTRIBUTE rule 20 action 'permit'
set policy prefix-list BGP-REDISTRIBUTE rule 20 description 'Allow SL WDC07'
set policy prefix-list BGP-REDISTRIBUTE rule 20 ge '23'
set policy prefix-list BGP-REDISTRIBUTE rule 20 prefix '192.168.48.0/20'
set policy prefix-list BGP-REDISTRIBUTE rule 30 action 'permit'
set policy prefix-list BGP-REDISTRIBUTE rule 30 description 'Allow SL services'
set policy prefix-list BGP-REDISTRIBUTE rule 30 prefix '10.0.0.0/8'
set policy prefix-list BGP-REDISTRIBUTE rule 40 action 'permit'
set policy prefix-list BGP-REDISTRIBUTE rule 40 description 'Allow SL services'
set policy prefix-list BGP-REDISTRIBUTE rule 40 ge '9'
set policy prefix-list BGP-REDISTRIBUTE rule 40 prefix '10.0.0.0/8'
set policy prefix-list BGP-BACKBONE-DAL13 rule 10 action 'permit'
set policy prefix-list BGP-BACKBONE-DAL13 rule 10 description 'Allow DAL13'
set policy prefix-list BGP-BACKBONE-DAL13 rule 10 ge '23'
set policy prefix-list BGP-BACKBONE-DAL13 rule 10 prefix '192.168.16.0/20'
set policy prefix-list BGP-BACKBONE-IN description 'Inbound backbone routes from other sites'
set policy prefix-list BGP-BACKBONE-IN rule 10 action 'deny'
set policy prefix-list BGP-BACKBONE-IN rule 10 description 'Block default route'
set policy prefix-list BGP-BACKBONE-IN rule 10 prefix '0.0.0.0/0'
set policy prefix-list BGP-BACKBONE-IN rule 20 action 'deny'
set policy prefix-list BGP-BACKBONE-IN rule 20 description 'Block WDC07 primary'
set policy prefix-list BGP-BACKBONE-IN rule 20 ge '21'
set policy prefix-list BGP-BACKBONE-IN rule 20 prefix '192.168.48.0/20'
set policy prefix-list BGP-BACKBONE-IN rule 30 action 'deny'
set policy prefix-list BGP-BACKBONE-IN rule 30 description 'Block loopbacks'
set policy prefix-list BGP-BACKBONE-IN rule 30 ge '25'
set policy prefix-list BGP-BACKBONE-IN rule 30 prefix '192.168.253.0/24'
set policy prefix-list BGP-BACKBONE-IN rule 40 action 'deny'
set policy prefix-list BGP-BACKBONE-IN rule 40 description 'Block backbone peering'
set policy prefix-list BGP-BACKBONE-IN rule 40 ge '25'
set policy prefix-list BGP-BACKBONE-IN rule 40 prefix '192.168.254.0/24'
set policy prefix-list BGP-BACKBONE-IN rule 999 action 'permit'
set policy prefix-list BGP-BACKBONE-IN rule 999 description 'Allow everything else'
set policy prefix-list BGP-BACKBONE-IN rule 999 ge '1'
set policy prefix-list BGP-BACKBONE-IN rule 999 prefix '0.0.0.0/0'
set policy prefix-list BGP-BACKBONE-INT rule 10 action 'permit'
set policy prefix-list BGP-BACKBONE-INT rule 10 description 'Allow int'
set policy prefix-list BGP-BACKBONE-INT rule 10 ge '23'
set policy prefix-list BGP-BACKBONE-INT rule 10 prefix '192.168.0.0/20'
set policy prefix-list6 BGP-BACKBONE-DAL13-V6 rule 10 action 'permit'
set policy prefix-list6 BGP-BACKBONE-DAL13-V6 rule 10 description 'Allow DAL13'
set policy prefix-list6 BGP-BACKBONE-DAL13-V6 rule 10 ge '64'
set policy prefix-list6 BGP-BACKBONE-DAL13-V6 rule 10 prefix 'fd52:d62e:8011:1000::/52'
set policy prefix-list6 BGP-BACKBONE-IN-V6 description 'Inbound backbone routes from other sites'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 10 action 'deny'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 10 description 'Block default route'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 10 prefix '::/0'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 20 action 'deny'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 20 description 'Block WDC07 primary'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 20 ge '53'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 20 prefix 'fd52:d62e:8011:2000::/52'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 30 action 'deny'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 30 description 'Block peering and stuff'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 30 ge '53'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 30 prefix 'fd52:d62e:8011:f000::/52'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 999 action 'permit'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 999 description 'Allow everything else'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 999 ge '1'
set policy prefix-list6 BGP-BACKBONE-IN-V6 rule 999 prefix '::/0'
set policy prefix-list6 BGP-BACKBONE-INT-V6 rule 10 action 'permit'
set policy prefix-list6 BGP-BACKBONE-INT-V6 rule 10 description 'Allow int'
set policy prefix-list6 BGP-BACKBONE-INT-V6 rule 10 ge '64'
set policy prefix-list6 BGP-BACKBONE-INT-V6 rule 10 prefix 'fd52:d62e:8011::/52'
set policy prefix-list6 BGP-REDISTRIBUTE-V6 rule 10 action 'deny'
set policy prefix-list6 BGP-REDISTRIBUTE-V6 rule 10 description 'Block WDC07 peering'
set policy prefix-list6 BGP-REDISTRIBUTE-V6 rule 10 prefix 'fd52:d62e:8011:23e3::/64'
set policy prefix-list6 BGP-REDISTRIBUTE-V6 rule 20 action 'permit'
set policy prefix-list6 BGP-REDISTRIBUTE-V6 rule 20 description 'Allow WDC07'
set policy prefix-list6 BGP-REDISTRIBUTE-V6 rule 20 ge '64'
set policy prefix-list6 BGP-REDISTRIBUTE-V6 rule 20 prefix 'fd52:d62e:8011:2000::/52'
set policy route-map BGP-REDISTRIBUTE rule 10 action 'permit'
set policy route-map BGP-REDISTRIBUTE rule 10 description 'Allow WDC07 and services IPv4'
set policy route-map BGP-REDISTRIBUTE rule 10 match ip address prefix-list 'BGP-REDISTRIBUTE'
set policy route-map BGP-REDISTRIBUTE rule 10 set origin 'igp'
set policy route-map BGP-REDISTRIBUTE rule 20 action 'permit'
set policy route-map BGP-REDISTRIBUTE rule 20 description 'Allow WDC07 and services IPv6'
set policy route-map BGP-REDISTRIBUTE rule 20 match ipv6 address prefix-list 'BGP-REDISTRIBUTE-V6'
set policy route-map BGP-REDISTRIBUTE rule 20 set origin 'igp'
set policy route-map BGP-BACKBONE-IN rule 10 action 'permit'
set policy route-map BGP-BACKBONE-IN rule 10 match ip address prefix-list 'BGP-BACKBONE-IN'
set policy route-map BGP-BACKBONE-IN rule 20 action 'permit'
set policy route-map BGP-BACKBONE-IN rule 20 match ipv6 address prefix-list 'BGP-BACKBONE-IN-V6'
set policy route-map BGP-BACKBONE-IN rule 30 action 'permit'
set policy route-map BGP-BACKBONE-IN rule 30 match large-community large-community-list 'ANYCAST_ALL'
set policy route-map BGP-BACKBONE-OUT rule 10 action 'permit'
set policy route-map BGP-BACKBONE-OUT rule 10 match large-community large-community-list 'ANYCAST_WDC07'
set policy route-map BGP-BACKBONE-OUT rule 10 set metric '+100'
set policy route-map BGP-BACKBONE-OUT rule 20 action 'permit'
set policy route-map BGP-BACKBONE-OUT rule 20 match as-path 'INT'
set policy route-map BGP-BACKBONE-OUT rule 20 match ip address prefix-list 'BGP-BACKBONE-INT'
set policy route-map BGP-BACKBONE-OUT rule 20 set metric '+100'
set policy route-map BGP-BACKBONE-OUT rule 30 action 'permit'
set policy route-map BGP-BACKBONE-OUT rule 30 match as-path 'INT'
set policy route-map BGP-BACKBONE-OUT rule 30 match ipv6 address prefix-list 'BGP-BACKBONE-INT-V6'
set policy route-map BGP-BACKBONE-OUT rule 30 set metric '+100'
set policy route-map BGP-BACKBONE-OUT rule 40 action 'permit'
set policy route-map BGP-BACKBONE-OUT rule 40 match as-path 'DAL13'
set policy route-map BGP-BACKBONE-OUT rule 40 match ip address prefix-list 'BGP-BACKBONE-DAL13'
set policy route-map BGP-BACKBONE-OUT rule 40 set metric '+100'
set policy route-map BGP-BACKBONE-OUT rule 50 action 'permit'
set policy route-map BGP-BACKBONE-OUT rule 50 match as-path 'DAL13'
set policy route-map BGP-BACKBONE-OUT rule 50 match ipv6 address prefix-list 'BGP-BACKBONE-DAL13-V6'
set policy route-map BGP-BACKBONE-OUT rule 50 set metric '+100'
set policy route-map BGP-BACKBONE-OUT rule 999 action 'permit'
set policy route-map BGP-BACKBONE-OUT rule 999 call 'BGP-REDISTRIBUTE'
set policy route-map BGP-BACKBONE-OUT rule 999 description 'Allow redistributed routes'
set policy as-path-list DAL13 rule 10 action 'permit'
set policy as-path-list DAL13 rule 10 description 'Alow anything from or via DAL13'
set policy as-path-list DAL13 rule 10 regex '.*4242420668.*'
set policy as-path-list INT rule 10 action 'permit'
set policy as-path-list INT rule 10 description 'Allow anything from or via int'
set policy as-path-list INT rule 10 regex '.*4242420666.*'
set policy large-community-list ANYCAST_ALL rule 10 action 'permit'
set policy large-community-list ANYCAST_ALL rule 10 description 'Allow all anycast from anywhere'
set policy large-community-list ANYCAST_ALL rule 10 regex '4242420696:100:.*'
set policy large-community-list ANYCAST_WDC07 rule 10 action 'permit'
set policy large-community-list ANYCAST_WDC07 rule 10 description 'Allow all anycast from wdc07'
set policy large-community-list ANYCAST_WDC07 rule 10 regex '4242420696:100:3'

If you have any recommendations of changes, please let me know. I will try testing with bare FRR later on tonight.

trae32566 added a comment.EditedJun 10 2021, 7:32 AM

FRR appears to have no problems processing this extremely quickly:

vyos@cr01b-vyos# time vtysh -c "conf t" -c "router bgp 4242420670" -c "address-family ipv4 unicast" -c "neighbor BACKBONE route-map BGP-BACKBONE-OUT out"

real    0m0.118s
user    0m0.110s
sys     0m0.007s
[edit]
vyos@cr01b-vyos# vtysh -c 'show run bgpd' | grep 'route-map BGP-BACKBONE-OUT out'
  neighbor BACKBONE route-map BGP-BACKBONE-OUT out
[edit]

I should clarify that the commit does not take long, the set is what is taking forever. This makes me think there's some sort of validation going on in the background that might be hanging.

Viacheslav added a comment.Jun 10 2021, 11:40 AM

Commit with such policies:

[email protected]# time commit

real	0m8.514s
user	0m6.754s
sys	0m0.909s
[edit]
[email protected]#
trae32566 added a comment.EditedJun 10 2021, 4:15 PM

Again, the problem is not the time it takes to commit, but the time it takes to set. I will try reproducing it again and see if I can come up with an easier way. I would suggest trying adding the BGP config and prefix lists and route-maps without the export route-map applied, then commit, then try applying the export route-maps.

trae32566 closed this task as Resolved.Jun 12 2021, 7:20 AM
trae32566 claimed this task.

This appears to be fixed in the most recent rolling releases; I'm not sure how, but it's fixed.

c-po moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.Jun 13 2021, 11:20 AM
SrividyaA changed the task status from Resolved to Resolved N/A.Aug 31 2021, 6:17 PM
SrividyaA set Issue type to Unspecified (please specify).
syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.Aug 29 2022, 12:17 PM
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus (1.3.0) board.