Page MenuHomeVyOS Platform
Create Task
Maniphest T7665

Commit failed when attempted to create a new BGP instance for VRF linked with VXLAN(L3VNI).
Closed, ResolvedPublicBUG
Actions

Description

FRR does not allow to create a new BGP instance for VRF when linked with VXLAN (L3VNI).
Issue reported in FRR: https://github.com/FRRouting/frr/issues/16152

Configuration to reproduce:

set interfaces dummy dum0 address '192.0.2.1/32'
set interfaces vxlan vxlan2000 mtu '1500'
set interfaces vxlan vxlan2000 parameters nolearning
set interfaces vxlan vxlan2000 port '4789'
set interfaces vxlan vxlan2000 source-address '192.0.2.1'
set interfaces vxlan vxlan2000 vni '2000'
set protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set protocols bgp address-family l2vpn-evpn advertise-all-vni
set protocols bgp neighbor 192.0.2.9 address-family ipv4-unicast allowas-in number '10'
set protocols bgp neighbor 192.0.2.9 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor 192.0.2.9 remote-as 'internal'
set protocols bgp neighbor 192.0.2.9 update-source '192.0.2.1'
set protocols bgp neighbor 192.0.2.25  peer-group 'EVPN'
set protocols bgp parameters log-neighbor-changes
set protocols bgp parameters network-import-check
set protocols bgp parameters router-id '192.0.2.1'
set protocols bgp peer-group EVPN address-family l2vpn-evpn
set protocols bgp peer-group EVPN remote-as '100'
set protocols bgp peer-group EVPN update-source 'dum0'
set protocols bgp system-as '100'
set vrf name blue protocols bgp address-family ipv4-unicast redistribute connected
set vrf name blue protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 address-family ipv4-unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 remote-as '4209000400'
set vrf name blue protocols bgp system-as '200'
set vrf name blue table '2000'
set vrf name blue vni '2000'

Commit error:

[4231|mgmtd] sending configuration [4232|zebra] sending configuration
[4233|ripd] sending configuration [4234|ripngd] sending configuration
[4235|ospfd] sending configuration [4237|ldpd] sending configuration
[4236|ospf6d] sending configuration [4238|bgpd] sending configuration
[4239|isisd] sending configuration BGP is already running; AS is 100
line 1: Failure to communicate[13] to bgpd, line: router bgp 200
vrf blue  line 2: Warning[4]...: early exit from config file [4238|bgpd]
Configuration file[/etc/frr/frr.conf] processing failure: 13
[4243|babeld] sending configuration [4246|watchfrr] sending
configuration Waiting for children to finish applying config...
[4248|staticd] sending configuration [4249|bfdd] sending configuration
[4252|pim6d] sending configuration [4232|zebra] done [4236|ospf6d] done
[4246|watchfrr] done [4243|babeld] done [4235|ospfd] done [4248|staticd]
done [4234|ripngd] done [4231|mgmtd] done [4237|ldpd] done [4233|ripd]
done [4239|isisd] done [4249|bfdd] done [4252|pim6d] done [4257|zebra]
sending configuration [4256|mgmtd] sending configuration [4259|ripngd]
sending configuration [4260|ospfd] sending configuration [4261|ospf6d]
sending configuration [4262|ldpd] sending configuration [4263|bgpd]
sending configuration [4264|isisd] sending configuration [4258|ripd]
sending configuration [4268|babeld] sending configuration Waiting for
children to finish applying config... BGP is already running; AS is
100 line 1: Failure to communicate[13] to bgpd, line: router bgp
200 vrf blue  line 2: Warning[4]...: early exit from config file
[4271|watchfrr] sending configuration [4273|staticd] sending
configuration [4263|bgpd] Configuration file[/etc/frr/frr.conf]
processing failure: 13 [4274|bfdd] sending configuration [4277|pim6d]
sending configuration [4277|pim6d] done [4259|ripngd] done
[4273|staticd] done [4260|ospfd] done [4262|ldpd] done [4258|ripd] done
[4261|ospf6d] done [4257|zebra] done [4274|bfdd] done [4256|mgmtd] done
[4264|isisd] done [4268|babeld] done [4271|watchfrr] done

[[vrf name blue protocols bgp]] failed
Commit failed

Error during reboot:

[   41.044582] vyos-router[823]: Mounting VyOS Config...done.
[   84.014817] vyos-router[823]: Starting VyOS router: migrate configure failed!
[   84.360354] vyos-config[825]: Configuration error

Config which is deleted

vyos@vyos# compare
[vrf name blue protocols]
+ bgp {
+     address-family {
+         ipv4-unicast {
+             redistribute {
+                 connected
+             }
+         }
+         l2vpn-evpn {
+             advertise {
+                 ipv4 {
+                     unicast
+                 }
+             }
+         }
+     }
+     neighbor 10.10.7.2 {
+         address-family {
+             ipv4-unicast
+         }
+         remote-as "300"
+     }
+     system-as "200"
+ }

Output from FRR console:

vyos@vyos# vtysh

Hello, this is FRRouting (version 9.1.3).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

vyos# conf t
vyos(config)# router bgp  200 vrf blue
BGP is already running; AS is 100

The issue is not reproduced in VyOS 2025.07.28-0022-rolling

Details

Version
1.4.3
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

SrividyaA created this task.Jul 29 2025, 12:54 PM
SrividyaA triaged this task as High priority.Jul 29 2025, 12:59 PM
SrividyaA updated the task description. (Show Details)
fernando subscribed.Jul 29 2025, 1:00 PM

patch : https://github.com/FRRouting/frr/pull/16159/commits/f153b9a9b636e7ca16ef066e934d355ba922df2b

pasik subscribed.Jul 29 2025, 1:43 PM
Fabse subscribed.EditedJul 31 2025, 10:51 AM

Reboot issue is not present in latest rolling. Commit issue still persists on latest rolling. Reboot is fixed but if I want to change vrf system-as:

set vrf name blue protocols bgp system-as 'xyz'

and commit I get this error;

[4857|mgmtd] sending configuration [4858|zebra] sending configuration
[4859|ripd] sending configuration [4860|ripngd] sending configuration
[4861|ospfd] sending configuration [4862|ospf6d] sending configuration
[4863|ldpd] sending configuration [4864|bgpd] sending configuration
[4865|isisd] sending configuration [4869|babeld] sending configuration
BGP is already running; AS is 60664 line 1: Failure to communicate[13]
to bgpd, line: router bgp 4208000000 vrf blue  line 2: Warning[4]...:
early exit from config file [4872|watchfrr] sending configuration
[4874|staticd] sending configuration [4864|bgpd] Configuration
file[/etc/frr/frr.conf] processing failure: 13 Waiting for children to
finish applying config... [4875|bfdd] sending configuration [4878|pim6d]
sending configuration [4865|isisd] done [4860|ripngd] done [4859|ripd]
done [4857|mgmtd] done [4862|ospf6d] done [4858|zebra] done
[4872|watchfrr] done [4861|ospfd] done [4863|ldpd] done [4875|bfdd] done
[4874|staticd] done [4878|pim6d] done [4869|babeld] done [4882|mgmtd]
sending configuration [4884|ripd] sending configuration [4886|ospfd]
sending configuration [4887|ospf6d] sending configuration [4883|zebra]
sending configuration [4888|ldpd] sending configuration [4889|bgpd]
sending configuration [4885|ripngd] sending configuration [4890|isisd]
sending configuration BGP is already running; AS is 60664 line 1:
Failure to communicate[13] to bgpd, line: router bgp 4208000000 vrf blue
line 2: Warning[4]...: early exit from config file [4889|bgpd]
Configuration file[/etc/frr/frr.conf] processing failure: 13
[4894|babeld] sending configuration [4897|watchfrr] sending
configuration [4899|staticd] sending configuration [4900|bfdd] sending
configuration Waiting for children to finish applying config...
[4903|pim6d] sending configuration [4885|ripngd] done [4890|isisd] done
[4886|ospfd] done [4888|ldpd] done [4887|ospf6d] done [4883|zebra] done
[4882|mgmtd] done [4884|ripd] done [4894|babeld] done [4897|watchfrr]
done [4900|bfdd] done [4899|staticd] done [4903|pim6d] done
c-po claimed this task.Aug 25 2025, 9:42 AM
c-po added a project: VyOS 1.4 Sagitta (1.4.4).
c-po added a comment.Aug 25 2025, 9:50 AM
In T7665#230894, @fernando wrote:

patch : https://github.com/FRRouting/frr/pull/16159/commits/f153b9a9b636e7ca16ef066e934d355ba922df2b

Unfortunately this patch makes things even worse. It looks like a nice solution but it turns out that FRR 9.1 will internally get confused. BGP related config load tests fail, as FRR vtysh internally thinks that a VNI (e.g. 4000) is still active, but it's not exposed on vtysh -c "show run" leading to a full fault state.

@Fabse @fernando

as an "unbreak" measure one could change the above mentioned CLI config from. Turns out my initial idea in also exposing different BGP ASNs per VRF for the BGP process caused more harm then good and we should have used the local-as feature from the beginning.

set vrf name blue protocols bgp address-family ipv4-unicast redistribute connected
set vrf name blue protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 address-family ipv4-unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 remote-as '4209000400'
set vrf name blue protocols bgp system-as '4208000000'
set vrf name blue table '2000'
set vrf name blue vni '2000'

to

set vrf name blue protocols bgp address-family ipv4-unicast redistribute connected
set vrf name blue protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 address-family ipv4-unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 local-as 4208000000 no-prepend replace-as
set vrf name blue protocols bgp neighbor 10.10.7.2 remote-as '4209000400'
set vrf name blue protocols bgp system-as '60664'
set vrf name blue table '2000'
set vrf name blue vni '2000'

resulting in a reboot safe configuration.

vyos@vyos:~$ show bgp vrf blue ipv4
BGP table version is 5, local router ID is 10.10.7.1, vrf id 7
Default local pref 100, local AS 60664
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 10.0.0.0/24      10.10.7.2                0             0 4209000400 ?
 *  10.10.7.0/30     10.10.7.2                0             0 4209000400 ?
 *>                  0.0.0.0                  0         32768 ?
 *> 172.18.202.0/24  10.10.7.2                0             0 4209000400 ?
 *> 172.18.254.202/32
                     10.10.7.2                0             0 4209000400 ?
 *> 192.0.2.0/30     10.10.7.2                0             0 4209000400 ?

Displayed  5 routes and 6 total paths
vyos@vyos:~$ show bgp vrf blue summary

IPv4 Unicast Summary (VRF blue):
BGP router identifier 10.10.7.1, local AS number 60664 vrf-id 7
BGP table version 5
RIB entries 9, using 864 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.10.7.2       4 4209000400        15        12        5    0    0 00:07:29            5        5

Total number of neighbors 1
c-po changed the task status from Open to In progress.Aug 25 2025, 10:02 AM
Fabse added a comment.Aug 26 2025, 10:11 AM

Hi @c-po and thanks for the insight. I understand that this is a workaround for a reboot safe configuration if you're using vrf bgp neighbors.
We are also using vrf inside our infrastructure as well without vrf bgp neighbors and only redistribute connected routes and advertise them via EVPN.

set vrf name blue protocols bgp address-family ipv4-unicast redistribute connected
set vrf name blue protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name blue protocols bgp address-family l2vpn-evpn route-target export '4208000000:2000'
set vrf name blue protocols bgp address-family l2vpn-evpn route-target import '4208000000:2000'
set vrf name blue protocols bgp system-as '4208000000'
set vrf name blue table '2000'
set vrf name blue vni '2000'

If I reboot that system it won't load the proper config anymore. How would your workaround apply to this configuration?

c-po added a comment.EditedAug 27 2025, 12:04 PM

Hi @Fabse,

the solution to this issue as worked up in our private chat is - for the rest of the world:

For VyOS routers without an eBGP connection to anyone else within that VRF

set vrf name blue protocols bgp address-family ipv4-unicast redistribute connected
set vrf name blue protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name blue protocols bgp address-family l2vpn-evpn route-target export '4208000000:2000'
set vrf name blue protocols bgp address-family l2vpn-evpn route-target import '4208000000:2000'
set vrf name blue protocols bgp system-as '100'
set vrf name blue table '2000'
set vrf name blue vni '2000'

For VyOS routers with an eBGP connection to a remote peer

set vrf name blue protocols bgp address-family ipv4-unicast redistribute connected
set vrf name blue protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 address-family ipv4-unicast
set vrf name blue protocols bgp neighbor 10.10.7.2 remote-as 300
set vrf name blue protocols bgp neighbor 10.10.7.2 local-as 200 no-prepend replace-as
set vrf name blue protocols bgp system-as '100'
set vrf name blue table '2000'
set vrf name blue vni '2000'

This will prevent leaking the internal ASN 100 to the outside world. The outside world will see you only as AS 200 with routes originating from AS 200 or being in the transit path.

To avoid potential issues down the line - and given that there's no compelling technical reason to retain the system-as CLI node under per-VRF BGP configuration, which cannot be achieved through alternative means - the maintainers have collectively decided to deprecate the following command:

set vrf name <name> protocols bgp system-as <asn>

Starting with VyOS 1.4.4, this CLI command will be considered deprecated. While it will still be accepted, it will no longer have any operational effect. A deprecation warning will be displayed at commit time, indicating that the BGP ASN from the global BGP configuration is now used instead.

A migration script will handle the transition and perform the following actions:

  • Ensure a global BGP configuration exists; if not, initialize one.
  • Iterate over all configured VRFs to determine whether a BGP instance exists and whether its system-as value differs from the global system-as.
  • For any mismatches, update the VRF BGP instance to use the global system-as and apply the local-as ASN no-prepend replace-as option on all affected neighbors to preserve existing behavior.
  • If a neighbor is already configured with a local-as directive, that neighbor will be excluded from the migration process, as it already follows a custom configuration.
c-po closed this task as Resolved.Aug 28 2025, 7:01 AM
c-po moved this task from Backlog to Finished on the VyOS 1.4 Sagitta (1.4.4) board.
c-po mentioned this in T7760: bgp: remove per vrf instance system-as node.Aug 28 2025, 7:03 AM
c-po changed the status of subtask T7760: bgp: remove per vrf instance system-as node from Open to In progress.
c-po updated the task description. (Show Details)Aug 28 2025, 8:15 AM
c-po updated the task description. (Show Details)
c-po mentioned this in rVYOSONEXd871fe9c4c65: bgp: T7760: deprecate per bgp vrf instance system-as node.Sep 23 2025, 3:01 PM
c-po closed subtask T7760: bgp: remove per vrf instance system-as node as Resolved.Oct 2 2025, 3:28 PM
dmbaturin removed a project: VyOS 1.4 Sagitta (1.4.4).Wed, Nov 19, 2:20 PM