Page MenuHomeVyOS Platform
Create Task
Maniphest T3657

BGP neighbors ipv6 not able to establish with IPv6 link-local addresses
Closed, ResolvedPublic
Actions

Description

Hi

we have been detected an issues when we want a bgp session with an ipv6 link-local address , it is not possible to establish .let me show :

version :

vyos@vyos:~$ show version

Version:          VyOS 1.4-rolling-202106260417
Release Train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Sat 26 Jun 2021 07:40 UTC
Build UUID:       5cb5c98a-9eaf-4630-b563-f5e207950933
Build Commit ID:  848f1c917efc29

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  Bochs
Hardware model:   Bochs
Hardware S/N:
Hardware UUID:    8876859b-c954-4183-aad8-148843b3b890

Copyright:        VyOS maintainers and contributor

bgp peer config :

set protocols bgp address-family ipv6-unicast redistribute connected
set protocols bgp local-as '65000'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 address-family ipv6-unicast route-map import 'BGP-IPV6'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 ebgp-multihop '2'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 remote-as '65005'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 update-source 'fe80::5200:ff:fe07:1'
set protocols bgp parameters default no-ipv4-unicast
set protocols bgp parameters router-id '2.2.2.2'

show bgp ipv6 neighbors output :

vyos@vyos-ipv6-pl2#run show bgp ipv6 neighbors
BGP neighbor is fe80::5200:ff:fe0c:1, remote AS 65005, local AS 65000, external link
  BGP version 4, remote router ID 0.0.0.0, local router ID 2.2.2.2
  BGP state = Active
  Last read 00:11:49, Last write never
  Hold time is 180, keepalive interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds
  Update source is fe80::5200:ff:fe07:1

 For address family: IPv6 Unicast
  Not part of any update group
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Route map for incoming advertisements is *BGP-IPV6
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:11:49,  Waiting for peer OPEN
  External BGP neighbor may be up to 2 hops away.
BGP Connect Retry Timer in Seconds: 120
Next connect timer due in 12 seconds
Read thread: off  Write thread: off  FD used: -1

ipv6 settings :

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 50:00:00:07:00:01 brd ff:ff:ff:ff:ff:ff
    inet6 2001:db8:3333::15/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::5200:ff:fe07:1/64 scope link
       valid_lft forever preferred_lft forever]

vyos@vyos:~$ show ipv6 neighbors
IP Address                                         Device     State                LLADDR
----------                                         ------     -----                ------
2001:db8:3333::16                                  eth1       stale                50:00:00:0c:00:01
fe80::5200:ff:fe0c:1                               eth1       stale                50:00:00:0c:00:01

journal log :

Jun 28 12:42:36 vyos-ipv6-pl2 sudo[4993]: pam_unix(sudo:session): session closed for user root
Jun 28 12:42:36 vyos-ipv6-pl2 commit[4996]: Successful change to active configuration by user vyos on /dev/ttyS0
Jun 28 12:42:36 vyos-ipv6-pl2 bgpd[978]: [EC 100663299] can't bind socket for fe80::5200:ff:fe07:1 : Invalid argument
Jun 28 12:44:36 vyos-ipv6-pl2 bgpd[978]: [EC 100663299] can't bind socket for fe80::5200:ff:fe07:1 : Invalid argument
Jun 28 12:46:36 vyos-ipv6-pl2 bgpd[978]: [EC 100663299] can't bind socket for fe80::5200:ff:fe07:1 : Invalid argument
Jun 28 12:48:36 vyos-ipv6-pl2 bgpd[978]: [EC 100663299] can't bind socket for fe80::5200:ff:fe07:1 : Invalid argument

there is something strange , when you want a ping(v6) using the link-local address ,it shows a similar issues :

vyos@vyos:~$sudo /usr/sbin/ip vrf exec default /bin/ping6 -I fe80::5200:ff:fe07:1 fe80::5200:ff:fe0c:1

/bin/ping6: bind icmp socket: Invalid argument

if  I use the interfaces , it works 

$  sudo /usr/sbin/ip vrf exec default /bin/ping6 -I eth1 fe80::5200:f
/bin/ping6: Warning: source address might be selected on device other than: eth1
PING fe80::5200:ff:fe0c:1(fe80::5200:ff:fe0c:1) from :: eth1: 56 data bytes
64 bytes from fe80::5200:ff:fe0c:1%eth1: icmp_seq=1 ttl=64 time=2.00 ms
64 bytes from fe80::5200:ff:fe0c:1%eth1: icmp_seq=2 ttl=64 time=1.64 ms
64 bytes from fe80::5200:ff:fe0c:1%eth1: icmp_seq=3 ttl=64 time=1.66 ms
64 bytes from fe80::5200:ff:fe0c:1%eth1: icmp_seq=4 ttl=64 time=2.00 ms
64 bytes from fe80::5200:ff:fe0c:1%eth1: icmp_seq=5 ttl=64 time=1.73 ms

Details

Version
1.4-rolling-202106260417
Is it a breaking change?
Perfectly compatible
Issue type
Feature (new functionality)

Related Objects

Event Timeline

fernando created this task.Jun 28 2021, 1:45 PM
fernando created this object in space S1 VyOS Public.
fernando added a comment.EditedJun 28 2021, 2:47 PM

I add an extra commentary , it is config on FRR:

vyos@vyos:~$ sudo vtysh -c "show running"
Building configuration...

Current configuration:
!
frr version 7.5.1-20210625-00-g566fb3c1d

router bgp 65000
 bgp router-id 2.2.2.2
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor fe80::5200:ff:fe0c:1 remote-as 65005
 neighbor fe80::5200:ff:fe0c:1 ebgp-multihop 2
 neighbor fe80::5200:ff:fe0c:1 update-source fe80::5200:ff:fe07:1
 !
 address-family ipv6 unicast
  redistribute connected
  neighbor fe80::5200:ff:fe0c:1 activate
  neighbor fe80::5200:ff:fe0c:1 route-map BGP-IPV6 in
 exit-address-family
!
ipv6 prefix-list BGP-IPV6 seq 100 permit ::/0 le 48
!
route-map BGP-IPV6 permit 10
 match ipv6 address prefix-list BGP-IPV6

and if I change the update-source to physical interface 'eth1' , the issues continues :

vyos@vyos:~$ show configuration commands | match bgp
set protocols bgp address-family ipv6-unicast redistribute connected
set protocols bgp local-as '65000'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 address-family ipv6-unicast route-map import 'BGP-IPV6'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 ebgp-multihop '2'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 remote-as '65005'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 update-source 'eth1'
set protocols bgp parameters default no-ipv4-unicast
set protocols bgp parameters router-id '2.2.2.2'

default behavior is to use ipv6 link-local address .

Matwolf subscribed.Jun 28 2021, 4:36 PM
c-po subscribed.EditedJun 28 2021, 5:45 PM

I wonder why you use ebgp multihop wirh link local addresses?

Also FRR manual states:

When you connect to a BGP peer over an IPv6 link-local address, you have to specify the IFNAME of the interface used for the connection. T

Please try set protocols bgp neighbor <addr> interface eth1

Matwolf added a comment.EditedJun 28 2021, 6:36 PM
In T3657#97243, @c-po wrote:

Also FRR manual states:

When you connect to a BGP peer over an IPv6 link-local address, you have to specify the IFNAME of the interface used for the connection. T

Please try set protocols bgp neighbor <addr> interface eth1

FRR manual states that the command is deprecated: https://docs.frrouting.org/en/latest/bgp.html#clicmd-neighbor-PEER-interface-IFNAME

Also set protocols bgp neighbor <addr> interface eth1 is not available on 1.4-rolling-202106260417 (where I tested it)

c-po added a comment.EditedJun 28 2021, 6:59 PM

Hello @Matwolf,

even if FRR manual states the deprecation notice, we have our own layer of abstraction and will deal with it once it is required.
For the time beeing, I just checked the commands (using tab completion).

As I never used this before but this should work:

set protocols bgp neighbor fe80::5200:ff:fe0c:1 interface v6only remote-as '65005'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 update-source 'eth1'

But it triggers a real bug now with the remove-as validator which must be taken a look at. If this is to struggeling for you I recommend switching to 1.3.0-rc as 1.4 is the current bleeding edge version.

cpo@LR1.wue3# show protocols bgp
+local-as 100
+neighbor fe80::5200:ff:fe0c:1 {
+    interface {
+        v6only {
+            remote-as 65005
+        }
+    }
+    update-source eth1
+}
[edit]
cpo@LR1.wue3# commit
[ protocols bgp ]
Neighbor "fe80::5200:ff:fe0c:1" remote-as must be set!

[[protocols bgp]] failed
Commit failed

THis is definately an interesting use-case which should receive it's own dedicated smoketest

c-po changed the task status from Open to Confirmed.Jun 28 2021, 7:07 PM
c-po claimed this task.
c-po triaged this task as Normal priority.
fernando added a comment.Jun 28 2021, 7:39 PM
In T3657#97243, @c-po wrote:

I wonder why you use ebgp multihop wirh link local addresses?

I used it only for testing (but this command increment ttl in two).

however , I did the changed and the issues continues:

vyos@vyos-ipv6-pl2:~$ show configuration commands | match bgp
set protocols bgp address-family ipv6-unicast redistribute connected
set protocols bgp local-as '65000'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 address-family ipv6-unicast route-map import 'BGP-IPV6'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 remote-as '65005'
set protocols bgp neighbor fe80::5200:ff:fe0c:1 update-source 'eth1'
set protocols bgp parameters default no-ipv4-unicast
set protocols bgp parameters router-id '2.2.2.2'

the output show bgp ipv6 neighbors

vyos@vyos-ipv6-pl2:~$ show bgp ipv6 neighbors
BGP neighbor is fe80::5200:ff:fe0c:1, remote AS 65005, local AS 65000, external link
  BGP version 4, remote router ID 0.0.0.0, local router ID 2.2.2.2
  BGP state = Active
  Last read 00:14:49, Last write never
  Hold time is 180, keepalive interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds
  Update source is eth1

 For address family: IPv6 Unicast
  Not part of any update group
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Route map for incoming advertisements is *BGP-IPV6
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:14:49,  Waiting for peer OPEN
BGP Connect Retry Timer in Seconds: 120
Next connect timer due in 108 seconds
Read thread: off  Write thread: off  FD used: -1

also I add the journalclt log :

Jun 28 18:55:38 vyos-ipv6-pl2 bgpd[879]: [EC 100663299] can't bind socket for fe80::5200:ff:fe07:1 : Invalid argument
Jun 28 18:57:38 vyos-ipv6-pl2 bgpd[879]: [EC 100663299] can't bind socket for fe80::5200:ff:fe07:1 : Invalid argument
Jun 28 18:59:38 vyos-ipv6-pl2 bgpd[879]: [EC 100663299] can't bind socket for fe80::5200:ff:fe07:1 : Invalid argument


vyos@vyos-ipv6-pl2:~$ sudo route -6 | grep fe80::5200:ff:fe07:1
fe80::5200:ff:fe07:1/128       [::]                       Un   0   3     0 eth1
pasik subscribed.Jun 29 2021, 7:52 AM
c-po mentioned this in rVYOSONEX5a5c0cd2e6f5: bgp: T3657: fix remote-as validator for IPv6 link-local peering.Jun 29 2021, 4:34 PM
c-po mentioned this in rVYOSONEX6152084f5f49: smoketest: bgp: T3657: test ipv6 link-local peering.
c-po added a comment.Jun 29 2021, 4:36 PM

Bug confirmed and fixed,

vyos@vyos:~$ show configuration commands | grep bgp
set protocols bgp local-as '200'
set protocols bgp neighbor eth0.204 address-family ipv6-unicast
set protocols bgp neighbor eth0.204 interface v6only remote-as '100'
vyos@vyos:~$ show bgp ipv6 sum
IPv6 Unicast Summary:
BGP router identifier 172.18.254.201, local AS number 200 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 21 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
eth0.204 4 100 99 99 0 0 0 01:35:07 0 0

Total number of neighbors 1

Please check with the next rolling release - I will trigger an image build for you now

c-po closed this task as Resolved.Jun 29 2021, 4:36 PM
c-po changed Version from -VyOS 1.4-rolling-202106260417 to 1.4-rolling-202106260417.
c-po changed Is it a breaking change? from Unspecified (possibly destroys the router) to Stricter validation.
Matwolf added a comment.Jun 29 2021, 5:22 PM

Hello @c-po,

looking at your configuration I see you set the neighbor using the interface name.
But in that case how does FFR know which IP address to connect to initiate a BGP session? Works in passive mode only?

Do you know if it works specifing configuring the neighbor using the link-local address of the remote router and setting "update-source" using the interface name?

(when the new build will be available?)

Thanks!

c-po added a comment.EditedJun 29 2021, 6:26 PM

Hi @Matwolf,

the new build is already available. I am unsure if this works or is even supported by FRR.
Please consult FRR manual and try configuring this manually from vtysh.

If this works indeed, I will get it done for VyOS, too

Matwolf added a comment.Jun 29 2021, 10:44 PM

Hi @c-po ,

upgraded to 1.4-rolling-202106290839 but still not working for my setup...

As suggested, I tried to fiddle with vtysh and eventually I was able to make it work.... but unfortunately using the "deprecated command"....

As soon as I typed (into vtysh) the following configuration command, the BGP session came up immediately!

neighbor fe80::<of-the-remote-peer> interface wg1234

Could be related to the fact I'm using a wireguard interface to interconnect with my peer?

If I try to configure my neighbor using your config as template the session doesn't come up and I get the following log messages (I see them by enabling debug neighbor-events from vtysh):

Jun 30 00:30:38 dn42-it01 bgpd[1003]: [Event] fe80::remote-peer-link-local connection rejected(VRF default:my-asn-number:default) - not configured and not valid for dynamic

Seems like the remote peer is trying to establish a BGP session but the neighbor is not recognized by my router.

I'm sincerely out of ideas.... Is it possible that the only way to make it work is by using a deprecated command?

c-po added a comment.Jun 30 2021, 6:48 PM

Hi @Matwolf,

please stop the idea of "deprecated" command. VyOS commands are in no relation to FRR commands.
If (and when) the FRR syntax changes, we will ensure it will still work by either migrating the VyOS CLI configuration dynamically on upgrade or by adjusting to the FRR configuration "under the hood" with our Jinja2 template.

As the provided configuration mentioned in https://phabricator.vyos.net/T3657#97274 works, I wonder if there is any particular reason to "hardcode" the Link-Local neighbor addresses instead of using a way of "auto discovery" as mentioned in https://phabricator.vyos.net/T3657#97274. I am still trying to catch the use-case for this szenario.

Matwolf added a comment.Jun 30 2021, 7:04 PM

Hi @c-po,

i was referring to the FRR command as deprecated, not the corresponding VyOS command. The VyOS command is not even available in the last version of VyOS... I was able to try it only via vtysh...

The configuration you used in https://phabricator.vyos.net/T3657#97274 doesn't work in my specific setup and produces the error mentioned on my last comment...

I don't know why, but hard coding the address is, for my setup, the only way to make it work unfortunately...

c-po added a comment.Jun 30 2021, 7:38 PM

Please share your entire setup then somwe are able to help out.

Matwolf added a comment.Jun 30 2021, 9:30 PM

All of my neighbors are connected with me via wireguard interfaces (a different interface for every peering). I have no physical direct link with any peer.
All neighbors using IPv4 or ULA IPv6 addresses are working properly.

Every wireguard interface (7 in total, at the moment) has the same tunneled IPv4 address and IPv6 addresses (ULA or link-local) of the others.

Then I create a static route for every IPv4 or IPv6 tunneled address of my peers to force the traffic through the right wg interface.
I don't create static routes for link local addresses.

I don't manage the other routers, but I know that they are, tipically, using Bird for BGP. They are not VyOS routers as far as I know.

As I mentioned before, forcing the neighbor link-local address in the BGP configuration and then manually specifying the interface using vtysh (neighbor <linklocal-remote-addr> interface <wireguard-interface>) allow the BGP session to establish successfully.

The configuration of the neighbor using the interface name instead of its link local address followed by the use of VyOS config command
set interface v6only remote-as NNNN
doesn't allow a successful BGP session and produces the following bgp (debug) logs:

Jun 30 22:30:48 dn42-it01 bgpd[1003]: [Event] <remote-link-local-address> connection rejected(VRF default:<my-local-as-number>:default) - not configured and not valid for dynamic

Using tcpdump on the wireguard interface allows me to see the router advertisement ICMPv6 packets from my link-local address to ipv6-allnodes (but no responses... maybe they are simply being ignored) and then the BGP packets (Open Message) from the remote router link-local address to mine.

I'll be more than happy to provide all the extra info you'll need, but I don't think I can post ALL my config commands here... Maybe only some specific parts in case you need them ;)

Thanks for your help

c-po added a comment.Jul 1 2021, 7:39 AM

Please share your configuration.

edofullin subscribed.Sep 18 2021, 8:27 AM

Are there updates on this issue?

With the last rolling release I still have to do neighbor fe80::xxxx interface wgN from vtysh to make it work.
Also that change seems to be not permanent

c-po reopened this task as Needs testing.Sep 19 2021, 1:01 PM
c-po set Issue type to Unspecified (please specify).
c-po added a comment.EditedSep 25 2021, 2:30 PM

Actually the VyOS syntax is a bit different - you do not need to establish a "relationship" with a link-local address - multiple links could indeed share the same link local address causing conflicts and non-uniqueness in the config.

The neighbor x:x:x:x:x:x:x:x interface VTYSH option is currently not supported in VyOS 1.4 - but it might be worth adding it for 1.4

Config VyOS 1.4-rolling

cpo@LR1.wue3# show protocols bgp
 local-as 200
 neighbor eth1 {
     address-family {
         ipv6-unicast {
         }
     }
     interface {
         v6only {
             remote-as 100
         }
     }
 }
 parameters {
     default {
         no-ipv4-unicast
     }
 }
cpo@LR1.wue3:~$ show bgp ipv6 summary

IPv6 Unicast Summary:
BGP router identifier 172.18.254.201, local AS number 200 vrf-id 0
BGP table version 2
RIB entries 3, using 576 bytes of memory
Peers 1, using 21 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
eth1            4        100        26        35        0    0    0 00:08:01            2        2

Total number of neighbors 1
cpo@LR1.wue3:~$ show ipv6 route
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

B>* 2001:db8:100::/48 [20/0] via fe80::250:56ff:feb3:cdba, eth1, weight 1, 00:04:18
B>* 2001:db8:200::/48 [20/0] via fe80::250:56ff:feb3:cdba, eth1, weight 1, 00:04:18
C * fe80::/64 is directly connected, eth1, 00:12:15
C * fe80::/64 is directly connected, eth1, 00:12:16
C * fe80::/64 is directly connected, eth0.201, 06:12:20
C * fe80::/64 is directly connected, eth0, 06:12:20
C>* fe80::/64 is directly connected, dum0, 06:12:23

Config VyOS 1.3.0-rc6

cpo@LR2.wue3# show protocols bgp
 bgp 100 {
     address-family {
         ipv6-unicast {
             network 2001:db8:100::/48 {
             }
             network 2001:db8:200::/48 {
             }
         }
     }
     neighbor eth1 {
         address-family {
             ipv6-unicast {
             }
         }
         interface {
             v6only {
                 remote-as 200
             }
         }
     }
 }
cpo@LR2.wue3# show protocols static route6
 route6 2001:db8:100::/48 {
     blackhole {
     }
 }
 route6 2001:db8:200::/48 {
     blackhole {
     }
 }
c-po added a comment.Sep 25 2021, 2:57 PM

The next rolling will also have support for the set protocols bgp neighbor fe80::202 interface source-interface 'eth1' CLI command

c-po mentioned this in rVYOSONEX993daec92965: bgp: T3657: add "neighbor fe80::202 interface source-interface 'eth1'" command.Sep 25 2021, 2:57 PM
c-po closed this task as Resolved.Sep 25 2021, 2:57 PM
c-po changed Is it a breaking change? from Stricter validation to Perfectly compatible.
c-po changed Issue type from Unspecified (please specify) to Feature (new functionality).
pedro mentioned this in T941: BGP neighbours with IPv6 link-local addresses.Feb 15 2022, 11:26 PM