Zone-based Firewalls on Bridges would flag related DHCP traffic invalid
In progress, HighPublicBUG
Actions

Assigned To

Authored By

	alainlamar
	Aug 11 2024, 9:26 AM

Description

Running DHCP on bridge interfaces would work fine, until zone-based firewalls are set up, incorporating a state-policy to drop (and log) invalid packages.

The Bug:

Returning DHCP offers are flagged invalid when they should pass the slaved interface to the outside. This is being observed for Ethernet slaves, as well as for WiFi interfaces as a slave. In the latter case, a very weird packet is observed which is interpreted by tcpdump to be IPX. Traces of traffic and logs follow below.

DHCP request from "the outside" to VyOS within a KVM virtual machine (eth1 slave to br0 at 192.168.123.182, config is attached):

config.support.gz1 KBDownload

vyos@vyos:~$ sudo tcpdump -nn -i any 'udp or icmp or arp'
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
09:03:21.674935 eth1  B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:a3:c2:e8, length 300
09:03:21.674935 br0   B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:a3:c2:e8, length 300
09:03:21.675670 br0   Out IP 192.168.123.182.67 > 192.168.123.128.68: BOOTP/DHCP, Reply, length 268
09:03:29.628686 eth1  B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:a3:c2:e8, length 300
09:03:29.628686 br0   B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:a3:c2:e8, length 300
09:03:29.629516 br0   Out IP 192.168.123.182.67 > 192.168.123.128.68: BOOTP/DHCP, Reply, length 268
09:03:29.938639 eth0  Out IP 192.168.122.182.42244 > 18.193.41.138.123: NTPv4, Client, length 48
09:03:29.973327 eth0  In  IP 18.193.41.138.123 > 192.168.122.182.42244: NTPv4, Server, length 48
09:03:38.803348 eth0  Out IP 192.168.122.182.60963 > 122.248.201.177.123: NTPv4, Client, length 48
09:03:39.119455 eth0  In  IP 122.248.201.177.123 > 192.168.122.182.60963: NTPv4, Server, length 48
09:03:44.345958 eth1  B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:a3:c2:e8, length 300
09:03:44.345958 br0   B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:a3:c2:e8, length 300
09:03:44.346801 br0   Out IP 192.168.123.182.67 > 192.168.123.128.68: BOOTP/DHCP, Reply, length 268
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel
vyos@vyos:~$

/var/log/messages for the same time frame:

Aug 11 09:03:18 vyos sudo:     vyos : TTY=pts/3 ; PWD=/home/vyos ; USER=root ; COMMAND=/usr/bin/tcpdump -nn -i any 'udp or icmp or arp'
Aug 11 09:03:21 vyos kea-dhcp4[5771]: 2024-08-11 09:03:21.675 INFO  [kea-dhcp4.leases/5771.139682525615808] DHCP4_LEASE_ADVERT [hwtype=1 52:54:00:a3:c2:e8], cid=[no info], tid=0xc207cc14: lease 192.168.123.128 will be advertised
Aug 11 09:03:21 vyos kernel: [ 1605.415822] [STATE-POLICY-INV-D]IN= OUT=eth1 SRC=192.168.123.182 DST=192.168.123.128 LEN=296 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=276 
Aug 11 09:03:29 vyos kernel: [ 1613.369591] [STATE-POLICY-INV-D]IN= OUT=eth1 SRC=192.168.123.182 DST=192.168.123.128 LEN=296 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=276 
Aug 11 09:03:29 vyos kea-dhcp4[5771]: 2024-08-11 09:03:29.629 INFO  [kea-dhcp4.leases/5771.139682542401216] DHCP4_LEASE_ADVERT [hwtype=1 52:54:00:a3:c2:e8], cid=[no info], tid=0xc207cc14: lease 192.168.123.128 will be advertised
Aug 11 09:03:44 vyos kea-dhcp4[5771]: 2024-08-11 09:03:44.346 INFO  [kea-dhcp4.leases/5771.139682550793920] DHCP4_LEASE_ADVERT [hwtype=1 52:54:00:a3:c2:e8], cid=[no info], tid=0xc207cc14: lease 192.168.123.128 will be advertised
Aug 11 09:03:44 vyos kernel: [ 1628.086816] [STATE-POLICY-INV-D]IN= OUT=eth1 SRC=192.168.123.182 DST=192.168.123.128 LEN=296 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=276

DHCP client (the outsider):

$ sudo dhclient -v virbr2
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/virbr2/52:54:00:a3:c2:e8
Sending on   LPF/virbr2/52:54:00:a3:c2:e8
Sending on   Socket/fallback
DHCPDISCOVER on virbr2 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on virbr2 to 255.255.255.255 port 67 interval 15
DHCPDISCOVER on virbr2 to 255.255.255.255 port 67 interval 14
^C

Below is a trace for a WiFi NIC on real hardware (slave to br0 at 10.66.66.254) - timestamps in CEST (UTC +0200)

reading from file wifi-bridge_dhcp.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144
Warning: interface names might be incorrect
    1  18:15:31.080223 wlan1 B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from a8:3b:76:ba:b0:19, length 300
    2  18:15:31.080223 br0   B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from a8:3b:76:ba:b0:19, length 300
    3  18:15:31.082635 br0   Out IP 10.66.66.254.67 > 10.66.66.64.68: BOOTP/DHCP, Reply, length 280
    4  18:15:31.834435 wlan1 Out IPX 009011ff.fe:80:00:00:00:00.0000 > a83b76ba.b0:19:86:dd:60:0a.0094: [length 0 < 30] (invalid)
    5  18:15:31.834519 wlan1 M   IP6 fe80::aa3b:76ff:feba:b019.5353 > ff02::fb.5353: 0*- [0q] 2/0/0 (Cache flush) PTR mars.local., (Cache flush) AAAA fe80::aa3b:76ff:feba:b019 (136)
    6  18:15:31.834519 br0   M   IP6 fe80::aa3b:76ff:feba:b019.5353 > ff02::fb.5353: 0*- [0q] 2/0/0 (Cache flush) PTR mars.local., (Cache flush) AAAA fe80::aa3b:76ff:feba:b019 (136)
    7  18:15:33.922423 wlan1 Out IPX 009011ff.fe:80:00:00:00:00.0000 > a83b76ba.b0:19:86:dd:60:0a.0094: [length 0 < 30] (invalid)
    8  18:15:33.922508 wlan1 M   IP6 fe80::aa3b:76ff:feba:b019.5353 > ff02::fb.5353: 0*- [0q] 2/0/0 (Cache flush) PTR mars.local., (Cache flush) AAAA fe80::aa3b:76ff:feba:b019 (136)
    9  18:15:33.922508 br0   M   IP6 fe80::aa3b:76ff:feba:b019.5353 > ff02::fb.5353: 0*- [0q] 2/0/0 (Cache flush) PTR mars.local., (Cache flush) AAAA fe80::aa3b:76ff:feba:b019 (136)
   10  18:15:34.093314 wlan1 Out IPX 00103aff.fe:80:00:00:00:00.0000 > a83b76ba.b0:19:86:dd:60:00.0000: [length 0 < 30] (invalid)
   11  18:15:34.093396 wlan1 M   IP6 fe80::aa3b:76ff:feba:b019 > ff02::2: ICMP6, router solicitation, length 16
   12  18:15:34.093396 br0   M   IP6 fe80::aa3b:76ff:feba:b019 > ff02::2: ICMP6, router solicitation, length 16
   13  18:15:42.038389 wlan1 Out IPX 00103aff.fe:80:00:00:00:00.0000 > a83b76ba.b0:19:86:dd:60:00.0000: [length 0 < 30] (invalid)
   14  18:15:42.038470 wlan1 M   IP6 fe80::aa3b:76ff:feba:b019 > ff02::2: ICMP6, router solicitation, length 16
   15  18:15:42.038470 br0   M   IP6 fe80::aa3b:76ff:feba:b019 > ff02::2: ICMP6, router solicitation, length 16
   16  18:15:46.862489 wlan1 Out IPX 00000000.80:11:39:96:00:00.0000 > a83b76ba.b0:19:08:00:45:10.0148: ipx-#148 65505
   17  18:15:46.862574 wlan1 B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from a8:3b:76:ba:b0:19, length 300
   18  18:15:46.862574 br0   B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from a8:3b:76:ba:b0:19, length 300
   19  18:15:46.865692 br0   Out IP 10.66.66.254.67 > 10.66.66.64.68: BOOTP/DHCP, Reply, length 280
   20  18:15:58.175261 wlan1 Out IPX 00000000.80:11:39:96:00:00.0000 > a83b76ba.b0:19:08:00:45:10.0148: ipx-#148 65505
   21  18:15:58.175346 wlan1 B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from a8:3b:76:ba:b0:19, length 300
   22  18:15:58.175346 br0   B   IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from a8:3b:76:ba:b0:19, length 300
   23  18:15:58.177709 br0   Out IP 10.66.66.254.67 > 10.66.66.64.68: BOOTP/DHCP, Reply, length 280
   24  18:15:59.957039 wlan1 Out IPX 00103aff.fe:80:00:00:00:00.0000 > a83b76ba.b0:19:86:dd:60:00.0000: [length 0 < 30] (invalid)

/var/log/messages for the same time frame - timestamps in UTC

Aug 10 16:15:27 wifi6e hostapd: wlan1: STA a8:3b:76:ba:b0:19 IEEE 802.11: associated (aid 1)
Aug 10 16:15:28 wifi6e hostapd: wlan1: STA a8:3b:76:ba:b0:19 RADIUS: starting accounting session B0E1A64687F3D895
Aug 10 16:15:28 wifi6e hostapd: wlan1: STA a8:3b:76:ba:b0:19 WPA: pairwise key handshake completed (RSN)
Aug 10 16:15:31 wifi6e kernel: [  343.866936] [STATE-POLICY-INV-D]IN= OUT=wlan1 SRC=10.66.66.254 DST=10.66.66.64 LEN=308 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=288 
Aug 10 16:15:31 wifi6e kea-dhcp4[3265]: 2024-08-10 16:15:31.082 INFO  [kea-dhcp4.leases/3265.140136406619840] DHCP4_LEASE_ADVERT [hwtype=1 a8:3b:76:ba:b0:19], cid=[no info], tid=0x1881157a: lease 10.66.66.64 will be advertised
Aug 10 16:15:46 wifi6e kernel: [  359.650007] [STATE-POLICY-INV-D]IN= OUT=wlan1 SRC=10.66.66.254 DST=10.66.66.64 LEN=308 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=288 
Aug 10 16:15:46 wifi6e kea-dhcp4[3265]: 2024-08-10 16:15:46.864 INFO  [kea-dhcp4.leases/3265.140136415012544] DHCP4_LEASE_ADVERT [hwtype=1 a8:3b:76:ba:b0:19], cid=[no info], tid=0x1881157a: lease 10.66.66.64 will be advertised
Aug 10 16:15:58 wifi6e kernel: [  370.962012] [STATE-POLICY-INV-D]IN= OUT=wlan1 SRC=10.66.66.254 DST=10.66.66.64 LEN=308 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=288 
Aug 10 16:15:58 wifi6e kea-dhcp4[3265]: 2024-08-10 16:15:58.177 INFO  [kea-dhcp4.leases/3265.140136389834432] DHCP4_LEASE_ADVERT [hwtype=1 a8:3b:76:ba:b0:19], cid=[no info], tid=0x1881157a: lease 10.66.66.64 will be advertised

Not-a-fix Workaround:

Accepting all invalid packages would render DHCP operational again. However, it is clear that this is not a solution, but may serve as a hint for debugging.

vyos@vyos# set firewall global-options state-policy invalid action accept 
[edit]
vyos@vyos# commit
[edit]
vyos@vyos#

Details

Difficulty level: Unknown (require assessment)
Version: 1.5-rolling-202408100021 (nightly)
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Bug (incorrect behavior)

Event Timeline

alainlamar created this task.Aug 11 2024, 9:26 AM

alainlamar updated the task description. (Show Details)Aug 11 2024, 9:30 AM

pasik subscribed.Aug 11 2024, 9:31 AM

alainlamar updated the task description. (Show Details)Aug 11 2024, 9:37 AM

alainlamar updated the task description. (Show Details)

From IRC, could this be related?

https://www.suse.com/support/kb/doc/?id=000018589

Some kind of offloading is enabled which causes the CRC checksums to be calculated incorrectly and because of this causing nftables to think that the packet is invalid?

dmbaturin triaged this task as High priority.Aug 12 2024, 7:29 AM

Config to reproduce the issue:

set firewall global-options state-policy invalid action 'drop'
set firewall global-options state-policy invalid log
set interfaces bridge br0 address '192.168.150.1/24'
set interfaces bridge br0 member interface eth1
set interfaces bridge br0 member interface eth2

set service dhcp-server shared-network-name BRI subnet 192.168.150.0/24 option default-router '192.168.150.1'
set service dhcp-server shared-network-name BRI subnet 192.168.150.0/24 option name-server '8.8.8.8'
set service dhcp-server shared-network-name BRI subnet 192.168.150.0/24 range 1 start '192.168.150.201'
set service dhcp-server shared-network-name BRI subnet 192.168.150.0/24 range 1 stop '192.168.150.210'
set service dhcp-server shared-network-name BRI subnet 192.168.150.0/24 subnet-id '150'

Logs:

Aug 12 10:59:25 vyos-bri kernel: [STATE-POLICY-INV-D]IN= OUT=eth1 SRC=192.168.150.1 DST=192.168.150.201 LEN=317 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=297 
Aug 12 10:59:26 vyos-bri kea-dhcp4[2499]: 2024-08-12 10:59:26.481 INFO  [kea-dhcp4.leases/2499.140330684389056] DHCP4_LEASE_ADVERT [hwtype=1 00:50:79:66:68:02], cid=[01:00:50:79:66:68:02], tid=0xf3631f3d: lease 192.168.150.201 will be advertised
Aug 12 10:59:26 vyos-bri kernel: [STATE-POLICY-INV-D]IN= OUT=eth1 SRC=192.168.150.1 DST=192.168.150.201 LEN=317 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=297 
Aug 12 10:59:29 vyos-bri kea-dhcp4[2499]: 2024-08-12 10:59:29.482 INFO  [kea-dhcp4.leases/2499.140330684389056] DHCP4_LEASE_ADVERT [hwtype=1 00:50:79:66:68:02], cid=[01:00:50:79:66:68:02], tid=0xf3631f3d: lease 192.168.150.201 will be advertised
Aug 12 10:59:29 vyos-bri kernel: [STATE-POLICY-INV-D]IN= OUT=eth1 SRC=192.168.150.1 DST=192.168.150.201 LEN=317 TOS=0x10 PREC=0x00 TTL=128 ID=0 DF PROTO=UDP SPT=67 DPT=68 LEN=297

And bridge table:

vyos@vyos-bri# sudo nft list table bridge vyos_filter
table bridge vyos_filter {
        chain VYOS_FORWARD_filter {
                type filter hook forward priority filter; policy accept;
                jump VYOS_STATE_POLICY
                counter packets 3 bytes 1176 accept comment "FWD-filter default-action accept"
        }

        chain VYOS_INPUT_filter {
                type filter hook input priority filter; policy accept;
                jump VYOS_STATE_POLICY
                counter packets 3 bytes 1176 accept comment "INP-filter default-action accept"
        }

        chain VYOS_OUTUT_filter {
                type filter hook output priority filter; policy accept;
                jump VYOS_STATE_POLICY
                counter packets 4 bytes 384 accept comment "OUT-filter default-action accept"
        }

        chain VYOS_PREROUTING_filter {
                type filter hook prerouting priority filter; policy accept;
                counter packets 3 bytes 1176 accept comment "PRE-filter default-action accept"
        }

        chain VYOS_STATE_POLICY {
                ct state invalid log prefix "[STATE-POLICY-INV-D]" counter packets 3 bytes 951 drop
                return
        }
}
[edit]
vyos@vyos-bri#

n.fort changed the task status from Confirmed to In progress.Aug 16 2024, 5:44 PM

n.fort claimed this task.

PR: https://github.com/vyos/vyos-1x/pull/4018

dmbaturin added a project: Restricted Project.Sun, Sep 15, 5:03 PM

Zone-based Firewalls on Bridges would flag related DHCP traffic invalidIn progress, HighPublicBUGActions

Description

Details

Event Timeline

Zone-based Firewalls on Bridges would flag related DHCP traffic invalid
In progress, HighPublicBUG
Actions