Page MenuHomeVyOS Platform

Podman NAT/NAT66 conflicts with PBR
Open, NormalPublicBUG

Description

When attaching a network to containers, PBR table rules will be NAT'ed (both ipv4 and ipv6). The issue seems to be the Podman mark rules conflicting with PBR, see forum thread for more details. Removing the network from containers and only use allow-host-networks will remove the conflicting NAT rules after reboot.

Details

Version
2024.05.06-latest
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)
Forum thread
https://forum.vyos.io/t/policy-route-table-selection-causing-nat/14365/9

Event Timeline

c-po triaged this task as Normal priority.Sep 6 2024, 5:30 AM
c-po added a project: VyOS Rolling.

I believe I am hitting a related PBR/fwmark/container interaction issue on VyOS rolling.

Environment:

  • VyOS: 20260505-dev
  • This configuration previously worked on April 2026 rolling images
  • Router has PBR routing selected LAN clients through OpenVPN vtun10
  • Tailscale runs as a VyOS container with host networking, NET_ADMIN/NET_RAW, advertise-exit-node, and advertised routes
  • The issue appears when Tailscale/container netfilter handling is enabled
  • Disabling the Tailscale container stops the issue
  • Running Tailscale with --stateful-filtering=false --netfilter-mode=off also stops the issue

Relevant PBR config:
set policy route UK_VPN_CLIENTS interface eth1
set policy route UK_VPN_CLIENTS rule 10 source group address-group UK_VPN_CLIENTS
set policy route UK_VPN_CLIENTS rule 10 destination address !172.16.0.0/12
set policy route UK_VPN_CLIENTS rule 10 set table 100

Table 100:
set protocols static table 100 route 0.0.0.0/0 interface vtun10
set protocols static table 100 route 172.19.0.0/24 interface eth1

Observed RPDB rule:
100: from all fwmark 0x7fffff9b lookup UK_VPN

This appears to match VyOS's documented/generated table mark behaviour:
0x7fffffff - 100 = 0x7fffff9b

Expected route lookup works:
ip route get 1.1.1.1 from 172.19.0.204 iif eth1 mark 0x7fffff9b
=> 1.1.1.1 dev vtun10 table UK_VPN mark 0x7fffff9b

However, live packets dropped by the killswitch are logged with a different mark:
IN=eth1 OUT=eth0 SRC=172.19.0.204 DST=1.1.1.1 PROTO=ICMP MARK=0x7f00ff9b

Route lookup using the live mark goes to the wrong table/interface:
ip route get 1.1.1.1 from 172.19.0.204 iif eth1 mark 0x7f00ff9b
=> via 199.172.214.1 dev eth0

Temporary workaround confirms the mismatch:
ip rule add pref 90 fwmark 0x7f00ff9b/0xffffffff lookup UK_VPN

After adding that rule, traffic correctly uses vtun10.

Additional observation:
The difference between the expected mark and the live mark is the 0x00ff0000 byte:

Expected:
0x7fffff9b

Live packet:
0x7f00ff9b

This is consistent with something in the container/Tailscale/Podman netfilter path clearing or overwriting the 0x00ff0000 byte. This also matches the class of issue described here, where container/Podman mark handling conflicts with PBR.

Also observed:

  • Disabling the Tailscale container stops the issue.
  • Running Tailscale with --stateful-filtering=false --netfilter-mode=off stops the issue.
  • Rolling Tailscale back to v1.96.x did not by itself fix it, so this does not appear to be solely the Tailscale 1.98 src_valid_mark change.
  • net.ipv4.conf.all.src_valid_mark = 0 while reproducing.
  • With Tailscale disabled, or with Tailscale netfilter disabled, traffic goes through vtun10 as expected.
  • The issue was not seen on April 2026 rolling images with the same general configuration, but is reproducible on 20260505-dev. I have not isolated the exact VyOS commit that changed the behaviour.

This looks like a collision between VyOS-generated PBR table marks and container/Tailscale/Podman netfilter mark handling.

Questions:

  1. Is this the same root issue as T6689, or should this be tracked separately as a PBR live-packet mark vs RPDB mark mismatch?
  2. If it is the same issue, is the expected fix to reserve a non-overlapping mark namespace for VyOS PBR and container/Podman rules?
  3. Was there a change between the April 2026 rolling images and 20260505-dev that changed container, nftables, Podman, NAT, or PBR mark handling?