Page MenuHomeVyOS Platform

container: switch to netavark network stack
Closed, ResolvedPublicFEATURE REQUEST

Description

https://www.redhat.com/sysadmin/podman-new-network-stack

  • Better IPv6 support
  • Improved support for containers in multiple networks
  • Improved performance

As of now the IPv6 addressing of a container is automatic - lets hope we can assign fixed IPv6 addresses to containers.
It would also be nice to have containers reside in multiple networks - maybe even within a VRF if that's supported

Details

Version
-
Is it a breaking change?
Perfectly compatible

Event Timeline

c-po claimed this task.
c-po moved this task from Open to Finished on the VyOS 1.4 Sagitta board.

Hi guys,

found the bug report as well. I'm running the latest 1.4 rolling release from yesterday: 1.4-rolling-202304231929

VyOS container configuration:

mkdir -p /config/containers/adguard-home/{conf,work}

add container image adguard/adguardhome
set container network default-net prefix '10.170.7.0/24'
set container network default-net description 'default container network for podman'

set container name adguard-home image 'adguard/adguardhome'
set container name adguard-home network default-net address '10.170.7.53'
set container name adguard-home volume persistence destination '/opt/adguardhome/work'
set container name adguard-home volume persistence source '/config/containers/adguard-home/work'
set container name adguard-home volume persistence2 destination '/opt/adguardhome/conf'
set container name adguard-home volume persistence2 source '/config/containers/adguard-home/conf'

Wanted to add a little bit of context: the error message podman reports is the following:

Apr 24 09:14:27 gw-vpn.bap.rhr.de podman[4792]: 2023-04-24 09:14:27.016390827 +0200 CEST m=+0.171489223 container create ec28096c2d944e1be6695212ca4a23bf86d56c5ee2b5658ce1c08463f46cae5a (image=docker.io/adguard/adguardhome:latest, name=adguard-home, org.opencontainers.image.vers>
Apr 24 09:14:27 gw-vpn.bap.rhr.de podman[4792]: 2023-04-24 09:14:26.88163034 +0200 CEST m=+0.036728724 image pull  adguard/adguardhome
Apr 24 09:14:27 gw-vpn.bap.rhr.de podman[4792]: Error: netavark: No such file or directory (os error 2)

I stumbled across the following issue in the podman repository which basically says it is missing the iptables package dependency: https://github.com/containers/podman/issues/16958

I built a custom image with iptables package added and got the container running successfully using the following build commands:

export VYOS_BRANCH=current
git clone -b $VYOS_BRANCH --single-branch https://github.com/vyos/vyos-build
cd vyos-build
export VYOS_VERSION=$(jq -r ".${VYOS_BRANCH}" < data/versions)
./build-vyos-image \
  --architecture amd64 \
  --build-by xxx \
  --build-type release \
  --build-comment "VyOS with packages ready for prometheus node exporter and ansible" \
  --version "${VYOS_VERSION}-rolling-$(date +%Y-%m-%d-%H-%M)" \
  --custom-package git \
  --custom-package ansible \
  --custom-package iptables \
  iso

I'm not sure if this is a suitable way of "fixing" the problem since VyOS 1.4 seems to be using nftables under the hood and the netavark repository also states

Support for iptables and firewalld at present, with support for nftables planned in a future release

Hope this helps, let me know if I can support with more information.

Regards
Sebastian

Hi @crazylittle,

can you add some more detials? I just used your above container config and upgraded from a VyOS version that came with CNI to a version with netavark and I do not see that error.
netavark was added 2023-04-02.

I updated from 1.4-rolling-202303280317 -> 1.4-rolling-202304280615 and all went good using your above adguard container.

c-po reopened this task as Needs testing.Apr 28 2023, 4:34 PM

Hi @c-po,

I updated one of my servers to the latest rolling:

sfinke@gw-vpn.bap.rhr.de:~$ show version
Version:          VyOS 1.4-rolling-202304280615           <-- 28.04.2023
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Fri 28 Apr 2023 06:15 UTC
Build UUID:       65a4823e-a418-4850-83aa-5c6e98fcc2f8
Build commit ID:  cdb6d84e33ca42

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-42 05 8d 28 42 f0 35 64-0b 22 92 b1 bf 43 ef 11
Hardware UUID:    288d0542-f042-6435-0b22-92b1bf43ef11

Copyright:        VyOS maintainers and contributors

And indeed the container is starting there!

However, the iptables package is installed there now.

sfinke@gw-vpn.bap.rhr.de:~$ dpkg -l | grep iptables
ii  iptables                             1.8.9-2                          amd64        administration tools for packet filtering and NAT

sfinke@gw-vpn.bap.rhr.de:~$ aptitude why iptables
i   vyos-1x-smoketest               Depends  skopeo
i A skopeo                          Depends  golang-github-containers-common (> 0.49)
i A golang-github-containers-common Depends  container-network-stack
i A containernetworking-plugins     Provides container-network-stack
i A containernetworking-plugins     Depends  iptables

sfinke@gw-vpn.bap.rhr.de:~$ dpkg -l | grep smoke
ii  vyos-1x-smoketest                    1.4dev0-4328-g78a9eaaf8          all          VyOS build sanity checking toolkit

Looking at my server with the "older" rolling running:

sfinke@gw-vpn.anx84.nue.de:~$ show version
Version:          VyOS 1.4-rolling-202304231929            <-- 23.04.2023
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Sun 23 Apr 2023 19:29 UTC
Build UUID:       4b72573d-dff0-4771-a78f-aab0639ddf0d
Build commit ID:  25cf22e2ebcf2f

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  netcup
Hardware model:   KVM Server
Hardware S/N:
Hardware UUID:    eda87b59-b85a-4069-9a08-c051ffad1217

Copyright:        VyOS maintainers and contributors
sfinke@gw-vpn.anx84.nue.de:~$ dpkg -l | grep iptables
sfinke@gw-vpn.anx84.nue.de:~$ dpkg -l | grep smoke

No iptables installed, and also no vyos-1x-smoketest package.

Did something change with the build process in the meantime?

I want to describe my issues but I am not able to do so very easily since I don't have ipmi on my router.

I upgraded to from 1.4-rolling-202303310306 to 1.4-rolling-202304280615

My truncated configuration

set container network containers description 'Network for VyOS containers'
set container network containers prefix '192.168.254.0/24'
set container name coredns cap-add 'net-bind-service'
set container name coredns image 'registry.k8s.io/coredns/coredns:v1.10.1'
set container name coredns memory '0'
set container name coredns network containers address '192.168.254.2'
set container name coredns shared-memory '0'
set container name coredns volume coredns-corefile destination '/Corefile'
set container name coredns volume coredns-corefile mode 'ro'
set container name coredns volume coredns-corefile source '/config/containers/coredns/config/Corefile'
set container name coredns volume coredns-hosts destination '/config/hosts'
set container name coredns volume coredns-hosts mode 'ro'
set container name coredns volume coredns-hosts source '/config/containers/coredns/config/hosts'

set firewall name containers-lan default-action 'accept'
set firewall name containers-lan description 'From CONTAINERS to LAN'
set firewall name containers-local default-action 'accept'
set firewall name containers-local description 'From CONTAINERS to LOCAL'
set firewall name containers-wan default-action 'accept'
set firewall name containers-wan description 'From CONTAINERS to WAN'
set firewall name lan-containers default-action 'accept'
set firewall name lan-containers description 'From LAN to CONTAINERS'
set firewall name local-containers default-action 'accept'
set firewall name local-containers description 'From LOCAL to CONTAINERS'
set firewall name wan-containers default-action 'drop'
set firewall name wan-containers description 'From WAN to CONTAINERS'
set firewall name wan-containers enable-default-log
set firewall zone containers default-action 'drop'
set firewall zone containers description 'VyOS containers zone'
set firewall zone containers from lan firewall name 'lan-containers'
set firewall zone containers from local firewall name 'local-containers'
set firewall zone containers from wan firewall name 'wan-containers'
set firewall zone containers interface 'cni-containers'
set firewall zone lan from containers firewall name 'containers-lan'
set firewall zone local from containers firewall name 'containers-local'
set firewall zone wan from containers firewall name 'containers-wan'

set service dns forwarding allow-from '192.168.0.0/24'
set service dns forwarding cache-size '0'
set service dns forwarding listen-address '192.168.0.1'
set service dns forwarding name-server 1.0.0.1
set service dns forwarding name-server 1.1.1.1
set service dns forwarding name-server 8.8.4.4
set service dns forwarding name-server 8.8.8.8
set service dhcp-server shared-network-name LAN subnet 192.168.0.0/24 name-server '192.168.254.2'

set system name-server '192.168.254.2'

On 1.4-rolling-202303310306 (old):

I can PING and TRACEROUTE any IP on my containers network, e.g. my coredns container 192.168.254.2 from Vyos iteslf or any network that is allowed.

On 1.4-rolling-202304280615 (new):

I cannot PING and TRACEROUTE any IP on my containers network. I can only PING and TRACEROUTE 192.168.254.1 and not 192.168.254.2 from Vyos itself. So it appears that Vyos does not have access to my containers network anymore.

In either case when trying to PING or TRACEROUTE from a device on my LAN network I can PING and TRACEROUTE 192.168.254.2.

Since coredns is my DNS server this pretty much breaks my network :)

I hope this helps a bit.

No iptables installed, and also no vyos-1x-smoketest package.

We've released one rolling without the smoketest package to have a fresher one- That image contains a corener case bug where BGP EVPN instances can not removed if the l3vni still exists (already fixed in the latest rolling), that's why smoketest package is missing

Did something change with the build process in the meantime?

As of netavark and podman - there was also a dependency issue I remember some weeks back which will most likely fixed the issue

In either case when trying to PING or TRACEROUTE from a device on my LAN network I can PING and TRACEROUTE 192.168.254.2.

So with CNI it was possible to directly ping the container from VyOS. With netavark you can only ping from any other machine the container but not from VyOS itself?

Wouldn't it make sense to add iptables as a direct dependency then? Looking back at Debian Bullseye, iptables was still a direct dependency to the podman package. But as of Debian Bookworm, iptables is just a suggested package to podman.

root@bullseye:/# apt-cache depends podman | grep iptables
  Depends: iptables

root@bookworm:/# apt-cache depends podman | grep iptables
  Suggests: iptables

@onedr0p

set container network containers description 'Network for VyOS containers'
set container network containers prefix '192.168.254.0/24'
set firewall zone containers interface 'cni-containers'

Check the interface name. The container interface is called pod-containers!

@carazzim0 good find, I updated that and now everything appears to be working again!

This comment was removed by dcplaya.

If I pre-load my zone firewall with the new interface format (pod-$containerName) and upgrade to vyos-1.4-rolling-202304290647, it seems to upgrade seamlessly

@dcplaya yeah that was a transitional error/quirk as we renamed that one during the development cycle.

Is it good now? Can we close this or are there more issues?

@c-po I know we're on rolling for 1.4 but is there any way to communicate breaking changes like this in the future? If this was mentioned in the PR or ticket I don't think there would have been any confusion.

As for this tickets status it can be closed.

In T5082#148096, @onedr0p wrote:

@c-po I know we're on rolling for 1.4 but is there any way to communicate breaking changes like this in the future? If this was mentioned in the PR or ticket I don't think there would have been any confusion.

As for this tickets status it can be closed.

I was actually not thinking that someone smart enough to assign the cni podman network to a firewall. If I would have thought about that, I would have programmed a migration script automatically taking care about it.

Sorry for the disturbance. Usually breaking changes are avoided as much as possible, because it will also reduce the amount of tickets here - self protection ;)

This might actually beat us in an upgrade form 1.3 -> 1.4 in the future so I will align the pod network in the 1.3 release, too so this won't become an issue.