Page MenuHomeVyOS Platform
Create Task
Maniphest T3748

Container deletion bug
Closed, ResolvedPublicBUG
Actions

Description

Continuing from the bug in T3747

admin@edge# delete container name 
[edit]
admin@edge# delete container network 
[edit]
admin@edge# set container network dnsnet prefix 10.0.72.0/24
[edit]
admin@edge# set container name dns02 image ubuntu:focal
[edit]
admin@edge# set container name dns02 network dnsnet address 10.0.72.253
[edit]
admin@edge# commit

Leads to:

[ container ]
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Contact us using the online help desk if you have a subscription:
  https://support.vyos.io/
- Make sure you are running the latest version of VyOS available at:
  https://vyos.net/get/
- Consult the community forum to see how to handle this issue:
  https://forum.vyos.io
- Join us on Slack where our users exchange help and advice:
  https://vyos.slack.com

When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your 
  business policy requires it)
- and include all the information presented below

Report Time:      2021-08-12 19:28:13
Image Version:    VyOS 1.4-kroy-202108092020
Release Train:    sagitta

Built by:         kroy
Built on:         Mon 09 Aug 2021 20:20 UTC
Build UUID:       d3a7abb0-afe2-4bc7-b8b1-fe700e5bcc4e
Build Commit ID:  dc5e42b9daa97a

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Micro-Star International Co., Ltd.
Hardware model:   MS-7B79
Hardware S/N:     To be filled by O.E.M.
Hardware UUID:    7b297d94-c648-7a1d-acb6-2cf05d23dcbc

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/containers.py", line 267, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/containers.py", line 193, in apply
    _cmd(f'podman network rm {network}')
  File "/usr/libexec/vyos/conf_mode/containers.py", line 42, in _cmd
    return cmd(command)
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
FileNotFoundError: [Errno 2] failed to run command: podman network rm container_net
returned: 
exit code: 2

noteworthy:
cmd 'podman network rm container_net'
returned (out):

returned (err):
Error: "container_net" has associated containers with it. Use -f to forcibly delete containers and pods: network is being used

[[container]] failed
Commit failed
[edit]

Which happens even with solely this config:

admin@edge# show container
-network container_net {
-    prefix 10.0.72.0/24
-}

Details

Difficulty level
Unknown (require assessment)
Version
1.4-rolling
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)

Related Objects

Event Timeline

kroy created this task.Aug 12 2021, 7:36 PM
kroy added a comment.Aug 12 2021, 7:40 PM

To add to this, it looks like I'm not going to be able to get rid of that without rebooting:

admin@edge# set container network container_net prefix 10.0.72.0/24
[edit]
admin@edge# commit
save
[edit]
admin@edge# save
Saving configuration to '/config/config.boot'...
Done
[edit]
admin@edge# show ne

  Configuration path: [ne] is not valid
      
[edit]
admin@edge# show container 
 network container_net {
     prefix 10.0.72.0/24
 }
[edit]
admin@edge# delete container 
[edit]
admin@edge# commit
[ container ]
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Contact us using the online help desk if you have a subscription:
  https://support.vyos.io/
- Make sure you are running the latest version of VyOS available at:
  https://vyos.net/get/
- Consult the community forum to see how to handle this issue:
  https://forum.vyos.io
- Join us on Slack where our users exchange help and advice:
  https://vyos.slack.com

When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your 
  business policy requires it)
- and include all the information presented below

Report Time:      2021-08-12 19:39:15
Image Version:    VyOS 1.4-kroy-202108092020
Release Train:    sagitta

Built by:         kroy
Built on:         Mon 09 Aug 2021 20:20 UTC
Build UUID:       d3a7abb0-afe2-4bc7-b8b1-fe700e5bcc4e
Build Commit ID:  dc5e42b9daa97a

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Micro-Star International Co., Ltd.
Hardware model:   MS-7B79
Hardware S/N:     To be filled by O.E.M.
Hardware UUID:    7b297d94-c648-7a1d-acb6-2cf05d23dcbc

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/containers.py", line 267, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/containers.py", line 193, in apply
    _cmd(f'podman network rm {network}')
  File "/usr/libexec/vyos/conf_mode/containers.py", line 42, in _cmd
    return cmd(command)
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
PermissionError: [Errno 1] failed to run command: podman network rm container_net
returned: 
exit code: 1

noteworthy:
cmd 'podman network rm container_net'
returned (out):

returned (err):
Error: unable to find network configuration for container_net: network not found

delete [ container ] failed
Commit failed
[edit]
kroy added a comment.Aug 12 2021, 7:42 PM

It appears things are in such a state where that network doesn't actually exist:

admin@edge# show container 
 network container_net {
     prefix 10.0.72.0/24
 }
[edit]
admin@edge# sudo podman network ls
NAME    VERSION  PLUGINS
[edit]
Viacheslav added a subscriber: Viacheslav.Aug 13 2021, 5:28 AM

It seems impossible to delete network and container and add a new one in one commit

Error: "container_net" has associated containers with it. Use -f to forcibly delete containers and pods: network is being used
pasik added a subscriber: pasik.Aug 13 2021, 9:18 AM
c-po added a subscriber: c-po.Aug 13 2021, 10:31 AM

@Viacheslav we have other "objects" in VyOS which also require deletion first under certain circumstances.

L2TPv3 e.g. is always deleted first before we change anything.
https://github.com/vyos/vyos-1x/blob/current/src/conf_mode/interfaces-l2tpv3.py#L84-L97

On regular tunnel interfaces, we also sometimes delete the tunnel fist and then recreate it, if changing a parameter is not supported by Linux:
https://github.com/vyos/vyos-1x/blob/current/src/conf_mode/interfaces-tunnel.py#L123-L146

kroy added a comment.Aug 25 2021, 11:00 PM

Looks good to me now

kroy added a comment.EditedAug 25 2021, 11:17 PM

I found a new and exciting way to break it though. And maybe what we were talking about in Slack will be able to correct some of this.

The takeaway here is it's not overly difficult to get the config and the filesystem in an inconsistent state if anything goes wrong.

Here's the transcript:

transcript.txt3 KBDownload

Viacheslav closed this task as Resolved.Oct 15 2021, 2:18 PM
Viacheslav claimed this task.

Initial bug was Fixed, VyOS 1.4-rolling-202110130217

vyos@r1-roll# compare 
[edit container]
+name dns02 {
+    image ubuntu:focal
+    network dnsnet {
+        address 10.0.72.253
+    }
+}
+network dnsnet {
+    prefix 10.0.72.0/24
+}
-network net01 {
-    prefix 10.0.72.0/24
-}

Commit

vyos@r1-roll# commit
[ container ]
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/ubuntu:focal...
Getting image source signatures
Copying blob sha256:f3ef4ff62e0da0ef761ec1c8a578f3035bef51043e53ae1b13a20b3e03726d17
Copying config sha256:597ce1600cf4ac5f449b66e75e840657bb53864434d6bd82f00b172544c32ee2
Writing manifest to image destination
Storing signatures
597ce1600cf4ac5f449b66e75e840657bb53864434d6bd82f00b172544c32ee2
0

[edit]
vyos@r1-roll#