Page MenuHomeVyOS Platform
Create Task
Maniphest T5889

Migration NAT 5-to-6 bug
Closed, ResolvedPublicBUG
Actions

Description

Hi,

This is my cloud-init user-data

#cloud-config
vyos_config_commands:
  - set nat source rule 100 outbound-interface name 'eth0'
  - set nat source rule 100 translation address 'masquerade'

This is the output from cloud-init.log

2024-01-03 00:15:22,454 - cc_vyos_userdata.py[DEBUG]: Configuring command: "set nat source rule 100 outbound-interface name 'eth0'"
2024-01-03 00:15:22,455 - cc_vyos_userdata.py[DEBUG]: Marking node as tag: "['nat', 'source', 'rule']"
2024-01-03 00:15:22,455 - cc_vyos_userdata.py[DEBUG]: Configuring command: "set nat source rule 100 translation address 'masquerade'"
2024-01-03 00:15:22,455 - cc_vyos_userdata.py[DEBUG]: Marking node as tag: "['nat', 'source', 'rule']"
2024-01-03 00:15:22,455 - cc_vyos_userdata.py[DEBUG]: Configuration file saved: /opt/vyatta/etc/config/config.boot
2024-01-03 00:15:22,455 - handlers.py[DEBUG]: finish: modules-config/config-vyos_userdata: SUCCESS: config-vyos_userdata ran successfully
2024-01-03 00:15:22,456 - main.py[DEBUG]: Ran 3 modules with 0 failures

However the config.boot does not process the interface name correctly and fails to load as a result. This is the relevant config created in config.boot, note it is missing the interface name.

interfaces {
    ethernet eth0 {
        address "dhcp"
        hw-id "fa:16:3e:33:ab:24"
        mtu "1500"
        offload {
            gro
            gso
            sg
            tso
        }
    }
}
nat {
    source {
        rule 100 {
            outbound-interface {
                name ""
            }
            translation {
                address "masquerade"
            }
        }
    }
}

As you can see eth0 exists.

Thanks

Daniel

Details

Difficulty level
Unknown (require assessment)
Version
latest rolling release
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Unspecified (please specify)

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenBUGzsdcT5907 cloud-init root task for 1.5 and 1.4
ResolvedBUGViacheslavT5889 Migration NAT 5-to-6 bug

Event Timeline

dniasoff created this task.Jan 3 2024, 12:27 AM
dniasoff added a comment.Jan 3 2024, 12:32 AM

If I connect to the router and issue the commands

conf
load
set nat source rule 100 outbound-interface name eth0
commit

Then the issue is resolved and the config load successfully

pasik added a subscriber: pasik.Jan 3 2024, 7:25 AM
Viacheslav added a subscriber: Viacheslav.Jan 3 2024, 2:57 PM

Which version?

Viacheslav added a comment.Jan 5 2024, 6:05 PM

It seems issues with migrations

[  OK  ] Reached target cloud-init.target - Cloud-init target.
[   28.675603] vyos-router[1767]: Waiting for NICs to settle down: settled in 0sec..
[   31.572854] vyos-router[1767]: Mounting VyOS Config...done.
[   42.670126] vyos-router[1767]: Starting VyOS router: migrate configure failed!
[   43.268753] vyos-config[1773]: Configuration error
Viacheslav added a comment.EditedJan 8 2024, 9:00 AM

Ok the file does not have the Release version in config and migration do anyway
https://github.com/vyos/vyos-vm-images/blob/current/roles/install-config/templates/config.boot.j2

vyos@ci-router1# cat /config/config.boot.2024-01-08-083418.pre-migration 
nat {
    source {
        rule 100 {
            translation {
                address "masquerade"
            }
            outbound-interface {
                name "eth0"
            }
        }
    }
}
interfaces {
    ethernet eth0 {
        address "dhcp"
        hw-id "52:54:00:ff:97:48"
        mtu "1500"
    }
    loopback lo {
    }
}
service {
    ssh {
        client-keepalive-interval "180"
        port "22"
    }
}
system {
    config-management {
        commit-revisions "100"
    }
    host-name "ci-router1"
    login {
        user vyos {
            authentication {
                encrypted-password "*"
                plaintext-password "vyos"
            }
        }
    }
    ntp {
        server "time1.vyos.net"
        server "time2.vyos.net"
        server "time3.vyos.net"
    }
    syslog {
        global {
            facility all {
                level "notice"
            }
            facility protocols {
                level "debug"
            }
        }
    }
}

[edit]
vyos@ci-router1#

If we load this file we will get broken NAT

vyos@ci-router1# compare 
[system login user vyos authentication]
- encrypted-password "$6$rounds=656000$nJOga2IJ0SU3My7v$jLE8TBE7VfkXS2mFY4TanoQo6cXEsfHn.1eN/m3RZSpctUV4CCPPfqn.SVWoCaVAwqTqY7NdIhBDfmiB27hTJ."
+ encrypted-password "*"
+ plaintext-password "vyos"
[]
+ nat {
+     source {
+         rule 100 {
+             outbound-interface {
+                 name ""
+             }
+             translation {
+                 address "masquerade"
+             }
+         }
+     }
+ }
Viacheslav added a parent task: T5907: cloud-init root task for 1.5 and 1.4 .Jan 8 2024, 9:13 AM
dniasoff added a comment.Jan 8 2024, 11:42 AM

Thanks Viacheslav

jestabro added a subscriber: jestabro.EditedJan 15 2024, 6:00 PM

I think that this should be addressed at the specific migration script:
https://github.com/vyos/vyos-1x/blob/current/src/migration-scripts/nat/5-to-6
Although there are open questions about avoiding migration, and specific ways to avoid it (boot flag 'no-vyos-migrate'), neither are the issue here: migration scripts should 'do no harm', and one invariant that can be enforced is that migration scripts should be idempotent; the above is not, as a result of:
https://github.com/vyos/vyos-1x/blob/current/src/migration-scripts/nat/5-to-6#L53-L55
I suggest the patch below to restore idempotency (which clearly can't break anything, since one would not have an empty string as inbound/outbound-interface); thoughts ?

diff --git a/src/migration-scripts/nat/5-to-6 b/src/migration-scripts/nat/5-to-6
index de3830582..c83b93d84 100755
--- a/src/migration-scripts/nat/5-to-6
+++ b/src/migration-scripts/nat/5-to-6
@@ -51,8 +51,9 @@ for direction in ['source', 'destination']:
         for iface in ['inbound-interface','outbound-interface']:
             if config.exists(base + [iface]):
                 tmp = config.return_value(base + [iface])
-                config.delete(base + [iface])
-                config.set(base + [iface, 'interface-name'], value=tmp)
+                if tmp:
+                    config.delete(base + [iface])
+                    config.set(base + [iface, 'interface-name'], value=tmp)
jestabro added a comment.Jan 15 2024, 6:03 PM

(edited above for formatting)

Viacheslav renamed this task from Cloud-Init not processing user data correctly to Cloud-Init not processing user data correctly NAT.Jan 16 2024, 11:49 AM
Viacheslav renamed this task from Cloud-Init not processing user data correctly NAT to Migration NAT 5-to-6 bug.Jan 16 2024, 12:43 PM
Viacheslav changed the task status from Open to In progress.Jan 16 2024, 12:59 PM
Viacheslav claimed this task.

PR https://github.com/vyos/vyos-1x/pull/2833

Viacheslav changed the task status from In progress to Needs testing.Jan 16 2024, 2:21 PM
Viacheslav closed this task as Resolved.Jan 16 2024, 3:12 PM
Viacheslav moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.

Works fine

vyos@ci-router1:~$ cat /var/log/cloud-init-output.log 
Cloud-init v. 22.1-437-g3e026561-1~bddeb running 'init-local' at Tue, 16 Jan 2024 15:08:04 +0000. Up 17.93 seconds.
Cloud-init v. 22.1-437-g3e026561-1~bddeb running 'init' at Tue, 16 Jan 2024 15:08:09 +0000. Up 21.96 seconds.
ci-info: ++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------+---------------+--------+-------------------+
ci-info: | Device |  Up  |     Address     |      Mask     | Scope  |     Hw-Address    |
ci-info: +--------+------+-----------------+---------------+--------+-------------------+
ci-info: |  eth0  | True | 192.168.122.172 | 255.255.255.0 | global | 52:54:00:e3:4f:82 |
ci-info: |   lo   | True |    127.0.0.1    |   255.0.0.0   |  host  |         .         |
ci-info: |   lo   | True |     ::1/128     |       .       |  host  |         .         |
ci-info: +--------+------+-----------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
ci-info: +-------+---------------+---------------+---------------+-----------+-------+
ci-info: | Route |  Destination  |    Gateway    |    Genmask    | Interface | Flags |
ci-info: +-------+---------------+---------------+---------------+-----------+-------+
ci-info: |   0   |    0.0.0.0    | 192.168.122.1 |    0.0.0.0    |    eth0   |   UG  |
ci-info: |   1   | 192.168.122.0 |    0.0.0.0    | 255.255.255.0 |    eth0   |   U   |
ci-info: +-------+---------------+---------------+---------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: |   1   |  multicast  |    ::   |    eth0   |   U   |
ci-info: +-------+-------------+---------+-----------+-------+
Cloud-init v. 22.1-437-g3e026561-1~bddeb running 'modules:config' at Tue, 16 Jan 2024 15:08:10 +0000. Up 23.28 seconds.
Killed old client process
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/52:54:00:e3:4f:82
Sending on   LPF/eth0/52:54:00:e3:4f:82
Sending on   Socket/fallback
DHCPRELEASE of 192.168.122.172 on eth0 to 192.168.122.1 port 67
Status of zebra: FAILED
Status of mgmtd: FAILED
Status of staticd: FAILED
RTNETLINK answers: No such process
Cloud-init v. 22.1-437-g3e026561-1~bddeb running 'modules:final' at Tue, 16 Jan 2024 15:08:17 +0000. Up 30.49 seconds.
No 'final' modules to run under section 'cloud_final_modules'
vyos@ci-router1:~$ 


2024-01-16 15:08:17,252 - cc_vyos_userdata.py[DEBUG]: Configuring command: "set system host-name 'ci-router1'"
2024-01-16 15:08:17,252 - cc_vyos_userdata.py[DEBUG]: Configuring command: "set nat source rule 100 outbound-interface name 'eth0'"
2024-01-16 15:08:17,253 - cc_vyos_userdata.py[DEBUG]: Marking node as tag: "['nat', 'source', 'rule']"
2024-01-16 15:08:17,253 - cc_vyos_userdata.py[DEBUG]: Configuring command: "set nat source rule 100 translation address 'masquerade'"
2024-01-16 15:08:17,253 - cc_vyos_userdata.py[DEBUG]: Marking node as tag: "['nat', 'source', 'rule']"
2024-01-16 15:08:17,253 - cc_vyos_userdata.py[DEBUG]: Configuration file saved: /opt/vyatta/etc/config/config.boot
2024-01-16 15:08:17,253 - handlers.py[DEBUG]: finish: modules-config/config-vyos_userdata: SUCCESS: config-vyos_userdata ran successfully


vyos@ci-router1:~$ show conf com | match nat
set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 translation address 'masquerade'
vyos@ci-router1:~$ 


vyos@ci-router1:~$ show ver
Version:          VyOS 1.4-rolling-202401161432

Thanks @jestabro