Page MenuHomeVyOS Platform
Create Task
Maniphest T5947

[1.3.2 -> 1.4.0-RC1 Migration] Static ipv6 routes dropped
Needs testing, NormalPublicBUG
Actions

Description

Migrated from 1.3.2 to 1.4.0-rc1 and observed the following config changes pre- vs post-migration:

[protocols ospfv3]
...
[protocols static]
- interface-route6 ::/0 {
-     next-hop-interface lo {
-     }
- }
[protocols static table 10]
- interface-route6 ::/0 {
-     next-hop-interface tun0 {
-     }
- }
[protocols static table 20]
- interface-route6 ::/0 {
-     next-hop-interface tun1 {
-     }
- }
[service]
...

I only have route tables for ipv6, so cannot comment on correctness of comparable ipv4 migration.

Details

Difficulty level
Unknown (require assessment)
Version
1.4.0-rc1
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

kevinrausch created this task.Jan 15 2024, 11:55 PM
trae32566 added a subscriber: trae32566.Jan 16 2024, 5:45 AM

I think this is happening because in 1.3, interface-based static routes are under protocols static interface-route6, whereas in 1.4+, they're located in protocols static route6 <address> interface

kevinrausch added a comment.Jan 16 2024, 6:13 AM

Agreed, if it had migrated correctly, I'd see new entries under protocols static; however, the config comparison I posted doesn't have those replacement entries, and my ipv6 (lack of) connectivity confirms it didn't migrate as expected.

Viacheslav added a comment.Jan 16 2024, 1:38 PM

Could you get the original 1.3.x config file copy to the 1.4 instance and load config? I tested 1.3.5 looks good

load path_to_config_1.3
pasik added a subscriber: pasik.Jan 16 2024, 6:59 PM
kevinrausch added a comment.EditedJan 17 2024, 2:45 AM

Performed these steps:

  • rollback VM to 1.3.2 working baseline
  • save vyos132.cfg
  • add system image https://cdn.vyos.io/1.4.0-rc1/vyos-1.4.0-rc1-amd64.iso
  • reboot
  • compare 1, confirm routes dropped per T5947 description
  • load vyos132.cfg
  • compare 1, observe routes have been migrated as expected, below

So, the migration can work but doesn't during initial upgrade?

[protocols static]
- interface-route6 ::/0 {
-     next-hop-interface lo {
-     }
- }
[protocols static table 10]
- interface-route6 ::/0 {
-     next-hop-interface tun0 {
-     }
- }
+ route6 ::/0 {
+     interface tun0 {
+     }
+ }
[protocols static table 20]
- interface-route6 ::/0 {
-     next-hop-interface tun1 {
-     }
- }
+ route6 ::/0 {
+     interface tun1 {
+     }
+ }
[protocols static]
+ route6 ::/0 {
+     interface lo {
+     }
+ }
trae32566 added a comment.Jan 17 2024, 3:32 AM

I'd be curious if updating to 1.3.5 first, then moving to 1.4 has any impact on this issue. I only say this because I have two separate routers I just upgraded from 1.3.5 that both have the following (after upgrade) which looks to have been correctly migrated:

route6 ::/0 {
    next-hop 2001:db8:3a01:a5::1 {
    }
}
route6 2001:db8:3a01:2::/64 {
    blackhole {
    }
}
route6 2001:db8:3a01:c5::/64 {
    interface bond1.1258 {
    }
}
route6 2001:db8:3a01:fd::/64 {
    interface bond1.1258 {
    }
}
route6 fc00::/7 {
    blackhole {
    }
}
route6 2001:db8:8011:239d::/64 {
    interface bond0.925 {
    }
}
kevinrausch added a comment.Jan 17 2024, 3:44 AM

I lost my legacy LTS access, so I'll have to build 1.3-latest to test that.

If it does turn out that there's something about migrating from <1.3.5, is the official guidance that I'd need to upgrade all my devices to 1.3.5 (self-built since no LTS) before upgrading to 1.4?

kevinrausch added a comment.Jan 17 2024, 3:45 PM

Performed these steps:

  • clean, checkout 1.3.5, build, push to internal web host
  • rollback VM to 1.3.2 working baseline
  • save vyos132.cfg
  • add system image https://removed/vyos-1.3.5-private-amd64.iso
  • reboot
  • compare 1, no changes to config as expected
  • add system image https://cdn.vyos.io/1.4.0-rc1/vyos-1.4.0-rc1-amd64.iso
  • reboot
  • compare 1, confirm routes dropped per T5947 description
  • load vyos132.cfg
  • compare 1, observe routes have been migrated as expected, below

Also tried with RC2, same results:

  • rollback VM to 1.3.2 working baseline
  • save vyos132.cfg
  • add system image https://cdn.vyos.io/1.4.0-rc2/vyos-1.4.0-rc2-amd64.iso
  • reboot
  • compare 1, confirm routes dropped per T5947 description
  • load vyos132.cfg
  • compare 1, observe routes have been migrated as expected, below
Viacheslav changed the task status from Open to Needs testing.Jan 17 2024, 3:55 PM

@kevinrausch I cannot reproduce it, all works fine.
before update

vyos@r1# run show conf com | match route6
set protocols static interface-route6 2001:db8::/64 next-hop-interface lo
set protocols static table 10 interface-route6 2001:db8::/64 next-hop-interface tun0
set protocols static table 20 interface-route6 2001:db8::/64 next-hop-interface tun1
[edit]
vyos@r1# 
[edit]
vyos@r1# run show version 

Version:          VyOS 1.3.2
Release train:    equuleus

After update:

vyos@r1:~$ 
vyos@r1:~$ show conf com | match route6
set protocols static route6 2001:db8::/64 interface lo
set protocols static table 10 route6 2001:db8::/64 interface tun0
set protocols static table 20 route6 2001:db8::/64 interface tun1
vyos@r1:~$ 
vyos@r1:~$ show version 
Version:          VyOS 1.4.0-rc1
Release train:    sagitta

If you have issues, add steps to reproduce it with "set commands".
This will make life much easier for developers.

Close it or provide the actual steps to reproduce or config file.

Viacheslav triaged this task as Normal priority.Jan 17 2024, 3:55 PM
kevinrausch added a comment.Jan 17 2024, 7:37 PM

So, the problem seems to be with empty leaf nodes missing from my baseline config.

I cloned an identical VM with a fresh 1.3.2 installation, and pasted all of the commands as output by show configuration commands from the baseline machine into the new machine. After upgrading to 1.4.0-rc2 per above, the routes migrate as expected.

However, if I copy the config.boot from my baseline machine to the new machine and attempt to upgrade the same way, the routes are dropped per issue description.

baseline, which drops static routes upon upgrade:

interfaces {
    ethernet eth0 {
        address 172.20.43.11/24
        address 2001:db8:1::1/64
        duplex auto
        speed auto
    }
    ethernet eth1 {
        address 172.20.32.12/24
        address 2001:db8:2::1/64
        duplex auto
        speed auto
    }
    loopback lo
    tunnel tun0 {
        encapsulation sit
        multicast disable
        remote 1.1.1.1
        source-address 0.0.0.0
    }
    tunnel tun1 {
        encapsulation sit
        multicast disable
        remote 1.1.1.2
        source-address 0.0.0.0
    }
}
protocols {
    static {
        interface-route6 ::/0 {
            next-hop-interface lo
        }
        route 0.0.0.0/0 {
            next-hop 172.20.43.1
        }
        table 10 {
            interface-route6 ::/0 {
                next-hop-interface tun0
            }
        }
        table 20 {
            interface-route6 ::/0 {
                next-hop-interface tun1
            }
        }
    }
}
service {
    ssh {
        disable-password-authentication {
        }
        port 22
    }
}
system {
    config-management {
        commit-revisions 20
    }
    domain-name removed
    host-name removed
    login {
        user vyos {
            authentication {
                plaintext-password 'vyos'
            }
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
}
// Warning: Do not remove the following line.
// vyos-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack@3:conntrack-sync@2:dhcp-relay@2:dhcp-server@6:dhcpv6-server@1:dns-forwarding@3:firewall@5:https@2:interfaces@22:ipoe-server@1:ipsec@5:isis@1:l2tp@3:lldp@1:mdns@1:nat@5:ntp@1:pppoe-server@5:pptp@2:qos@1:quagga@8:rpki@1:salt@1:snmp@2:ssh@2:sstp@3:system@21:vrrp@2:vyos-accel-ppp@2:wanloadbalance@3:webproxy@2:zone-policy@1"
// Release version: 1.3.2

re-created by entering configuration commands, migrates correctly

interfaces {
    ethernet eth0 {
        address 172.20.43.11/24
        address 2001:db8:1::1/64
        duplex auto
        speed auto
    }
    ethernet eth1 {
        address 172.20.32.12/24
        address 2001:db8:2::1/64
        duplex auto
        speed auto
    }
    loopback lo {
    }
    tunnel tun0 {
        encapsulation sit
        multicast disable
        remote 1.1.1.1
        source-address 0.0.0.0
    }
    tunnel tun1 {
        encapsulation sit
        multicast disable
        remote 1.1.1.2
        source-address 0.0.0.0
    }
}
protocols {
    static {
        interface-route6 ::/0 {
            next-hop-interface lo {
            }
        }
        route 0.0.0.0/0 {
            next-hop 172.20.43.1 {
            }
        }
        table 10 {
            interface-route6 ::/0 {
                next-hop-interface tun0 {
                }
            }
        }
        table 20 {
            interface-route6 ::/0 {
                next-hop-interface tun1 {
                }
            }
        }
    }
}
service {
    ssh {
        disable-password-authentication
        port 22
    }
}
system {
    config-management {
        commit-revisions 20
    }
    domain-name removed
    host-name removed
    login {
        user vyos {
            authentication {
                plaintext-password 'vyos'
            }
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
}
// Warning: Do not remove the following line.
// vyos-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack@3:conntrack-sync@2:dhcp-relay@2:dhcp-server@6:dhcpv6-server@1:dns-forwarding@3:firewall@5:https@2:interfaces@22:ipoe-server@1:ipsec@5:isis@1:l2tp@3:lldp@1:mdns@1:nat@5:ntp@1:pppoe-server@5:pptp@2:qos@1:quagga@8:rpki@1:salt@1:snmp@2:ssh@2:sstp@3:system@21:vrrp@2:vyos-accel-ppp@2:wanloadbalance@3:webproxy@2:zone-policy@1"
// Release version: 1.3.2

These two configs are a sanitized/reduced version of my full config, and when diffing the full versions, there are quite a few empty leaf nodes missing from my baseline. Meaning, whatever fix is needed for this issue may need to be applied to other migration scripts.

Viacheslav added a comment.EditedJan 17 2024, 8:25 PM

Your original config looks corrupted.
For example
Loopback entry:

loopback lo

Expected entry:

loopback lo {
}

Interface route in config:

static {
    interface-route6 ::/0 {
        next-hop-interface lo
    }

Expected:

static {
    interface-route6 ::/0 {
        next-hop-interface lo {
        }
    }

They're what I see from /config/config.boot file on 1.3.2 on my node.

kevinrausch added a comment.Jan 17 2024, 9:01 PM

Well, it didn't start on 1.3.2, so I don't know how the empty leaf nodes could have disappeared over time.

Is there a way to force a full rewrite of the config.boot before attempting upgrades?

Viacheslav added a comment.Jan 18 2024, 8:00 AM
In T5947#172846, @kevinrausch wrote:

Well, it didn't start on 1.3.2, so I don't know how the empty leaf nodes could have disappeared over time.

Is there a way to force a full rewrite of the config.boot before attempting upgrades?

In general "commit/save" does all work

kevinrausch added a comment.Jan 18 2024, 1:43 PM

That doesn't seem to insert missing leaf nodes, so I rebuilt the installation by copy/paste of commands from existing VM, and it migrated the routes as expected upon upgrade to 1.4-rc2.

Unless you want me to convert this issue to a request for "fixup/sanitize missing leaf nodes", the issue can close, thanks for the support investigating.

Viacheslav added a comment.Jan 18 2024, 2:05 PM

Let's not close it now, as we do not fully understand the cause

Viacheslav added a comment.EditedJan 29 2024, 1:08 PM

The root issue could be in https://vyos.dev/T5611 T5611

Viacheslav added a subscriber: jestabro.Jan 29 2024, 2:02 PM
dmbaturin edited projects, added VyOS 1.4 Sagitta (1.4.0-epa1); removed VyOS 1.4 Sagitta.Feb 15 2024, 5:38 PM