Page MenuHomeVyOS Platform

Improving Boot Time for Large Firewall Configurations
Closed, WontfixPublicFEATURE REQUEST

Assigned To
Authored By
rps
Feb 4 2019, 8:28 PM
Referenced Files
F2727108: private.cfg
May 27 2022, 4:51 AM
F2649634: gb-glo-sg4ng1fw27-01_20220324.txt
Apr 8 2022, 7:13 AM
F2649632: image.png
Apr 8 2022, 7:09 AM
F2647713: image.png
Apr 7 2022, 10:30 AM
F2647711: image.png
Apr 7 2022, 10:30 AM
F2647618: image.png
Apr 7 2022, 8:33 AM
F524401: config.txt
May 4 2020, 8:45 AM

Description

Our largest instance has 23,080 lines for config.boot (mostly firewall rule configuration for 15 or so VLANs).

VyOS 1.1.8 to 1.2 configuration migration and boot: 25 min.

Save configuration time: 10 sec.

VyOS 1.2.0 saved config boot time: 19 min.

Commit configuration change time: 3 min 10 sec. (and yes the wait is terrifying)

I've traced back the majority of the boot slowdown to executing iptables for each rule insertion.

Updating the firewall to create a temporary file with changes and calling it using iptables-restore -n < $FILENAME the -n flag being the noflush and important if not replacing the entire ruleset.

Because iptables-restore performs an atomic change there are a few advantages:

  1. Error in any of the rules will cause the atomic change to fail before applied for all rules making error recovery much easier in terms of VyOS not getting out of sync with itself.
  2. The speed of the atomic commit is orders of magnitude faster for large rulesets (from 15 min to 30 sec in one example).
  3. Because iptables-restore with the -n flag allows for insert and removal operations rather than a full flush it is easy to combine add and remove operations such that there is no gap in policy between deletion and insertion of rules making changes safer from a traffic perspective.

For simple operations the syntax is identical to iptables with each line in the file being the same as what would be provided after iptables ending with the word COMMIT (testing needed).

In terms of the commit time I don't know if there is an easy way to address this until the filesystem is no longer used for storing the config tree as a directory structure.

I think adopting the atomic netfilter configuration would be one something easy enough to implement in 1.2.1+ though.

Details

Version
-

Event Timeline

syncer changed the task status from Open to Confirmed.Feb 5 2019, 2:17 PM
syncer triaged this task as Normal priority.
syncer edited projects, added VyOS 1.2 Crux (VyOS 1.2.2); removed VyOS 1.2 Crux.

I am affected too by this issue.

My configuration file has around 7500 lines, while 5000 of them are related to firewall (rules and address-groups)

With VyOS 1.2 boot time is 12 minutes, while in VyOS 1.8 the same config takes less than 3 minutes. commit times went from 10 sec to nearly one minute.

I'm also affected by this, but even with a relatively "small" configuration (2662 lines, at present, where more than half are firewall rules, 5 interfaces).

My system takes approximately 5 minutes to boot, and commits are painfully long.

Also affected by this. Reboots take almost 10 minutes before the device is usable. Commits take a long time as well.

I see similar topics on the Vyos forum.

Will this be picked up?

Another affected by this! Several devices with more than 20K config lines cause our automation scripts to take really really long to complete, and also the devices take much time to boot.
It would be great having this fixed in 1.3 :-)

I definitely am not using large port-ranges. A pretty standard setup using a zone-based firewall.

See attached for reference. Running on an apu2c4, reboot to usable time is 10 minutes. The device itself can run pfSense, ESXi server etc. with no real issues. Yes I know it isn't a powerhouse but the current setup doesn't explain such lengthy reboot and commit times.

See console output:

[ 31.095034] vyos-router[834]: Waiting for NICs to settle down: settled in 2s.
[ 38.543873] vyos-router[834]: Started watchfrr.
[ 38.582174] vyos-router[834]: Mounting VyOS Config...done.
[ 706.098852] vyos-router[834]: Starting VyOS router: migrate rl-system firewa.
[ 706.119589] vyos-config[6524]: Configuration success

We don't do any firewalling — we have lots of prefix-lists for filtering eBGP sessions. Right now we're looking at a router that's taken more than 1h20minutes to boot up — and it is still not finished — on modern Xeon CPUs. That's doubled in length since adding a prefix-list of around 5000 entries (roughly double the total number of prefix-list entries as before).

There's definitely something wrong with how the configuration is applied. Is there an O(n^2) problem?

@csalcedo maybe you use large port-ranges https://phabricator.vyos.net/T2189

Thanks @Viacheslav but not, we already hit that bug some years ago and we made a workaround by defining the ports in the rule without using named port-ranges.

We mainly have lots of firewall rules with lots of firewall groups (address groups and network groups, but not port-groups).

also having lots of NAT rules makes the vyos config handling and boot time very slow..

I'm also affected by this. My configuration has about 5k ip prefixes in network group for policy based routing.

Do we have any news when this feature (-n option) will be included? We have lot of vyos on 1.2.X with booting times between 20 and 40 minutes and commits about 3-5 minutes.

@daniel.arconada it should be fixed in 1.4
Can you check it?

Hi @Viacheslav I have tried this morning upgrading from vyos 1.2.6-S1 to vyos-1.4-rolling-202204060217-amd64.iso but something went wrong with the firewall config load and all the firewall rules and objets were removed from config so impossible to test.. :(

Should i try with another 1.4-rolling version?

tried also with vyos-1.4-rolling-202203070319-amd64.iso and same issue with the migration

image.png (294×1 px, 99 KB)

Note: migrating from 1.2.6-s1 to 1.3.1-S1 exactly same error. I think there is another opened bug for this pendent to be fixed.

Errors detail with vyos-config-debug on the grub:

image.png (243×1 px, 77 KB)

image.png (287×1 px, 60 KB)

@n.fort yes . This is an example os our AC-1 enviroment (the one i have used for the first test)

firewall {
    all-ping enable
    broadcast-ping disable
    config-trap disable
    group {
        address-group CENTREON_SERVERS {
            address 10.5.69.24
        }
        address-group CLUSTER_ADDRESSES {
            address 10.255.255.4
            address 10.255.255.5
            address 10.5.104.16
            address 10.5.105.16
            address 172.16.255.254
            address 10.5.104.14
            address 10.5.105.14
            address 10.5.104.13
            address 10.5.105.13
            address 10.5.104.12
            address 10.5.105.12
            address 10.5.105.248
            address 10.5.104.107
            address 10.5.105.124
            address 10.5.104.216
            address 10.5.105.174
            address 10.5.104.43
            address 10.5.104.133
            address 10.5.105.162
            address 10.5.104.111
            address 10.5.105.121
            address 10.5.104.93
            address 10.5.104.69
            address 10.5.105.90
            address 10.5.104.20
            address 10.5.105.135
            address 10.5.104.116
            address 10.5.105.242
            address 10.5.104.210
            address 10.5.105.55
            address 10.5.104.240
            address 10.5.105.161
            address 10.5.104.230
            address 10.5.104.156
            address 10.5.105.47
            address 10.5.104.249
            address 10.5.105.127
            address 10.5.104.232
            address 10.5.105.52
            address 10.5.104.140
            address 10.5.105.201
            address 10.5.104.79
            address 10.5.105.42
            address 10.5.104.84
            address 10.5.105.54
            address 10.5.104.181
            address 10.5.105.91
            address 10.5.104.141
            address 10.5.105.196
            address 10.5.105.155
            address 10.5.104.213
            address 10.5.105.56
            address 10.5.104.63
            address 10.5.105.179
            address 10.5.106.229
            address 10.5.107.144
            address 10.5.106.96
            address 10.5.106.98
            address 10.5.107.28
            address 10.5.107.18
            address 10.5.107.20
            address 10.5.107.23
            address 10.5.107.19
            address 10.5.107.25
            address 10.5.107.26
        }
        address-group CMK_SATELLITES {
            address 10.5.66.45
            address 10.5.66.46
            address 10.5.66.47
            address 10.5.66.48
            address 10.5.66.49
            address 10.5.66.50
            address 10.5.66.85
        }
        address-group DHCP_SERVERS {
            address 10.255.241.13
            address 10.255.241.14
            address 10.255.242.13
            address 10.255.242.14
            address 10.255.243.13
            address 10.255.243.14
            address 10.255.244.13
            address 10.255.244.14
            address 10.255.245.13
            address 10.255.245.14
            address 10.255.246.13
            address 10.255.246.14
            address 10.255.247.13
            address 10.255.247.14
            address 10.255.248.13
            address 10.255.248.14
            address 10.255.249.13
            address 10.255.249.14
            address 10.5.104.14
            address 10.5.105.14
            address 10.5.104.13
            address 10.5.105.13
        }
        address-group DNSCACHE_SERVERS {
            address 10.255.255.4
            address 10.255.255.5
            address 10.5.104.12
            address 10.5.105.12
        }
        address-group DT_BLOCKED {
            address 172.16.255.254
        }
        address-group DT_FW2F58C_1 {
            address 10.5.106.96
        }
        address-group DT_SMTP_BLOCKED {
            address 172.16.255.254
            address 10.5.107.18
            address 10.5.107.20
            address 10.5.107.19
            address 10.5.107.28
            address 10.5.107.25
            address 10.5.107.26
        }
        address-group DT_VPN-1708 {
            address 10.5.107.18
            address 10.5.107.20
            address 10.5.107.19
        }
        address-group G-ALL_OPEN {
            address 172.16.255.254
        }
        address-group G-ICMP {
            address 172.16.255.254
            address 10.5.104.16
            address 10.5.105.16
        }
        address-group G-20-TCP {
            address 172.16.255.254
        }
        address-group G-21-TCP {
            address 172.16.255.254
        }
        address-group G-22-TCP {
            address 172.16.255.254
            address 10.5.105.179
            address 10.5.107.18
            address 10.5.107.20
            address 10.5.106.98
            address 10.5.107.19
            address 10.5.106.96
            address 10.5.107.28
            address 10.5.107.25
            address 10.5.107.26
        }
        address-group G-25-TCP {
            address 172.16.255.254
        }
        address-group G-53-TCP {
            address 172.16.255.254
        }
        address-group G-53-UDP {
            address 172.16.255.254
        }
        address-group G-80-TCP {
            address 172.16.255.254
            address 10.5.105.179
            address 10.5.107.18
            address 10.5.107.20
            address 10.5.106.98
            address 10.5.107.19
            address 10.5.106.96
            address 10.5.107.28
            address 10.5.107.25
            address 10.5.107.26
        }
        address-group G-110-TCP {
            address 172.16.255.254
        }
        address-group G-143-TCP {
            address 172.16.255.254
        }
        address-group G-443-TCP {
            address 172.16.255.254
            address 10.5.105.179
            address 10.5.107.18
            address 10.5.107.20
            address 10.5.107.19
            address 10.5.106.96
            address 10.5.107.25
            address 10.5.107.26
        }
        address-group G-465-TCP {
            address 172.16.255.254
        }
        address-group G-587-TCP {
            address 172.16.255.254
        }
        address-group G-993-TCP {
            address 172.16.255.254
        }
        address-group G-995-TCP {
            address 172.16.255.254
        }
        address-group G-1433-TCP {
            address 172.16.255.254
        }
        address-group G-3306-TCP {
            address 172.16.255.254
        }
        address-group G-3389-TCP {
            address 172.16.255.254
            address 10.5.107.28
        }
        address-group G-8080-TCP {
            address 172.16.255.254
            address 10.5.106.96
        }
        address-group G-8443-TCP {
            address 172.16.255.254
            address 10.5.105.179
        }
        address-group G-8447-TCP {
            address 172.16.255.254
        }
        address-group G-10000-TCP {
            address 172.16.255.254
        }
        address-group LAN_ADDRESSES {
            address 10.255.255.2
            address 10.255.255.3
        }
        address-group NAGIOS_PROBES {
            address 10.5.104.16
            address 10.5.105.16
        }
        address-group NAS_ARRAYS {
            address 10.5.94.251
            address 10.5.94.252
            address 10.5.94.253
            address 10.5.94.254
        }
        address-group NAS_DOMAIN_CONTROLLERS {
            address 10.5.94.16
            address 10.5.94.17
        }
        address-group NLB_ADDRESSES {
            address 10.5.69.22
            address 10.5.69.23
        }
        network-group MANAGEMENT_NETWORKS {
            network 10.22.6.0/24
            network 10.23.53.0/24
        }
        network-group NAS_NETWORKS {
            network 10.5.94.0/24
        }
        network-group RFC1918 {
            network 10.0.0.0/8
            network 172.16.0.0/12
            network 192.168.0.0/16
        }
        network-group TRANSFER_NETS {
            network 10.5.116.32/27
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name LAN-INBOUND {
        default-action drop
        rule 10 {
            action drop
            description "Anti-spoofing non-cluster addresses"
            source {
                group {
                    address-group !CLUSTER_ADDRESSES
                }
            }
        }
        rule 20 {
            action drop
            description "Drop traffic to datacenter transfer net"
            destination {
                group {
                    network-group TRANSFER_NETS
                }
            }
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
        rule 400 {
            action drop
            description Anti-spoofing_10.255.255.2
            source {
                address 10.255.255.2
                mac-address !00:50:56:9d:28:54
            }
        }
        rule 401 {
            action drop
            description Anti-spoofing_10.5.106.96
            source {
                address 10.5.106.96
                mac-address !00:50:56:30:a8:19
            }
        }
        rule 402 {
            action drop
            description Anti-spoofing_10.5.106.98
            source {
                address 10.5.106.98
                mac-address !00:50:56:3d:1d:67
            }
        }
        rule 403 {
            action drop
            description Anti-spoofing_10.5.107.18
            source {
                address 10.5.107.18
                mac-address !00:50:56:34:6d:b4
            }
        }
        rule 404 {
            action drop
            description Anti-spoofing_10.5.107.19
            source {
                address 10.5.107.19
                mac-address !00:50:56:3a:ec:56
            }
        }
        rule 405 {
            action drop
            description Anti-spoofing_10.5.107.25
            source {
                address 10.5.107.25
                mac-address !00:50:56:2a:0c:f3
            }
        }
        rule 406 {
            action drop
            description Anti-spoofing_10.5.107.20
            source {
                address 10.5.107.20
                mac-address !00:50:56:34:6d:b4
            }
        }
        rule 407 {
            action drop
            description Anti-spoofing_10.5.107.23
            source {
                address 10.5.107.23
                mac-address !00:50:56:30:a8:19
            }
        }
        rule 408 {
            action drop
            description Anti-spoofing_10.5.107.26
            source {
                address 10.5.107.26
                mac-address !00:50:56:2a:0c:f3
            }
        }
        rule 410 {
            action drop
            description Anti-spoofing_10.255.255.3
            source {
                address 10.255.255.3
                mac-address !00:50:56:9d:b6:18
            }
        }
        rule 411 {
            action drop
            description Anti-spoofing_10.5.107.28
            source {
                address 10.5.107.28
                mac-address !00:50:56:30:fc:74
            }
        }
        rule 420 {
            action drop
            description Anti-spoofing_10.255.255.4
            source {
                address 10.255.255.4
                mac-address !00:50:56:ff:ff:ff
            }
        }
        rule 425 {
            action drop
            description Anti-spoofing_10.255.255.5
            source {
                address 10.255.255.5
                mac-address !00:50:56:ff:ff:ff
            }
        }
        rule 430 {
            action drop
            description Anti-spoofing_10.5.104.12
            source {
                address 10.5.104.12
                mac-address !00:50:56:ff:ff:ff
            }
        }
        rule 435 {
            action drop
            description Anti-spoofing_10.5.105.12
            source {
                address 10.5.105.12
                mac-address !00:50:56:ff:ff:ff
            }
        }
        rule 440 {
            action drop
            description Anti-spoofing_probe_10.5.104.16
            source {
                address 10.5.104.16
                mac-address !00:50:56:9d:05:df
            }
        }
        rule 450 {
            action drop
            description Anti-spoofing_probe_10.5.105.16
            source {
                address 10.5.105.16
                mac-address !00:50:56:9d:3b:22
            }
        }
        rule 460 {
            action drop
            description Anti-spoofing_10.5.104.13
            source {
                address 10.5.104.13
                mac-address !00:50:56:b1:56:62
            }
        }
        rule 465 {
            action drop
            description Anti-spoofing_10.5.104.14
            source {
                address 10.5.104.14
                mac-address !00:50:56:b1:3e:95
            }
        }
        rule 470 {
            action drop
            description Anti-spoofing_10.5.105.13
            source {
                address 10.5.105.13
                mac-address !00:50:56:b1:56:62
            }
        }
        rule 475 {
            action drop
            description Anti-spoofing_10.5.105.14
            source {
                address 10.5.105.14
                mac-address !00:50:56:b1:3e:95
            }
        }
        rule 507 {
            action drop
            description Anti-spoofing_10.5.105.248
            source {
                address 10.5.105.248
                mac-address !00:50:56:14:78:42
            }
        }
        rule 508 {
            action drop
            description Anti-spoofing_10.5.104.107
            source {
                address 10.5.104.107
                mac-address !00:50:56:31:e6:20
            }
        }
        rule 509 {
            action drop
            description Anti-spoofing_10.5.105.124
            source {
                address 10.5.105.124
                mac-address !00:50:56:1c:bd:df
            }
        }
        rule 510 {
            action drop
            description Anti-spoofing_10.5.104.216
            source {
                address 10.5.104.216
                mac-address !00:50:56:14:49:e3
            }
        }
        rule 511 {
            action drop
            description Anti-spoofing_10.5.105.174
            source {
                address 10.5.105.174
                mac-address !00:50:56:19:11:3e
            }
        }
        rule 512 {
            action drop
            description Anti-spoofing_10.5.104.43
            source {
                address 10.5.104.43
                mac-address !00:50:56:2b:20:76
            }
        }
        rule 513 {
            action drop
            description Anti-spoofing_10.5.104.133
            source {
                address 10.5.104.133
                mac-address !00:50:56:31:c2:cb
            }
        }
        rule 514 {
            action drop
            description Anti-spoofing_10.5.105.162
            source {
                address 10.5.105.162
                mac-address !00:50:56:3b:25:8e
            }
        }
        rule 515 {
            action drop
            description Anti-spoofing_10.5.104.111
            source {
                address 10.5.104.111
                mac-address !00:50:56:06:53:1d
            }
        }
        rule 516 {
            action drop
            description Anti-spoofing_10.5.105.121
            source {
                address 10.5.105.121
                mac-address !00:50:56:35:06:11
            }
        }
        rule 517 {
            action drop
            description Anti-spoofing_10.5.104.93
            source {
                address 10.5.104.93
                mac-address !00:50:56:2d:ae:73
            }
        }
        rule 518 {
            action drop
            description Anti-spoofing_10.5.104.69
            source {
                address 10.5.104.69
                mac-address !00:50:56:00:d4:cb
            }
        }
        rule 519 {
            action drop
            description Anti-spoofing_10.5.105.90
            source {
                address 10.5.105.90
                mac-address !00:50:56:12:c2:cc
            }
        }
        rule 520 {
            action drop
            description Anti-spoofing_10.5.104.20
            source {
                address 10.5.104.20
                mac-address !00:50:56:1d:f0:44
            }
        }
        rule 521 {
            action drop
            description Anti-spoofing_10.5.105.135
            source {
                address 10.5.105.135
                mac-address !00:50:56:25:f8:27
            }
        }
        rule 522 {
            action drop
            description Anti-spoofing_10.5.104.116
            source {
                address 10.5.104.116
                mac-address !00:50:56:17:8c:0d
            }
        }
        rule 523 {
            action drop
            description Anti-spoofing_10.5.105.242
            source {
                address 10.5.105.242
                mac-address !00:50:56:32:ff:63
            }
        }
        rule 524 {
            action drop
            description Anti-spoofing_10.5.104.210
            source {
                address 10.5.104.210
                mac-address !00:50:56:0c:08:0a
            }
        }
        rule 525 {
            action drop
            description Anti-spoofing_10.5.105.55
            source {
                address 10.5.105.55
                mac-address !00:50:56:2e:4f:ee
            }
        }
        rule 526 {
            action drop
            description Anti-spoofing_10.5.104.240
            source {
                address 10.5.104.240
                mac-address !00:50:56:3b:96:b6
            }
        }
        rule 527 {
            action drop
            description Anti-spoofing_10.5.105.161
            source {
                address 10.5.105.161
                mac-address !00:50:56:07:03:b0
            }
        }
        rule 528 {
            action drop
            description Anti-spoofing_10.5.104.230
            source {
                address 10.5.104.230
                mac-address !00:50:56:1f:09:79
            }
        }
        rule 529 {
            action drop
            description Anti-spoofing_10.5.104.156
            source {
                address 10.5.104.156
                mac-address !00:50:56:38:a0:3c
            }
        }
        rule 530 {
            action drop
            description Anti-spoofing_10.5.105.47
            source {
                address 10.5.105.47
                mac-address !00:50:56:34:54:24
            }
        }
        rule 531 {
            action drop
            description Anti-spoofing_10.5.104.249
            source {
                address 10.5.104.249
                mac-address !00:50:56:11:1b:10
            }
        }
        rule 532 {
            action drop
            description Anti-spoofing_10.5.105.127
            source {
                address 10.5.105.127
                mac-address !00:50:56:1e:a4:09
            }
        }
        rule 533 {
            action drop
            description Anti-spoofing_10.5.104.232
            source {
                address 10.5.104.232
                mac-address !00:50:56:16:ca:27
            }
        }
        rule 534 {
            action drop
            description Anti-spoofing_10.5.105.52
            source {
                address 10.5.105.52
                mac-address !00:50:56:29:01:78
            }
        }
        rule 535 {
            action drop
            description Anti-spoofing_10.5.104.140
            source {
                address 10.5.104.140
                mac-address !00:50:56:22:72:e6
            }
        }
        rule 536 {
            action drop
            description Anti-spoofing_10.5.105.201
            source {
                address 10.5.105.201
                mac-address !00:50:56:07:0f:34
            }
        }
        rule 537 {
            action drop
            description Anti-spoofing_10.5.104.79
            source {
                address 10.5.104.79
                mac-address !00:50:56:10:e2:4a
            }
        }
        rule 538 {
            action drop
            description Anti-spoofing_10.5.105.42
            source {
                address 10.5.105.42
                mac-address !00:50:56:0a:e5:39
            }
        }
        rule 539 {
            action drop
            description Anti-spoofing_10.5.104.84
            source {
                address 10.5.104.84
                mac-address !00:50:56:12:b1:21
            }
        }
        rule 540 {
            action drop
            description Anti-spoofing_10.5.105.54
            source {
                address 10.5.105.54
                mac-address !00:50:56:37:34:55
            }
        }
        rule 541 {
            action drop
            description Anti-spoofing_10.5.104.181
            source {
                address 10.5.104.181
                mac-address !00:50:56:2c:3d:08
            }
        }
        rule 542 {
            action drop
            description Anti-spoofing_10.5.105.91
            source {
                address 10.5.105.91
                mac-address !00:50:56:12:f6:fb
            }
        }
        rule 543 {
            action drop
            description Anti-spoofing_10.5.104.141
            source {
                address 10.5.104.141
                mac-address !00:50:56:19:da:2b
            }
        }
        rule 544 {
            action drop
            description Anti-spoofing_10.5.105.196
            source {
                address 10.5.105.196
                mac-address !00:50:56:15:a5:ee
            }
        }
        rule 546 {
            action drop
            description Anti-spoofing_10.5.105.155
            source {
                address 10.5.105.155
                mac-address !00:50:56:35:46:14
            }
        }
        rule 547 {
            action drop
            description Anti-spoofing_10.5.104.213
            source {
                address 10.5.104.213
                mac-address !00:50:56:3c:7e:8e
            }
        }
        rule 548 {
            action drop
            description Anti-spoofing_10.5.105.56
            source {
                address 10.5.105.56
                mac-address !00:50:56:38:fd:e2
            }
        }
        rule 549 {
            action drop
            description Anti-spoofing_10.5.104.63
            source {
                address 10.5.104.63
                mac-address !00:50:56:07:49:e9
            }
        }
        rule 555 {
            action drop
            description Anti-spoofing_10.5.105.179
            source {
                address 10.5.105.179
                mac-address !00:50:56:15:fa:c3
            }
        }
        rule 563 {
            action drop
            description Anti-spoofing_10.5.106.229
            source {
                address 10.5.106.229
                mac-address !00:50:56:1b:2a:d6
            }
        }
        rule 565 {
            action drop
            description Anti-spoofing_10.5.107.144
            source {
                address 10.5.107.144
                mac-address !00:50:56:12:b7:64
            }
        }
        rule 1500 {
            action drop
            description "Block port 11211-udp"
            protocol udp
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
                port 11211
            }
        }
        rule 1510 {
            action drop
            description "Test Drive - Outgoing traffic blocked"
            destination {
                group {
                    network-group !NAS_NETWORKS
                }
            }
            source {
                group {
                    address-group DT_BLOCKED
                }
            }
        }
        rule 1520 {
            action drop
            description "Deny outgoing SMTP to new contracts"
            destination {
                port smtp
            }
            protocol tcp
            source {
                group {
                    address-group DT_SMTP_BLOCKED
                }
            }
        }
        rule 1600 {
            action accept
            description "Allow unicast requests to DHCP servers"
            destination {
                group {
                    address-group DHCP_SERVERS
                }
                port bootps
            }
            protocol tcp_udp
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
        rule 1610 {
            action accept
            description "Allow DNS queries to dnscache servers"
            destination {
                group {
                    address-group DNSCACHE_SERVERS
                }
                port 53
            }
            protocol tcp_udp
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
        rule 1620 {
            action accept
            destination {
                group {
                    address-group NAS_ARRAYS
                }
            }
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
        rule 1630 {
            action accept
            description "Kerberos authentication to Domain Controllers"
            destination {
                group {
                    address-group NAS_DOMAIN_CONTROLLERS
                }
                port 88
            }
            protocol tcp_udp
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
        rule 1640 {
            action drop
            description "Deny rest of the traffic to NAS"
            destination {
                group {
                    network-group NAS_NETWORKS
                }
            }
        }
        rule 2000 {
            action accept
            description "TOP port - SSH"
            destination {
                group {
                    address-group G-22-TCP
                }
                port ssh
            }
            protocol tcp
        }
        rule 2001 {
            action accept
            description "TOP port - RDESKTOP"
            destination {
                group {
                    address-group G-3389-TCP
                }
                port 3389
            }
            protocol tcp
        }
        rule 2002 {
            action accept
            description "TOP port - HTTP"
            destination {
                group {
                    address-group G-80-TCP
                }
                port http
            }
            protocol tcp
        }
        rule 2003 {
            action accept
            description "TOP port - HTTPS"
            destination {
                group {
                    address-group G-443-TCP
                }
                port https
            }
            protocol tcp
        }
        rule 2004 {
            action accept
            description "TOP port - DOMAIN TCP"
            destination {
                group {
                    address-group G-53-TCP
                }
                port domain
            }
            protocol tcp
        }
        rule 2005 {
            action accept
            description "TOP port - DOMAIN UDP"
            destination {
                group {
                    address-group G-53-UDP
                }
                port domain
            }
            protocol udp
        }
        rule 2006 {
            action accept
            description "TOP port - SMTP"
            destination {
                group {
                    address-group G-25-TCP
                }
                port smtp
            }
            protocol tcp
        }
        rule 2007 {
            action accept
            description "TOP port - IMAP"
            destination {
                group {
                    address-group G-143-TCP
                }
                port imap2
            }
            protocol tcp
        }
        rule 2008 {
            action accept
            description "TOP port - POP3"
            destination {
                group {
                    address-group G-110-TCP
                }
                port pop3
            }
            protocol tcp
        }
        rule 2009 {
            action accept
            description "TOP port - MSSQL TCP"
            destination {
                group {
                    address-group G-1433-TCP
                }
                port ms-sql-s
            }
            protocol tcp
        }
        rule 2010 {
            action accept
            description "TOP port - MYSQL TCP"
            destination {
                group {
                    address-group G-3306-TCP
                }
                port mysql
            }
            protocol tcp
        }
        rule 2011 {
            action accept
            description "TOP port - FTPDATA"
            destination {
                group {
                    address-group G-20-TCP
                }
                port ftp-data
            }
            protocol tcp
        }
        rule 2012 {
            action accept
            description "TOP port - FTP"
            destination {
                group {
                    address-group G-21-TCP
                }
                port ftp
            }
            protocol tcp
        }
        rule 2013 {
            action accept
            description "TOP port - SSMTP"
            destination {
                group {
                    address-group G-465-TCP
                }
                port ssmtp
            }
            protocol tcp
        }
        rule 2014 {
            action accept
            description "TOP port - SMTPS"
            destination {
                group {
                    address-group G-587-TCP
                }
                port 587
            }
            protocol tcp
        }
        rule 2015 {
            action accept
            description "TOP port - IMAPS"
            destination {
                group {
                    address-group G-993-TCP
                }
                port imaps
            }
            protocol tcp
        }
        rule 2016 {
            action accept
            description "TOP port - POP3S"
            destination {
                group {
                    address-group G-995-TCP
                }
                port pop3s
            }
            protocol tcp
        }
        rule 2017 {
            action accept
            description "TOP port - TOMCAT"
            destination {
                group {
                    address-group G-8080-TCP
                }
                port 8080
            }
            protocol tcp
        }
        rule 2018 {
            action accept
            description "TOP port - Alternative HTTPS"
            destination {
                group {
                    address-group G-8443-TCP
                }
                port 8443
            }
            protocol tcp
        }
        rule 2019 {
            action accept
            description "TOP port - 10000/TCP"
            destination {
                group {
                    address-group G-10000-TCP
                }
                port 10000
            }
            protocol tcp
        }
        rule 2020 {
            action accept
            description "TOP port - 8447/TCP"
            destination {
                group {
                    address-group G-8447-TCP
                }
                port 8447
            }
            protocol tcp
        }
        rule 2040 {
            action accept
            description "TOP port - All ports open"
            destination {
                group {
                    address-group G-ALL_OPEN
                }
            }
        }
        rule 2050 {
            action accept
            description "ICMP group"
            destination {
                group {
                    address-group G-ICMP
                }
            }
            protocol icmp
        }
        rule 2100 {
            action accept
            description FW2F58C_1-TCP-ALLOW-ANY
            destination {
                group {
                    address-group DT_FW2F58C_1
                }
                port 28
            }
            protocol tcp
        }
        rule 2101 {
            action accept
            description VPN-1708-ANY-ALLOW-10.5.80.17
            destination {
                group {
                    address-group DT_VPN-1708
                }
            }
            source {
                address 10.5.80.17
            }
        }
        rule 2102 {
            action accept
            description VPN-1708-ANY-ALLOW-10.5.81.17
            destination {
                group {
                    address-group DT_VPN-1708
                }
            }
            source {
                address 10.5.81.17
            }
        }
        rule 8500 {
            action drop
            description "Deny traffic to any private address"
            destination {
                group {
                    network-group RFC1918
                }
            }
            disable
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
        rule 8510 {
            action accept
            description "Default allow rule"
            destination {
                group {
                    address-group !CLUSTER_ADDRESSES
                }
            }
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
    }
    name LOCAL-LAN {
        default-action drop
        rule 2 {
            action accept
            destination {
                address 10.255.255.1
            }
            protocol icmp
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
        rule 4 {
            action accept
            destination {
                group {
                    address-group LAN_ADDRESSES
                }
            }
            source {
                group {
                    address-group LAN_ADDRESSES
                }
            }
        }
        rule 10 {
            action accept
            description "Multicast para VRRP"
            destination {
                address 224.0.0.18
            }
            source {
                group {
                    address-group LAN_ADDRESSES
                }
            }
        }
    }
    name LOCAL-SYNC {
        default-action drop
        rule 5 {
            action accept
            description "Permitir trafico sync entre nodos"
            destination {
                address 10.5.68.60/30
            }
            source {
                address 10.5.68.60/30
            }
        }
    }
    name LOCAL-WAN {
        default-action drop
        description "External connections from VLAN2701 to this system"
        rule 10 {
            action accept
            description "Allow intra-vlan 2701 connections"
            destination {
                address 82.223.45.160/27
            }
            source {
                address 82.223.45.160/27
            }
        }
        rule 20 {
            action accept
            description "Allow Arsys desktops to contact this system"
            source {
                group {
                    network-group MANAGEMENT_NETWORKS
                }
            }
        }
        rule 30 {
            action accept
            description "Allow intra-vlan traffic"
            destination {
                group {
                    network-group TRANSFER_NETS
                }
            }
            source {
                group {
                    network-group TRANSFER_NETS
                }
            }
        }
    }
    name WAN-INBOUND {
        default-action drop
        rule 10 {
            action accept
            description "Management from HN-ES"
            source {
                group {
                    network-group MANAGEMENT_NETWORKS
                }
            }
        }
        rule 20 {
            action accept
            description "Connections from Load Balancer to Frontends - TCP Proxy"
            destination {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
            source {
                group {
                    address-group NLB_ADDRESSES
                }
            }
        }
        rule 30 {
            action accept
            description "Allow external probes"
            destination {
                group {
                    address-group NAGIOS_PROBES
                }
            }
            protocol icmp
        }
        rule 40 {
            action accept
            description "Allow Centreon servers traffic to VMs"
            destination {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
            source {
                group {
                    address-group CENTREON_SERVERS
                }
            }
        }
        rule 50 {
            action accept
            description "Allow CMK to check dnscache servers - TCP"
            destination {
                group {
                    address-group DNSCACHE_SERVERS
                }
                port 22,53,6556
            }
            protocol tcp
            source {
                group {
                    address-group CMK_SATELLITES
                }
            }
        }
        rule 65 {
            action accept
            description "Allow CMK to check dnscache servers - UDP"
            destination {
                group {
                    address-group DNSCACHE_SERVERS
                }
                port 53
            }
            protocol udp
            source {
                group {
                    address-group CMK_SATELLITES
                }
            }
        }
        rule 70 {
            action accept
            description "Allow CMK to check dnscache servers - ICMP"
            destination {
                group {
                    address-group DNSCACHE_SERVERS
                }
            }
            protocol icmp
            source {
                group {
                    address-group CMK_SATELLITES
                }
            }
        }
        rule 80 {
            action accept
            description "Allow CMK to check monitoring sensors - TCP"
            destination {
                group {
                    address-group NAGIOS_PROBES
                }
                port 6556
            }
            protocol tcp
            source {
                group {
                    address-group CMK_SATELLITES
                }
            }
        }
        rule 90 {
            action accept
            description "Allow CMK to check monitoring sensors - ICMP"
            destination {
                group {
                    address-group NAGIOS_PROBES
                }
            }
            protocol icmp
            source {
                group {
                    address-group CMK_SATELLITES
                }
            }
        }
        rule 2000 {
            action accept
            description "TOP port - SSH"
            destination {
                group {
                    address-group G-22-TCP
                }
                port ssh
            }
            protocol tcp
        }
        rule 2001 {
            action accept
            description "TOP port - RDESKTOP"
            destination {
                group {
                    address-group G-3389-TCP
                }
                port 3389
            }
            protocol tcp
        }
        rule 2002 {
            action accept
            description "TOP port - HTTP"
            destination {
                group {
                    address-group G-80-TCP
                }
                port http
            }
            protocol tcp
        }
        rule 2003 {
            action accept
            description "TOP port - HTTPS"
            destination {
                group {
                    address-group G-443-TCP
                }
                port https
            }
            protocol tcp
        }
        rule 2004 {
            action accept
            description "TOP port - DOMAIN TCP"
            destination {
                group {
                    address-group G-53-TCP
                }
                port domain
            }
            protocol tcp
        }
        rule 2005 {
            action accept
            description "TOP port - DOMAIN UDP"
            destination {
                group {
                    address-group G-53-UDP
                }
                port domain
            }
            protocol udp
        }
        rule 2006 {
            action accept
            description "TOP port - SMTP"
            destination {
                group {
                    address-group G-25-TCP
                }
                port smtp
            }
            protocol tcp
        }
        rule 2007 {
            action accept
            description "TOP port - IMAP"
            destination {
                group {
                    address-group G-143-TCP
                }
                port imap2
            }
            protocol tcp
        }
        rule 2008 {
            action accept
            description "TOP port - POP3"
            destination {
                group {
                    address-group G-110-TCP
                }
                port pop3
            }
            protocol tcp
        }
        rule 2009 {
            action accept
            description "TOP port - MSSQL TCP"
            destination {
                group {
                    address-group G-1433-TCP
                }
                port ms-sql-s
            }
            protocol tcp
        }
        rule 2010 {
            action accept
            description "TOP port - MYSQL TCP"
            destination {
                group {
                    address-group G-3306-TCP
                }
                port mysql
            }
            protocol tcp
        }
        rule 2011 {
            action accept
            description "TOP port - FTPDATA"
            destination {
                group {
                    address-group G-20-TCP
                }
                port ftp-data
            }
            protocol tcp
        }
        rule 2012 {
            action accept
            description "TOP port - FTP"
            destination {
                group {
                    address-group G-21-TCP
                }
                port ftp
            }
            protocol tcp
        }
        rule 2013 {
            action accept
            description "TOP port - SSMTP"
            destination {
                group {
                    address-group G-465-TCP
                }
                port ssmtp
            }
            protocol tcp
        }
        rule 2014 {
            action accept
            description "TOP port - SMTPS"
            destination {
                group {
                    address-group G-587-TCP
                }
                port 587
            }
            protocol tcp
        }
        rule 2015 {
            action accept
            description "TOP port - IMAPS"
            destination {
                group {
                    address-group G-993-TCP
                }
                port imaps
            }
            protocol tcp
        }
        rule 2016 {
            action accept
            description "TOP port - POP3S"
            destination {
                group {
                    address-group G-995-TCP
                }
                port pop3s
            }
            protocol tcp
        }
        rule 2017 {
            action accept
            description "TOP port - TOMCAT"
            destination {
                group {
                    address-group G-8080-TCP
                }
                port 8080
            }
            protocol tcp
        }
        rule 2018 {
            action accept
            description "TOP port - Alternative HTTPS"
            destination {
                group {
                    address-group G-8443-TCP
                }
                port 8443
            }
            protocol tcp
        }
        rule 2019 {
            action accept
            description "TOP port - 10000/TCP"
            destination {
                group {
                    address-group G-10000-TCP
                }
                port 10000
            }
            protocol tcp
        }
        rule 2020 {
            action accept
            description "TOP port - 8447/TCP"
            destination {
                group {
                    address-group G-8447-TCP
                }
                port 8447
            }
            protocol tcp
        }
        rule 2040 {
            action accept
            description "TOP port - All ports open"
            destination {
                group {
                    address-group G-ALL_OPEN
                }
            }
        }
        rule 2050 {
            action accept
            description "ICMP group"
            destination {
                group {
                    address-group G-ICMP
                }
            }
            protocol icmp
        }
        rule 2100 {
            action accept
            description FW2F58C_1-TCP-ALLOW-ANY
            destination {
                group {
                    address-group DT_FW2F58C_1
                }
                port 28
            }
            protocol tcp
        }
        rule 2101 {
            action accept
            description VPN-1708-ANY-ALLOW-10.5.80.17
            destination {
                group {
                    address-group DT_VPN-1708
                }
            }
            source {
                address 10.5.80.17
            }
        }
        rule 2102 {
            action accept
            description VPN-1708-ANY-ALLOW-10.5.81.17
            destination {
                group {
                    address-group DT_VPN-1708
                }
            }
            source {
                address 10.5.81.17
            }
        }
    }
    receive-redirects disable
    send-redirects disable
    source-validation disable
    state-policy {
        established {
            action accept
        }
        invalid {
            action drop
        }
        related {
            action accept
        }
    }
    syn-cookies enable
    twa-hazards-protection disable
}
high-availability {
    vrrp {
        group eth3-223 {
            advertise-interval 3
            authentication {
                password Ng-1p223
                type plaintext-password
            }
            interface eth3
            preempt-delay 30
            priority 5
            transition-script {
                backup /config/scripts/backup.sh
                fault /config/scripts/backup.sh
                master /config/scripts/master.sh
            }
            virtual-address 10.255.255.1/32
            virtual-address 169.254.169.254/32
            vrid 223
        }
        sync-group VRRP-GROUP {
            member eth3-223
        }
    }
}
interfaces {
    ethernet eth0 {
        address 10.5.64.89/24
        description Management
        duplex auto
        hw-id 00:50:56:9d:a4:47
        smp-affinity auto
        speed auto
    }
    ethernet eth1 {
        description MicroVLANs
        duplex auto
        hw-id 00:50:56:9d:92:98
        smp-affinity auto
        speed auto
        vif 1412 {
            address 10.5.116.57/27
            description "MicroVLAN privada"
            firewall {
                in {
                    name WAN-INBOUND
                }
                local {
                    name LOCAL-WAN
                }
            }
        }
        vif 2701 {
            address 82.223.45.167/27
            description "MicroVLAN publica"
            firewall {
                in {
                    name WAN-INBOUND
                }
                local {
                    name LOCAL-WAN
                }
            }
        }
    }
    ethernet eth2 {
        address 10.5.68.62/30
        description Sync
        duplex auto
        firewall {
            local {
                name LOCAL-SYNC
            }
        }
        hw-id 00:50:56:9d:db:bc
        smp-affinity auto
        speed auto
    }
    ethernet eth3 {
        address 10.255.255.3/20
        description "Customers LAN"
        duplex auto
        firewall {
            in {
                name LAN-INBOUND
            }
            local {
                name LOCAL-LAN
            }
        }
        hw-id 00:50:56:9d:b6:18
        policy {
            route SRC-ROUTE
        }
        smp-affinity auto
        speed auto
    }
    loopback lo {
        address 10.5.64.89/32
    }
}
nat {
    destination {
        rule 5 {
            description cloud-init
            destination {
                address 169.254.169.254
                port http
            }
            inbound-interface eth3
            protocol tcp
            translation {
                address 10.5.74.214
            }
        }
    }
    source {
        rule 10 {
            description "NAT to Internet"
            outbound-interface eth1.2701
            source {
                address 10.5.96.0/20
            }
            translation {
                address masquerade
            }
        }
        rule 20 {
            description "NAT to Internet"
            outbound-interface eth1.2701
            source {
                address 10.5.82.0/21
            }
            translation {
                address masquerade
            }
        }
    }
}
policy {
    community-list 100 {
        rule 10 {
            action permit
            regex 65500:2001
        }
    }
    community-list 101 {
        description "Global Micro"
        rule 10 {
            action permit
            regex 65500:200
        }
    }
    community-list 200 {
        rule 10 {
            action permit
            regex "65500:20**"
        }
    }
    prefix-list RFC1918-Routes {
        rule 1 {
            action permit
            prefix 10.0.0.0/8
        }
        rule 2 {
            action permit
            prefix 172.16.0.0/12
        }
        rule 3 {
            action permit
            prefix 192.168.0.0/16
        }
    }
    prefix-list Service-NETs {
        rule 1 {
            action permit
            ge 32
            prefix 0.0.0.0/0
        }
    }
    route SRC-ROUTE {
        rule 100 {
            destination {
                address 10.5.64.0/23
            }
            set {
                table 10
            }
            source {
                group {
                    address-group CLUSTER_ADDRESSES
                }
            }
        }
    }
    route-map ALLOW-RFC1918-Routes {
        rule 10 {
            action permit
            match {
                ip {
                    address {
                        prefix-list RFC1918-Routes
                    }
                }
            }
        }
        rule 15 {
            action permit
            match {
                community {
                    community-list 100
                }
            }
        }
        rule 18 {
            action permit
            match {
                community {
                    community-list 101
                }
            }
        }
        rule 20 {
            action deny
        }
    }
    route-map Any-Site-2 {
        rule 10 {
            action permit
            match {
                community {
                    community-list 200
                }
            }
        }
        rule 20 {
            action deny
        }
    }
    route-map CLOUD-Service-NETs {
        rule 10 {
            action permit
            match {
                ip {
                    address {
                        prefix-list Service-NETs
                    }
                }
            }
            set {
                community 65500:2024
            }
        }
        rule 20 {
            action deny
        }
    }
    route-map None {
        rule 10 {
            action deny
        }
    }
}
protocols {
    bgp 8560 {
        address-family {
            ipv4-unicast {
                redistribute {
                    static {
                    }
                }
            }
        }
        neighbor 10.5.116.38 {
            address-family {
                ipv4-unicast {
                    route-map {
                        export CLOUD-Service-NETs
                        import ALLOW-RFC1918-Routes
                    }
                    weight 150
                }
            }
            description RouteServer1-vyos
            password xxxxxxxx
            remote-as 8560
            timers {
                holdtime 5
                keepalive 1
            }
        }
        neighbor 10.5.116.39 {
            address-family {
                ipv4-unicast {
                    route-map {
                        export CLOUD-Service-NETs
                        import ALLOW-RFC1918-Routes
                    }
                    weight 125
                }
            }
            description RouteServer2-frr
            password xxxxxxxx
            remote-as 8560
        }
        neighbor 10.5.116.40 {
            address-family {
                ipv4-unicast {
                    route-map {
                        export CLOUD-Service-NETs
                        import ALLOW-RFC1918-Routes
                    }
                    weight 100
                }
            }
            description RouteServer3-bird
            password xxxxxxxx
            remote-as 8560
        }
        parameters {
            log-neighbor-changes
            router-id 10.5.64.89
        }
    }
    static {
        interface-route 10.5.104.16/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.20/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.43/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.63/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.69/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.79/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.84/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.93/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.107/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.111/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.116/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.133/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.140/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.141/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.156/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.181/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.210/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.213/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.216/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.230/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.232/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.240/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.104.249/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.16/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.42/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.47/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.52/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.54/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.55/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.56/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.90/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.91/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.121/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.124/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.127/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.135/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.155/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.161/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.162/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.174/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.179/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.196/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.201/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.242/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.105.248/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.106.96/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.106.98/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.106.229/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.18/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.19/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.20/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.23/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.25/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.26/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.28/32 {
            next-hop-interface eth3 {
            }
        }
        interface-route 10.5.107.144/32 {
            next-hop-interface eth3 {
            }
        }
        route 0.0.0.0/0 {
            next-hop 82.223.45.161 {
            }
        }
        route 10.0.0.0/8 {
            next-hop 10.5.116.33 {
                distance 150
            }
        }
        route 10.5.94.0/24 {
            next-hop 10.5.116.44 {
            }
        }
        route 172.16.0.0/12 {
            next-hop 10.5.116.33 {
                distance 150
            }
        }
        route 192.168.0.0/16 {
            next-hop 10.5.116.33 {
                distance 150
            }
        }
        table 10 {
            route 0.0.0.0/0 {
                next-hop 10.5.116.33 {
                }
            }
        }
    }
}
service {
    lldp {
        legacy-protocols {
            cdp
        }
        snmp {
            enable
        }
    }
    snmp {
        community 1Trpq25 {
            authorization ro
        }
        contact network@arsys.es
        description es-lgr-lp4ngp2fw24-02
        listen-address 10.5.64.89 {
            port 161
        }
        location NGCS
        trap-target 192.168.0.224 {
            community 1Trpq25
            port 162
        }
    }
    ssh {
        listen-address 10.5.64.89
        listen-address 10.5.68.62
        port 22
    }
}
system {
    config-management {
        commit-revisions 20
    }
    conntrack {
        expect-table-size 8192
        hash-size 262144
        table-size 2097152
        timeout {
            icmp 30
            other 120
            tcp {
                close 10
                close-wait 60
                established 3600
                fin-wait 30
                last-ack 30
                syn-recv 5
                syn-sent 5
                time-wait 5
            }
            udp {
                other 10
                stream 10
            }
        }
    }
    host-name es-lgr-lp4ngp2fw24-02
    ip {
        arp {
            table-size 2048
        }
    }
    ipv6 {
        disable
    }
    login {
        user arsys_admin {
            authentication {
                encrypted-password xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                plaintext-password ""
            }
            level admin
        }
        user arsys_operator {
            authentication {
                encrypted-password xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                plaintext-password ""
            }
            level admin
        }
        user provisioning {
            authentication {
                encrypted-password xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
                plaintext-password ""
                public-keys arpinsertion-ac1@por-ngcs.lan {
                    key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
                    type ssh-rsa
                }
            }
            level admin
        }
        user syncuser {
            authentication {
                encrypted-password xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
                plaintext-password ""
            }
            level admin
        }
        user vyos {
            authentication {
                encrypted-password $xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/
                plaintext-password ""
                public-keys larquitectura@arsys.es {
                    key AAAAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
                    type ssh-rsa
                }
                public-keys lplataforma-red@arsys.es {
                    key AAAAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
                    type ssh-rsa
                }
            }
            level admin
        }
    }
    name-server 10.4.131.17
    name-server 10.4.132.17
    ntp {
        server por-ntp1.por-ngcs.lan {
        }
        server por-ntp2.por-ngcs.lan {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level info
            }
        }
        host 10.5.66.54 {
            facility all {
                level notice
            }
            facility protocols {
                level info
            }
            facility user {
                level err
            }
        }
        user all {
            facility all {
                level emerg
            }
        }
    }
    time-zone Europe/Madrid
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack-sync@1:conntrack@1:dhcp-relay@2:dhcp-server@5:firewall@5:ipsec@5:l2tp@1:mdns@1:nat@4:ntp@1:pptp@1:qos@1:quagga@3:ssh@1:system@9:vrrp@2:vyos-accel-ppp@1:wanloadbalance@3:webgui@1:webproxy@1:webproxy@2:zone-policy@1" === */
/* Release version: 1.2.1 */

Thanks for sharing.
It this ok?

/* Release version: 1.2.1 */

I understood you were on 1.2.6 and trying to upgrade 1.3 and 1.4.
Nevertheless, I'll take your config and tests upgradings

@n.fort Sorry yes this AC-1 vyos was on 1.2.1 I will a test tomorrow with a version 1.2.6-S1 to have the path i was mention..

With shared config, I'm not getting high times while loading config (at least not that high as exposed in this task)

After sanitizing config, these are the commit times for complete load configuration:

### time commit on 1.2.1
real	0m30.514s
user	0m22.390s
sys	0m4.014s


### time commit on 1.3.1-S1
real    0m37.424s
user    0m23.288s
sys     0m6.716s


### time commit for 1.4-rolling-202204060217
real    0m22.628s
user    0m7.525s
sys     0m4.733s

And tips for upgrading your current config:

  • Avoid using "set system ipv6 disable"... Remove that config during process. You can look here and in forum about issues related to this config.
  • Change services port names on firewall rules (found problems with bootps and ssmtp).
  • Source nat rule 20 -- ipv4 source address is wrong, and validators in 1.4 version detected it.
  • If possible, move to/from LTS images: 1.2.1 --> 1.2.8 --> 1.3.1-S1 [ --> 1.4.X]

Hi, I'm sorry for the confusion but the configuration I uploaded is not from a production machine. It's just an example of a small configuration of a pre-production vyos that I was having trouble upgrading.
Tomorrow if I will upload one of the ones that take 25-30 minutes to boot on modern CPUs (xeon gold).

@n.fort the config i would like to upload to this ticket has 43727 lines.. Its a text file of 1.3 Mbps. Its it possible to attach the file instead of paste it on the comments?

Update: File uploaded. Thanks cgb{F2649634}

Not sure if this works but there is an upload button in the toolbar:

image.png (52×416 px, 4 KB)

@n.fort I have upgraded following the path 1.2.6-S1 --> 1.3-S1 --> 1.4 Following the tips of not using port names for bootps and ssmtp and not using "set system ipv6 disable" and everything went fine.

Reboot time has improved and takes about half as before. Commit time is about the same.
I have used a a vyos with 600Ks of configuration (about 25k-30k lines)

1.2.6-S1
commit time : 56 sg
boot time : 16 min 30 sg

1.3.1-S1
commit time : 56 sg
boot time : 18 min

1.4.rolling
commit time : 56 sg
boot time : 7 min 40 sg <--- very good improvement.

Did similar tests with your big config >20k lines:

> Boot time on 1.2.6-S1 --> 14m
> Boot time on 1.2.8 --> 14m
> Boot time on 1.3.1-S1 --> 18m
> Boot time on 1.4-rolling-202204100814 --> 8m

I'm still having issues moving past anything higher than 1.2.8. Booting 1.2.8 looks thusly:

[ 23.004967] vyos-router[956]: Started watchfrr.
[ 23.028300] vyos-router[956]: Mounting VyOS Config...done.
[ 350.621308] vyos-router[956]: Starting VyOS router: migrate rl-system firewal
l configure.
[ 350.783229] vyos-config[13662]: Configuration success

Okay, ~6 minutes. I can live with this. Anything higher (tried 1.3.1-S1 and 1.4-rolling-20220526) just never completes (well, maybe it does, but I'm cannot wait longer on a production router for more than about 30 minutes). I've done about all that I can (see T2088), removing large port ranges (managing them outside of vyos), I dont have any port names, just numbers, and I just cannot figure out why my relatively simple (6 interfaces, no vlans, a few wireguard peers, mostly masquerading and a few hairpin nat/reflection rules) configuration would now take 5x or more time to load. I'm happy to post my anonimized config if that might help.

@panachoi If you can share the anonymized config that works in 1.2.8 that would be useful. I'd expect migrating to 1.4 to see a decent improvement in firewall load times.

@panachoi , for me moving to 1.4 rolling release did the trick. Boot times went from > 10 mins in 1.2 to 2-3 minutes in 1.4. Hope that helps

1.4 rolling does not help me, so there must be something "wrong" with my configuration. I've attached the private config, it would be awesome if someone might find what's broken.

For a better analysis, can you share your firewall and nat config without hidden data? You can send it to my email: n.fort@vyos.io

1.4 rolling does not help me, so there must be something "wrong" with my configuration. I've attached the private config, it would be awesome if someone might find what's broken.

...
        rule 54 {
            action drop
            description "Block Steam Gaming"
            destination {
                address 0.0.0.0/0
                group {
                    port-group steam
                }
            }
            disable
            log enable
            protocol all
            source {
                group {
                    address-group Chollo
                }
            }
            state {
                new enable
            }
        }
...

^ This rule is throwing an error on migration "Protocol must be tcp, udp, or tcp_udp when specifying a port or port-group"

There are some non-firewall related errors too that need to be addressed:

Trap target "xxx.xxx.141.30" requires a community to be set!

[[service snmp]] failed

You cannot use NetFlow engine-id "2" together with NetFlow protocol
version "5"!

[[system flow-accounting]] failed

Okay, thats the only rule where I was using a port-group combined with protocol all; the others that use protocol all dont have a port or port group in the rule, so they are okay?

Finally, I'm a bit confused about the netflow stanza. the command line says this:

set system flow-accounting netflow engine-id 
Possible completions:
   <0-255>      NetFlow engine-id (version 5 and 9 only)

I've deleted the engine-id from the netflow stanza, and changed the firewall rule to use tcp_udp instead of all, and will try (with 1.3.1-S1 first) again. I dont mind if it takes a bit longer to boot, as long as it does!

Still not much luck here. But I've let the boot run a bit longer, and notice the following:

The firewall appears to be "up", in the sense that I can get out via NAT, Wireguard VPN peers are up, etc. but:

There is no login prompt, and I cannot login via ssh. I finally get this on the console:

[ 1900.345709] Out of memory: Killed process 1373 (python3) total-vm:8593632kB, anon-rss:7641268kB, file-rss:2420kB, shmem-rss:0kB, UID:0 pgtables:15028kB oom_score_adj:0
[ 1901.759022] vyos-router[842]: Starting VyOS router: migrate rl-system firewall configure/usr/libexec/vyos/init/vyos-router: line 104:  1373 Killed                  sg ${GROUP} -c "$vyos_libexec_dir/vyos-boot-config-loader.py $BOOTFILE"
[ 1901.876065] vyos-router[842]:  failed!

And then after that its dead.
So there's still something wrong with my config under anything > 1.2.8. Time to fire up a vm to try to load the config into a new installation, I guess ?

So, I just spun up a brand new VM, copied the config to it and loaded it without issue.

Upon typing commit, I'm kicked out of the console, and back at the default configuration.

I've debugged this further, by breaking up my configuration into various sections (system, interfaces, firewall,nat,service,vpn etc) and running them on a new VM.

It seems the following items kept the configuration from committing properly:

interfaces openvpn:
encryption aes256
hash sha512

Although the hash seems to be acceptable,
**encryption aes256**
was not migrated to

**encryption cipher aes256**

Also, the wireguard configration was not migrated properly?, in 1.2.8, there was no named-key or default-key (although I guess it was default in any case?) , so the keys were just in /config/auth/wireguard/{private,public}.key

It seems that now they are kept in /config/auth/wireguard/default-key or /config/auth/wireguard/named-key. Once I manually corrected this (by setting the configuration to use a named-key, and then copying the keys into that directory), that worked to apply.

In 1.2.8, the radius server stanza looked like this:

radius-server xxx.xxx.xxx.20 {
    port 1812
    secret deadbeef
    timeout 3
}
radius-source-address xxx.xxx.xxx.254

But this is now:

radius {
     server xxxx.xxx.xxx.20 {
         key deadbeef
         port 1812
     }
     source-address xxx.xxx.xxx.254
 }

And finally, I think, it definitely didn't/does not like my "root" user in the configuration. This is probably a dead body left over from the vyatta days, maybe.

Boot time is now about half (from 300+ seconds), to 170.

Yes, you error with "root" user is a known issue: T4281.

Also to avoid error"out of memory", you should configure VM with at least 1G of RAM in 1.3/1.4

But regarding current task, you were able to test your firewall in 1.3? If possible, can you share times for 1.4?

This vm started out with 4G of memory and 2CPUs; I doubled quickly everything when I hit the out of memory error the first time, but that didn't help. I can quickly install the latest rolling and test

Sorry its taken me so long to follow up on this

1.3.1-S1:

[   23.519140] vyos-router[762]: Mounting VyOS Config...done.
[  144.868974] vyos-router[762]: Starting VyOS router: migrate rl-system firewall configure.
[  145.083468] vyos-config[11473]: Configuration success

1.4-rolling-202206090217

[   22.268820] vyos-router[813]: Mounting VyOS Config...done.
[  100.824597] vyos-router[813]: Starting VyOS router: migrate configure
[  100.827663] vyos-router[8778]:  failed!
[  101.019804] vyos-router[8805]: /opt/vyatta/etc/config/scripts/vyos-postconfig-bootup.script: line 9: /sbin/ipset: No such file or directory
[  101.610294] vyos-config[821]: Configuration error

The error is because 1.4 has fully migrated to nftables, I guess, so ipset doesn't work anymore, and I have not adapted my postboot script for this (yet).

I just saw that somehow, installing the rolling image, my firewall rules did not get migrated correctly, the entire firewall block is empty; even after disabling the offending postboot commands. So something is still amiss

Seems, that for whatever reason, it doesn't like some network ranges:

Error: 192.168.45.0/22 is not a valid IPv4 address range
Error: 192.168.45.0/22 is not a valid IPv4 prefix
Error: 192.168.45.0/22 is not a valid IPv4 address

Invalid value
Value validation failed
Set failed

This is here (I dont actually need this rule anymore, but...)

set firewall name Internet_In rule 30 source address '192.168.45.0/22'

Whats wrong with this VLSM ?

Yes. New 1.4 has more restricted checks on addresses and networks.
Actually, if you are using /22, the correct network for this case is 192.168.44.0/22.
You can use this online tool for checking ipv4 networks and subnets.

Indeed, I figured that out. I also found that my openvpn config was not migrated properly (T3642?); all of the tls configuration stuff (previously kept under /config/auth somewhere) was gone. After doing run import pki for all of the necessary bits it was able at least to commit openvpn properly.

Here's the current major issue for me:

In 1.2/1.3, I was managing some ipsets outside of the configuration, just creating the appropriate (empty) group, and using outside scripts to actually fill them. This is no longer working with 1.4; if an empty set/group is referenced in a rule, the rule will fail to commit, even if you can actually create such a group with no entries without any warnings. This was extremely useful, because one could dynamically modify sets, which seems not (yet?) to be (easily??) possible anymore.

It would appear that named sets are not being used, as the following configuration stanzas:

set firewall group network-group gaming description 'Game Hosting IPs'
set firewall group network-group gaming network '62.80.105.0/24'
set firewall name Internet_Out rule 15 action 'drop'
set firewall name Internet_Out rule 15 description 'Drop Gaming'
set firewall name Internet_Out rule 15 destination group network-group 'gaming'
set firewall name Internet_Out rule 15 log 'enable'

generates the following rule in nftables

ip daddr { 62.80.105.0/24 } log prefix "[Internet_Out-15-D] " counter packets 0 bytes 0 drop comment "Internet_Out-15"

I suppose that this is a side effect of the way that nftables works.

OTOH, shouldn't/can't the above firewall group generate something like:

table ip filter {
          define gaming_set = {62.80.105.0/24}
          set gaming = {
                          type ipv4_addr; flags interval;
                          elements = $gaming_set
           }

and so on? Of couse this doesn't necessarily solve the problem of commiting on a empty set

Thanks for the pointer, but I think it should still be considered a "bug" that you can no longer use an empty group (I'm just going to assume that this would apply to any kind of group, but most are probably using this for host/network groups, as this is where it would be most useful). Judging from the comments in T4147, I'm clearly not the only one who was taking advantage of managing sets outside of the system. Alas, my boot times for 1.4 (what this discussion is about) are not really valid, as my configuration didn't really get migrated from 1.3.1->1.4, or better said, it doesn't actually commit, and I actually ended up with a mostly empty firewall config on boot, which is perhaps why its booting so quickly now.

We can't do more due to old backend on the 1.3
If there will be a specific options to improve it should be a separate task
Close it.