Adding a large port-range will take ~ 20 minutes to commit
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	c-po
	Mar 31 2020, 4:47 PM

Description

I just enabled all high-ports for communication in a firewall rule as the connection tracking and the underlaying protocoll just sucks.

 port-group SONOS-PORTS {
     description "Ports used by SONOS"
-    port 58627
     port 1900
     port 1901
     port 6969
     port 3400-3401
     port 1443
+    port 20000-65535
 }

vyos@vyos# time commit
real    23m44.800s
user    9m9.063s
sys     14m20.008s

That means a reboot will take additional 20 minutes.

A reason could be: https://github.com/vyos/vyatta-cfg-firewall/blob/8d373bd48b8142692dc704dbdbc0c6b314c0913b/lib/Vyatta/IpTables/IpSet.pm#L413 according to @zsdc
https://github.com/vyos/vyatta-cfg-firewall/commit/835304e5aaa252e8b0bcf4651629cd089e670147

Details

Difficulty level: Hard (possibly days)
Version: 1.2.5-epa2
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Bug (incorrect behavior)

Related Objects

Mentioned In: 1.3.3
T4558: Incomplete error message when duplicate firewall port-group used
T4017: Adding firewall port ranges makes commit/boot MASSIVELY slow
T1230: Improving Boot Time for Large Firewall Configurations
Mentioned Here: T2199: Rewrite firewall in new XML/Python style

Event Timeline

c-po triaged this task as High priority.Mar 31 2020, 4:47 PM

c-po created this task.

c-po updated the task description. (Show Details)

c-po added a subscriber: zsdc.

pasik added a subscriber: pasik.Apr 1 2020, 7:00 AM

How about parallel loops?
https://metacpan.org/pod/Parallel::Loops

As an option use key -exist
Then we can get rid of these Perl checks.

root@roll01:/home/sever# ipset -N myports bitmap:port range 0-65535
root@roll01:/home/sever# ipset add myports 80
root@roll01:/home/sever# 
root@roll01:/home/sever# ipset add myports 80-100 -exist
root@roll01:/home/sever# ipset add myports 20-100 -exist

root@roll01:/home/sever# time ipset add myports 20-44000 -exist

real	0m0.007s
user	0m0.000s
sys	0m0.007s

I have investigated this a bit. Most operations for ports are doing one-by-one. Deleting as I see is always done in this way. Adding a range is done by a single command, but checking ports are doing one-by-one.
If we skip/change mentioned checking for adding ports, this should decrease initial commit time. But when we try to change/delete ports, the issue will back.
I think that there should be better to reimplement the whole firewall group section in Python, instead of fixing this logic now.

T2199 for the firewall rewrite - free for the taking. I wouldn't stray much from the old code logic, as some things have hidden meanings. Especially leaving checks out could introduce bugs unless you're absolutely sure they can be bypassed.

syncer edited projects, added VyOS 1.3 Equuleus; removed VyOS 1.2 Crux.Apr 12 2020, 1:03 PM

Viacheslav mentioned this in T1230: Improving Boot Time for Large Firewall Configurations.May 4 2020, 8:37 AM

mpueschel added a subscriber: mpueschel.Sep 21 2020, 12:16 PM

panachoi added a subscriber: panachoi.Jul 8 2021, 11:46 AM

erkin set Issue type to Bug (incorrect behavior).Aug 30 2021, 7:50 AM

erkin removed a subscriber: Active contributors.

syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.Nov 6 2021, 11:25 AM

c-po mentioned this in T4017: Adding firewall port ranges makes commit/boot MASSIVELY slow.Nov 24 2021, 6:24 PM

Viacheslav updated the task description. (Show Details)Jul 2 2022, 9:13 AM

Viacheslav mentioned this in T4558: Incomplete error message when duplicate firewall port-group used.Jul 22 2022, 12:57 AM

masterit added a subscriber: masterit.Jul 23 2022, 7:25 PM

syncer edited projects, added VyOS 1.3 Equuleus (1.3.3); removed VyOS 1.3 Equuleus (1.3.0).Aug 29 2022, 7:05 AM