Page MenuHomeVyOS Platform

Add capability to use local and external dynamic-lists for firewall rules but also for various policies such as access-list, route-maps etc.
Closed, ResolvedPublicFEATURE REQUEST

Description

It would be nice if it would be possible to add dynamic lists to VyOS firewall regarding address, network, interface, port, mac and domain groups.

Similar to the "External Dynamic List" feature of PaloAlto Networks firewalls:

https://docs.paloaltonetworks.com/pan-os/11-0/pan-os-admin/policy/use-an-external-dynamic-list-in-policy

The purpose is that you then dont have to create something that interacts with the HTTP API of the VyOS but rather just dump a plain textfile either onto the VyOS box or the VyOS box will fetch this plain file.

Example for sources who uses plain files are https://www.team-cymru.com/bogon-reference-http among others.

Other usecases are various SEIM systems who can export a plain text file of ports, address, networks, as-numbers etc to be blocked (or allowed).

For VyOS the dynamic list could either be loaded locally (someone uploads a textfile to lets say /config/custom directory) or externally (the VyOS box fetches the file from external source through scp, sftp, ftps, http, https...).

Suggestion something like:

set firewall dynamic-list <NAME> source 'local' type 'network' path '/config/custom/blacklist_network.txt'

set firewall dynamic-list <NAME> source 'https' type 'interface' path 'https://example.com/blacklist/interface.txt'

There should also be possible to define how often these lists should be updated. Either dynamically like "every 1 minute" or statically "at 14:15:00 every day", preferly syntax from crontab could be used for this such as:

set firewall dynamic-list <NAME> refresh '15 14 * * *'

For commits VyOS would use the local cache of these files and if the cached file doesnt exist it will fetch it.

There should also for op-mode exist a way to manually refresh these caches upon request:

update firewall dynamic-list all

update firewall dynamic-list <NAME>

Same with some op-mode to refresh the firewall itself once the lists have been updated:

update firewall ruleset

A spinoff of this feature request would be that the dynamic-lists should also be accessible for use by routing protocols:

set policy dynamic-list <NAME> source 'local' type 'access-list' path '/config/custom/blacklist_access-list.txt'

set policy dynamic-list <NAME> source 'https' type 'route-map' path 'https://example.com/blacklist/routemap.txt'

set policy dynamic-list <NAME> refresh '15 14 * * *' action 'access-list <NAME>'
...
set policy dynamic-list <NAME> refresh '15 14 * * *' action 'large-community-list <NAME>'

update policy dynamic-list all

update policy dynamic-list <NAME>

update policy access-list <NAME>
update policy prefix-list <NAME>
update policy route <NAME>
update policy route6 <NAME>
update policy route-map <NAME>
update policy local-route <NAME>
update policy local-route6 <NAME>
update policy as-path-list <NAME>
update policy community-list <NAME>
update policy extcommunity-list <NAME>
update policy large-community-list <NAME>

The "update policy" op-mode is to recommit the policy once the "update policy dynamic-list" have completed its refresh of the dynamic-list.

Basically the same function as being called by "set policy dynamic-list <NAME> refresh".

Details

Version
-
Is it a breaking change?
Perfectly compatible
Issue type
Feature (new functionality)

Event Timeline

Note that the PR for T4797 was never updated for requested changes:
https://github.com/vyos/vyos-1x/pull/1648

On the other hand, it is agreed that this would be a useful feature: @Apachez if you are able/willing to provide an alternative/updated PR, we will close the previous one.

@Apachez I would need this feature in another feature (https://vyos.dev/T6040) to avoid a double implementation.

Unknown Object (User) subscribed.Apr 20 2024, 11:42 AM
syncer lowered the priority of this task from High to Wishlist.May 19 2024, 8:04 AM
syncer edited projects, added VyOS 1.5 Circinus; removed VyOS 1.4 Sagitta (1.4.0-GA).

@Apachez could you please assign this task to me?

I've submitted a PR (https://github.com/vyos/vyos-1x/pull/4326) to implement this feature using the logic described below by @sarthurdev:

I think it being called and centered around blacklisting is too specific. I'd be more inclined to see it as a firewall group, perhaps like the functionality of domain groups:

set firewall group remote-address-list <name> url <url>
set firewall group remote-network-list <name> url <url>

The nftables sets could be dynamically updated by the existing domain resolver daemon, those groups could then be available for use in standard firewall rules.

The configuration syntax in my implementation is slightly different to your suggestions, but still implements a majority of the functionality.

Viacheslav changed the task status from Open to In progress.Jan 31 2025, 11:36 AM
Viacheslav assigned this task to Embezzle.
dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.
Embezzle changed the task status from In progress to Needs testing.Mar 25 2025, 6:21 PM

Tested as working in: VyOS 2025.03.26-0018-rolling

Thank you very much @Embezzle - This is a feature I've been hoping for for a long time (at the moment I have a script that runs via VyOS cron and updates a firewall filter and commits it) - really appreciate you taking the time to do this piece of work.
Much Love.

Any chance this might be backported to 1.4 ?

We tested the solution implemented by @Embezzle and it seems to work.

But we found a problem which might come from the fact that we have tested it with over 90000 entries.
The command show firewall group DEMO detail fails with the following error when you stop displaying the results by pressing q:

Traceback (most recent call last):
  File "/usr/libexec/vyos/op_mode/firewall.py", line 794, in <module>
    show_firewall_group(args.name)
  File "/usr/libexec/vyos/op_mode/firewall.py", line 699, in show_firewall_group
    output_firewall_vertical(rows, header, adjust=False)
  File "/usr/libexec/vyos/op_mode/firewall.py", line 188, in output_firewall_vertical
    print(tabulate.tabulate(transformed_rule, tablefmt="presto"))
BrokenPipeError: [Errno 32] Broken pipe

It works when you scroll to the end (by entering G) and then press q or just return.

It seems like it's not related with the changes itself but by the general output.

Still, I hope to see the future soon in a release channel like the Stream Releases.

Regards
Markus

@Embezzle you are only supporting IP addresses but no prefixes as 192.0.0.8/24 ?
Do you have plans to also support networks/prefixes?

A lot of the original proposed features would be left aside if the implementation would stop here.
Or maybe I am the only one complaining here.

@Embezzle you are only supporting IP addresses but no prefixes as 192.0.0.8/24 ?
Do you have plans to also support networks/prefixes?

What makes you say this? The PR smoketests show a /24 example.
Comments later in the PR show subnet mask examples etc.

You are right, when a correct "base" address for the prefix is used.
If not the prefix is silently discarded (192.0.0.0/24 vs. 192.0.0.8/24).
I am ok with it. Just important for documentation.

You are right, when a correct "base" address for the prefix is used.
If not the prefix is silently discarded (192.0.0.0/24 vs. 192.0.0.8/24).
I am ok with it. Just important for documentation.

It is probably worth noting that it is not just remote-groups that will not accept this format, it will also be rejected by network-groups:

vyos@vyos# set firewall group network-group test network 192.0.0.8/24

  Error: 192.0.0.8/24 is not a valid IPv4 prefix



  Invalid value
  Value validation failed
  Set failed