Page MenuHomeVyOS Platform

Implement a firewall blacklisting solution
Open, LowPublicFEATURE REQUEST

Description

Before VyOS 1.4 and the move to NFTables we (ADESTIS) had a blacklisting solution for VyOS which was based on script files and the task scheduler.
During that time it was easy to manipulate ipsets.

With VyOS 1.4 and NFTables it's different and my plan was to implement a blacklisting solution within VyOS itself
which can be configured through the VyOS config.
The plan is to create a python module for the implementation itself so that if needed it could be reused from other parts of VyOS.

In order to discuss the features I think it's a good starting point to outline the configuration structure because it makes it easier to understand the different parts:

firewall
    blacklisting
        settings
            abuseipdb
                key
        group <group-name>
            scheduling
                task <task-name>
            sources
                generic-source <url>
                abuseipdb
                    - confidence-level <level>
            filter
                network <prefix>
                firewall-group <group-name>
            destination
                file <location>
                firewall-group <group-name>
            events
                - after-update
                    - script <location>
                    - [arguments]
                - before-update
                    - script <location>
                    - [arguments]

(common entries like description and disable are not named here but planned for all nodes/tagnodes)

The plan is to have one or multiple blacklisting groups which itself can query either generic urls or specific (at the moment we only have AbuseIP DB).
The collection and aggregation of the data is done according to the schedules.
Here it would be nice to reuse the existing system task-scheduler but for the moment it could get hard because it is not implemented as module.
(So there might be a separate task to first make it modular)

The aggregated data could then be filtered to remove whitelisting entries.
The filtered data could then be pushed out to either a file or to an firewall group.
For the last option another type of firewall group is required (if https://vyos.dev/T5493 will then exists already, maybe this could be used).

I am currently planing with events so that before and after updates custom actions could be triggered.

Details

Version
-
Is it a breaking change?
Perfectly compatible
Issue type
Feature (new functionality)

Event Timeline

adestis created this object in space S1 VyOS Public.
Viacheslav changed the subtype of this task from "Task" to "Feature Request".Feb 13 2024, 3:42 PM

@c-po in task T4797 (https://github.com/vyos/vyos-1x/pull/1648) you mentioned that the author should use the system task-scheduler feature.
I would understand this the same way as the author did:
The implementation of scheduling should be a module which could be reused by other features/extensions but the "cron" data should be stored separate from
what the user can schedule via the config.
But maybe you see it differently?

I probably can continue here in summer at first.

@adestis did your previous solution account for non-IP address characters in a given blocklist? For example the https://www.spamhaus.org/drop/dropv6.txt list has a bunch of stuff that would need to be ignored.

If not, to save you some trouble, I found some reference code in ipgrep that works well in processing files like above. One small edit is needed to include the ‘/‘ character in the exclusion along with swapping ip_address for ip_network. Those two changes let it handle CIDR notation.

https://github.com/n-st/ipgrep.py

Hi Giggum,
our previous solution was IPv4 only and not so nice integrated in VyOS,
therefore there are several reasons why a rework is a good idea.

I like your hint about text (comments) in the output.
I will take into account that there is not only an IP address but also comments.
We probably work with a regex, so this should not be the problem as long as there are not more than one IP per line.

IPv4 and IPv6 must also be treated differently because they need to end up in different firewall groups.

Fortunately, the script you found has an MIT license which means that we can use it and add additional code on top.

Regards
Markus

A lot of work was lost in mid of the year when my not pushed changes got lost due to a hardware failure.
So I had to do the work again.

For now, we have implemented the interface definition and i would like to discuss this before investing more time in an implementation which will not make it in the release some day.

Here is an example configuration:

firewall {
    blacklisting {
        group a_simple_example {
            destination {
                firewall-group my-blacklist
            }
            scheduling {
                task daily {
                    interval 1d
                }
            }
            sources {
                generic-source http://yet.another.url/
            }
        }
        group extended_example {
            destination {
                file /config/blacklist.txt
                firewall-group blacklist
            }
            events {
                after-update {
                    script /config/scripts/after-blacklist-update.sh {
                    }
                }
                before-update {
                    script /config/scripts/before-blacklist-update.sh {
                        arguments some-arguments
                    }
                }
            }
            filter {
                network whitelist {
                    network 192.168.0.0/24
                    network 172.16.0.0/20
                }
                network whitelist2 {
                    network 1.1.1.1/32
                }
                network whitelist3 {
                    firewall-group whitelist
                }
            }
            scheduling {
                task t1 {
                    interval 1d
                }
                task t2 {
                    interval 1h
                }
            }
            sources {
                abuseipdb {
                    confidence-level 70
                    key 1234
                }
                generic-source http://my.url/list.txt
                generic-source http://another.url/list2.txt
            }
        }
        settings {
            abuseipdb {
                key 111
            }
        }
    }
}

There were some discussions about reusing the task scheduler component.
The problem is see it that it will be hard because it is not implemented as module.

Any ideas / feedback?

Regards
Markus

I think it being called and centered around blacklisting is too specific. I'd be more inclined to see it as a firewall group, perhaps like the functionality of domain groups:

set firewall group remote-address-list <name> url <url>
set firewall group remote-network-list <name> url <url>

The nftables sets could be dynamically updated by the existing domain resolver daemon, those groups could then be available for use in standard firewall rules.

You are right with the naming we should consider this.
Regarding differentiation between network and address lists:
The current sources are returning mixed lists containing addresses and prefixes so I would not consider having both.

Since we want to add features there (which we require internally) I would suggest s.th. else.
I will discuss this internally and provide a new suggestion.

/ Markus