@c-po in task T4797 (https://github.com/vyos/vyos-1x/pull/1648) you mentioned that the author should use the system task-scheduler feature.
I would understand this the same way as the author did:
The implementation of scheduling should be a module which could be reused by other features/extensions but the "cron" data should be stored separate from
what the user can schedule via the config.
But maybe you see it differently?

I probably can continue here in summer at first.

@adestis did your previous solution account for non-IP address characters in a given blocklist? For example the https://www.spamhaus.org/drop/dropv6.txt list has a bunch of stuff that would need to be ignored.

If not, to save you some trouble, I found some reference code in ipgrep that works well in processing files like above. One small edit is needed to include the ‘/‘ character in the exclusion along with swapping ip_address for ip_network. Those two changes let it handle CIDR notation.

https://github.com/n-st/ipgrep.py

Hi Giggum,
our previous solution was IPv4 only and not so nice integrated in VyOS,
therefore there are several reasons why a rework is a good idea.

I like your hint about text (comments) in the output.
I will take into account that there is not only an IP address but also comments.
We probably work with a regex, so this should not be the problem as long as there are not more than one IP per line.

IPv4 and IPv6 must also be treated differently because they need to end up in different firewall groups.

Fortunately, the script you found has an MIT license which means that we can use it and add additional code on top.

Regards
Markus

A lot of work was lost in mid of the year when my not pushed changes got lost due to a hardware failure.
So I had to do the work again.

For now, we have implemented the interface definition and i would like to discuss this before investing more time in an implementation which will not make it in the release some day.

Here is an example configuration:

firewall {
    blacklisting {
        group a_simple_example {
            destination {
                firewall-group my-blacklist
            }
            scheduling {
                task daily {
                    interval 1d
                }
            }
            sources {
                generic-source http://yet.another.url/
            }
        }
        group extended_example {
            destination {
                file /config/blacklist.txt
                firewall-group blacklist
            }
            events {
                after-update {
                    script /config/scripts/after-blacklist-update.sh {
                    }
                }
                before-update {
                    script /config/scripts/before-blacklist-update.sh {
                        arguments some-arguments
                    }
                }
            }
            filter {
                network whitelist {
                    network 192.168.0.0/24
                    network 172.16.0.0/20
                }
                network whitelist2 {
                    network 1.1.1.1/32
                }
                network whitelist3 {
                    firewall-group whitelist
                }
            }
            scheduling {
                task t1 {
                    interval 1d
                }
                task t2 {
                    interval 1h
                }
            }
            sources {
                abuseipdb {
                    confidence-level 70
                    key 1234
                }
                generic-source http://my.url/list.txt
                generic-source http://another.url/list2.txt
            }
        }
        settings {
            abuseipdb {
                key 111
            }
        }
    }
}

There were some discussions about reusing the task scheduler component.
The problem is see it that it will be hard because it is not implemented as module.

Any ideas / feedback?

Regards
Markus

I think it being called and centered around blacklisting is too specific. I'd be more inclined to see it as a firewall group, perhaps like the functionality of domain groups:

set firewall group remote-address-list <name> url <url>
set firewall group remote-network-list <name> url <url>

The nftables sets could be dynamically updated by the existing domain resolver daemon, those groups could then be available for use in standard firewall rules.

You are right with the naming we should consider this.
Regarding differentiation between network and address lists:
The current sources are returning mixed lists containing addresses and prefixes so I would not consider having both.

Since we want to add features there (which we require internally) I would suggest s.th. else.
I will discuss this internally and provide a new suggestion.

/ Markus

"blackhole" is a common term in regards of dropping traffic.

Ref:

https://www.team-cymru.com/ty/cisco-router-traditional-bogons

https://datatracker.ietf.org/doc/html/rfc7999