Page MenuHomeVyOS Platform

External address/network lists for firewall (Local and remote)
In progress, WishlistPublicFEATURE REQUEST

Description

I'm thinking of creating a PR for this feature but wanted to discuss it a bit since it will require config, code, scheduled task and documentation.

My thought for config is (I'm using CINSARMY as an example but it could be any txt file that is one entry per line, Cisco Talos, 3coresec, Spamhause, blocklist.de, etc etc)

firewall {
  group {
    external-list CINSARMY {
      url https://cinsscore.com/list/ci-badguys.txt
      interval 12h
    }
    external-list MANUALBANS {
      url file:///config/user-data/ip-bans.txt
      interval 10m
    }
  }
  name OUTSIDE-IN {
    default-action-drop
    rule 1 {
      action drop
      source {
        group {
          external-list CINSARMY
        }
      }
    }
  }
}

it would be url of file, url would use curl to fetch the file on interval, file would just load the local file from /config/user-data/

both would be processed and put into 1 of 2 groups, ipv4, ipv6 since community lists like CINS can have a mix

It would create nftable groups for both types using name just like address and network groups, so L_CINSARMY, L6_CINSARMY, etc etc. and you populate them on interval.

Then in the config we would need to allow using them in rules. In the VyOS config we just need one entry both variants which simplifies the config for the user and allows for shorter config display since lists like CINSARMY is 30k entries which make show unusable.

I've currently build a bash script that does most of this work, but I'd love to add this functionality directly into VyOS for speed (Python) and future proofing.

I'd like to open this for discussion to make sure the value is worth the effort before I start work.

  • Is the feature something the community would like to see?
  • Does proposed the config layout make sense?
  • Does the backend flow makes sense?
  • Would there be a good show firewall op to list the tables or search for an IP for debugging?
  • Does nftables have a swap action like ipset has, if so I'd build the tables in tmp groups and swap and remove the tmp as this would be faster (Need to research this)
  • Anything else I forgot to mention or consider for this task?

UPDATE 1:

  • Renamed item from blocklist to list as @n.fort pointed out, it could be used for more then blocklists
  • only ipv4 and ipv6 needs to be separated, nft don't need to separate address and network this most be relics from the ipset switch to nft
  • Determined this PR will be very similar to geoip, with the exception of each list will require it's on cronjob based on interval, reusing the system task-scheduler code and adding one param for the output file should be able to handle this very easily. We would only reuse the backend codes as to not bloat the config, thus tasks will not show under system task-scheduler in the config.

UPDATE 2:

  • in the PR it was asked to move the task-scheduler logic into the python vyos package, this is now complete and part of the PR
  • in the PR I was asked to change the lists category to external-list which is a WIP, was hoping more would weigh in on this change before I executed it, but it's all I have left now

Details

Difficulty level
Normal (likely a few hours)
Version
1.4
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible
Issue type
Feature (new functionality)

Event Timeline

TheSin- created this object in space S1 VyOS Public.

From my point of fiew, looks interesting.
The proposed structure and behaviour doesn't look that different than what is currently in geoip filtering: external URLs with data, and sync from time to time.

Under the hood, I would suggest to only create one group in nftables: type ipv4_address with flag interval (create address and groups in any nighltly and check nftables sets to see what I mean)

I don't agree on naming "set firewall group blocklist", since may be used for accepting connections rather than dropping them.

I didn't look deep into the nft groups, so I wasn't sure if we could mix ipv4/6 and addresses and networks, if we can then I agree one group would be best, though I'm sure ipv4/6 would still need to separate but checking each line for : makes that task super easy and fast.

I'd like to have a per list interval though, some like local files you might want checked an updated way more often then online lists, and in some cases there maybe upstream limits on how often you can check for updates. I will certainly look at the geoid code for this as it does sound very similar.

You are correct the name blocklist might not be best as there could be other use cases, maybe just list?

sarthurdev triaged this task as Wishlist priority.Nov 3 2022, 7:44 PM
sarthurdev changed the subtype of this task from "Task" to "Feature Request".
sarthurdev added a project: VyOS 1.4 Sagitta.
sarthurdev added a subscriber: sarthurdev.

After a few hours of digging I do think this request would be very similar to geoip, only ipv4, and ipv6 groups would be required per list.

The notable exception is that unlike geoip, each list could have it's own interval as previously stated, and each list can be including in rules, but we should also allow for lists to be nested into groups.

Most of this work is pretty straight forward, the intervals section might be the hardest part unless we reuse the tasks code and just add the calls to a separate crontab, vyos-lists or something which would allow for each list to have an interval. The script would just call list-update with a specific list name as a param. Just an initial thought on how to handle it. This might require the internal tasks scheduler functions to be refactored a little to add a param to specify the cronjob file.

PR was added, I'm just trying to learn the documentation system now. Though to be frank documentation has never been my strong suit.

TheSin- changed the task status from Open to In progress.Nov 8 2022, 9:01 PM
TheSin- claimed this task.
TheSin- changed Difficulty level from Unknown (require assessment) to Normal (likely a few hours).
TheSin- changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.

In the PR it was requested to change the group category from lists to external-list which I'm fine with, but before I do the work to rename files and fields, does everyone agree with this change?

task-scheduler logic was moved into vyos.task_scheduler so it can be imported properly and used by other modules

TheSin- renamed this task from Blocklists (Local and remote) to External address/network lists for firewall (Local and remote).Nov 9 2022, 5:56 PM
TheSin- updated the task description. (Show Details)

list/lists in config and op-mode now moved to external-list

Added file:// parser to vyos.remote.download and used that to simplify the code, no need to check if it's local now.

I'd be happy to test anything that implements this. Previously, I (judging from the forum, I'm not the only one) using this EdgeOS-BL-Mgmt with 1.3.x.

It would be great if there was something ending up in 1.4 that implements a simple way to add a network group filled with lists that are maintained externally; important would be the ability to have multiple URLS which are then merged, so that one could use a single firewall rule with the group.

@TheSin- we also need this feature because our old blacklisting implementation is no longer working since the new nft implementation replaces sets after changes in the config.
So we have to go the official way and I just stumbled over your feature request when I started to make my own implementation.
Even when I have not implemented any core parts for VyOS yet, I offer my help with the implementation of this feature.