When building up my firewall rules I made an address and network group the same name, which it also allowed.

After creating the network group and doing commit it bugged out with the message:

ipset v6.38: Kernel support protocol versions 6-7 while userspace supports protocol versions 6-6
Set cannot be destroyed: it is in use by a kernel component
Error: type mismatch [address] [network]

I had to actually remove the network group, commit, remove the address group with the same name. And then recreate the network group.