It seems like when offloading is used, and VLAN interfaces is added to the flowtable, the system fails to load the entire firewall section during boot.
By changing this in my config:
set firewall flowtable OFFLOAD interface 'eth0' set firewall flowtable OFFLOAD interface 'eth1' set firewall flowtable OFFLOAD interface 'eth1.10' set firewall flowtable OFFLOAD interface 'eth1.20'
To:
set firewall flowtable OFFLOAD interface 'eth0' set firewall flowtable OFFLOAD interface 'eth1'
Got it working, so either it should be fixed, so VLAN interfaces work, of the CLI should prevent adding VLAN interfaces to flowtables in the first place.
With the VLAN interfaces defined, the following error is thrown at startup: Starting VyOS router: migrate configure failed!