Downloaded the latest rolling, the only thing I have done with the rolling was installing it on a fresh Proxmox VM. I created two firewall groups with the same name - one for address-group and the other is for port-group.

+    group {
+        address-group emby {
+            address 10.0.7.10
+        }
+        port-group emby {
+            port 8096
+            port 8920
+        }
+    }

When I tried to commit, I got the error below. However, I was able to commit and save.

vyos@vyos# commit
[ firewall group address-group emby ]
Error: type mismatch [port] [address]

When creating a firewall name, the error below is what I got. Also, I get a commit failed.

vyos@vyos# compare 
[edit firewall]
+name trust_to_server {
+    default-action drop
+    rule 100 {
+        action accept
+        destination {
+            group {
+                address-group emby
+                port-group emby
+            }
+        }
+        source {
+            address 10.0.11.0/24
+        }
+    }
+}
[edit]
vyos@vyos# commit
[ firewall name trust_to_server rule 100 destination group address-group emby ]
Group [emby] has not been defined

[[firewall name trust_to_server]] failed
Commit failed
[edit]
vyos@vyos# 
[edit]
vyos@vyos#

I discarded the candidate config; deleted the "port-group emby" and created a new "port-group emby_app". Then I recreated the same firewall rules and I got the same commit error above despite renaming the port-group. The odd part is the "address-group emby" exist.

The workaround that I find is to delete the "address-group emby" then commit and save. Recreate the "address-group emby" then commit and save. Recreate the same firewall rules with the new port-group emby_app then commit and save. The workaround let me commit the firewall rules. It seems to be related to the config.boot. I have to keep saving the changes I made; otherwise, it would fail to commit.

The main problem is that you use the same name for different group types.

ipset -N emby bitmap:port --from 1 --to 65535
ipset -A emby 8096
ipset -A emby 8920

What we get

root@r5-roll:/home/vyos# ipset -L
Name: emby
Type: bitmap:port
Revision: 3
Header: range 1-65535
Size in memory: 8264
References: 0
Number of entries: 2
Members:
8096
8920

So now it trying to set IP address in the group for "Type: bitmap:port" or trying to recreate the group with type "Type: hash:ip"

# ipset -N emby hash:ip family inet
ipset v6.38: Kernel support protocol versions 6-7 while userspace supports protocol versions 6-6
Set cannot be created: set with the same name already exists

So we should prevent to use the same names for different types of groups.
The names should be unique.