I just found out it was possible to use UDP Unicast for Conntrack-Sync instead of the Multicast I was using.
I changed my configuration from Multicast to Unicast and committed the changes. Removed the mcast-group command and added "peer <x.x.x.x>" to the end of the Interface command.
However this didn't make "show conntrack-sync" show Unicast, it still showed Multicast.
So I did a "restart conntrack-sync" but this didn't work either. Doing a "ps afux" showed still a very early (boot) PID for /usr/sbin/conntrackd
Only a "sudo kill <pid of conntrackd>" killed it (so it wasn't running at all)
Then I had to manually restart it with "restart conntrack-sync"
This changed Conntrack Sync to actually use UDP instead of Multicast.
I verified this both with
a) Taking TCP dumps on other hosts and still seeing Multicast traffic
b) The "show conntrack-sync statistics" command was saying it was Multicast traffic, but now I've fix it it states UDP Traffic.
I think that any changes to the conntrack-sync section should result in a proper restart of conntrack-sync, or at the very minimum "restart conntrack-sync" should properly restart it, not require me to kill the existing conntrackd before issuing it.