When building VyOS then mksquashfs is being used to create the filesystem.squashfs file.
However since the syntax is compiled dynamically (by vyos-live-build/scripts/build/binary_rootfs) its hard to tell afterwards what the full syntax of mksquashfs really were to create that squashfs-file.
Suggestion to add some logging at around https://github.com/vyos/vyos-live-build/blob/current/scripts/build/binary_rootfs#L324 what the variable ${MKSQUASHFS_OPTIONS} ultimately ended up to be.