I finally managed to narrow this down further. This problem is caused by enabling the extended nexthop capability. FRR intentionally sends RAs when this capability is enabled, althought so far I don't understand why. I opened a discussion in the FRR repo: https://github.com/FRRouting/frr/discussions/15994

I don't think this is a bug in FRR, but rather a configuration issue. I'm not really familiar with FRR, but as far as I can tell, sending RAs is an intended feature of it that can be disabled per interface: https://docs.frrouting.org/en/latest/ipv6.html#clicmd-ipv6-nd-suppress-ra
Not sure why it's enabled by default, since I couldn't find anything in the generated /etc/frr/frr.conf that would enable it.

It works as expected now on 1.4-rolling-202211190627, but my system failed to boot with the old key types in the config, so I had to remove them before switching to the new image. Thanks for the quick fix!