Page MenuHomeVyOS Platform

VRRP transition scripts never run — two defects in src/system/keepalived-fifo.py
In progress, HighPublicBUG

Description

VRRP transition scripts never run — two defects in src/system/keepalived-fifo.py

Report body for vyos.dev. Addresses, hostnames and identities replaced with documentation
equivalents. Line numbers refer to rolling at b7e948397 (2026-08-27).


Summary

transition-script was configured on a VRRP instance for months and never executed once.
Two independent defects in src/system/keepalived-fifo.py combine to produce this. Both are
still present in rolling, and both are visible by inspection.

Neither produces any error, warning or log line. The configuration is accepted, the daemon
runs, the transitions happen — and nothing fires.


Defect 1 — the notify regex cannot match names containing :

src/system/keepalived-fifo.py:109

regex_notify = re.compile(r'^(?P<type>\w+) "(?P<name>[\w-]+)" (?P<state>\w+) (?P<priority>\d+)$', re.MULTILINE)

The name group accepts [\w-]+ — letters, digits, underscore, hyphen. It does not
accept a colon.

VyOS itself accepts colons in VRRP group names: set high-availability vrrp group foo:11 ...
commits without complaint, and foo:11 is a widespread naming convention. For any such
instance the regex never matches, notify_message is None, and the block at lines 122-142
that dispatches transition-script is never reached.

Observed, with instances named <cluster>:<vrid> and a sync group named <cluster>:

Received message: INSTANCE "cluster:999v6" MASTER 100      <- nothing follows
Received message: GROUP "cluster" FAULT 0
GROUP cluster changed state to FAULT                       <- regex matched

The logger.info at line 126 only appears for the sync group, whose name has no colon.
Instance transitions are silently dropped.

This is the worst kind of failure: the CLI accepts a configuration that its own dispatcher
cannot process, and says nothing at any point.

Design question for the maintainers

Two directions, and the choice belongs to the project:

  1. Widen the regex to the character set the CLI actually accepts for VRRP group names.
  2. Reject those names at configuration time.

Option 1 is non-breaking and fixes existing deployments silently affected today. Option 2 is
arguably more correct but breaks working configurations on upgrade. Either is better than the
current silent mismatch. I have not sent a patch precisely because this needs deciding first.


Defect 2 — the FIFO reader truncates messages

src/system/keepalived-fifo.py:150-164

def pipe_wait(self):
    self.pipe_read = os.open(self.pipe_path, os.O_RDONLY | os.O_NONBLOCK)
    while self.stopme.is_set() is False:
        time.sleep(0.250)
        try:
            message = os.read(self.pipe_read, 500)
            if message:
                for line in message.decode().strip().splitlines():
                    self.message_queue.put(line)
                self.message_event.set()

os.read() is bounded to 500 bytes and no residual buffer is kept between iterations. A
transition involving enough VRRP instances produces more than that in one burst: the read
splits mid-line, the tail of the chunk is queued as an incomplete fragment, and the head of
the next chunk is queued as another.

Observed on a pair with 14 instances in one sync group:

ROUP cluster changed state to MASTER          <- the "G" ended the previous read
Received message: INSTANCE "cl
uster:999v6" BACKUP 200

n_type then holds ROUP, the elif n_type == 'GROUP' at line 136 fails, and no script
runs. Whether a given transition fires is decided by where the 500-byte boundary happens to
fall, which makes it look intermittent.

This affects any deployment with enough VRRP instances to exceed 500 bytes of notifications in
a single transition — it is not specific to unusual configurations.

Suggested fix

Keep the incomplete trailing line and process only complete ones:

self.pipe_read = os.open(self.pipe_path, os.O_RDONLY | os.O_NONBLOCK)
buf = str()  # residual buffer: keeps the incomplete trailing line between reads
while self.stopme.is_set() is False:
    time.sleep(0.250)
    try:
        message = os.read(self.pipe_read, 500)
        if message:
            buf += message.decode()
            # only complete lines are processed, the tail is kept for the next read
            lines = buf.split('\n')
            buf = lines.pop()
            queued = False
            for line in lines:
                line = line.strip()
                if line:
                    self.message_queue.put(line)
                    queued = True
            if queued:
                self.message_event.set()

Running in production since 2026-08-27. Transition scripts fire reliably in both directions;
before the change they fired perhaps one time in three.

I can open a PR for this one if it is useful — it is self-contained and does not depend on how
defect 1 is resolved.


Interaction between the two

They must both be fixed. Working around defect 1 alone — by moving transition-script from
the instance to the sync group, whose name has no colon — makes scripts fire *sometimes*,
because defect 2 then truncates GROUP into ROUP on roughly two transitions out of three.
That intermittency is what makes this hard to diagnose from the outside.


Environment

Affected coderolling @ b7e948397, unchanged in this area for a long time
Observed on2026.06.20-0050-rolling and 2026.08.14-0025-rolling
TopologyHA pair, 14 VRRP instances (IPv4 + IPv6) in a single sync group, no-preempt, rfc3768-compatibility
Instance naming<cluster>:<vrid> — the colon that defect 1 chokes on

Both defects are demonstrable by reading the source; the logs above are the observed
manifestation rather than the proof.

Details

Version
rolling
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

rockfish added a subscriber: Viacheslav.

@Viacheslav which option would you choose for Defect 1 ? regex extension ?
I probably shoudn't have claimed this one, should I ?

if that helps, I worked on a fix for defect 2 and created a draft PR
https://github.com/vyos/vyos-1x/pull/5430

Following up on the design question, because I framed it wrong.

I assumed the choice was between widening the regex and constraining the name at configuration time. There is no constraint to restore: <tagNode name="group"> in interface-definitions/high-availability.xml.in carries no <constraint> on the node name, and neither does sync-group. VyOS accepts whatever the CLI tokenizer allows.

So [\w-]+ is not a character class missing a colon, it's a restriction with no counterpart anywhere in the schema. The defect is also wider than I originally reported: dots, spaces and slashes fail the same way.

INSTANCE "cluster:234" BACKUP 200 current: no match
INSTANCE "with.dots" BACKUP 200 current: no match
INSTANCE "with space" MASTER 100 current: no match
INSTANCE "with/slash" FAULT 0 current: no match

Option 2 would mean inventing a constraint that doesn't exist today and breaking every configuration using anything outside [\w-] on upgrade. That's not something to do in a bug fix, so I've stopped waiting on the choice and sent a patch for option 1: https://github.com/vyos/vyos-1x/pull/5468

The name is delimited by double quotes in the notify format, so the fix is to stop constraining it in that position and match on the delimiter instead:

python
regex_notify = re.compile(r'^(?P<type>\w+) "(?P<name>[^"]+)" (?P<state>\w+) (?P<priority>\d+)$', re.MULTILINE)

Still one line at a time under re.MULTILINE, still rejects a truncated fragment.

Update on #5468 (defect 1 fix).

dmbaturin caught a follow-on bug in the fix itself: widening the notify regex to accept a name with a dot in it (or a colon, space, slash - the same class defect 1 originally reported) only fixes matching that name out of the keepalived notify line. Two lines further down, the code that turns a matched name into a transition-script command built the lookup path with an f-string - f'group.{name}.transition_script.{state}' - and handed it to dict_search(), which splits on '.'. So a name containing a literal dot broke the lookup again, one call downstream of the regex, with the exact same silent no-op symptom defect 1 describes.

Pushed a follow-up commit on #5468:

  • Both lookups (group and sync-group) now use dict_search_args(), which indexes by exact key and never parses the name - correct for any character the CLI's tag-node tokenizer accepts, not only dots.
  • Extracted the lookup into one lookup_transition_script() helper instead of duplicating it across the INSTANCE and GROUP branches.
  • Guarded the script's bottom half (argument parsing, config load, FIFO creation, thread startup) behind if __name__ == '__main__':, so the module can actually be imported for unit testing. It could not be before - KeepalivedFifo.__init__() parses argv and loads the live VRRP config on construction, so importing the module used to mean running it.
  • Added src/tests/test_keepalived_fifo.py: 16 tests covering the notify regex and the transition-script lookup together, for names containing a dot, a colon, a space and a slash, plus the lookup's edge cases (no script configured, script configured for a different state, missing group key, empty config dict).

This closes the gap between "the name matches" and "the name's script actually runs" for the same character set defect 1 is about - the fix and its test coverage should now hold for the whole class, not just the colon case originally reported.

Also rebased #5468 onto rolling, so it now sits on top of #5429 and defect 2's fix (#5430), both merged since this PR was opened.

Viacheslav changed the task status from Open to In progress.Wed, Sep 16, 9:01 AM