Page MenuHomeVyOS Platform
Create Task
Maniphest T4062

VRRP IPSEC-AH : sequence number xxxxxxx already processed. Packet dropped. Local(xxxxxxx)
Resolved (N/A)PublicBUG
Actions

Description

Hi guys,

Seems there is a bug in VRRP, both nodes entered MASTER state rendering the VRRP address unavailable.
In the log there are two main lines that popout

Dec 09 02:07:27 Keepalived_vrrp[3090]: (IPv4_VLAN99) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 09 07:28:19 Keepalived_vrrp[3090]: (IPv4_VLAN10) Entering MASTER STATE
Dec 09 07:28:19 Keepalived_vrrp[3090]: (IPv4_VLAN10) IPSEC-AH : sequence number 7626336 already processed. Packet dropped. Local(7626336)

As soon as I reboot the first node everything started working again on node 2 as expected and the master was able to take over again.

vyos@EU-GW03:~$ show version

Version:          VyOS 1.3.0-epa3
Release train:    equuleus

Built by:         Sentrium S.L.
Built on:         Sun 31 Oct 2021 17:38 UTC
Build UUID:       383e45ad-b32a-4359-8183-9baacc8e69d9
Build commit ID:  bb511522cc3bb2-dirty

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-42 39 f0 01 50 fb b9 8f-6d 44 a4 9d be a0 66 17
Hardware UUID:    4239f001-50fb-b98f-6d44-a49dbea06617

Copyright:        VyOS maintainers and contributors

VRRP config

vyos@EU-GW03:~$ show configuration commands | strip-private | match vrrp
set high-availability vrrp group IPv4_VLAN10 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN10 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN10 interface 'eth1.10'
set high-availability vrrp group IPv4_VLAN10 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN10 priority '200'
set high-availability vrrp group IPv4_VLAN10 virtual-address 'xxx.xxx.10.1/24'
set high-availability vrrp group IPv4_VLAN10 vrid '10'
set high-availability vrrp group IPv4_VLAN75 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN75 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN75 interface 'eth1.75'
set high-availability vrrp group IPv4_VLAN75 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN75 priority '200'
set high-availability vrrp group IPv4_VLAN75 virtual-address 'xxx.xxx.75.1/24'
set high-availability vrrp group IPv4_VLAN75 vrid '75'
set high-availability vrrp group IPv4_VLAN98 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN98 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN98 interface 'eth1.98'
set high-availability vrrp group IPv4_VLAN98 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN98 priority '200'
set high-availability vrrp group IPv4_VLAN98 virtual-address 'xxx.xxx.98.1/24'
set high-availability vrrp group IPv4_VLAN98 vrid '98'
set high-availability vrrp group IPv4_VLAN99 authentication password xxxxxx
set high-availability vrrp group IPv4_VLAN99 authentication type 'ah'
set high-availability vrrp group IPv4_VLAN99 interface 'eth1.99'
set high-availability vrrp group IPv4_VLAN99 preempt-delay '180'
set high-availability vrrp group IPv4_VLAN99 priority '200'
set high-availability vrrp group IPv4_VLAN99 virtual-address 'xxx.xxx.99.1/24'
set high-availability vrrp group IPv4_VLAN99 vrid '99'
set high-availability vrrp group IPv4_WAN authentication password xxxxxx
set high-availability vrrp group IPv4_WAN authentication type 'ah'
set high-availability vrrp group IPv4_WAN interface 'eth0'
set high-availability vrrp group IPv4_WAN preempt-delay '180'
set high-availability vrrp group IPv4_WAN priority '200'
set high-availability vrrp group IPv4_WAN virtual-address 'xxx.xxx.84.192/25'
set high-availability vrrp group IPv4_WAN vrid '1'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN10'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN75'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN98'
set high-availability vrrp sync-group VLAN member 'IPv4_VLAN99'
set high-availability vrrp sync-group VLAN member 'IPv4_WAN'{F2220989}

Greetings,

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.3.0-epa3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

boevering created this task.Dec 9 2021, 8:33 AM
pasik added a subscriber: pasik.Dec 9 2021, 10:23 AM
m.korobeinikov changed the task status from Open to Needs testing.Dec 10 2021, 12:51 AM
Viacheslav added a subscriber: Viacheslav.Dec 21 2021, 6:37 PM

@boevering Do you know how to reproduce it?

boevering added a comment.EditedDec 22 2021, 3:50 PM

@Viacheslav the only way is by letting it run.
As adviced in the slack I upgraed to differt version, just now it dropped again.
This time it's differtent as the backup still sayes it still the backup node but all traffic to the VRRP address is offline.

Master: Version:          VyOS 1.3-beta-202112090443
Slave: Version:          VyOS 1.4-rolling-202112090318

In the logging of the Master node:

vyos@EU-GW03:~$ show log vrrp
-- Logs begin at Wed 2021-12-22 08:46:04 CET, end at Wed 2021-12-22 16:43:56 CET. --
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN99) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780985)
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN10) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780985)
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN75) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780985)
Dec 22 16:15:23 Keepalived_vrrp[2976]: (IPv4_VLAN98) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780985)

Log of the slave

Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN75) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN10) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN98) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 07:12:28 Keepalived_vrrp[3130]: (IPv4_VLAN99) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
Dec 22 16:14:46 Keepalived_vrrp[3130]: A thread timer expired 4.842168 seconds ago
Dec 22 16:14:46 Keepalived_vrrp[3130]: (IPv4_VLAN99) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: VRRP_Group(VLAN) Syncing instances to MASTER state
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN10) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) Entering MASTER STATE
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN99" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN99 changed state to MASTER
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN98) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_WAN) Entering MASTER STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780969)
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) Master received advert from 10.10.75.253 with higher priority 200, ours 100
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN75) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: VRRP_Group(VLAN) Syncing instances to BACKUP state
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN10) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN98) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN99) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_WAN) Entering BACKUP STATE
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN99) IPSEC-AH : sequence number 8780969 already processed. Packet dropped. Local(8780969)
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN10) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780966)
Dec 22 16:14:56 Keepalived_vrrp[3130]: (IPv4_VLAN98) IPSEC-AH : sequence number 8780966 already processed. Packet dropped. Local(8780966)
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN10" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN10 changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN75" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN75 changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN98" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN98 changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_WAN" MASTER 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_XS4ALL changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: GROUP "VLAN" MASTER 0
Dec 22 16:14:56 keepalived-fifo.py[3144]: GROUP VLAN changed state to MASTER
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN75" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN75 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN10" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN10 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN98" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN98 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_VLAN99" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_VLAN99 changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: INSTANCE "IPv4_WAN" BACKUP 100
Dec 22 16:14:56 keepalived-fifo.py[3144]: INSTANCE IPv4_XS4ALL changed state to BACKUP
Dec 22 16:14:56 keepalived-fifo.py[3144]: Received message: GROUP "VLAN" BACKUP 0
Dec 22 16:14:56 keepalived-fifo.py[3144]: GROUP VLAN changed state to BACKUP
syncer edited projects, added VyOS 1.3 Equuleus (1.3.2); removed VyOS 1.3 Equuleus.Aug 14 2022, 1:05 PM
dmbaturin edited projects, added VyOS 1.3 Equuleus (1.3.3); removed VyOS 1.3 Equuleus (1.3.2).Aug 14 2022, 6:32 PM
syncer edited projects, added VyOS 1.3 Equuleus (1.3.4); removed VyOS 1.3 Equuleus (1.3.3).Jul 12 2023, 9:46 PM
syncer edited projects, added VyOS 1.3 Equuleus (1.3.5); removed VyOS 1.3 Equuleus (1.3.4).Aug 25 2023, 9:29 PM
syncer edited projects, added VyOS 1.3 Equuleus (1.3.6); removed VyOS 1.3 Equuleus (1.3.5).Dec 17 2023, 11:36 PM
Viacheslav closed this task as Resolved N/A.Jan 20 2024, 3:04 AM

There are no other reports
close it

Reopen if you know the steps to reproduce it.