Page MenuHomeVyOS Platform
Create Task
Maniphest T6902

Commit and Save takes long time when more than 100's ipsec tunnel are already configured
Open, NormalPublicBUG
Actions

Description

{F5047466}Commit and save took very long time to be executed (more then 5 mins). The running configuration contains more than 500 static routes and 200 ipsec tunnels, firewall and nat rules.
When small modification made related to the ipsec authentication, the commit took more than 2 minutes to finish the execution. Also high cpu consumption was noticed during the commit.

vyos@vpn01# show vpn ipsec authentication psk TEST
 id 6.6.6.6
>secret "secret-pwd"

vyos@vpn01# time commit

real    2m15.850s
user    0m12.903s
sys    0m25.419s

CPU Usage{F5046130}

57007       - S  54% unionfs-fuse -o cow -o allow_other /opt/vyatta/config/tmp/changes_only_56984=RW:/opt/vyatta/config/active=RO /opt/vyatta/config/tmp/new_config_56984
57303       - S  30% /opt/vyatta/sbin/my_commit

Attaching the configuration

Details

Version
VyOS 1.4.0
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

SrividyaA created this task.Wed, Nov 20, 11:29 AM
Viacheslav triaged this task as Normal priority.Wed, Nov 20, 12:21 PM
Viacheslav added a parent task: T6302: The root task for bugs and improvements related to commit time and boot.
pasik subscribed.Wed, Nov 20, 12:41 PM
SrividyaA updated the task description. (Show Details)Thu, Nov 21, 6:21 AM
marco_agostani subscribed.Wed, Nov 27, 4:27 PM

In addiction we experiment problem even with api and graphql

in the end this code took from 24 s to 80s to being execute

session_env=$(cli-shell-api getSessionEnv $$)
eval $session_env
cli-shell-api setupSession
cli-shell-api --show-working-only --show-show-defaults --show-ignore-edit showConfig
cli-shell-api teardownSession

and it seems the API call it multiple times .

jestabro subscribed.Wed, Nov 27, 4:39 PM

There is current work that will be replacing the legacy show command, removing this bottleneck; related tasks will link here to keep track of progress.

Apachez subscribed.Thu, Nov 28, 10:06 PM

Wouldnt be surprised if it turns out to be the same root cause as this is related to:

https://vyos.dev/T5388

marco_agostani added a comment.Fri, Nov 29, 9:19 AM

Dear Apachez
I observed a big load and time spent in 4 unionfs procs.
So probably you're point in hte right direction

marco_agostani added a comment.Mon, Dec 2, 9:39 AM

Investigating a little bit show big time diff here

cli-shell-api --show-active-only --show-show-defaults --show-ignore-edit showConfig

took an average of 3 secs

cli-shell-api --show-working-only --show-show-defaults --show-ignore-edit showConfig

took always more than 24 secs
it seems something related to this call in Cpp code (cnode-algorithm.cpp)

wroot.reset(new CfgNode(*cstore, rpath, false, true));