## NOTE: the apt configuration was taken from the vyos/vyos-build:current docker -> /etc/apt/sources
sudo sh -c "echo '
# deb http://snapshot.debian.org/archive/debian/20220228T000000Z bullseye main
deb http://deb.debian.org/debian bullseye main
# deb http://snapshot.debian.org/archive/debian-security/20220228T000000Z bullseye-security main
deb http://security.debian.org/debian-security bullseye-security main
# deb http://snapshot.debian.org/archive/debian/20220228T000000Z bullseye-updates main
deb http://deb.debian.org/debian bullseye-updates main
' > /etc/apt/sources.list"

sudo apt update
sudo apt install linux-tools
# sudo ln -sf /usr/lib/linux.*tools.*/perf /usr/bin/perf

/usr/libexec/vyos/init/vyos-router:
--------------------------------------------------
# load the initial config
load_bootfile ()
{
    log_progress_msg configure
    (   
        if [ -f /etc/default/vyatta-load-boot ]; then
            # build-specific environment for boot-time config loading
            source /etc/default/vyatta-load-boot
        fi

        if grep -q vyos-debug /proc/cmdline; then
            sudo sysctl -w kernel.perf_event_paranoid=-1
            sudo sysctl -w kernel.kptr_restrict=0 
            echo 0 > /tmp/vyos-config-status

            sg ${GROUP} -c "time $vyos_libexec_dir/vyos-boot-config-loader.py $BOOTFILE" &
            ps aux | grep vyos-boot-config-loader

            _pid=$(pgrep -f "python3.*vyos-boot-config-loader.py")
            perf record --max-size 500M -p $_pid  \
                         -g --call-graph dwarf   \
                         -o /home/vyos/perf.data \
                         -- sleep 60 &
        else
            sg ${GROUP} -c "$vyos_libexec_dir/vyos-boot-config-loader.py $BOOTFILE"
        fi
    )
}
------------------------------------------------
Technical nuances:
1. It can be assumed that the profiler will generate ~10MB/sec of samples for the config loader,
        meaning -> 2 min config load ~ 1.5G samples
   You should remember the amount of system available memory to avoid unpleasant situations.
   Processing 2G of samples takes > 0.5G RAM. Otherwise, perf gets OOM killed when performing
        `perf script --header > perf.hist`

2. Perf can analyze the subprocess or attach to one already active.
   Instead of running in the foreground, we send the loader process to the background.
   Using the '--max-size' flag becomes problematic if you run a command as a subprocess.
   Perf will kill the loader once the size limit is reached. However, we only want to collect the samples.

   Therefore, we attach it to the loader already running in the background.
   In addition, I'm providing the `sleep`  subprocess to limit sampling in time.
   I have also seen a strange bug when directly attaching perf would fail to terminate correctly,
  s meaning that collected data is not usable.

3.       echo 0 > /tmp/vyos-config-status
   This is a trick around the systemd, so it considers the initialization successful and drops you
   to the bash prompt.

4. Perf flags:
   If both --max-size and `sleep ..` is provided, the process is terminated on the first one to become true.
   You can avoid profiling the program initialization by delaying the start with the `-D <ns>` flag.

5. I'm using `time` to get bash execution metrics; therefore, you will see multiple similar entries in
   the process table. To get the right one, include python in the grep.
        pgrep -f "python3.*vyos-boot-config-loader.py"

- The boot process has failed, the prompt is stuck, bash is not initialized
- You reboot the device, it tries to read the config, fails once again as there is no free memory
- Config was not loaded, you cannot log in; it is a loop

- Stop on grub, edit the menu entry, append `init=/bin/bash` to the `linux` field
- Boot into the root shell and remove the generated dumps or edit grub config to disable profiling
- Reboot into the regular system

sudo modprobe nbd max_part=10
sudo qemu-img resize VyOS-hda.qcow2 +5G
sudo qemu-nbd -c /dev/nbd0 VyOS-hda.qcow2
sudo gparted /dev/nbd0
# extend the used partition
# https://linuxconfig.org/how-to-resize-a-qcow2-disk-image-on-linux

disabled configure || load_bootfile
log_end_msg $?

1. Your `perf script --header` may return a bunch of errors like
    `DWARF data block is larger than specified  (number) < (number)`
    It likely means you have messed up something in debug flags, like `-ggdb3` instead of `-ggdb`
    Just git clean + rebuild the packages before testing.
    TODO: test dwarf specific flags
2. If the package got stripped during compilation, the perf would track only the kernel stack.

1. You can generate a standart flamegraph 
    $ git clone https://github.com/brendangregg/FlameGraph
    $ cd FlameGraph
    $ ./stackcollapse-perf.pl --all < perf.hist  > out.folded
    $ ./flamegraph.pl --hash  --bgcolor=green --count=pages < out.folded > out.svg
   You will likely want to open this file in browser so the interactive features are working
       i.e., zoom-in/out on stack frame, interactive search, highlight ...
   However, it is pretty basic and not as functional as alternatives.
2. `profiler.firefox.com` is the most convenient/comprehensive profiler I have found.
    Features that I have found useful:
    - Call Tree / Flame Graph / Stack Chart
    - Local data hosting. Analyzed profiles can be easily shared between developers/users.
      The project user can try to analyze the collected samples locally and share them in 1 click if developer support is needed.
      I.e., the default config-sample: https://share.firefox.dev/3LDZ1B6
    - Interactive stack visualization widget with call stack preview on mouse hover
      Click on the function, and it will be highlighted on the widget
    - Accumulation of trace time for the stack frame
    - Focus on the function filter. Displays only samples with chosen function.
      It makes it easy to see the used time on the widget + you can hover on the inactive process area
      to see what was happening outside of the main processing logic.
    - Interactive search filter synced with stack preview widget
    - Convenient keyboard controls
3.  `www.speedscope.app`. Features close to the previous, time-ordered call stack instead of the widget



Here are some of the metrics that I`m considering for this analysis:
1. The `time` summary, can be found at `artifacts.tar.gz -> /*/vyatta/cfg-stdout.log`
2. Difference against the cumulative time logged in `vyatta/cfg-stdout.log`
3. Time spent on debug print itself + timer manipulations
   output_internal             | debug print
   high_resolution_clock::now  | timer
4. Main processing points
   commit::doCommit            | main processing function
   _commit_exec_prio_subtree   | tree processing
                               | vyatta-cfg/src/commit/commit-algorithm.cpp:662
   \ _commit_check_cfg_node    | vyatta-cfg/src/commit/commit-algorithm.cpp:673
   \ _commit_exec_cfg_node     | vyatta-cfg/src/commit/commit-algorithm.cpp:674
      \ _exec_node_actions     | vyatta-cfg/src/commit/commit-algorithm.cpp:295
         \ _executeTmplActions | template execution
                               | vyatta-cfg/src/cstore/cstore.cpp:2035
           \_system_out        | method that evaluates code from node.def files, i.e.
                               |   run, syntax:expression:, commit:expression: 
5. Anything outside the `commit::doCommit`

Accumulated time for template processing: 4.532s
NOTE:  4.5sec is `real-time`, meaning it includes `user + sys + sleep`

Process time
    May 01 21:20:01 vyos vyos-router[1315]: real        0m6.176s
    May 01 21:20:01 vyos vyos-router[1315]: user        0m1.630s
    May 01 21:20:01 vyos vyos-router[1315]: sys        0m1.250s

 Time spent for debugging purposes (focust on function)
                       ::now < 100ms
    ::output_internal = 1s

Looking at these metrics, we can assume that config tree initialization is a relatively short process compared to template execution.
It took us 4.5s for templates and the rest 1.7 for initialization and the following fs sync.

    commit::doCommit 
    Traced running time : 4.7sec
    In Kernel space     : 64% ~ 3sec
    In User space       : 36% ~ 1.7sec
    |\
    | \ _commit_exec_prio_subtree
    |   Traced running time : 4.6sec
    |   In Kernel space     : 60%
    |   In User space       : 40%
    |
    |\
    | \ _commit_check_cfg_node
    |   Traced running time : 2.6sec
    |   In Kernel space     : 65%
    |   In User space       : 35%
    |   \
    |   \ _exec_node_actions
    |     Traced running time : 2.6sec
    |     In Kernel space     : 65%
    |     In User space       : 35%
    |
    |\
    | \ _commit_exec_cfg_node
        Traced running time : 1.9sec
        In Kernel space     : 60%
        In User space       : 30%
        \
        \ _exec_node_actions
          Traced running time : 1.85sec
          In Kernel space     : 57%
          In User space       : 43%

--------------------------------------------------------------------------
    _exec_node_actions
    Traced running time : 4.5sec
    In Kernel space     : 63%
    In User space       : 37%
    |
    |\
    | \ _executeTmplActions
    |   Traced running time : 4sec
    |   In Kernel space     : 67%
    |   In User space       : 33%
    |   \
    |   \ execute_list                   <=== function traced by VYOS_DEBUG
    |     Traced running time : 2.5sec
    |     In Kernel space     : 84%      <====== main processing
    |     In User space       : 16%
    |
    |\
    | \ cstore::Cstore::output_internal  <====== function printing to the file
    |   Traced running time : 1sec
    |   In Kernel space     : 50%
    |   In User space       : 50%
    |
    |\            
       \ std::tr1::shared_ptr::~shared_ptr
          It takes 450ms to deallocate tmp (?) `Cpath` objects used in the function

5. Anything outside the `commit::doCommit.`
   Before the method - two milliseconds to initialize the config tree objects
   Before the finish - 0.5s system call `clone,` likely the fs update after loading

In Attachment vyos configuration with 298 Vlans and 298 DHCP servers.

Every commit takes about 11 seconds.
A reboot takes about 12 minutes to load all vyos configurations.

Accumulated time for template processing: 214.248s => 3m34s
Process time
real        7m12.155s
user        1m57.035s
sys        2m6.238s

We have spent  only half of the processing time for the template execution.
The question is what was happening the rest of 3m and if there this can be optimized.

https://github.com/jestabro/profiling-tools

# NOTE: I had to downgrade this package to resolve the installation conflict
# sudo apt-get install python3-pkg-resources=45.2.0-1
  sudo apt-get install python3-pip
  sudo python3 -m pip install memray
  PATH+=":/home/vyos/.local/bin"

---------------------------------------------------
  python3 -m memray run --live /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py
  python3 -m memray run -o output.bin profiling-tools/memray/mandelbrot.py
  python3 -m memray flamegraph output.bin
  scp $vyos:memray-flamegraph-output.html .

- Tracing user process. The user needs to install the debug symbols, and he can conveniently collect `perf.data` for the following analysis, i.e
    $ monitor trace "$pid" 10sec "$out_file"
          .... performance profile was collected to $out_file
- An CLI top-level operation modifier
  This can be usefull to profile other CLI commands that might take extra time like:
         load / add / delete / generate ...
     configure: commit, load, merge, save, show ...
    $ trace "$cmd ..."
          ..... performance profile collected to "$cmd".data
    # trace load $cfg_file
    # trace commit
    # trace save
         ....

Python/Perl methods demangling (name decoding)
The issue - `perf` dropped the Python support a couple of years ago.
When collecting the performance profile, you cannot see the frame names for python processes.
Instead, it will display some generic python runtime method.
It should not be hard to integrate some open-source python profiler if there is interest.
  https://github.com/benfred/py-spy
  https://github.com/P403n1x87/austin#examples=
Any interest in the memory profilers?
  https://github.com/bloomberg/memray