Hi @aserkin ! I should have some time to help you. For this issue, I would need

• artifacts collected from running zebra under Valgrind
• a reference configuration that I can replicate in a lab

I will ping you over Slack for additional details and assistance.

I have created a draft pull request for FRR, but I can still see a bunch of odd bugs.
I'm going to activate it after additional testing by the team.
Most issues involve Wireshark's inability to parse packets correctly and display an exception, although the demons seem to run fine.
https://github.com/FRRouting/frr/pull/14788

I`m back after a long break and will follow up on this feature.
Here is a summary of things that have happened since the last update:

1. In my absence, the feature testing got broken as a result of migrating from the `mininet` to the `munet` framework
From debugging, I have identified a root cause to be - bpf fs was not inherited by the `munet` router.
The solution is to hop into the router mount namespace for the test run.

Because there is a long-running development for operation data retrieval, we can postpone this ticket until an effort is finished.
Then, I can open a feature request or visit the yang meeting and start a discussion about the data pagination functionality.
Currently, my idea is to simulate pagination at the fs level by having a split of requested JSON.
This solution involves:

• Fetching an operation data from the demon

vtysh -c "show yang operational-data /frr-vrf:lib/vrf[name='default']/frr-zebra:zebra/ribs zebra" > big.json

• Fetching flat data stream and formating it 1 item (prefix) per line with jq -c option

jq -c '."frr-vrf:lib" .vrf[0] ."frr-zebra:zebra" .ribs .rib[0] .route[]' big.json | split -l 100 -d

• Splitting result by the number of lines (objects), saving into files on the filesystem

Now, UI can display 1 of the resulting files at a time. These files may be regenerated on page refresh.
Such a solution will utilize extra disk space, although it can be avoided by using pipes and other
streaming utilities like awk/sed

I cannot attach the rest of the artifacts due to size limits.
Publishing them on git instead: https://github.com/volodymyrhuti/QPPB_docs

Hi @zsdc! This seems to be related to T4028. The relevant commits are:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
commit 92980561382fc04380414a6e2f6ca6746c2fe5e9 ┃
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━
Author: Donald Sharp <[email protected]>
Date:   Mon Apr 19 19:23:45 2021 -0400

Hi @aserkin! It looks like you have some frr server misbehavior. It sends up/down events with an unexisting vrf id.
Could you make/describe the setup that causes the issue to appear? Thanks

DEMO Notes:
=====================

1) You need to load the XDP program before starting frr so that
   it can find the LPM map on plugin initialization.
   To keep it simple, the VTY interface was not implemented for now.
   XDP side is accessible via `bpftool`
3) I`m monitoring packets for TOS/DSCP changes to see if marking happens
   But in another approach tag is associated with the packet and then
   read by the TC classifier
4) These are two traffic shaping examples.
   The point is that you have two options for marking:
4.1) Modifying the TOS byte and installing the u32 tc filter to match the value.
   This has a limited range of possible values (8 bits) + needs to modify the packet.
4.2) Using a custom BPF classifier.
    The XDP side extends the packet context and saves the value.
    Afterward, the classifier may read the context and control the shaping behavior
    by setting the `skb->tc_classid` or one of the fields mentioned below.

Therefore, BPF programs attached to the tc BPF hook can, for instance,
read or write the skb’s mark, pkt_type, protocol, priority, queue_mapping,
napi_id, cb[] array, hash, tc_classid or tc_index, vlan metadata, the XDP
transferred custom metadata and various other information. All members of
the struct __sk_buff BPF context used in tc BPF are defined in the
linux/bpf.h system header.

https://docs.cilium.io/en/stable/bpf/#tc-traffic-control

DEMO
===============================================

To demonstrate the feature let's look at the following topology

Changes on the FRR side:

• Convert xdp helper library to an optional plugin + bgp hook
• Minor fixes + cleanups
• Figured out most of the permission problems

Changes on the XDP side:

• Convert mappings from legacy iproute format to the latest libbpf one
• New mappings improve debugging experience by implementing pretty-printing for XDP map dumping
• Added an xdp-loader for xdp-tools repo

The latest version of the demo can be found here:

1. volodymyrhuti/frr/tree/QPPB_DEMO_V1.3
2. volodymyrhuti/xdp_qppb

The latest version of the demo can be found here:

You can find the latest version of the demo implementation here:

1. volodymyrhuti/linux/tree/QPPB_DEMO_V1.1
2. volodymyrhuti/frr/tree/QPPB_DEMO_V1.1

Demo QPPB implementation supporting bgp-policy destination mode:

Memray:

In order to keep useful tracing/debugging tooling in a single place, the @jestabro has created the repo:

https://github.com/jestabro/profiling-tools

Since there is no vyatta package yet, you need to either compile it by hand or install it from the apt
as explained before. Some examples to play around with:

# NOTE: I had to downgrade this package to resolve the installation conflict
# sudo apt-get install python3-pkg-resources=45.2.0-1
  sudo apt-get install python3-pip
  sudo python3 -m pip install memray
  PATH+=":/home/vyos/.local/bin"

Analysis:

I have collected the profiling data for the following configurations:

Gotchas:

If you are running a small QEMU device and it has run out of the memory, the scenario is following:

- The boot process has failed, the prompt is stuck, bash is not initialized
- You reboot the device, it tries to read the config, fails once again as there is no free memory
- Config was not loaded, you cannot log in; it is a loop

TBD: GUI

VyOS users can configure the front-end interface, called vycontroll, to examine the configuration state.
A detailed description can be found at:
https://vycontrol.com/
https://github.com/vycontrol/vycontrol
https://docs.vyos.io/en/equuleus/configuration/service/https.html
https://brezular.com/2021/05/01/vycontrol-web-ui-for-vyos-firewall/

FRR Debugging

Recently, I had to triage/debug a bunch of issues that involved running a legacy build of frr.
This involved:

• Triaging issue down to the place when it was introduced. Otherwise, verifying that feature was never working at all.
• Comparing the execution flow between legacy/master versions to identify the divergence
• Building & running multiple (legacy/master) frr versions in parallel
• Doing deep analysis within gdb

Since the last update, I have simplified the CLI interface:

1. I have removed the global iterator and incapsulated the iteration state into the vty structure.
   This way, each vtysh client has its private iteration state for the following requests.
   It should be possible to query multiple data nodes simultaneously and asynchronously.

Ongoing activity:

1. Stabilization
-  I have seen a corner case that would crash inside the northbound callbacks.
-  I can see some validation failure logs, although the resulting output seems good for me.
-  Daniil was concerned about memory leaks associated with iteration state.
   After additional research - this is not a problem, but I can imagine cases where we would
   fail to handle a malformed XPath and leak resources on the stuck unwinding
   I need to do some testing with Valgrind.
2. Scale testing
3. Async support for multiple vtysh clients. The current demo assumes that there is only one client.
   I want to map the iteration state to the vtysh client/socket so multiple requests may be executed in parallel
4. A debugging instruction
   I have used some complicated debugging flow when merging the feature.
   This should be useful for other (non-C) devs.
5. Finishing the documentation
6. advanced XPath filtering support?

Recently, I had a conversation with the VMware team lead - Pushpasis Sarkar.
He has described the ongoing development and explained the use case they are interested in.
From the conversation:

1. The latest proposal draft: 
   Page 72-73 `Retrieve Operational Data - Retrieving Containers and Leaf members`
   Page 84-85 `Retrieve Operational Data - Retrieving Large List elements` + comments
   Page 86 `Retrieve Operational Data - Retrieving Containers and Leaf members` + comments.

There was some effort to introduce profiling into the system before, but nothing was developed.
The ticket was opened to verify that the timing values displayed in /var/log/vyatta are correct.
The vyos-debug flag enables tracing for actions described in the templates.
This will be a step-by-step walkthrough of the system profiling, as I have found this to have a bunch of non-obvious technical nuances that might get you stuck.

Pull https://github.com/FRRouting/frr/pull/11004 was merged. This needs retesting on the latest FRR