The problem:
---------------------------------------------------------------------------------------------------------
The current FRR implementation lets you fetch particular config objects in one iteration.
This becomes problematic when querying a bulk state data (e.g. millions of routes) as it holds the CLI/Object
until it is fully displayed. There is no way to limit requests to "n" first objects, etc...
In such a case, you are reading the entire object and skipping the uninteresting data, which is very inefficient.
There should be an easy-to-use and robust mechanism to split (paginate) the data into manageable pieces.
Solution:
---------------------------------------------------------------------------------------------------------
This problem has already received some attention from the frr community, but the proper solution was not
developed/finished yet.
```
1. Previously, the northbound architect created a pull with the solution, but after multiply rounds of
review, he stopped active contributions to the project. So that PR was not merged in the end.
https://github.com/FRRouting/frr/pull/6371
2. Currently, the community with engineers from the `VMware` team is working on the new
`Centralised Management Daemon (MGMTD)`
One of the development goals is -
'13. Support for batching and pagination for display of large sets of operation data.`
https://github.com/FRRouting/frr/wiki/FRR-Centralized-Management-Requirements
The pull request tracking the development: https://github.com/FRRouting/frr/pull/10000
Is in development for about 2 years, ~20k lines of code, doesn't look like it will be merged soon
```
The [2] is being actively developed, but a lot of work must be done before it can be used in practice.
It is good enough for some basic tests, but the DB connection is available only for `staticd` and part of `zebra`.
More details will be described in the follow-up comments.
Regarding the [1]:
After trying to develop some hacky solutions, I realized that most of my ideas are already implemented in the dropped PR.
Since there have not been many changes to the northbound architecture, it is possible to merge it back by hand and customize it for our needs.
The basic demo can be found at:
https://github.com/volodymyrhuti/frr/tree/oper_data_pagination_dev
The PR introduces a new cli flags to perform data fetching with pagination
```
Fetching #n first elements
show yang operational-data <xpath> max-elements <n> <demon>
Fetching all elements by #n elements per iteration
show yang operational-data /frr-interface:lib max-elements 3 repeat zebra
```
To demonstrate how it can be extended, I have introduced a new flag `next` that does an iteration
starting from the previous one.
```
vova | sudo vtysh -c "show yang operational-data /frr-vrf:lib/vrf[name='default']/frr-zebra:zebra/ribs max-elements 5 zebra"
{
"frr-vrf:lib": {
"vrf": [
{
"name": "default",
"state": {
"active": false
},
"frr-zebra:zebra": {
"ribs": {
"rib": [
{
"afi-safi-name": "frr-routing:ipv4-unicast",
"table-id": 254,
"route": [
{
"prefix": "0.0.0.0/0",
"route-entry": [
{
"protocol": "kernel",
"metric": 600,
"selected": [null],
"installed": [null],
"internal-flags": 8,
"internal-status": 16
}
]
}
]
}
]
}
}
}
]
}
}
vova | sudo vtysh -c "show yang operational-data /frr-vrf:lib/vrf[name='default']/frr-zebra:zebra/ribs next zebra"
{
"frr-vrf:lib": {
"vrf": [
{
"name": "default",
"state": {
"active": false
},
"frr-zebra:zebra": {
"ribs": {
"rib": [
{
"afi-safi-name": "frr-routing:ipv4-unicast",
"table-id": 254,
"route": [
{
"prefix": "0.0.0.0/0",
"route-entry": [
{
"protocol": "kernel",
"uptime": "2022-06-09T15:31:47.509602+03:00",
"nexthop-group": {
"id": 10,
"nexthop": [
{
"nh-type": "ip4-ifindex",
"vrf": "0",
"gateway": "192.168.0.1",
"interface": "wlp3s0",
"active": [null],
"fib": [null],
"bh-type": "null",
"onlink": false
}
]
}
}
]
},
{
"prefix": "172.17.0.0/16",
"route-entry": [
{
"protocol": "connected",
"distance": 0,
"metric": 0,
"selected": [null],
"installed": [null],
"internal-flags": 8,
"internal-status": 16,
"uptime": "2022-06-09T15:31:47.509603+03:00",
"nexthop-group": {
"id": 8,
"nexthop": [
{
"nh-type": "ifindex",
"vrf": "0",
"gateway": "",
"interface": "docker0",
"active": [null],
"fib": [null],
"bh-type": "null",
"onlink": false
}
]
}
}
]
},
{
"prefix": "192.168.250.0/24",
"route-entry": [
{
"protocol": "connected",
"distance": 0,
"metric": 0,
"selected": [null],
"installed": [null],
"internal-flags": 8,
"internal-status": 16,
"uptime": "2022-06-09T15:31:47.509602+03:00",
"nexthop-group": {
"id": 7,
"nexthop": [
{
"nh-type": "ifindex",
"vrf": "0",
"gateway": "",
"interface": "anbox0",
"active": [null],
"fib": [null],
"bh-type": "null",
"onlink": false
}
]
}
}
]
}
]
}
]
}
}
}
]
}
}
vova | sudo vtysh -c "show yang operational-data /frr-vrf:lib/vrf[name='default']/frr-zebra:zebra/ribs next zebra"
{
}
```
---------------------------------------------------------------------------------------------------------
Details on the development and the timeline will be in the following comments.