Any hope for DPDK?
Event Timeline
@dmbaturin you better explaining about DPDK
Basically, it's not that simple, we have internal talks but there is no real plan or ETAs yet
To many things need to be solved, and this is big chunk of work = lot of people = lot of money and/or time
Hm, I belive it should be relatively easy to make vyos "forget" about some interfaces, on which you plan to use your separate dpdk-enabled software and to just compile dpdk into main distribution. Is it enough for your needs, @Caesar305 or you need some specific application or you are talking about making all firewall stuff work over dpdk (which sounds like A VERY VERY HUGE task)?
"DPDK support" involved a lot of low-level contributions to a lot of different projects. Essentially you need to re-implement major parts of Linux on a case-by-case basis which is outside of the scope for VyOS right now.
There are several people working on DPDK support in their respective areas and over time the hope is that VyOS will be able to pull in that work once it's been contributed to the associated project.
For example DPDK support for:
- Bridging
- Forwarding
- Connection Tracking Matching
- Filtering
- Tunnel Encapsulation (e.g. GRE and VXLAN)
Are all very separate efforts with separate implementations.
A lot of what you're seeing today in terms of FOSS support for DPDK is for bridging (e.g. OVS). This is what "CloudRouter" is talking about when they claim to have DPDK support. But DPDK for bridging and not forwarding or filtering isn't very useful for a OS that's purpose is to be a router or firewall instead of a fast bridge.
Some commercial development kits support DPDK with non-free implementations (like 6WIND) and with a little work VyOS could be built using them but the licensing would be very expensive.
The most comprehensive FOSS effort I've seen so far is http://www.openfastpath.org/ which appears to be an open source effort to do what 6WIND does.
If they're actually successful it would bring most of what we need in terms of DPDK support to Linux in general and not just VyOS.
The hope is the VyOS will be the OS of choice to show off projects like OFP.
Well, I think this question can't be correctly answered until it is correctly stated. So I suggest waiting @Caesar305 for some clarifications. @rps 's answer implies that "support" means "ALL the routing stack works over dpdk" which seems to be really far now. But another option is the ability to run specific dpdk software on dedicated ports (e.g. traffic generator software for load testing of external equipment or high performance network sniffer) - this task seems to be achievable, if it's requested and donated for :)
@mickvav I think when people ask "does it support DPDK" it's because they've read that using DPDK will allow forwarding and possible filtering and NATing of traffic at 10 Gbps+ rates. VyOS offering some DPDK stuff and saying "mission accomplished" would leave a bad taste in people's mouths the same way CloudRouter is claiming DPDK support when it's only for bridged traffic.
But you're right, we should ask @Caesar305 what he really means by DPDK specifically ... it can mean a lot of things.
@rps Yes, when I refer to DPDK support I mean full support for routing, NAT, firewall, etc. I understand this would be a huge undertaking which is why I suggested some kind of bounty or possible sponsorship from several companies willing to pitch in and help.
Ok, @Caesar305, than I'll ask another stupid question - why do you think that if someone will implement FULL linux bridging/routing/firewall stack with DPDK, he will get some significant profit from this decision? May be I miss something, but if all these things are already implemented in kernel, they are just already there, so DPDK seems to be extremely effective if you make it do specific things by throwing away all unneeded things, if you implement everything in userspace application instead of kernel you will benefit only on simplicity of debugging, am I right?
@mickvav Well, the entire point of VyOS is to simplify routing configuration on a Linux platform / kernel. All the things vyos does can be done with manually tinkering of configuration files, etc. So yes, implementing DPDK will allow for simplified troubleshooting and configuration.
@mickvav I think you're misunderstanding the benefit of DPDK. It's essentially fastpath for Intel-based platforms and if implimented correctly can be the difference between 10 Gbps and 100 Gbps on the same hardware. Obviously being able to scale VyOS to that level would be game-changing. It's important, just likely not in scope for VyOS at this time ...
I agree that DPDK would be a game changer for VyOS. It would nullify the Brocade 5600 vRouter and would make VyOS the absolute go to routing software for high PPS networks. I'm already seeing an increase in interest around the BSDRP router project which is using netmap (BSD's version of DPDK in simple terms). The point being that, as more and more networks move towards 10Gbit / 100Gbit networking, VyOS is going to lag behind anyone else who implements DPDK or netmap routing first.
VPP might be a better starting point than "DPDK".
It is already packaged and has a lot of the userspace stuff sorted out already. https://wiki.fd.io/view/VPP/Installing_VPP_binaries_from_packages#Ubuntu.2FDebian
It's an interesting idea, I've even tried this stuff couple of days ago, but it seems to be under heavy development, although seems to be a motion in right direction - snippets of code in documentation doesn't work, things which they demonstrate in videos are already moved in another modules and so on. So to make these things importable into vyos, they should first be made workable.
Thus, if someone needs this stuff to be integrated into vyos, he has to achive some simple goals:
- 1. VPP stuff stabilizes in it's API (at least to some level).
- 2. There is a simple way to "say" to VPP things like these:
- 2.a vpp, use interfaces ethA and ethB, act on them with IP addresses IPA and IPB,
- 2.b vpp, please route packages from ethA to ethB and firewall/nat/whatever them according to ruleset C
- 2.c vpp, please, apply traffic shaping, ip source or destination or port or protocol or whatever else restirictions to traffic (do things, which netfilter does)
- 2.d vpp, please, interact with quagga router according to ruleset D (get routes from quagga/notify quagga about interface changes and so on)
- ....
- 3. Make sure that things from part 2 does not require recompilation on any configuration change OR that vpp can change it's runtime without out-of-service and package loss.
So it seems to me (and fd.io guys tell it in their videos :) ) that it takes about a month (!) of full-time hacking around to understand vpp deeply enough to just make it do what you expect it should do. After this moment we can start talking about integration of VyOS CLI and things like this. As VyOS hacking is not my main job, and even if I'll step in this effort, it will take a long time. Or someone has to either find a freelancer for this project or get to deal with @syncer or @dmbaturin or get to some agreement with my employer to assign this thing to me as a primary task :).
It seems to me that this thing as a whole may take a couple of month of full time to archive, but you know that programmer's estimations on times are almost always inaccurate :).
I'll make a move here and suggest that until FOSS projects to implement DPDK support see more maturity that VyOS doesn't go down the rabbit hole of that for now; I think a side project, maybe "HP-VyOS" (for High-Performance VyOS) take on trying to build a version of VyOS that can leverage experimental code like DPDK or VPP.
IMHO it's important that VyOS proper has a focus on reliability, stability, and improving configuration management for now (e.g. vyconfd, rollback without reboot, remote API, etc). This is especially the case as the project is in a "reputation building" phase, where adoption is gaining momentum.
The performance is already very good. If a group of people want to take on building an experimental "HP-VyOS", I think it would be fine to make it an official project of VyOS that we can provide support infrastructure for (wiki, mirrors, phabricator), but at this point I think it would be too time-consuming and drastic of a change to be the focus of VyOS proper. There is a high risk of problems and alienating future use of VyOS otherwise. That may all change in a year or two, as supporting projects become more mature.
Agree with @rps completely
something like that can appear ocassionally
but we need to set priorities correct
It seems that we need VyOS HCL(Hardware Compatibility List)
For those who looking for certain numbers in performance
Once we will see(or even participate) in some open source DPDK implementation (forwarding or any other in scope of interest) we can start talking about future of it in VyOS
Would the http://www.opendataplane.org/ and http://www.openfastpath.org/index.php/service/technicaloverview/ not work better in VyOS?
Use this to create a fastpath interface and the linux OS can just that.
Does openfastpath really work? Have you tried it? It all looks great, and if it works reliably, we indeed should integrate it.
Here is a howto on the openfastpath - https://www.howtoforge.com/tutorial/opendataplane-with-open-fast-path-on-ubuntu/
I'm going to try setup a lab to test it.
How did you go with OFP? I can compi;e and run it on vyos platform but first I tried in on Ubuntu and found it much worse than stock linux perhaps there are some setup that is required for it to function optimally. I also tried vpp and that doubled the pps at 64bytes.
@jeff What exactly did you do when you "tried it with vpp"? Can you tell us a bit about what hardware you tested on and what your numbers were?
Has anyone tried to do something with the howtoforge: https://www.howtoforge.com/tutorial/opendataplane-with-open-fast-path-on-ubuntu/
tutorial?
Is there a something to be done with this?
I can try to build it using docker.
I have tried this and I could not seem to get any data what-so-ever to go over the open fastpath interface, so I don't think it is a viable solution.
I tried doing some basic routing with ofp and it seemed to work but the shipped dpdk version does not compile for my kernel (4.10), so I can't test that.
So it looks like vpp progressing a bit.
https://github.com/FRRouting/frr/wiki/Alternate-forwarding-planes:-VPP
Maybe until we get FRR integrated it will be ready for stesting