I'm running latest LEDE kernel 4.4.39 on the EdgeRouter Lite 3 and I'm not positively surprised by the performance of the device. I'm aware the LEDE/OpenWrt can't support the hardware acceleration available, but still, I was expecting a bit more then 130 Mbit/s to be possible The device is advertised to run near link speed.
I'm wondering, am I really running into an hardware limit, or could there be something wrong in the kernel or did I configure something wrong? Or does the hardware acceleration really improve speed that much?
The setup I'm using: gbit internet <-> eht0 <-> eth0.34 <-> NAT <-> bridge <-> eth1
And am just downloading a big test file from the download test server of my provider.
Results: 130 Mbit/s, 16000 packets/s
ksoftirqd/0 is running at ~ 100% cpu, so that's where the bottleneck is. A simple state-full firewall is used with NAT enabled. Could it be caused by the vlan(34) and bridge interfaces in between, that will require the kernel to move each packet many times through multiple interfaces?
The EdgeRouter Lite only has a 500MHz dual core MIPS CPU. They are most likely only capable of reaching gigabit speeds when using the stock firmware from Ubiquiti, enabling whatever form of acceleration they have (hardware / special kernel modules / ...).
For comparison:
A Zyxel ZyWALL USG 60 has a 400MHz dual core MIPS CPU which hits around 70% usage on a 120Mbit connection with only the firewall active and all other UTM features off using their stock firmware. AFAIK Zyxel has no form of hardware or software acceleration in place at the moment.
So if you want to get the advertised gigabit speed from the EdgeRouter, you either have to use the stock firmware or wait until OpenWRT/LEDE can utilize the same form of acceleration Ubiquiti can.
IIRC a friend of mine also told me the performance of the EdgeRouter was impacted (even on the stock firmware) when he enabled bridging as that happens in software and affects the CPU. Maybe it also disabled acceleration.
Just a question:
Why do you want to replace the Ubiquiti firmware? It's not your usual home router crap and gets updated quite often.
I have a vague memory of getting about 200+ using FreeBSD (also software only) but that was a good 2 years ago or so. It ran fine but no QoS if you need that.
wait until OpenWRT/LEDE can utilize the same form of acceleration Ubiquiti can.
In most supported devices the hardware acceleration will never be implemented because in vendor sources/firmwares it is done with very ugly hacks that would make some system components incompatible with other devices (so you would need to keep dozens of forks of them, each with a device-specific hack to use such hardware acceleration).
Thanks for the responses! I've been digging quite some more into this and after some testing I found that indeed the performance, when not using offloading, is limited to about 130-150 mbit/s, even with the stock Ubiquity firmware (which does not enable offloading by default). I tested with a 1000m download over HTTP.
Performance is impacted by packet size, which directly relates to number of packets / sec that need to be processed. And performance is impacted by other actions that need to take place, like QoS, Vlan, bridging and nat.
I guess a MIPS 400/500 MHz CPU generally is limited then to about 150 - 200 mbit/s, if no offloading is used / available.
The offloading features are proprietary unfortunately. So either use the stock kernel, or OpenWRT/LEDE, but then accept the rather big performance penalty.
What I wasn't aware of / didn't expect for such a low cost device, is that Ubiquiti keeps posting firmware updates regularly. Which is great.
With stock firmware I easily get 900 mbit/s, with ip and vlan offloading enabled.
On the question why LEDE/OpenWrt: I'm not that interested in all the software features, including a UI, that is available in the Ubiquity firmware. I'd like to be able to configure anything by myself and keep full control. Which is by the way also one of the reasons why I started the DebWrt project, combining the power of OpenWrt/LEDE with Debian.
For those interested, I figured out how to compile the Ubiquiti kernel for the erlite using the GPL sources package, provided on the Ubiquiti download site. The offloading modules are proprietary, but as suggested here, the offloading kernel modules provided by the normal firmware can be used with a self compiled kernel. Changes have been incorporated in DebWrt. For information here.
With enough interest I could have a look how to extend OpenWrt/LEDE with a new target based on the the Ubiquiti GPL sources.
There's a long, proud tradition of nerds extracting performance-enhancing binary blobs from one project and making them work well in another. When you get the time and personal interest, go for it.
@MartB
SFE definetely cannot outperform Cavium's proprietary offload yet. With SFE you can get around 1.3 Gbit/sec half-duplex, while Cavium is close to 1 Gbit/sec full-duplex.
I'm running LEDE on my ERL with SFE support in the kernel, and when doing SQM with CAKE you can push 200 Mbit/s full-duplex through it. Which is not bad compared to the 50 Mbit/s you get with CAKE on the stock firmware.