Netgear R7800 exploration (IPQ8065, QCA9984)

Yup. Personally—I only noticed the huge latency spikes. Modem is an SB8200 fwiw, and it's Broadcom based, so no PUMA issue.

I managed to compile and install the qca-nss-gmac driver. The nss-gmac driver replaces the standard Linux stmmac driver. The nss-gmac driver seems to be working properly on my R7800. I'm not sure if the driver actually uses the additional 2 NSS CPU cores tho. It doesn't seem like it is using it.

I've uploaded the codes needed to compile and run with the nss-gmac driver here:

Let me know if you would like a pre-build binary instead for testing. My image is based on the lede-17.01 branch, with my custom changes as well. For those experiencing latency spikes, do assess if you can try out the qca-nss-gmac driver and see if that resolves your issue.

I will try a prebuilt image.

Details of my builds can be found in the community build thread for R7800.

After re-looking into the source code, looks like the NSS cores are not used, as it requires the NSS firmware to be uploaded. But the nss-gmac driver is required for further work on the nss-drv to be done. I'll see if I can get the nss-drv successfully loaded. That would be the one to drive the NSS cores and perform the offload magic.

Might be actually better here, or you might start an own thread about that nss-gmac test build.

I was under the impression that using the NSS cores needed SSDK and various incantations to offload packet processing ... I may be wide of the mark though.

I managed to get most of the NSS stuff building and running successfully, but I couldn't resolve the patches required to get SSDK to work on the current 17.01 kernel, there might be updated stuff in the upstream codeaurora tree now though.

Came across this:


Seems more up-to-date than the github of Paul Chambers...


This guy provides improved factory firmwares.
@quarky This could provide more info on the nss-stuff, see /package/nss*

@quarky your changes do not enable nss-gmac driver as your dts entries are missing the corresponding compatibility.
I've been trying to enable it but failed, smth in the driver itself leads to a boot loop.
Here's he DTS patch
https://pastebin.com/1SQEhsBc

@quarky I have just tried your build (nss) it looks like the latency overhead introduced by the router has decrease to 0.5ms or less (down from 1+ ms), but I cannot confirm for sure as I only had 15 minutes or so. The latency spikes remained unfortunately.
Two more observations: 1) I could not get an IP from a LAN port until I removed my VLAN's and then it all worked. 2) The router went into a reboot loop once I sysupgrad'ed to my firmware. Had to recover the router via TFTP.

Not only that, looking at the official firmware (and the codeaurora stuff, along with descriptions of how the NSS is initialized and works) you'll need the following:

kmod-qca-nss-connmgr
kmod-qca-nss-drv
kmod-qca-nss-gmac
kmod-qca-ssdk-nohnat

Connmgr monitors conntrack events and creates rules that offload onto the NSS cores (see here: https://lkml.org/lkml/2015/1/21/804)

My changes includes those DTS entry for the GMACs devices. The nss-gmac drivers is running in my build. Next step is to get the nss-drv driver working. I’ve disabled the stmmac Ethernet driver, so the only Ethernet driver has to be the nss-gmac.

get serial and see why it bootloop

Yes, so getting the nss-gmac driver up and running is crucial for the next step, from what I’ve read. The one that does actual offloading is the nss-drv driver, which depends on nss-gmac.

Links from a few post earlier provided more pieces of the puzzle. Hopefully it’ll be enough to activate the NSS cores.

Just found out that nss drv has double compatibility string

I addeded that as the DTS entry for lede is different from the driver. Adding that line in the nss-gmac driver makes it easier for me to switch back to the standard driver without fuss.

The slight decrease in latency may be coincidental. I was hoping the nss-Gmac driver may be more efficient in handling the gmac interfaces. My R7800 is configured with tag VLAN for the WAN port and it’s working. So I don’t think it’s VLAN issue.

Could you be running a master build before flashing with mine? The configurations may be different. Best to reset to default if testing different builds.

Yes, I was running master and I did reset the router after to test your build.

@quarky with nss-gmac driver enabled could you run the following command and provide output pls
cat /sys/kernel/debug/clk/gmac*/clk_rate

Here's the output from my R7800:

root@sbrt-r7800:~# ls /sys/kernel/debug/clk/gmac*
/sys/kernel/debug/clk/gmac_core1_clk:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate

/sys/kernel/debug/clk/gmac_core1_src:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate

/sys/kernel/debug/clk/gmac_core2_clk:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate

/sys/kernel/debug/clk/gmac_core2_src:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate

/sys/kernel/debug/clk/gmac_core3_clk:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate

/sys/kernel/debug/clk/gmac_core3_src:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate

/sys/kernel/debug/clk/gmac_core4_clk:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate

/sys/kernel/debug/clk/gmac_core4_src:
clk_accuracy clk_enable_count clk_flags clk_notifier_count clk_phase clk_prepare_count clk_rate
root@sbrt-r7800:~# cat /sys/kernel/debug/clk/gmac*/clk_rate
266666666
266666666
266666666
266666666
266666666
266666666
266666666
266666666