Netgear R7800 exploration (IPQ8065, QCA9984)

Here the output :

root@LEDE:~# cat /proc/interrupts
           CPU0       CPU1
 16:    3282539    2952357       GIC  18 Edge      gp_timer
 18:         33          0       GIC  51 Edge      qcom_rpm_ack
 19:          0          0       GIC  53 Edge      qcom_rpm_err
 20:          0          0       GIC  54 Edge      qcom_rpm_wakeup
 26:          0          0       GIC 241 Edge      29000000.sata
 27:    3432167          0       GIC  67 Edge      qcom-pcie-msi
 28:    9562915          0       GIC  89 Edge      qcom-pcie-msi
 29:     183179          0       GIC 202 Edge      adm_dma
 30:    7136044          0       GIC 255 Level     eth0
 31:    9659778          0       GIC 258 Level     eth1
 32:          0          0       GIC 130 Level     bam_dma
 33:          0          0       GIC 128 Level     bam_dma
 40:          2          0   msmgpio   6 Edge      gpio-keys
 88:          2          0   msmgpio  54 Edge      gpio-keys
 99:          2          0   msmgpio  65 Edge      gpio-keys
103:          0          0   PCI-MSI   0 Edge      aerdrv
104:    3432167          0   PCI-MSI   1 Edge      ath10k_pci
136:          0          0   PCI-MSI   0 Edge      aerdrv
137:    9562915          0   PCI-MSI   1 Edge      ath10k_pci
169:         11          0       GIC 184 Level     msm_serial0
170:          2          0       GIC 187 Level     1a280000.spi
171:         67          0       GIC 142 Level     xhci-hcd:usb1
172:          0          0       GIC 237 Level     xhci-hcd:usb3
IPI0:          0          0  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:     149144    5975393  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:      72281    3751366  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          1          0  IRQ work interrupts
IPI7:          0          0  completion interrupts
Err:          0

How can I possibly move services to the second CPU?

[edit] oops, wrong values, this should be the correct one
[edit2] hmm, just testing here and i can't move wireless for some reason, but i can move eth0 and 1:

echo 2 >/proc/irq/30/smp_affinity
echo 2 >/proc/irq/31/smp_affinity

That should move eth0 and 1 over to the second cpu, while keeping Wifi on the original one. You'll have to run it after each reboot.

Let me know how this works out, am really curious if it makes a difference

Thanks @johnnysl for your help.
At first sight, it looks ok, less multicast TV macro-blocs when running wireless high-rate transfers (QoS is my next fight).
I'll perform wired speedtest later, and keep in touch.
Just wondering:

cat /proc/irq/default_smp_affinity
3

Why 3 ?

Bitmask.
1 = core 0
2 = core 1
4 = core 2 ... Etc.

3 = both core 0 and 1

Exactly, it is binary translated.
1=01
2=10
3=11

So 3 would mean it could do multi cpu sharing, but in reality it just only grabs the first one.

@hnyman: any idea why cpu affinity of the wifi interfaces is not allowed to be changed? (Which is no issue on the wrt1900ac)

Thanks for that clarification !
I've noticed another strange behavior: the R7800 has a higher latency than my previouns WNDR3800. The ping for any remote destination is 0,6ms higher. It's also longer to answer ping request as you can see on my smokeping chart:

Do you think it's related to the SoC architecture or something like that ?

I must admit, that switching eth0 over to the 2nd core makes aria download way faster. Probably this is due to USB and wan have been sharing the same 1St core.

I haven't looked at the IPQ806x target kernel options if there is anything about multi-core balancing, but maybe we need to pay attention to those.

I would have assumed that Linux somehow distributes the workload to cores, but apparently not, at least not automatically & optimally.

As far as I have found out, irq load is managed by apic that puts all hardware irq onto core0. The only way to balance the load is to use irqbalance package that is absent in lede

Simplest might be possible to add the affinity setting commands to /etc/rc.local, but I will look into irqbalance: https://github.com/Irqbalance/irqbalance

EDIT: you noticed the irqbalance, while I was writing this.

I was able to get irqbalance compiled. It distributed interrupts rather nicely.

Makefile https://gist.github.com/hnyman/c01b7d1d5e00cc9eea89f6eb4e7e0f27#file-irqbalance-makefile
Additional patch (patches/100-disable-ui-compilation.patch):
https://gist.github.com/hnyman/c01b7d1d5e00cc9eea89f6eb4e7e0f27#file-patch-100-disable-ui-compilation-patch

I made a rather rough first version: I disabled practically all optional functionality, opted to use the external glib2 (that adds a large package dependency) and disabled compilation of the UI, as that gave some errors on the first build attempts.

Below is what happened. I ran "oneshot IRQ balancing" and after a while it can be noticed that 27/qcom-pcie-msi, 31/eth1, 104/ath10k_pci got moved. So both fixed and wlan interrupts got split to different cores.

root@LEDE:/tmp# irqbalance --oneshot
...
root@LEDE:/tmp# cat /proc/interrupts
           CPU0       CPU1
 16:    1038754     541144       GIC  18 Edge      gp_timer
 18:         33          0       GIC  51 Edge      qcom_rpm_ack
 19:          0          0       GIC  53 Edge      qcom_rpm_err
 20:          0          0       GIC  54 Edge      qcom_rpm_wakeup
 26:          0          0       GIC 241 Edge      29000000.sata
 27:     689539      11246       GIC  67 Edge      qcom-pcie-msi
 28:     770553          0       GIC  89 Edge      qcom-pcie-msi
 29:     182217          0       GIC 202 Edge      adm_dma
 30:     329046          0       GIC 255 Level     eth0
 31:     535468      23525       GIC 258 Level     eth1
 32:          0          0       GIC 130 Level     bam_dma
 33:          0          0       GIC 128 Level     bam_dma
 40:          2          0   msmgpio   6 Edge      gpio-keys
 88:          2          0   msmgpio  54 Edge      gpio-keys
 99:          2          0   msmgpio  65 Edge      gpio-keys
103:          0          0   PCI-MSI   0 Edge      aerdrv
104:     689539      11246   PCI-MSI   1 Edge      ath10k_pci
136:          0          0   PCI-MSI   0 Edge      aerdrv
137:     770553          0   PCI-MSI   1 Edge      ath10k_pci
169:         14          0       GIC 184 Level     msm_serial0
170:          2          0       GIC 187 Level     1a280000.spi
171:          0          0       GIC 142 Level     xhci-hcd:usb1
172:        591          0       GIC 237 Level     xhci-hcd:usb3
IPI0:          0          0  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:      64319      94950  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:      16372     411777  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          1          2  IRQ work interrupts
IPI7:          0          0  completion interrupts

IRQ affinity distribution:

root@LEDE:/tmp# cat /proc/irq/16/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/18/smp_affinity
1
root@LEDE:/tmp# cat /proc/irq/19/smp_affinity
1
root@LEDE:/tmp# cat /proc/irq/20/smp_affinity
2
root@LEDE:/tmp# cat /proc/irq/26/smp_affinity
1
root@LEDE:/tmp# cat /proc/irq/27/smp_affinity
2
root@LEDE:/tmp# cat /proc/irq/28/smp_affinity
1
root@LEDE:/tmp# cat /proc/irq/29/smp_affinity
2
root@LEDE:/tmp# cat /proc/irq/30/smp_affinity
1
root@LEDE:/tmp# cat /proc/irq/31/smp_affinity
2
root@LEDE:/tmp# cat /proc/irq/32/smp_affinity
1
root@LEDE:/tmp# cat /proc/irq/33/smp_affinity
2
root@LEDE:/tmp# cat /proc/irq/40/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/88/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/99/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/103/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/104/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/136/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/137/smp_affinity
3
root@LEDE:/tmp# cat /proc/irq/169/smp_affinity
2
root@LEDE:/tmp# cat /proc/irq/170/smp_affinity
1
root@LEDE:/tmp# cat /proc/irq/171/smp_affinity
2
root@LEDE:/tmp# cat /proc/irq/172/smp_affinity
1

I pushed irqbalance to the packages repo:
https://github.com/openwrt/packages/commit/c5913bd12d73c4c781b79c16246a8e9c8d236b8f

EDIT:
Would be great if somebody can figure out how to avoid the glib2 dependency. Sounds crazy to pull in a 900 kB library to get 3-4 list functions :frowning:

When I set the configure options in Makefile to disable external glib2, I got errors during compilation. I did not look closer into them, yet, but just decided to use the external glib2 to get buildbot to compile the first versions...

@hnyman - just make it use its bundled glib-local which is merely a tiny stub adding a few linked list functions.

I suppose it's better to exclude wifi affinity change, because sometimes it doesn't broadcast after reboot if it's balanced.

Ps I've put irqbalance into startup.

Update:
Wed Feb 8 02:42:05 2017 user.notice : IRQ 27 was BANNED.
Wed Feb 8 02:42:05 2017 daemon.warn irqbalance: WARNING: MSI interrupts found in /proc/interrupts
Wed Feb 8 02:42:05 2017 daemon.warn irqbalance: But none found in sysfs, you need to update your kernel
Wed Feb 8 02:42:05 2017 daemon.warn irqbalance: Until then, IRQs will be improperly classified

Updated: adm_dma should be excluded as well

Actually it seems that the most correct way is to put eth0 and eth1 onto core1 manually. Irqbalance leads to numerous issues :confused:

Wow, that seems to be a bit faster than ipq806x Linksys EA8500:
| r3042 | ARMv7 Processor rev 0 (v7l) | 9.37 | ARMv7 Processor rev 0 (v7l) | 45.57 | Qualcomm (Flattened Device Tree) | 1.0.2k | 113688660 | 140577330 | 106261500 | 50258130 | 20346880 | 7244030 | 53185330 | 42715280 | 41551190 | 107.1 | 4769.5 472.4 | 472.8 |

Yeah, that is the goal. I tried it yesterday, but like I said, I got some compilation errors. I wanted to quickly see it irqbalance works at all, so I switched it to use external glib2 for now.

But I will try to get the internal glib stub compiled.

[quote="jow, post:59, topic:285"]
just make it use its bundled glib-local which is merely a tiny stub adding a few linked list functions.
[/quote]Using the internal stub works ok by itself, but it looks like there are recent upstream commits that have bypassed the "external glib on/off" selection and force glib types to everywhere via glib.unix.h requirement :frowning:
https://github.com/Irqbalance/irqbalance/commit/d1993bcde2a524346a9508754671f096ec129ad1

I will try reverting that.

And I will file bug at upstream. It makes no sense to provide a config option and internal stub for glib, if that is overlooked in code.

I managed to disable glib2 usage in irqbalance by reverting an upstream commit. The packages feed repo has been updated with https://github.com/openwrt/packages/commit/c42ecd05a1c91070bc86b4d8254972562b6e0c67

1 Like

This is likely related to the interrupt mitigation in the SOC ethernet controller. (It can be disabled through the ethernet driver though.)

What does that mean? Do the have something like interrupt coalescing running inside the ethernet driver?