Strict Priority setting per interface using tc Networking

I want to set up per interface priority as I have 4 LAN interfaces to connect different LAN devices :

Test Setting As below with two LAN interface:

WAN port - eth0

LAN port - ath0, ath1

tc qdisc del dev eth0 root
tc qdisc add dev eth0 root handle 1: prio bands 8 priomap 0 1 2 3 4 5 6 7

tc filter add dev eth0 protocol ip parent 1: prio 1 handle 1 fw flowid 1:1
tc filter add dev eth0 protocol ip parent 1: prio 2 handle 2 fw flowid 1:2
tc filter add dev eth0 protocol ip parent 1: prio 3 handle 3 fw flowid 1:3
tc filter add dev eth0 protocol ip parent 1: prio 4 handle 4 fw flowid 1:4
tc filter add dev eth0 protocol ip parent 1: prio 5 handle 5 fw flowid 1:5
tc filter add dev eth0 protocol ip parent 1: prio 6 handle 6 fw flowid 1:6
tc filter add dev eth0 protocol ip parent 1: prio 7 handle 7 fw flowid 1:7
tc filter add dev eth0 protocol ip parent 1: prio 8 handle 8 fw flowid 1:8

As platform has to use ebtable marking:

//ath0 - upload
 ebtables -D FORWARD -i ath0 -o br0 -j mark --mark-set 1 --mark-target ACCEPT 
 ebtables -A FORWARD -i ath0 -o br0 -j mark --mark-set 1 --mark-target ACCEPT

 //ath0 - download
ebtables -D FORWARD -i br0 -o ath0 -j mark --mark-set 1 --mark-target ACCEPT 
ebtables -A FORWARD -i br0 -o ath0 -j mark --mark-set 1 --mark-target ACCEPT

 //ath1 - upload
 ebtables -D FORWARD -i ath1 -o br0 -j mark --mark-set 2 --mark-target ACCEPT 
 ebtables -A FORWARD -i ath1 -o br0 -j mark --mark-set 2 --mark-target ACCEPT

 //ath1 - download
ebtables -D FORWARD -i br0 -o ath1 -j mark --mark-set 2 --mark-target ACCEPT 
ebtables -A FORWARD -i br0 -o ath1 -j mark --mark-set 2 --mark-target ACCEPT

different status Checks :

check class:

~ # tc -s -d class show dev eth0
class prio 1:1 parent 1: 
 Sent 278885308 bytes 196045 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:2 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:3 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:4 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:5 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:6 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:7 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:8 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0

check filters :

~ # tc -s -d filter show dev eth0
filter parent 1: protocol ip pref 1 fw 
filter parent 1: protocol ip pref 1 fw handle 0x1 classid 1:1 
filter parent 1: protocol ip pref 2 fw 
filter parent 1: protocol ip pref 2 fw handle 0x2 classid 1:2 
filter parent 1: protocol ip pref 3 fw 
filter parent 1: protocol ip pref 3 fw handle 0x3 classid 1:3 
filter parent 1: protocol ip pref 4 fw 
filter parent 1: protocol ip pref 4 fw handle 0x4 classid 1:4 
filter parent 1: protocol ip pref 5 fw 
filter parent 1: protocol ip pref 5 fw handle 0x5 classid 1:5 
filter parent 1: protocol ip pref 6 fw 
filter parent 1: protocol ip pref 6 fw handle 0x6 classid 1:6 
filter parent 1: protocol ip pref 7 fw 
filter parent 1: protocol ip pref 7 fw handle 0x7 classid 1:7 
filter parent 1: protocol ip pref 8 fw 
filter parent 1: protocol ip pref 8 fw handle 0x8 classid 1:8

check qdisc setting :

# tc -s -d qdisc show dev eth0
qdisc prio 1: root refcnt 2 bands 8 priomap  0 1 2 3 4 5 6 7 1 1 1 1 1 1 1 1
 Sent 278886988 bytes 196085 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0

check ebtable :

~ # ebtables -LV
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 4, policy: ACCEPT
-i ath0 -o br0 -j mark --mark-set 0x1 --mark-target ACCEPT
-i br0 -o ath0 -j mark --mark-set 0x1 --mark-target ACCEPT
-i ath1 -o br0 -j mark --mark-set 0x2 --mark-target ACCEPT
-i br0 -o ath1 -j mark --mark-set 0x2 --mark-target ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

I try to send traffic on both interfaces and it passes but from class status check packet count, it seems that only marking 1 is working :

class prio 1:1 parent 1: 
 Sent 278907565 bytes 196399 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:2 parent 1: 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0

Does anyone has any idea, what I might be missing ?
Thanks in advance

What you're missing is that Mark doesn't determine priority, and your filters don't either so it's always using the default.

https://www.systutorials.com/docs/linux/man/8-tc-prio/

The TOS based default is totally out of date... Someone should update it to use DSCP. But you can use filters... You just need to design the filter right.

as per my knowledge, tc respect TOS but tc filter can also add to that priority listing:

tc filter add dev eth0 protocol ip parent 1: prio 1 handle 1 fw flowid 1:1

this is filter which says- mark 1 (handle 1) packet send to 1:1 class

priomap
The priomap maps the priority of a packet to a class. The priority can either be set directly from userspace, or be derived from the Type of Service of the packet.

so definitely prio still respect TOS??? and filter can add to this priority.

Current stetting try to mark packet using ebtable and then tc filter pass mark 1 packet to class 1:1 which has highest priority

is this still make sense ??

OR it means we have to update TOS or DSCP for setting priority but I dont want to make any change to TOS or DSCP as its leaving packet

... but it does not by default and your config also does not.
"priority" in priomap is an attribute of the datastructure holding the packet, not the packet itself.

also note that unrestricted use of priority queueing will lead to complete drain-out of the lower prio's traffic (aka network error on ath1 when ath0 is downloading) which is not practical for most applications.

have you tried sqm with piece_of_cake?

I think perhaps ebtables isn't doing the marking:

http://ebtables.netfilter.org/misc/ebtables-man.html#lbBD

says "The mark target can be used in every chain of every table. It is possible to use the marking of a frame/packet in both ebtables and iptables, if the bridge-nf code is compiled into the kernel."

do you have bridge-nf as a module? Have you enabled it in sysctl? As soon as you do, your bridge may stop working the way you think it should, so you may need to alter your firewall rules.

I think you are correct about the "handle x" which I thought meant something else other than "fwmark x" but https://www.linux.org/docs/man8/tc-fw.html confirms your usage.

This can happen, but there are plenty of cases where it wouldn't and strict priority is reasonable. For example suppose you have a whole network of IP telephones connected to one of your bridge ports. Maybe there are 10 telephones. You know that no more than 10 phone calls will be active. Each phone call uses say 100kbit/s or thereabouts, so you know less than about 1Mbps will be going to or from this port. If you want to give this port strict priority so that jitter and loss is minimized, it's a perfectly reasonable application. There are many other possibilities like this: a remote controlled robot or remote sensor device, a game console, a VOIP media server, etc.

The key is that, yes, you can starve the other ports, so you should consciously make a choice as to whether that's a concern etc. To mitigate this problem you could police the port so that it doesn't allow more than a fixed maximum bandwidth for example. Then if something goes haywire and your VOIP phones are hit by an IoT botnet it won't take over the link.