Maybe try SFQ+HFSC with such small bandwidth to use.
Okay, so this is borked (but not your fault, we do have an issue currentky with cakes auto-auto-overhead-adjustments), you also have GRO/GSO enabled. But
The "maxpacket 1492" fits with what one expects on a pppoe interface, so if you would add 8 bytes you and up with 1500 which certainly is too little, you will almost certainly need at least 18 bytes for ethernet (6 dstMAC, 6 srcMAC, 2 ethertype, 4 frame check sequence) and 4 bytes for VDSL2/PTM; potentially another 4 to 8 bytes for single or double VLAN tags. My ISP currently uses 1526 frames, so on a pppoe interface (where the PPPoE header is not yet added) I actually need to request 1526-1492 = 34 bytes of overhead to properly describe the link to sqm.
Now there is a bigger issue with current cake visible in:
So cake will add variable overhead for ingress and egress. So I would try to add the following to the advanced option strings: "raw overhead 34 mpu 64", this should sidestep cake's broken auto-auto-adjustments and still properly describe the link.
Why does this not directly show up in tests? Well, bandwidth and overhead are not independent variables, specifying too much overhead effectively reduces the bandwidth and setting the bandwidth too low allows for too small an overhead. Since one often really does not know the exact numbers for neither but empirically tries to get a working combination it simply is not guaranteed that the set one ends up with truly describes the links limits well or just happen to work well-enough.
In case you ask yourself, yes the cake developer is working on fixing the auto-auto-adjusments making things simpler in the future but no ETA yet.
1 Like
I would be wary to recommend that, for some time now there are bug reports of kernel warnings caused by the hfsc module being loaded (even if not actively used) that might indicate that hfsc is broken in current linux kernels.
And I really really would recommend to stick to fq_codel as compared to SFQ (SFQ was cool in its day, but fq_codel should work better in almost all situations, so I really would at least compare both and only select SFQ if it really performs noticeably better)
Hfsc works great on my link, using a Debian based router and 4.14.x kernel, I recently started using it in a manual config. Very nice once you understand it. Real-time classes are perfect for voice streams. I use it with fq_codel on the non rt classes, and fast fiber link.
An example of my service from ATT in the United States for the curious
Modem-Router Combo Arris NVG599 (passthrough mode)
TP-Link WiFi Router VERSION="18.06.0-rc2"
Downstream Sync Rate (kbps) 28190 / 31831
Upstream Sync Rate (kbps) 6884 / 6158|
Downstream Max Attainable Rate (kbps) 40100 / 36719
Upstream Max Attainable Rate (kbps) 8631 / 7543
Modulation VDSL2 / VDSL2
Data Path Interleaved / Interleaved
My Cake recipe:
config queue 'eth1'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option itarget 'auto'
option etarget 'auto'
option verbosity '5'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option qdisc_advanced '1'
option squash_dscp '1'
option squash_ingress '1'
option qdisc_really_really_advanced '1'
option interface 'eth0.2'
option enabled '1'
option debug_logging '1'
option linklayer 'none'
option download '60022'
option upload '13042'
option iqdisc_opts 'ingress mpu 64 nat raw overhead 8'
option eqdisc_opts 'ack-filter-aggressive mpu 64 nat raw overhead 8 '
tc -s -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 869141068 bytes 2443052 pkt (dropped 0, overlimits 0 requeues 4)
backlog 0b 0p requeues 4
maxpacket 1514 drop_overlimit 0 new_flow_count 15 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8019: dev eth0.2 root refcnt 2 bandwidth 13042Kbit besteffort triple-isolate nat ack-filter-aggressive split-gso rtt 100.0ms raw overhead 8 mpu 64
Sent 93748626 bytes 661195 pkt (dropped 85, overlimits 213204 requeues 0)
backlog 0b 0p requeues 0
memory used: 304800b of 4Mb
capacity estimate: 13042Kbit
min/max network layer size: 42 / 1514
min/max overhead-adjusted size: 64 / 1522
average network hdr offset: 14
Tin 0
thresh 13042Kbit
target 5.0ms
interval 100.0ms
pk_delay 780us
av_delay 70us
sp_delay 14us
backlog 0b
pkts 661280
bytes 93774164
way_inds 20385
way_miss 991
way_cols 0
drops 18
marks 0
ack_drop 67
sp_flows 1
bk_flows 1
un_flows 0
max_len 1514
quantum 398
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 1565493766 bytes 1141579 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 801a: dev ifb4eth0.2 root refcnt 2 bandwidth 60022Kbit besteffort triple-isolate nat wash ingress split-gso rtt 100.0ms raw overhead 8 mpu 64
Sent 1581461464 bytes 1141569 pkt (dropped 10, overlimits 1670648 requeues 0)
backlog 0b 0p requeues 0
memory used: 178560b of 4Mb
capacity estimate: 60022Kbit
min/max network layer size: 60 / 1514
min/max overhead-adjusted size: 68 / 1522
average network hdr offset: 14
Tin 0
thresh 60022Kbit
target 5.0ms
interval 100.0ms
pk_delay 159us
av_delay 40us
sp_delay 10us
backlog 0b
pkts 1141579
bytes 1581475872
way_inds 17710
way_miss 996
way_cols 0
drops 10
marks 0
ack_drop 0
sp_flows 2
bk_flows 1
un_flows 0
max_len 1514
quantum 1514
I have non-interleaved VDSL2. Since 18.06 I've been using an overhead of 12, it seems to work well. I did not derive this number via heavy testing. 8 will also work just fine, but I decided to go with 12 because it seemed to feel a tiny bit better with some informal tests I ran.
I don't think the previous recommended number of 34 which I was using on 17.05 is optimal.
1 Like
Please note, that on VDSL2 you will have the following mandatory components:
"PTM-framing" VDSL2 (IEEE 802.3-2012 61.3 relevant for VDSL2): 1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS), = 4 Byte
COMMON (VDSL2/Ethernet): 6 (dest MAC) + 6 (src MAC) + 2 (ethertype) + 4 Byte Frame Check Sequence (FCS) = 18 Byte
So you will have at least an overhead of 22 Bytes on top of IP frames.
The 34 above have two more non-mandatory components:
VLAN (IEEE 802.1Q): 4 Byte VLAN
PPPoE: 2 Byte PPP + 6 Byte PPPoE
for a total of:
4 + 18 + 4 + 8 = 34 Bytes.
if you use pppoe and instantiate sqm on pppoe-wan the kernel does not "see" the 8 byte pppoe header (if however you instantiate sqm on the actual ethernet interface connected to the real vdsl modem, the kernel already knows about these 8 bytes,then you would only configure overhead 26, but I would recommend for pppoe connections to always instantiate on pppoe-wan, as that will work better with hotplug)
Best Regards
EDIT: Since Ethernet frames are at least 64 bytes from the destination MAC address to the FCS, and VDSL2/PTM adds 4 more bytes per-packet-overhead, the MPU on VDSL2/PTM equals 64+4=68 bytes.
ATTENTION: The MPU for true ethernet is 84 bytes, as ethernet L1 adds an additional 20 bytes of effective per packet overhead on top of the 64 Byte L2 frame.
2 Likes
PTM-TC introduces one flag
octet (on average), one address octet, one control octet
and two CRC octets, which adds up to five octets.
I believe you are talking about ITU G.993.1 Annex H: which states the following sizes: 1 Byte Opening Flag Sequence (OFS), 1 Byte Address Field (AF), 1 Byte Control Field (CF), 2 Byte TC-CRC (PTM-FCS), 1 Byte Closing Flag Sequence (CFS might be avoided when sending PTM frames back-2-back).
I will note that this seems to apply to VDSL1 only, VDSL2 works according to ITU-T Rec. G.993.2 (02/2006) in Annex K it references Rec. ITU-T G.992.3 (04/2009) Annex N which in turn references 61.3.3.1 of [IEEE 802.3]. I also note that PTM also requires sync-bytes (every 65th byte will be a sync byte) but these are packet size independent and best modeled as a static rate reduction from the sync values (a static reduction of 100-100*64/65 = 1.54% will do nicely).
PTM-TC preemption (if supported at all) works by adding 2 new values for the 65th sync octet, and support for short frames will introduce another flag C(j1), but that IMHO is irrelevant for links that send ethernet frames with ethernet-FCS as ethernet guarantees these to be >= 64 bytes and hence they will never trigger the short frame support (I assume rare runt frames will be discarded before being sent over the link, but if one has significant numbers of runt frames one has bigger problems than getting the PTM overhead wrong).
Now, I am not an expert in these matter and my interpretation might be wrong, so if you have references that clearly show my arguments and references above to be incorrect please let me know (ideally by also posting those references you have in mind).
I think you meant you reference https://www.itu.int/rec/T-REC-G.993.2, NOT G.992.3 : Asymmetric digital subscriber line transceivers 2 (ADSL2)
Argh, so VDSL2 is defined in G.993.2, but 993.2 in Annex K references G.992.3 Annex N:
" K.3.8 Functionality
The functionality of the PTM-TC shall implement 64/65-octet encapsulation as defined in AnnexN/G.992.3 [10], and shall include encapsulation, packet error monitoring, data rate decoupling, and frame delineation."
I guess I should make that chain reference clearer.
Forgive me if I sounded brash in my previous post, but as I may be a little thick in my understandings' of the documents; it appears as though the PTM-TC is a constant. It doesn't change as the "technology" is upgraded. Except for the latest GFAST, which is still in the works.
Well, compare ITU-T Rec. G.993.1 (06/2004) Annex H with IEEE 802.3-2012 61.3, you will notice that down on the wire these are quite different. VDSL2's PTM uses a fixed 64/65-encoding while VDSL1's PTM basically uses HDLC, these are quite different beasts at least on that level.
True. SO do we include both 4 bytes for PTM and the 64/65 in our calculations or just the 4 ptm bytes for overhead and goodput?
To some degree yes. IMHO the fact that it is one of the ADSL standard that introduced PTM is indicative that the ITU planned an exit route from the previously envisioned all-ATM world. It just seems that switching customers to VDSL2 is a better upgrade path for ISPs (who would face considerable upgrade pain if they would use PTM on ADSL-links given all the old ATM-only CPE equipment in the field).
I recommend to take both into account, but using slightly different methods:
The four "named" PTM bytes are per-packet and hence need to be accounted per packet and hence increase the per-packet-overhead to be accounted for.
The sync byte after every 64 "payload-bytes" however are independent of packet size, as the standard clearly shows they are continuously sent even if the link idles; it is far easier to fold these into the setting the shaper-rate as a simple static one-time reduction by 100-100*64/65 = 1.54%. That is much less involved compared to doing that "extension" for every packet (and the per packet method will introduce more rounding error, at least for reasonably fast links).
Side-note: ATM/AAL5 is not suited for such a static accounting, but there the additions are guaranteed to be integer numbers of bytes so at least there is no rounding error...
It makes perfect sense now. You are indeed a "DSL" wealth of knowledge lol
it's difficult to approximate my wan ingress max rate goodput as there is a shaper on the line (which is could be a good thing given what shaper is actually in use). My best guess from my modem <> dslam is that the synce rate for ingress 60022kbits, with some form of overhead averaging about 1.2%, which leaves an average of 59300kbits. This is where the shaper comes in, leaving about 55500kbits for throughput. The egress side show the same 1.2% but is not shaped in the modem, which is in ip passthrough mode. it's throughput is about 12800kbits, leaving 12064 on the table.
It might be better to move a discussion for you specific link to a new thread, instead of adding it to the end of something old?
It's possible I'm getting confused here - but is the idea that I look at the max_packet size and then set the overhead using the above? I currently have it set to 26 (optimizing for the same ISP using VDSL2 and DHCP - which is why I'm replying here), but if I look at the output of tc, it seems the maximum packet size is 1514.
Presumably also this is a calculation that persists - as I can change the overhead setting in the GUI (which stops and starts sqm), but the packet size persists.