VPN Policy-Based Routing + Web UI - ARCHIVE #1

Gives me

Chain PREROUTING (policy ACCEPT 2146 packets, 432K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  any    any     anywhere             anywhere             match-set wanroute dst /* openvpn-policy-routing */ MARK or 0x10000
    0     0 MARK       all  --  any    any     anywhere             anywhere             match-set tun0route dst /* openvpn-policy-routing */ MARK or 0x20000
    0     0 MARK       all  --  any    any     anywhere             anywhere             match-set tun1route dst /* openvpn-policy-routing */ MARK or 0x40000
    1    32 MARK       all  --  any    any     MBP.lan  anywhere             /* openvpn-policy-routing */ MARK or 0x20000

Chain INPUT (policy ACCEPT 1215 packets, 205K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 906 packets, 219K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   18  1132 TCPMSS     tcp  --  any    eth1.2  anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: wan (mtu_fix) */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  any    tun0    anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: wan (mtu_fix) */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  any    tun1    anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: wan (mtu_fix) */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 1423 packets, 649K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 2329 packets, 869K bytes)
 pkts bytes target     prot opt in     out     source               destination 

@dziny -- please try the latest ipk, I've fixed the mangle table flush problem there for now and will try to implement a solution similar to sqm-scripts which @hnyman described here: https://github.com/openwrt/packages/pull/4022

I suspect @hnyman is a professional educator in some capacity.

In build 4.1.0-1 I've implemented proper (sqm-like) management of iptables.

1 Like

Not really, but I hate to see people struggling with the same problems that have been discussed earlier in connection to other apps.

The easy way to learn good practices here is to look at the source code how other similar apps have solved the challenges earlier. In your case I guess SQM and mwan3 are probably the closest peers (as they use firewall marks and may modify routing rules).

@hnyman -- I've tried the master snapshot build earlier today and noticed that PROCD now gives an error for the command-less PROCD instances. I'm wondering if you know of a workaround (or a similar command-less PROCD script I can look at) so that no warning is displayed?

No help this time. I am no procd specialist.

But if there is a change in procd behaviour, you might need to check procd changelog:
https://git.lede-project.org/?p=project/procd.git;a=summary

I have retested the same scenario as above with the updated ipk. There was one difference, namely now whatismyipaddress.com from my MBP laptop gives me IP of my tun1 interface not tun0. But the desktop is honouring the rule and gives me IP of tun0 interface. Everything else behaves as before.

I wonder if there could something sneaky, like one of the clients uses ipv6 instead of ipv4 and thus does not follow the ipv4-base rule?

I have ipv6 disconnected for now so that should not be an issue. Neither sqm nor mwan3 are installed.

I appreciate you're sticking with it. Can you please post output of below on your router:

nslookup www.whatismyip.com
nslookup whatismyipaddress.com
ipset save
iptables -L -v -t mangle

The match-set rules (for domains in the ipset) are the first rules created in the chain, unless I misunderstand how it works, they should absolutely have priority over all other rules. I'm as puzzled as you are as to why it doesn't work.

This is what I'm seeing:

root@LEDE:~# nslookup www.whatismyip.com
nslookup: can't resolve '(null)': Name does not resolve

Name:      www.whatismyip.com
Address 1: 104.27.192.92
Address 2: 104.27.193.92
Address 3: 2400:cb00:2048:1::681b:c05c
Address 4: 2400:cb00:2048:1::681b:c15c
root@LEDE:~# nslookup whatismyipaddress.com
nslookup: can't resolve '(null)': Name does not resolve

Name:      whatismyipaddress.com
Address 1: 104.82.154.103 a104-82-154-103.deploy.static.akamaitechnologies.com
Address 2: 2a02:26f0:104:280::25ad
Address 3: 2a02:26f0:104:29b::25ad
root@LEDE:~# ipset save
create wanroute hash:ip family inet hashsize 1024 maxelem 65536
create tun0route hash:ip family inet hashsize 1024 maxelem 65536
add tun0route 104.103.184.231
add tun0route 104.82.154.103
create tun1route hash:ip family inet hashsize 1024 maxelem 65536
root@LEDE:~# iptables -L -v -t mangle
Chain PREROUTING (policy ACCEPT 133K packets, 40M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 133K   40M OVPBR_MARK  all  --  any    any     anywhere             anywhere            [goto]  mark match 0x0/0xff0000

Chain INPUT (policy ACCEPT 52961 packets, 13M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 50014 packets, 16M bytes)
 pkts bytes target     prot opt in     out     source               destination         
  717 40716 TCPMSS     tcp  --  any    eth1.2  anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: wan (mtu_fix) */ TCPMSS clamp to PMTU
  540 32416 TCPMSS     tcp  --  any    tun0    anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: wan (mtu_fix) */ TCPMSS clamp to PMTU
  139  8896 TCPMSS     tcp  --  any    tun1    anywhere             anywhere             tcp flags:SYN,RST/SYN /* !fw3: wan (mtu_fix) */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 85218 packets, 25M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 135K packets, 41M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OVPBR_MARK (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  any    any     anywhere             anywhere             match-set wanroute dst MARK xset 0x10000/0xff0000
   85 21639 MARK       all  --  any    any     anywhere             anywhere             match-set tun0route dst MARK xset 0x20000/0xff0000
    0     0 MARK       all  --  any    any     anywhere             anywhere             match-set tun1route dst MARK xset 0x30000/0xff0000
 3006  424K MARK       all  --  any    any     MBP.lan       anywhere             MARK xset 0x30000/0xff0000

Everything looks to be in order to me. My only suspicion would be the IPv6 routing is getting in a way.

I'll try to implement IPv6 routing soon, hopefully that would fix the problem.

1 Like

To help troubleshoot issues (and because I wanted a break from the routing coding and because learning json-parsing LEDE functions was fun) I've implemented the extra option: support.

root@EA8500:~# /etc/init.d/openvpn-policy-routing
Syntax: /etc/init.d/openvpn-policy-routing [command]

Available commands:
    ...
	support	Generates output required to troubleshoot routing issues
		Use '-d' option for more detailed output
		Use '-p' option to automatically upload data to paste.ee
		List domain names after options to include their lookup in report

Essentially if you run /etc/init.d/openvpn-policy-routing support it will print out information (similar to what has been requested earlier in this thread) which might be helpful to troubleshoot problems. With -d option it will be more detailed (by default output of some commands is grepped/trimmed to OVPBR-related items). With -p option it will automatically upload unsanitized output to paste.ee under my account, so I'll see a new paste (no need for you to publicly share the link). While paste.ee folks don't seem to index pastes (and promise not to look at them), be aware that your unsanitized data will be publicly available.

Finally if there're issues with specific domains, you can add domain names to be resolved as part of the support output after either/or both of -d and -p.

Full command can look like: /etc/init.d/openvpn-policy-routing support -d -p whatismyip.com whatismyipaddress.com .

I've set up a second tunnel, added a single dnsmasq ipset rule for it and I can't replicate your issue on my IPv6-lacking setup.

I'll see what I can do about IPv6.

Short update. I have installed the latest version from your website. I'm still seeing the same issue. dnsmasq ipset rule is only respected if there is no route based rule for the given IP. But if there is one the option is not respected.
I have tried to run
/etc/init.d/openvpn-policy-routing support -d -p
but the paste fails (have curl and certificated uploaded).

@dziny thank you! I've been waiting for paste.ee to resolve the token-related issue and was using temporary account until then, they've fixed the issue and deleted temporary account (like I've asked) without letting me know. :wink: In build 4.1.2-3 I've fixed the non-working paste.ee.

I'm puzzled and I hope someone more knowledgable about iptables can look at the code I've submitted. In the mean time, if you don't mind trying two things:

  1. Add 1-2 domains you want to be in ipset to the "internal" ipset management with resolve options: https://github.com/stangri/openwrt-packages/blob/openvpn-policy-routing/net/openvpn-policy-routing/files/README.md#internal-domain-based-policies-handling. It does the same thing dnsmasq does, but while dnsmasq can afford to resolve and add domains to ipset on request, internal handling resolves domains ahead of time. Make sure to set resolve=1 in the config like it says in README.

  2. If #1 fails, remove the offending domains from both dnsmasq's and internal ipsets (luci app allows you to configure dnsmasq ipsets only if it's installed and running, so you'd have to use uci commands or edit config file directly), resolve domain(s) you want routed differently and add their IP addresses with IP-based policies.

Please let me know how it goes when you have the time to test it. If #1 doesn't work, but #2 does, I will change the way the domain-based routing is managed internally to resolve domains and use their IP addresses in IP-based policies instead of adding domains to ipsets.

PS. If you try something and it doesn't work after /etc/init.d/openvpn-policy-routing reload and /etc/init.d/dnsmasq restart, try rebooting the router (if you can).

I have uploaded 2 configs to paste.ee

Just to confirm #1 fails to work, same symptoms. But the I tried #2 and I think I found the issue with the order in which the rules are implemented. I had 2 configs:

config policy
        option comment 'whatismyipaddress'
        option remote_addrs '104.103.184.231'
        option gateway 'tun0'

config policy
        option gateway 'tun1'
        option comment 'MBP'
        option local_addrs '192.168.8.16'

and things did not work as expected, everything went on my MBP though tun1 (last rule). So I swapped the order of the rules

config policy
        option gateway 'tun1'
        option comment 'MBP'
        option local_addrs '192.168.8.16'

config policy
        option comment 'whatismyipaddress'
        option remote_addrs '104.103.184.231'
        option gateway 'tun0'

and it worked! Everything except 104.103.184.231 (whatismyipaddress.com) went through tun1 but http://whatismyipaddress.com used tun0.
But this is exactly the opposite of what I expected. I think this might be the reason the ipset based rules are also failing! They are interpreted after my IP based rules. Weird isn't it?

1 Like

I guess what's happening is that once match happens, it continues to match based on other rules.

I've reversed (fully) the order of rules in 4.1.3-1. You can verify with /etc/init.d/openvpn-policy-routing support.

The rules should now be executed in the order they're listed in the luci app with domain-based rules having top priority. Hopefully. :wink:

Please test and confirm.

PS. dnsmasq wasn't very happy about me juggling the ipsets on my router, so it's best to reboot after you install the updated package.

2 Likes

Grab 4.1.3-3 I should have fixed the bug with duplicate rules if you ran /etc/init.d/openvpn-policy-routing start multiple times.

1 Like

I need to do more tests. But what I have tested so far (the setup above) finally seems to be OK!!! Thanks. If I understand correctly each rule applies a mark on a packet. So it's the final mark after it went through all rules is then used for routing. It makes sense now why it wasn't working before...

1 Like