VPN Policy-Based Routing + Web UI - ARCHIVE #1

in 4.1.4-21 I fixed a bug where an IPv6 table creation would generate an error if you have IPv6 thru VPN, but no IPv6 on LAN. If there's no IPv6 on LAN, no IPv6 tables would get created.

Hey I posted in the open-wrt forums, but this might be a better place since its more popular so more will see an answer.

Version info

root@OpenWrt:~# opkg list-installed | grep policy
luci-app-openvpn-policy-routing - git-17.080.69173-773734e27-4
openvpn-policy-routing - 4.1.4-21

I am currently running into two issues. 1) I cannot get domain policy to work. and 2) I cannot get remote-port policy to work. If I do a policy on just an IP/IP range, it seems fine.

For the domain policy

opkg update; opkg remove dnsmasq; opkg install ipset iptables dnsmasq-full
Package ipset (6.24-1) installed in root is up to date.
Package iptables (1.4.21-2) installed in root is up to date.
Package dnsmasq-full (2.76-1) installed in root is up to date.

I go to domains, and I have this as my "Domains Policies"

/whatismyipaddress.com/hulu.com/netflix.com/wanroute

Netflix still prevents me, and the whatiymyipaddress still shows my VPN IP.

As for my remote-port based policy.

Basically, I have remote ports 1001-65535 set to WAN for my local desktop. Some things, like EC2 instances, properly work with my WAN ip if I set my security group /firewall settings to allow my WAN IP. But stuff like Teamspeak and Steam, seem to still be using my VPN IP. I am not sure where this might be comming from.
One example, with my VPN on, my steam speeds are slow because of my VPN provider for whatever reason. It slowls builds up. But when I turn off my VPN, my speeds sky rocket. VPN provider issue aside, Steam is supposed to be using the following ports.

27000 through 27037

All of these should be included in my broad IP selection of 1001-65535 to WAN. So I am not sure what the issue is.

I also tried hosting a game server on port 2302, and when people connect, it shows them the VPN IP, so it seems my remote-port based routing is not working as intended?

root@OpenWrt:~# opkg list_installed | grep dnsmasq
dnsmasq-full - 2.76-1
root@OpenWrt:~# grep ipset /etc/config/dhcp
        list ipset '/hulu.com/netflix.com/nhl.com/whatismyipaddress.com/wanroute'

openvpn-policy-routing file

config openvpn-policy-routing 'config'
        option strict_enforcement '1'
        option verbosity '2'
        option enabled '1'

config domain-policy
        list ipset '/hulu.com/netflix.com/wanroute'
        list ipset '/whatismyipaddress.com/wanroute'

config policy
        option gateway 'wan'
        option comment 'FireTV'
        option local_addrs '192.168.1.150'

config policy
        option gateway 'wan'
        option comment 'MyPC-WAN'
        option remote_ports '1001-65535'

My DHCP Config

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.auto'
        option localservice '1'
        list ipset '/hulu.com/netflix.com/nhl.com/whatismyipaddress.com/wanroute'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv6 'server'
        option ra 'server'
        list dhcp_option '6,209.222.18.222,209.222.18.218'
        option ra_management '1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'

My DNSMasq config

# auto-generated config file from /etc/config/dhcp
conf-file=/etc/dnsmasq.conf
dhcp-authoritative
domain-needed
localise-queries
read-ethers
bogus-priv
expand-hosts
local-service
domain=lan
server=/lan/
ipset=/hulu.com/netflix.com/nhl.com/whatismyipaddress.com/wanroute
dhcp-leasefile=/tmp/dhcp.leases
resolv-file=/tmp/resolv.conf.auto
stop-dns-rebind
rebind-localhost-ok
dhcp-broadcast=tag:needs-broadcast
addn-hosts=/tmp/hosts
conf-dir=/tmp/dnsmasq.d
user=dnsmasq
group=dnsmasq




dhcp-range=lan,192.168.1.100,192.168.1.249,255.255.255.0,12h
dhcp-option=lan,6,209.222.18.222,209.222.18.218
no-dhcp-interface=br-wan

With DHCP option 6 you're instructing your clients to use OpenDNS for name resolution, so dnsmasq doesn't have a chance to resolve the domain names and add their IPs to the actual ipset.

That should be covered by local ports, not remote ports.

Me neither, can you run /etc/init.d/openvpn-policy-routing support (either /etc/init.d/openvpn-policy-routing support -p so it automatically uploads the paste under my account or PM me/post the output here) so I can verify if the port iptables rule was set up properly?

PS. Outstanding job on posting your current config details.

Did it just now with -p, so you should see it.

I guess I am confused, not a networking expert. So, I thought localports were the ones that were randomized on connection when you connected to a website? I guess I will need to fix that also then.

As for the dhcp option 6 you are talking about. I am confused on that, what should I be setting it to?

Not seeing the paste -- did you get any errors?

Local ports are the ports on your network for services like Plex, game servers, etc.

You can either remove option 6 from dhcp config or set it to your router's IP address (192.168.1.1).

Yeah, it was missing the following so I just installed them and ran it again.Surprised they were missing, I would assume libopenssl and ca-bundle were be a part of any linux build...

ERROR: curl, libopenssl or ca-bundle were not found!

EDIT: I also tried removing "list dhcp_option '6,209.222.18.222,209.222.18.218'" entirely, and I also changed it simply to 3, neither of those seemed to have fixed my netflix/hulu issue. I also added whatismyipaddress.com to it so I can more easily check my IP. so now my domain policies looks like

/whatismyipaddress.com/netflix.com/hulu.com/wanroute

Still not seeing the paste. And I've just done one myself to test if it still works -- and it does.

Did you restart dnsmasq afterwards? Also, your clients might have dns cache and not request resolution for known domain names. So I'd suggest restarting both the router and the clients.

Weird... well here it is.

============================================================
Dnsmasq version 2.76 Copyright (c) 2000-2016 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify

Routes/IP Rules
default 71.231.52.1 0.0.0.0 UG 0 0 0 br-wan
0: from all lookup local
32766: from all lookup main
32767: from all lookup default

IP Tables
OPR_CHAIN all -- anywhere anywhere [goto] mark match 0x0/0xff0000
-N OPR_CHAIN
-A OPR_CHAIN -s 192.168.1.70/32 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A OPR_CHAIN -m set --match-set wanroute dst -c 112 40060 -j MARK --set-xmark 0x10000/0xff0000
-A OPR_CHAIN -m set --match-set tun0route dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000

IPv6 Tables
OPR_CHAIN all anywhere anywhere [goto] mark match 0x0/0xff0000
-N OPR_CHAIN

Domain-based routing settings
dnsmasq.cfg02411c.ipset: /hulu.com/netflix.com/nhl.com/whatismyipaddress.com/wanroute
openvpn-policy-routing.cfg059dfa.domain-policy: /whatismyipaddress.com/netflix.com/hulu.com/wanroute

Current ipsets
create wanroute hash:ip family inet hashsize 1024 maxelem 65536
add wanroute 52.38.48.218
add wanroute 52.25.169.106
add wanroute 54.200.222.12
add wanroute 54.186.252.5
add wanroute 54.69.111.95
add wanroute 54.149.239.243
add wanroute 52.88.254.85
add wanroute 52.35.90.140
create tun0route hash:ip family inet hashsize 1024 maxelem 65536
create wanlist list:set size 8
add wanlist wanroute
create tun0list list:set size 8
add tun0list tun0route

I have restarted my router since then, I will do a dnscache flush and try again.

Just flushed my DNS, issue seems to still persist.I also tried on my cell phone and tablet. Both of which I actually havent used Netflix on yet, and the issue seems to persist with netflix.

So there's no rule for external ports there. Do you get any errors when you do /etc/init.d/openvpn-policy-routing reload? I just tested a similar rule and it worked on my system.

Safest way would be to remove option 6 from dhcp config, reboot router and reboot all clients. They might have cached dhcp option 6. You're the second person in this thread for whom domain-based routing didn't work because of dhcp option 6 until it was removed.

PS. You can remove ipset from /etc/config/openvpn-policy-routing, ipsets there are being ignored if the working and ipset-supporting dnsmasq is found on your router.

Ah yes, that was my bad, I deleted stuff and was going to rebuild from start, and sent you that dnsmasq before I redid everything.

I also completed deleted the dns option 6 line previous, before reloading everything.

Here is me running /init.d/openvpn-policy-routing reload. Looks like its failing on the domain part. Not very verbose though for output.

root@OpenWrt:~# /etc/init.d/openvpn-policy-routing reload
creating table wan/br-wan/71.231.52.1/wanroute                                [✓]
creating table PIA_VPN/tun0/10.63.10.5/tun0route                              [✓]
routing 'Local IP' 192.168.1.70 to *.*.*.* via wan                            [✓]
routing 'MyPC-WAN' 192.168.1.1/24 to *.*.*.*:1001-65535 via wan               [✓]
routing domains policies ✗
ERROR: failed to add whatismyipaddress.com to ipset wanroute failed to add netflix.com to ipset wanroute failed to add hulu.com to ipset wanroute
openvpn-policy-routing 4.1.4-21 started wan/71.231.52.1 PIA_VPN/10.63.10.5 ✓

Here is an updated dnsmasq.

root@OpenWrt:~# /etc/init.d/openvpn-policy-routing support
openvpn-policy-routing 4.1.4-21 running on OpenWrt Bleeding Edge. WAN (IPv4): wan/71.231.52.1.
============================================================
Dnsmasq version 2.76  Copyright (c) 2000-2016 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
============================================================
Routes/IP Rules
default         10.63.10.5      128.0.0.0       UG    0      0        0 tun0
default         71.231.52.1     0.0.0.0         UG    0      0        0 br-wan
32748:  from all fwmark 0x20000 lookup 201
32749:  from all fwmark 0x10000 lookup 200
============================================================
IP Tables
OPR_CHAIN  all  --  anywhere             anywhere            [goto]  mark match 0x0/0xff0000
-N OPR_CHAIN
-A OPR_CHAIN -p tcp -m multiport --dports 1001:65535 -c 504 32838 -j MARK --set-xmark 0x10000/0xff0000
-A OPR_CHAIN -s 192.168.1.70/32 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A OPR_CHAIN -m set --match-set wanroute dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A OPR_CHAIN -m set --match-set tun0route dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
============================================================
IPv6 Tables
OPR_CHAIN  all      anywhere             anywhere            [goto]  mark match 0x0/0xff0000
-N OPR_CHAIN
============================================================
Domain-based routing settings
dnsmasq.cfg02411c.ipset: /hulu.com/netflix.com/nhl.com/whatismyipaddress.com/wanroute
openvpn-policy-routing.cfg059dfa.domain-policy: /whatismyipaddress.com/netflix.com/hulu.com/wanroute
============================================================
Current ipsets
create wanroute hash:ip family inet hashsize 1024 maxelem 65536
create tun0route hash:ip family inet hashsize 1024 maxelem 65536
create wanlist list:set size 8
add wanlist wanroute
create tun0list list:set size 8
add tun0list tun0route
============================================================

Oh, my bad, it's been a while since I've looked at the code. The ipsets in the /etc/config/openvpn-policy-routing are not being ignored if they exist. I suspect that the errors you're seeing are caused by the duplicate ipsets for the same domains in dhcp and openvpn-policy-routing configs. They will be gone if you remove ipsets from /etc/config/openvpn-policy-routing.

Just to make sure it's the issue with duplicates, could you post the output of:

which dig
which resolveip

But as you can see the iptables rule for the remote ports is now set so it should work.

EDIT: Ignore That

Anyways, I reflashed my device. I will rebuild to where I was, make sure DNS option is disabled and check everything again.

Alright, after flashing my device and reconfiguring it, it looks like the domain portion works just fine. I am using netflix on my desktop right now and it is launching on my tablet!

However, my local ports portion does not seem to be taking right now still. Here is some updated info.

Routing reload

root@OpenWrt:~# /etc/init.d/openvpn-policy-routing reload
creating table wan/eth1/71.231.52.1/wanroute                                  [✓]
creating table wan6/eth1/fe80::e11:67ff:fe02:4822/wanroute6                   [✗]
creating table PIA_VPN/tun0/10.3.10.5/tun0route                               [✓]
routing 'MyWAN' 192.168.1.134 to *.*.*.*:1025-65535 via wan                   [✓]
routing domains policies ✓
openvpn-policy-routing 4.1.4-22 started wan/71.231.52.1 PIA_VPN/10.3.10.5 ✓

DHCP config

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.auto'
        option localservice '1'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv6 'server'
        option ra 'server'
        option ra_management '1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'

openvpn-policy-router

============================================================
Dnsmasq version 2.76  Copyright (c) 2000-2016 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
============================================================
Routes/IP Rules
default         10.3.10.5       128.0.0.0       UG    0      0        0 tun0
default         71.231.52.1     0.0.0.0         UG    0      0        0 eth1
32754:  from all fwmark 0x30000 lookup 202
32755:  from all fwmark 0x10000 lookup 200
============================================================
IP Tables
OPR_CHAIN  all  --  anywhere             anywhere            [goto]  mark match 0x0/0xff0000
-N OPR_CHAIN
-A OPR_CHAIN -s 192.168.1.134/32 -p tcp -m multiport --dports 1025:65535 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A OPR_CHAIN -m set --match-set wanroute dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A OPR_CHAIN -m set --match-set tun0route dst -c 0 0 -j MARK --set-xmark 0x30000/0xff0000
============================================================
IPv6 Tables
OPR_CHAIN  all      anywhere             anywhere            [goto]  mark match 0x0/0xff0000
-N OPR_CHAIN
-A OPR_CHAIN -m set --match-set wanroute6 dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
============================================================
Domain-based routing settings
openvpn-policy-routing.cfg059dfa.domain-policy: /whatismyipaddress.com/netflix.com/hulu.com/wanroute
============================================================
Current ipsets
create wanroute hash:ip family inet hashsize 1024 maxelem 65536
add wanroute 52.40.40.180
add wanroute 23.214.22.169
add wanroute 23.39.25.103
create wanroute6 hash:ip family inet6 hashsize 1024 maxelem 65536
create tun0route hash:ip family inet hashsize 1024 maxelem 65536
create wanlist list:set size 8
add wanlist wanroute
add wanlist wanroute6
create tun0list list:set size 8
add tun0list tun0route
============================================================

openvpn-policy-routing

config openvpn-policy-routing 'config'
option strict_enforcement '1'
option verbosity '2'
option enabled '1'

config policy
option gateway 'wan'
option comment 'MyWAN'
option local_addrs '192.168.1.134'
option remote_ports '1025-65535'

config domain-policy
list ipset '/whatismyipaddress.com/netflix.com/hulu.com/wanroute'

My DNSMasq (which seems to have changed a bit)

conf-file=/etc/dnsmasq.conf
dhcp-authoritative
domain-needed
localise-queries
read-ethers
bogus-priv
expand-hosts
local-service
domain=lan
server=/lan/
dhcp-leasefile=/tmp/dhcp.leases
resolv-file=/tmp/resolv.conf.auto
stop-dns-rebind
rebind-localhost-ok
dhcp-broadcast=tag:needs-broadcast
addn-hosts=/tmp/hosts
conf-dir=/tmp/dnsmasq.d
user=dnsmasq
group=dnsmasq

dhcp-range=lan,192.168.1.100,192.168.1.249,255.255.255.0,12h
no-dhcp-interface=eth1

Maybe because there's no policy for local ports?

May I ask you a few things tho:

  1. You keep creating an ipset in the /etc/config/openvpn-policy-routing rather than /etc/config/dhcp -- why and how? Are you editing the config file from CLI/shell or you're adding domains thru Web UI and that's where they're being added?
  2. I've noticed that the wan6 table errors out on creation and since I'm yet to set up 6-in-4 on my router I'm wondering if you could run some commands on yours.

Sorry, I meant REMOTE ports. My remote ports is still not working. Also, I think I was wrong. I checked netflix again this morning and its not working. I am wondering if I mistakenly had my VPN off. Going to check all my configs again.

Everything I am doing is from the GUI, I don't do anything from the CLI.

And yeah I can run some commands for the wan6 stuff.

Ah, you're on OpenWrt, aren't ya? The way dnsmasq works is a bit different between LEDE and OpenWrt, that's why the luci app doesn't detect you have dnsmasq working and lets you edit the ipset in the /etc/config/openvpn-policy-routing instead of /etc/config/dhcp.

Let me see if I can fix that quickly.

1 Like

I guess thats something else that confuses me. I thought i installed lede? Although it looks like its still OpenWRT. I was attempting to upgrade from OpenWRT to lede to fix a kernel issue I was having.

lede-17.01.0-r3205-59508e3-kirkwood-linksys-viper-squashfs-sysupgrade.tar

Khm, then luci app should have let you edit ipset in /etc/config/dhcp.

Since you left out the first line of the support call above, I can't tell which OS are you running now. :wink:

Sorry, late reply because I apparently hit the daily limit of posts for a new user.

root@OpenWrt:~# /etc/init.d/openvpn-policy-routing support
openvpn-policy-routing 4.1.4-22 running on OpenWrt Bleeding Edge. WAN (IPv4): wan/71.231.52.1. WAN (IPv6): wan6/fe80::e11:67ff:fe02:4822.
============================================================
Dnsmasq version 2.76  Copyright (c) 2000-2016 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify

So yeah, despite using a lede file, I some how ended up with OpenWRT. I assumed it was a fork of OpenWRT, thus why the logo was the same, b ut after some research, i see its quite different. So yeah, not sure what happened, but that explains a lot of the confusion.