VPN Bypass (split tunneling) Service + Luci UI

nnsense · April 30, 2018, 9:32pm

After I've read your comment, I've digged around a bit more about mwan3, and I've found a site stating that mwan is only needed for multiple wan connection, which is not my case.
As far as I know, since I've restored the router to default before adding vpn bypass, mwan3 is enabled by default on gl.inet b1300.
Anyway, since I really don't need that feature, I've removed it with opkg remove luci-app-mwan3 mwan3.
I've rebooted the router, and now the ip 192.168.8.115, which was the only configured in vpnbypass, has no connectivity to the router and (obviously) to internet.
Basically, whatever IP I configure in vpnbypass, it will get no access to the router.

Now iptables-save | grep xmark output is the following:

-A VPNBYPASS -p tcp -m multiport --sports 32400 -j MARK --set-xmark 0x10000/0xff0000
-A VPNBYPASS -s 192.168.8.115/32 -j MARK --set-xmark 0x10000/0xff0000
-A qos_Default -p udp -m mark --mark 0x0/0xf0 -m length --length 0:500 -j MARK --set-xmark 0x22/0xff
-A qos_Default -p icmp -j MARK --set-xmark 0x11/0xff
-A qos_Default -p tcp -m mark --mark 0x0/0xf0 -m tcp --sport 1024:65535 --dport 1024:65535 -j MARK --set-xmark 0x44/0xff
-A qos_Default -p udp -m mark --mark 0x0/0xf0 -m udp --sport 1024:65535 --dport 1024:65535 -j MARK --set-xmark 0x44/0xff
-A qos_Default_ct -p tcp -m mark --mark 0x0/0xf -m tcp -m multiport --ports 22,53 -m comment --comment "ssh, dns" -j MARK --set-xmark 0x11/0xff
-A qos_Default_ct -p udp -m mark --mark 0x0/0xf -m udp -m multiport --ports 22,53 -m comment --comment "ssh, dns" -j MARK --set-xmark 0x11/0xff
-A qos_Default_ct -p tcp -m mark --mark 0x0/0xf -m tcp -m multiport --ports 20,21,25,80,110,443,993,995 -m comment --comment "ftp, smtp, http(s), imap" -j MARK --set-xmark 0x33/0xff
-A qos_Default_ct -p tcp -m mark --mark 0x0/0xf -m tcp -m multiport --ports 5190 -m comment --comment "AOL, iChat, ICQ" -j MARK --set-xmark 0x22/0xff
-A qos_Default_ct -p udp -m mark --mark 0x0/0xf -m udp -m multiport --ports 5190 -m comment --comment "AOL, iChat, ICQ" -j MARK --set-xmark 0x22/0xff
-A qos_Default_ct -p tcp -m mark --mark 0x0/0xf -m tcp -m multiport --ports 22,53 -m comment --comment "ssh, dns" -j MARK --set-xmark 0x11/0xff
-A qos_Default_ct -p udp -m mark --mark 0x0/0xf -m udp -m multiport --ports 22,53 -m comment --comment "ssh, dns" -j MARK --set-xmark 0x11/0xff
-A qos_Default_ct -p tcp -m mark --mark 0x0/0xf -m tcp -m multiport --ports 20,21,25,80,110,443,993,995 -m comment --comment "ftp, smtp, http(s), imap" -j MARK --set-xmark 0x33/0xff
-A qos_Default_ct -p tcp -m mark --mark 0x0/0xf -m tcp -m multiport --ports 5190 -m comment --comment "AOL, iChat, ICQ" -j MARK --set-xmark 0x22/0xff
-A qos_Default_ct -p udp -m mark --mark 0x0/0xf -m udp -m multiport --ports 5190 -m comment --comment "AOL, iChat, ICQ" -j MARK --set-xmark 0x22/0xff

Here my whole iptables-save

stangri · May 2, 2018, 3:47am

Didn't realize it was a stock GL-Inet firmware. Latest LEDE ipq40xx snapshot images should work on B1300 just fine btw.

I'll post more commands which output I'll need to troubleshoot the service working on a stock GL-Inet firmware in a few days if that's the route you want to take.

nnsense · May 2, 2018, 10:25pm

I really need VPNBypass functionality, turning on and off my VPN every time I need a different service is a pain and using two different wifi AP providing different gateways is not a solution since the same device could have services that should got through VPN and other through local gateway.. So, yes, I would follow any advice to make it work.

Thanks

stangri · May 7, 2018, 2:08am

Just FYI, the latest ipq40xx-platform snapshots for B1300 seem to work just fine, so you may want to consider switching to vanilla LEDE.

Please post the output of:

dnsmasq --version
ip rule list
iptables -v -t mangle -S VPNBYPASS
ipset save

nnsense · May 7, 2018, 4:28pm

Hi

Outputs...

root@GL-B1300:~# dnsmasq --version
Dnsmasq version 2.78  Copyright (c) 2000-2017 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC no-ID loop-detect inotify

root@GL-B1300:~# ip rule list
0:      from all lookup 128
1:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

root@GL-B1300:~# iptables -v -t mangle -S VPNBYPASS
-N VPNBYPASS
-A VPNBYPASS -p tcp -m multiport --sports 32400 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPNBYPASS -s 192.168.8.115/32 -c 271 26669 -j MARK --set-xmark 0x10000/0xff0000

root@GL-B1300:~# ipset save
create vpnbypass hash:ip family inet hashsize 1024 maxelem 65536

Anyway if it sort my issues out, I will soon reinstall with the latest ipq40xx-platform snapshots for B1300.

Thanks

stangri · May 9, 2018, 6:00am

According to the counts, the packets are being marked. I'm guessing something in GL-Inet stock firmware is getting in the way.

nydssa · July 2, 2018, 5:23pm

Hi all,

I've got my configuration to work on the following hardware:
-WRT3200ACM
-LuCI openwrt-18.06 branch

One thing I can't get to work and I don't know why: Netflix. When connected to my VPN, I've excluded my own machine to perform some tests. I can see the IP from my ISP and can surf to any website, except for netflix.

It won't connect.. When disabling the VPN and surfing via my ISP, there is no issue. Netflix over VPN is also no problem, only slower.

Anyone got an idea?

Thank you in advance!

stangri · July 2, 2018, 10:06pm

Search the current and the archived threads for the VPN Policy Routing package for the word Netflix, I believe someone has posted a solution to Netflix problem there.

wilfortune · July 7, 2018, 2:12am

Did you have to do anything else? I cannot communicate with clients connected to VLANs inside the tunnel. Bypassing the VLANs I want excluded does seem to work though..

Edit: I could not access any client active on a VLAN that was included in the tunnel, either remotely or locally, despite whatever combination of local/remote ports I tried. Is VLAN compatibility a known issue? I only see @algebro mentioning them in this thread, so I'm inclined to believe I'm either doing something very wrong or communication with VLANs is not supported.

Cheers either way @stangri, crazy how simple you've made this. I had to shut down the VPN and your scripts because I must be able to RDP into hosts within the tunnel, and I had some impatient people to pacify. Would like to give it another go when time permits.

Keep up the good work

stangri · July 7, 2018, 9:20am

Have you set up VLANs to restrict them to the tunnel/WAN?

Yes, quite possibly neither this, nor VPR would work well if you've already tried to configure routing via VLANs.

wilfortune · July 10, 2018, 9:49pm

Perhaps I should describe my layout, and maybe you/someone can spot where I've made mistakes.

"lan" -> "eth0.63" -> 192.168.46.0/24
"vlan_d" -> "eth0.64" -> 192.168.47.0/24
"vlan_m" -> "eth0.65" -> 192.168.48.0/24
"vlan_p" -> "eth0.66" -> 192.168.49.0/24

Each VLAN has its own firewall zone but "lan" is the only one allowed to forward to others.

All of "vlan" firewall zones forward to "tunout", which then forwards to "wan". I designed it that way in the hopes that if the VPN went down, it would be a backup countermeasure against any temporary leakage.

The "lan" subnet is the only one directed outside the tunnel via vpnbypass.

What I've tried so far

Add line in /etc/iproute2/rt_tables => 10 dmz
ip rule add from 192.168.46.0/24 table dmz
For each subnet I want access to, added ip route add #SUBNET# via #GATEWAY# dev #VLAN.ID# table dmz

At this point, I was able to access the VLANs in the tunnel, but it did not survive a reboot/restart, so I tried adding it via /etc/config/network:

config rule
	option src '192.168.46.0/24'
	option lookup 'dmz'
	option priority '32764'

config route
	option interface 'vlan_d'
	option target '192.168.47.0/24'
	option gateway '192.168.47.1'
	option table 'dmz'

config route
	option interface 'vlan_m'
	option target '192.168.48.0/24'
	option gateway '192.168.48.1'
	option table 'dmz'

config route
	option interface 'vlan_p'
	option target '192.168.49.0/24'
	option gateway '192.168.49.1'
	option table 'dmz'

After much trial and error, I noticed that the rule for fwmark 0x10000 was superseding my routing rule:

root@LEDE:/etc/init.d# ip rule list
0:      from all lookup local
32764:  from all fwmark 0x10000 lookup 200
32765:  from 192.168.46.0/24 lookup dmz
32766:  from all lookup main
32767:  from all lookup default

So I added to /etc/firewall.user:

ip rule delete fwmark 0x10000 table 200
ip rule delete from 192.168.46.0/24 table dmz

ip rule add fwmark 0x10000 table 200
ip rule add from 192.168.46.0/24 table dmz

While that did not survive a restart/reboot, wasn't planing to reboot that often once things were setup properly. Running service firewall restart or /etc/init.d/firewall restart after rebooting enabled the access I needed, and I grew tired of trying to fix it any further.

That's when I decided to run some DNS leak tests. My ISP's DNS servers were all over the place, so I set about trying to fix that (per this post), and now have modified my /etc/config/dhcp thusly:

config dnsmasq
        ... unchanged values...
#       option resolvfile '/tmp/resolv.conf.auto'
        option noresolv '1'
        list server '209.222.18.222'
        list server '209.222.18.218'
        list server '8.8.8.8'
        list server '8.8.4.4'
        option localservice '1'
        list ipset '/hulu.com/netflix.com/vpnbypass'
        list ipset '/hbo.com/vpnbypass'

config dhcp 'lan'
        ... unchanged values...
        list dhcp_option '6,8.8.8.8,8.8.4.4'

This config works until I restart the firewall to regain local access. After I've done that, hosts connected to the "lan" subnet all have the VPN's IP address, not the one from the ISP.

At this point, I'm inclined to give up. I have no idea why the port forwarding in vpnbypass works remotely, but not locally. My feeling now is that it might be better to run the VPN client on a physically separate LEDE device or remove the "lan" subnet exclusion from vpnbypass.

Unless I'm doing something horribly wrong? Any takers?

I'm sill pretty new to all of this, and while I realize that LEDE wasn't for beginners, I would appreciate any help/guidance anyone may have for me. I'm closer than I was, so that feels like something.

stangri · July 11, 2018, 2:49am

Not that it matters much in your situation, but this:

Should prevent the domain-based policies from working.

As far as not being able to access the VLAN devices from LAN once you set any policies specifically for them -- that's the expected behaviour (at least with the current implementation of VPN Bypass/VPR). You may have a rule for LAN to forward to VLANs but if VLANs are marked to go into the table routing directly to WAN, you will not receive any packets back.

wilfortune · July 13, 2018, 6:04pm

Gotcha, thank you for the quick replies.

I have a feeling I'm doing something wrong somewhere, so I'll try again later.

bialy39 · November 7, 2018, 8:28am

Hello.
Is it possible to bypass vpn by apps installed on router itself?
I2pd and TOR installed on router are still using vpn tunnel.
Do somebody know how to make them using normal internet connection?

Regards.

kgursu · December 21, 2018, 12:09am

Hi, esp. @stangri.

Why is opkg suggesting me to downgrade VPN Bypass on Snapshots channel?

luci-app-vpnbypass git-18.350.50885-91a73be-4 » git-18.247.53383-f6dd876-7

Thanks.

cybrnook · January 17, 2019, 4:28am

@stangri what's up stranger! You still supporting this package, or is all the magic sauce going it's the VPN policy package now?

I got a fellow Maple Syrup lover I am getting setup and I am going to get him on one of your packages. Wanted to ask you first where I should direct him.

stangri · January 17, 2019, 5:03am

The vpnbypass is in the official repo and if their default OpenVPN config enables VPN routing by default, the vpnbypass would be less hassle for them to use, especially for simple configs.

For extended configs I'd recommend vpn-policy-routing tho.

cybrnook · January 17, 2019, 3:09pm

I assume enabling vpnbypass in the luci menuconfig will take care of the needful dependencies like unticking dnsmasq and integrating ipset iptables dnsmasq-full instead, or this has to also be taken into account?

dbischof90 · January 19, 2019, 4:52pm

Hey @stangri , while I haven't been able so far to set up the vpn-policy-routing package I thought I try this in parallel.
Turns out it's not really starting either. Maybe something is wrong with my system

I can see the following system log:

Sat Jan 19 17:39:50 2019 user.notice vpnbypass [6353]:  : iptables -t mangle -A VPNBYPASS -m set --match-set vpnbypass dst -j MARK --set-mark 0x010000/0xff0000
Sat Jan 19 17:39:50 2019 daemon.err modprobe: xt_set is already loaded
Sat Jan 19 17:39:50 2019 daemon.err modprobe: ip_set is already loaded
Sat Jan 19 17:39:50 2019 daemon.err modprobe: ip_set_hash_ip is already loaded
Sat Jan 19 17:39:50 2019 user.notice vpnbypass [6353]: service started with TID: 200; FW_MARK: 0x010000
Sat Jan 19 17:39:50 2019 user.notice vpnbypass [6353]: service monitoring interfaces: wan expressvpntun  ✓

etc/config/vpnbypass:

config vpnbypass 'config'
	list localport '22'
	list remoteport '22'
	option enabled '1'

etc/config/dhcp:

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.auto'
	option nonwildcard '1'
	option localservice '1'
	list ipset '/netflix.com/vpnbypass'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config domain
	option name 'router'
	option ip '192.168.1.1'

config domain
	option name 'drucker'
	option ip '192.168.1.122'

config host
	option name 'drucker'
	option dns '1'
	option mac '0E:80:62:D8:F4:F6'
	option ip '192.168.1.122'

iptables-save | grep xmark:

-A VPNBYPASS -p tcp -m multiport --dports 22 -j MARK --set-xmark 0x10000/0xff0000
-A VPNBYPASS -p tcp -m multiport --sports 22 -j MARK --set-xmark 0x10000/0xff0000

Again, the goal for now is:

route netflix/amazon outside of the VPN
open the SSH port (and potentially others later but that as a first example)
route the rest of the traffic normally through ExpressVPN

Am I missing something?

dk4dk4 · January 22, 2019, 2:48am

root@OpenWrt:~# dnsmasq --version
Dnsmasq version 2.80  Copyright (c) 2000-2018 Simon Kelley

root@OpenWrt:~# ip rule list
0:      from all lookup local
32765:  from all fwmark 0x10000 lookup 200
32766:  from all lookup main
32767:  from all lookup default

Hi Stangri,
I have just configured the VPNBYPASS module and so far so good.

Do you have to reboot after applying new rules?
Can you enter port ranges in a field say 8080-8088
Can you bypass for an IP and Port combo:
I want 192.168.1.2 to use the VPN tunnel, but I want only 192.168.1.2:443 to bypass the tunnel?

I do not want to exclude 443 for all devices, as some are PCs and some are mobiles with different 443 capabilities each..

Thanks!