I used to run a similar setup. It was nice, but had to be maintained, and remote access is not exactly the thing I would like to see breaking.
I switched to Tailscale. It has an ample free tier for individuals and everything just works. It is really a good product.
I have some doubts about its capacity to scale, from a maintenance perspective. The interface is sometimes tiring (especially search) and some decisions are a bit counterintuitive. I, however, really hope they are doing fine because it is fantastic.
I run a similar setup. I didn't end up going for a full router-side hub inside the LAN, to avoid extra hops for things when they're inside the network.
Some of my notes:
* I have multiple servers outside the LAN that have WG connections back to specific interior systems for things like metrics/log-shipping, monitoring, and backups. All of those are "point-to-point" (the cloud-hosted systems have multiple Peers for different internal systems). This means that they can only talk to the things I expect them to, w/o me needing to also handle that at the firewall level.
* The Wireguard Mac and iOS apps are very solid. I use them heavily on multiple devices, and they can be trained for OnDemand activation based on SSIDs. This means that when my phone tries to load my NVR dashboard, it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub). The iOS app is reliable enough that I set it up once on my partner's phone so they could access the cameras and then never had to think about it again.
* Because the only things you need to set up a peer link are shared pubkeys and to agree on IPs, wiring up my endpoints via Puppet was super smooth and adding/adjusting has likewise been smooth. My systems generate a keypair during setup, publish their public key where the other nodes can find it, and all I have to do is update the map of "hey, this server get this WG-internal IP, and this is who should link to it".
> it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub)
Appropriate netmasks in the routing tables do this regardless of physical interfaces. For example, if your local network has netmask 255.255.255.128, the local link is prioritised over the tunnel link which could have netmask 255.255.255.0, but you have to decide which nets/subnets are useful to you.
Not really? That would rely on my local devices sharing a subnet with my phone/laptop, and would also blow up in fun ways if I ever got on a hotel wifi that used similar subnets to my home network.
You should pick a random network (like in 10.x.x.x if ipv4) and a smaller mask for your VPN than any public wifi would ever choose. Then a yet smaller mask for your home network. The probability of collision then is tiny.
I don’t think you’re quite grokking the problem space here.
The goal is “I want my phone to always connect to a given hostname. When I’m on my home WiFi, that connection should be direct. When I’m not, that connection needs to bounce through the WireGuard tunnel”.
The target system isn’t even on the same subnet of my home LAN as my phone. So short of injecting custom routes, netmasks aren’t helping me at all. But even if they were, short of running split horizon DNS, I’d need my WireGuard subnet to be a superset of my home LAN, and for my home LAN to be a portion of that, and also my WireGuard subnet can’t conflict with whatever network I’m on. So I’ve now made a bunch of complex constraints and my home LAN is taking constraints from my remote access flow.
Or I just tell WireGuard to route the IP of the LAN-side system and to disable itself if I’m home.
I understand your goal. I've run this solution for more than a decade, originally with openvpn and multiple subnets.
Yes, it does require your local networks to be subnets of the VPN. If you can't control dhcp, then you can add static alias IPs to the local interfaces and thus have "another" local subnet for this purpose. In these cases, the wireguard link might need two addresses (AllowedIPs: 10.x.x.x/31) where one of them is configured on the local interface with the local subnet mask, the wireguard ip configured with the wider VPN mask.
Adding IP aliases doesn't seem to be possible on phones though ...
My solution to this is a pair of udp ports relayed to each other on a VPS via socat. Then you can point a home WireGuard server (behind cgnat etc) at one, and clients at the other, with no need to trust the vps
Or you can setup Yggdrasil by installing it on each machine and adding a few public volunteer peers to the configuration. My own summary of WireGuard->Nebula->Yggdrasil journey: https://news.ycombinator.com/item?id=43967082
I've looked at Yggdrasil multiple times now, and I always end up backing away because it's security critical software intentionally facing the public internet and talking to untrusted peers, but it's a smaller project with no security audit. In contrast, wireguard has a lot of eyes on it, simpler code that does less, only exposes a listen-only invisible UDP port to untrusted parties, and uses formal verification to ensure that its smaller codebase is secure. Perhaps I'm paranoid, but it's a really hard sell for me to use anything but wireguard:\
It's almost as if these people are offended someone would not use tailscale. Do you also find stories about people hiking and comment "I just look up scenic locations on google"? Why would you think your +1 to the easy, commercial route is at all interesting to a forum literally having Hacker in the name?
It is quite different here as the OP stated headscale which is own hosted and I understand they use just the tailscale clients. That is mine setup too after using tailscale first and then wanted something own hosted without my traffic going through someone else's network infrastructure (internet excluded of course)
In my opinion Tailscale is the realistic option for most people. The author is familiar with Tailscale having worked with it previously, but my interpretation is that he wanted to get more familiar with the underlying Wireguard technology.
“Each Tailscale agent in your distributed network streams its logs to a central log server (at log.tailscale.com). This includes real-time events for open and close events for every inter-machine connection (TCP or UDP) on your network.”
That reminds me of a complaint that I had with a visualization library that kept pushing the open source project to be tied into their hosting racket. I brought up how lots of security organizations at the time wouldn't understand how much of a massive problem it was for the end user to decide to make the data immediately available to the entire Internet. It feels wrong that such implications could be missed from a set of environment variables just being missing. Usually you want things to fail safely ...
Agreed it's a bit annoying how non-discoverable this is. Wish it was in some sort of onboarding prompt. But I can see Tailscale's argument for making it the way it is.
Tailscale is really targeting the business market, especially since their product is basically free for personal use. In a corporate environment, I imagine that the client logs are actually hugely valuable to the corporate customers themselves. It lets them see who is accessing what and is super critical when doing a post-mortem after a hack. (also no actual traffic content is logged)
But I still keep client logging disabled for my personal use.
Remember that the iOS iPhone app does not handle domain names with an A and AAAA correctly.
It stops at the first IPv4 address it encounters, which means it'll go over an 464XLAT proxy instead of using a direct AAAA connection on networks like T-Mobile.
Is there an easy setup so that this could be done with VPS relaying the traffic but not decrypting it?
So far, I’m thinking of something like FRP to proxy incoming UDP from internet to the private network. I’m not sure if that works if only outgoing 443 is open at private network.
An issue with his remote setup is that the remote VPS decrypts packets from the remote laptop, then re-encrypts them for the LAN — this means that the remote VPS can see the plaintext of all those packets. He’ll need to layer TLS or something similar, or run Wireguard over Wireguard.
As somebody with a very similar setup, all the things I'm making accessible over the Wireguard network are HTTPS, SSH, etc. The handful of things that couldn't do native TLS (or were irritating to get configured with automated TLS certs) I stuck behind nginx for TLS.
Short of setting up dns validation and using 3rd party dns service as many registrars don't support API for dns management, how is domain validation done for acquiring TLS certs when serving only via wireguard ?
There's the private CA route but its a pain to setup the certs on all (mobile) devices and Android makes it very scary and hard.
I use https://github.com/go-acme/lego and DNS validation. I'm not sure what DNS provider you're using, but a ton of them have robust APIs. In my case, I'm using Route53 which is notable in that I can generate API credentials that can only update the specific record needed for a particular name's ACME validation record.
What options are available to use a remote VPS to facilitate connecting Wireguard directly through the CGNAT? It seems most "client" devices are going to be behind at least some kind of NAT as well.
STUN and TURN is what tail/headscale use, basically you can use whatever you want to map ports but if you want to hole-punch you are probably going to have to use something off the shelf or write your own client to integrate wg and the hole-punching logic.
The refusal to use a control plane tells me the author would probably enjoy openvpn much more than vanilla wireguard.
It’s so nice to edit openvpn configuration, maybe some ccd file, restart the openvpn server and see clients reconnect one by one and pick up new options (eg: nee routes etc)
As a random aside: in a past life, I was helping somebody with a project where we needed a VPN connecting two locations. We were working with OpenVPN, and the nodes we had to handle the connection were beefy with large uplinks but we were hitting the single-core limitation so it didn't matter. We ended up building a proof of concept that launched one OpenVPN instance per core and then bonded them together, which let us get way closer to line rate.
Thankfully we never had to try them for real to see how horribly that would have gone under real load.
AES-ni is enabled and Linux confirms it’s enabled and openSSL has it enabled, but I found no easy way to check if it’s actually being used (I found a link long ago but lost it :( )
I was using either AES-256-GCM or AES-256-CBC.
It could also be default configs not set right. Brief google search tells me to tweak myriad of buffers and config options… Some saying without changing buffers they were limited to 100mbps for example. Lots said changing to udp/changing mtu/buffer/etc helped…
I agree with you that it should be fine/fast enough. That was my expectation too! However my testing in real life showed it not to be and it’s a common issue for openvpn. The easiest solution seems to be wire guard rather then tweaking random stuff with no idea what’s bottlenecks
I use defined.net (managed Nebula) for my homelab and side project overlay networks, and it works great. They have a very generous free tier and I automate enrollment with a set of systemd units that I wrote. Configuration is very easy and the Nebula android app does what I need so I can access everything over my phone. A few friends use wg-easy to manage their wireguard setups, and they sing its praise.
I still use wireguard for simple point to point tunnels into my datacenter rack but anything important I use Nebula.
I did the p2p between any sensible nodes in my network then added routing layer via BGP so any node can "see" any other node regardless of underlying mesh.
The access limits are better served by firewalls than limiting tunelling
FWIW I found using one wireguard interface/instance too restrictive. The problem is that wireguard is an L3 interface that has its own internal IP routing table, so only one node can serve as a default (horizon) route for the instance. So for my setup, on each node I have a separate wg interface for every peer node. Every packet that goes into a given interface is destined to come out to the one peer on the other side.
I ask because using Wireguard for internal LAN connections is excessive in many (most?) situations. I get zero trust, but not trusting anything on your LAN introduces way too many operational hurdles and overhead.
Having been travelling just now with Wireguard as my primary VPN the main thing I noticed is it's completely unusable on airport wifi.
So the big feature I need is any sort of gateway which works over port 80 or 443 natively - ideally at a subpath so it's easy put behind a reverse proxy.
This. Our courtesy corp network (for people’s personal devices) has a number of open ports on TCP. But almost none on UDP. Had to put an exception (favor really) with network security to open a port on UDP for my VPN
Some IsPs block WireGuard regardless of the port - I’ve got entry nodes on 53, 443 and a high port and some places (Egypt was quite common) block them all - I assume with deep packet inspection looking for the WireGuard headers
I used to run a similar setup. It was nice, but had to be maintained, and remote access is not exactly the thing I would like to see breaking.
I switched to Tailscale. It has an ample free tier for individuals and everything just works. It is really a good product.
I have some doubts about its capacity to scale, from a maintenance perspective. The interface is sometimes tiring (especially search) and some decisions are a bit counterintuitive. I, however, really hope they are doing fine because it is fantastic.
The Plan B is to host Headscale
Tailscale is awesome. Something like it should have been part of IPv6
I run a similar setup. I didn't end up going for a full router-side hub inside the LAN, to avoid extra hops for things when they're inside the network.
Some of my notes:
* I have multiple servers outside the LAN that have WG connections back to specific interior systems for things like metrics/log-shipping, monitoring, and backups. All of those are "point-to-point" (the cloud-hosted systems have multiple Peers for different internal systems). This means that they can only talk to the things I expect them to, w/o me needing to also handle that at the firewall level.
* The Wireguard Mac and iOS apps are very solid. I use them heavily on multiple devices, and they can be trained for OnDemand activation based on SSIDs. This means that when my phone tries to load my NVR dashboard, it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub). The iOS app is reliable enough that I set it up once on my partner's phone so they could access the cameras and then never had to think about it again.
* Because the only things you need to set up a peer link are shared pubkeys and to agree on IPs, wiring up my endpoints via Puppet was super smooth and adding/adjusting has likewise been smooth. My systems generate a keypair during setup, publish their public key where the other nodes can find it, and all I have to do is update the map of "hey, this server get this WG-internal IP, and this is who should link to it".
> it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub)
Appropriate netmasks in the routing tables do this regardless of physical interfaces. For example, if your local network has netmask 255.255.255.128, the local link is prioritised over the tunnel link which could have netmask 255.255.255.0, but you have to decide which nets/subnets are useful to you.
Not really? That would rely on my local devices sharing a subnet with my phone/laptop, and would also blow up in fun ways if I ever got on a hotel wifi that used similar subnets to my home network.
You should pick a random network (like in 10.x.x.x if ipv4) and a smaller mask for your VPN than any public wifi would ever choose. Then a yet smaller mask for your home network. The probability of collision then is tiny.
I don’t think you’re quite grokking the problem space here.
The goal is “I want my phone to always connect to a given hostname. When I’m on my home WiFi, that connection should be direct. When I’m not, that connection needs to bounce through the WireGuard tunnel”.
The target system isn’t even on the same subnet of my home LAN as my phone. So short of injecting custom routes, netmasks aren’t helping me at all. But even if they were, short of running split horizon DNS, I’d need my WireGuard subnet to be a superset of my home LAN, and for my home LAN to be a portion of that, and also my WireGuard subnet can’t conflict with whatever network I’m on. So I’ve now made a bunch of complex constraints and my home LAN is taking constraints from my remote access flow.
Or I just tell WireGuard to route the IP of the LAN-side system and to disable itself if I’m home.
I understand your goal. I've run this solution for more than a decade, originally with openvpn and multiple subnets.
Yes, it does require your local networks to be subnets of the VPN. If you can't control dhcp, then you can add static alias IPs to the local interfaces and thus have "another" local subnet for this purpose. In these cases, the wireguard link might need two addresses (AllowedIPs: 10.x.x.x/31) where one of them is configured on the local interface with the local subnet mask, the wireguard ip configured with the wider VPN mask.
Adding IP aliases doesn't seem to be possible on phones though ...
My solution to this is a pair of udp ports relayed to each other on a VPS via socat. Then you can point a home WireGuard server (behind cgnat etc) at one, and clients at the other, with no need to trust the vps
Or you can setup Yggdrasil by installing it on each machine and adding a few public volunteer peers to the configuration. My own summary of WireGuard->Nebula->Yggdrasil journey: https://news.ycombinator.com/item?id=43967082
I've looked at Yggdrasil multiple times now, and I always end up backing away because it's security critical software intentionally facing the public internet and talking to untrusted peers, but it's a smaller project with no security audit. In contrast, wireguard has a lot of eyes on it, simpler code that does less, only exposes a listen-only invisible UDP port to untrusted parties, and uses formal verification to ensure that its smaller codebase is secure. Perhaps I'm paranoid, but it's a really hard sell for me to use anything but wireguard:\
AFAIK Yggdrasil does not listen by default and you can make an entirely private Yggdrasil network.
But given both are in Go I would not expect RCEs to be common in either.
I just use Tailscale (headscale)... It's Wireguard in essence but much easier to deploy and maintain.
Without fail, in every post on HN mentioning Wireguard:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
It's almost as if these people are offended someone would not use tailscale. Do you also find stories about people hiking and comment "I just look up scenic locations on google"? Why would you think your +1 to the easy, commercial route is at all interesting to a forum literally having Hacker in the name?
It is quite different here as the OP stated headscale which is own hosted and I understand they use just the tailscale clients. That is mine setup too after using tailscale first and then wanted something own hosted without my traffic going through someone else's network infrastructure (internet excluded of course)
Don’t gatekeep. Headscale is interesting, non commercial and absolutely relevant to the discussion.
In my opinion Tailscale is the realistic option for most people. The author is familiar with Tailscale having worked with it previously, but my interpretation is that he wanted to get more familiar with the underlying Wireguard technology.
Important to know the security tradeoffs of the Tailscale commercial product. At the very least people will learn about all the pieces.
The privacy tradeoffs are much worse :/ https://tailscale.com/kb/1011/log-mesh-traffic
“Each Tailscale agent in your distributed network streams its logs to a central log server (at log.tailscale.com). This includes real-time events for open and close events for every inter-machine connection (TCP or UDP) on your network.”
RE: https://kieranhealy.org/blog/archives/2013/06/09/using-metad...
I believe setting the "TS_NO_LOGS_NO_SUPPORT=1" env disables this logging.
That reminds me of a complaint that I had with a visualization library that kept pushing the open source project to be tied into their hosting racket. I brought up how lots of security organizations at the time wouldn't understand how much of a massive problem it was for the end user to decide to make the data immediately available to the entire Internet. It feels wrong that such implications could be missed from a set of environment variables just being missing. Usually you want things to fail safely ...
Agreed it's a bit annoying how non-discoverable this is. Wish it was in some sort of onboarding prompt. But I can see Tailscale's argument for making it the way it is.
Tailscale is really targeting the business market, especially since their product is basically free for personal use. In a corporate environment, I imagine that the client logs are actually hugely valuable to the corporate customers themselves. It lets them see who is accessing what and is super critical when doing a post-mortem after a hack. (also no actual traffic content is logged)
But I still keep client logging disabled for my personal use.
Yep, despite the extremely FUD-y variable name, though there's still no way to opt out on iOS or Android: https://github.com/tailscale/tailscale/issues/13174
There's an unmerged PR for the Android client: https://github.com/tailscale/tailscale-android/pull/695
Headscale isn't commercial
Remember that the iOS iPhone app does not handle domain names with an A and AAAA correctly.
It stops at the first IPv4 address it encounters, which means it'll go over an 464XLAT proxy instead of using a direct AAAA connection on networks like T-Mobile.
Is there an easy setup so that this could be done with VPS relaying the traffic but not decrypting it?
So far, I’m thinking of something like FRP to proxy incoming UDP from internet to the private network. I’m not sure if that works if only outgoing 443 is open at private network.
An issue with his remote setup is that the remote VPS decrypts packets from the remote laptop, then re-encrypts them for the LAN — this means that the remote VPS can see the plaintext of all those packets. He’ll need to layer TLS or something similar, or run Wireguard over Wireguard.
As somebody with a very similar setup, all the things I'm making accessible over the Wireguard network are HTTPS, SSH, etc. The handful of things that couldn't do native TLS (or were irritating to get configured with automated TLS certs) I stuck behind nginx for TLS.
Short of setting up dns validation and using 3rd party dns service as many registrars don't support API for dns management, how is domain validation done for acquiring TLS certs when serving only via wireguard ?
There's the private CA route but its a pain to setup the certs on all (mobile) devices and Android makes it very scary and hard.
I use https://github.com/go-acme/lego and DNS validation. I'm not sure what DNS provider you're using, but a ton of them have robust APIs. In my case, I'm using Route53 which is notable in that I can generate API credentials that can only update the specific record needed for a particular name's ACME validation record.
Here's a list of supported providers: https://go-acme.github.io/lego/dns/
And in case you're curious, the API perms dance to do specific-record updates: https://github.com/armorfret/terraform-aws-r53-certbot/blob/...
Author here! Indeed, it is mostly HTTPS terminated by Caddy in the server at home. Otherwise, it is SSH.
WireGuard over WireGuard (WireGuard end-to-end encrypted hub and spoke) example https://www.procustodibus.com/blog/2021/12/wireguard-e2ee-hu...
What options are available to use a remote VPS to facilitate connecting Wireguard directly through the CGNAT? It seems most "client" devices are going to be behind at least some kind of NAT as well.
STUN and TURN is what tail/headscale use, basically you can use whatever you want to map ports but if you want to hole-punch you are probably going to have to use something off the shelf or write your own client to integrate wg and the hole-punching logic.
The refusal to use a control plane tells me the author would probably enjoy openvpn much more than vanilla wireguard.
It’s so nice to edit openvpn configuration, maybe some ccd file, restart the openvpn server and see clients reconnect one by one and pick up new options (eg: nee routes etc)
Only if you’re happy with your CPU being pegged at 100% whenever you go beyond 40 mbit/s.
I’d rather use wireguard and get 800 mbit/s.
Hasn't openvpn had hardware acceleration for some time now? Those days are long gone.
Nope it’s still busted and worse of all it’s single core bound!
As a random aside: in a past life, I was helping somebody with a project where we needed a VPN connecting two locations. We were working with OpenVPN, and the nodes we had to handle the connection were beefy with large uplinks but we were hitting the single-core limitation so it didn't matter. We ended up building a proof of concept that launched one OpenVPN instance per core and then bonded them together, which let us get way closer to line rate.
Thankfully we never had to try them for real to see how horribly that would have gone under real load.
It’s surprising to me how bad the scaling is too. A 200megabits/s i what I was able to get on a fairly modern 10900k!
This with the default cipher? From what I can see, AES on a modern CPU should be orders of magnitude faster than that.
AES-ni is enabled and Linux confirms it’s enabled and openSSL has it enabled, but I found no easy way to check if it’s actually being used (I found a link long ago but lost it :( )
I was using either AES-256-GCM or AES-256-CBC.
It could also be default configs not set right. Brief google search tells me to tweak myriad of buffers and config options… Some saying without changing buffers they were limited to 100mbps for example. Lots said changing to udp/changing mtu/buffer/etc helped…
I agree with you that it should be fine/fast enough. That was my expectation too! However my testing in real life showed it not to be and it’s a common issue for openvpn. The easiest solution seems to be wire guard rather then tweaking random stuff with no idea what’s bottlenecks
Are OpenVPN and Wireguard really comparable?
Wireguard is creating a series of point-to-point links with crypto that's basically impossible to mess up.
OpenVPN sets up a hub and spokes and has far more baggage carried around with it.
tinc is good for setup like that as it automatically meshes the nodes
I use defined.net (managed Nebula) for my homelab and side project overlay networks, and it works great. They have a very generous free tier and I automate enrollment with a set of systemd units that I wrote. Configuration is very easy and the Nebula android app does what I need so I can access everything over my phone. A few friends use wg-easy to manage their wireguard setups, and they sing its praise.
I still use wireguard for simple point to point tunnels into my datacenter rack but anything important I use Nebula.
https://github.com/quickvm/defined-systemd-units
I did the p2p between any sensible nodes in my network then added routing layer via BGP so any node can "see" any other node regardless of underlying mesh.
The access limits are better served by firewalls than limiting tunelling
This sounds interesting. Could you say more? Do you run BGP on the hosts themselves over the Wireguard tunnels? Do you self-manage a Wireguard mesh?
FWIW I found using one wireguard interface/instance too restrictive. The problem is that wireguard is an L3 interface that has its own internal IP routing table, so only one node can serve as a default (horizon) route for the instance. So for my setup, on each node I have a separate wg interface for every peer node. Every packet that goes into a given interface is destined to come out to the one peer on the other side.
Was this meant as a demonstrator?
I ask because using Wireguard for internal LAN connections is excessive in many (most?) situations. I get zero trust, but not trusting anything on your LAN introduces way too many operational hurdles and overhead.
I use Tailscale with headscale and it is very simple. I used to have manual WG but this is way easier.
Having been travelling just now with Wireguard as my primary VPN the main thing I noticed is it's completely unusable on airport wifi.
So the big feature I need is any sort of gateway which works over port 80 or 443 natively - ideally at a subpath so it's easy put behind a reverse proxy.
Are you saying the protocol is being blocked? I have used Wireguard over 443 at airports with no issues.
UDP can be blocked. Doesn't affect browsing DNS is local anyway.
However it's arranged generally rovust VPN requires being able to send TCP over a well known port - which Wireguard by default doesn't do.
So I'm currently looking for an Android VPN app which can multiplex via a long live lived TLS connection.
This. Our courtesy corp network (for people’s personal devices) has a number of open ports on TCP. But almost none on UDP. Had to put an exception (favor really) with network security to open a port on UDP for my VPN
Some IsPs block WireGuard regardless of the port - I’ve got entry nodes on 53, 443 and a high port and some places (Egypt was quite common) block them all - I assume with deep packet inspection looking for the WireGuard headers