Maybe so, as what we're doing in absence of proper docs is rightly classified as something worse than alpha. The best docs they had so far up to this point:
These are great, for someone who knows Kubernetes. You should understand that creating the tailscale subnet router as a pod directly means the connection is not resilient. It's also key to understand that tailscale will break if you have more than one instance of subnet router at a time, so substituting Deployment in place of where these docs use Pod is not a really good choice without some fine tuning because of the risk that a rolling update creates another copy of the pod before the old one has shut down.
Maybe stateful set, if there was a way to permanently imply that statefulset can only have one replica. I appreciate the link anyway. I figured all this out on my own, and I'm using tailscale productively with Kubernetes based on the old docs, with my open source project. Tailscale has a great and generous free software tier for supporting OSS maintainers. :tada:
That's really neat. Cloudflare tunnel for external customer egress, and Tailscale for internal tool egress. No more costly cloud specific load balancers !
My only issue with Tailscale was that it can't seem to stay logged in longer than something like 45 or 90 days.. making it a fun toy, but not for enterprise use.
As someone who travels a lot with machines all over the world, if a node goes offline I can ask someone to reboot a machine .. but there is no way I am giving random people credentials to my machines and network to fix issues.
There is an option to disable key expiry in the machine settings, unless you're talking about a different issue / bug. In my case, simply turning off key expiry is enough to keep the machine online for months inside tailscale network so far.
Hey ! I see Tailscale Funnels as maybe a good replacement for Ngrok, but not for Cloudflare Tunnels.
Your Funnels are in Beta, MUST use your tailnet’s domain name, have bandwitdh limits, no failover and no load balancing. If my website goes down, I close shop. Cloudflare Tunnels are just way more mature for production loads. CF Tunnels technically don't have load balancing, but if you set multiple Tunnels with the same ID, you get some sort of load balancing AND failovers if a tunnel goes down. And after that, they have a paid Load Balancer option.
Even for internal admin portals, the mention that "Traffic over Funnel is subject to bandwidth limits." with absolutely no defined numbers is just a turn off. If you added a number to that, like a limit of MBPS or GB/Month of transfer, it would be something I can bring to my colleagues, something we can discuss and weight on. For now, with no number, it's just a threat.
Everything else about Tailscale is chefskiss tho ;)
We have collectively decided as an industry to place infinite trust in a handful of vendors for the sake of convenience.
Auth providers (OIDC), cloud providers, and numerous software repositories and SaaS providers effectively have root on the entire universe. A major compromise of any number of these large vendors could expose millions of systems to total compromise.
I'd say nation states, which means the US and possibly other nation states effectively have root on virtually all SaaS systems.
It's exactly the architecture I'd promote if I wanted a total panopticon, but I'm not suggesting a conspiracy. Ease of use is the most powerful force in computing and history has shown that people will trade privacy, security, freedom, cost, and virtually anything else for it.
It does make a certain amount of sense. Time is non-fungible; no amount of money can buy more of it. So ease of use by saving time is extremely valuable and commands a high price.
The problem is that IMO the cost of all this SaaS automation is higher than most people understand. There's some rather huge hidden costs here.
That just moves the trust requirement up the chain, and assumes that the image wasn’t compromised pre-signing (either intentionally or unintentionally)
What's the alternative? I don't know of any reason to think that tarballs or <os> installers are any better. I suppose I could clone the code and look for security flaws myself but I'm no expert and on something like nginx it's certain to be a waste of my time.
While I understand the argument you're making, the exposure of running "curl | bash" on your local machine is much higher than running arbitrary code inside of a container.
Even if you specify a hash, are you actually checking all of the code and binaries in that image?
What about the image base (e.g. Alpine, Debian) and their packages?
That depends on how you're running it. You wouldn't want a compromised image to be the base for your build containers, which get access to your product's source code via volume mount, or to secret keys via environment variables.
I'm a big fan of Nix Flakes but they appear to have the same issue here as "curl | bash".
I mention this because Flakes can do much of what Docker can do, but do not use containers or virtualization.
The official NixPkgs are carefully vetted. Nix Flakes make it more convenient to pull in sources from outside NixPkgs.
As Nix Flakes become more prevalent, perhaps there needs to be some way to containerize them. There are more benefits to Nix Flakes than native execution so containerizing them somehow might be an acceptable compromise.
One thing that i think is really missing is redundancy on the proxies
Currently there's no way to have two proxies that listen on the same tailscale hostname/ip
Ideally in Kubernetes every pod is redundant to allow downscaling of nodes efficiently, so this means we have to eat a minute or so of downtime randomly every now and then on our tailscale endpoints
I think an alternative solution would be nice, where services are just registered with their service discovery, though I suppose that would require them providing an official API for that part of their product.
In my current setup for Tailscale + Kubernetes, I just use their subnet router[0] and add the kubedns server for the cluster.local domains to their MagicDNS. Having proper service discovery would make this into a nice round solutionl.
55 comments
[ 2.8 ms ] story [ 225 ms ] threadKeep it simple st*pid!
Keep it simple, smarty
https://tailscale.com/kb/1185/kubernetes/
These are great, for someone who knows Kubernetes. You should understand that creating the tailscale subnet router as a pod directly means the connection is not resilient. It's also key to understand that tailscale will break if you have more than one instance of subnet router at a time, so substituting Deployment in place of where these docs use Pod is not a really good choice without some fine tuning because of the risk that a rolling update creates another copy of the pod before the old one has shut down.
Maybe stateful set, if there was a way to permanently imply that statefulset can only have one replica. I appreciate the link anyway. I figured all this out on my own, and I'm using tailscale productively with Kubernetes based on the old docs, with my open source project. Tailscale has a great and generous free software tier for supporting OSS maintainers. :tada:
TrueCharts charts have Tailscale support, but not all my charts are from there, and also I'm kind of avoiding them.
Also, it's nice if you deploy something without a chart.
As someone who travels a lot with machines all over the world, if a node goes offline I can ask someone to reboot a machine .. but there is no way I am giving random people credentials to my machines and network to fix issues.
API keys have 90 day expiry but you can get around that with an oauth app that has credentials that don't expire
Your Funnels are in Beta, MUST use your tailnet’s domain name, have bandwitdh limits, no failover and no load balancing. If my website goes down, I close shop. Cloudflare Tunnels are just way more mature for production loads. CF Tunnels technically don't have load balancing, but if you set multiple Tunnels with the same ID, you get some sort of load balancing AND failovers if a tunnel goes down. And after that, they have a paid Load Balancer option.
Even for internal admin portals, the mention that "Traffic over Funnel is subject to bandwidth limits." with absolutely no defined numbers is just a turn off. If you added a number to that, like a limit of MBPS or GB/Month of transfer, it would be something I can bring to my colleagues, something we can discuss and weight on. For now, with no number, it's just a threat.
Everything else about Tailscale is chefskiss tho ;)
The whole industry does it, and it's the same as "curl | bash" to specify image:tag and not image@hash.
If I were TAO I'd apply for a job at Docker or their hosting provider.
And that's why I have an in-between step.
Harbor.io allows you to configure it as a proxy with approval mechanism and cve scanning
We won't even do this for webpages, but we find it a fine default for code that executes inside critical infrastructure.
It's utter madness. Cool to see someone is doing something about it.
Auth providers (OIDC), cloud providers, and numerous software repositories and SaaS providers effectively have root on the entire universe. A major compromise of any number of these large vendors could expose millions of systems to total compromise.
It's exactly the architecture I'd promote if I wanted a total panopticon, but I'm not suggesting a conspiracy. Ease of use is the most powerful force in computing and history has shown that people will trade privacy, security, freedom, cost, and virtually anything else for it.
It does make a certain amount of sense. Time is non-fungible; no amount of money can buy more of it. So ease of use by saving time is extremely valuable and commands a high price.
The problem is that IMO the cost of all this SaaS automation is higher than most people understand. There's some rather huge hidden costs here.
Harbor + Notary + admission controllers - AKA private image repository with image signing.
Sigstore. Another method for signing & verifying artifacts.
While I understand the argument you're making, the exposure of running "curl | bash" on your local machine is much higher than running arbitrary code inside of a container.
Even if you specify a hash, are you actually checking all of the code and binaries in that image?
What about the image base (e.g. Alpine, Debian) and their packages?
I mention this because Flakes can do much of what Docker can do, but do not use containers or virtualization.
The official NixPkgs are carefully vetted. Nix Flakes make it more convenient to pull in sources from outside NixPkgs.
As Nix Flakes become more prevalent, perhaps there needs to be some way to containerize them. There are more benefits to Nix Flakes than native execution so containerizing them somehow might be an acceptable compromise.
You may be interested in nix snapshotter, which can run nix packages in containerd.
https://news.ycombinator.com/item?id=37407758
Currently there's no way to have two proxies that listen on the same tailscale hostname/ip
Ideally in Kubernetes every pod is redundant to allow downscaling of nodes efficiently, so this means we have to eat a minute or so of downtime randomly every now and then on our tailscale endpoints
Not ideal for anything customer facing but fine enough for staff.
These are the relevant Github issues to follow, hopefully they address these someday:
https://github.com/tailscale/tailscale/issues/465 https://github.com/tailscale/tailscale/issues/4324
Now imagine running derp-server with in the DC with your k8s.
In my current setup for Tailscale + Kubernetes, I just use their subnet router[0] and add the kubedns server for the cluster.local domains to their MagicDNS. Having proper service discovery would make this into a nice round solutionl.
[0]: https://tailscale.com/kb/1185/kubernetes/#subnet-router