For folks interested in the Kubernetes networking model, I recommend looking elsewhere. This post appears to be content marketing wrapped in a thin shell of introductory tutorial for an approach that isn't used by non-trivial deployments.
First, if you want to do a networking tutorial, start with something simple -- an HTTP server and curl, for example. You want to be able to tcpdump the traffic to understand what's going on under the hood. The first half of the blog post is some massively complex magic -- CRDs? An operator? Just why? -- and there's no reason to run an Elasticsearch instance just to test out packet routing.
Second, you almost certainly don't want to use kube-proxy. It uses (abuses?) iptables/nftables in a way that will make your sysadmins cry tears of blood. For small deployments, every major cloud provider (AWS, Azure, GCP, etc) has a CNI plugin that lets you allocate pod IPs out of a dedicated NAT prefix. For larger or bare-metal deployments, either use IPv6 natively (if available) or 6to4 (if on an IPv4-only network). I wrote a tutorial on the 6to4 approach[0], but honestly if you have someone on staff who is familiar with the Linux kernel network configs they'll probably have a better idea of how to set it up to work with your system.
Third, you probably want to avoid getting super-magical with your DNS. Approaches like that described in the article (coredns configured to directly resolve non-namespaced Kubernetes service names) have poor performance once you get beyond toy-sized clusters, and having to hunt down all the places your code does a single-name lookup is not fun. Instead, configure a "normal" DNS server (or equivalent non-DNS address resolver) to read Kubernetes-announced endpoints in bulk (with caching, etc), and use hostnames like `myservice.mynamespace.mycluster.yourproddomain.com`, which lets you (1) figure out where your packets are getting routed to, and (2) provision mTLS certificates to pods that let them authenticate themselves as a given service identity. Yes, it's longer, but your future self (or future underlings) will thank you.
Thanks for the comment. We totally agree with your suggested approach.
The reason we went for the example and set up in the blog post is that most people aren’t as familiar with the very basics. Therefore, we wanted to set up a plausible example (which is why we deployed ES instead of just a couple of busyboxes) and kept the examples self-contained.
> Second, you almost certainly don't want to use kube-proxy. It uses (abuses?) iptables/nftables […] every major cloud provider ([…] Azure […])
…yeah… the IP "pool" for Services on Azure does come from a pool of NAT IPs, but it's still an imaginary pool? (It's not a real IP in the cloud's network — which is also more NAT IPs.) AFAICT, it's not different than how the article describes. kube-proxy runs in the cluster, even. (This is w/ the Azure CNI.)
In fact, the Service IP pool is so imaginary, if you accidentally allocate the same CIDR in the real Vnet, AKS gets all sorts of mad at you, and withholds upgrades until you repent. Support, similarly, won't do anything, no matter how obviously unrelated your actual problem is.
Contrast to GCP's GKE: service IPs show up in a VPC, IIRC as "secondary" IPs for a subnet. That does happen closer to how I think you're describing it. (GKE's model is night and day better. I hate finding out that, somehow, somewhere, in our past, someone created an overlapping network in Azure. It's a PITA to deal with, every time it happens.)
(Also, how much I'd love to jettison everything that has anything to do with NAT for IPv6.)
Also, IIRC, if you debug an AKS node, you'll see the iptables "abuse" you mention. And yes, it makes me cry tears of blood. I think IPVS is supposed to straighten some of that out and make me cry less tears of blood?
> provision mTLS certificates to pods that let them authenticate themselves as a given service identity.
You can do this with core-dns being your resolver, and using the normal unadorned service names. (CoreDNS really doesn't matter here.) It just changes what name you need to stick in the subjectAltNames. You'll need to "be your own CA" for this, of course. But cert-manager makes that pretty easy to do. (No more difficult than getting a cert from a real CA, and if anything … potentially easier.)
I've not had a problem with CoreDNS? Ours seems to cope fine at about 500 requests / min. The bigger problem is stuff in the cluster being really irresponsible about how many DNS queries they make…
Am I the only one who dislikes these hand drawn diagrams? At least the text parts of it. Pod looks like Pool, watch* looks like watcher and rewrite looks like rent. Look at the diagrams with both print as well as hand drawn text next to each other and tell me which one is easier to scan.
Objectively, there is no argument against print text being superior to hand written in every way from legibility, density and universality. If one only happened to have a real hand drawn diagram, you can of course excuse any lack of readability, but to go out of your way and intentionally degrade digital content to the level of hand drawn should be inexcusable. This is purely style over substance and that should have no place in technical content.
Yes, fortunately there is HNers that provide the right link to the (clickbait) title. But using HN for this «smart» ads leaves a bad taste and most likely backfires. Enough spam already
Not sure what you're getting at, cilium is built upon ebpf and xdp but that's not really anything to do with using bgp to announce networking directly instead of proxy traffic redirection. Those technologies seem to be used to simplify what needs to be announced from the clusters point of view but keep traffic flowing.
Confusingly there's also ebgp and ibgp to read up on. This is a dyslexics dream stack.
18 comments
[ 2.7 ms ] story [ 51.2 ms ] threadFirst, if you want to do a networking tutorial, start with something simple -- an HTTP server and curl, for example. You want to be able to tcpdump the traffic to understand what's going on under the hood. The first half of the blog post is some massively complex magic -- CRDs? An operator? Just why? -- and there's no reason to run an Elasticsearch instance just to test out packet routing.
Second, you almost certainly don't want to use kube-proxy. It uses (abuses?) iptables/nftables in a way that will make your sysadmins cry tears of blood. For small deployments, every major cloud provider (AWS, Azure, GCP, etc) has a CNI plugin that lets you allocate pod IPs out of a dedicated NAT prefix. For larger or bare-metal deployments, either use IPv6 natively (if available) or 6to4 (if on an IPv4-only network). I wrote a tutorial on the 6to4 approach[0], but honestly if you have someone on staff who is familiar with the Linux kernel network configs they'll probably have a better idea of how to set it up to work with your system.
Third, you probably want to avoid getting super-magical with your DNS. Approaches like that described in the article (coredns configured to directly resolve non-namespaced Kubernetes service names) have poor performance once you get beyond toy-sized clusters, and having to hunt down all the places your code does a single-name lookup is not fun. Instead, configure a "normal" DNS server (or equivalent non-DNS address resolver) to read Kubernetes-announced endpoints in bulk (with caching, etc), and use hostnames like `myservice.mynamespace.mycluster.yourproddomain.com`, which lets you (1) figure out where your packets are getting routed to, and (2) provision mTLS certificates to pods that let them authenticate themselves as a given service identity. Yes, it's longer, but your future self (or future underlings) will thank you.
[0] https://john-millikin.com/stateless-kubernetes-overlay-netwo...
The reason we went for the example and set up in the blog post is that most people aren’t as familiar with the very basics. Therefore, we wanted to set up a plausible example (which is why we deployed ES instead of just a couple of busyboxes) and kept the examples self-contained.
Btw, really liked your link. Great blog post.
…yeah… the IP "pool" for Services on Azure does come from a pool of NAT IPs, but it's still an imaginary pool? (It's not a real IP in the cloud's network — which is also more NAT IPs.) AFAICT, it's not different than how the article describes. kube-proxy runs in the cluster, even. (This is w/ the Azure CNI.)
In fact, the Service IP pool is so imaginary, if you accidentally allocate the same CIDR in the real Vnet, AKS gets all sorts of mad at you, and withholds upgrades until you repent. Support, similarly, won't do anything, no matter how obviously unrelated your actual problem is.
Contrast to GCP's GKE: service IPs show up in a VPC, IIRC as "secondary" IPs for a subnet. That does happen closer to how I think you're describing it. (GKE's model is night and day better. I hate finding out that, somehow, somewhere, in our past, someone created an overlapping network in Azure. It's a PITA to deal with, every time it happens.)
(Also, how much I'd love to jettison everything that has anything to do with NAT for IPv6.)
Also, IIRC, if you debug an AKS node, you'll see the iptables "abuse" you mention. And yes, it makes me cry tears of blood. I think IPVS is supposed to straighten some of that out and make me cry less tears of blood?
> provision mTLS certificates to pods that let them authenticate themselves as a given service identity.
You can do this with core-dns being your resolver, and using the normal unadorned service names. (CoreDNS really doesn't matter here.) It just changes what name you need to stick in the subjectAltNames. You'll need to "be your own CA" for this, of course. But cert-manager makes that pretty easy to do. (No more difficult than getting a cert from a real CA, and if anything … potentially easier.)
I've not had a problem with CoreDNS? Ours seems to cope fine at about 500 requests / min. The bigger problem is stuff in the cluster being really irresponsible about how many DNS queries they make…
[1] https://excalidraw.com/
Author here.
I’ve used Excalidraw and downloaded an external library with the k8s icons.
Also played around with non-default colors to help with info hierarchy.
Hope that helps!
If you need further info, my Twitter account is linked at the bottom of the post. Feel free to DM me there.
Objectively, there is no argument against print text being superior to hand written in every way from legibility, density and universality. If one only happened to have a real hand drawn diagram, you can of course excuse any lack of readability, but to go out of your way and intentionally degrade digital content to the level of hand drawn should be inexcusable. This is purely style over substance and that should have no place in technical content.
We’re writing a blog post about that right now. Should be up next week.
If you’re curious and would be up for a review, I’d be happy to send it to you first so you can read earlier!
bgp: border gateway protocol ebpf: extended berkely packet filter
Confusingly there's also ebgp and ibgp to read up on. This is a dyslexics dream stack.