Great article, I've been able to learn more about why a cluster setup could utilize linkerd. However this statement confused me a little:
However, Finagle is written in Scala, which Gould concedes is "not for everyone".
linkerd is written in Scala [1] and linkerd is written in Rust [2]. If he is optimizing for other people’s contributions, is this really the best way to go?
> linkerd is written in Scala [1] and linkerd is written in Rust [2]
Though your links are correct your statement isn't. Linkerd is written in Scala, but the Rust bit is linkerd-tcp, a TCP loadbalancer, which linkerd (the Scala one) doesn't handle as that covers HTTP and gRPC with optionally encrypting connections between linkers using TLS.
> If he is optimizing for other people’s contributions, is this really the best way to go?
But is he optimising for other people's contributions? Using a C/Rust for something like a TCP proxy makes a lot more sense to me than having a JVM do that. And if he perceives Scala as "not for everyone", maybe adding Rust means he might get more contributions to those components?
I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two. By spinning it out as a standalone component that acts as a proxy, like linkerd is doing, Scala doesn't get in the way and what Finagle offers can be used with a mesh of microservices written in a myriad of languages.
Sorry, not typing -tcp on linkerd-tcp was completely a typo on my end.
> I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two.
Thanks, this is the answer I have been looking for.
I appreciate the sentiment behind linkerd et al but I think we are really over-complicating things and tying ourselves into knots by going head-first into this world. I can't help but think we're going to regret it in a few years.
The complexity introduced not only by Docker, but also k8s and all of the other things competing for a spot in the middle of your cluster, is immense and widely underestimated.
Your applications need to be aware and adapted to the k8s model of automatic pod termination, no hard storage, etc. Google has had over 10 years with the best computer scientists money can buy to perfect their stack. Most companies don't.
For most companies, k8s/linkerd/Docker deployments are a case of overengineering run frighteningly amok.
There are many-a-snafu lying down this path. I'm very worried about how casually everyone is willing to jump aboard.
One particular worry: most Dockerfiles are built on opaque images registered with an image registry, and many of the publishers are not well-known. If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance). These are frequently distributed by complete strangers. Building "from scratch" off the base image distributed by the distro itself is considered "the old way", at least among those I've worked with.
My understanding is that most public image repositories are unmoderated. The moment a widely-used image gets pwned, there is going to be panic.
Source: been converting entire company to Docker/k8s
> Building "from scratch" off the base image distributed by the distro itself is considered "the old way"
Then I don't want to be new. I started using a few of the Docker Hub images (mostly `openjdk`), but I've transitioned completely to building all runtime images off of `debian:jessie`. And even that image I may want to remove eventually, just so that our CI system isn't affected by Docker Hub outages.
Yes, it's insane. I can only attribute it to that I don't work on problem of any scale (amount of apps, servers, deployments per day, etc.) and all of these people do. Who really wants to spend their time and effort running a Kubernetes master and nodes, 3+ Consul/etcd nodes, and now some fancy load balancers? The ratio of companies that would benefit from that to the number of companies working on products in the space is way out of wack.
"Have we made an unnecessarily contorted solution that you don't need to solve any of your real-world problems? Don't worry, you can still claim to be cool by paying us money." What a solution!
I would not take the claims of scale at face value. People make this claim to boost their own egos, and the reality is that many of the people who do this would be more than fine on a handful of servers. They just want to feel important.
My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here and there, there was demand for it because there's demand for anything and everything with Google's name attached to it, regardless of applicability or propriety or any good sense (case in point: companies pretending to be smart by "integrating" TensorFlow), and they released k8s in response to the clamor for the mysterious Google-backed orchestration platform, much to the detriment of non-Googlers everywhere.
Maybe it's a coordinated plan to harm young startups that may challenge them. Or, alternate theory: perhaps it's a coordinated plan to get people to buy far more cloud instances than they actually need, since k8s literally cannot run on a single instance; to run it locally, you have to run virtual machines.
I understand that k8s has improved and that many people use it (myself among them), but it maintains that vibe of an internal tool; something so convoluted and complex and inside baseball that you'd only voluntarily undertake it if someone said your paycheck was at risk.
Just FYI, over 50% of contributions come from !Google, and that's part of the reason that so many people (CoreOS, Redhat, Apprenda, Deis, thousands of internal folks) have rebased on it.
I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.
I have more experience with Kubernetes now than I did then, though I think a lot of those interface snafus, which is most of what the old post discussed, are still valid complaints.
You mentioned back then that if people aren't aware of some kube featureset, they end up reimplementing it. You mentioned logging. Can you tell me what you recommend for logging?
Right now, we have everything writing logs to stdout in the container, which gets recorded by kube in something like /var/log/kube/containers/ * . Then we have a fluentd container that reads the logs and uploads them to an external machine, which is running fluentd containers that receive the input stream, transform it according to rules, and pipe it out to a cloud-based log aggregator.
Is that how it's supposed to work? I know we used to do it somewhat differently, but had trouble with resource consumption that was disrupting our pods.
And here's a potentially legitimate Kubernetes issue: the scheduler is only reading the load generated from containers administered by Kubernetes, so even though the box is being thrashed, k8s will not be aware of the problem and continues to report the node's resources as healthy. In our case, the dockerd process was using 4.5 cores. Kubernetes seemed unaware and it was affecting our performance because another busy pod was on the server and was not getting clamped by its quota (since it, technically, was not hitting the quota for itself). Shouldn't these be defined relative to overall system resources, and not just the containers that kube can see?
It's possible Kubernetes was aware, but that we didn't know how to see it. A colleague told me the interface I was looking at for pod performance data reflects only the quotas, not the actual utilization, so maybe k8s knew (why isn't there a `kubetop` that can be run on the host to supervise?). I also hear that automatic rescheduling is still in draft stage, so if k8s detects high load on a kubelet, instead of moving the pod, it will just not assign new pods to it.
About addressing. Right now, we have dev, stage, and prod. We have a container that takes all traffic targeting the kubelet on port 80 and proxies it out to the pod using servicelb. It's set up to forward from http://app-label.env-name.example.com to a pod matching that label. It's annoying that we have to run an external container for this in the first place and do extra configuration on it, but it's also annoying that since servicelb is not aware of the beta label attempting to simulate hostnames, we can't individually address instances of apps. Is that how this is supposed to work, or is there a ready-built solution that makes this much easier that we're overlooking?
----
>I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
The best way to improve Kubernetes would be to:
a) simplify the terminology;
b) simplify the kubectl command, and particularly make it follow semi-standard Unix utility conventions (examples: -f should mean force or not be used (and yes, I know some other tools commit this sin too, but the association with "force" is nerve-wracking on a production environment), should probably be -r or -o; --from-file is ambiguous, and should be --read-literal or -r --raw or something like that);
c) simplify the configuration syntax and files;
d) provide simple ways to get real storage and real IPs (I haven't thoroughly investigate...
As a Kubernetes user, I don't understand the "vibe of an internal tool" comment. It feels nothing like that to me.
Kubernetes is big and complex, to be sure, but to me its state of being unfinished is not because it's "internal" anything, but because — since Google couldn't just open-source Borg and Omega, which presumably have a decade of Google-specific cruft that wouldn't translate to a reusable tool — it had to be built from scratch, and is only slowly accruing the various features needed to approximate the maturity of a full-blown orchestration system of the kind that Google uses already.
In other words, Kubernetes works great and all, but it's also immature in many ways: Ingress is painful, secrets management not secure enough; we're only now getting stateful sets, affinities, federation, role-based security, etc.; we're still waiting for things like scheduled jobs and custom metrics support in the autoscaler; Kubelet is still an awkward SPOF; etc. But those shortcomings have nothing to do with it being an internal tool, but rather because there's just a lot of work left to be done.
> If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance)
I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1].
> My understanding is that most public image repositories are unmoderated.
To the best of my understanding, the official images are maintained by Docker folks, or people associated with the upstream projects, or some mixture of both.
> most Dockerfiles are built on opaque images registered with an image registry
Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
So yes, while it's true there are a lot of public images out there with questionable provenance, there's usually a solid, well-supported foundation to base your own images on.
>I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images.
Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
While it is nice that there are a few known-good repositories for projects popular enough to secure the approval of Docker Inc., the habit of casually importing images is still rampant, afaict.
>Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
Well, here's an example [0] of a base Dockerfile. I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something. After chasing the tree up, this is
a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
and b) a good example of the "brilliance" of the Dockerfile format. That massive set of line continuations is necessary because if you implement the Dockerfile "naively", by making each modification a "RUN" command, you end up with an image that is very bloated due to the way Docker's caching layer functions. Bonus feature: if any of those commands is to return a non-0 exit code, the line will abort.
Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
I'm not sure. You can find more information about them here[0].
> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
I'm not sure about this either. It looks like Rancher and OpenShift support whitelisting registries, so this may have to be handled at a higher level. I believe you can always tell whether something being pulled from Docker Hub is official because it won't have a leading path component indicating the contributor (e.g. mwpmaybe/ruby:2.4-alpine is a non-official image whereas ruby:2.4-alpine is official).
> the habit of casually importing images is still rampant
This is just my two cents, but that seems like a policy problem, not a technology problem. Enforce a rule for your project/team/organization that containers have to use an official image, or a private image based on an official image. I do agree that a simple per-registry namespace whitelist would be a nice feature to have in dockerd.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
I believe official images must be based on other official images.
> b) a good example of the "brilliance" of the Dockerfile format
I don't think it was ever meant to be brilliant, just simple. I'm definitely looking forward to the next generation of Dockerfile formats that ameliorate this issue and many others.
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
I believe so. This is a complaint I have with Docker Hub. I personally feel that the Dockerfiles used to create public images should always be published along with the images.
>This is just my two cents, but that seems like a policy problem, not a technology problem.
Software that makes dangerous things too easy is a technology problem. This is why Linux distros require signed packages by default.
Furthermore, one of Docker's popular appeals is "never get stuck in dependency hell again: just install someone's container!" You see more and more projects saying "The best way to install this is just to use our Docker container." The culture is definitely not leading to a security-minded perspective.
Compare with something like Ansible Playbooks, where the commands executed are always visible.
>I don't think it was ever meant to be brilliant, just simple.
Requiring long continuations for reasonable image sizes is not simple. Without the continuations, final image sizes regularly bloat to be larger than a typical distro ISO. On the other hand, exchanging the cached layers for size nullifies some of Docker's claimed value, specifically wrt atomicity and build time.
> Requiring long continuations for reasonable image sizes is not simple.
We're just arguing semantics at this point. I mean "simple" like Assembler is simple: rudimentary instructions requiring a certain amount of understanding and sophistication to wield correctly. Not "simple" as in easy, intuitive, and footgun-free.
> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
I believe most of your questions have a (mostly) positive answer. Admittedly the Docker project moves fast, so I've tried to provide a link or two for each:
> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
The official images are all signed with Notary[0] (released <2 years ago). The "Content Trust" feature makes use of TLS plus a number of keys[1]. It can be enabled in required environments to prevent the pulling of non-signed images (`export DOCKER_CONTENT_TRUST=1`).
> I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something.
There is a helpful Chrome extension, OctoLinker[2], that makes the `FROM parent` clickable (among other non-Docker things). I'm sure there are similar extensions, but this is the one I currently use.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
The `FROM` line in the Dockerfile would likely be the first thing to undergo scrutiny, for obvious reasons. Also, all the layers go through security scanning[3], some issues can be fixed[4] and others are tracked upstream (e.g. by Debian in the case of the Ruby image you linked[5]).
> you end up with an image that is very bloated due to the way Docker's caching layer functions
You can now use `docker build --squash`[6] to combine multiple image layers while still benefiting from layer caching during builds. Also, the final image won't contain files that are added in previous layers and removed in later ones. (And with multi-stage builds recently merged, soon multiple inheritance will be easier.)
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it
I would say groups or sole developers that want others to use their images will usually provide documentation and links to the Dockerfiles. Of course, this isn't always possible or desired (e.g. see closed-source projects like Windows Server Core[7], or the countless people using Docker Hub as a free image host.)
Thanks for this post. It does indeed answer most of my follow-up questions.
I wanted to let you know that I gave `docker build --squash` a try today. This is the output I got:
"--squash" is only supported on a Docker daemon with experimental features enabled
so that feature is not yet mainlined.
Also, I encountered another issue today when trying to do a `docker push`. It retried a cache layer multiple times before conceding with `open /dev/mapper/docker-254:0-xxx-yyy-zzz: no such file or directory` on one of the layers. I had to rebuild the image with `--no-cache` to get it to push.
Not doing anything unusual/fancy, just an ordinary docker push, which worked fine on other images before and after. No auto-cleanup scripts running in the background or anything that should cause a layer to go mysteriously missing.
It's perfectly normal to use base images and build your own. We do this for most things.
One simple reason is control: You want to be able to follow CVEs and upgrade your images in line with patches. If every image branches off some random Docker Hub image, the only way to do that would be some kind of image vulnerability scanner, which I believe exists, but wouldn't help you actually patch anything easily.
Public images are still super nice to bootstrap things. You can start with a public image, then move to a custom-built one.
Lastly: I wouldn't conflate the Docker image challenge with the operational complexity of using Kubernetes and microservices. They are completely orthogonal in my mind. Docker has all sorts of issues that we need to solve sooner rather than later; Kubernetes' design, on the other hand, is sound.
I recognize that Docker and k8s are distinct, and I think rkt looks much smarter than Docker. Now that k8s has released a stable version capable of running non-Docker containers (as of last week), it's fair to separate them.
However, I'm still struggling to see the real value proposition offered by either.
The story on Docker or Docker-style containers without orchestration is that they're supposed to be easy to ensure you get an identical environment everywhere and more fully utilize your hardware resources. While that does seem to happen if you incorporate docker-compose or some other metadata about what ports to map and volumes to mount, the pains associated with using Docker have far, far outweighed the one-time cost of configuring a new application or getting a new build to work for me. And we already had pretty good tooling for this pre-container, with things like Ansible. It wasn't exact, but it was much more comfortable.
Being a Puppet user (until we've finally retired that nightmare), I couldn't be happier with this migration to containers.
Docker itself is problematic for all sorts of reasons, but it at least does the most important thing, which is to run pre-packaged code in isolated containers. With Puppet, as with Ansible, you just can't guarantee an isolated environment, nor is the environment easily "replicable", in the sense that if you need more instances of the exact same thing (e.g. more data processing workers), there's nothing in the system built to treat things as N-of-anything. The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems.
Meanwhile, Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes, where in Docker Engine they're just low-level primitives you need to manage manually. It also handles exec'ing into an existing container. The pod as the unit of orchestration is also much nicer than Docker's single-process containers.
For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally). Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing, I rest assured that what is running is what I deployed.
Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc. We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources.
Docker, to me, isn't particularly interesting without Kubernetes, although it has its uses. It's terrific to be able to "boot" a temporary Ubuntu or Elasticsearch or whatever in order to do something that I can just tear down afterwards. Build/test containers are great, both locally and in a CI environment.
>The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems.
If you're operating the same script from the same image (say, a distro's official ISO), they should work effectively the same way. It is true that bootstrapping nodes may break, but unlikely. It shouldn't be common.
There's no reason something like Ansible couldn't be extended to build containers. Playbooks effectively perform the same function as Dockerfiles: they list commands to execute off of a base image. Playbooks are somewhat more flexible in that they're frequently used to address already-running machines.
For the record, push-centric configuration management like Chef and Puppet is much worse than Salt or Ansible, which can connect to arbitrary nodes and execute commands.
>Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes
Maybe we adopted too early but none of these things are "neatly" captured by Kubernetes. We have complex superstructures over all of them. They may be neater than running them in Docker alone, but they're much more complex than running against bare nodes.
Particularly, addressing application instances and getting stateful storage have been challenges in Kubernetes, just as they are in Docker. StatefulSets went stable in December (3 months ago) and I haven't tried them yet, so maybe those concerns are assuaged.
The ideal interface for k8s would be, essentially, an aware hypervisor. If each pod acted as a subnet and each container acted like a machine in that subnet, meaning it had an IP that was accessible according to external firewall and DNS rules and otherwise acted like a normal machine, that'd be much easier to accept. Neither Docker nor Kubernetes are amenable to this.
>It also handles exec'ing into an existing container.
The interface for this is practically the same as it is on Docker. You do `kubectl exec -it <pod> /bin/ash` instead of `docker exec -it <container_id> /bin/ash`. The only extra thing it handles is making the container appear local to the master, instead of requiring one to get on the correct kubelet.
>For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally).
The value in this being running more applications on a single cloud instance? If you have the instance/VM as the unit of orchestration, it works the same way: a coordinator (load balancer) divides the traffic among the units, and then stops transparently when you deregister or when a health check fails.
>Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing
"Orchestration, scheduling, autoscaling etc." refer, more or less, to the same thing: getting your applications running somewhere. Containers and k8s in no way have a monopoly on that functionality, and my experience has not been that they've significantly simplified it.
>I rest assured that what is running is what I deployed.
If running a playbook over a fresh node isn't good enough, you can use an image system to ensure it's a binary copy of what you built. There were several competing APIs for managing this in a platform-agnostic manner.
>Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc.
I know that k8s includes some logic to attempt to automate the failover/deployment process, and that...
I know I was just downvoted for answering your questions before, but I thought I'd reply here just to share what I've hit so far during today's work day using Docker. Just today.
1. `docker push` fails because of a mysteriously-vanished cache layer, forcing `docker build --no-cache` to work.
2. `docker-compose stop [service]` broke in a weird way, failing to detect the service's stoppage, creating a duplicate service of the same name that I couldn't do anything with. Had to clear out the duplicate container with `docker rm -fv ...` and now trying to figure out how to make `docker-compose` happy again.
These are totally normal, conventional operations that should not be choking for unexplained reasons.
And people suggest running production workloads on this, for some reason.
and bonus:
3. Tried the `docker build --squash` feature suggested by another commenter, to find it requires "experimental support" to be enabled in the Docker daemon. I have enough problems with docker without that, so I'm going to pass.
> Facilities that linkerd provides to assist with this include service discovery, load balancing, encryption, tracing and logging, handling retries, expiration and timeouts, back-offs, dynamic routing, and metrics.
Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly.
Also, regarding https://linkerd.io/, what's with landing pages having almost no useful information on them. "linker∙d is a transparent proxy that adds service discovery, routing, failure handling, and visibility to modern software applications" what's that even mean?
> Fast, lightweight, and performant
> Handles tens of thousands of requests per second per instance with minimal latency overhead. Scales horizontally with ease.
Well, since you're calling yourself "transparent" I would hope it'd be fast!
> Any language, any environment
> Runs as a transparent proxy alongside existing applications, integrates with existing infrastructure.
So, it's just proxying and retrying things and doing tls for me? Why isn't this better handled in the library I'm using to make the call in the first place?
> Latency-aware load-balancing
> Balances request traffic using real-time performance, reducing tail latencies across your application.
Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
> Runtime traffic routing
> Provides dynamic, scoped, logical routing rules, enabling blue-green deployments, staging, canarying, failover, and more.
OK, some of that does sound useful, but some of it still feels like something the caller should handle.
> Drop-in service discovery
> Integrates with most service discovery systems, decoupling applications from specific implementations.
What's this mean? Why not just use SRV records?
> Production-tested and proven at scale
> Powers the production infrastructure of banks, artificial intelligence companies, social networks, government labs, and more.
I know why this is here, but it feels like _everyone_ claims the same thing, so, to me, it just becomes noise.
> "linkerd allows you to drop in a transparent layer of application resilience and provides the operational affordances critical for modern, cloud native environments."
> —Oliver Gould, CTO, Buoyant
This sounds doesn't make sense to me, you can't "drop in" resiliency. It needs to be built. The whole thing just feels like it's all marketing and doesn't tell me exactly what problems it's solving for me as a developer.
------------
Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
Disclaimer: I'm still researching linkerd and haven't tried to use it.
One component we're interested in is its "automatic" tracing. It claims to be able to annotate requests with Zipkin stats without requiring any instrumentation at the software level.
You are correct that other solutions exist for service discovery, but that's an in-demand feature right now as well, and afaik the main reason that people use linkerd.
>Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
Quick, turn back now from this whole containerization thing, because I assure you this is basically the answer for everything in this space. And you will be aggressively accused of being an old fogie for resisting it. Simplicity is just not appreciated anymore; at this point, "containers" may as well be an antonym for it (and by extension, an antonym for "sane engineering").
I'm personally a big BSD jails fan. The idea of containerization is nice, but people often forget that it requires tooling and care, just like anything. And isn't a magic bullet
> > Latency-aware load-balancing > Balances request traffic using real-time performance, reducing tail latencies across your application.
> Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.
> Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
So? If you're happy with your solution, keep it! But maybe others haven't solved all these problems, or don't like your approach to how you've solved them for other reasons or don't even want to spend time solving these problems in the first place. For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
> Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.
Wouldn't those stats get out of date really quickly if you're not constantly talking to that machine?
> For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
But now you have a new component that you need to test and get to know. You didn't get rid of those 10 things, you just compartmentalized them in something else that needs to be maintained.
> Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
Very true, I just don't see the particular advantages to using more basic and well-known tools like DNS and TLS.
No programming language's HTTP/RPC stack supports SRV records. They are just not a thing. Changing your microservices to know about SRV is not a great solution either. Using something like linkerd frees your application from concerns like how discovery and transport is done. SRV is also not compatible with most components out there. e.g. you can't tell memcached/redis to resolve SRV records. You should take a look at how Kubernetes also does not do SRV records, instead it creates an IP address per service for intra-cluster communication. A/AAAA names work with legacy applications and are backwards compatible. That's pretty much why open source community has not been rolling out a major solution based on SRV.
Multiple A records are also fine. Also more supported as you pointed out.
Unless you're constantly talking to a service, your stats will get old quickly. It could also cause everyone to target a specific host creating problems.
I'm still not sure what service discovery is required beyond knowing the name of a service.
Linkerd doesn't free me from anything. I still need to know how it's done, how it's configured, and how to fix it when it breaks.
I don't think SRV records are the right answer; your networking layer should be k8s aware and issue a watch command on Endpoints; it'll be updated immediately-ish when the servers changes. This is similar to how finagle's zookeeper server set is supposed to work.
What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production. But I think it's straddling a very narrow section; small users shouldn't care about this and just rely on the round robin, zipkin is a PITA to run anyway. Large users will probably want to write their own libraries (zipkin is a lot better if you put traces in your process)
> your networking layer should be k8s aware and issue a watch command on Endpoints;
Or, I could use standard networking concepts and not build my entire app to suit one new thing that I may not want to stay with forever, or even more than just test.
> What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production
There are already TCP/IP, DNS, DHCP, &c libraries for all languages, and that aren't tied to a specific stack. Why should I build into this stack and not into a more common subset?
DNS for service discovery is fraught with perils. Many implementations don't respect TTLs, or don't actually use multiple records, or don't do something smart like power of two.
Even if you pick all your impls carefully, you have to wait for your TTL instead of a push mechanism for changes.
If you wanted to implement the watch as a library, code wise, it's maybe ~500 lines per language, using the k8s client lib. I could hammer it out in two days.
The whole point of Linkerd is moving logic out of your app. Linkerd is like a library but, instead of being in-process, it inserts itself between the app and whatever the app needs to talk to. The app just dials http://foo.svc/, and Linkerd sends the connection to the right place; it doesn't need to know where foo.svc is.
Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices. Linkerd replaces this with something that is explicitly designed to deal with inter-app communication. (I believe one way to run Linkerd is as a small sidecar app inside the same Docker container the man app, as opposed to a big central server, which is also a neat way of doing it.)
Re "why not a library" — sure, you can build all sorts of discovery, distribution and resilience into the client itself. But that consolidates all of this intelligence into just that client. What if you use multiple languages? What about third-party client libs (e.g. Memcached, Redis, PostgreSQL) that you don't control and that you don't want to reimplement? You'll find yourself reimplementing the same thing over and over.
> you can't "drop in" resiliency
I recommend reading the documentation, because drop-in resilience is really the whole point. The guys who built Linkerd previously developed Finagle [1], which is Twitter's RPC library, and when they talk about network resilience, they actually know what they're talking about [2].
> The whole point of Linkerd is moving logic out of your app. Linkerd is like a library but, instead of being in-process, it inserts itself between the app and whatever the app needs to talk to. The app just dials http://foo.svc/, and Linkerd sends the connection to the right place; it doesn't need to know where foo.svc is.
That's just DNS.
> Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices.
Why not DNS round-robin?
> I recommend reading the documentation, because drop-in resilience is really the whole point.
Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
That's not just DNS. Well, it might be - it depends on your environment.
For example: In a world where you have some kind of containerization, or scheduler responsible for deploying your services you typically to have services (or different instances of your services), running on ports determined at runtime. You might solve this with an off-host load balancer, so your frontend applications don't need to worry about it, but now you need to worry about those other things: circuit breaking, retry mechanisms, etc between your application and its middle-tear load balancer.
A common deployment strategy for something like linkerd (or Envoy) would be as a sidecar process, on the same host where you don't need to worry about the reliability of the network to talk to a remote service, because something else is taking care of 90% of your concerns for you.
> Why not DNS round-robin?
There's functionality offered by load balancing at L7 that you can't do with round-robin DNS. You end up having to push that complexity into each service you develop - retry mechanisms, circuit breaking, and routing decisions if you're going to try do green/blue or canary deployments.
You might say "solve this with a library", but the reality is you have to reinvent the wheel, you have to keep that library in sync to ensure the behaviour is consistent, and if you're in an organization that develops services in "N" languages, that challenge can be even greater.
> Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
Absolutely. Your services still need to be able to fail gracefully, but this just moves the complexity of a whole host of other issues to a single, and common spot.
>That's not just DNS. Well, it might be - it depends on your environment.
In short, another problem introduced by containers and not relevant to non-containers. Why can't a container get its own IP and port like everything else? FreeBSD jails do. Why make it opaque and put the "ingress" component inside of k8s or Docker? It'd be reasonable for k8s to act as a DHCP/DNS server, but not to swallow the whole world.
No, it's a bit more complicated than that, as chrisboulton explains (so I won't go into detail).
You could use a dynamic DNS server such as Consul for discovery, and people do. Last I checked, you still had to fight eagerly caching clients, but that's probably doable. But there's still a host of other issues that Linkerd addresses.
Again, I strongly recommend reading about Finagle to understand what this is trying to solve.
Another thing Linkerd brings to the mix is instrumentation and tracing. Instead of building this into every single sender and receiver, Linkerd can collect tracing data for you and make it available for other subsystems to ingest, such as Zipkin or Prometheus.
Round-robin may be sufficient for your needs. However, the load balancing techniques that linkerd employs solved real production issues for us. These techniques allowed us to be paged less frequently for minor errors and we could more reliably make changes to the system without causing user-facing issues.
Lyft has a tech similar to linkerd called Envoy: https://github.com/lyft/envoy. It is a bit lower level than linkerd (based on APIs rather than already integrated with common services) and written in C++.
I'm working on an adapter which should offer a better integration with Kubernetes. The end goal is to have envoy be able to replace our Ingress controllers. Not having to run the JVM on every machine is also nice.
> Not having to run the JVM on every machine is also nice.
Personally, Envoy compiling to a binary makes it much more likely that I would experiment with it than with linkerd. I have hesitated trying out linkerd because we don't run anything JVM based in any of our stack, and we aren't currently excited about introducing any.
I am definitely going to take a look at Envoy. Thanks for referencing it!
48 comments
[ 3.4 ms ] story [ 123 ms ] thread[1] https://github.com/linkerd/linkerd [2] https://github.com/linkerd/linkerd-tcp
Though your links are correct your statement isn't. Linkerd is written in Scala, but the Rust bit is linkerd-tcp, a TCP loadbalancer, which linkerd (the Scala one) doesn't handle as that covers HTTP and gRPC with optionally encrypting connections between linkers using TLS.
> If he is optimizing for other people’s contributions, is this really the best way to go?
But is he optimising for other people's contributions? Using a C/Rust for something like a TCP proxy makes a lot more sense to me than having a JVM do that. And if he perceives Scala as "not for everyone", maybe adding Rust means he might get more contributions to those components?
I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two. By spinning it out as a standalone component that acts as a proxy, like linkerd is doing, Scala doesn't get in the way and what Finagle offers can be used with a mesh of microservices written in a myriad of languages.
> I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two.
Thanks, this is the answer I have been looking for.
The complexity introduced not only by Docker, but also k8s and all of the other things competing for a spot in the middle of your cluster, is immense and widely underestimated.
Your applications need to be aware and adapted to the k8s model of automatic pod termination, no hard storage, etc. Google has had over 10 years with the best computer scientists money can buy to perfect their stack. Most companies don't.
For most companies, k8s/linkerd/Docker deployments are a case of overengineering run frighteningly amok.
There are many-a-snafu lying down this path. I'm very worried about how casually everyone is willing to jump aboard.
One particular worry: most Dockerfiles are built on opaque images registered with an image registry, and many of the publishers are not well-known. If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance). These are frequently distributed by complete strangers. Building "from scratch" off the base image distributed by the distro itself is considered "the old way", at least among those I've worked with.
My understanding is that most public image repositories are unmoderated. The moment a widely-used image gets pwned, there is going to be panic.
Source: been converting entire company to Docker/k8s
Then I don't want to be new. I started using a few of the Docker Hub images (mostly `openjdk`), but I've transitioned completely to building all runtime images off of `debian:jessie`. And even that image I may want to remove eventually, just so that our CI system isn't affected by Docker Hub outages.
My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here and there, there was demand for it because there's demand for anything and everything with Google's name attached to it, regardless of applicability or propriety or any good sense (case in point: companies pretending to be smart by "integrating" TensorFlow), and they released k8s in response to the clamor for the mysterious Google-backed orchestration platform, much to the detriment of non-Googlers everywhere.
Maybe it's a coordinated plan to harm young startups that may challenge them. Or, alternate theory: perhaps it's a coordinated plan to get people to buy far more cloud instances than they actually need, since k8s literally cannot run on a single instance; to run it locally, you have to run virtual machines.
I understand that k8s has improved and that many people use it (myself among them), but it maintains that vibe of an internal tool; something so convoluted and complex and inside baseball that you'd only voluntarily undertake it if someone said your paycheck was at risk.
I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.
Disclosure: I work at Google on Kubernetes
Not to toot the same horn.
I have more experience with Kubernetes now than I did then, though I think a lot of those interface snafus, which is most of what the old post discussed, are still valid complaints.
You mentioned back then that if people aren't aware of some kube featureset, they end up reimplementing it. You mentioned logging. Can you tell me what you recommend for logging?
Right now, we have everything writing logs to stdout in the container, which gets recorded by kube in something like /var/log/kube/containers/ * . Then we have a fluentd container that reads the logs and uploads them to an external machine, which is running fluentd containers that receive the input stream, transform it according to rules, and pipe it out to a cloud-based log aggregator.
Is that how it's supposed to work? I know we used to do it somewhat differently, but had trouble with resource consumption that was disrupting our pods.
And here's a potentially legitimate Kubernetes issue: the scheduler is only reading the load generated from containers administered by Kubernetes, so even though the box is being thrashed, k8s will not be aware of the problem and continues to report the node's resources as healthy. In our case, the dockerd process was using 4.5 cores. Kubernetes seemed unaware and it was affecting our performance because another busy pod was on the server and was not getting clamped by its quota (since it, technically, was not hitting the quota for itself). Shouldn't these be defined relative to overall system resources, and not just the containers that kube can see?
It's possible Kubernetes was aware, but that we didn't know how to see it. A colleague told me the interface I was looking at for pod performance data reflects only the quotas, not the actual utilization, so maybe k8s knew (why isn't there a `kubetop` that can be run on the host to supervise?). I also hear that automatic rescheduling is still in draft stage, so if k8s detects high load on a kubelet, instead of moving the pod, it will just not assign new pods to it.
About addressing. Right now, we have dev, stage, and prod. We have a container that takes all traffic targeting the kubelet on port 80 and proxies it out to the pod using servicelb. It's set up to forward from http://app-label.env-name.example.com to a pod matching that label. It's annoying that we have to run an external container for this in the first place and do extra configuration on it, but it's also annoying that since servicelb is not aware of the beta label attempting to simulate hostnames, we can't individually address instances of apps. Is that how this is supposed to work, or is there a ready-built solution that makes this much easier that we're overlooking?
----
>I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
The best way to improve Kubernetes would be to:
a) simplify the terminology;
b) simplify the kubectl command, and particularly make it follow semi-standard Unix utility conventions (examples: -f should mean force or not be used (and yes, I know some other tools commit this sin too, but the association with "force" is nerve-wracking on a production environment), should probably be -r or -o; --from-file is ambiguous, and should be --read-literal or -r --raw or something like that);
c) simplify the configuration syntax and files;
d) provide simple ways to get real storage and real IPs (I haven't thoroughly investigate...
Kubernetes is big and complex, to be sure, but to me its state of being unfinished is not because it's "internal" anything, but because — since Google couldn't just open-source Borg and Omega, which presumably have a decade of Google-specific cruft that wouldn't translate to a reusable tool — it had to be built from scratch, and is only slowly accruing the various features needed to approximate the maturity of a full-blown orchestration system of the kind that Google uses already.
In other words, Kubernetes works great and all, but it's also immature in many ways: Ingress is painful, secrets management not secure enough; we're only now getting stateful sets, affinities, federation, role-based security, etc.; we're still waiting for things like scheduled jobs and custom metrics support in the autoscaler; Kubelet is still an awkward SPOF; etc. But those shortcomings have nothing to do with it being an internal tool, but rather because there's just a lot of work left to be done.
I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1].
> My understanding is that most public image repositories are unmoderated.
To the best of my understanding, the official images are maintained by Docker folks, or people associated with the upstream projects, or some mixture of both.
> most Dockerfiles are built on opaque images registered with an image registry
Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
So yes, while it's true there are a lot of public images out there with questionable provenance, there's usually a solid, well-supported foundation to base your own images on.
0. https://hub.docker.com/_/ruby/
1. https://github.com/docker-library/official-images/tree/maste...
Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
While it is nice that there are a few known-good repositories for projects popular enough to secure the approval of Docker Inc., the habit of casually importing images is still rampant, afaict.
>Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
Well, here's an example [0] of a base Dockerfile. I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something. After chasing the tree up, this is
a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
and b) a good example of the "brilliance" of the Dockerfile format. That massive set of line continuations is necessary because if you implement the Dockerfile "naively", by making each modification a "RUN" command, you end up with an image that is very bloated due to the way Docker's caching layer functions. Bonus feature: if any of those commands is to return a non-0 exit code, the line will abort.
Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
[0] https://github.com/docker-library/ruby/blob/752c5f7cf44870ce...
I'm not sure. You can find more information about them here[0].
> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
I'm not sure about this either. It looks like Rancher and OpenShift support whitelisting registries, so this may have to be handled at a higher level. I believe you can always tell whether something being pulled from Docker Hub is official because it won't have a leading path component indicating the contributor (e.g. mwpmaybe/ruby:2.4-alpine is a non-official image whereas ruby:2.4-alpine is official).
> the habit of casually importing images is still rampant
This is just my two cents, but that seems like a policy problem, not a technology problem. Enforce a rule for your project/team/organization that containers have to use an official image, or a private image based on an official image. I do agree that a simple per-registry namespace whitelist would be a nice feature to have in dockerd.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
I believe official images must be based on other official images.
> b) a good example of the "brilliance" of the Dockerfile format
I don't think it was ever meant to be brilliant, just simple. I'm definitely looking forward to the next generation of Dockerfile formats that ameliorate this issue and many others.
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
I believe so. This is a complaint I have with Docker Hub. I personally feel that the Dockerfiles used to create public images should always be published along with the images.
0. https://docs.docker.com/docker-hub/official_repos/
Software that makes dangerous things too easy is a technology problem. This is why Linux distros require signed packages by default.
Furthermore, one of Docker's popular appeals is "never get stuck in dependency hell again: just install someone's container!" You see more and more projects saying "The best way to install this is just to use our Docker container." The culture is definitely not leading to a security-minded perspective.
Compare with something like Ansible Playbooks, where the commands executed are always visible.
>I don't think it was ever meant to be brilliant, just simple.
Requiring long continuations for reasonable image sizes is not simple. Without the continuations, final image sizes regularly bloat to be larger than a typical distro ISO. On the other hand, exchanging the cached layers for size nullifies some of Docker's claimed value, specifically wrt atomicity and build time.
We're just arguing semantics at this point. I mean "simple" like Assembler is simple: rudimentary instructions requiring a certain amount of understanding and sophistication to wield correctly. Not "simple" as in easy, intuitive, and footgun-free.
In Kubernetes land, you can use the ImagePolicyWebhook admission controller. https://kubernetes.io/docs/admin/admission-controllers/#imag...
I agree that Dockerfiles are awful.
> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
The official images are all signed with Notary[0] (released <2 years ago). The "Content Trust" feature makes use of TLS plus a number of keys[1]. It can be enabled in required environments to prevent the pulling of non-signed images (`export DOCKER_CONTENT_TRUST=1`).
> I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something.
There is a helpful Chrome extension, OctoLinker[2], that makes the `FROM parent` clickable (among other non-Docker things). I'm sure there are similar extensions, but this is the one I currently use.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
The `FROM` line in the Dockerfile would likely be the first thing to undergo scrutiny, for obvious reasons. Also, all the layers go through security scanning[3], some issues can be fixed[4] and others are tracked upstream (e.g. by Debian in the case of the Ruby image you linked[5]).
> you end up with an image that is very bloated due to the way Docker's caching layer functions
You can now use `docker build --squash`[6] to combine multiple image layers while still benefiting from layer caching during builds. Also, the final image won't contain files that are added in previous layers and removed in later ones. (And with multi-stage builds recently merged, soon multiple inheritance will be easier.)
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it
I would say groups or sole developers that want others to use their images will usually provide documentation and links to the Dockerfiles. Of course, this isn't always possible or desired (e.g. see closed-source projects like Windows Server Core[7], or the countless people using Docker Hub as a free image host.)
[0]: https://github.com/docker/notary
[1]: https://docs.docker.com/engine/security/trust/trust_key_mng/
[2]: https://octolinker.github.io/
[3]: https://docs.docker.com/docker-hub/official_repos/#how-do-i-...
[4]: https://github.com/docker-library/official-images/pulls?utf8...
[5]: https://github.com/docker-library/ruby/issues/117
[6]: https://docs.docker.com/engine/reference/commandline/build/#...
[7]: https://hub.docker.com/r/microsoft/windowsservercore/
I wanted to let you know that I gave `docker build --squash` a try today. This is the output I got:
so that feature is not yet mainlined.Also, I encountered another issue today when trying to do a `docker push`. It retried a cache layer multiple times before conceding with `open /dev/mapper/docker-254:0-xxx-yyy-zzz: no such file or directory` on one of the layers. I had to rebuild the image with `--no-cache` to get it to push.
Not doing anything unusual/fancy, just an ordinary docker push, which worked fine on other images before and after. No auto-cleanup scripts running in the background or anything that should cause a layer to go mysteriously missing.
One simple reason is control: You want to be able to follow CVEs and upgrade your images in line with patches. If every image branches off some random Docker Hub image, the only way to do that would be some kind of image vulnerability scanner, which I believe exists, but wouldn't help you actually patch anything easily.
Public images are still super nice to bootstrap things. You can start with a public image, then move to a custom-built one.
Lastly: I wouldn't conflate the Docker image challenge with the operational complexity of using Kubernetes and microservices. They are completely orthogonal in my mind. Docker has all sorts of issues that we need to solve sooner rather than later; Kubernetes' design, on the other hand, is sound.
However, I'm still struggling to see the real value proposition offered by either.
The story on Docker or Docker-style containers without orchestration is that they're supposed to be easy to ensure you get an identical environment everywhere and more fully utilize your hardware resources. While that does seem to happen if you incorporate docker-compose or some other metadata about what ports to map and volumes to mount, the pains associated with using Docker have far, far outweighed the one-time cost of configuring a new application or getting a new build to work for me. And we already had pretty good tooling for this pre-container, with things like Ansible. It wasn't exact, but it was much more comfortable.
Docker itself is problematic for all sorts of reasons, but it at least does the most important thing, which is to run pre-packaged code in isolated containers. With Puppet, as with Ansible, you just can't guarantee an isolated environment, nor is the environment easily "replicable", in the sense that if you need more instances of the exact same thing (e.g. more data processing workers), there's nothing in the system built to treat things as N-of-anything. The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems.
Meanwhile, Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes, where in Docker Engine they're just low-level primitives you need to manage manually. It also handles exec'ing into an existing container. The pod as the unit of orchestration is also much nicer than Docker's single-process containers.
For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally). Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing, I rest assured that what is running is what I deployed.
Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc. We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources.
Docker, to me, isn't particularly interesting without Kubernetes, although it has its uses. It's terrific to be able to "boot" a temporary Ubuntu or Elasticsearch or whatever in order to do something that I can just tear down afterwards. Build/test containers are great, both locally and in a CI environment.
What pains are you referring to, specifically?
If you're operating the same script from the same image (say, a distro's official ISO), they should work effectively the same way. It is true that bootstrapping nodes may break, but unlikely. It shouldn't be common.
There's no reason something like Ansible couldn't be extended to build containers. Playbooks effectively perform the same function as Dockerfiles: they list commands to execute off of a base image. Playbooks are somewhat more flexible in that they're frequently used to address already-running machines.
For the record, push-centric configuration management like Chef and Puppet is much worse than Salt or Ansible, which can connect to arbitrary nodes and execute commands.
>Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes
Maybe we adopted too early but none of these things are "neatly" captured by Kubernetes. We have complex superstructures over all of them. They may be neater than running them in Docker alone, but they're much more complex than running against bare nodes.
Particularly, addressing application instances and getting stateful storage have been challenges in Kubernetes, just as they are in Docker. StatefulSets went stable in December (3 months ago) and I haven't tried them yet, so maybe those concerns are assuaged.
The ideal interface for k8s would be, essentially, an aware hypervisor. If each pod acted as a subnet and each container acted like a machine in that subnet, meaning it had an IP that was accessible according to external firewall and DNS rules and otherwise acted like a normal machine, that'd be much easier to accept. Neither Docker nor Kubernetes are amenable to this.
>It also handles exec'ing into an existing container.
The interface for this is practically the same as it is on Docker. You do `kubectl exec -it <pod> /bin/ash` instead of `docker exec -it <container_id> /bin/ash`. The only extra thing it handles is making the container appear local to the master, instead of requiring one to get on the correct kubelet.
>For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally).
The value in this being running more applications on a single cloud instance? If you have the instance/VM as the unit of orchestration, it works the same way: a coordinator (load balancer) divides the traffic among the units, and then stops transparently when you deregister or when a health check fails.
>Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing
"Orchestration, scheduling, autoscaling etc." refer, more or less, to the same thing: getting your applications running somewhere. Containers and k8s in no way have a monopoly on that functionality, and my experience has not been that they've significantly simplified it.
>I rest assured that what is running is what I deployed.
If running a playbook over a fresh node isn't good enough, you can use an image system to ensure it's a binary copy of what you built. There were several competing APIs for managing this in a platform-agnostic manner.
>Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc.
I know that k8s includes some logic to attempt to automate the failover/deployment process, and that...
1. `docker push` fails because of a mysteriously-vanished cache layer, forcing `docker build --no-cache` to work.
2. `docker-compose stop [service]` broke in a weird way, failing to detect the service's stoppage, creating a duplicate service of the same name that I couldn't do anything with. Had to clear out the duplicate container with `docker rm -fv ...` and now trying to figure out how to make `docker-compose` happy again.
These are totally normal, conventional operations that should not be choking for unexplained reasons.
And people suggest running production workloads on this, for some reason.
and bonus:
3. Tried the `docker build --squash` feature suggested by another commenter, to find it requires "experimental support" to be enabled in the Docker daemon. I have enough problems with docker without that, so I'm going to pass.
Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly.
Also, regarding https://linkerd.io/, what's with landing pages having almost no useful information on them. "linker∙d is a transparent proxy that adds service discovery, routing, failure handling, and visibility to modern software applications" what's that even mean?
> Fast, lightweight, and performant > Handles tens of thousands of requests per second per instance with minimal latency overhead. Scales horizontally with ease.
Well, since you're calling yourself "transparent" I would hope it'd be fast!
> Any language, any environment > Runs as a transparent proxy alongside existing applications, integrates with existing infrastructure.
So, it's just proxying and retrying things and doing tls for me? Why isn't this better handled in the library I'm using to make the call in the first place?
> Latency-aware load-balancing > Balances request traffic using real-time performance, reducing tail latencies across your application.
Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
> Runtime traffic routing > Provides dynamic, scoped, logical routing rules, enabling blue-green deployments, staging, canarying, failover, and more.
OK, some of that does sound useful, but some of it still feels like something the caller should handle.
> Drop-in service discovery > Integrates with most service discovery systems, decoupling applications from specific implementations.
What's this mean? Why not just use SRV records?
> Production-tested and proven at scale > Powers the production infrastructure of banks, artificial intelligence companies, social networks, government labs, and more.
I know why this is here, but it feels like _everyone_ claims the same thing, so, to me, it just becomes noise.
> "linkerd allows you to drop in a transparent layer of application resilience and provides the operational affordances critical for modern, cloud native environments." > —Oliver Gould, CTO, Buoyant
This sounds doesn't make sense to me, you can't "drop in" resiliency. It needs to be built. The whole thing just feels like it's all marketing and doesn't tell me exactly what problems it's solving for me as a developer.
------------
Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
One component we're interested in is its "automatic" tracing. It claims to be able to annotate requests with Zipkin stats without requiring any instrumentation at the software level.
You are correct that other solutions exist for service discovery, but that's an in-demand feature right now as well, and afaik the main reason that people use linkerd.
>Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
Quick, turn back now from this whole containerization thing, because I assure you this is basically the answer for everything in this space. And you will be aggressively accused of being an old fogie for resisting it. Simplicity is just not appreciated anymore; at this point, "containers" may as well be an antonym for it (and by extension, an antonym for "sane engineering").
> Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.
> Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
So? If you're happy with your solution, keep it! But maybe others haven't solved all these problems, or don't like your approach to how you've solved them for other reasons or don't even want to spend time solving these problems in the first place. For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
Wouldn't those stats get out of date really quickly if you're not constantly talking to that machine?
> For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
But now you have a new component that you need to test and get to know. You didn't get rid of those 10 things, you just compartmentalized them in something else that needs to be maintained.
> Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
Very true, I just don't see the particular advantages to using more basic and well-known tools like DNS and TLS.
Unless you're constantly talking to a service, your stats will get old quickly. It could also cause everyone to target a specific host creating problems.
I'm still not sure what service discovery is required beyond knowing the name of a service.
Linkerd doesn't free me from anything. I still need to know how it's done, how it's configured, and how to fix it when it breaks.
What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production. But I think it's straddling a very narrow section; small users shouldn't care about this and just rely on the round robin, zipkin is a PITA to run anyway. Large users will probably want to write their own libraries (zipkin is a lot better if you put traces in your process)
OK, A/AAAA are more common.
> your networking layer should be k8s aware and issue a watch command on Endpoints;
Or, I could use standard networking concepts and not build my entire app to suit one new thing that I may not want to stay with forever, or even more than just test.
> What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production
There are already TCP/IP, DNS, DHCP, &c libraries for all languages, and that aren't tied to a specific stack. Why should I build into this stack and not into a more common subset?
Even if you pick all your impls carefully, you have to wait for your TTL instead of a push mechanism for changes.
If you wanted to implement the watch as a library, code wise, it's maybe ~500 lines per language, using the k8s client lib. I could hammer it out in two days.
Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices. Linkerd replaces this with something that is explicitly designed to deal with inter-app communication. (I believe one way to run Linkerd is as a small sidecar app inside the same Docker container the man app, as opposed to a big central server, which is also a neat way of doing it.)
Re "why not a library" — sure, you can build all sorts of discovery, distribution and resilience into the client itself. But that consolidates all of this intelligence into just that client. What if you use multiple languages? What about third-party client libs (e.g. Memcached, Redis, PostgreSQL) that you don't control and that you don't want to reimplement? You'll find yourself reimplementing the same thing over and over.
> you can't "drop in" resiliency
I recommend reading the documentation, because drop-in resilience is really the whole point. The guys who built Linkerd previously developed Finagle [1], which is Twitter's RPC library, and when they talk about network resilience, they actually know what they're talking about [2].
[1] https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rp...
[2] https://monkey.org/~marius/funsrv.pdf
That's just DNS.
> Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices.
Why not DNS round-robin?
> I recommend reading the documentation, because drop-in resilience is really the whole point.
Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
That's not just DNS. Well, it might be - it depends on your environment.
For example: In a world where you have some kind of containerization, or scheduler responsible for deploying your services you typically to have services (or different instances of your services), running on ports determined at runtime. You might solve this with an off-host load balancer, so your frontend applications don't need to worry about it, but now you need to worry about those other things: circuit breaking, retry mechanisms, etc between your application and its middle-tear load balancer.
A common deployment strategy for something like linkerd (or Envoy) would be as a sidecar process, on the same host where you don't need to worry about the reliability of the network to talk to a remote service, because something else is taking care of 90% of your concerns for you.
> Why not DNS round-robin?
There's functionality offered by load balancing at L7 that you can't do with round-robin DNS. You end up having to push that complexity into each service you develop - retry mechanisms, circuit breaking, and routing decisions if you're going to try do green/blue or canary deployments.
You might say "solve this with a library", but the reality is you have to reinvent the wheel, you have to keep that library in sync to ensure the behaviour is consistent, and if you're in an organization that develops services in "N" languages, that challenge can be even greater.
> Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
Absolutely. Your services still need to be able to fail gracefully, but this just moves the complexity of a whole host of other issues to a single, and common spot.
In short, another problem introduced by containers and not relevant to non-containers. Why can't a container get its own IP and port like everything else? FreeBSD jails do. Why make it opaque and put the "ingress" component inside of k8s or Docker? It'd be reasonable for k8s to act as a DHCP/DNS server, but not to swallow the whole world.
No, it's a bit more complicated than that, as chrisboulton explains (so I won't go into detail).
You could use a dynamic DNS server such as Consul for discovery, and people do. Last I checked, you still had to fight eagerly caching clients, but that's probably doable. But there's still a host of other issues that Linkerd addresses.
Again, I strongly recommend reading about Finagle to understand what this is trying to solve.
Another thing Linkerd brings to the mix is instrumentation and tracing. Instead of building this into every single sender and receiver, Linkerd can collect tracing data for you and make it available for other subsystems to ingest, such as Zipkin or Prometheus.
Round-robin may be sufficient for your needs. However, the load balancing techniques that linkerd employs solved real production issues for us. These techniques allowed us to be paged less frequently for minor errors and we could more reliably make changes to the system without causing user-facing issues.
See https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-b...
I'm working on an adapter which should offer a better integration with Kubernetes. The end goal is to have envoy be able to replace our Ingress controllers. Not having to run the JVM on every machine is also nice.
I am definitely going to take a look at Envoy. Thanks for referencing it!