I can't get over how much of a smell the requirement of an RPC interface in order to interface with tools that may or may not have any good reason to be running on an ongoing basis is to me.
I'm sure there may be cases where you interact with the containers frequently enough that spawning a process each time is actually a worthwhile optimization, but more and more of these containerisation systems are becoming an unholy mess of daemons that needs to run in order to run and manage containers that need not depend on anything but the host init/systemd.
E.g. one of the really appealing things of rkt for me is the simplicity - depending on the level of isolation everything is running either direclty under systemd, or under an individual isolator like systemd-nspawn.
I disliked this tendency towards a herd of daemons intensely when Docker continued as it started and used HTTP for volume/network plugins, and I dislike it just as much now.
It's as if someone sat down and thought long and hard about how to add more complexity and more "fun" failure modes.
It gets better. Take a look at rktlet, a CRI implementation for rkt (EDIT: I originally mistakenly wrote Docker). Specifically the runtime [1], which ends up shelling out to the "rkt" binary.
So you end up running a new daemon that communicates with Kubernetes via gRPC, that then spawns rkt anyway. So you get to RPC and fork/exec.
I'm sure that ends up "optimized away" at some point by e.g. having a rkt CRI implementation that just links in the relevant rkt code. But I'm left wondering why we need this complexity in the first place.
I don't get your argument. If I'm reading this correctly, you're arguing that system calls and/or a call to a shared library function are cleaner than RPC to another process?
The overhead of RPC in an application like this is tiny and the cost of an additional process on 2016 equipment is non-existent.
It's more things that can fail that now needs monitoring. (EDIT:) And in the specific case of rktlet you still end up forking/execing anyway.
The overhead isn't necessarily a big deal (and can easily go the other way - if the request frequency is high enough, it's cheaper to keep the process around), but it does also potentially add up.
The usual reasons to choose this pattern are "because versioning" and "location transparency".
When the resources you interact with are local OR you have tight over the environment, you're totally right, there's little reason to introduce "middle procs".
When there's a risk that different versions of things will manipulate resources in incompatible ways then the rpc layer makes sense.
What if the thing you're calling via an RPC abstraction is not a resident deamon but instead it gets loaded on demand via socket activation and then after performing the request it shuts off? Or a process gets forked and a RPC request/response get passed via stdio? Would that smell better? I'm honestly curious.
(I'm not saying that this would make sense for the CRI API, I didn't even looked at it)
Socket activation is a possible workaround, of course, but you then still have the gPRC dependencies. At least that removes the issue of keeping the service running.
In general for an API as simple as the CRI API I'd prefer forking and passing data via stdio or the environment any day.
In practice the GRPC interface can be "statically linked" into the kubelet binary and it then just becomes a plugin API that implementations can build. I believe this is how "dockershim" works today.
The advantage of the GRPC framework for developing though are tremendous, I'm building my own CRI implementation and I can iterate extremely quickly due to the fact that it can use RPC.
If the separate processes can be avoided, that's quite a bit better, though also entirely different from the descriptions on their pages.
What advantage do you get from gRPC over e.g. taking a JSON document on stdin or using environment variables? To me the issue isn't that things happens in a separate process, but that I really don't like solutions that ends up adding even more moving wheels without very good reason.
The way I see GRPC in the context of the CRI is a very well defined API. i.e. kubelet makes function (or in this case rpc) calls to my implementation of the api and I return results to it.
I tend to think of it like the VFS in the Linux kernel. If Linux was a microkernel, it would talk to each FS module over an RPC interface, since its not, they are standard function calls as statically linked together.
I'm unsure what your Q really is, how would I easily have function request/response layer with json documents or stdin? It seems more complicated. But as I don't quite understand perhaps I'm missing the point.
I also dont think its adding more moving wheels, its just taking one wheel out of kubelet (i.e. dockertools part of kubelet) and moving it elsewhere (i.e. dockershim as a cri implementation). The only increase in "moving wheels" is that even in the non "statically linked" scenario, one has to distribute 2 binaries to each node. kubelet and the cri implementation, but in practice I'm not sure thats really such a big deal.
Currently quite a few people from the OCI community (including myself) are working on implementing a CRI-compliant runtime[1] around runC and the various OCI specifications as well as the containers/image and containers/storage projects. There's a lot of cool design to ocid which means that it doesn't require a daemon to be constantly running.
I am curious how do you manage developing this while the OCI runtime is not yet finalized, do you just make sure they are compliant with the current RC of the spec?
Do you have any more specific pointers regarding using it without a daemon? As the examples seem to start with starting a daemon, unless I misunderstand something. If it doesn't need that, then that's a big plus in my book.
Though, I'm getting more and more disillusioned in general with where these specs are heading - the complexity seems to be skyrocketing for sometimes very little benefit.
Not necessarily specific to Kubernetes and/or OCI - Docker is a prime offender.
E.g. typical example: the highly coupled nature of many of the networking alternatives where routing, fabric and ip allocation gets muddled all up, when there are well developed, stable, well tested independent and orthogonal alternatives for tunnelling and route propagation there's a serious level of Not Invented Here syndrome at work in many of the container projects. I'm sure some people need all the complexity, but I'm getting more and more tempted to ditch many of the higher level tools in favour of composing smaller, simpler tools.
(Incidentally I'll make one prediction: one good thing likely to come from CRI is that I suspect it will lead to a new array of Kubernetes "replacements" from simpler composable tools; the APIs don't look all that bad - I just don't like the RPC dependency)
> Do you have any more specific pointers regarding using it without a daemon? As the examples seem to start with starting a daemon, unless I misunderstand something
At the moment, the RPC requirement means that you need to have a process that can accept RPC requests (a "daemon" if you like). However, unlike Docker (and containerd), ocid's lifetime is not tied to the lifetime of its containers -- which is one of the main downsides of Docker/containerd IMO. So in principle you could have ocid set up to only start up when kubelet is telling it to do anything. The real benefit of the design behind ocid is that in the future we could switch to a fork-exec model with the kubelet and it would still work.
For example, currently kubernetes is adding a requirement for runtimes to include a "kpod" binary that can do container and image operations even if the kubelet is down. My hope is that eventually they will just make the kubelet shell out to this binary so the CRI is defined through some sort of "here are the cli flags you need to accept" interface.
> the complexity seems to be skyrocketing for sometimes very little benefit.
I wouldn't call the current CRI "complicated", it's just that the gRPC requirement IMO is a bit too much. However, I would hope that since ocid and rkt both don't require daemons (well, rkt requires systemd but that's a given) that they'll reconsider their method of communicating with container runtime runners.
21 comments
[ 2.8 ms ] story [ 30.4 ms ] threadI'm sure there may be cases where you interact with the containers frequently enough that spawning a process each time is actually a worthwhile optimization, but more and more of these containerisation systems are becoming an unholy mess of daemons that needs to run in order to run and manage containers that need not depend on anything but the host init/systemd.
E.g. one of the really appealing things of rkt for me is the simplicity - depending on the level of isolation everything is running either direclty under systemd, or under an individual isolator like systemd-nspawn.
I disliked this tendency towards a herd of daemons intensely when Docker continued as it started and used HTTP for volume/network plugins, and I dislike it just as much now.
It's as if someone sat down and thought long and hard about how to add more complexity and more "fun" failure modes.
Also, Go doesn't dlopen AFAIK.
So you end up running a new daemon that communicates with Kubernetes via gRPC, that then spawns rkt anyway. So you get to RPC and fork/exec.
I'm sure that ends up "optimized away" at some point by e.g. having a rkt CRI implementation that just links in the relevant rkt code. But I'm left wondering why we need this complexity in the first place.
[1] https://github.com/kubernetes-incubator/rktlet/tree/master/r...
The overhead of RPC in an application like this is tiny and the cost of an additional process on 2016 equipment is non-existent.
The overhead isn't necessarily a big deal (and can easily go the other way - if the request frequency is high enough, it's cheaper to keep the process around), but it does also potentially add up.
When the resources you interact with are local OR you have tight over the environment, you're totally right, there's little reason to introduce "middle procs".
When there's a risk that different versions of things will manipulate resources in incompatible ways then the rpc layer makes sense.
And if/when the "different versions of things" are an issue, they can just as well affect the RPC layer.
(I'm not saying that this would make sense for the CRI API, I didn't even looked at it)
In general for an API as simple as the CRI API I'd prefer forking and passing data via stdio or the environment any day.
The advantage of the GRPC framework for developing though are tremendous, I'm building my own CRI implementation and I can iterate extremely quickly due to the fact that it can use RPC.
What advantage do you get from gRPC over e.g. taking a JSON document on stdin or using environment variables? To me the issue isn't that things happens in a separate process, but that I really don't like solutions that ends up adding even more moving wheels without very good reason.
Kubernetes is extremely convoluted as it is.
I tend to think of it like the VFS in the Linux kernel. If Linux was a microkernel, it would talk to each FS module over an RPC interface, since its not, they are standard function calls as statically linked together.
I'm unsure what your Q really is, how would I easily have function request/response layer with json documents or stdin? It seems more complicated. But as I don't quite understand perhaps I'm missing the point.
I also dont think its adding more moving wheels, its just taking one wheel out of kubelet (i.e. dockertools part of kubelet) and moving it elsewhere (i.e. dockershim as a cri implementation). The only increase in "moving wheels" is that even in the non "statically linked" scenario, one has to distribute 2 binaries to each node. kubelet and the cri implementation, but in practice I'm not sure thats really such a big deal.
[1]: https://github.com/kubernetes-incubator/cri-o
Though, I'm getting more and more disillusioned in general with where these specs are heading - the complexity seems to be skyrocketing for sometimes very little benefit.
Not necessarily specific to Kubernetes and/or OCI - Docker is a prime offender.
E.g. typical example: the highly coupled nature of many of the networking alternatives where routing, fabric and ip allocation gets muddled all up, when there are well developed, stable, well tested independent and orthogonal alternatives for tunnelling and route propagation there's a serious level of Not Invented Here syndrome at work in many of the container projects. I'm sure some people need all the complexity, but I'm getting more and more tempted to ditch many of the higher level tools in favour of composing smaller, simpler tools.
(Incidentally I'll make one prediction: one good thing likely to come from CRI is that I suspect it will lead to a new array of Kubernetes "replacements" from simpler composable tools; the APIs don't look all that bad - I just don't like the RPC dependency)
At the moment, the RPC requirement means that you need to have a process that can accept RPC requests (a "daemon" if you like). However, unlike Docker (and containerd), ocid's lifetime is not tied to the lifetime of its containers -- which is one of the main downsides of Docker/containerd IMO. So in principle you could have ocid set up to only start up when kubelet is telling it to do anything. The real benefit of the design behind ocid is that in the future we could switch to a fork-exec model with the kubelet and it would still work.
For example, currently kubernetes is adding a requirement for runtimes to include a "kpod" binary that can do container and image operations even if the kubelet is down. My hope is that eventually they will just make the kubelet shell out to this binary so the CRI is defined through some sort of "here are the cli flags you need to accept" interface.
> the complexity seems to be skyrocketing for sometimes very little benefit.
I wouldn't call the current CRI "complicated", it's just that the gRPC requirement IMO is a bit too much. However, I would hope that since ocid and rkt both don't require daemons (well, rkt requires systemd but that's a given) that they'll reconsider their method of communicating with container runtime runners.