I don't completely agree that this is the cause, as CRDs also create revenue-generating demand from controller software that acts on them.
But I absolutely agreed with this:
> There are applications out there that are storing their complete state in CRDs. Please do not do this. The Kubernetes API server is not a general-purpose datastore.
A thousand times, yes. The API server is good at what it does. It is not a suitable replacement for a real, honest-to-Codd database. The idea that laying hands on a connection to a postgresql or mysql server is somehow more onerous than spinning up an entire goddamn cluster is bonkers. Bon. Kers.
There are a number of reasons. One is that states don't stay simple for very long. Just the other day I saw someone ask "how do I find all the children of a parent resource?". The answer was to iterate over everything that might have that parent resource as an OwnerRef. In a normalised schema this would be simple. In a document database it is not.
Another problem: there are no multi-document transactions. If I rely on updating A and B together, I am out of luck. I will instead need to account for A and B being out of sync, which involves 4 permutations: A,B; A, !B; !A, B; !A, !B. As a CRD controller author I am now responsible for a shitty database. Where Kinds have crisp hierarchies this is fine, the supervising controller can inspect and repair the lower documents. But many things are not hierarchical in nature and you can easily wind up with control loop fights.
Another is that there's a difference between "I already have a cluster" and "I am using the API server as a database". Because CRDs wind up in a single global document store, folks inevitably want them to have hard isolation. RBAC, namespaces etc get you some of the way but it's basically just hard. Eventually some bright spark suggests putting things into a different cluster, at which point the balance of ease flips the other way.
CRDs are super useful extension points (note, for example, I am writing a book about Knative), but they are not suitable for most application and business logic purposes.
Well, I agree that you can't really use CRD as an alternative to a database. But wouldn't you say that there are cases where it makes sense to use CRD to keep metadata? Keeping results of job runs for example.
7 comments
[ 5.8 ms ] story [ 32.1 ms ] threadBut I absolutely agreed with this:
> There are applications out there that are storing their complete state in CRDs. Please do not do this. The Kubernetes API server is not a general-purpose datastore.
A thousand times, yes. The API server is good at what it does. It is not a suitable replacement for a real, honest-to-Codd database. The idea that laying hands on a connection to a postgresql or mysql server is somehow more onerous than spinning up an entire goddamn cluster is bonkers. Bon. Kers.
Another problem: there are no multi-document transactions. If I rely on updating A and B together, I am out of luck. I will instead need to account for A and B being out of sync, which involves 4 permutations: A,B; A, !B; !A, B; !A, !B. As a CRD controller author I am now responsible for a shitty database. Where Kinds have crisp hierarchies this is fine, the supervising controller can inspect and repair the lower documents. But many things are not hierarchical in nature and you can easily wind up with control loop fights.
Another is that there's a difference between "I already have a cluster" and "I am using the API server as a database". Because CRDs wind up in a single global document store, folks inevitably want them to have hard isolation. RBAC, namespaces etc get you some of the way but it's basically just hard. Eventually some bright spark suggests putting things into a different cluster, at which point the balance of ease flips the other way.
CRDs are super useful extension points (note, for example, I am writing a book about Knative), but they are not suitable for most application and business logic purposes.
This is misleading, the object _instances_ of a CRD belong to a namespace, unless they are cluster wide