Not really (unfortunately). Kubernetes can leverage a number [1] of container runtimes - OpenShift uses CRI-O, for instance. There's also the problem that Docker can refer to a container runtime, orchestration engine (Swarm, which is analogous to k8s), CLI tools (`docker` and `docker-compose`), or HTTP API (registry). At my last few gigs the maintenance of separate configs for docker[-]compose and k8s has been a consistent pain point.
The container ecosystem is pretty complicated :-).
Wow this is such a beautiful read. Has anything like this been done before for some other topic?
P.S. This sentence is hard to grasp for me
"This Unawareness helps Decouple systems that produce events from otters that read events." https://www.gentlydownthe.stream/#/20
I'm glad people are enjoying this. I wrote a story two years ago about Queues / Kinesis - and am in the process of getting it illustrated. I doubt it'll make any money, but it's good to know that people enjoy this type of thing!
> "This Unawareness helps Decouple systems that produce events from otters that read events."
It simply means that the producer doesn't need to maintain a list of listeners. It just throws the event into the stream, and assumes that anyone who wants to read it will be able to do so.
I don't have a link but the NSA has a coloring book about cryptography that's in the public domain, since it was created by the government with tax dollars and has no copyright notice. I actually have a copy I've been meaning to scan, I'm just afraid of ending up on a list if I do. :)
Current version:
"First, they dropped large stones into the river, splitting each topic into a smaller number of streams, or Partitions."
I know nothing about Kafka, but I think maybe this should be:
"First, they dropped large stones into the river, splitting each topic into a number of smaller streams, or Partitions."
Could be just me. I see Apache kafka more nightmarish than bureaucracy in 'The Castle' or Police/judiciary in 'The Trial' from author Franz Kafka. Mr Franz btw stole his last name from this very famous Apache Kafka project.
Kafka is very configurable .. and teams struggle to optimize the production deployment configuration and tend to second guess their decisions. Getting a managed deployment from Confluent etc. mitigates these issues as you can count on proven SMEs to deliver a well optimized solution. However, if you were to DIY , you might not be so blessed. Disclosure: I never did a DIY Kafka prod deploy -- my employer was already a Confluent customer.
Unfortunately we do not have something to discuss. All these consumer/producer/broker failing randomly, system behaving erratically is me doing it wrong. Those white papers from Confluent and testimonial from customers are proof that Kafka works fine.
Individual Kafka consumers and producers tend to have simple behavior, which is good, and initially plugging them together yields simple, predictable systems, but people tend to keep going until they experience pain, at which point they have a system that is right at the limit of what they can understand. Then further feature work pushes them over the limit into darkness.
I think Kafka needs the equivalent of OpenAPI and Redoc, a simple spec and document generator, but for groups of consumers and producers rather than single applications. This would increase the tractability of complex systems, but it would also let you see the system getting more complex over time, even when you haven't reached the pain point yet.
Thanks for the tip; that's a really cool project. It looks like the documentation generator for it works on a single file at a time, and a single file defines a single service. I was thinking of something that works on the level of a system of interacting services. I want a tool that reads the API specifications for a group of services and documents the interactions between them. I'd like to look at service X and see that it emits message Y on queue Z, and then see which services read message Y on queue Z, jump to their documentation, etc. I think the AsyncAPI format is perfect to build on, though. I'll start there if I decide to take a hack at it.
I'm taking a closer look today, and it looks to me like one AsyncAPI document, which can be defined using multiple files, defines one application. If you look at the "fixed fields" section under the root object, the second field is "id"[0]:
id Identifier Identifier of the application the AsyncAPI document is defining.
This format is fine, but the tool I'm looking for is something that will read the definitions of multiple AsyncAPI documents (multiple applications) and show how their inputs and outputs connect, so I can answer a question like, "When application X publishes message Y on channel Z, which applications consume that message?"
AsyncAPI gets me 90% there by defining the service spec and providing code to parse it. It's possible somebody has already written the rest; I'll have to see.
Different strokes, I guess; I can't count the number of people that openly gush about _why's Ruby book with the cats or whatever, but to me it just read like the ravings of a highly functional something-opath. But reasonable people can disagree, and I'm sure I'll get downvoted for disagreeing with the hivemind.
My initial impression was "more of this garbage" - but it was really well done in the sense that it distilled the core functionality of the system in an easy-to-understand form.
The guy above you mentioned manga-guides to stuff, which utterly fail at their job, which again, is to distill key-information in an entertaining, easy-to-read, general (but fully accurate) manner.
I don't really get these weird ways of explaining different technologies, just give me a straightforward text description. But straightforward text isn't going anywhere and it doesn't hurt me if people want to read mangas about foxes or whatever.
Yea that's where I was going with asking why. It feels like it's a trendy thing is to convert a technical thing into a children's book. (Which is odd.. children don't need that book.. so it's for an adult that wants to consume children's literature)
Note: I'm not against creative attempts to explain technical concepts. But the form to me seems odd, and that it feels like we're producing very short tutorials in a childrens format. That's even weirder.
To me, the attraction is less about the specific art used to communicate the concept, and more about the careful attention to a fully-formed analogy that explains the tech in completely different terms.
These kinds of explanations tend to focus on the most critical/important concepts, and help validate (or dispel) assumptions I've made about the tech.
This focus on analogy also lets the author tell the story faster, because I already understand:
- What an otter is
- That rivers flow
- The water flowing down a river that forks will be spread across those forks
- etc...
Depending on the strength of the analogy, it's possible to get the reader on the same page much faster than an intro/tutorial that must first explain foundational concepts just to get to the basics of the technology itself.
Do people find these analogies helpful? The concepts aren't that difficult, the audience for them is already technical, and adding a cutesy abstraction about it makes it harder to understand.
The art and animation in this is great, but I feel like the author's talents are wasted on a document with no audience. Make a kids' book instead!
I've never really been a fan of analogies, because I feel like in the end I just need to first understand what the thing is underneath, so I can understand how the author has built their analogy... so the task of understanding why they used this analogy is equivalent to understanding the thing itself.
I never found it helpful from a purely technical perspective, but I found it extremely eye-opening as an unorthodox approach to programming that really captured my imagination. It was definitely something that encouraged me to dig deeper into Ruby and do more explorative "creative" coding.
There's no shortage of dry technical documentation, so seeing something akin to outsider art in that space was really refreshing.
Personally I would love to see more technical books come with a soundtrack!
I like the novelty of it. I have a copy of The Manga Guide to Databases at my desk that I occasionally 'lend out' to people that mess up my databases. It won't turn anyone into a db hero but it's a decent primer at least.
Because to me it seems like a very pure way of separating a concept from implementation. In language and metaphors that are easy to understand and fun to read. I really like it. Now, in a very short time I can decide if Kafka can solve my problem or whether I should move on. I for one store information presented in this way much better, it's feasible that in 5 years I'm presented with a problem and the Otters pop into my mind.
> Using deft allegory, the authors have provided an insightful and intuitive explanation of one of Unix's most venerable networking utilities. Even more stunning is that they were clearly working with a very early beta of the program, as their book first appeared in 1933, years (decades!) before the operating system and network infrastructure were finalized.
The fact that since January 2000 nearly 16k people "found this helpful" opened my eyes to how Amazon book reviews really do have the potential to be "classic".
From the review:
"The Story About Ping has earned a place on my bookshelf, right between Stevens' Advanced Programming in the Unix Environment, and my dog-eared copy of Dante's seminal work on MS Windows, Inferno. Who can read that passage on the Windows API ("Obscure, profound it was, and nebulous, So that by fixing on its depths my sight -- Nothing whatever I discerned therein."), without shaking their head with deep understanding. But I digress."
I think the otters should question some of the things that are meant to be axiomatic. Is tight coupling really a scalability problem? Otters should be prepared to defend these statements. Nixie's song might be the vacuous equivalent of "Mongo is web scale".
Also: did adding the stream really decouple the otters, or did it just make the coupling less visible? After all, the consuming otters may still depend a great deal on the precise behaviors of the producing otters. It's just that now the producing otters don't know in what ways other otters depend on them, making it harder for them to make changes without harming their otter-dependents.
You need some other animals - Beavers maybe? - who are in the same forest and who just have a simple system of scratching messages onto the side of a single large tree somewhere that everyone goes to look at regularly. Then at the end see how many picnics the beavers missed vs how many otters lost their fucking minds from the complexity of the system and then see who comes out ahead.
Well, fishing and mining are both resource extraction but apart from that have nothing in common. Kafka and REST both have APIs but apart from that have nothing in common.
I read the whole thing. That's why I have the question. So they used to have messages send to everyone (websocks?), now they have the messages persistent in different channels for people to consume. Isn't this just different entry points of a Rest API?
I'm sorry you're getting downvoted, but I think this question is legitimate because the book is peddling Kafka as if it's the only way to do event sourcing. Event sourcing is what you should compare with REST APIs, Kafka is one way of doing it, but you can do the same with any database, as long as you have a way to write things in and read things out and organize them, you can achieve event sourcing.
With REST APIs (first few pages of the book), services talk directly with each other, with event sourcing (the rest of the book) services talk with an event store (Kafka in the book) as the intermediary.
WOW, this is so nice. The drawings are really cute and the animations give it its own style. Bit of feedback the color palet all over the place. Could use a bit of hierarchy to help the focus of the viewer and reuse same colors to make it more coherent.
No worries. ±1 day is universally race-condition territory. Optimizing at that level is super tricky and incredibly hard to get right, which is why I wanted to mention it.
I've been trying to put my finger on why Kafka so well captured the imagination of many distributed systems engineers. My best answer is, "low-cost publish and multi-consumer data-sharded subscribe is the key to resilient horizontal scaling and parallelism."
Kafka has its flaws, but it really served us well. We have Python Data Engineers who focus on distributed system design[1], and Kafka is one of the team's least finicky open source components, but it is used everywhere, and it basically enables the entire rest of the real-time data processing stack.
232 comments
[ 3.1 ms ] story [ 211 ms ] threadNow I want a Docker children's book please.
The container ecosystem is pretty complicated :-).
[1] https://kubernetes.io/docs/setup/production-environment/cont...
P.S. This sentence is hard to grasp for me "This Unawareness helps Decouple systems that produce events from otters that read events." https://www.gentlydownthe.stream/#/20
https://medium.com/hackernoon/intuitive-rl-intro-to-advantag...
It simply means that the producer doesn't need to maintain a list of listeners. It just throws the event into the stream, and assumes that anyone who wants to read it will be able to do so.
RSS vs an email list, I guess.
https://poignant.guide/
More generally there's this list of similar fiction https://fiftysevendegreesofrad.github.io/hard-comp-fi-fictio...
Otherwise quite well done!
Is there a typo at https://www.gentlydownthe.stream/#/22 ?
Current version: "First, they dropped large stones into the river, splitting each topic into a smaller number of streams, or Partitions."
I know nothing about Kafka, but I think maybe this should be: "First, they dropped large stones into the river, splitting each topic into a number of smaller streams, or Partitions."
(my emphasis for both versions) "
"Then, the otters would decide which part of the river to put the message in"
I'd imagine a child looking at this going like "why are they doing this to the river". Why are they are throwing things into the river. :D
I think Kafka needs the equivalent of OpenAPI and Redoc, a simple spec and document generator, but for groups of consumers and producers rather than single applications. This would increase the tractability of complex systems, but it would also let you see the system getting more complex over time, even when you haven't reached the pain point yet.
(I view OpenAPI as a near-failure, but, good luck to everyone trying.)
https://www.asyncapi.com/docs/specifications/v2.0.0#referenc...
AsyncAPI gets me 90% there by defining the service spec and providing code to parse it. It's possible somebody has already written the rest; I'll have to see.
[0] https://www.asyncapi.com/docs/specifications/v2.0.0#fixed-fi...
The guy above you mentioned manga-guides to stuff, which utterly fail at their job, which again, is to distill key-information in an entertaining, easy-to-read, general (but fully accurate) manner.
Note: I'm not against creative attempts to explain technical concepts. But the form to me seems odd, and that it feels like we're producing very short tutorials in a childrens format. That's even weirder.
These kinds of explanations tend to focus on the most critical/important concepts, and help validate (or dispel) assumptions I've made about the tech.
This focus on analogy also lets the author tell the story faster, because I already understand:
- What an otter is
- That rivers flow
- The water flowing down a river that forks will be spread across those forks
- etc...
Depending on the strength of the analogy, it's possible to get the reader on the same page much faster than an intro/tutorial that must first explain foundational concepts just to get to the basics of the technology itself.
The art and animation in this is great, but I feel like the author's talents are wasted on a document with no audience. Make a kids' book instead!
But I can't deny they are very popular!
I'm going to try and fit this into as many sentences as I can get away with!
There's no shortage of dry technical documentation, so seeing something akin to outsider art in that space was really refreshing.
Personally I would love to see more technical books come with a soundtrack!
Different people, different preferences I guess.
This prompted me to make an awesome repo to collect children's books on technical topics (which I have seen a few on HN).
https://github.com/searchableguy/awesome-illustrated-guides
I couldn't find a similar list. If there's one, let me know.
Google Machine Learning - https://cloud.google.com/products/ai/ml-comic-1
Google Federated Learning - https://federated.withgoogle.com/
Also, why’s guide to ruby, though it’s probably out-of-date: https://en.m.wikipedia.org/wiki/Why%27s_(poignant)_Guide_to_...
https://www.amazon.com/review/R2VDKZ4X1F992Q
The fact that since January 2000 nearly 16k people "found this helpful" opened my eyes to how Amazon book reviews really do have the potential to be "classic".
Your question makes as much sense as: why is <new database X> any different from SQL?
OK. So I didn't get what Kafka is.
With REST APIs (first few pages of the book), services talk directly with each other, with event sourcing (the rest of the book) services talk with an event store (Kafka in the book) as the intermediary.
Kafka has its flaws, but it really served us well. We have Python Data Engineers who focus on distributed system design[1], and Kafka is one of the team's least finicky open source components, but it is used everywhere, and it basically enables the entire rest of the real-time data processing stack.
[1]: https://www.parse.ly/careers/python_data_engineer
That said, I feel like the otters could have just made a bulletin board to solve their problem