According to the others on this thread it's related to the date when Kubernetes was released. It wasn't clear to me, but thank you, I think you raise another good point.
How tedious to have to constantly refute FUD when it's easy to find the answers with Google.
Anyone involved in Kubernetes, near CoreOS at the time, or really anywhere in the space at the time (instead of looking back at it with anger), knows this all to be false. CoreOS was setting direction for etcd, and understandably adding features for one of its bigger users (and in fact, some of those features are used by things of larger scale than k8s).
Kubernetes itself was started by Googlers, many of whom are still there or left to go... do Kubernetes at Red Hat (IBM) or as a startup, or at Microsoft. But to act like it was an outside project started by people who had previously quit, or are somehow unqualified to work on an orchestrator, is just an an angry untruth. Every major committer to Kubernetes besides a handful of RH folks were at Google when Kubernetes 1.0 came out. I'm happy to be corrected but I know it's hip af to hate k8s (just like two days ago https://news.ycombinator.com/item?id=23807556)
Thanks. I just wanted to know what the author meant; it wasn't obvious that I needed to Google the quote to figure out the meaning. I certainly hope that I wasn't adding any FUD to the discussion.
How about that Kubernetes was released in 2014, not 2015? It's the first thing that shows up when you Google "Kubernetes" - Initial release: 7 June 2014; 6 years ago.
Evidently, OP couldn't even be bothered to gather basic facts about the tech he trashes.
Thank you. I'm sorry that you feel like I was trashing the tech. I really didn't mean to come off that way by asking a question about a statement that the author made.
Agreed. The writing style is so full of bile that any actual point just drowns in it and makes the author seem petty and bitter about the world. Not a good read, even for those that agree with the author about unneeded complexity.
I feel like this page actually comes off as a pretty good recommendation for systemd. Like it's a piece of software with normal bugs and the usual crop of hard decisions.
Almost like with etcd, it started simple, but got expanded into a huge incomprehensible monster by the original developers. Politically and business-wise, systemd is a clear win for Lennaert and his clique, as well as the whole Redhat/IBM.
But I am sure that future historians will view the impact of systemd on computing as largely negative.
The systemd/RH relationship is complicated. Red Hat doesn't use most of systemd's features--they don't even ship systemd-networkd on RHEL8, preferring NetworkManager instead. On the flipside, I see more use of systemd's features on Arch or Debian.
I don't think you can frame systemd as some kind of RH trojan horse when so little of it makes it into RHEL/Fedora.
So while Arch ships everything, and being Arch it doesn't really have a default, its recommended networking system isn't systemd-networkd but rather netctl, which is basically everything systemd-networkd should have been.
`netctl` was explicitly written because systemd didn't have a network daemon. The past 3-4 years the community has in general recommended against using `netctl`.
It was never removed from the ISO because the releng maintainer didn't put that much thought into it, but I'm happy to tell you `netctl` was replaced with `iwd` on this month ISO release.
> But I am sure that future historians will view the impact of systemd on computing as largely negative.
I don't really think so. They probably will point out a lot problems with at some point somehow got fixed but it should be net-neutral or positive.
The thing is todays linux is running manny services and to do so nicely you want to have some form of service manager which does startup/shutdown/restart/circicute braking and helps with interconnect. Which is what systemd does and what mainly differentiates it from many previous systems which where mainly "just" start-up helpers. Through there are IMHO a large amount of problems with systemd the general approach is IMHO good, just the implementation isn't so much.
Sorry, no. Systemd doesn't add anything for service management that didn't previously exist elsewhere other than the unit file syntax for service definition. What it did is pull as many disparate aspects of "how do you run a thing" under its umbrella as it could, so it controls as much of the environment as possible and doesn't have to care about playing nice with anyone else. That's the general approach.
It's a logical fallacy to assume, because systemd was the thing that came along and resulted in the problems in sysv being fixed, that systemd's approach or implementation were necessary, desirable, or the best option available.
You may not realize you're inciting a flame war. If you google for this there's no shortage of arguments going back 10 years and continuing to this day
"You can't discuss this topic because of a flame war" is ceding territory to the assholes. I'm willing to accept that when it comes to talking about political topics here, because this isn't a political forum, but I refuse to accept that we can't talk about technical topics here because some people refuse to keep a civil tongue.
Similar opinion here. I put it off for when I could learn it better, and so far it's been fine... sometimes nice too when it has a feature I want that's already built-in (depending on version level and all that).
Ditto. my personal experience with systemd has been "wow, it's easier than ever to write an init script that runs when it's needed".
Looking at the components that make up a modern UNIX system, it's definitely time to think "how can we make a new operating system, evolved from UNIX, but with a more coherent core running the majority of run time orchestration".
I'm mostly a happy systemd user. Unit files are much easier to work with and more consistent than init scripts, the faster bootup (at least on my arch and Ubuntu systems, maybe you could make Gentoo even more minimal with initscripts), and the container/isolation features won me over.
That said, I can understand the objection to software that would previously support BSD or macOS now only providing systemd units or depending on other parts of the systemd project that in turn depend on the init making at a pain for portability across unices. Or software like gnome taking up huge amounts of time for the BSDs to support.
Unit files are great. SystemD should have stayed there. But then it handles networking(including configuring network interfaces), and time synchronization, and name resolution, and the kitchen sink.
From an administrator's perspective it's the same problem with every declarative run control system: there is a layer of indirection between what's written in the configuration files and what the machine does (as opposed to an imperative system, where it's a bunch of shell commands that are literally the actions the machine takes).
With systemd in particular the indirection is notably (needlessly) opaque, with a big spaghetti mess of unit files symlinked in 18 different places, some of which have semantic meaning to the init system and some of which do not.
I have used systemd successfully in production and have a positive opinion of it. I like that it is integrated, documented, and self consistent.
None of the negative ideological complaints against it resonate with me, as none of them seem to impact me (yet, anyways). My experience with past init systems is that they are cryptic and error prone.
It violates the old school UNIX Philosophy of (1) simple standalone tools (2) text as the universal interface.
(1) systemd is a monolith that aspires to do almost everything from booting to networking to sound. It's in many ways a Kubernetes sans distributed systems.
(2) systemd uses binary logs, abandoning "text is the universal interface"
systemd is also a giant single point of failure.
Personally I don't see the big deal. The actual reason people hate systemd is that it's the type of design a software company like Microsoft would come up with rather than one that a bunch of gritty basement hackers would create. They have found refuge in OSes like Gentoo and OpenBSD, so systemd served as a kind of explusion of this type of programmer/user from the Linux ecosystem.
i went trying to learn how to make my linux box do something at startup.
as somebody learning how to use my linux, systemd clouded the discovery process of learning how to linux in the intended way.
i ended up using systemd, but now that's the thing i'm familiar with if i want to add some service to my computer... so it has infected me with some non-linux-y knowledge that won't go away soon.
Also compare it to doing things the Windows way: Something to run on startup? Put it in the "Startup" (magic) folder. Want something to do the things a service does, where it gets auto-restarted and auto-run and even logs success and failure? Like three clicks can run arbitrary anything using the scheduled task GUI and things get auto-logged in the Windows Event Log
i guess if i wanted windows, i could use windows... right?
edit for tone:
sry, that sounded saltier than i wanted. i actually went to linux because i was convinced that was the better way to do things, and i wanted to do things the better way.
There is https://blog.darknedgy.net/technology/2020/05/02/0/ which puts systemd in its historical context, explains some of the criticism, and then goes into the technical details. It's a long post and the tone makes it difficult to continue reading at times, but I learnt a lot from reading it.
One practical problem I've had is with journald. The log files are useless outside of the system they were generated on. They could have just used a simple text file and avoided major headaches.
I for one think the team behind it and especially Poettering have incredibly bad taste in software, which bleeds into their designs, and possibly the worst attitudes and culture of any major OS initiative, making Linus look tame (hell, at least if he got bent out of shape he was usually right, and not just throwing a tantrum because he could get away with it). I don’t necessarily disagree with it’s goals, even, I just think the implementation’s not great and the team are, largely, self-righteous jerkasses. See links others have posted here, their attitude and demeanor’s pretty out-there.
Branding perhaps. If 50 tools had been released, called "newInit", "newDns", "newNTP" etc, with each tool succeeding or failing (upstart) on it's own merits, that would be one thing.
There is a perception that systemd has taken over the entire system as a cancer - replacing things that have worked for decades just because it's cool.
One annoyance I get is this: I've just taken an Ubuntu 1604 server running apache and b0rked the config file.
$ sudo systemctl restart apache2.service
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.
$
That's crazy. "Something broke, now go look here or there for why". Compare under an old ubuntu 1404 machine I have lying around
$ sudo service apache2 reload
* Reloading web server apache2
* The apache2 configtest failed. Not doing anything.
Output of config test was:
AH00526: Syntax error on line 1 of /etc/apache2/sites-enabled/sd.conf:
Invalid command 'blah', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.
That is so much more helpful.
You run the first and it outputs the last few lines of log. Why not output that when I call restart in the first place?
Another one is taking over DNS from resolv.conf. Muscle memory from years of "cat /etc/resolv.conf" (with a tab somewhere in the middle) has been replaced with the ungainly "systemd-resolve --status".
Now all of this is great if you're running servers as cattle - if you have thousands of machines doing web-scale stuff you don't want to see what's happening on a single machine, your layers of automation abstract it all away, your system can cope with machines going down, you probably don't even have an ssh account.
If you are looking after individual machines though, systemd came in and turned everything upside down to presumably make things easier for cattle owners.
I think he/she made it clear it is needlessly replacing parts of the job.
The question is, should one feel any one way about that. I’d imagine it would be the most annoying thing on earth, so it’s a valid emotion being expressed.
> I think he/she made it clear it is needlessly replacing parts of the job
Maybe? There weren't any actual reasons for why the author so vehemently disliked all of the technologies mentioned, and there was certainly no acknowledgement of the potential benefits of them.
"This is systems. You are to be trying to be clever. Please stop." is a damn good rule of thumb and I think lots of people tend to go all-in on k8s without considering whether their use of the features will actually justify the level of additional clever so introduced.
To me, that being the 'actual reasons' was implicit from the rant - I don't disagree that it was a rant rather than an argument.
The author is arguing that simple, easy to use, and easy for hobbyists and upstarts to learn tools are being replaced by complex systems designed by and for the 1% of companies who will actually need the complexity.
They are arguing that "House Builders Inc" has invented an automated robotic nail gun and is encouraging standards industries to recommend the only things built with such a system are compliant.
Lamenting about complexity just for the sake of lamenting about complexity and then adding a few potshots at Google/FB/etc. doesn't exactly make for a thought-provoking, nuanced, or interesting argument, though
> This just reads as a systems engineer who is angry about new tooling/processes replacing parts of their job.
but it doesn't its needs a bunch of work to upgrade/migrate replace. which is what they are complaining about. having to replace perfectly good stuff, with _new_ stuff.
Unless you are bringing teams together onto one platform, k8s is never going to save you cash/engineer hours.
I don't experience the sadness that the writer of the blog mentions. However I do understand what they mean. Personally, I've decided I don't really like huge projects, and instead have focused on smaller iOS apps. They don't have dependencies, except for what the Apple platform offers. There's a backend, but it's really abstracted away for me through a RESTful API. And I'm able to continually and without fail, deliver something of value to my customer.
> the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys
I maintain a (/the only?) etcd3 library for Node.js[0], and used etcd extensively on my former team.
None of these things are new to etcd3 API. All of these are present on v2 as well[1], whose API the author extolls, or are built within clients on etcd's base APIs (e.g. there's no 'lock' API, only leases). However, with etcd3 we get stricter typing, better performance, and better semantics (e.g. watch streams and lease streams over polling) thanks to GRPC.
In general these rich APIs allow 'average' engineers to build complex distributed apps more correctly. I've built reliable sharding, hash rings, elections, and so on based on etcd's API--none in more than a hundred or two lines of code (more in Go, less in Node.js). All of these are classic hard problems that etcd makes easy. Sure, there's innumerable standalone services for each of these things, but often there's no need to take the cost of many tools when one would work.
I feel like the blog post could largely be summed up as "stop threatening to remove the normal HTTP API, some of us find that a lot easier to debug and ease of debugging is essential for a core operational component of a distributed system" - which is, I would argue, an entirely reasonable thing to want.
I quite enjoyed the rantiness on a "being entertained" basis but it did rather work against effectively making the core point.
> stop threatening to remove the normal HTTP API, some of us find that a lot easier to debug
I've seen the trend toward complexity in other projects too, and it harms not just ease of debugging, but ease of hacking.
Take, for example, Swagger UI[0]
v2 was so simple. It was vanilla JS using jQuery. I, as an embedded systems developer, was able to easily hack it so it could read in the OpenAPI JSON from a database and I even added a little search box so you could filter down the APIs you wanted to see. Super fast and easy and worked just the way I wanted it to!
Starting with Swagger UI v3, it became... extremely labyrinthine by comparison. It was completely re-written in React JS and now I need a bunch of new tooling to make changes and everything was broken out into dozens of different modular files so I couldn't find where I needed to make a given change, also not to mention I've never used React so it felt like the barrier to hackability was dramatically increased.
I'm sure full time React folks love the new architecture because it's so much <cleaner/safer/scalable/etc>, but for me the change was extremely confusing and made the tool unhackable (I tried for a few hours to get it to do what I wanted, but it started looking like I was just going to have to learn all of React and I threw in the towel), and so I'm permanently stuck on v2 for now.
> Complaining about an open source project changing technologies because you can't be bothered to learn them is a bit rich
Web technology is its own beast. I invested a long time learning and mastering AngularJS only for all that work and knowledge to be flushed down the toilet over the next few years. Web tech has terrible ROI so that's why I "can't be bothered" to learn the latest one. I'm salty specifically about Swagger jumping on <insert latest framework> instead of sticking with simple vanilla JS that everyone understands.
There's a lot of justified hatred for kubernetes, but for fuck's sake why did anyone adopt it in the first place, if they didn't understand what they were getting into? It's like the startups from 10 years ago who absolutely had to use MongoDB because "scale", and then they never get off the ground because they're trying to implement ACID from first principles.
Since hackernews is mostly junior devs or senior principle architects (formerly junior devs), I expect there's a lot of stockholm syndrome going on in these comments.
It's possible that many people (like me) had been trying to solve a similar non-business problem for years, including rolling our own solutions and the like, and when an open source option widely backed emerged and looked like a possible standard, we accept some warts in exchange for a broad, general purpose, flexible, automatable, and well-thought out (yes) solution.
Of course K8s is not perfect, and it's overkill for small to medium apps (I think hype train convinced a lot of people they would need to scale to massive cloud levels when really they didn't), but if you have ever needed K8s (especially for a complex microservice system at big enterprise level) then you know the value and you remember the proprietary vendor-locked era of sadness before K8s emerged.
There is a cost for not getting on a hype train. CEOs rather waste a couple of billion dollars on that machine learning/AI hype train, because if they don't hop on that train, Wall street folks see these companies old-school, not innovating. So, they bring in Chief AI officer and write about about how machine learning gonna help their companies in the Quarterly reports. Even if they lose $2B in this AI venture, just treat as insurance premium to keep the stock price going up.
This. I cannot tell you the number of times where I've seen teams use Mongo and then reinvent a schema as part of the development contracts. It's enraging.
Engineering-driven-development, where the purpose of engineering is engineering and business problems take a backseat in favor of hiring lots of engineers, giving them toys to play with and having buzzwords to on your careers page. In this case, your objective is to fill up your stack with as many shiny things as possible regardless of whether they are necessary or appropriate to solve the business problem at hand.
This strongly correlates with bullshit VC-funded companies that wouldn't be profitable even on a good day so their only objective is to raise as much funding as possible (the business problem is secondary, because even solving the business problem perfectly wouldn't make them profitable) and chasing fads and buzzwords is sadly a valid strategy for raising capital.
Oh man, thank you for this. I cannot stand the engineering culture of Google and I especially cannot stand the engineering culture of Google wannabes. Google ruins everything they touch, and I'm really sick of the engineering worship they most certainly do not deserve. Every single piece of software to come out of Google is overengineered, broken garbage.
Your statements are overly hostile and hyperbolic. There's been plenty of great software out of Google and lots of good ideas taken for granted. I'll give you folks trying to build things the Google way without Google tools or scale is more often than not the wrong thing to do, but that doesn't invalidate the good.
Go was made by a small group of smart people with debatable loyalty to Google. Google just ran with it, and they haven't done a great job with it since.
I'm not sure what you mean by debatable loyalty. The creators of the language have all worked at the company for more than 10 years and support the product for internal users.
Can you put your grinding axe away and stick entirely to facts and technical merits?
Look, Go has way more to do with the experience of its creators than with their contact with the engineering culture of Google. The influences from Plan 9, Inferno, Limbo, and so on are all extremely apparent in Golang's design and were all born outside of Google. Very little of Google's spirit made it into Go.
I don't know how you got from "debatable loyalty to Google" to "the language was mostly inspired by Bell Labs tech, rather than Google", which is perfectly fine. But it doesn't explain your animosity and refusal to discuss based on technical merits, rather than just (what is clearly) axe-grinding.
Note that Go ended up being a major language within Google for a wide range of systems (even replacing systems built previously by the Go developers, such as sawzall), so rather than saying "debatable loyalty to google", might it not be better to say that the developers worked to transform Google from the inside to be less dependent on C++ and Java?
I retract "debatable loyalty to Google". I meant something more along the lines of "debatably attributable to Google's engineering", i.e. a statement directly in support of my main argument.
Gmail sucks. They use it as a vehicle for anti-competitive "standards" and it's needlessly difficult to deliver mail to. Their new JavaScript-riddled frontends (which they seem to develop and then shrug off every couple of years) have awful performance and are terrible at doing what has ultimately a very simple task for 30 years.
Google Maps is fine. OSM is superior technically but Google Maps only wins because it has access to better data. It's not an example of great engineering, it's just okay.
Google Docs is pretty good. At least it was, the last time I used it was several years ago. I can't comment on the code, which is mainly what I'm commenting on in this thread: their engineering quality. Throw enough monkeys at a typewriter and maybe Google Docs comes out, but who knows what it looks like on the inside.
Engineering quality is about nothing other than meeting the spec- how well does google accomplish that?
It sounds like you have some issues with the design itself, maybe some issues with the business practices. Those are issues with the spec, not the engineering. The engineering team delivers the spec, they do it smoothly, they do it at scale
Highly used? So what? Might does not make right. They're definitely not cost efficient. Google's standard play is throwing tens of millions of dollars at a problem, which is not good engineering, even if it works.
Edit: you changed your comment out from under me and I fail to rouse the enthusiasm to reply to your new comment.
I thought each of these were good until I tried something different. I think people moved from legacy things like Windows Live Mail, MapQuest, and Word 2000 to these services, and then never noticed the rest of the world got better outside of Google-land.
Gmail operates what is probably an industry-worst mail client. I have helped people using AOL Mail in 2020, and it is a superior experience. It performs better. FastMail is a paid service, and it's quality matches that: FastMail can run in circles around Gmail every single day.
Google Docs/Sheets is Office without all of the reasons businesses use Word. You can try to patch in some of it's holes with add-ons (I had some experience with a Mail Merge add-on for Gmail/Sheets, and it was fine, I guess, but a paid service from a third party) but with a lot of added jank. Collaborative editing was it's claim to fame, but you can do that just fine in Office 365, with a far more capable app suite.
Google Maps, Drew covered the point pretty well: Google just has the money for more data. It's prohibitively expensive to use that data yourself as a service, and others who have access to less data still manage a comparable and sometimes better product than Maps. Usually while better respecting your privacy.
This is much more about your opinion, and your axe to grind against google, rather than an intelligent discussion about the merits of software.
Personally, I really like JAX and XLA. These are two really powerful systems that work together to implement high performance machine learning and theoretical physics/biology. JAX can basically take derivatives of python functions (useful for training) and XLA optimizes the underlying compute graph to execute quickly on different physical platforms.
I initially thought this was serious hyperbole and I still think it may overstate things a bit, but when I thought about it I actually had a really hard time coming up with counterexamples. Meanwhile it’s easy for Microsoft and Apple. Huh. Best I can do is products that were great at launch... like 12-15 years ago.
Bazel is fucking awful. Most miserable years of my career is when I was tasked with maintaining a build system based on it. Same for gRPC, which TFA also does a good job of shutting down.
I'm going to recommend at this point, you take a breather from your invective, cool off for a bit, and return when you have something valuable to contribute. You're going to have to do better when commenting on Hacker News: you're just attacking stuff because you don't like it, rather than making technical explanations.
> Bazel is fucking awful. Most miserable years of my career is when I was tasked with maintaining a build system based on it. Same for gRPC, which TFA also does a good job of shutting down.
Counter-anecdata: my most miserable years of my career were when dealing with bespoke build and API systems that weren't Bazel and gRPC.
Blaze/Bazel has its flaws (including poor integration into the world outside google), but what are you comparing it against? Any general purpose build system I've seen that is not bazel based and not nix is a flaming pile of garbage: almost nothing else can even figure out a correct dependency tree, let alone which parts of it have changed and need recompilation. Bazel also uses a familiar, readable and yet concise subset of python for build description, whereas most other build systems use customer languages with syntaxes and semantics that are comically bad if you don't have to use them and tragically so if you do.
Again, gRPC has its flaws (and I'm not a fan of HTTP2), but what are you comparing it to? For internal services REST is acceptable for extremely simple, stable and low volume APIs. For everything else it is an obvious disaster: terrible CPU/network/memory performance, you need to write more code than you would have to with gRPC and get zero type safety unless you add some json schema monstrosity on top of it at which point your performance will likely take a further nose dive. Also, good luck evolving the protocol – gRPC is carefully designed to support this. Far from doing a good job, "TFA" quotes some extremely clueless article on why protobuf sucks that was ripped to shreds when posted on HN previously.
The only compelling alternative to gRPC I am aware of is capnproto (which is derivative, but in some ways nicer), but it has far less eco system maturity and mindshare.
Yes, sorry I was imprecise, it would have been better to have written "derivative of protobuf". To the extent the RPC part is derivative, it's probably mostly derivative of E?
Bazel is the most complicated garbage I've ever had the displeasure of using. It takes tens of thousand of lines of support code just to add a new tool to your toolchain.
> It takes tens of thousand of lines of support code just to add a new tool to your toolchain.
No, I've added custom compilers and languages to Bazel, and it was not nearly 'tens of thousands of line'. Sure, it's complex, but such is life with inversion of control. It's difficult to get around this complexity if you want to follow the design goals of Bazel.
> gRPC, k8s: see TFA
The article says nothing about why gRPC or k8s are bad. It just uses very emotional language to handwave into the general direction of 'bloat' and 'too complex'.
I'm serious. Say what you will about the high complexity of the web platform, or the rising dominance of Chromium. It's a seriously impressive piece of engineering. Have you ever watched the Chromium build process in action? It builds hundreds of static libraries, then statically links them all into one monster executable or DLL (depending on the platform). BTW, part of what enables static linking at this massive scale is the gn build system, which AFAIK is inspired by Blaze (the internal ancestor of Bazel). So IMO that's a major point in favor of that type of build system.
BTW, to answer your point about throwing tens of millions of dollars at a problem: some problems are just unavoidably that big, especially when you factor in things that are required for a piece of end-user-facing software to be usable by the whole world, such as internationalization, accessibility, and in general, complex but usable UIs.
This a rant post that backs up very few of its assertions. Though, the author may not have been trying to write for serious consumption. Sometimes it's therapeutic to have a good rant.
I'm not sure what there is for us to discuss. Nice rant, I guess? The post does not attempt to persuade or change opinions (which, again: cool. sometimes it's nice to have a good rant).
But it's not about kubernetes. It's about simple API's becoming complicated for no reason (according to the author, I have no experience on etcd) using etcd as an example
The author doesn't detail _what_ is more complicated about the gRPC API, other than the fact that it's gRPC. One could implement the exact same API in HTTP or with gRPC; so without specific examples, it's kind of a meaningless critique.
Surely, most consumers of a database like etcd are using a client library, in which case why does it matter if the API is HTTP or gRPC?
Tangential: after reading some of the comments, I was surprised that the blog post was only like 250 words; the author really says very little.
why does everything with a negative valence attract this criticism? surely there are plenty of times in one's life where he or she says negative things about something (a coworker, a boss, a partner, an acquaintance) without any intent to foster a "productive discussion". as though that is something owed to products / services / people / experiences / whatever that we hate for whatever reason. as though those things would somehow magically improve with any amount of "productive discussion". how about not everyone take it personally when someone dislikes something purely because of the experience that they have had with it? given the comments section, the only way for this "discussion" (some call it a post) to have engendered this tone of discussion while still being on the front page is either (1) everyone disagrees with the post, but hate-upvotes it anyway or (2) the points being made strike a chord with people. and regardless of which it is, this whatever the opposite of saber rattling is that causes this kind of reaction clearly misses the point.
> This guy used to run infra at uber and was incredibly salty about every single new technology. There were a lot of bad ones, but every conversation was about as constructive, free of evidence, and bitter as this blog post.
I'm going to assume you're a developer?
Infra people are usually much more apprehensive to take on new technology. Crucially I would describe classically trained sysadmins as 'pessimists to the core'. This is why there's memes of operations saying 'no'.
This is what devops was all about, the shared responsibility of it all. I'm going to assume that uber was perfect and got devops exactly right- but adding technology should in my mind always be met with the absolute most critical eye imaginable; and if he's a classically trained sysadmin then it probably comes from that place of being once bitten twice shy.
I'm explaining that the majority of the people who were sysadmins in 2005 are more likely to be pessimists than optimists and are change/risk averse, I don't think that's controversial and definitely is a meme.[0]
> [thanks for] helping me empathize with toxic people.
The apprehensiveness of sysadmins May have been justified in the world 5 years ago but today it sounds somewhat out of place. Note that critically evaluating new technologies is still an important skill and many infrastructure people I work with are extremely cautious about adopting new tech without spiking/getting to know it. But the sysadmins with the penchant for saying no is probably one of the reasons the devops movement actually kicked off: developers and infrastructure folks could see the immense productivity gains from being able to ship code quickly to production and got onboard the technologies that enabled this after being frustrated with the amount of time that traditional software deployment processes took.
So in today’s world, that kind of attitude is hardly productive. Skeptical? Absolutely. But open minded.
It is and it isn't. I remember the first time I thought "this software is just X that everybody forgot about, reinvented again". That was 2005. Not coincidentally that was a decade after I started working as a sysadmin.
Venkatesh Rao says that there comes a point in your life when you realize things you thought were permanent are temporary, and things you thought were temporary are permanent; he uses "40" as a good rubric for that developmental stage. IT goes through many pendula, whether it be containerization vs. amalgamation, or thin clients vs. thick. What, over time, you learn to hold on to is the tools that have lasted decades and will probably continue to. Right now the pendulum is starting to swing back towards amalgamation, and it will probably swing back towards containerization again in another decade. Whatever the tech is at the time, it can be good to reconsider whether your views will have changed not on the technology, but on the larger pendulum it's riding.
I find it hard to parse anything meaningful from your comment, sorry. I don’t mean to be rude, perhaps blunt.
Please name what infrastructure technology has lasted decades, I am curious to know. If you try hard enough yes you can see it’s all cyclical, you can say, MULTICS was the OG cloud. OK, fine, but what use is that to be as a practical software developer building things? Perhaps with age you start seeing everything as being like something you’ve already seen so it’s not that exciting anymore?
> A decade is a long time in software engineering.
Is it?
Code I wrote literally 20 years ago is still running in production, and I get paid to work on an app with plenty of code around that was first committed in 2009.
If you build a system in k8s today, and it's a success, there's good reason to believe you'll be on k8s in 2030.
That’s pretty cool, really. I can’t really find studies on average software life time, so all I really have is anecdotes. I can’t imagine software running for that long on eg public clouds without any kind of major refactor considering how quickly technology changes today, and how aggressively cloud costs push companies to change. Specifically, I can see something like “Serverless” coming to maturity and completely changing the paradigm (again!) on how software is architected. But it’s possible I’m wrong and like you say, k8s will be around for much longer than I anticipate.
And this is why every single project out there is now a house of cards (or should I say house of YAML files) using insanely complicated technologies (like Kubernetes) with very "interesting" failure modes to say the least.
This attitude works today because of engineering-driven-development; the whole purpose of engineering is engineering and business priorities took a backseat in favor of buzzwords on the careers page and an obligatory "engineering blog" (describing how they solve self-inflicted problems), however when it comes to reliability and solving problems a large majority of projects can get away with much simpler, old-school technologies.
Almost everything in your comment is wrong. Kubernetes has enabled a whole host of observability tooling (eg opentracing), promoted a culture where app logs are easily and always accessible, enabled 0 downtime deployments for teams without dedicate infra specialists and so much more. It has made deploying reliable applications a lot more easier than ever before.
Services today scale to handle a lot more users and traffic than they did not so long ago; and these reliability guarantees are the norm rather than an exception.
Kubernetes does indeed have advantages but also brings a whole layer of complexity, overhead and moving parts. From my experience, in many cases the theoretical advantages don't end up being worth the tradeoff and/or don't even end up being implemented. Furthermore the particular things you mention (tracing, centralized logging & no-downtime deploys) can be done just as easily without Kubernetes.
I disagree about not needing dedicated infrastructure specialists. Kubernetes' complexity, learning curve and failure modes would make me uncomfortable operating without having a dedicated "devops" person (or sysadmin as we used to call them) while I am perfectly comfortable managing a few virtual machines (or even bare metal hosts) with a load-balancer in front of it. I recommend building systems in a way that can easily fit in your mind, and there's only so many abstraction layers and moving parts you can fit in there before you overload.
When it comes to scaling, not every application needs to scale and even when it needs to, it's trivial to scale stateless app servers without Kubernetes. You can scale quite far without Kubernetes, and when you're past that point you'll realize your main bottleneck is your data store and Kubernetes (or similar) can't magically solve that.
Your answer to everything I said is “actually, it’s not that hard without kubernetes”. Maybe it’s not for you. For most developers, it absolutely is. And that’s why kubernetes is popular.
Fads don’t form out of thin air. There’s always some value that they provide. To someone experienced with setting up infrastructure, the tasks may seem trivial, and the value add is low. For others who don’t, having a dead simple way of easily adding tooling around their applications is a godsend. Why is this so fucking hard for you to understand?
I am approaching this from a developer's perspective. Kubernetes and similar (even local Docker) introduces an extra layer of indirection that you often have to fight with. Sometimes it's worth it, sometimes it's not. For me, if my application is misbehaving in production, I prefer being able to just SSH into the machine and figure out what's wrong than fight with the container layer, its authentication system, command-line syntax just to obtain a shell inside the container. When I am developing locally, I prefer having all my files on the local filesystem instead of having to worry about volume redirection and "docker exec".
I am not an expert in setting up infrastructure by any means. In fact if I were I would probably use and promote these technologies. But in my opinion, adding another layer of abstraction doesn't magically solve the problems of the underlying stack (it won't protect you against obscure Linux kernel behavior, but now you have yet another moving part and potential variable which you need to account for when troubleshooting) but still gets in the way when you're trying to do something simple that doesn't even require any of the advantages the container technology is offering.
When it comes to "adding tooling around their applications", I am not sure what you mean but I will assume you refer to your previous examples, in which case I do not see how container technologies change the game at all. Tracing and centralized logging require your application to talk to a centralized log server (for logs, you can also output to stdout and have systemd/syslog collect and send them to the logging server) and container technologies don't change anything here.
I am not saying that container orchestration technologies provide no value. I am saying that they are often overkill for the task at hand and introduce extra complexity, moving parts and management overhead.
I'm an infrastructure type and everything you said rings true for me.
Kubernetes and abstractions of its ilk (shipping containers for example) have a place, but every abstraction comes with some form of trade-offs, be that performance or transparency.
Dealing with a node brown-out in kubernetes is much worse than dealing with a network, host or service outage because the troubleshooting steps involved evolve fractally.
That said, obviously there is value- but it's good to critically assess the value instead of just jumping in.
If you dislike the article, go ahead and say why. Leaving negative personal comments about the person who wrote it approximates your own comments: unconstructive, free of evidence and bitter
Given that Uber is infamous for its extreme NIH syndrome, unnecessarily reinventing the wheel and spending enormous amounts of dev effort to create internal versions of products that already exist, I think it's relevant to the article to point out that the author almost probably had a key role in that culture, and what his philosophy leads to if taken unchecked.
I haven't even read the article and hence don't really have an opinion either way, but "[not] constructive" and "free of evidence" seem like pretty specific criticisms to me.
Its safe to say the unix crowd recreated the Windows Registry. I guess it turns out the Microsoft guys weren't idiots its just a messy problem that requires a messy solution.
etcd is not a windows registry- it's more similar to active directory, but with consensus protocols to implement the multi-machine requirements for consistency.
It's mostly a rant about someone not accepting that extra performance can come at the cost of complexity.
While I could argue with all the points that he's making, my main counterpoint is this: junior devs don't care that their http/2 server uses way more "complex" code then their http/1 server, it's just a flag away (or in most cases, automatic). Senior devs worth their salt also don't care, if I design an application that is going to run on kubernetes I now know it will run in the big cloud providers and on premise without major changes. It forces you to accept that your app will die and needs to be able to run from a cold start without any issues. I can't count the number of machines I've encountered over the years that couldn't be rebooted because the maintainers monkeypatched the crap out of it while running EoL distributions, libraries and web servers.
And now I've realized I've become a geek yelling at the cloud as well.
The performance benefits are aimed squarely at the large, entrenched interests who can absorb the increased costs as a rounding error. HTTP/2 and SPDY/QUIC look, to me, to be about decreasing costs and increasing efficiency for large web hosts and erecting barriers-to-entry for competitors.
I mean, who are these newcomers we're talking about? There's basically never a reason to roll your own web server. I guess if you want to compete with Nginx, sure, the barrier of entry is a higher now? But that's an incredibly niche case.
HTTP/2 is more complex to implement from scratch (but still quite doable, even as an individual), but it has built-in support in every language & framework worth its salt now, so you don't need to do that.
If you're using an existing implementation, that's usually just as easy to do as HTTP/1.1, because they have almost exactly the same semantics (that's an explicit goal from the spec), and so most implementations have almost exactly the same API.
In practice, it's a syntax & connection management change on the wire that's mostly invisible as a developer building on top of it, plus a set of optional extra features (like Server Push) that you can use if you want or ignore if you don't.
Can't speak for QUIC/HTTP3, since I haven't touched them yet, but I'd be a surprised if that's a hugely different story.
if you wanted simple, there was redis or memcached.
it's an arbitrary cutoff to call etcd simple (it implements Raft, for fuck's sake, it implies a distributed system, multiple odd number of nodes to avoid split-brain!)
systemd is simple, fixed config format, comes with every distro, default works, has extensive documentation, compared to undocumented distro-specific init scripts written in sh (or bash, or worse).
sure, if someone just wants to hack on their 8bit toaster, then they might find a simple shell script simpler.
Heroku also forces you to accept your app will die and needs to run from a cold start. Surely there's more to this k8s thing than "it's cheaper than heroku" ?
> HTTP/2 a.k.a. SPDY is a comically bloated Layer 4/5/7 mega-combo protocol designed to replace HTTP. It takes something simple and (most importantly!) comprehensible and debuggable for junior programmers and replaces it with an insanely over-complicated system that requires tens of thousands of lines of code to implement the most minimal version of, but which slightly reduces page load time and server costs once you reach the point of doing millions of requests per second. I am filled WITH RAGE just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web
This resonates with me, and I'd like to add HTTP/2 can only bring advantages if you actually go all the way to push/bundle resources into responses and have a strategy/priority when to push eagerly vs serve lazily; I'm even much more worried about upcoming QUIC (and DoH) because there's no impl in sight.
> HTTP/2 can only bring advantages if you actually go all the way to push/bundle resources into responses
Compared to well-optimized HTTP/1 (e.g. using minified CSS and sprite-sheets), sure. Compared to most HTTP/1 deployments, though: no. HTTP/2 gives you tons of advantages "for free" that you need build-time processes to attain in HTTP/1. With HTTP/2, you can do "the naive thing" that'd you'd have done on the 1995 Web in Notepad, and it'll be the optimal thing.
Also, HTTP/2 means less OS packet-switching overhead server-side if you have ancillary connections (e.g. websockets) open against the host, since those also get muxed into the same carrier socket.
Also, mobile clients wake up less, because there's only one TCP socket to do idle-keepalive on.
HTTP/2 also means that TCP's Nagling has more to work with, and so is less likely to end up needing to waste bandwidth on emitting many undersized packets—it can just pack N short requests into the same TCP jumbo frame, since they're all going to the same place.
I would also point out an indirect "advantage": HTTP/2 makes it cheaper to serve ads proxied through the first-party host (as HTTP/2 flows) than for the client to hit the third-party ad servers directly. People can still block the ads/trackers either way, but served inline to the origin like this, the people who don't block ads, will get a better experience.
I agree these things can be useful, but again none of these come out of the box (if I haven't overlooked or misunderstood something). Including "doing the naive thing"; I mean how do you expect your web server to automatically push CSS or SVG sprites/fonts unless you're relying on the server to intercept/parse your HTML for delivery-time optimizations a la PageSpeed and make heuristic scheduling decisions? Unless you're putting in the effort to optimize your payloads you will just result in as many roundtrips with HTTP/1.1 + keep-alive.
Web browsers are limited in the number of concurrent socket connections they'll open to a given origin. This matters not-at-all in HTTP/2, since everything is going over a single socket; while mattering quite a lot in HTTP/1, since dependent resources being loaded in parallel must be loaded on separate sockets. If you only have six parallel sockets to work with, then if your page is, say, an image gallery, then the Javascript file that makes it work (loaded at the bottom of the body) might be blocked waiting behind the loading of e.g. some large image higher up in the body. The previous requests need to entirely finish (= a round trip) before the next requests on the same socket can start. Keep-alive does nothing to fix that.
HTTP/1.1 pipelining partially mitigates this, allowing the client to "queue up" a list of all the dependent resources it wants from each socket; but it suffers from head-of-line blocking. Which sounds like some arcane thing, but in practice it means that big things might block the loading of small things. (The browser doesn't know how big things are, so it can't effectively schedule them; and the server must dumbly queue results up in the same order the client requested them, because that's the only way the HTTP pipeline's implicit flow sequence counters will match up.)
HTTP/2 is a full mitigation for this problem, since—even without a heuristic "prioritization strategy" for the delivery of dependent-resource flows—the "oblivious" strategy is still a good one: if you attempt to deliver all the resources in the queue concurrently; and you do so by delivering one fixed-size chunk of each flow per iteration, in a round-robin fashion; then you'll end up finishing delivery of resources smallest-to-largest—which means you'll usually deliver the most-critical resources first, no matter where in the dependent-resource queue they started.
Or, in short:
HTTP/1.0 = O(N) required roundtrips for a page with N resources.
HTTP/1.1 with pipelining = O(1) required roundtrips for a page with N resources (followed by O(N) bytes streamed half-duplex), but the page can still take nearly the same amount of time to become interactive as if it were O(N) roundtrips, because of effectively worst-case scheduling.
HTTP/1.2 = O(1) RTTs + O(N) half-duplex bytes for N resources, loading "intelligently" such that the page becomes interactive in O(log N) time.
I really appreciate your going into these details, but I still don't understand the "oblivious" strategy thing which supposedly improves baseline performance OOTB when the criterion is the time to first render of above-the-fold content given otherwise same conditions. You say the effect of delivering all resources concurrently is that "the most-critical resources [are delivered first], no matter where in the dependent-resource queue they started". But this isn't proven; it's just a different heuristic to apply to traffic shaping assuming small resources are needed early vs assuming dependent resources (+ HTML markup itself) are loaded asynchronously in the order the browser parses an HTML DOM (the default behaviour, and the one authors can influence directly). You're not magically increasing the bandwidth by multiplexing so something has to give.
The "oblivious approach" is "just a different heuristic to apply", yes. It's just one that happens to work especially well for HTML rendering, given that large resources tend to be depended upon in a way (e.g. an img or video tag) that gives them a pre-download bounding-box size, allowing the rest of the page to render around them and to not reflow once they're loaded; while the types of resources that tend to be small (like CSS or JS files) are mostly depended on with the semantics that they can potentially reflow the page entirely when they finish loading, which means the browser completely inhibits interactivity (and/or rendering, depending on the browser) until those resources finish loading.
The only things that kind of break this heuristic are:
• large single-file Javascript SPAs. These are usually just marked with the `async` attribute, such that the initial DOM of the page can first render, then be gradually enhanced when the SPA loads. But with HTTP/2 + ES6, you can also just not pack the SPA into a single file, instead relying on ES6 modules, which will each be individually smaller and therefore will end up being delivered first by the content-oblivious round-robin chunk-delivery strategy.
• web fonts, which are large and will necessarily cause a complete reflow of the page once loaded. Currently, browsers make a special loading-precedence exception for web fonts; though it doesn't matter as much right now, as they're still mostly served from third-party CDNs rather than as first-party HTTP/2 flows.
AFAIK, for regular web-page retrieval, these are exactly the total set of things that currently benefit from being server-pushed along with the first response; everything else just gets handled well even without server-push.
(If you're curious, server-push is really designed for the use-case of pushing secondary API responses along with an initial API response; to allow for GraphQL-like "single-round-trip resource-hierarchy/-graph walking" in a way that's more friendly for caching layers than comingling the results into a single resource-representation. It makes the most sense in a Firebase-like system, where un-asked-for server-pushed resources can be obliviously accepted and dumped into the client-side in-memory set of synced entities asynchronously to the parsing of the initial response; and then, once the dependency is parsed out on the client side, the client can discover that it already has the entity it wants in its in-memory entity store, and doesn't even need to make the request.)
Apologies if I've missed some part of the explanation that answers this. The way your previous reply described it, the benefit of HTTP/2 is loading smaller things earlier. But given that goal, couldn't the browser just prioritize JS/CSS files over images/videos for the same effect, without any new protocol that sets in stone some heuristic purely based on content size?
> the benefit of HTTP/2 is loading smaller things earlier
No, the benefit of HTTP/2 is a lack of head-of-line blocking. Head-of-line blocking can be easily seen when big things block small things, but that's not what it is. What it is, is when something doesn't make progress because another thing is being waited for.
Imagine a multimedia container file-format where you can't interleave audio frames with video frames, but rather need to put the whole audio track first, or the whole video track first. This format would be unsuited to streaming, because downloading the first chunk of the file would only get you some of one track, rather than useful (if smaller) amounts of all the tracks required for playback. Note that this is true no matter which way you order the tracks within the file—whether the audio (smaller) or video (larger) track comes first, it's still blocking the progress of the other track.
HTTP/2 is like a streaming multimedia container format: it interleaves the progress of the things it loads, allowing them to be loaded concurrently.
This doesn't just mean that small things requested later can be prioritized over large things requested early (though it does mean that.) It also means that, for example, if you load N small Javascript files that each require a compute-intensive step to parse+load (GPU compute shaders, say), then you won't have to wait for the compute-heavy load process of the previous files to complete, before you begin downloading the next ones; but rather you can concurrently download, parse, and load all such script files at once. Insofar as they don't express interdependencies, this will be a highly-parallelizable process, much like serving independent HTTP requests is a highly-parallelizable process for a web server.
One benefit of HTTP/2's lack of head-of-line blocking, that would be more talked-about if we had never developed websockets, is that with HTTP/2, you get a benefit very much like websockets, just using regular HTTP primitives. You can request a Server-Sent Events (SSE) stream as one flow muxed into your HTTP/2 connection, and receive timely updates on it, no matter what else is being muxed into the connection at the same time. Together with the ability to make normal API requests as other flows over the same connection, this does everything most people want websockets for. So the use-case where websockets are the best solution shrinks dramatically (down to when you need a time-linearized, stateful, connection-oriented protocol over HTTP.)
> new protocol that sets in stone some heuristic
Note that there's actually no explicit specification of the order in which HTTP/{2,3} flows should be delivered. What I'm calling "content-oblivious round-robin chunk scheduling" is just the simplest-to-implement strategy that could possibly meet HTTP/2's non-head-of-line-blocking semantics (and so likely the strategy used by many web servers, save for the ones that have been highly-optimized at this layer.) But both clients and servers are free to schedule the chunks of HTTP flows onto the socket however they like. (They can even impose a flow concurrency cap, simulating browsers' HTTP/1.1 connection limit and starving flows of progress. It'd make the client/server a non-conformant HTTP/{2,3} server, but it'd still work, as what progress "should" be being made is unknowable to the peer.)
It's a bit like saying an OS or VM has a "soft real-time guarantee" for processes. Exactly how does the OS scheduler choose what process will run next on each core? Doesn't really matter. It only matters that processes don't break their "SLA" in terms of how long they go without being scheduled.
The advantages listed above all come from the fact that HTTP/2 runs all traffic over a single multiplexed TCP connection, rather than HTTP/1.1, where clients are forced to open many independent TCP connections (because each is a single channel where responses must be returned sequentially, in the order they were requested, and so block one another).
That happens totally automatically, there's nothing special you need to do to enable that.
There are also other potential advantages once you look at push etc, but using TCP as it was meant to be used really does give you many advantages for free.
That's not true at all. HTTP/2 is hugely impactful without using push at all—multiplexing beyond 4/8 streams enables radically different, performant bundling strategies, all "for free" with zero configuration, just by the act of using the protocol.
it shoves everything down a single TCP connection. This means that if you loose a packet (and 4g is lossy) it stalls the _entire_ queue.
Instead of addressing the main complaint (that multiplexing everything down a single TCP connection is a fundamentally stupid idea) they come up with an entirely _new_ protocol.
It also fails to understand what HTTP2 has turned into: a file transport protocol, with some RPC built in.
So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
To be clear, it's not like there are any HTTP/2-only servers in the world. HTTP/2 is something a client connection upgrades to. One might think of HTTP/2 and HTTP/3 not precisely as versions of HTTP, but rather as variants—HTTP/2 is "optimized binary HTTP, fixed-station profile" and HTTP/3 is "optimized binary HTTP, mobile-station profile."
(Mind you, they are versions relative to each-other, because HTTP/3 is strictly better at doing what HTTP/2 does than HTTP/2 is, so there's no reason to use HTTP/2 if you have HTTP/3.)
But as I was saying: nobody forces mobile devices to use HTTP/2; and by-and-large, they don't. HTTP/1.1 still has its use-cases; neither HTTP/2 nor HTTP/3 was designed to obviate HTTP/1.1's entire set of use-cases.
You know how SSH has sub-protocols, e.g. SFTP? HTTP/2 and HTTP/3 are to HTTP/1.1, as SFTP is to SSH. It's a mode you can put the connection into, that will optimize it for a certain profile of use-cases. No more, no less.
(You know what else is a mode you can put HTTP/1.1 into? Websockets! Again, a complementary use-case.)
> So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
If you think you're so smart, write an RFC (or even just, build a reference server) that's competitive in production performance for its use-cases over similar links.
There comes a point when explicit OSI-style encapsulating layering becomes an active hindrance; and for the use-case of "a server serving a million concurrent requests"—the FAANG-scale problem—they passed that point long ago.
Yes, we mere mortal client users and backend developers might not find such use-cases relevant; but take the perspective of e.g. an internet backbone ops engineer. By volume, most of the traffic going around the Internet goes to these sites. Optimizing just the traffic going to these sites—with a special protocol clients only use when speaking to these sites—makes the whole Internet faster, by reducing in size the huge blob of bandwidth-contention the requests to these few sites create.
Also, a design hint, if you actually want to try to build something competitive to HTTP/3: most of the problem being solved by HTTP/3 is that certain things are known to be optimal, but despite that, nobody can just force their corporate overlords to mandate a switch to those things right away. So HTTP/3 needs to be optimized for the case where you do the most fundamentally-performant things (e.g. using zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access); and also for the cases where you do less fundamentally-performant things (like generating and streaming a JSONL or CSV stream line-by-line as the output rows gets computed by an in-process dataflow engine.) One might call HTTP/3 an arbitrary custom protocol, that happens to have semantics such that it can be downgraded to HTTP/1.1 by a link-to-link proxy. And that is a hard constraint to optimize under.
well you do not upgrade your http/1.1 connection. http/2 works by using ALPN on TLS connections, that actually try to get the right protocol.
yeah there is h2c, which does upgrade the connection, but that only works in backend scenarios without tls.
I think if you're making this trade-off and "optimizing just the traffic going to these [big] sites—with a special protocol clients only use when speaking to these sites—makes the whole Internet faster", then I guess you shouldn't be surprised to be criticized by the majority (according to your own words) not having these problems, yet having to shoulder additional complexity for little or no benefit, and even only serving to increase asymetry on the web. Why can't the "big sites" (it's just Google who's behind QUIC anyways, isn't it?) not then create their own network and take their enormous traffic there? The web was created for easy self-publishing after all.
> yet having to shoulder additional complexity for little or no benefit
They don't, though? HTTP/2 and HTTP/3 are voluntary to all parties concerned; whether you're a client, a server, or a gateway, if you don't speak those protocols but choose to just speak HTTP/1.1, then it's the peer that has to cope with that, not you.
(There isn't even any fancy forward-compatibility needed in TLS to support the semantics of the ALPN extension. If you just use an old TLS library that ignores the unknown extension data in the TLS stream, the other side will continue on assuming you didn't understand the question, and therefore aren't an HTTP/{2,3} server.)
HTTP/{2,3} are like a local language of a culture, spoken by immigrants when they run into other immigrants from the same culture. If either party is not an immigrant from that culture, it just doesn't come up.
> Why can't the "big sites" not then create their own network and take their enormous traffic there?
That's called circuit switching (i.e. the thing telco and cable services do that's not the Internet), and it's the thing the packet-switched Internet effectively obsoleted. From an Internet engineer's perspective, if you have two data streams, it's strictly better engineering to just feed them into a switch that linearizes those packets onto a high-bandwidth line "as they come" (and upgrade the bandwidth of the switch+line as needed, so that no signal is ever starved of line-time), than to try to time-divide or frequency-divide the pipe; let alone to keep those packets isolated on two separate networks of pipes. Then you'd need to maintain two networks of pipes! (And the people working at FAANG are still fundamentally Internet engineers who believe in Internet principles, rather than telecom principles.)
But besides that, how would that network be delivered into people's homes? Unless you're proposing that these services take the form of their own additional cable going into your house/SIM in your phone, this network has to merge into the regular Internet somewhere. And it's exactly at that point when that traffic once again contends with the rest of the traffic on the Internet. Even if it's only on the last mile, it's still getting in the way.
> it's just Google who's behind QUIC anyways, isn't it?
SPDY and QUIC are the names of "prototype standards" developed by Google. HTTP/2 and HTTP/3 are standards inspired by SPDY and QUIC, developed by HTTPWG, with Google as just one participant in that conversation.
The other backers of the standard are, of course, the groups whose interests are aligned behind having more-efficient HTTP: carriers, bigcorps, switch/NAT/WAF hardware manufacturers, cellular ISPs, etc.
But I see your deeper point—you're saying that this is all Google's solution to Google's problem, so shouldn't the onus be on Google to solve every downstream problem as well?
Well, it is and it isn't. Google is solving this problem for us right now, but it's not a Google-exclusive problem. TCP was created by DARPA, but maintaining a consistent stream over packet loss/reordering is not a DARPA-specific problem. They just happened to be the first group to need a solution for that problem.
The reason HTTP/2 and HTTP/3 are public standards, rather than things going on secretly only between Google Chrome and Google's backend servers, is that other parties see value in them—not just present value to themselves, but also future value.
New big uses of internet bandwidth arise every day. Netflix started sucking up half the Internet ten years ago, and it's already dropped down to less than 15% because other even larger use-cases have eclipsed it.
HTTP/2 and HTTP/3 are engineered to allow small businesses a path to grow into the ...
> HTTP/2 is something a client connection upgrades to.
to which I as a normal person on a normal phone has no control.
> Also, a design hint, if you actually want to try to build something competitive to HTTP/3
I'd need to control a major browser's code base.
Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
> perspective of e.g. an internet backbone ops engineer.
They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
> If you think you're so smart, write an RFC
I already work for a FAANG during the day, I'm not going to put free hours in at night. And I'm certainly not going to get involved in cross company politics.
> a point when explicit OSI-style encapsulating layering
OSI has never been correct, its a fib told to students to make them think there is any kind of order past tcp/udp. I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms. yes its very neat, but
> zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access
from the protocol level, its not actually that much different, unless you are doing some sort of multiplexing down a single TCP pipe.... now, if you trying to balance throughput and latency on the same connection, thats can be really tricky. An answer is not to use the same connection. You do what FASP does which is use UDP for bulk and TCP for control/accounting.
> like generating and streaming a JSONL
you can easily use TCP at one packet a minute, or maxing out a 10 gig link. (I know, I've written a FASP replacement.)
> semantics such that it can be downgraded to HTTP/1.1
you can downgrade 10gig-t to 10m/s that has no bearing on its performance, unless you terminate it badly. Its just a preamble, then for the life of the connection, an utter irrelevance.
> you can downgrade 10gig-t to 10m/s that has no bearing on its performance, unless you terminate it badly. Its just a preamble, then for the life of the connection, an utter irrelevance.
You misinterpreted "downgrade" here (though maybe I chose a bad word for this.) I didn't mean that the proxy itself can force both sides to speak HTTP/1.1 to it. I meant that a proxy can be speaking HTTP/{2,3} to one side, while speaking HTTP/1.1 to the other, by translating ("downgrading") HTTP/{2,3} messages into their equivalent HTTP/1.1 messages. This is the reason that HTTP/{2,3} are HTTP in a real sense: the messages that flow over them still interoperate with the rest of the HTTP ecosystem, through bilingual gateways. The messages are a different bijective encoding of the same HTTP application-layer message-flow semantics.
(Nginx, by default, is such a gateway: it speaks HTTP/2 to clients if they offer, but speaks HTTP/1.1 to upstream backends even if they know how to speak HTTP/2. This is because HTTP/2 is useful to carry flows over the Internet, but has few advantages within a low-contention low-packet-loss LAN.)
This also has other implications, like the fact that browsers and caching proxies don't need to change the way they store cached HTTP resources, just because the resources come in over HTTP/{2,3}. They can just transcribe the messages to disk as if they had been the equivalent HTTP/1.1 messages.
> Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
The point is to take a stupid thing that people will keep on doing no matter what (for better or for worse), and help them to do it as fast (or as un-slow) as possible, without trying to make it non-stupid in the process (because that won't be the same thing any more, and so people won't use your smart thing, but will continue doing the stupid thing.)
You know Formula 1 racing? In F1, there are rules against doing most things that could make cars go fast. The point of the competition is to make the cars go fast anyway, without breaking those rules. If you make the car go as fast as it can, it's not an F1 car any more. To win, you've gotta make the fastest F1 car, not the fastest car.
That's HTTP/{2,3}: the cars are the flows, and the rules are HTTP's must-be-all-things-to-all-people semantics and manifold optimized-for usecases. The goal of adding HTTP extensions/versions/etc. is to make your server (or client) talk to clients (or servers) as efficiently as possible, without breaking any random peer from 1996 speaking to you through a bilingual gateway. If you did break those peers, you might have the fastest protocol, but you wouldn't have the fastest HTTP protocol.
> They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
That's an infinite regress, as the people who you're handing the problem off to are also Internet backbone engineers. If your company owns 30% of the Internet's traffic, your leaf-node AS is part of the backbone, and that AS's problems are the Internet backbone's problems. Just like the major universities were part of the backbone until ~1990.
Though, keep in mind, this isn't about fancy application-layer-aware routing or anything. HTTP/{2,3} have bandwidth savings, and that savings is passed directly onto the Internet backbone those messages travel along, in the form of more headroom for other stuff. It's not about needing to pay ...
It took me a bit to understand what you were saying here:
> I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms.
...but I think I got it now. I was arguing with the wrong thing here.
Yes, there's a reason that FTP and RTP do what they do. (Though in FTP's case it had more to do with TCP-port parity gender; multiple data sockets per control socket was just a convenient outgrowth of the existing architecture.)
But crucially, HTTP/{2,3} are about accelerating existing traffic; and existing HTTP "apps" (the HTML + JS + CSS that contain an implicit dependent-resource digraph that gets turned into flow-requests by the browser) have no indication in them for what's going to be a "control-plane" request vs. a "data-plane" request. HTTP itself is an oblivious carrier re: these use-cases. So HTTP/{2,3} must continue that trend, and be oblivious carriers as well. To do otherwise would force a rewrite of all HTTP "apps" to add such indications.
However, in HTTP/1.1, you got the benefit of this purpose-isolated signalling "for free" anyway, because you'd probably be making your data requests from a CDN origin, while making control requests to an API-subdomain origin. Those are separate origins, so they got separate (keepalive, pipelined) TCP sockets.
In HTTP/{2,3}, you can choose to shove everything into one socket by putting both your control-plane and your data-plane into the same origin.
But, crucially, you don't have to. Nobody's stopping your HTTP "app" from just having separate control and data origins; and so ending up with two HTTP/{2,3} sockets open, one for control signals, the other for BLOB data.
And, of course, existing sites previously engineered for HTTP/1.1 won't shove everything into a single socket, since they've already separated the control origin from the data origin. (They've likely got several data origins, so they'll likely want to re-engineer with upstream proxying to compact those down into one data origin; but all they have to do while doing this is to choose not to also compact the control origin in there as well. The choice to not change anything is, in this case, the right choice!)
And also keep in mind that most websites aren't, like, photographer's portfolios or direct video hosts. There are few websites directly embedding same-origin BLOBs. There are many websites embedding "Binary Not-so-large OBjects", like blogs that embed same-origin 500px-wide medium-quality optimized-JPEG images. And those websites don't have such a large disparity between control signalling and data signalling that they have any need to separate them—those images aren't much larger than the average compressed-JSON XHR response!
And because of that, the "default" advice for HTTP/{2,3} transitioning (that you should follow if you're 99% of websites) can just be "put everything on one origin"; whereas "split control signals to a different origin, thereby forcing it to a separate socket with separate TCP window scaling parameters" can just be one of those arcane PageSpeed things you learn from an infrastructure consultant, a bit like choosing what type of select(2)-alike kernel ABI your webserver should use to receive packets.)
>but will completely cut out future generations from system programming for the web
There's so many things where my competence is based on understanding the simple preceding system plus a good intuition of how the class of systems tend to be bungled up over the years. We don't pay down the accumulated complexity, and newcomers don't get the same luxury I do.
It really becomes noticeable with highschoolers and I have to chose between teaching useless but illustrative groundwork, or just explaining and expecting a combination of taking it on faith and memorization.
The misconception is the idea that anyone can easily implement a compliant HTTP 1.1 server, or even client. The protocol may appear simple on paper but parsing it correctly and securely is quite tricky. IMO, one of the benefits of HTTP 2 and 3 is they force people to use libraries and remove the illusion that interacting with HTTP bytes on the wire is a reasonable thing to do.
> IMO, one of the benefits of HTTP 2 and 3 is they force people to use libraries
What a strange comment. "The benefit of this protocol is that it's so complex that nobody understands it, so they have to use an existing implementation."
What a weird slippery slope argument. There is a world of difference between "so complex that nobody understands it" and "I can telnet to port 80 and fetch a webpage using keyboard and fingers"
That's pretty much in the "cut out future generations from system programming for the web" territory, and complexity for the sake of complexity. Also, you have to implement HTTP/2 in addition to HTTP/1.1.
It's a big jump to go from "this protocol is more complicated than the previous one" to "cut out future generations from system programming". It's not _that much_ more complicated that a competent programmer who's familiar with sockets and bytes can't ever accomplish it.
As discussed at length in another sub-thread here, the complexity solves real problems, it's not for its own sake. But if you're not convinced of that, nothing I say here is going to change that.
·
If you've worked with me, you'd know that I take simplicity as a virtue to an extent that's almost detrimental to getting stuff done. I despise complexity for its own sake. I wish as much as anyone that we could start over and redesign IP, TCP, TLS, HTTP, HTML, JS, and a few more things, with all our accumulated knowledge, into a simple and clean application and content distribution system that's easy to understand and observe. But path dependence :( We have to work with what we've got.
It's the picking out of HTTP 1.1 as some sort of golden age of clean and simple and observable that bothers me. It isn't.
The value of being able to build an HTTP 1.1 server is not in putting a hand-crafted server on the internet and serving traffic. It's in the experience of building the server and later applying that detailed byte-by-byte knowledge in real work. It's also in teaching young engineers that yes, they can learn something by digging into the layers that surround their own code. Part of the dysfunction of "enterprise" programming is the programmed helplessness of people who are surrounded on all sides by complex, opaque technology that turns curiosity into a liability. We are replacing a "whoa, I can see how it works!" experience with a "fuck, I guess have to leave this part to those bigcorp engineers" experience.
(I'm not agreeing with the author, just pointing out why he might care about the ability for an individual to implement the protocols the web is built on.)
I agree it's incredibly important to dig into layers under (and over) the one you're working at! I'm very grateful for the experience of building a simple unix-like kernel and a simplified TCP & IP implementation as school projects. And I'd encourage anyone to do the same with HTTP 1 or 2 or 3. The difference is that no one thinks that their educational/toy TCP implementation is suitable for production use, while people do seem to (mistakenly) think that about HTTP.
What I don't get with the complaint is that HTTP/1 still works fine. If you are hosting (or writing) your own server, you can stick with that and be fine. I have heard ZERO talk about browsers deprecating HTTP/1 support.
For the companies implementing http/2, those improvements DO matter. They have different needs.
Author is really good at discrediting themselves straight off the bat in one sentence:
> for a ~~bullshit~~ unsuccessful project called CoreOS Container Linux that was EOL'd several years ago
CoreOS was actually quite successful to me as an outside observer. It had decent paid user base as well as people using it without paying. It has showed people that Chrome OS can be used to build atomically updating host OS with a readonly fs for container hosting. It has inspired many other container hosts to come like RancherOS and Google’s Container-Optimized OS. Similarly, it was probably one of the reasons why Red Hat was interested in the acquisition.
Furthermore, CoreOS was EOL'ed last month; not several years ago.
I can keep going on why etcd API was switched to gRPC and what benefits this offers, even at small scale. But I don't think it's worth anyone’s time convincing the author otherwise. Based on their tone, it's clear to me that they have trouble with using software when things get a tad bit complicated. Usually, there's community decision-making behind these decisions, and they're often deliberated for months, backed with prototypes and data. I'm pretty sure the author doesn't care, however.
My team ran about 50% of a Fortune 500 software company on CoreOS up until a few months ago when we migrated to Flatcar. We even paid a lucrative support contract while we ran it. I can count the number of major OS issues we had with it over several years on one hand- and we've had even better success with Flatcar so far.
Calling CoreOS unsuccessful is a massive misunderstanding of the market.
Let’s keep this chain going: Your viewpoint on this company using it is what shows a massive misunderstanding of the market.
Clearly, people are using it successfully and enjoying it (or other similar products). Just because you don’t like it it doesn’t change the reality of the situation.
1. I repeated an argument made in the article itself. That cannot be considered flamebait.
2. I've always expressed strong opinions in my posts. Sometimes this results in multiple upvotes, sometimes in multiple downvotes.
3. About being "unsubstantive" - I underscored the main concept/rationale for Core Linux, and linked to its Wikipedia page. You could argue that it's not substantive _enough_, but when you phrase it this way it's as though you merely dislike the fact that I repeated the inflammatory term from the article.
Chris from Kinvolk here. Happy to see you're having success with Flatcar.
We, of course, agree that CoreOS Container Linux was a huge success. The uptake that we've seen in Flatcar usage, especially since the CoreOS EOL date on May 26th, has been extraordinary. So from what we a can see, the market is there for a minimal Linux for containers and and we're happy to continue filling that need with Flatcar.
I'm not exactly sure what you're asking. But Flatcar Container Linux is completely open source. In fact, everything we do at Kinvolk is. We do not build open-core products.
That sounds really interesting!
I currently work on a side-project which is a distributed application and I plan on incorporating kubernetes/cloud-native into some of the core concepts, so I will check it out once I'm at this part of the implementation!
Could you share some info about the switch to gRPC, not for the authors benefit, but for the rest of HN? As someone not familiar with this story, I’d be very curious for your insights!
I want to make constructive comments about this blog post, but the way it conducts itself makes doing so extremely difficult because nothing about it is constructive. This is troublesome for HN - I almost wish I could flag it for being impossible to discuss without turning into a flame war. It feels like several disjointed complaints based on a desire to jam any and all inconveniences into a unified narrative about Modern Software and Google. Dismissing CoreOS is the opener, but there’s almost nothing on the page that doesn’t wave red flags in some way, and I’m trying desperately to be open and objective.
It is essentially a rant with not much substance. If this was a comment on HN it would be flagged probably. Like to hit at so many things in 3-4 paragraphs you aren't making an argument for or against something, you are just making a scattershot blast on like most of systems stuff today.
I have worked with (and under) big names in system software and so many of them think saying "terrible", "garbage", and "trash" will motivate people to do productive corrections or improvements. Nope.
Times have changed and the very people he criticizes are infinitely better at branding and inspiration. There's a reason they're the ones gaining traction.
Like you, I want to be constructive, and I do agree with the author's core complaints. He just doesn't realize he's actively self-sabotaging and harming his own reputation.
Many reasons to switch from JSON to gRPC: * gRPC uses HTTP/2 which means you can concurrently make multiple requests on a single TCP connection, while on a typical JSON API which probably uses http/1.1, you can't. * Type safety of protobuf types. * Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane. * Client libraries are automatically generated, and not error prone. * RPCs are already optimized for bytes on the wire efficiency, whereas JSON is not. This matters a great deal as Kubernetes objects get large in size/quantity over time but controllers still work effectively by not spending too much CPU on encoding/decoding like they do on JSON. * Similar to the previous point, most json decoders don't reuse objects, so every decoded object is a new alloc, whereas gRPC can Reset() and reuse the same object while decoding/encoding. * Builtin authentication primitives (such as JWT/tokens or even adding TLS to client and/or server). * gRPC has support for interceptors which are like middleware functions you can inject to requests/responses on both client and server-side for logging, authorizing etc.
The list goes on, but something to note is that etcd was not developed by Google (and as far as I know, not by ex-Googlers). Both etcd and gRPC are owned by the same open source foundation, so it's natural that they make use of an existing technology.
> gRPC uses HTTP/2 ... while on a typical JSON API which probably uses http/1.1, you can't.
What are you actually talking about? An HTTP API by itself doesn't actually care about which http version is used, that's something only the HTTP server that serves the API should care about (nowadays, pretty much any http server in any language supports HTTP/2 and 1.1).
> Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane.
Even HTTP/1.1 has mechanisms for that, like long polling and chunk encoding (which allows infinite streams to keep a dual channel of communication open where each chunk can be treated as a message - with "headers" and all).
> Client libraries are automatically generated, and not error prone.
This is true, but we've had similar technology since the SOAP days... you could easily do the same with a XML-based HTTP API.
By the way, most of your points are against JSON, not a HTTP API per se, which usually would allow a number of formats , including XML and JSON at least.
> Builtin authentication primitives...
HTTP also has that when you include cookies and something like OAuth/OpenID.
> gRPC has support for interceptors which are like middleware functions
This kind of thing is better done by using the specific platform you're running on (depends on the language) so I don't see something like this as something desirable on the specific RPC framework you're using. With HTTP, caching, logging etc are trivial to do and one of the strongest advantages of using HTTP in the first place, so I think you're confusingly making a point for HTTP here, unless your focus is on the RPC side of things? In which case, you would have to make a case on why RPC is a better fit than HTTP for etcd, which you haven't.
So all in all, I found your points utterly unconvincing, but presented with so much conviction that you're actually right that I could not resist to respond (even if I don't want at all to get into a pointless discussion on the merits of HTTP VS RPC or JSON VS Protobuffers)!
By using gRPC, we get much more without thinking too much. Also, bi-directional streaming is instantaneous and it is not like afterthought long polling.
It is pretty straightforward and not too hard to understand the API. Designing APIs with protobuf makes things convenient and it brings lots of already written tools with it.
Since gRPC built on top of HTTP/2, we thought gRPC as easiest and performant way of writing a HTTP API with good defaults.
"we get much more without thinking too much" is precisely one of the reasons for the original push to REST (and against RPC at that time SOAP and CORBA and RMI etc.) was made in our industry 10-15 years ago. By falsely representing a _remote_ resource as if it was a _local_ one we open a whole can of worms; not just performance, but resilience, infrastructure issues, etc. Transferring documents over HTTP with a common language of HTTP's verbs was supposed to get programmers to model their applications as the sets of resources that they are, discourage them from making excessive round trips, make debugging easier, and make use of standard HTTP load balancing hardware.
Yes it's easy to make that RPC call. But should you?
I've been off doing other things in the intervening period, but while I had my back turned the industry seems to have turned its back on REST and gone whole hog on RPC. Again.
At first I was thinking this was just internally here at Google, where protobufs and gRPC reign supreme. But it seems to have taken hold everywhere.
What did I miss? Why have we swung this way. Again. Is the pendulum going to go back?
> By falsely representing a _remote_ resource as if it was a _local_ one we open a whole can of worms
I've heard this argument before, but how does gRPC itself cause this issue to manifest? I'm curious to hear what your opinions are on a better alternative, and how not to represent a remote resource as a local one.
But the fact that it presents the remote resource in an API which resembles a local object means that programmers often get lazy in the manner of which they think about these things. The REST semantic is supposed to make this more explicit.
A remote object is not an object in your program or even your computer. It's something you're taking from something that is computationally miles and miles away. Compared to the microseconds it takes to dispatch a local call, it's an eternity away, even on a local network of the highest speed.
Accumulate those latencies over thousands of dispatches, and trouble can ensue.
I am reminded of an observation from when I first joined Google, coming out of their acquisition of the scrappy awesome ads company I worked at (Admeld).
We had a little service that kept track of ad impression caps / budgeting. We didn't want to serve an ad a single time more than the customer wanted us to, etc. Serving many thousands of ads per second, a process distributed across multiple machines in multiple data centres, this is a bit of a tricky shared state problem. The people who came before me had designed a rather clever solution which used a form of backoff to trickle down the number of ads served as they got closer and closer to budget cap, and to synchronize this state across clusters (this was before there were Rafty services to make this kind of thing easier, BTW).
We had a bit of a show and tell with Google when we first joined. I wanted to know how they were handling this problem, since their scale was many times ours, so I asked the question and got a puzzled/annoyed look:
"Oh we just make an RPC call to our budgeting server."
Summary: if you're at Google you don't have to worry as much about these problems. You still do, but there's an insane amount of infrastructure and horsepower and an army of SREs to help make it happen.
So, yeah, my point is -- just because Google does something or has invented something doesn't mean it's the best way to do it, especially in a smaller more cost conscious organization.
Thank you for taking the time to write this up, and the interesting anecdote.
What I don't see though, is how is making a (g)RPC call any different from making a REST call? Like you said, the REST call is supposedly more explicit, but at the end of the day, it seems more of a convention and some hard underlying difference. What's the difference between `httpClient.get("...")` and `grpcClient.foo(...)`?
The latter encourages you to model things as remote procedure calls, the former, well, in my experience it's open to incompetence and abuse, too, so... ehh... but done properly... well, go read the Roy Fielding paper :-)
I mean, internally at Google we have load balancing for grpc (I'm sure the outside world does now, too, but it was new to me when I joined) -- but load balancing HTTP requests containing readable JSON or XML documents, that's far more sysadmin friendly, wouldn't you say? Off the shelf infrastructure, nothing exotic, easier to monitor. Same goes for caching, for proxies, etc.
Being able to just stick a URL for a given resource in your browser, or hit it with wget/curl to read it, that's a serious bonus.
In general URLs follow conventions similar to those down by our Unix forefathers, when they designed the filesystem API. We are all familiar with this model. And in some ways REST done right is very Unix philosophy -- provide a consistent model upon which a bunch of little tools can interoperate.
I could go on... have to go put my daughter to bed tho
I don't think that's accurate... as the article claims (and it seems believable) it has only appeared in etcd because of Xooglers interfering with the project in the name of Kubernetes (also from Google). Or do you actually have examples of gRPC being used in many other non-Google-related projects?
Entirely agreed, CoreOS Container Linux was a great project that served its purpose well. Yes it went EOL last month and we just moved to one its successors, Fedora CoreOS, also works well. The authors tone is incredibly off putting.
Yeh I don't even know what I'm doing and kept a 3 node cluster running with 100% uptime of small/jokey internal apps for 2 years and everything up-to-date. Replaced one node due to failure and often pull the power from one at random as a party trick.
Curious why the author declares Container Linux bullshit/unsuccessful? I thought it was a wonderful project that got better over its (brief) lifetime. The active/passive upgrade was an absolute blessing. If the declaration is it was unsuccessful because it no longer exists, Flatcar and Fedora CoreOS are pretty straightforward successors. fcos is basically functionally equivalent if you don't use rpm-ostree layers and if you use podman in place of any rkt containers. If the complaint is "systemd," that's fine—it's not the OS for you—but I don't think that makes it bullshit.
I didn't like having to move cl machines to fcos but it really wasn't that bad and I still get coordinated, active/passive upgrades. ¯\_(ツ)_/¯
I run k8s in production and I think this is a really bizarre axe to grind that sort of smells like someone who got upset by how steep the kubernetes learning curve is. Which, in a way, is understandable.
> 1) Add hundreds of new failure modes to your software
In my entirely anecdotal experience, it removes error modes. It turns out that just because k8s offers a feature (it offers many!) doesn't mean you're required to use it.
> 2) Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML
Portability, put another way, is a requirement to maintain n different deployment mechanisms because you have no meaningful control over the system on which your software is deployed. This is unavoidable depending on what it is you're deploying and where, but for those of us who have the ability to make those decisions the old notion of portability described by automake and others is actually a bad contract to agree to.
That said, the YAML but is horrible and should be abolished. This is a great criticism.
> 3) Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking
The author isn't considering that I am not, and never will be, Google's entire SRE team. I consider it a general win to be able to apply some subset of their encoded knowledge to the problem of keeping my services running with minimal downtime even if you pick up some k8s-specific baggage along the way.
You know i actually agree with you but, as a manager of other engineers, people never stop complaining about having to edit large amounts of YAML that describe complex objects. At a certain point im forced to acknowledge that its just not ergonomic. Now, I will say that perhaps whats actually happening is the objects that the describe are so complex and numerous that the problem is actually that you need better tools, not that YAML as a format is bad. YAML as used to configure, say Open API specifications, is something people always make my problem. I don't really know why this is because I don't have an issue hacking this stuff up in VIM but people really just don't seem to like it.
Of all things I would definitely not call it easy to read and write. It has super-weak (see this for examples: https://github.com/cblp/yaml-sucks) dynamic typing, which coupled with requiring awful templating to get anything non-trivial done it has been the cause of many outages.
For real world Kubernetes needs to be addressed I'd argue we need Dhall or something with Dhall-like semantics and goals to have first class support.
YAML can be validated using JSON Schema/OpenAPI, and somebody does maintain Kubernetes schema definitions, including Kustomize, which can then be used in an IDE for live validation. For example, the Kubernetes extension for VSCode does this.
If you use Helm, writing and validating schemas for Helm value files is trickier, and not something I believe anyone is doing. It should be easier to do. Ideally you should simply put this at the top of a value file:
$schema: ./schema.yaml
But from what I know, there's no editor/extension, that supports validating through such a declaration. Secondly, Helm doesn't have a way to statically analyze value access. Something like {{ .Values.missTypedKey }} will only be caught when you generate the manifests.
Mostly that it's a single projection of an underlying data model. Where the projection is a snug fit, great. But typically folks want sets of things and suddenly it's a bit chatty.
Nothing at all. Until you start 'templateizing' it (looking at you, Helm).
It's definitely better than JSON (it can even have comments, imagine that). Most people don't bother reading the spec or even examples though, and don't realize how good it actually is.
In the Kubernetes and Ansible/Salt world it's frequently templatized, and then you need to worry about indentation that you place or your template engine places. Every day I pray that Helm will incorporate support for jsonnet so I never need to touch YAML again.
Why not parse the YAML- manipulate it like you would any other data structure in your language of choice, and then generate the YAML from there?
I don't understand why you would want to directly edit markup language without parsing it. YAML's goal is to represent a data structure so that a human can interact with it. It accomplishes that goal well. If you want to edit that data structure programmatically- no markup language makes sense to use, represent the data structure using your programming language's native representations.
I made a script for expressing similar deployments (ie homepage-en, homepage-de) in a variety of contexts (test, prod) and their permutations in a single, succinct json file.
I agree with, and have felt the same pain as you here. But this isn't YAML's fault, IMO, it is that the templaters, like Helm, aren't safe, and don't escape their outputs.
If have a string, and I use that in the middle of a YAML template, I expect a string — I don't expect "300" to become (integer) 300; give me a function for that conversion.
Any HTML templating system that behaved this way would be frowned upon due to the potential for XSS. There's little risk of XSS here, but it's still annoying to get the wrong results.
The features intended to make it easy to use, in the end lead to surprising behavior, often with values having non-string types where you expected strings. There are some examples in https://www.arp242.net/yaml-config.html.
Your position here seems to be that YAML is a reasonable configuration language. It kinda is.
But that's not how it's used in Kubernetes. It's used as a declarative DSL that you use to specify your deployments, and it is HORRIFIC. Extremely nested things where the equivalent actual code to do that would be shorter. Things that are inexpressible. Backwards-incompatible changes to your YAML that actually make it impossible to deploy changes that bring your YAML up to spec if your cluster's master node upgraded first.
YAML gives nothing here. jsonnet really isn't a drastic improvement, IMO, except that it lets you express some repetitive parts without having to figure out how many times a template engine is going to indent your YAML.
YAML actually has one of the worst default behaviors I've seen: automatic conversion of certain strings to bools.
This wouldn't be a problem if YAML required that all strings be quoted...but it doesn't, and there's no "strict" mode to force quotes, so users are extremely likely to end up doing something that behaves unexpectedly.
If you're lucky, you'll spot the problem quickly instead of wasting half a day on it.
YAML is easy to read and write in tutorial documents. It becomes a complete mess in real life and whitespace based syntax is horrendous to debug. We have had endless heartaches due to minor yaml issues and today employ tooling to minimize the amount of yaml engineers write to avoid these issues.
It's always worth noting that k8s isn't designed to give outsiders the power of Borg the way Google SREs have the power of Borg.
it's designed to give outsiders something better, based on learned experience of where the toil and cocked-foot-guns are working with Borg. Borg is, by modern standards, extremely legacy infrastructure and even Google is trying to replace it with something better internally.
the problem is that k8s is a half arsed implementation of borg/tupperware.
Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.)
The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale.
Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.
> The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.)
Hmm, I thought you could specify larger node cidr at cluster creation time so you can have >254 ip addresses on each node? The default is /24 but you should be able to specify a larger block, unless you're on GKE (not sure if you can do that on GKE).
The network provides simple and easily controllable ip allocation. You don't need to do any of it, just offload it to AWS/GC/Azure.
Slam another network card/VLAN, give it a seperate ip range from the control network, and everything is handled for you, quickly, securely and efficiently.
He is partially right because i think containers or vagrant are good tools for development in 80% of all cases...but on the other hand i like having monolith's in testing and production...so there's that ;)
As far as I can tell, the only thing this article is saying is that the author doesn't like gRPC, preferring hand-rolled APIs. (The rest of the rant doesn't really talk enough about the problems to respond to. The author doesn't like Kubernetes. The author doesn't like systemd. The author doesn't like software-defined networking. No reason is given as to why, so there is really no way to have a constructive conversation about it.)
Hand rolled APIs are easier to understand, but harder to maintain. It's great if you're only ever going to have one client, but once you need more than one, it's sure tedious to write and rewrite it for every language you want to support. Using gRPC means that you can auto-generate the client, and while they might not be as wonderful as writing each one of them by hand, at least you can get a client for whatever language you're using. And, the clients all behave the same way -- trying to figure out how to add interceptors to every bespoke client you need is quite tedious. (Look at how long it took AWS to get contexts in go, or how hard it is to add OpenTelemetry to the random hand-rolled HTTP client, etc. With gRPC, you just do those things once!)
Using protos as the transport layer lets you make backwards-compatible changes smoothly; adding fields is safe, renaming fields is safe, etc. The same is not true of using JSON -- if you call something "foo", you can't just one day rename it to "bar". Clients won't know what "bar" is. So you have to update clients and servers at the same time, and you can never "make before break". You see this all the time when someone rolls out a client/server update for a browser app -- your browser cached the Javascript, and it can't talk to the server anymore until that cache expires. It's nasty. I don't understand why people do that to themselves.
gRPC also adds defined semantics for TCP connection length; with HTTP/1.1, maybe you can reuse your connection, maybe you can't, it depends. You can't have multiple requests in flight on the same connection, even if you can reuse it. HTTP/2 fixes this, but gRPC has first-class channels and behavior is well-understood for request/response, streams, etc.
It is unfamiliar and not as easy to debug on the command-line as "curl http://api.example/foo", but once you get up and running, easy things are easy and hard things are possible.
As for Kubernetes, I dunno, it hasn't been bad for me. I tend to use the managed offerings, so I don't have to spare 5 machines for an etcd cluster / masters, or maintain them. I build a container and Kubernetes ensures that it runs forever. If it dies, it's restarted. If more replicas are added, they start receiving traffic.
I can manage 100% of the configuration in Git, so if my cluster or cloud provider blows up, I can re-apply somewhere else and have a 99.9% chance of it all working within 15 minutes. Before containers and k8s, production felt very much like a "yolo" thing to me. Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release. You needed to distribute root credentials, hoping that you could fully trust everyone on your team to not mess anything up. It mostly worked, but through sheer brute force rather than any system working behind the scenes to make things run smoothly. With k8s, you can delegate this tedium to software. A developer can update a config file, have the PR approved, and software will ensure that the new version of the software starts, is assigned some load, and the old version is shut down. It's smooth, hard for someone to manually mess up, and quite productive.
I get that people have made their own ad-hoc orchestration an...
thanks for the succinct explanation of gGRPC over HTTP / REST / whatever.
I've been trying to figure that out for a while.
Think your analysis is spot on overall.
The author strikes me as the person who was saying "why can't I just keep writing Assembly?" in the 90s .
The answer is you can keep using <X technology at a lower abstraction level>.
But don't be mad when people are using the new abstraction layer to build interesting stuff because they don't have to worry about the lower level as much.
> Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release.
that's not really true, configuration management on mutable systems was pretty widespread before everyone started doing immutable infra.
> In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers). I would go so far as to say that Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd.
Oh for crying out loud. This Kubernetes bashing is getting so old.
Frankly, I think it's the best thing that has happened in the past few years. Boohoo it requires some YAML. Yeah. And then you get lots of value out of the box.
Systemd bashing is a bit more deserved, but this horse has been dead for a while now.
And there's also some CoreOS bashing too. CoreOS is/was spetacular.
>The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 [appendex a] is just a list of the 61 most popular headers from Google properties.
I thought this was moderately-funny cheeky banter, but I wanted to see what implementation decision they were making fun of with this silly misrepresentain. Quote RFC 7541: "The static table was created from the most frequent header fields used by popular web sites." Oh.
Why do you think that is a bad header compression scheme for a static protocol where the majority of traffic contains those headers/values repeated over and over? That indexed table shaves about 30% off header size.
When I read these anti-kubernetes articles, of which there is one every couple of months, I think of an analogy with tailors. I mean, a made-to-measure garment is strictly superior to what you can buy from Old Navy. If you take a tailor into Old Navy you would probably get a similar rant about how badly made the clothing there is. I also recognize that everything that Kubernetes/Docker is doing could be replicated more simply and with greater craftsmanship.
I think the unnoticed problem is that the tailors (old school system administrators and dev ops) are being asked to become the managers of Old Navy (kubernetes cluster administrators). Their entire skill set is opposed to this new role. I can't really blame them for being frustrated.
To keep stretching my already thin analogy, there are still tailors in this world. However, most people buy their clothes at outlets like Old Navy. Whining about the quality of the clothes and the crummy manufacturing process at Old Navy won't change that.
> When I read these anti-kubernetes articles, of which there is one every couple of months, I think of an analogy with tailors. I mean, a made-to-measure garment is strictly superior to what you can buy from Old Navy. If you take a tailor into Old Navy you would probably get a similar rant about how badly made the clothing there is. I also recognize that everything that Kubernetes/Docker is doing could be replicated more simply and with greater craftsmanship.
But down in reality, most companies are probably running their own made-to-measure deployment and operation schemes with lower quality and consistency than Old Navy.
Of course, just as many tailors back in the day were likely to make clothing worse than Old Navy makes today. We fetishize the top 1% of masters as if the average quality is anywhere near that.
Again, it reminds me of a carpenter friend who hates Ikea. Yeah, I get it, Ikea is really bad compared to custom built furniture. No one who buys it expects anything else.
This story is older than the industrial revolution. Craftsman being replaced by technology. We even instinctually know this is going to happen to knowledge workers but we seem blind to it when it happens to us.
647 comments
[ 4.1 ms ] story [ 368 ms ] threadAnyone involved in Kubernetes, near CoreOS at the time, or really anywhere in the space at the time (instead of looking back at it with anger), knows this all to be false. CoreOS was setting direction for etcd, and understandably adding features for one of its bigger users (and in fact, some of those features are used by things of larger scale than k8s).
Kubernetes itself was started by Googlers, many of whom are still there or left to go... do Kubernetes at Red Hat (IBM) or as a startup, or at Microsoft. But to act like it was an outside project started by people who had previously quit, or are somehow unqualified to work on an orchestrator, is just an an angry untruth. Every major committer to Kubernetes besides a handful of RH folks were at Google when Kubernetes 1.0 came out. I'm happy to be corrected but I know it's hip af to hate k8s (just like two days ago https://news.ycombinator.com/item?id=23807556)
Google is still the plurality contributor, followed by Red Hat. If you count Red Hat and IBM jointly, they overtake Google[0].
Disclosure: I work for VMware, which competes with Red Hat/IBM.
[0] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1&var...
Evidently, OP couldn't even be bothered to gather basic facts about the tech he trashes.
I'll take that quote into my fortune file.
But I am sure that future historians will view the impact of systemd on computing as largely negative.
I don't think you can frame systemd as some kind of RH trojan horse when so little of it makes it into RHEL/Fedora.
`netctl` was explicitly written because systemd didn't have a network daemon. The past 3-4 years the community has in general recommended against using `netctl`.
It was never removed from the ISO because the releng maintainer didn't put that much thought into it, but I'm happy to tell you `netctl` was replaced with `iwd` on this month ISO release.
I don't really think so. They probably will point out a lot problems with at some point somehow got fixed but it should be net-neutral or positive.
The thing is todays linux is running manny services and to do so nicely you want to have some form of service manager which does startup/shutdown/restart/circicute braking and helps with interconnect. Which is what systemd does and what mainly differentiates it from many previous systems which where mainly "just" start-up helpers. Through there are IMHO a large amount of problems with systemd the general approach is IMHO good, just the implementation isn't so much.
It's a logical fallacy to assume, because systemd was the thing that came along and resulted in the problems in sysv being fixed, that systemd's approach or implementation were necessary, desirable, or the best option available.
FWIW I really like systemd, but I agree the scope creep is real and isn't all positive.
Also: https://suckless.org/sucks/systemd/
Personally I avoided it for a while because of the hate, but haven't really had issues with it since using it.
Looking at the components that make up a modern UNIX system, it's definitely time to think "how can we make a new operating system, evolved from UNIX, but with a more coherent core running the majority of run time orchestration".
Even https://nosystemd.org/ mostly just lists bugs
After moving all my services from varius init systems to systemd, I'd never go back
That said, I can understand the objection to software that would previously support BSD or macOS now only providing systemd units or depending on other parts of the systemd project that in turn depend on the init making at a pain for portability across unices. Or software like gnome taking up huge amounts of time for the BSDs to support.
https://wiki.archlinux.org/index.php/Systemd-nspawn
With systemd in particular the indirection is notably (needlessly) opaque, with a big spaghetti mess of unit files symlinked in 18 different places, some of which have semantic meaning to the init system and some of which do not.
None of the negative ideological complaints against it resonate with me, as none of them seem to impact me (yet, anyways). My experience with past init systems is that they are cryptic and error prone.
(1) systemd is a monolith that aspires to do almost everything from booting to networking to sound. It's in many ways a Kubernetes sans distributed systems.
(2) systemd uses binary logs, abandoning "text is the universal interface"
systemd is also a giant single point of failure.
Personally I don't see the big deal. The actual reason people hate systemd is that it's the type of design a software company like Microsoft would come up with rather than one that a bunch of gritty basement hackers would create. They have found refuge in OSes like Gentoo and OpenBSD, so systemd served as a kind of explusion of this type of programmer/user from the Linux ecosystem.
as somebody learning how to use my linux, systemd clouded the discovery process of learning how to linux in the intended way.
i ended up using systemd, but now that's the thing i'm familiar with if i want to add some service to my computer... so it has infected me with some non-linux-y knowledge that won't go away soon.
edit for tone:
sry, that sounded saltier than i wanted. i actually went to linux because i was convinced that was the better way to do things, and i wanted to do things the better way.
i trust neckbeards over most others.
There is a perception that systemd has taken over the entire system as a cancer - replacing things that have worked for decades just because it's cool.
One annoyance I get is this: I've just taken an Ubuntu 1604 server running apache and b0rked the config file.
That's crazy. "Something broke, now go look here or there for why". Compare under an old ubuntu 1404 machine I have lying around That is so much more helpful.You run the first and it outputs the last few lines of log. Why not output that when I call restart in the first place?
Another one is taking over DNS from resolv.conf. Muscle memory from years of "cat /etc/resolv.conf" (with a tab somewhere in the middle) has been replaced with the ungainly "systemd-resolve --status".
Now all of this is great if you're running servers as cattle - if you have thousands of machines doing web-scale stuff you don't want to see what's happening on a single machine, your layers of automation abstract it all away, your system can cope with machines going down, you probably don't even have an ssh account.
If you are looking after individual machines though, systemd came in and turned everything upside down to presumably make things easier for cattle owners.
The question is, should one feel any one way about that. I’d imagine it would be the most annoying thing on earth, so it’s a valid emotion being expressed.
Maybe? There weren't any actual reasons for why the author so vehemently disliked all of the technologies mentioned, and there was certainly no acknowledgement of the potential benefits of them.
To me, that being the 'actual reasons' was implicit from the rant - I don't disagree that it was a rant rather than an argument.
They are arguing that "House Builders Inc" has invented an automated robotic nail gun and is encouraging standards industries to recommend the only things built with such a system are compliant.
It's a lamentation of the introduction of complexity into a formerly-simple set of APIs.
but it doesn't its needs a bunch of work to upgrade/migrate replace. which is what they are complaining about. having to replace perfectly good stuff, with _new_ stuff.
Unless you are bringing teams together onto one platform, k8s is never going to save you cash/engineer hours.
I maintain a (/the only?) etcd3 library for Node.js[0], and used etcd extensively on my former team.
None of these things are new to etcd3 API. All of these are present on v2 as well[1], whose API the author extolls, or are built within clients on etcd's base APIs (e.g. there's no 'lock' API, only leases). However, with etcd3 we get stricter typing, better performance, and better semantics (e.g. watch streams and lease streams over polling) thanks to GRPC.
In general these rich APIs allow 'average' engineers to build complex distributed apps more correctly. I've built reliable sharding, hash rings, elections, and so on based on etcd's API--none in more than a hundred or two lines of code (more in Go, less in Node.js). All of these are classic hard problems that etcd makes easy. Sure, there's innumerable standalone services for each of these things, but often there's no need to take the cost of many tools when one would work.
0. https://github.com/microsoft/etcd3
1. https://etcd.io/docs/v2/api/
I quite enjoyed the rantiness on a "being entertained" basis but it did rather work against effectively making the core point.
I've seen the trend toward complexity in other projects too, and it harms not just ease of debugging, but ease of hacking.
Take, for example, Swagger UI[0]
v2 was so simple. It was vanilla JS using jQuery. I, as an embedded systems developer, was able to easily hack it so it could read in the OpenAPI JSON from a database and I even added a little search box so you could filter down the APIs you wanted to see. Super fast and easy and worked just the way I wanted it to!
Starting with Swagger UI v3, it became... extremely labyrinthine by comparison. It was completely re-written in React JS and now I need a bunch of new tooling to make changes and everything was broken out into dozens of different modular files so I couldn't find where I needed to make a given change, also not to mention I've never used React so it felt like the barrier to hackability was dramatically increased.
I'm sure full time React folks love the new architecture because it's so much <cleaner/safer/scalable/etc>, but for me the change was extremely confusing and made the tool unhackable (I tried for a few hours to get it to do what I wanted, but it started looking like I was just going to have to learn all of React and I threw in the towel), and so I'm permanently stuck on v2 for now.
[0] https://github.com/swagger-api/swagger-ui
Fork Swagger v2 and make your own improvements. No one is stopping you and it's what open source is all about after all.
Web technology is its own beast. I invested a long time learning and mastering AngularJS only for all that work and knowledge to be flushed down the toilet over the next few years. Web tech has terrible ROI so that's why I "can't be bothered" to learn the latest one. I'm salty specifically about Swagger jumping on <insert latest framework> instead of sticking with simple vanilla JS that everyone understands.
> Fork Swagger v2 and make your own improvements
Yeah, that's exactly what I did...
Better get out of here before they notice us.
Of course K8s is not perfect, and it's overkill for small to medium apps (I think hype train convinced a lot of people they would need to scale to massive cloud levels when really they didn't), but if you have ever needed K8s (especially for a complex microservice system at big enterprise level) then you know the value and you remember the proprietary vendor-locked era of sadness before K8s emerged.
Same with all these technologies.
Kubernetes is a grassroots phenomenon of developers experimenting with self-immolation while padding the resume.
Major difference. Your example comes from the top. Kubernetes comes from the bottom, and is more insidious in my opinion.
This strongly correlates with bullshit VC-funded companies that wouldn't be profitable even on a good day so their only objective is to raise as much funding as possible (the business problem is secondary, because even solving the business problem perfectly wouldn't make them profitable) and chasing fads and buzzwords is sadly a valid strategy for raising capital.
Can you put your grinding axe away and stick entirely to facts and technical merits?
Note that Go ended up being a major language within Google for a wide range of systems (even replacing systems built previously by the Go developers, such as sawzall), so rather than saying "debatable loyalty to google", might it not be better to say that the developers worked to transform Google from the inside to be less dependent on C++ and Java?
That right there is evidence they don't have Google's best interests at heart.
Google Maps is fine. OSM is superior technically but Google Maps only wins because it has access to better data. It's not an example of great engineering, it's just okay.
Google Docs is pretty good. At least it was, the last time I used it was several years ago. I can't comment on the code, which is mainly what I'm commenting on in this thread: their engineering quality. Throw enough monkeys at a typewriter and maybe Google Docs comes out, but who knows what it looks like on the inside.
It sounds like you have some issues with the design itself, maybe some issues with the business practices. Those are issues with the spec, not the engineering. The engineering team delivers the spec, they do it smoothly, they do it at scale
Edit: you changed your comment out from under me and I fail to rouse the enthusiasm to reply to your new comment.
Gmail operates what is probably an industry-worst mail client. I have helped people using AOL Mail in 2020, and it is a superior experience. It performs better. FastMail is a paid service, and it's quality matches that: FastMail can run in circles around Gmail every single day.
Google Docs/Sheets is Office without all of the reasons businesses use Word. You can try to patch in some of it's holes with add-ons (I had some experience with a Mail Merge add-on for Gmail/Sheets, and it was fine, I guess, but a paid service from a third party) but with a lot of added jank. Collaborative editing was it's claim to fame, but you can do that just fine in Office 365, with a far more capable app suite.
Google Maps, Drew covered the point pretty well: Google just has the money for more data. It's prohibitively expensive to use that data yourself as a service, and others who have access to less data still manage a comparable and sometimes better product than Maps. Usually while better respecting your privacy.
Personally, I really like JAX and XLA. These are two really powerful systems that work together to implement high performance machine learning and theoretical physics/biology. JAX can basically take derivatives of python functions (useful for training) and XLA optimizes the underlying compute graph to execute quickly on different physical platforms.
Google doesn't get credit for Go, Bell Labs does.
Counter-anecdata: my most miserable years of my career were when dealing with bespoke build and API systems that weren't Bazel and gRPC.
Again, gRPC has its flaws (and I'm not a fan of HTTP2), but what are you comparing it to? For internal services REST is acceptable for extremely simple, stable and low volume APIs. For everything else it is an obvious disaster: terrible CPU/network/memory performance, you need to write more code than you would have to with gRPC and get zero type safety unless you add some json schema monstrosity on top of it at which point your performance will likely take a further nose dive. Also, good luck evolving the protocol – gRPC is carefully designed to support this. Far from doing a good job, "TFA" quotes some extremely clueless article on why protobuf sucks that was ripped to shreds when posted on HN previously.
The only compelling alternative to gRPC I am aware of is capnproto (which is derivative, but in some ways nicer), but it has far less eco system maturity and mindshare.
I would argue that while Cap'n Proto's serialization is derivative of Protobuf, the RPC protocol is wildly different from gRPC.
> but it has far less eco system maturity and mindshare.
That's certainly true.
(I'm the author of Cap'n Proto.)
2. protocol buffers
rest is vaporware
Ever had to maintain a Gerrit instance?
Bazel is the most complicated garbage I've ever had the displeasure of using. It takes tens of thousand of lines of support code just to add a new tool to your toolchain.
Google does not get credit for Go's design.
gRPC, k8s: see TFA
Yes. [1]
> It takes tens of thousand of lines of support code just to add a new tool to your toolchain.
No, I've added custom compilers and languages to Bazel, and it was not nearly 'tens of thousands of line'. Sure, it's complex, but such is life with inversion of control. It's difficult to get around this complexity if you want to follow the design goals of Bazel.
> gRPC, k8s: see TFA
The article says nothing about why gRPC or k8s are bad. It just uses very emotional language to handwave into the general direction of 'bloat' and 'too complex'.
[1] - https://cs.hackerspace.pl/hscloud/-/tree/devtools/gerrit
I'm serious. Say what you will about the high complexity of the web platform, or the rising dominance of Chromium. It's a seriously impressive piece of engineering. Have you ever watched the Chromium build process in action? It builds hundreds of static libraries, then statically links them all into one monster executable or DLL (depending on the platform). BTW, part of what enables static linking at this massive scale is the gn build system, which AFAIK is inspired by Blaze (the internal ancestor of Bazel). So IMO that's a major point in favor of that type of build system.
BTW, to answer your point about throwing tens of millions of dollars at a problem: some problems are just unavoidably that big, especially when you factor in things that are required for a piece of end-user-facing software to be usable by the whole world, such as internationalization, accessibility, and in general, complex but usable UIs.
I'm not sure what there is for us to discuss. Nice rant, I guess? The post does not attempt to persuade or change opinions (which, again: cool. sometimes it's nice to have a good rant).
There's a million reasons to hate Kubernetes, the author couldn't be bothered to venture beyond the lowest hanging fruit (yaml)?
If I could downvote this I would.
Surely, most consumers of a database like etcd are using a client library, in which case why does it matter if the API is HTTP or gRPC?
Tangential: after reading some of the comments, I was surprised that the blog post was only like 250 words; the author really says very little.
>There were a lot of bad ones
Sounds like he had a point, then. Also, apparently HN would rather read your rather salty gossip, than comment on the article on its own merits. Nice!
I'm going to assume you're a developer?
Infra people are usually much more apprehensive to take on new technology. Crucially I would describe classically trained sysadmins as 'pessimists to the core'. This is why there's memes of operations saying 'no'.
This is what devops was all about, the shared responsibility of it all. I'm going to assume that uber was perfect and got devops exactly right- but adding technology should in my mind always be met with the absolute most critical eye imaginable; and if he's a classically trained sysadmin then it probably comes from that place of being once bitten twice shy.
No, but thanks for explaining my career to me and helping me empathize with toxic people.
I'm explaining that the majority of the people who were sysadmins in 2005 are more likely to be pessimists than optimists and are change/risk averse, I don't think that's controversial and definitely is a meme.[0]
> [thanks for] helping me empathize with toxic people.
You're welcome. :)
[0]: https://books.google.se/books?id=0VRnDwAAQBAJ&pg=PA490&lpg=P...
And, yeah, sysadmins used to say no a lot but younger ones tend not to because they haven't been rubbed wrong by the silo's of yore.
So in today’s world, that kind of attitude is hardly productive. Skeptical? Absolutely. But open minded.
Venkatesh Rao says that there comes a point in your life when you realize things you thought were permanent are temporary, and things you thought were temporary are permanent; he uses "40" as a good rubric for that developmental stage. IT goes through many pendula, whether it be containerization vs. amalgamation, or thin clients vs. thick. What, over time, you learn to hold on to is the tools that have lasted decades and will probably continue to. Right now the pendulum is starting to swing back towards amalgamation, and it will probably swing back towards containerization again in another decade. Whatever the tech is at the time, it can be good to reconsider whether your views will have changed not on the technology, but on the larger pendulum it's riding.
Please name what infrastructure technology has lasted decades, I am curious to know. If you try hard enough yes you can see it’s all cyclical, you can say, MULTICS was the OG cloud. OK, fine, but what use is that to be as a practical software developer building things? Perhaps with age you start seeing everything as being like something you’ve already seen so it’s not that exciting anymore?
Is it?
Code I wrote literally 20 years ago is still running in production, and I get paid to work on an app with plenty of code around that was first committed in 2009.
If you build a system in k8s today, and it's a success, there's good reason to believe you'll be on k8s in 2030.
And this is why every single project out there is now a house of cards (or should I say house of YAML files) using insanely complicated technologies (like Kubernetes) with very "interesting" failure modes to say the least.
This attitude works today because of engineering-driven-development; the whole purpose of engineering is engineering and business priorities took a backseat in favor of buzzwords on the careers page and an obligatory "engineering blog" (describing how they solve self-inflicted problems), however when it comes to reliability and solving problems a large majority of projects can get away with much simpler, old-school technologies.
Services today scale to handle a lot more users and traffic than they did not so long ago; and these reliability guarantees are the norm rather than an exception.
I disagree about not needing dedicated infrastructure specialists. Kubernetes' complexity, learning curve and failure modes would make me uncomfortable operating without having a dedicated "devops" person (or sysadmin as we used to call them) while I am perfectly comfortable managing a few virtual machines (or even bare metal hosts) with a load-balancer in front of it. I recommend building systems in a way that can easily fit in your mind, and there's only so many abstraction layers and moving parts you can fit in there before you overload.
When it comes to scaling, not every application needs to scale and even when it needs to, it's trivial to scale stateless app servers without Kubernetes. You can scale quite far without Kubernetes, and when you're past that point you'll realize your main bottleneck is your data store and Kubernetes (or similar) can't magically solve that.
Fads don’t form out of thin air. There’s always some value that they provide. To someone experienced with setting up infrastructure, the tasks may seem trivial, and the value add is low. For others who don’t, having a dead simple way of easily adding tooling around their applications is a godsend. Why is this so fucking hard for you to understand?
I am not an expert in setting up infrastructure by any means. In fact if I were I would probably use and promote these technologies. But in my opinion, adding another layer of abstraction doesn't magically solve the problems of the underlying stack (it won't protect you against obscure Linux kernel behavior, but now you have yet another moving part and potential variable which you need to account for when troubleshooting) but still gets in the way when you're trying to do something simple that doesn't even require any of the advantages the container technology is offering.
When it comes to "adding tooling around their applications", I am not sure what you mean but I will assume you refer to your previous examples, in which case I do not see how container technologies change the game at all. Tracing and centralized logging require your application to talk to a centralized log server (for logs, you can also output to stdout and have systemd/syslog collect and send them to the logging server) and container technologies don't change anything here.
I am not saying that container orchestration technologies provide no value. I am saying that they are often overkill for the task at hand and introduce extra complexity, moving parts and management overhead.
Kubernetes and abstractions of its ilk (shipping containers for example) have a place, but every abstraction comes with some form of trade-offs, be that performance or transparency.
Dealing with a node brown-out in kubernetes is much worse than dealing with a network, host or service outage because the troubleshooting steps involved evolve fractally.
That said, obviously there is value- but it's good to critically assess the value instead of just jumping in.
https://standalone-sysadmin.com/the-difference-between-site-...
While I could argue with all the points that he's making, my main counterpoint is this: junior devs don't care that their http/2 server uses way more "complex" code then their http/1 server, it's just a flag away (or in most cases, automatic). Senior devs worth their salt also don't care, if I design an application that is going to run on kubernetes I now know it will run in the big cloud providers and on premise without major changes. It forces you to accept that your app will die and needs to be able to run from a cold start without any issues. I can't count the number of machines I've encountered over the years that couldn't be rebooted because the maintainers monkeypatched the crap out of it while running EoL distributions, libraries and web servers.
And now I've realized I've become a geek yelling at the cloud as well.
HTTP/2 is more complex to implement from scratch (but still quite doable, even as an individual), but it has built-in support in every language & framework worth its salt now, so you don't need to do that.
If you're using an existing implementation, that's usually just as easy to do as HTTP/1.1, because they have almost exactly the same semantics (that's an explicit goal from the spec), and so most implementations have almost exactly the same API.
In practice, it's a syntax & connection management change on the wire that's mostly invisible as a developer building on top of it, plus a set of optional extra features (like Server Push) that you can use if you want or ignore if you don't.
Can't speak for QUIC/HTTP3, since I haven't touched them yet, but I'd be a surprised if that's a hugely different story.
I also disagree with much of the rest of that, but the clouds are getting sick of me today.
if you wanted simple, there was redis or memcached.
it's an arbitrary cutoff to call etcd simple (it implements Raft, for fuck's sake, it implies a distributed system, multiple odd number of nodes to avoid split-brain!)
systemd is simple, fixed config format, comes with every distro, default works, has extensive documentation, compared to undocumented distro-specific init scripts written in sh (or bash, or worse).
sure, if someone just wants to hack on their 8bit toaster, then they might find a simple shell script simpler.
but the post is not lamenting that.
This resonates with me, and I'd like to add HTTP/2 can only bring advantages if you actually go all the way to push/bundle resources into responses and have a strategy/priority when to push eagerly vs serve lazily; I'm even much more worried about upcoming QUIC (and DoH) because there's no impl in sight.
Compared to well-optimized HTTP/1 (e.g. using minified CSS and sprite-sheets), sure. Compared to most HTTP/1 deployments, though: no. HTTP/2 gives you tons of advantages "for free" that you need build-time processes to attain in HTTP/1. With HTTP/2, you can do "the naive thing" that'd you'd have done on the 1995 Web in Notepad, and it'll be the optimal thing.
Also, HTTP/2 means less OS packet-switching overhead server-side if you have ancillary connections (e.g. websockets) open against the host, since those also get muxed into the same carrier socket.
Also, mobile clients wake up less, because there's only one TCP socket to do idle-keepalive on.
HTTP/2 also means that TCP's Nagling has more to work with, and so is less likely to end up needing to waste bandwidth on emitting many undersized packets—it can just pack N short requests into the same TCP jumbo frame, since they're all going to the same place.
I would also point out an indirect "advantage": HTTP/2 makes it cheaper to serve ads proxied through the first-party host (as HTTP/2 flows) than for the client to hit the third-party ad servers directly. People can still block the ads/trackers either way, but served inline to the origin like this, the people who don't block ads, will get a better experience.
HTTP/1.1 pipelining partially mitigates this, allowing the client to "queue up" a list of all the dependent resources it wants from each socket; but it suffers from head-of-line blocking. Which sounds like some arcane thing, but in practice it means that big things might block the loading of small things. (The browser doesn't know how big things are, so it can't effectively schedule them; and the server must dumbly queue results up in the same order the client requested them, because that's the only way the HTTP pipeline's implicit flow sequence counters will match up.)
HTTP/2 is a full mitigation for this problem, since—even without a heuristic "prioritization strategy" for the delivery of dependent-resource flows—the "oblivious" strategy is still a good one: if you attempt to deliver all the resources in the queue concurrently; and you do so by delivering one fixed-size chunk of each flow per iteration, in a round-robin fashion; then you'll end up finishing delivery of resources smallest-to-largest—which means you'll usually deliver the most-critical resources first, no matter where in the dependent-resource queue they started.
Or, in short:
HTTP/1.0 = O(N) required roundtrips for a page with N resources.
HTTP/1.1 with pipelining = O(1) required roundtrips for a page with N resources (followed by O(N) bytes streamed half-duplex), but the page can still take nearly the same amount of time to become interactive as if it were O(N) roundtrips, because of effectively worst-case scheduling.
HTTP/1.2 = O(1) RTTs + O(N) half-duplex bytes for N resources, loading "intelligently" such that the page becomes interactive in O(log N) time.
The only things that kind of break this heuristic are:
• large single-file Javascript SPAs. These are usually just marked with the `async` attribute, such that the initial DOM of the page can first render, then be gradually enhanced when the SPA loads. But with HTTP/2 + ES6, you can also just not pack the SPA into a single file, instead relying on ES6 modules, which will each be individually smaller and therefore will end up being delivered first by the content-oblivious round-robin chunk-delivery strategy.
• web fonts, which are large and will necessarily cause a complete reflow of the page once loaded. Currently, browsers make a special loading-precedence exception for web fonts; though it doesn't matter as much right now, as they're still mostly served from third-party CDNs rather than as first-party HTTP/2 flows.
AFAIK, for regular web-page retrieval, these are exactly the total set of things that currently benefit from being server-pushed along with the first response; everything else just gets handled well even without server-push.
(If you're curious, server-push is really designed for the use-case of pushing secondary API responses along with an initial API response; to allow for GraphQL-like "single-round-trip resource-hierarchy/-graph walking" in a way that's more friendly for caching layers than comingling the results into a single resource-representation. It makes the most sense in a Firebase-like system, where un-asked-for server-pushed resources can be obliviously accepted and dumped into the client-side in-memory set of synced entities asynchronously to the parsing of the initial response; and then, once the dependency is parsed out on the client side, the client can discover that it already has the entity it wants in its in-memory entity store, and doesn't even need to make the request.)
No, the benefit of HTTP/2 is a lack of head-of-line blocking. Head-of-line blocking can be easily seen when big things block small things, but that's not what it is. What it is, is when something doesn't make progress because another thing is being waited for.
Imagine a multimedia container file-format where you can't interleave audio frames with video frames, but rather need to put the whole audio track first, or the whole video track first. This format would be unsuited to streaming, because downloading the first chunk of the file would only get you some of one track, rather than useful (if smaller) amounts of all the tracks required for playback. Note that this is true no matter which way you order the tracks within the file—whether the audio (smaller) or video (larger) track comes first, it's still blocking the progress of the other track.
HTTP/2 is like a streaming multimedia container format: it interleaves the progress of the things it loads, allowing them to be loaded concurrently.
This doesn't just mean that small things requested later can be prioritized over large things requested early (though it does mean that.) It also means that, for example, if you load N small Javascript files that each require a compute-intensive step to parse+load (GPU compute shaders, say), then you won't have to wait for the compute-heavy load process of the previous files to complete, before you begin downloading the next ones; but rather you can concurrently download, parse, and load all such script files at once. Insofar as they don't express interdependencies, this will be a highly-parallelizable process, much like serving independent HTTP requests is a highly-parallelizable process for a web server.
One benefit of HTTP/2's lack of head-of-line blocking, that would be more talked-about if we had never developed websockets, is that with HTTP/2, you get a benefit very much like websockets, just using regular HTTP primitives. You can request a Server-Sent Events (SSE) stream as one flow muxed into your HTTP/2 connection, and receive timely updates on it, no matter what else is being muxed into the connection at the same time. Together with the ability to make normal API requests as other flows over the same connection, this does everything most people want websockets for. So the use-case where websockets are the best solution shrinks dramatically (down to when you need a time-linearized, stateful, connection-oriented protocol over HTTP.)
> new protocol that sets in stone some heuristic
Note that there's actually no explicit specification of the order in which HTTP/{2,3} flows should be delivered. What I'm calling "content-oblivious round-robin chunk scheduling" is just the simplest-to-implement strategy that could possibly meet HTTP/2's non-head-of-line-blocking semantics (and so likely the strategy used by many web servers, save for the ones that have been highly-optimized at this layer.) But both clients and servers are free to schedule the chunks of HTTP flows onto the socket however they like. (They can even impose a flow concurrency cap, simulating browsers' HTTP/1.1 connection limit and starving flows of progress. It'd make the client/server a non-conformant HTTP/{2,3} server, but it'd still work, as what progress "should" be being made is unknowable to the peer.)
It's a bit like saying an OS or VM has a "soft real-time guarantee" for processes. Exactly how does the OS scheduler choose what process will run next on each core? Doesn't really matter. It only matters that processes don't break their "SLA" in terms of how long they go without being scheduled.
The advantages listed above all come from the fact that HTTP/2 runs all traffic over a single multiplexed TCP connection, rather than HTTP/1.1, where clients are forced to open many independent TCP connections (because each is a single channel where responses must be returned sequentially, in the order they were requested, and so block one another).
That happens totally automatically, there's nothing special you need to do to enable that.
There are also other potential advantages once you look at push etc, but using TCP as it was meant to be used really does give you many advantages for free.
it shoves everything down a single TCP connection. This means that if you loose a packet (and 4g is lossy) it stalls the _entire_ queue.
Instead of addressing the main complaint (that multiplexing everything down a single TCP connection is a fundamentally stupid idea) they come up with an entirely _new_ protocol.
It also fails to understand what HTTP2 has turned into: a file transport protocol, with some RPC built in.
So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
(Mind you, they are versions relative to each-other, because HTTP/3 is strictly better at doing what HTTP/2 does than HTTP/2 is, so there's no reason to use HTTP/2 if you have HTTP/3.)
But as I was saying: nobody forces mobile devices to use HTTP/2; and by-and-large, they don't. HTTP/1.1 still has its use-cases; neither HTTP/2 nor HTTP/3 was designed to obviate HTTP/1.1's entire set of use-cases.
You know how SSH has sub-protocols, e.g. SFTP? HTTP/2 and HTTP/3 are to HTTP/1.1, as SFTP is to SSH. It's a mode you can put the connection into, that will optimize it for a certain profile of use-cases. No more, no less.
(You know what else is a mode you can put HTTP/1.1 into? Websockets! Again, a complementary use-case.)
> So, instead of optimising for file transfer, a control layer, and a communication/serialisation layer, they came up with a horrid mush that is http3. (or the googlenet)
If you think you're so smart, write an RFC (or even just, build a reference server) that's competitive in production performance for its use-cases over similar links.
There comes a point when explicit OSI-style encapsulating layering becomes an active hindrance; and for the use-case of "a server serving a million concurrent requests"—the FAANG-scale problem—they passed that point long ago.
Yes, we mere mortal client users and backend developers might not find such use-cases relevant; but take the perspective of e.g. an internet backbone ops engineer. By volume, most of the traffic going around the Internet goes to these sites. Optimizing just the traffic going to these sites—with a special protocol clients only use when speaking to these sites—makes the whole Internet faster, by reducing in size the huge blob of bandwidth-contention the requests to these few sites create.
Also, a design hint, if you actually want to try to build something competitive to HTTP/3: most of the problem being solved by HTTP/3 is that certain things are known to be optimal, but despite that, nobody can just force their corporate overlords to mandate a switch to those things right away. So HTTP/3 needs to be optimized for the case where you do the most fundamentally-performant things (e.g. using zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access); and also for the cases where you do less fundamentally-performant things (like generating and streaming a JSONL or CSV stream line-by-line as the output rows gets computed by an in-process dataflow engine.) One might call HTTP/3 an arbitrary custom protocol, that happens to have semantics such that it can be downgraded to HTTP/1.1 by a link-to-link proxy. And that is a hard constraint to optimize under.
They don't, though? HTTP/2 and HTTP/3 are voluntary to all parties concerned; whether you're a client, a server, or a gateway, if you don't speak those protocols but choose to just speak HTTP/1.1, then it's the peer that has to cope with that, not you.
(There isn't even any fancy forward-compatibility needed in TLS to support the semantics of the ALPN extension. If you just use an old TLS library that ignores the unknown extension data in the TLS stream, the other side will continue on assuming you didn't understand the question, and therefore aren't an HTTP/{2,3} server.)
HTTP/{2,3} are like a local language of a culture, spoken by immigrants when they run into other immigrants from the same culture. If either party is not an immigrant from that culture, it just doesn't come up.
> Why can't the "big sites" not then create their own network and take their enormous traffic there?
That's called circuit switching (i.e. the thing telco and cable services do that's not the Internet), and it's the thing the packet-switched Internet effectively obsoleted. From an Internet engineer's perspective, if you have two data streams, it's strictly better engineering to just feed them into a switch that linearizes those packets onto a high-bandwidth line "as they come" (and upgrade the bandwidth of the switch+line as needed, so that no signal is ever starved of line-time), than to try to time-divide or frequency-divide the pipe; let alone to keep those packets isolated on two separate networks of pipes. Then you'd need to maintain two networks of pipes! (And the people working at FAANG are still fundamentally Internet engineers who believe in Internet principles, rather than telecom principles.)
But besides that, how would that network be delivered into people's homes? Unless you're proposing that these services take the form of their own additional cable going into your house/SIM in your phone, this network has to merge into the regular Internet somewhere. And it's exactly at that point when that traffic once again contends with the rest of the traffic on the Internet. Even if it's only on the last mile, it's still getting in the way.
> it's just Google who's behind QUIC anyways, isn't it?
SPDY and QUIC are the names of "prototype standards" developed by Google. HTTP/2 and HTTP/3 are standards inspired by SPDY and QUIC, developed by HTTPWG, with Google as just one participant in that conversation.
The other backers of the standard are, of course, the groups whose interests are aligned behind having more-efficient HTTP: carriers, bigcorps, switch/NAT/WAF hardware manufacturers, cellular ISPs, etc.
But I see your deeper point—you're saying that this is all Google's solution to Google's problem, so shouldn't the onus be on Google to solve every downstream problem as well?
Well, it is and it isn't. Google is solving this problem for us right now, but it's not a Google-exclusive problem. TCP was created by DARPA, but maintaining a consistent stream over packet loss/reordering is not a DARPA-specific problem. They just happened to be the first group to need a solution for that problem.
The reason HTTP/2 and HTTP/3 are public standards, rather than things going on secretly only between Google Chrome and Google's backend servers, is that other parties see value in them—not just present value to themselves, but also future value.
New big uses of internet bandwidth arise every day. Netflix started sucking up half the Internet ten years ago, and it's already dropped down to less than 15% because other even larger use-cases have eclipsed it.
HTTP/2 and HTTP/3 are engineered to allow small businesses a path to grow into the ...
to which I as a normal person on a normal phone has no control.
> Also, a design hint, if you actually want to try to build something competitive to HTTP/3
I'd need to control a major browser's code base.
Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
> perspective of e.g. an internet backbone ops engineer.
They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
> If you think you're so smart, write an RFC
I already work for a FAANG during the day, I'm not going to put free hours in at night. And I'm certainly not going to get involved in cross company politics.
> a point when explicit OSI-style encapsulating layering
OSI has never been correct, its a fib told to students to make them think there is any kind of order past tcp/udp. I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms. yes its very neat, but
> zero-copy binary serialization protocols like Capn Proto from a process with direct userland Ethernet access
from the protocol level, its not actually that much different, unless you are doing some sort of multiplexing down a single TCP pipe.... now, if you trying to balance throughput and latency on the same connection, thats can be really tricky. An answer is not to use the same connection. You do what FASP does which is use UDP for bulk and TCP for control/accounting.
> like generating and streaming a JSONL
you can easily use TCP at one packet a minute, or maxing out a 10 gig link. (I know, I've written a FASP replacement.)
> semantics such that it can be downgraded to HTTP/1.1
you can downgrade 10gig-t to 10m/s that has no bearing on its performance, unless you terminate it badly. Its just a preamble, then for the life of the connection, an utter irrelevance.
You misinterpreted "downgrade" here (though maybe I chose a bad word for this.) I didn't mean that the proxy itself can force both sides to speak HTTP/1.1 to it. I meant that a proxy can be speaking HTTP/{2,3} to one side, while speaking HTTP/1.1 to the other, by translating ("downgrading") HTTP/{2,3} messages into their equivalent HTTP/1.1 messages. This is the reason that HTTP/{2,3} are HTTP in a real sense: the messages that flow over them still interoperate with the rest of the HTTP ecosystem, through bilingual gateways. The messages are a different bijective encoding of the same HTTP application-layer message-flow semantics.
(Nginx, by default, is such a gateway: it speaks HTTP/2 to clients if they offer, but speaks HTTP/1.1 to upstream backends even if they know how to speak HTTP/2. This is because HTTP/2 is useful to carry flows over the Internet, but has few advantages within a low-contention low-packet-loss LAN.)
This also has other implications, like the fact that browsers and caching proxies don't need to change the way they store cached HTTP resources, just because the resources come in over HTTP/{2,3}. They can just transcribe the messages to disk as if they had been the equivalent HTTP/1.1 messages.
> Lets get one thing clear, HTTP is, has never been and almost never will be efficient. The entire HTML/JS/HTTP web is never going to be efficient. If the web was a meritocracy based on protocols, HTTP would have died a death at CERN.
The point is to take a stupid thing that people will keep on doing no matter what (for better or for worse), and help them to do it as fast (or as un-slow) as possible, without trying to make it non-stupid in the process (because that won't be the same thing any more, and so people won't use your smart thing, but will continue doing the stupid thing.)
You know Formula 1 racing? In F1, there are rules against doing most things that could make cars go fast. The point of the competition is to make the cars go fast anyway, without breaking those rules. If you make the car go as fast as it can, it's not an F1 car any more. To win, you've gotta make the fastest F1 car, not the fastest car.
That's HTTP/{2,3}: the cars are the flows, and the rules are HTTP's must-be-all-things-to-all-people semantics and manifold optimized-for usecases. The goal of adding HTTP extensions/versions/etc. is to make your server (or client) talk to clients (or servers) as efficiently as possible, without breaking any random peer from 1996 speaking to you through a bilingual gateway. If you did break those peers, you might have the fastest protocol, but you wouldn't have the fastest HTTP protocol.
> They'd terminate the connection as close to the client as possible and palm it off to the local FAANG pop and make it someone else's problem. That and adding pop's for porn hub and torrent filters will knock out >70% of your precovid interconnect traffic.
That's an infinite regress, as the people who you're handing the problem off to are also Internet backbone engineers. If your company owns 30% of the Internet's traffic, your leaf-node AS is part of the backbone, and that AS's problems are the Internet backbone's problems. Just like the major universities were part of the backbone until ~1990.
Though, keep in mind, this isn't about fancy application-layer-aware routing or anything. HTTP/{2,3} have bandwidth savings, and that savings is passed directly onto the Internet backbone those messages travel along, in the form of more headroom for other stuff. It's not about needing to pay ...
> I'm talking about not putting control signals in the same TCP connection that you are shoving tonnes of data down. That's never going to scale past 70-100ms.
...but I think I got it now. I was arguing with the wrong thing here.
Yes, there's a reason that FTP and RTP do what they do. (Though in FTP's case it had more to do with TCP-port parity gender; multiple data sockets per control socket was just a convenient outgrowth of the existing architecture.)
But crucially, HTTP/{2,3} are about accelerating existing traffic; and existing HTTP "apps" (the HTML + JS + CSS that contain an implicit dependent-resource digraph that gets turned into flow-requests by the browser) have no indication in them for what's going to be a "control-plane" request vs. a "data-plane" request. HTTP itself is an oblivious carrier re: these use-cases. So HTTP/{2,3} must continue that trend, and be oblivious carriers as well. To do otherwise would force a rewrite of all HTTP "apps" to add such indications.
However, in HTTP/1.1, you got the benefit of this purpose-isolated signalling "for free" anyway, because you'd probably be making your data requests from a CDN origin, while making control requests to an API-subdomain origin. Those are separate origins, so they got separate (keepalive, pipelined) TCP sockets.
In HTTP/{2,3}, you can choose to shove everything into one socket by putting both your control-plane and your data-plane into the same origin.
But, crucially, you don't have to. Nobody's stopping your HTTP "app" from just having separate control and data origins; and so ending up with two HTTP/{2,3} sockets open, one for control signals, the other for BLOB data.
And, of course, existing sites previously engineered for HTTP/1.1 won't shove everything into a single socket, since they've already separated the control origin from the data origin. (They've likely got several data origins, so they'll likely want to re-engineer with upstream proxying to compact those down into one data origin; but all they have to do while doing this is to choose not to also compact the control origin in there as well. The choice to not change anything is, in this case, the right choice!)
And also keep in mind that most websites aren't, like, photographer's portfolios or direct video hosts. There are few websites directly embedding same-origin BLOBs. There are many websites embedding "Binary Not-so-large OBjects", like blogs that embed same-origin 500px-wide medium-quality optimized-JPEG images. And those websites don't have such a large disparity between control signalling and data signalling that they have any need to separate them—those images aren't much larger than the average compressed-JSON XHR response!
And because of that, the "default" advice for HTTP/{2,3} transitioning (that you should follow if you're 99% of websites) can just be "put everything on one origin"; whereas "split control signals to a different origin, thereby forcing it to a separate socket with separate TCP window scaling parameters" can just be one of those arcane PageSpeed things you learn from an infrastructure consultant, a bit like choosing what type of select(2)-alike kernel ABI your webserver should use to receive packets.)
There's so many things where my competence is based on understanding the simple preceding system plus a good intuition of how the class of systems tend to be bungled up over the years. We don't pay down the accumulated complexity, and newcomers don't get the same luxury I do.
It really becomes noticeable with highschoolers and I have to chose between teaching useless but illustrative groundwork, or just explaining and expecting a combination of taking it on faith and memorization.
What a strange comment. "The benefit of this protocol is that it's so complex that nobody understands it, so they have to use an existing implementation."
Bram's Law - The easier a piece of software is to write, the worse it's implemented in practice.
I'm not sure whether I agree with him but it's not obviously wrong.
[0] https://web.archive.org/web/20170629144859/http://www.advoga...
As discussed at length in another sub-thread here, the complexity solves real problems, it's not for its own sake. But if you're not convinced of that, nothing I say here is going to change that.
·
If you've worked with me, you'd know that I take simplicity as a virtue to an extent that's almost detrimental to getting stuff done. I despise complexity for its own sake. I wish as much as anyone that we could start over and redesign IP, TCP, TLS, HTTP, HTML, JS, and a few more things, with all our accumulated knowledge, into a simple and clean application and content distribution system that's easy to understand and observe. But path dependence :( We have to work with what we've got.
It's the picking out of HTTP 1.1 as some sort of golden age of clean and simple and observable that bothers me. It isn't.
(I'm not agreeing with the author, just pointing out why he might care about the ability for an individual to implement the protocols the web is built on.)
For the companies implementing http/2, those improvements DO matter. They have different needs.
Why is this bad?
It allows us to make smaller purpose REST api endpoints and do many calls, instead of trying to bundle everything together in larger endpoints.
Being able to parallelize cheaply greatly simplified our architecture and API design, and it costs us nothing.
> for a ~~bullshit~~ unsuccessful project called CoreOS Container Linux that was EOL'd several years ago
CoreOS was actually quite successful to me as an outside observer. It had decent paid user base as well as people using it without paying. It has showed people that Chrome OS can be used to build atomically updating host OS with a readonly fs for container hosting. It has inspired many other container hosts to come like RancherOS and Google’s Container-Optimized OS. Similarly, it was probably one of the reasons why Red Hat was interested in the acquisition.
Furthermore, CoreOS was EOL'ed last month; not several years ago.
I can keep going on why etcd API was switched to gRPC and what benefits this offers, even at small scale. But I don't think it's worth anyone’s time convincing the author otherwise. Based on their tone, it's clear to me that they have trouble with using software when things get a tad bit complicated. Usually, there's community decision-making behind these decisions, and they're often deliberated for months, backed with prototypes and data. I'm pretty sure the author doesn't care, however.
Calling CoreOS unsuccessful is a massive misunderstanding of the market.
Just because people buy snake oil, it doesn't make it a good product.
Clearly, people are using it successfully and enjoying it (or other similar products). Just because you don’t like it it doesn’t change the reality of the situation.
https://en.wikipedia.org/wiki/Container_Linux
(for those who don't know, it was renamed from CoreOS to Container Linux)
https://news.ycombinator.com/newsguidelines.html
2. I've always expressed strong opinions in my posts. Sometimes this results in multiple upvotes, sometimes in multiple downvotes.
3. About being "unsubstantive" - I underscored the main concept/rationale for Core Linux, and linked to its Wikipedia page. You could argue that it's not substantive _enough_, but when you phrase it this way it's as though you merely dislike the fact that I repeated the inflammatory term from the article.
For example, we've gone a step further than CoreOS did and have a fully open-sourced update server, Nebraska (https://github.com/kinvolk/nebraska). We also generate a list of contents and licenses for each build. Here is an example from the most recent stable: https://stable.release.flatcar-linux.net/amd64-usr/2512.2.1/...
Times have changed and the very people he criticizes are infinitely better at branding and inspiration. There's a reason they're the ones gaining traction.
Like you, I want to be constructive, and I do agree with the author's core complaints. He just doesn't realize he's actively self-sabotaging and harming his own reputation.
https://coreos.com/os/eol/
Many reasons to switch from JSON to gRPC: * gRPC uses HTTP/2 which means you can concurrently make multiple requests on a single TCP connection, while on a typical JSON API which probably uses http/1.1, you can't. * Type safety of protobuf types. * Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane. * Client libraries are automatically generated, and not error prone. * RPCs are already optimized for bytes on the wire efficiency, whereas JSON is not. This matters a great deal as Kubernetes objects get large in size/quantity over time but controllers still work effectively by not spending too much CPU on encoding/decoding like they do on JSON. * Similar to the previous point, most json decoders don't reuse objects, so every decoded object is a new alloc, whereas gRPC can Reset() and reuse the same object while decoding/encoding. * Builtin authentication primitives (such as JWT/tokens or even adding TLS to client and/or server). * gRPC has support for interceptors which are like middleware functions you can inject to requests/responses on both client and server-side for logging, authorizing etc.
The list goes on, but something to note is that etcd was not developed by Google (and as far as I know, not by ex-Googlers). Both etcd and gRPC are owned by the same open source foundation, so it's natural that they make use of an existing technology.
Was a "JSON mapping for etcd's protocol buffer message definitions" the original "HTTP API"?
"A Critique of the Remote Procedure Call Paradigm" (https://pdfs.semanticscholar.org/e125/57a7582881a62040eee68b...)
"A Note on Distributed Computing" (https://github.com/papers-we-love/papers-we-love/blob/master...)
What are you actually talking about? An HTTP API by itself doesn't actually care about which http version is used, that's something only the HTTP server that serves the API should care about (nowadays, pretty much any http server in any language supports HTTP/2 and 1.1).
> Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane.
Even HTTP/1.1 has mechanisms for that, like long polling and chunk encoding (which allows infinite streams to keep a dual channel of communication open where each chunk can be treated as a message - with "headers" and all).
> Client libraries are automatically generated, and not error prone.
This is true, but we've had similar technology since the SOAP days... you could easily do the same with a XML-based HTTP API.
By the way, most of your points are against JSON, not a HTTP API per se, which usually would allow a number of formats , including XML and JSON at least.
> Builtin authentication primitives...
HTTP also has that when you include cookies and something like OAuth/OpenID.
> gRPC has support for interceptors which are like middleware functions
This kind of thing is better done by using the specific platform you're running on (depends on the language) so I don't see something like this as something desirable on the specific RPC framework you're using. With HTTP, caching, logging etc are trivial to do and one of the strongest advantages of using HTTP in the first place, so I think you're confusingly making a point for HTTP here, unless your focus is on the RPC side of things? In which case, you would have to make a case on why RPC is a better fit than HTTP for etcd, which you haven't.
So all in all, I found your points utterly unconvincing, but presented with so much conviction that you're actually right that I could not resist to respond (even if I don't want at all to get into a pointless discussion on the merits of HTTP VS RPC or JSON VS Protobuffers)!
For example, whole etcd API is located at https://github.com/etcd-io/etcd/blob/4c6881ffe4b3bae257c0720...
It is pretty straightforward and not too hard to understand the API. Designing APIs with protobuf makes things convenient and it brings lots of already written tools with it.
Since gRPC built on top of HTTP/2, we thought gRPC as easiest and performant way of writing a HTTP API with good defaults.
Yes it's easy to make that RPC call. But should you?
I've been off doing other things in the intervening period, but while I had my back turned the industry seems to have turned its back on REST and gone whole hog on RPC. Again.
At first I was thinking this was just internally here at Google, where protobufs and gRPC reign supreme. But it seems to have taken hold everywhere.
What did I miss? Why have we swung this way. Again. Is the pendulum going to go back?
I've heard this argument before, but how does gRPC itself cause this issue to manifest? I'm curious to hear what your opinions are on a better alternative, and how not to represent a remote resource as a local one.
But the fact that it presents the remote resource in an API which resembles a local object means that programmers often get lazy in the manner of which they think about these things. The REST semantic is supposed to make this more explicit.
A remote object is not an object in your program or even your computer. It's something you're taking from something that is computationally miles and miles away. Compared to the microseconds it takes to dispatch a local call, it's an eternity away, even on a local network of the highest speed.
Accumulate those latencies over thousands of dispatches, and trouble can ensue.
I am reminded of an observation from when I first joined Google, coming out of their acquisition of the scrappy awesome ads company I worked at (Admeld).
We had a little service that kept track of ad impression caps / budgeting. We didn't want to serve an ad a single time more than the customer wanted us to, etc. Serving many thousands of ads per second, a process distributed across multiple machines in multiple data centres, this is a bit of a tricky shared state problem. The people who came before me had designed a rather clever solution which used a form of backoff to trickle down the number of ads served as they got closer and closer to budget cap, and to synchronize this state across clusters (this was before there were Rafty services to make this kind of thing easier, BTW).
We had a bit of a show and tell with Google when we first joined. I wanted to know how they were handling this problem, since their scale was many times ours, so I asked the question and got a puzzled/annoyed look:
"Oh we just make an RPC call to our budgeting server."
Summary: if you're at Google you don't have to worry as much about these problems. You still do, but there's an insane amount of infrastructure and horsepower and an army of SREs to help make it happen.
So, yeah, my point is -- just because Google does something or has invented something doesn't mean it's the best way to do it, especially in a smaller more cost conscious organization.
What I don't see though, is how is making a (g)RPC call any different from making a REST call? Like you said, the REST call is supposedly more explicit, but at the end of the day, it seems more of a convention and some hard underlying difference. What's the difference between `httpClient.get("...")` and `grpcClient.foo(...)`?
I mean, internally at Google we have load balancing for grpc (I'm sure the outside world does now, too, but it was new to me when I joined) -- but load balancing HTTP requests containing readable JSON or XML documents, that's far more sysadmin friendly, wouldn't you say? Off the shelf infrastructure, nothing exotic, easier to monitor. Same goes for caching, for proxies, etc.
Being able to just stick a URL for a given resource in your browser, or hit it with wget/curl to read it, that's a serious bonus.
In general URLs follow conventions similar to those down by our Unix forefathers, when they designed the filesystem API. We are all familiar with this model. And in some ways REST done right is very Unix philosophy -- provide a consistent model upon which a bunch of little tools can interoperate.
I could go on... have to go put my daughter to bed tho
I don't think that's accurate... as the article claims (and it seems believable) it has only appeared in etcd because of Xooglers interfering with the project in the name of Kubernetes (also from Google). Or do you actually have examples of gRPC being used in many other non-Google-related projects?
I didn't like having to move cl machines to fcos but it really wasn't that bad and I still get coordinated, active/passive upgrades. ¯\_(ツ)_/¯
> 1) Add hundreds of new failure modes to your software
In my entirely anecdotal experience, it removes error modes. It turns out that just because k8s offers a feature (it offers many!) doesn't mean you're required to use it.
> 2) Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML
Portability, put another way, is a requirement to maintain n different deployment mechanisms because you have no meaningful control over the system on which your software is deployed. This is unavoidable depending on what it is you're deploying and where, but for those of us who have the ability to make those decisions the old notion of portability described by automake and others is actually a bad contract to agree to.
That said, the YAML but is horrible and should be abolished. This is a great criticism.
> 3) Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking
The author isn't considering that I am not, and never will be, Google's entire SRE team. I consider it a general win to be able to apply some subset of their encoded knowledge to the problem of keeping my services running with minimal downtime even if you pick up some k8s-specific baggage along the way.
YAML 1.1 (the most commonly implemented version) also lacks a great way of declaring sets (i.e. unordered collections of unique values).
I actually quite like most of the language a lot but the corner cases can be quite complicated.
For real world Kubernetes needs to be addressed I'd argue we need Dhall or something with Dhall-like semantics and goals to have first class support.
If you use Helm, writing and validating schemas for Helm value files is trickier, and not something I believe anyone is doing. It should be easier to do. Ideally you should simply put this at the top of a value file:
But from what I know, there's no editor/extension, that supports validating through such a declaration. Secondly, Helm doesn't have a way to statically analyze value access. Something like {{ .Values.missTypedKey }} will only be caught when you generate the manifests.Nothing at all. Until you start 'templateizing' it (looking at you, Helm).
It's definitely better than JSON (it can even have comments, imagine that). Most people don't bother reading the spec or even examples though, and don't realize how good it actually is.
k8s API supports JSON almost exclusively, basically one could write in Whitespace then convert to it.
This scourge affects JSON too, such as Azure's ARM templates, which also crowbar logic into the format. An abomination!
I don't understand why you would want to directly edit markup language without parsing it. YAML's goal is to represent a data structure so that a human can interact with it. It accomplishes that goal well. If you want to edit that data structure programmatically- no markup language makes sense to use, represent the data structure using your programming language's native representations.
It's a bash script though.
Example json:
If have a string, and I use that in the middle of a YAML template, I expect a string — I don't expect "300" to become (integer) 300; give me a function for that conversion.
Any HTML templating system that behaved this way would be frowned upon due to the potential for XSS. There's little risk of XSS here, but it's still annoying to get the wrong results.
matchExpressions:
Rather than, if (app=="myservice")But that's not how it's used in Kubernetes. It's used as a declarative DSL that you use to specify your deployments, and it is HORRIFIC. Extremely nested things where the equivalent actual code to do that would be shorter. Things that are inexpressible. Backwards-incompatible changes to your YAML that actually make it impossible to deploy changes that bring your YAML up to spec if your cluster's master node upgraded first.
YAML gives nothing here. jsonnet really isn't a drastic improvement, IMO, except that it lets you express some repetitive parts without having to figure out how many times a template engine is going to indent your YAML.
This wouldn't be a problem if YAML required that all strings be quoted...but it doesn't, and there's no "strict" mode to force quotes, so users are extremely likely to end up doing something that behaves unexpectedly.
If you're lucky, you'll spot the problem quickly instead of wasting half a day on it.
Compare roughly equivalent Compose/Swarm and k8s configs, and you'll see that the k8s one is about 8x bigger.
it's designed to give outsiders something better, based on learned experience of where the toil and cocked-foot-guns are working with Borg. Borg is, by modern standards, extremely legacy infrastructure and even Google is trying to replace it with something better internally.
Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.)
The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale.
Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.
Hmm, I thought you could specify larger node cidr at cluster creation time so you can have >254 ip addresses on each node? The default is /24 but you should be able to specify a larger block, unless you're on GKE (not sure if you can do that on GKE).
The network provides simple and easily controllable ip allocation. You don't need to do any of it, just offload it to AWS/GC/Azure.
Slam another network card/VLAN, give it a seperate ip range from the control network, and everything is handled for you, quickly, securely and efficiently.
If you're AWS, Azure, or Google, this is a feature, not a bug. It makes it easy to sell k8s-as-a-service.
Hand rolled APIs are easier to understand, but harder to maintain. It's great if you're only ever going to have one client, but once you need more than one, it's sure tedious to write and rewrite it for every language you want to support. Using gRPC means that you can auto-generate the client, and while they might not be as wonderful as writing each one of them by hand, at least you can get a client for whatever language you're using. And, the clients all behave the same way -- trying to figure out how to add interceptors to every bespoke client you need is quite tedious. (Look at how long it took AWS to get contexts in go, or how hard it is to add OpenTelemetry to the random hand-rolled HTTP client, etc. With gRPC, you just do those things once!)
Using protos as the transport layer lets you make backwards-compatible changes smoothly; adding fields is safe, renaming fields is safe, etc. The same is not true of using JSON -- if you call something "foo", you can't just one day rename it to "bar". Clients won't know what "bar" is. So you have to update clients and servers at the same time, and you can never "make before break". You see this all the time when someone rolls out a client/server update for a browser app -- your browser cached the Javascript, and it can't talk to the server anymore until that cache expires. It's nasty. I don't understand why people do that to themselves.
gRPC also adds defined semantics for TCP connection length; with HTTP/1.1, maybe you can reuse your connection, maybe you can't, it depends. You can't have multiple requests in flight on the same connection, even if you can reuse it. HTTP/2 fixes this, but gRPC has first-class channels and behavior is well-understood for request/response, streams, etc.
It is unfamiliar and not as easy to debug on the command-line as "curl http://api.example/foo", but once you get up and running, easy things are easy and hard things are possible.
As for Kubernetes, I dunno, it hasn't been bad for me. I tend to use the managed offerings, so I don't have to spare 5 machines for an etcd cluster / masters, or maintain them. I build a container and Kubernetes ensures that it runs forever. If it dies, it's restarted. If more replicas are added, they start receiving traffic. I can manage 100% of the configuration in Git, so if my cluster or cloud provider blows up, I can re-apply somewhere else and have a 99.9% chance of it all working within 15 minutes. Before containers and k8s, production felt very much like a "yolo" thing to me. Most of the world set up some VPSs, logged in, configured them, and prayed that everything would work well. Your website went "down for maintenance" every time you did a release. You needed to distribute root credentials, hoping that you could fully trust everyone on your team to not mess anything up. It mostly worked, but through sheer brute force rather than any system working behind the scenes to make things run smoothly. With k8s, you can delegate this tedium to software. A developer can update a config file, have the PR approved, and software will ensure that the new version of the software starts, is assigned some load, and the old version is shut down. It's smooth, hard for someone to manually mess up, and quite productive.
I get that people have made their own ad-hoc orchestration an...
I've been trying to figure that out for a while.
Think your analysis is spot on overall.
The author strikes me as the person who was saying "why can't I just keep writing Assembly?" in the 90s .
The answer is you can keep using <X technology at a lower abstraction level>.
But don't be mad when people are using the new abstraction layer to build interesting stuff because they don't have to worry about the lower level as much.
that's not really true, configuration management on mutable systems was pretty widespread before everyone started doing immutable infra.
> cancerous user-hostile protocols of HTTP/2
I know not to take the article seriously. I'm simply not interested in hearing from someone with that kind of attitude.
Oh for crying out loud. This Kubernetes bashing is getting so old.
Frankly, I think it's the best thing that has happened in the past few years. Boohoo it requires some YAML. Yeah. And then you get lots of value out of the box.
Systemd bashing is a bit more deserved, but this horse has been dead for a while now.
And there's also some CoreOS bashing too. CoreOS is/was spetacular.
I thought this was moderately-funny cheeky banter, but I wanted to see what implementation decision they were making fun of with this silly misrepresentain. Quote RFC 7541: "The static table was created from the most frequent header fields used by popular web sites." Oh.
https://www.keycdn.com/blog/http2-hpack-compression
I think the unnoticed problem is that the tailors (old school system administrators and dev ops) are being asked to become the managers of Old Navy (kubernetes cluster administrators). Their entire skill set is opposed to this new role. I can't really blame them for being frustrated.
To keep stretching my already thin analogy, there are still tailors in this world. However, most people buy their clothes at outlets like Old Navy. Whining about the quality of the clothes and the crummy manufacturing process at Old Navy won't change that.
But down in reality, most companies are probably running their own made-to-measure deployment and operation schemes with lower quality and consistency than Old Navy.
Again, it reminds me of a carpenter friend who hates Ikea. Yeah, I get it, Ikea is really bad compared to custom built furniture. No one who buys it expects anything else.
This story is older than the industrial revolution. Craftsman being replaced by technology. We even instinctually know this is going to happen to knowledge workers but we seem blind to it when it happens to us.