Dozens of people wrote to me, aghast at the idea that I provision servers manually, wondering how I could refuse to use the rube goldberg machines that define the modern "cloud-scale" computing trends. It looks like it didn't take long for my approach to be validated.
Salt is complicated as hell. k8s, Docker, Ansible, they're all the same. Complicated systems break more often and more severely than simple systems. It's a simple fact, and I hope that our industry gets over its collective devops fetish sooner rather than later.
No, Salt is distinctive: it uses resident agents, unlike other tools; infamously, they managed to implement e=1 RSA in their transport protocol. You cannot reasonably extrapolate from Salt to Terraform or Ansible.
I can, and I will. It's not just security that's affected, it's availability, reliability, and maintainability as well, and it fosters a culture which doesn't value any of those things nearly enough, leading to a bunch of hulking monoliths lumbering over to the pile of VC money for dinner. I have no confidence in any of these systems.
A system like Ansible, in particular, doesn't install anything resident on the servers it manages, nor manages state which could be exploited on a management machine. And in one sense, it's like writing a shell script and running it on the server via SSH. Manual provisioning scales well enough for a single server but is much less secure when you scale past one server, or need to be able to rebuild that server in a hurry, like when your cloud provider inevitably deletes it.
Ansible will faithfully deploy any security vulnerabilities you tell it to across your whole fleet, in just seconds! Don't make the mistake of thinking it's immune to errors.
Your servers ought to be simple, too. Once you're simple enough, the problems shrink and shrink until they're barely there. Or, you can go as complex as possible and just hide them in mountains of garbage. "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies."
Okay - I'm intrigued. How do you recommend managing infrastructure which consists of thousands of servers? With k8s - you just delete the pod and it gets recreated with a clean image within a few seconds. What's your proposal?
I recommend not having thousands of servers. Who has thousands of servers? Maybe 10 companies, worldwide, have an actual demonstrable need for that. The rest of the companies are just engaging in devops masturbation.
What about the plethora of companies that provide B2B IT services? If your company’s business is “deploy $insert_products on-premise for each customer”, and you’re doing good business, you’re going to quickly approach thousands of servers.
I was doing some work for a bank last month. It’s a reasonably small bank, and you likely haven’t heard of it. They have thousands of servers. They had over 100 logstash instances alone. All their desktops were virtualized, and that was a couple thousand servers at least.
They have thousands of servers, but should they? I'm sure they must, banks are, after all, world-famous for their technical accumen. It's good to know that my money is as safe as I always suspected it was.
I did suggest to them, for the sake of reliability, that they should move their entire infrastructure to one incredibly large server. Because a single point of failure is easier to maintain (less points that can fail). But sadly, they didn’t listen.
Seriously though, payment infrastructure actually is known for its reliability and fault tolerance. Banking systems don’t go offline terribly often.
Is your reasonably small bank known for its reliability and fault tolerance? The main reason banks don't go offline is because the core critical infrastructure is running on 50 year old mainframes that no one is allowed to touch because all of the greybeards with actual talent who made them are pushing up daisies.
Nowhere did I say you should be on one incredibly large server, nor that you should have a single point of failure. That wouldn't be simple, either, because it would fail to support the prime directive, or would require a great deal of gymnastics to. It's about balance. You don't need thousands of servers to make a reliable system.
> Is your reasonably small bank known for its reliability and fault tolerance?
I don’t know if they have a specific reputation for it. But their services were very reliable and resilient.
> The main reason banks don't go offline is because the core critical infrastructure is running on 50 year old mainframes that no one is allowed to touch because all of the greybeards with actual talent who made them are pushing up daisies.
This isn’t really true at all. The banks central ledger will likely be running on DB2, and that will almost never fail. But most of their infrastructure runs on more “modern” systems (DB2 is still modern if we’re being honest, it’s just more specialized). When you swipe your card somewhere, nearly all of the systems the transaction passes through are ordinary web services, running on ordinary architecture. That’s where the greatest risk of service disruption is. It’s just the final side effect is updating a record in a DB2 database somewhere.
Most banks also outsource a lot of the core system maintenance directly to IBM. A typical software or infrastructure engineer in a bank will have absolutely no contact with those systems.
> Nowhere did I say you should be on one incredibly large server
No, you just seem to think that reducing the number of components in your infrastructure is the key to reliability. If you want fault tolerance, then you generally want redundancy, and you want your services to fail gracefully (ie, not take down other non-dependant services when they do). Both of those things require deploying more servers.
If points of failure are a concern, then you also need to account for the fact that every single time you make a manual change you are creating a new point of human failure (with humans generally being the least reliable part of any well designed system). If you automate a change, you can spend more time scrutinizing it for mistakes, and be reasonably sure it’s deployed exactly the same everywhere. Combine that with blue/green, rolling or canary deployments, and automated testing, and you end up with something much more reliable than shelling your way across the entire infrastructure to deploy a single change. To suggest that deploying less infrastructure is a superior alternative really just comes across as ludditism.
You might think so, but if that was the reason your bank would be offline a lot. Everything that these old monoliths process come in batches, at predictable intervals.
What makes banking systems reliable is that there's so much built in latency in every operation that nobody really notices if something is down for six hours.
> What makes banking systems reliable is that there's so much built in latency in every operation that nobody really notices if something is down for six hours.
This is true for settlement, but not authorization. When you spend your money it could be hours or days until the transaction is correctly reflected in your balance. But in order to spend it in the first place the transaction has to be authorized, and this happens in real time. Depending on where you live, who you bank with, what kind of card you’re using, how you’re using it... this authorization may rely heavily on your bank being available for online transaction processing, or not very much. But unless you’re doing a transaction with one of those imprint devices, your transaction must be authorized in real time by somebody.
I sit close by to the people that deal with this at work, so my understanding is very limited. Payment processors, service providers, reconciliation, yatta yatta. We used to joke around with them about carrying cash, just in case. We stopped joking around when we realized they all did.
Payment processor could mean a number of different things. It could mean a tiny gateway operator. The sort of thing that can fail quite happily without impact transaction processing (most services you use will be backed by redundant gateways).
I carry some cash too. But the times I’ve needed it were for things like an outage of the internet connection at a restaurant, or an issue with a payment terminal provider (or for the very uncommon cash-only merchant).
> Is your reasonably small bank known for its reliability and fault tolerance? The main reason banks don't go offline is because the core critical infrastructure is running on 50 year old mainframes that no one is allowed to touch because all of the greybeards with actual talent who made them are pushing up daisies.
> Nowhere did I say you should be on one incredibly large server, nor that you should have a single point of failure. That wouldn't be simple, either, because it would fail to support the prime directive, or would require a great deal of gymnastics to. It's about balance. You don't need thousands of servers to make a reliable system.
Heh those things go offline every night for 2 hour maintenance.
Also fun when those nice single points of failure crash (they do)
Source: work at a shop trying to _get out of_ greybeard mainframe to get more reliablity
Most banks have a window of a couple of hours per night where access to the core system is restricted for settlement. It's kind of a necessity of the business model. However most banks (including every bank I've ever worked at, which is quite a few), don't shut down services during that time. ATM, credit and debit cards, everything else... All those systems run all the time. They just have an eventual consistency model. The compromise that the banks tend to make, is that certain types of fraud are more tolerated during that window, rather than sacrificing availability.
Oof. I can't see how this is anything other than a huge assumption to think that you know better than each and every business that has thousands of servers. The smartest people I know generally assume the least of situations they are not intimately familiar.
How large is the company you work for (employees or revenue)? How many servers do you have to support that company?
> How large is the company you work for (employees or revenue)? How many servers do you have to support that company?
“SourceHut [of which ddevault is the founder] has 10 dedicated servers and about 30 virtual machines, all of which were provisioned by hand... [It] made 4 thousand dollars in Q1 with two employees and an intern.” (from https://sourcehut.org/blog/2020-04-20-prioritizing-simplitit...)
> Oof... The smartest people I know generally assume the least of situations they are not intimately familiar.
Yeah, I generally think highly of Drew’s opinions on technical matters, but this thread is just painful to read.
The thing I don’t like about the sorts of claims made here is that when people advocate for simplicity like this, they often just have a highly opinionated perspective of what should be simple. So they’re not really advocating for simplicity, they’re just advocating for prioritising whatever thing they think it is that’s important. Maybe that thing should be important, but it usually depends on the context, so you end up with bad general purpose advice, which usually doesn’t accurately portray the trade-offs it’s advocating.
Heck, IMO even one server is risky to manage manually. If you're making all your config changes from a prompt it's easy to forget how or why you set something up a specific way and end up with a huge pain when you need to do it again. A config management tool gives you executable documentation for exactly how the server is set up should you ever need to recreate it from scratch.
I tried that, but I'm not sufficiently superhuman. Things slowly but inexorably drift out of sync as changes get made in one place but not the other, despite my best efforts. Declaring how things should be and then letting the computer make it so is both less work and more reliable. (This is why I call it "executable documentation".)
Tell that to companies doing extremely large-scale machine learning. Or any cloud infrastructure provider. Or CDNs. Or literally any video production company that owns a render farm. Or any company doing large-scale media transcoding/streaming.
Maybe "don't have thousands of servers" is just a bad take :)
Many companies operate at a large scale. If your company or your product doesn't.... That doesn't mean your way is the correct way for anyone.
You haven't caught on that no one has said automation replaces testing or other processes but that seems to be the drum you're banging on. Ansible or whatever tool doesn't replace testing, monitoring, pentests, whatever else you want to do. Automation lets you fix the issues faster and replace servers faster and more reliably, once you have done the testing that you have to do either way.
At a large Australian bank I contracted to about 10 years ago, it was suspected that they had upwards of 13,500 servers, including many mainframes. They had so many servers that there were more than a few just sitting uncollected and still packaged in the storage vault, abandoned several years ago when the project they were intended for was cancelled/put on hold etc. Nobody every claimed them.
Another part of the problem is that in the past they invested in every platform under the sun that ran the financial applications they required, so there was a lot of legacy platforms that they couldn't yet rationalize away (AIX, Solaris etc).
The other problem is that they are incredibly risk averse, and thus will build new platforms extremely slowly and only after immense testing will they cut over from the old platform to the new platform. They are typically averse to consolidation or rationalisation since the ongoing stability of the application is the most important factor to their business.
A virtualization umbrella project that I was working on 10 years ago for this bank that covered several "core" platforms (AIX, VMware, Solaris, etc) is still under way as we speak, with many applications still to be migrated from old platform to the new one, despite the original "replacement" hardware now being obsoleted by the manufacturer already, according to those I know that still work there. I wonder if they'll ever finish.
Last company I worked at, the application stack, after lots and lots of trimming, and memory tweaking, ran 14 servers. Hot Failover doubled that, Disaster recovery ran 3x. So, that was 42 servers per customer. We had 40+ customer - so, at last count, were at 1600+ servers, just for production applications - not including development, marketing, sales, finances, etc... And it was just a run-of-the-mill data ingestion company - nothing special. 1600 servers is only 80 racks - that's a small data center as things go - there are hundreds, if not thousands of companies with data centers that large.
Now, at my current gig - we have 43 nodes - but each node can run 100+ k8s pods (and, they are beefy 768 GB nodes). Our administration overhead, in terms of people/effort/auditing is a fraction of what it was at the last gig. Also - Disaster and failover is just a matter of bringing up the clusters and persistent stores in another region. No need to keep compute on standby. This week, we're even doing a migration of a cluster from GKE to Azure. And it started out on AWS (though not in production) - I'm not in the sysadmin world anymore, but from what I've seen, things like k8s have made it a heckuva lot easier to do big things with fewer people.
Well, the sort of technique in https://news.ycombinator.com/item?id=23071459 long pre-dates things like that (but not sophisticated resource managers, of course) before we knew it was so difficult.
You will also faithfully deploy any security vulnerabilities you decide to, during your manual deployment process. I’m not sure how it is that you think you’ve dunked on Ansible here.
Deploying your less-complicated design consistently through automation reduces complexity; it doesn’t increase it. I don’t think we are operating from similar definitions of “mathematical fact”.
Some automation is good, but no one reaches for these tools because they need "some" automation. They do it because they have already obscenely over-complex applications which require these levels of complexity just to stand them up. Or worse, they don't have that level of complexity, but they use these tools anyway, which means that they account for more than half of the software's complexity in the first place.
"Some automation is good" is not a statement compatible with (here I paraphrase) "tools that run a consistent set of shell commands on an arbitrary set of machines are the same as tools that install resident agents on those machines". It's OK if you didn't know how Salt (or Ansible, maybe) worked, but you should just acknowledge that and move on rather than doubling down.
Don't resort to acusations of ignorance just because you aren't making your point effectively. I have hands-on experience with every tool I've mentioned except for k8s, which isn't being discussed in detail. I am under no illusions that the security profile of Salt vs Ansible is different. If that's all you're getting at, then fine, I concede, do your little victory lap.
My point is and has been much broader: these tools solve the same class of problems which are symptoms of a broken culture, and solve them poorly. It's no surprise to me that any and all of them have problems, have had problems, and will have more problems - because they're broken by design, and they "solve" problems that we shouldn't have.
Well, I can manage a system running hundreds or thousands of heterogeneous stateless nodes consistently from an image essentially simply managed "by hand" (maybe with judicious help from things like package auto-updates). Otherwise I could add something like Ansible with its configuration -- probably rather larger -- similarly managed by hand. Obviously the latter increases complexity and scope for problems; observationally it also decreases reliability and availability.
Substantial experience says the simple way is the best to manage HPC systems, in particular, and it generalizes, like to managing teaching "clusters" with few resources[1]. There are minimal trades-off from just using things you already know -- package management, revision control, and initrd/pivot_root -- against using tools which are distinctly non-trivial, but make you look good for solving the extra problems and justify more manpower. I've used a cluster-building system from a maintainer and contributors with Ansible expertise and been unpleasantly surprised, if not as much as by several cluster vendors' products.
Of course you automate things like monitoring, alerts, and some problem mitigation, for which simple configuration management frees effort. Accidental and essential complexity, someone once talked about.
I feel like the resident agent thing is the wrong focus.
Goofing RSA is a more troubling sign, because it means there probably isn't anybody who understands how this stuff works in the room where plans to use it are made, and that means there will be security bugs forever.
Presumably other people don't agree with me or else there wouldn't be important stuff running on Salt?
I don't even know if they still use the transport protocol they came up with. I remember more clearly the Red Hat security guy claiming, on the same Github issue thread, that e=3 RSA was dangerous because of GPUs, which is almost an equally fucky thing to say.
The agent thing, though, would squick me out almost no matter which team implemented it.
Isn't it fairly common for config management tools to use agents? Chef and Puppet both do.
Once your configuration management is tied into your CI/CD pipeline, I don't really see much of a difference between an agent running on each machine vs a master node that has a root ssh key into your whole fleet.
And sshd is a fairly common thing to fall over in my experience (though particularly on badly-managed clusters). Presumably more so on otherwise-inaccessible systems, by the laws of life. The agent saved systems isolated in places like x-ray hutches a number of times back when I thought using cfengine was a good idea.
Or you just don’t expose these services externally. Even with Ansible, why open up sshd to publicly facing interfaces. Use a jumpbox or VPN. Why increase your threat surface?
I agree that keeping a system as simple as possible is a good idea. But, you may want to take a step back and consider why (as of this writing) every single comment from you—and nobody else’s—is grey in this thread. (It’s not often on HN that you see two luminaries such as you and 'tptacek get into such a vigorous and apparently one-sided debate.)
I was going to attempt to summarize both sides of the argument here, but instead I think it’s better if I just suggest that you take some time to attempt doing that yourself and try to understand the points that everyone else is trying to make. I don’t think it’s your point about excess complexity that’s garnering the downvotes (at least, I hope it isn’t), but rather (a) that you’ve tarred a very broad swathe of software with exactly the same brush, refusing to acknowledge the significant differences in design, implementation, and use between such diverse softwares as Kubernetes and Ansible; and (b) the dogmatism about there being no conceivable need for such any such tools under any circumstances whatsoever.
I’d be very interested in a blog post (if one hasn’t been published already) about how system management is done at Sourcehut. A lot of people (myself included) turn to the tools you’re disparaging to help them manage things like change tracking and testing server configurations before applying to production, and it would be interesting to read about how you manage these sorts of problems without introducing a tool to do it.
I just received a notice that many CentOS 6 systems managed by RamNode were also compromised; it seems like they were customer instances, but in that case, the malicious parties installed BTC miners on all the servers, and one of my tiny VPSes was collateral damage for a couple hours, as they worked to contain the situation.
I came at this article from the angle that "salt" referred to the cryptographic type, but it's actually some configuration management framework thingie. So there is not a cryptographic vulnerability in CT, but rather one particular log was compromised because of the software they use to run their servers. Very confusing.
This doesn't impact on the security promises CT gives us in practice because it's just one log and all extant CT policies require multiple logs.
However some of the unfinished work on CT means that we're more vulnerable here than would be ideal.
For example: An adversary who does have the SCT private key can make any SCT. So they can make one which says this log saw a certificate last month, before their compromise, because that's just a date change and they get to pick what is signed. This claim would be untrue, but nobody would automatically notice that.
In theory the complete CT system could detect this SCT - at least probabilistically - because clients which are shown the SCT could periodically check whether a log consistent with that SCT actually exists and report mysterious SCTs that don't match. But that isn't yet built, a Chrome which is shown an SCT from a trusted log promising the certificate was logged last month just takes that as gospel truth.
Now for Google this is pretty satisfactory because their CT policy requires that you use their (Google's) logs. But for a third party like Apple it's a bit less comfortable. If the Google logs suffered a security breach, Apple has no reassurance that they would find there was a problem short of Google voluntarily letting them know.
54 comments
[ 3.0 ms ] story [ 85.8 ms ] threadhttps://sourcehut.org/blog/2020-04-20-prioritizing-simplitit...
Dozens of people wrote to me, aghast at the idea that I provision servers manually, wondering how I could refuse to use the rube goldberg machines that define the modern "cloud-scale" computing trends. It looks like it didn't take long for my approach to be validated.
Salt is complicated as hell. k8s, Docker, Ansible, they're all the same. Complicated systems break more often and more severely than simple systems. It's a simple fact, and I hope that our industry gets over its collective devops fetish sooner rather than later.
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ansible
Your servers ought to be simple, too. Once you're simple enough, the problems shrink and shrink until they're barely there. Or, you can go as complex as possible and just hide them in mountains of garbage. "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies."
Seriously though, payment infrastructure actually is known for its reliability and fault tolerance. Banking systems don’t go offline terribly often.
Nowhere did I say you should be on one incredibly large server, nor that you should have a single point of failure. That wouldn't be simple, either, because it would fail to support the prime directive, or would require a great deal of gymnastics to. It's about balance. You don't need thousands of servers to make a reliable system.
I don’t know if they have a specific reputation for it. But their services were very reliable and resilient.
> The main reason banks don't go offline is because the core critical infrastructure is running on 50 year old mainframes that no one is allowed to touch because all of the greybeards with actual talent who made them are pushing up daisies.
This isn’t really true at all. The banks central ledger will likely be running on DB2, and that will almost never fail. But most of their infrastructure runs on more “modern” systems (DB2 is still modern if we’re being honest, it’s just more specialized). When you swipe your card somewhere, nearly all of the systems the transaction passes through are ordinary web services, running on ordinary architecture. That’s where the greatest risk of service disruption is. It’s just the final side effect is updating a record in a DB2 database somewhere.
Most banks also outsource a lot of the core system maintenance directly to IBM. A typical software or infrastructure engineer in a bank will have absolutely no contact with those systems.
> Nowhere did I say you should be on one incredibly large server
No, you just seem to think that reducing the number of components in your infrastructure is the key to reliability. If you want fault tolerance, then you generally want redundancy, and you want your services to fail gracefully (ie, not take down other non-dependant services when they do). Both of those things require deploying more servers.
If points of failure are a concern, then you also need to account for the fact that every single time you make a manual change you are creating a new point of human failure (with humans generally being the least reliable part of any well designed system). If you automate a change, you can spend more time scrutinizing it for mistakes, and be reasonably sure it’s deployed exactly the same everywhere. Combine that with blue/green, rolling or canary deployments, and automated testing, and you end up with something much more reliable than shelling your way across the entire infrastructure to deploy a single change. To suggest that deploying less infrastructure is a superior alternative really just comes across as ludditism.
What makes banking systems reliable is that there's so much built in latency in every operation that nobody really notices if something is down for six hours.
This is true for settlement, but not authorization. When you spend your money it could be hours or days until the transaction is correctly reflected in your balance. But in order to spend it in the first place the transaction has to be authorized, and this happens in real time. Depending on where you live, who you bank with, what kind of card you’re using, how you’re using it... this authorization may rely heavily on your bank being available for online transaction processing, or not very much. But unless you’re doing a transaction with one of those imprint devices, your transaction must be authorized in real time by somebody.
I carry some cash too. But the times I’ve needed it were for things like an outage of the internet connection at a restaurant, or an issue with a payment terminal provider (or for the very uncommon cash-only merchant).
> Nowhere did I say you should be on one incredibly large server, nor that you should have a single point of failure. That wouldn't be simple, either, because it would fail to support the prime directive, or would require a great deal of gymnastics to. It's about balance. You don't need thousands of servers to make a reliable system.
Heh those things go offline every night for 2 hour maintenance.
Also fun when those nice single points of failure crash (they do)
Source: work at a shop trying to _get out of_ greybeard mainframe to get more reliablity
How large is the company you work for (employees or revenue)? How many servers do you have to support that company?
“SourceHut [of which ddevault is the founder] has 10 dedicated servers and about 30 virtual machines, all of which were provisioned by hand... [It] made 4 thousand dollars in Q1 with two employees and an intern.” (from https://sourcehut.org/blog/2020-04-20-prioritizing-simplitit...)
> Oof... The smartest people I know generally assume the least of situations they are not intimately familiar.
Yeah, I generally think highly of Drew’s opinions on technical matters, but this thread is just painful to read.
Anyone whose properly managed more than a few servers realizes and understands this.
The correct way would be to just document the changes.
Maybe "don't have thousands of servers" is just a bad take :)
You haven't caught on that no one has said automation replaces testing or other processes but that seems to be the drum you're banging on. Ansible or whatever tool doesn't replace testing, monitoring, pentests, whatever else you want to do. Automation lets you fix the issues faster and replace servers faster and more reliably, once you have done the testing that you have to do either way.
Another part of the problem is that in the past they invested in every platform under the sun that ran the financial applications they required, so there was a lot of legacy platforms that they couldn't yet rationalize away (AIX, Solaris etc).
The other problem is that they are incredibly risk averse, and thus will build new platforms extremely slowly and only after immense testing will they cut over from the old platform to the new platform. They are typically averse to consolidation or rationalisation since the ongoing stability of the application is the most important factor to their business.
A virtualization umbrella project that I was working on 10 years ago for this bank that covered several "core" platforms (AIX, VMware, Solaris, etc) is still under way as we speak, with many applications still to be migrated from old platform to the new one, despite the original "replacement" hardware now being obsoleted by the manufacturer already, according to those I know that still work there. I wonder if they'll ever finish.
Now, at my current gig - we have 43 nodes - but each node can run 100+ k8s pods (and, they are beefy 768 GB nodes). Our administration overhead, in terms of people/effort/auditing is a fraction of what it was at the last gig. Also - Disaster and failover is just a matter of bringing up the clusters and persistent stores in another region. No need to keep compute on standby. This week, we're even doing a migration of a cluster from GKE to Azure. And it started out on AWS (though not in production) - I'm not in the sysadmin world anymore, but from what I've seen, things like k8s have made it a heckuva lot easier to do big things with fewer people.
My point is and has been much broader: these tools solve the same class of problems which are symptoms of a broken culture, and solve them poorly. It's no surprise to me that any and all of them have problems, have had problems, and will have more problems - because they're broken by design, and they "solve" problems that we shouldn't have.
Substantial experience says the simple way is the best to manage HPC systems, in particular, and it generalizes, like to managing teaching "clusters" with few resources[1]. There are minimal trades-off from just using things you already know -- package management, revision control, and initrd/pivot_root -- against using tools which are distinctly non-trivial, but make you look good for solving the extra problems and justify more manpower. I've used a cluster-building system from a maintainer and contributors with Ansible expertise and been unpleasantly surprised, if not as much as by several cluster vendors' products.
Of course you automate things like monitoring, alerts, and some problem mitigation, for which simple configuration management frees effort. Accidental and essential complexity, someone once talked about.
1. http://workshop.openafs.org/afsbpw09/thu_2_1.html
Goofing RSA is a more troubling sign, because it means there probably isn't anybody who understands how this stuff works in the room where plans to use it are made, and that means there will be security bugs forever.
Presumably other people don't agree with me or else there wouldn't be important stuff running on Salt?
The agent thing, though, would squick me out almost no matter which team implemented it.
Once your configuration management is tied into your CI/CD pipeline, I don't really see much of a difference between an agent running on each machine vs a master node that has a root ssh key into your whole fleet.
I was going to attempt to summarize both sides of the argument here, but instead I think it’s better if I just suggest that you take some time to attempt doing that yourself and try to understand the points that everyone else is trying to make. I don’t think it’s your point about excess complexity that’s garnering the downvotes (at least, I hope it isn’t), but rather (a) that you’ve tarred a very broad swathe of software with exactly the same brush, refusing to acknowledge the significant differences in design, implementation, and use between such diverse softwares as Kubernetes and Ansible; and (b) the dogmatism about there being no conceivable need for such any such tools under any circumstances whatsoever.
I’d be very interested in a blog post (if one hasn’t been published already) about how system management is done at Sourcehut. A lot of people (myself included) turn to the tools you’re disparaging to help them manage things like change tracking and testing server configurations before applying to production, and it would be interesting to read about how you manage these sorts of problems without introducing a tool to do it.
However some of the unfinished work on CT means that we're more vulnerable here than would be ideal.
For example: An adversary who does have the SCT private key can make any SCT. So they can make one which says this log saw a certificate last month, before their compromise, because that's just a date change and they get to pick what is signed. This claim would be untrue, but nobody would automatically notice that.
In theory the complete CT system could detect this SCT - at least probabilistically - because clients which are shown the SCT could periodically check whether a log consistent with that SCT actually exists and report mysterious SCTs that don't match. But that isn't yet built, a Chrome which is shown an SCT from a trusted log promising the certificate was logged last month just takes that as gospel truth.
Now for Google this is pretty satisfactory because their CT policy requires that you use their (Google's) logs. But for a third party like Apple it's a bit less comfortable. If the Google logs suffered a security breach, Apple has no reassurance that they would find there was a problem short of Google voluntarily letting them know.