I see this sentiment all the time, but it could never apply to the defense sector. Do you really want our nations enemies to have access to the source code of fighter jets, cruise missiles, the nuclear weapons program, or missile defense systems?
how are those systems going to be secure without being audited by the market?
the reality is that this military industrial complex is just a way to print money and funnel it into the hands of a few criminals. all of the major news publications openly told you that the USA has been attacked over 100 times in the middle east AND WE LEFT. The public is going to find out very soon that this massive, expensive military can't actually defend against or defeat other militaries. That when you spend $1000 on a hammer everything looks really flashy and impressive AT FIRST with your fancy jets and missile systems, but after enough time, and after enough of the money has been funneled out, you realize it's a paper army. the USA is so weak that Iran is pushing them out of the middle east and there's nothing the theives that run the military can do about it. but don't worry. they already manufactured a Palestine sympathy story ahead of time to explain why they have to pull out and run away like the cowards they are. and your son's and neighbors sons will pay the price with their lives
How are you equating a "code review" to open source, the difference is the difference between asking a couple dudes for advice and asking the entire world for advice. I'm sure the argument sounded really good in your head, but let's be adults here when it comes to the United States military
Because auditable does not mean the same thing as audited. It is silly that people keep pushing that dead argument after Heartbleed pounded a stake through its heart. Audits are time consuming, challenging, and boring. Experience shows even critical, high profile projects hardly get any review at all by the "world", let alone actual audits by competent domain experts.
Quality verification depends on auditing and auditing depends on competent, trusted review and testing. Global transparency is not a substitute for auditing except when discussing absolute rock-bottom standards. Reviewable is significantly better than unreviewed and unreviewable, but that is the lowest possible bar. Unfortunately, software does, as a general rule, have rock-bottom auditing standards, so FOSS does provide meaningful assurance increases in many use cases. But, that is a artifact of the abysmal quality standards rather than any sort of inherent auditing advantage that FOSS provides. That is not to say that global transparency is not valuable for other reasons, but it provides no meaningful quality verification or auditing advantage in serious applications versus local transparency (to the trusted reviewer).
If you want to see how this works in practice, look at literally every other industry.
the market will determine what should and should not be audited, according to supply and demand. thinking that a couple employees that were not paid for with profit but with printed and taxed government dollars is not and never will be a substitute for the free market, which is a synonym for humanity. not respecting that fact will always result in loss and inefficiency. and no amount of paragraphs can undo that.
The market has been horrid in determining that. There are a few dozen massive companies funding a significant amount of the menial and boring work done on FOSS. Possibly because they have more to lose from their competitors building moats and wide-scale incidents.
Thankfully governments are now also funding FOSS work but this really doesn't align with what you're envisioning, I think.
Open Source != GPL. Nor on Github. I don't know what the parent fully meant, but I do know that some agencies have full access to some Microsoft source code.
From where I sit, I find it absurd that everything we use isn't open source. Again, not free, but the code given to purchasers.
You don't need closed source to protect IP, and the proof is in all of these API lawsuits, and copyright law.
I don't want my missiles to not have code I cannot edit, and stepping back from the top secret sphere, tangential I am appalled at how crappy car firmwares are closed source.
In the old days, a country's national transport agencies could look at evey linkage, every rod, every part of a car design.
Now 90% of the design is hidden. And with electric cars, it's even more firmware.
And the idea that OTA updates are a thing for cars. The madness. The absolute madness.
How much do you want to bet that charging firmwares are remotely updatable?
Now imagine that 9am on Monday, every electric car explodes?
Even today, that would mean an immense number of houses on fire. How could the fire department handle it?
And how could it be handled as the fires spread? And what if lots of other infra goes up?
And that's today. What about when 90% of transportationn is electric? Even if not a single house or building burned, or person was hurt,
how would you replace all those buses, trucks, delivery vehicles, and cars? As COVID showed us, you cannot ramp up and down overnight.
And what if it happened to all our allies? Would they still sell part to us?
What of all the tractors are electric, and we miss corn and wheat planting season?
My point in this tangent is... no one is even looking at the important bits. And to reiterate, how much do you want to bet charging firmwares are remotely updated? Really, they should be air gapped from the entire rest of the car.
Letting potential hackers have access in this way, is just plain lazy and reckless.
Nobody said random users should be able to edit. FOSS means the code is available, not that they're going to take patches. (See sqlite for an extreme case - code is public domain, but they more or less don't take contributions)
> I don't want my missiles to not have code I cannot edit, and stepping back from the top secret sphere, tangential I am appalled at how crappy car firmwares are closed source.
I read that as that whoever's missiles they are should have full visibility and control of their code; I am assuming that "my missiles" means the government, not that the poster personally owns missiles.
Code available means everyone else has a leg up developing their own weapon system or finding flaws in yours. You don't really want adversaries looking through the source code of your targeting and guidance systems.
> I do know that some agencies have full access to some Microsoft source code.
Access does not mean open source. Open source = OSI and what I think GP meant. And I hold the sentiment that government funded development should be open source.
I specifically said "and other licenses", so why would you respond, as if I acted as if GPL was the only license?
No matter. One entity doesn't get to unilaterally redefine a century old term. You claim you want precision, well then specify OSI, a subset of open source.
(And yes, the licenses it approves are indeed a subset.)
One entity is the de facto organization that I, and many others trust to shepard this important definition for society.
Using precise language is useful for building software (and in general life).
I like that OSI exists and helps people understand a specific and desired definition for open source.
There’s a lot of thought on this topic by really smart and skilled people. The aspects of open source that I think are most important are captured in OSI.
For me, I don’t care if I can see the source, if it’s not free and usable. I can decompile code and patch for myself without a license. And I can pay for code in escrow.
But what’s important is that I can modify and redistribute. And that I can be part of a virtuous community creating things together.
This is actually not a bad idea for an international peace treaty. If you make weapons, they must be open source. If the real power in these tools is the secrecy behind their design and implementation, seems like a great way to suck the power out of them.
This strategy would fall apart when you encounter an adversary willing to sacrifice its own people. Nations that care about their own people need to keep a technological advantage on the battlefield.
Believe it or not, USA government cares at least a little. USA did not institute the draft even when fighting on two fronts (Afghanistan and Iraq). USA made up for the lack in manpower with technology: Reaper drones, cluster munitions, night vision, precise artillery, overwhelming air power.
Don't you find it a bit frightening that this tech is daily giving a smaller and smaller subset of people in the world the unilateral ability to project their will onto the world? At least if they tried to draft in an unjust war, the people could withhold their physical support. This was pretty effective during the Vietnam area, but it's a bargaining chip we've lost.
As we saw with 702, we really have very little leverage left in what our gov does anymore, the threat of being voted out even carries little power, exactly because they're systematically removing humans from every loop they can for the purpose of working unilaterally even when--perhaps especially when--it's against the will of the people. The spooks themselves have literally said exactly as much in their fight against privacy.
To me, the latest 702 is more terrifying than any adversary. When we, who the government should be beholden to, cannot even know the text of the laws which apply to us, it really puts a fine point on how the those who run our government view themselves in relation to its citizenry.
"The liberties of people never were, nor ever will be, secure, when the transactions of their rulers may be concealed from them." -Patrick Henry
Well if the history in the past 3 years is relevant I would say that the power of elites to project their will onto the rest of the world is small. In both sides: Russia is still grinding through a war that may be their second Afghanistan; and on the western side it took US Congress half a year to approve some funds so someone else (Ukraine) would to fight China’s gas station.
From my PoV, much of the Ukraine war was the result of the powers in the US exercising their control over the world, and it sure didn't take long to dispatch two carrier battlegroups to the aid of Israel.
I mean, all treaties are at risk of failure from bad actors, even implicit ones like mutually-assured-destruction, and in the later case it has been shown to be incredibly effective. But I don't feel like grinding the threat of powerful weapons against each other to be a particularly wise long term solution to security.
I think it's a pretty obvious ipso facto that the more, and more advanced weapons, that are placed into the global battlefield, the less actual security (in life and liberty) we can expect.
The problem isn't that there are adversaries that are willing to sacrifice their people, almost every war is one of attrition after all, it's that leaders that would be adversaries like this are inevitable in a governmental structure that is delegated a monopoly on violence.
Open-source doesn't mean "not secure" - the nuclear codes would be in the .env obv >.>
Seriously though - the software itself would be separated from secrets architecturally. And because it's open and anyone in the world can contribute it could be superior code than what some private government tech contractor could come up with.
For the equipment use case like jets and missiles, a separate directive component (driver) would likely be necessary anyway, not just for security/privacy but because different nations have different equipment. We use F-16s, M-16s, Autel drones, etc.
This driver/directive component would be the manufacturer's IP - like an nvidia GPU driver. Think of it like we'd all be running/contributing the same OS but because we have different hardware and security needs there are still some private components.
The idea/goal here is that our defense companies would overall benefit from it. Not just because the software is improved, but you can justify funding, build curricula around it - might benefit the defense industry to modernize their tech practices.
The problem is that not all the secrets are so easily extracted - sometimes the design/software is the secret.
If you put all the design up for nuclear weapons but just kept the nuclear codes secret it's great that no one can fire ours, but people could implement the design on their own with different codes.
To use a more realistic example, consider air defence missile systems use to shoot down incoming missiles and drones. The secrets here aren't keys, it's software-driven behaviours including the approach to identify radar tracks as hostile or noise, identify when to commit a missile and how to target it, the code implicitly contains the vulnerabilities where the radar tracking is less effective and more evadable and how the system tries to mitigate this, etc.
When you take away all the secret behaviours, you quickly end up with not much more than just the drivers connecting the hardware to the logic, which isn't a lot of code that's driving the funding. How you set a missile tubes tilt and direction is trivial stuff and is mostly reused from old platforms that were funded decades ago.
Additionally, there is some talk about how this is kind of like security through obscurity. When it comes to things like weapons and other high-tech capabilities, security isn't only security against being owned, it's security against your opponent closing the technological gap. Unfair wars, where you have a significant tech lead on your opponent, means your population bleeds less.
Decreasingly the case, and we might be at the point I can say "poorly architected" if that's the case. A random 19 year old developer in the military with access to the secrets would be a big security hole too.
That's what we should be comparing with regards to open-source vs not open-source - in either case access to weaponry would of course be heavily gated.
Some of the other arguments you make here are no different than conventional arguments against open-source:
> the real code would still be private
Precisely the point, get the crowd to optimize the low-risk parts, build communities around those libraries and frameworks and recruit from it
> opponents might catch up
To remain an industry leader it's actually better to get everyone playing your game rather than trying to compete and stay ahead in a wild west scenario. You want to capture it really, so you can control it and be the leader in it. Open-source is one great way to do that (browser vendors come to mind).
You can say "poorly architected" all you want, but it's true. Military capability is frequently determined by software-implemented behaviours, not data you can plug into a generic public framework.
>19 year old developer with access would be a big security hole
It's true, they are. That's why militaries and defence companies go to great lengths to vet their staff and why even within vetted staff, sensitive material is compartmentalised to minimise the risk from any given individual. Even despite that, military secrets are still leaked on an all too regular basis.
Gating access is compartmentalisation. If you're being brought onto, say, missile development, you absolutely will have to submit to both vetting (knowing who you are prior to access) and compartmentalisation (permitting access only to your relevant secrets throughout).
I'm not saying that just because you have some kind of clearance you will get access to everything, but it's part of the preconditions to your own relevant access.
Yes, those security clearances are the real gates, not anything in your JavaScript codebase, and that's the point - there's already clearance in place within the military, there should be nothing in a codebase that can bypass a security clearance requirement anyway.
A very secure codebase is designed in a way that all the sensitive parts are separated from the parts general users (and developers) have access to - it shouldn't be all imbued together such that sensitive parts about missiles are exposed to login APIs etc. as it seems like you were saying.
It may even be lower risk than not to open-source as the public is more likely to find and fix actual security quirks that a private contractor might miss (or could even be paid as a spy to purposely leave vulnerable).
There's also the community/recruitment aspect. AI/LLM companies are cleverly open-sourcing major parts of their work while keeping the only important part that makes them valuable private - it's a win:win as they keep their secrets yet provide for and stimulate a developer community.
Certainly. It’s not like giving them the source code would increase risk significantly, if the software is designed well. I think it would actually strengthen as more researchers would study and submit contribs.
I think Linux is as or more secure than windows and it’s open source. There’s tons of sensitive systems that are open source.
It’s a design fallacy that security through obscurity is good.
> It’s not like giving them the source code would increase risk significantly
I like free software a lot, and do not know much about weapon development, but would not the software reveal a lot about capabilities of the weapon platform? The argument to keep the software private might not be motivated just by attempt to hide security holes, but also by desire to hide what the weapon can do, what are the operational limits, etc.
The point is simple - obscurity as an addition increases security.
Reverse engineering is not trivial and raises the bar.
Edit. Oh, your link supports that
>The obvious disadvantage of this approach is that it doesn't prevent the bugs from being exploited - it only make the meaningful exploitation very hard or even impossible. But if one is concerned also about e.g. DoS attacks, then Security by Obscurity will not prevent them in most cases. The other problem with obfuscating the code is the performance (compiler cannot optimize the code for speed) and maintenance (if we got a crash dump on an "obfuscated" Windows box, we couldn't count on help from the technical support). Finally there is a problem of proving that the whole scheme is correct and that our obfuscator (or e.g. ASLR engine) doesn't introduce bugs to the generated code and that we will not get random crashes later (that we would be most likely unable to debug, as the code will be obfuscated).
> Do you really want our nations enemies to have access to the source code of fighter jets, cruise missiles, the nuclear weapons program, or missile defense systems
They likely already do. Between exploiting remote vulnerabilites (a nation-state intrusion will never even be noticed -- they aren't going to encrypt all the files and ask for ransom) and old-fashioned spycraft, it would be really amazing if all the contractors involved on those projects had perfect security.
From my time working in defense contracting, quite a lot of the software was open source, built on open source components (just like everything else) or even hosted on official military GitHub repos. Code written by the (federal) government itself cannot even be copyrighted (in the US). Obviously you are not going to find the classified stuff on GitHub, but even there it was very common for the government to take all of the source code from one contractor and hand it to another one to work on. You also design your systems so that as much as possible can be unclassified, because unclassified development is much, much easier.
There's a couple places in the book Skunk Works where the author laments the government's procurement strategy for planes that always ensured different manufacturers got at least some contracts so they wouldn't go out of business and leave the government reliant on just one supplier. He derided it as a form of socialism! Alas, it was a reasonable approach.
> Public services are built with public money. So unless there’s a good reason not to do so, the code they’re based should be made available for people to reuse and build on.
> Open source code can be reused by developers working in government, avoiding duplication of work and reducing costs for government as a whole. And publishing source code under an open licence means that you’re less likely to get locked in to working with a single supplier.
Obviously there are exceptions made, but you have to explain why the exception is necessary. An auth library would be an example of where you might not want the code pubic.
Anything used will become a target. How the hell is MS a bigger threat than the rest? If anything a lot of the industry is a way bigger threat and spends less on security.
I don’t understand your comment. Microsofts failures are a threat precisely because it is so used, especially by the US govt. And they may spend a lot on security, but their recent failures have been pretty amateurish. A recent breach they had was due to an old, non-2fa service account with a weak password and privileged access. See also the CISA report about last years breach.
And if it were ten open-source projects then they'd have to defend themselves against nation-state attackers. They're not ready for it. The researchers that demonstrated an attack on Linux got vilified instead of the maintainers that had misplaced their trust. *Researchers* not a truly sophisticated and a well-funded threat actor. Do you see the issue?
Do you really think the alternatives are more diligent with their defense, 2FA, have spent as much time and effort on security?
I truly don't like Microsoft but this seems like shitting on them but I really don't see the point. Anything they use is going to be a "national security threat." I'd gladly see other vendors stepping up, but it's enormously difficult and we're not really there yet.
But AWS alternatives ARE just a smattering of open source stuff.
Getting everything into AWS doesn't solve anything by itself, either. Then they'd still need to get everything off of windows, office, exchange, AD, etc. etc. Which is a ridiculous amount of work and they'd be fighting bugs and issues for years and years at their scale.
As much as I like ragging on the softies--and lol my good sir--it's the existence of power that is a threat. The current captain, or their whims, or our belief in them, they are a sense only and no guarantee.
Larger ships carry more people and can thus sink with more, let's cut the ships in half?
Point being, there's a lot of infrastructure that kinda doesn't make sense split up. If you want resiliency, build a second system in parallel not make management of an existing one times more expensive.
There's really no reason for all sensitive data and email to be stored on the servers of one company which has tens of thousands of employees who might be insider threats not to just one customer, but all of them.
You do realise you'll have half a ship, not two ships then?
Defence in depth is expensive, infrastructure at scale is expensive, it would be difficult and more expensive to have a bunch of small operators reach the same level considering the initial cost and overhead.
I'm sure though that such equally viable alternatives could be built, but not by splitting existing ones.
I'm not a fan of Microsoft, but this is some amazing blame shifting. The root cause of the problem is the government single-sourcing a vendor and being incapable of negotiating with said vendor. The US government is 10% of Microsoft's annual revenue just on security services (if I read the article correctly) but is failing to negotiate. The right answer here is if the situation is that bad, make a very public long-term commitment to shift to something else & up-level your IT department to be able to execute multi-year projects competently. Instead of meeting Microsoft on the business playing field, it's trying to use scary "national security threat" verbiage to try to bully them in the court of public opinion.
Regarding security, at the end of the day, the US government is a huge target for adversaries. You can't outsource your security practices & if MS software is really that much worse they should be fixing their purchasing requirements. The reality though is that whatever software the US government would switch to would become the focus of adversarial research.
Security is frequently only as strong as the weakest link. If you’re an adversary, would you spend more time on an org’a poor security practices (pentagon failing audits) or a more difficult software vendor?
The audits in question are financial. There’s no doubt some overlap with security – e.g. failing to replace out of date software due to financial constraints exposes them to risk – but it’s not a simple relationship.
It's an enormous organization (one of the largest in the world) that is largely getting audited on physical objects that either spend 99% of their life sitting in stockpiles, or getting actively used and discarded.
Passing an audit isn't having its checkbook balanced. Passing an audit is like your employer delivering a full and accurate account of how many chairs it owns, as well as providing all the necessary paper trail for how all of them were acquired, moved, and discarded.
Doable in a 5-person company, or a 50-person company, next to impossible in a 5,000-person company, actually impossible in a 500,000 person company.
And the DoD has hundreds of thousands of different types of such items to keep track of, all of which were tracked by disparate physical-fingers-counting-stuff accounting systems, that were historically not up to the expected standards. Updating them and reconciling all of them is a long process, which the DoD is working through.
I hate war departments as much as the next liberal, but I also recognize that this is an incredibly difficult ask.
And that the failures are generally not caused by Lt. Billy Bob filching Stinger missiles to sell to some warlord in East Oblastan.
The grift in the DoD comes from contracts for buying more crap that it doesn't need, or for paying good money for contract work that doesn't get done (Iraq, Afghanistan), not from having stuff that was bought 'fall off the back of the truck'. The organization "has* an audit problem, but it's main issue is one of procurement.
Wow, thank you for the detailed write-up. Your points make a lot of sense and I do think the general hate for the military industrial complex makes people look at these failed audits as malicious, probably prematurely.
Do you think the Pentagon will be able to pass that audit anytime soon (within the decade)?
I’d imagine that these proclamations are part of the negotiation.
I wouldn’t want to be a Microsoft account executive on the US govt contract right now; they’re about to have a massive load of additional requirements.
And if they don’t play ball, possibly antitrust to weaken their stranglehold on being the only real enterprise player.
I’m not saying any of this is the right approach, but it’s a tool in the governments toolbox.
any commercial channel filled with this much money will attract hordes of smiling useful idiots filling every possible niche. Now you have another problem, managing useful idiots over time.
I assume it's the $20 billion in security services statement compared against their ~$200 billion yearly revenue. I'm not sure those security services are all for the U.S. govt though.
My assumption is that the revenue figure is not exclusively the US government spending. Otherwise what percentage of their yearly revenue is security services to all the other countries as well as businesses in the world? The US government is of course a large customer, but a considerable part of the world runs on windows and other Microsoft products.
Well you can't not outsource your security because gov payscale limits do not match market reality. You have to realise that a ton of people who should be directly employed by NSA etc. are actually working for their contracts for this reason.
> Partly this is due to the concentration of wealth, inaccessible to taxing.
This has nothing to do with it. Contractors cost notably more, so if the goal was economizing it’d be an obvious step to cut out the middlemen by hiring staff directly.
The problem is that there’s an entire political ideology holding that government is inherently wasteful and its adherents will oppose any attempt to track market salaries because that allows them both to say they’re saving money at the time and later to cite the struggling/failed project as proof that they were right.
Yup. When a VC-backed company goes bankrupt, no one bats an eye. When Solyndra's loans go bad, even though it was a tiny fraction of the government's green energy portfolio, you get headlines and congressional hearings.
And when a big contractor has an 8+ figure write-off, they have a PR team working to prevent accountability and none of the people who were talking about private-sector innovation and efficiency will be asked whether they had a more realistic position now.
How many middle class workers were robbed through income tax to pay for the $524million dollars lost on solindra? Government money totally ruins any private business, to the tune of a huge party where everyone takes as much as possible with no accountability
How many orders of magnitude more money is “robbed” by Comcast/Verizon/Charter, the medical industry, manufacturers who hiked prices up during the pandemic, etc.? Large organizations of all persuasion need oversight but that doesn’t mean we should give up on the concept any more than Enron meant we should give up on the stock market or private energy companies.
> The problem is that there’s an entire political ideology holding that government is inherently wasteful and its adherents will oppose any attempt to track market salaries because that allows them both to say they’re saving money at the time and later to cite the struggling/failed project as proof that they were right
Why is it a surprise that employees who are essentially unfirable don't perform well? Aside from a very small percentage where the hiring culture is exceptional, it's almost universal. This is a problem with governments across at all levels in almost all countries.
How many times do we have to match this movie before realizing that just increasing the salaries won't make a difference, and instead will be wasteful.
> Why is it a surprise that employees who are essentially unfirable don't perform well?
This is a great example of that political dogma: notice that you’ve accepted as an article of faith the trope that government employees can’t be fired or disciplined or that this is not true of contractors, despite neither of those being true?
If your goal is successful projects, what you’re looking for is accountability and managerial discretion. The managers you think can’t direct civil servants directly aren’t magically more capable of selecting and overseeing contracts, either, and without technical staff they won’t have someone they can turn to for advice who doesn’t have a financial conflict of interest.
> This is a great example of that political dogma: notice that you’ve accepted as an article of faith the trope that government employees can’t be fired or disciplined or that this is not true of contractors, despite neither of those being true
Nice strawman. It is harder to fire govt employees than to fire private employees. Disagree?
Just look up at-will employment law that doesn't apply to government entities. Looks like you're the one following political dogma and accepting articles of faith and tropes.
Otherwise why would ignore that a very consequential law is different for private vs govt employees?
> The managers you think can’t direct civil servants directly aren’t magically more capable of selecting and overseeing contracts, either.
Where did I say they cannot? Please stop with the strawmen. They just don't have enough incentive because it's very hard to fire them, unlike managers in the private sector.
Lets ignore the personal back and forth, and get back to the argument.
It is harder to fire govt employees than to fire private employees. Disagree?
At-will employment law doesn't apply to government entities.
A very consequential law is different for private vs govt employees.
> The managers you think can’t direct civil servants directly aren’t magically more capable of selecting and overseeing contracts, either.
Never said in my comment they cannot direct civil servants. They just don't have enough incentive because it's very hard to fire them, unlike managers in the private sector.
That's like the farthest I've seen goalposts move within a span of two comments. You went from "essentially unfirable" to simply "harder to fire" and continues to act like you're so right. Like it's almost pointless to discuss something with someone who pretends those are even close to the same.
That's three comments where folks are attacking me instead of addressing the topic because they know they lost the argument. Sad to see that happen. And also look up the meaning of the word essentially.
> Lets ignore the personal back and forth, and get back to the argument.
I mean, were you not attempting to make an argument when you said that government employees were “essentially unfirable”? Your argument just sucked/was factually incorrect and you want to steer away from the “interpersonal” aspect of someone pointing that out.
It’s gauche to point this out on HN but you aren’t engaging in good faith here - wnd when someone else engages you in actual good-faith you fall back to juvenile debate-club attempts to frame them as the wrongdoer.
That’s highly rude and a bad attitude and approach to bring to this community. Act better and take some accountability for your own misbehavior.
I had two options, either continue the personal back and forth by debating that you folks don't understand the meaning of the word 'essentially', or take the high ground and debate the topic.
Sad that both you and the other poster lost the argument so hard that you have to resort to ad hominems and make it about me instead of the topic. Thats very telling. And also look up the meaning of the word essentially.
> Why is it a surprise that employees who are essentially unfirable.
A government needs employees right? One way to attract and retain employees is to offer competitive wages, another is to offer a relatively low workload and high job stability. Government worker salaries are easy targets and the people who set them are elected officials, so its not a surprise most governments today lean on job security over wages.
> This has nothing to do with it. Contractors cost notably more, so if the goal was economizing it’d be an obvious step to cut out the middlemen by hiring staff directly.
It certainly has something to do with it.
The market rates for engineers was distorted because FAANG had a lot of money to throw around, so therefore hiring staff at government pay rates is quite difficult and is subject to the General Schedule (https://federalnewsnetwork.com/pay-benefits/2023/12/biden-fi...).
Contractors work from bills and projects, where the money is carved out; I agree it’s terribly inefficient.
The other issue is how difficult it is to get into government vs getting in as a contractor.
I tried the former many times without success; the latter? Easier than FAANG or finance.
Yes, the GS scale is a problem. My point was that it’s not a problem because we’re trying to save money – if that were the case, someone would notice that raising the cap to allow a $300k civil service job is cheaper than allowing the same job to be performed by a $500k contractor who takes home less and is replaced more frequently.
Politics enters the picture because the pay cap is derived from the salaries for politicians rather than what expertise is valued at on the open market:
Yes, but “saving money” is your point and strawman; I never said anything about saving money, nor did I imply it.
GS in general cannot grow when the market pay was distorted by both lack of tax funds; and when those driving market pay has a disproportionate amount of wealth to corner the labor market, in order to prevent the hiring of engineers by other industries.
I think you misunderstood: I agree that lack of tax funds is a problem in other areas but in this specific case it isn’t because the same or greater amount of money is already being spent.
This is a straw man. Even if the top richest people paid an additional 16 billion in taxes that would run the gov for like a day. Our problem is with spending.
> Naturally government pay would lag behind even the more mediocre H1Bs.
Those "mediocre H1Bs" work their ass off compared to non-H1Bs because they get laid off/fired and deported on short notice with barely enough or sometimes no time to sell their belongings if they don't perform [1]. Meanwhile it's next to impossible to fire a govt employee for bad performance.
Maybe the solution is to fill the govt with mediocre H1B employees, not pay govt employees even more without accountability if they don't perform.
> Those "mediocre H1Bs" work their ass off compared to non-H1Bs because they get laid off/fired and deported on short notice with barely enough or sometimes no time to sell their belongings if they don't perform [1]
H1Bs wouldn’t be so easy to fire if the skillset were rare or difficult
to fill.
Instead, companies are
gaming the H1B lottery to create much lower-paid indentured servants while a sliver of the business rakes in 40% to 60% of the spread.
And the Federal government employees are paid at the same
level or less.
I don’t find the H1B abuse defensible, but certainly a useful discussion point on relative salaries.
> H1Bs wouldn’t be so easy to fire if the skillset were rare or difficult to fill.
By the same token a good chunk of government employees would be paid higher if their skillset were rare or difficult to fill, and they worked hard. The good ones that work hard switch to the private sector, even if the pay isn't much higher, so that they can get stuff done and not be surrounded by folks just existing while collecting a paycheck.
Within your disdain for government workers is a small sliver of truth. One problem with the Federal government--and why contractors are preferred--is that Government Schedule pay is decent for some jobs and insufficient for others.
For software engineers--and the reason contracting is better--is that government schedule pay is far below market. But you see the same problem even in private industries like Lockheed Martin, where the brightest engineers (with security clearances) are really doubling/moonlighting with other jobs and the managers look the other way.
Also: The private sector is extremely inefficient, when it's handling areas of high upfront costs/high risk. Private military companies
If you want to defund government, start by tearing down the TSA which nobody in any party believes is anything but a job-creating sham. Of course, the loudest calls for defunding or non-growth are for deparatments that provide real value--like the FDA or the IRS.
So while you're happy the government is being defunded, most US citizens don't feel like promoting a national security problem for short-term gains.
That number is meaningless without also considering US GDP (even if you write trillion in caps). The US averages about 14% government spending as a fraction of GDP, placing it at ~90/140. For the size of the US economy, spending should be significantly higher.
>...The US government's Bureau of Economic Analysis as of Q3 2023 estimates $10,007.7 billion in annual total government expenditure and $27,610.1 billion annual total GDP which is 36.2%.[1]
Is your entire argument that it's a really big number? Are we afraid of big numbers? How much do you think it should spend? It's the federal government, they do a lot of stuff. Stuff costs money. We can complain about how they spend the money, or that the money is being wasted or stolen, but pointing out that it's a big number isn't a very convincing argument.
Is this actual spending, or are you rolling pass-through savings programs into this? Cause the caveat there is people always seem to want to count the outflow but not the associated inflow…
The US government isn't in need of a thousand high-skilled hackers. They are in need of a million normal employees with some basic security awareness. Anyone with a modicum of skill can find thousands of areas to improve. The issue is that almost nobody is in a position to get anything changed. Even basic software choices are a multi-year epic.
The NSA isn't actually supposed to be a massive Statsi-like bureaucracy, it's meant to crack codes used in wartime so that our troops know what the enemy will do next.
> If a US citizen needs to be spied on by the US government, the FBI is supposed to do that.
Semi-autonomous collections systems generally aren't capable of determining citizenship.
Once the data is at rest, much of it is made available to a surprisingly large number of federal and state agencies - and even local agencies via fusion centers, etc.
The system is built for dissemination. Within that, some checks on citizenship could be possible for some data. But even then, once a foreign person is part of the data/comms transaction, all the related data is available - citizenship of the other participants is no barrier.
Well I obviously have no clue what the numbers are on the offensive side doubt they are smaller than a thousand. But there are def thousands of cyber security professionals working for government through myriad of contractors purely because of caps.
The government is able to employ tons of smart people that could be making way more money elsewhere.
They might not get the absolute best security people in the world, but they could get good enough - as good as they're getting from MS for a fraction of the price.
Additionally, government salaries aren't terrible when you factor in the pension. Most people want the money now. But if you want financial security in the future - that's a reason a lot of people chose to work for the Fed.
At least in the Shuttle era, I once heard it said in a way that was something along the lines of “contracted support is basically working to rule; NASA employee components are working to solve problems.”
Referring here to contractors as the body of private sector contract organizations that had outsourcing contracts for various support roles, not the individual people. Speaking to working environment.
> Additionally, government salaries aren't terrible when you factor in the pension. Most people want the money now. But if you want financial security in the future - that's a reason a lot of people chose to work for the Fed.
This varies by agency and field. The older pension system was replaced with a newer model a while back so a prospective federal employee is looking at the combination of effectively a 401k, a pension of 1% top salary per year of service, and social security. That’s not bad but your salary is capped at under $200k so you're not getting anyone with IT skills turning down FAANG positions unless it’s for a cause they support (NASA, the VA, etc. can do that a lot more effectively than Agriculture, etc.). In other professions, of course, that can be pretty different – if you’re an academic who isn’t able to/interested in switching careers to ad-tech, that might be a great job compared to getting in a cage fight for a handful of tenure-track positions.
This matters a lot for security because the defense needs to be everywhere. NASA will never run out of people who want to work on robots because rovers are cool; the IRS needs people who can modernize internal business systems and that’s not only not cool but will get mockery from the more clueless people they know.
> The right answer here is if the situation is that bad, make a very public long-term commitment to shift to something else & up-level your IT department to be able to execute multi-year projects competently.
The problem is that there isn't much in terms of alternatives, especially not if you prefer to have one software / vendor / tech stack.
- In groupware, there used to be Lotus Notes, but that went down the drain years ago. Thunderbird can do everything Outlook can (i.e. provide an email client, calendar and address book), but there is no official(ly supported) Thunderbird server software suite so there's always a potential for subtle bugs between whatever one chooses for directory, email and calendar backends.
- for AD there's obviously Samba but it, again, lacks a management UI that supports all of its features, so yet another potential for issues.
- the Office suite alternatives are even more of a nightmare, both in terms of usability, stability as well as compatibility with the millions of legacy files originating from Office. Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice. (And I'm not sure stuff like MS Access even has a FOSS counterpart)
- And then, there's all the other stuff that integrates with AD for authentication/authorization. In a lot of cases, it's "either use MS AD or you're on your own when you hit issues".
- finally, Windows itself. Essentially, the US Government would have to sink billions of dollars into ReactOS development to make it compatible enough with mainstream Windows versions to run all the legacy software that people use - and no, WINE alone is not enough, not for anything that deals with hardware directly. And I wouldn't assume it's possible to even hire enough developers that are skilled to develop for WINE/ReactOS and fulfill the project requirements of never having been exposed to Windows source code.
Microsoft has achieved an insane amount of vendor lock-in, even Apple with all its financial and technological might or Valve (who invested a huge amount of money and work into getting WINE feature-rich enough to run a ton of AAA games on their Steam Deck) have been able to even come close. They can provide as shitty a service/software as they want, their audience literally has no other choice.
(Me personally, I'll keep my Windows 7 and 10 VMs alive for as long as I can, but no way in hell I'm ever moving to the ad-ridden, bling-bling flashy pseudo-hipster-UI disaster that is Windows 11)
If you want to use Windows outside of an air gapped environment, it’s reckless to use anything less than Windows 10, and even for that you have only about a year left unless you pay for extended support.
Also, as someone who actually uses Windows 11 Professional all day, every working day, I honestly haven’t encountered these adverts everyone is always referring to (maybe it’s because I’m not in the US).
As for the UI, initial release didn’t allow ungrouping of Windows on taskbar and context menus hid most things behind an extra click. This was crap, but a major update a while back has resolved these.
I’ve tried Linux, I occasionally need to use it or macOS for development of our cross platform Electron based product, and I would still choose Windows as it really has consistently “just worked” for me since around Windows 2000.
But if macOS or Linux works well for you, I’m happy you’re happy and perhaps you in turn could be happy for people who are happy with Windows (and ideally without thinking they must be deluded or something).
I used to think the way I’d try to tackle a challenge like that would be essentially embracing the ChromeOS model by going web-first for everything so you’d avoid accumulating new accidental dependencies on proprietary software, and trying to shift legacy apps to WINE, terminal servers, etc. Obviously you’d need thousands of exceptions but the important part would be removing the inertia which means that new code starts out being non-portable.
Unfortunately, these days the browser engine market is pretty lopsided so that strategy would need to be paired with something like a requirement to test in Gecko and WebKit because I trust Google’s long-term management even less than Microsoft.
Wine can run apps that access hardware without issues.
The main problem is the catch-22: there's not much point in developing competitors to Microsoft's stack if businesses aren't going to ever consider them, and government departments won't consider alternatives if they can't get everything from a single vendor in a 100% risk free manner thanks to the "nobody ever got fired for buying Microsoft" and "one throat to choke" mentality you get there.
Governments do this to themselves. The USG doesn't even have to pick Microsoft. They could potentially sign contracts with Apple for workstation hardware and services, that would encourage and feed the alternative ecosystem based around Apple, or they could fund Linux, etc. They don't though. It's just soooo much easier to sign one giant contract, because that minimizes work for them and it's not really important to get value for money if you're in the public sector. Signing a great deal won't get you a big bonus or anything like that, but taking risks can get you blocked from promotions. So, why risk anything?
> Wine can run apps that access hardware without issues.
Depends on the hardware. Stuff like mice, keyboards or game controllers yes, as long as they're supported by the host OS stack (don't get me started on bluez and the clusterfuck that's Linux audio in general). But anything dealing with user-mode USB drivers (looking at you Samsung ODIN or a truckload of webcams) or more exotic hardware is out of luck.
> Governments do this to themselves. The USG doesn't even have to pick Microsoft. They could potentially sign contracts with Apple for workstation hardware and services, that would encourage and feed the alternative ecosystem based around Apple, or they could fund Linux, etc. They don't though.
Apple shot themselves into their own foot here by completely discontinuing their Intel x86 lineup. You can run Windows stuff on the M series SoCs but performance is atrocious (unless it's ARM Windows, but good luck finding software compatible with that oddity, no thanks to Qualcomm here who couldn't be arsed to put out actually usable chips for years).
As for Linux, there is already a huge amount of government funding into Linux because of servers and their support contracts. That's how RH, SuSE and the other commercial Linux distributions are surviving. The problem is the desktop software stuff, here the chicken-egg problem comes into force - and made worse by the fact that unlike Microsoft who can just decree whatever they want and the rest of the world has to accept it, FOSS projects are mostly driven-by-consensus, and if you're a commercial or government entity needing a feature you have to either fork off with all the cost that entails or herding cats and playing petty politics with people from all around the world (who might just block your idea out of principle after finding out you're working on behalf of the government/Monsanto/whoever is problematic these days, on top).
I think that's maybe more due to lack of demand/effort than any fundamental technical limitation. You can run USB drivers in user mode on both Linux and Windows. If someone paid enough for such support to be added, it'd appear.
A lot of the software getting hacked isn't even truly Windows-specific stuff to begin with though. VPN appliances, firewalls, they're probably all running out of date Linux distros. Active Directory servers could have some open source competitor too, aren't they mostly Kerberos? And for the rest, ChromeOS could knock it off too with a mix of web apps and RDP.
> You can run USB drivers in user mode on both Linux and Windows.
Yeah, but you'd need to rewrite the whole Windows app for a native Linux or macOS port because WINE to my knowledge can't access devices even if there were a Linux/macOS driver.
> Active Directory servers could have some open source competitor too, aren't they mostly Kerberos?
AD is so much more than just LDAP+Kerberos. You have a fully integrated file store, native and direct integration into Exchange (on-prem and cloud), group policies for users and machines, DNS, DHCP, NTP... one vendor, one tech stack, one management way. Set it up and you're done, and even a novice can set up a minimal working small-business environment in a day.
With Linux, you have to integrate/configure Samba (LDAP, GPO, file storage), Heimdall/MIT Kerberos, postfix/exim (Mail), no idea what one could even use for calendar sync, BIND (DNS), ISC DHCP server, Jitsi (Video calls), Mattermost (Teams chats). Each of these software pieces have their own configuration syntax, nothing integrates automatically on its own, guides are sometimes horribly outdated, there's subtle differences in behavior between distributions, some of these tools have web UIs for management, others are CLI, and all of them look and behave differently.
"Wine includes some limited support for direct hardware access, including running native Windows drivers. This comes with several restrictions, but we still have been able to successfully access some native devices."
"For example, the SteelSeries USB mouse driver offers the ability to modify the mouse's LED settings as well as simply driving the mouse, and in this case Wine is able to pass through the LED settings"
>Thunderbird can do everything Outlook can (i.e. provide an email client, calendar and address book), but there is no official(ly supported) Thunderbird server software suite so there's always a potential for subtle bugs between whatever one chooses for directory, email and calendar backends.
Thunderbird is a lot more reliable than outlook - I use both daily. But yeah there's no 'thunderbird software suite'. I don't think that's really an issue though. Outlook breaks regularly, and our desktop support group had to deploy an 'outlook reset' app because the vendor is unresponsive. Being that unresponsive lessens the attractiveness of a 'software suite'.
>Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice.
If I find MS word can't open a file, Libreoffice always does. I've never had a problem opening a libreoffice writer or MS Word file. I'm no 'power user' though.
The reason businesses and organizations stick to microsoft is because it's what they're used to, not because the apps themselves are better.
As for microsoft as a vendor - they are as obtuse a vendor as I've ever dealt with. Nearly always 100% unresponsive - even with projects involving hundreds of thousands of yearly dollars in licensing fees. I think being unresponsive is some kind of corporate culture thing with them. They're like the McDonalds of IT vendors. You get what's on the menu and that's it. If it doesn't work, well too bad because they know the perceived cost of switching is believed to be too high.
> But yeah there's no 'thunderbird software suite'. I don't think that's really an issue though.
For me, it is. Outlook natively integrates with Teams, Onedrive and the rest of the Office suite. Everything is available at once, with one click, in a reasonably similar UX/UI pattern (okay, outside of the horror that is Teams with direct chats, meeting chats and Teams team chats). Let's just take a Teams meeting invite... in Outlook it directly opens the Teams meeting, in Thunderbird first the web browser pops up, which then opens Teams, and leaves a stray tab behind. It may be just ten seconds, but it's so much more annoying.
we have a company that has a monopoly that is honestly unimaginable (how does someone monopolize computation of all things...)
and we know that all monopolies require government enforcement to prevent others from competing...
then we know that there's no such thing as a conflict between Microsoft and the us government
the us govt is to serve Microsoft and that's that. any conversation like this about the merits of Microsoft as a market participant are laughable and disingenuous
theres no programmer alive that knows the history of Microsoft that would speak about them as you have just done, as an honest company trying to make a product
Then go after them for monopolistic abuses. Arguably Lina Khan is the first FTC chair to be making noise about it yet the only thing about Microsoft has been their OpenAI partnership because it's sexy & nothing about their traditional marketplace participation. The EU is going after them a bit more aggressively & having some success. Ultimately this would still be the US government's failings - claiming that makes Microsoft a national security threat is a joke & disingenous.
I know the history of MS anti-trust fairly well having grown up during the core part of their monopolist years - now they're part of the familiar oligopaly that's strangling tech.
> and we know that all monopolies require government enforcement to prevent others from competing...
Do we know that? That may be true of government run monopologies, but I'm pretty sure most marketplace monopolies rely on lack of government enforcement. For example, here's some analysis showing how US case law makes the government wary of going after Amazon for driving Quidsi out of business & forcing an acquisition by Amazon: https://cei.org/blog/amazons-private-labels-dont-threaten-co.... Ironically US does enforce rules against dumping from international players which shows that it realizes the harm that such activity can have, it just chooses to allow it for domestic players abusing smaller domestic players.
The era of global botnets, worms, and ransomware can be credited to Microsoft, as well.
Lax concern and slow turnaround for CVEs, poor update performance that led to much of the world turning off automatic updates, and updates being blocked on non-activated installs, if they weren't just easily broken gave root (literally) to fleets of vulnerable internet connected Windows systems being leveraged against the world at large for roughly a decade until Windows 10 brought (and forced) endpoint security practices into the 21th century.
It's reassuring that their official stance hasn't changed; "This is how big we are, and it'd be a real tragedy if we couldn't afford to stop something like what we caused before from happening again...", instead of being sued into responsibility by the world at large.
(Uh oh, not 5 minutes in and it looks like Microsoft PR is already here to downvote the trouble away.)
“NotPetya” and StuxNet were sophisticated worms/viruses which leveraged flaws in Microsoft products to ultimately bring down infra (ie, Iran nuclear program) or bring down countries (ie, Ukraine suffered attacks to energy sector)
Part of the reason it is downvoted is the snide insult at the end about downvotes. It took conscious decision and effort to edit a snide insult into the post where there was none before. As one of the people being derogatorily called "Microsoft PR", this is unnecessarily antagonistic of GP towards me on HN, especially when I did nothing to GP.
Commenting on your downvotes is off-topic, against the rules, and when done as a snide insult like above: rude. GP might experience better results if they edited their post to remove the snide, insulting comments about downvoters.
It would definitely be possible for government to be good at things ... in the olden days of tech development, very good people were employed and empowered at government positions with technology roles. I'm thinking back to later 90s when I filled out my financial student aid application. That was an _extremly_ complicated web product for the time, built entirely by the government, and it completely worked and was easy to use. Commercial products like turbotax on the web didn't get parity of complexity and robustness for a good decade more than that.
The 'outsource everything' 'not allowed to compete with private industry' mentalities are what has the made the government unable to function in a quality manner ... Its virtually impossible for the government to just hire some people to a team to build some shit -- instead they are _required_ to create bids for contractors to bid on and then incredibly formal contract management processes that are just incredibly disfunctional by design ...
It doesn't have to be this way -- its a political result going back to the 'small government' movement which was ultimately about proving that government has to be bad at everything by imposing rules to ensure that result in as many places as possible ...
Agreed. The US government is actually the most technologically accomplished organization in human history. Examples: Everything NASA has done and does, nuclear weapons, nuclear-powered aircraft carriers, the Internet, the NSA's capabilities, etc.
More control yes but also: more transparency, more vendor choices across the lifecyle. Also more control over telemetry: the current MS stack uploads far more than necessary to only keep things running; government users should not be surveilled, their data should not be sold, etc.
I call bullshit. Someone, somewhere long ago deep inside of the bowels of the NSA decided to deprioritize actual security, and the use of the capability security model. They knowingly did this, because actually secure computing (which they already had at the time[1,2,3]) represented a threat to the NSA, or so they thought at the time. The trade-offs seemed acceptable, because there were systemic approaches at the time that could keep things secure via other means, such as air-gapping, TEMPEST screening, etc.
Much like the consumer decision on the other side of that secrecy barrier, when it was decided the MULTICS was too complex, and you really didn't need all of that complexity (to allow mutli-level security), the easing of hardware requirements, and systems design time all seemed worth it.
Here we are 50 years later, and both decisions have proven to be disastrous. We've swiss-cheesed our infrastructure in a manner that can only be hinted at metaphorically. The closest analogies I know are:
Imagine the power grid and modern society, with absolutely zero circuit breakers or fuses.
Imagine running an economy without being able to divide wealth into uniform small amounts, dollars, cents, etc.
In both cases, everything becomes all or nothing. This is what we have with computers, all or nothing. You have to trust software you run, or you simply can't run it. It's the same with the "License Agreement", nobody actually reads them, and nobody tries to enforce them, yet there they are.
It's possible to have secure computing, we had it in the 1980s, but we didn't realize it. It was crude, but effective, because we had systems with no persistent internal storage, and we could write protect the storage we had. Thus it was possible to manage data, and the side effects allowed, in a simple, powerful way by simply not putting data at risk, making copies, and using those instead.
Until we reverse the bad decisions of the 1970s, we're going to keep up the deranged behavior, blaming everything and everyone, and ignoring the one way out of this.
You shouldn't ever have to trust a program you execute to behave in the manner you expect. Your computer should enforce the limits you expect, and the program should have zero ways around those limits.
Your assessment of the current state of affairs is spot on. However, there are a few additions.
A. It is not metaphorically swiss-cheesed, the swiss-cheese model [1] of software security is officially endorsed. Just stack enough trash and maybe it will be okay is now accepted policy.
B. It was not driven by the NSA. It was actually driven by commercial software vendors like Microsoft and Cisco who could not meet minimum security standards, so demanded the standards be lowered to allow them to make sales. Any time the NSA gets the drivers seat in setting government security procurement standards they almost always push for actual security since the NSA is the government; they are protecting themselves. It is reasonable to distrust them for commercial and external systems, but you can trust their standards for their own and government systems are not meaningfully compromised.
You should probably try building a pure capability based operating system and making it usable, before blithely saying the entire industry got security wrong. Capabilities aren't magic. UNIX has had many forms of capability for a long time, NeXT/macOS/iOS uses them extensively as well, and there are still vulnerabilities regardless.
Also, capabilities are irrelevant to the types of hack being discussed on this thread where the vectors are forgotten admin accounts, non-patched software, etc.
Capabilities are magic, if you don't have them. People confuse permission flags, such as those found in smartphones, etc.. or locking down permissions a-prior as in app-armor, with capabilities.
The ability, at run time, to pick a file, and give only that file, folder, or set of resources to a piece of software is essential to secure computing. As far as I know, NeXT/macOS/iOS et all don't have that ability.
CSRB's report on the Exchange Online breach that dropped a couple weeks ago was pretty damning. Microsoft had a situation where a threat actor had access to the entirety of Exchange Online, and possibly their entire cloud. CSRB describes the entire incident as completely avoidable, and resulting from Microsoft's inadequate security culture, and it calls Microsoft out for making public statements about the breach and its response that it knew to be inaccurate.
The ONLY way that breach got detected was because the State department bought the premium package with extra logging that let them see when mailboxes get opened. It turned out, Microsoft had a signing key that could create access tokens for anything in their cloud, and it was stolen by Storm-0558. (More precisely, the key was only supposed to be useful for a portion of their services, but a bug allowed Storm-0558 to bypass that scope limitation.) And they used that to go read the e-mails of the State department and a bunch of other organizations, and private individuals. There was nothing customers could do to prevent the attack, and apparently no other indication in their logs that it was taking place, besides this category of entry that was gatekept behind a premium subscription package.
Microsoft generated the key in 2016 and discontinued it years prior to the incident, but it was never revoked. Microsoft didn't even bother with key rotations anymore after 2021 because one time they fucked it up and it caused an outage, so they decided to just not do that anymore. Also, Microsoft apparently didn't have any means of detecting the obvious use of a zombie key.
Also, Microsoft still doesn't really know how they got the key. They made a blog post about their theory, representing it as something they were highly confident in based on the evidence. After 6 months of pressure from the government, Microsoft finally updated the post to admit that they had no evidence of critical parts of what they claimed, and several key points in their narrative were factually incorrect.
Then earlier this year, Microsoft got hacked AGAIN because they had an unused-but-active test account with a guessable password and no MFA, and it was authorized for access to e-mail boxes of (at a minimum) numerous members of Microsoft senior leadership.
> Microsoft didn't even bother with key rotations anymore after 2021 because one time they fucked it up and it caused an outage, so they decided to just not do that anymore.
Key rotation is almost like restoring from backups. It's an absolutely necessary capability and practice.
You'd be surprised at how little cloud vendors give a shit about security internally. Story time: I recently went ahead and implemented key rotation for one of our authz services, since it had none, and was reprimanded for "not implementing it like Google". Fun fact: Google's jwks.json endpoint claims to be "certs" from the path (https://www.googleapis.com/oauth2/v3/certs). They are not certs - there is no X.509 wrapper, no stated expiration, no trust hierarchy. Clients are effectively blind when performing token validation with this endpoint, and it's really shitty.
Other nonsense I've seen: leaking internally signed tokens for external use (front-channel), JWTs being validated without a kid claim in the header - so there's some sketchy coupling going on, skipping audience validation, etc...
Not much surprises me anymore when it comes to this kinda stuff - internally, I suspect most cloud providers operate like "feature factories" and security is treated as a CYA/least-concern thing. Try pushing for proper authz infrastructure inside your company and see what kinda support you'll get.
Are there any large companies that don't operate like feature factories? It seems to be such a common issue and the natural result of the incentive structure.
although this is a valid insight, it reduces the detail of the conversation into "yes or no" on a topic that is not a "yes or no" topic.. it is behavior and messaging among a dozen critical functions of business. Almost every business is different in their mix.. perhaps faced with similar rhetoric, law says "show me an example then we can discuss" instead of "classify all examples then apply to a situation"
Not necessarily. Scheduled key rotation has a lot of conceptual problems:
1. As MS found, revoking old keys is very risky because doing so creates outages. But if you don't do it then changing keys is useless. This isn't a problem specific to Microsoft. Lots of companies have learned this lesson the hard way.
2. It assumes that attackers don't just use stolen keys immediately (e.g. to issue more keys, change passwords, create new accounts etc). In practice they usually do.
3. It assumes that if you change the keys the attackers can't just immediately re-steal the new keys.
So it's only really a useful practice in one very specific scenario: you do something that boots undetected attackers out of your network without realising that's what it did, and the attackers need ongoing access that only that key can provide, and they can't use that key to elevate permissions in a more permanent way like by creating a new account on the system or stealing a user password. Pretty specific scenario.
Unfortunately, key rotation also comes with big downsides. Any software that works with keys has to be built to tolerate a change silently, because now it's a regular occurrence instead of a rare one (where maybe a bit of disruption can be absorbed). That creates complexity and therefore bugs. And because it's a repetitive piece of fiddly and complex work that can break your entire service if you get it wrong it inevitably gets automated, and that in turn means that you end up with a large collection of highly privileged subsystems that have the power to silently change keys in ways admins won't notice because they are expecting it: exactly the sort of thing attackers will immediately focus on.
Overall it's not an obviously winning move. Opportunity cost matters too. Whilst you're setting up all the infrastructure to do this, ironing out the bugs, cleaning up after the outages etc, your competitors might be investing in other kinds of security best practices that are more effective. It's especially useless here because MS don't know how the key was stolen to begin with, so there's no reason to think that if they changed it that would have had any effect. Most likely it could have just been immediately restolen and all the effort would have been theatre.
> Microsoft has a shocking level of control over IT within the US federal government
Apple has same within US schools oll over the country and nobody even cares. Your government will change but your children will be with you for a while.
I've been told by current military folks that they are forced to use outdated windows (the ones without security updates) on official military computers on base. So this is where they access emails, surf the web, and all of that. They had to use IE instead of the evergreen edge browser.
It's well known among the people who serve that it's a joke.
Not just on base. I had a conversation with a three star general. He remarked that while our warships had separate software and hardware for systems and fire control, the rest of of the ship’s IT ran on Windows. Supposedly, there was no connection between the two, but LOL.
Why LOL? These are separate systems/networks. Even warships need boring admin things like email, internet, identity management, etc. Plus the systems that go on these ships are heavily customized not just straight out of the box.
Reminds me of Gary McKinnon, who "hacked" US military computers by using default passwords.... what difference does.it make to use newer OSes if ITops sucks?
To be more accurate, the leadership of Microsoft's lack of prioritizing security, aka basic quality of product, is a national security threat.
A similar claim could obviously be made of Boeing. Just imagine what is happening in their military contracts which we are not allowed to hear about. Looking at the projects which we are allowed to know about, airliners and Boeing's Starliner, clearly Boeing management needs to be put out to pasture.
The core issue is cutting corners, for profit. This is not an issue which is easy to handle in our system. It seems that the best we can do is name and shame. Let's do that at least.
While this brings to mind that scene [1] from Steve Carrel's "Space Force" show, you can't blame the scorpion for stinging you if you give him a ride on your back.
The dependency on (pretty much) a single vendor across the entire military organization is such a monumental single point of failure, I can't imagine what level of obtuse bureaucracy led them there. It seems self-inflicted at best; if it wasn't MS it would've been some other corp.
IMO the military (especially given their incredible size and access to resources) should have 100% rolled their own IT solutions from first principles, and built upon that.
Even if you pretended Microsoft didn't exist, I don't really see how any similar alternative actually presents a secure alternative. There's a mess of vulnerabilities and complexity at every layer of any modern computing stack you can think of.
If you ACTUALLY want security, you're going to be dealing with computers which are 100x slower than what we're used to with a tiny fraction of the features. For the most part I'm not convinced people actually want secure computer systems, they want computer systems which are productive and they'll just cope with the regular security breaches.
I've been hearing about iOS exploits for eons, and exploits against Safari and iMessage specifically happen all the time. See: All the stuff the NSO group has done. Apple has also been getting hit by unpatchable hardware level vulnerabilities because quite simply their hardware is designed to be fast, not designed to be secure.
I'm also not saying this based on people reporting vulnerabilities, I'm pointing out the software we're using is fundamentally impossible to secure on merit of its complexity and the tradeoffs people make that neglect security.
The US govt can also do a lot better. The very agency they formed to counter cyber threats and alert against(CISA) itself got hacked because they failed to patch or remediate, and led to a serious leak of sensitive chemical industry information among others. Because they failed to follow their own security advisory. And they won't even put out a report detailing the hack like MS did.
> According to an early report on the breach, an anonymous source said that the compromised systems were the Infrastructure Protection (IP) Gateway, which houses critical information about the interdependency of U.S. infrastructure, and the Chemical Security Assessment Tool (CSAT), which houses private sector chemical security plans.
> CSAT is an online portal that contains highly sensitive information that determines which facilities are considered high-risk under the Chemical Facility Anti-Terrorism Standards (CFATS).
> CISA declined to confirm or deny which of their systems were taken offline.
> In late February, CISA had already issued a warning that cyber threat actors are exploiting previously identified vulnerabilities in Ivanti Connect Secure and Ivanti Policy Secure gateways. Ivanti Connect Secure is a widely deployed SSL VPN, while Ivanti Policy Secure (IPS) is a network access control (NAC) solution.
> Now, CISA itself has fallen victim to a cyberattack involving Ivanti products.
I'd guess that, before the US ever does anything about its Microsoft Achilles heel, other countries will realize that they face even more threats from depending on Microsoft.
(This applies to a number of reckless tech companies upon which countries and other companies create dependencies. But MS is one of the most concerning to me.)
223 comments
[ 3.6 ms ] story [ 242 ms ] threadand an aside note do you really want the national defense to be one thumb drive away from total collapse?
the reality is that this military industrial complex is just a way to print money and funnel it into the hands of a few criminals. all of the major news publications openly told you that the USA has been attacked over 100 times in the middle east AND WE LEFT. The public is going to find out very soon that this massive, expensive military can't actually defend against or defeat other militaries. That when you spend $1000 on a hammer everything looks really flashy and impressive AT FIRST with your fancy jets and missile systems, but after enough time, and after enough of the money has been funneled out, you realize it's a paper army. the USA is so weak that Iran is pushing them out of the middle east and there's nothing the theives that run the military can do about it. but don't worry. they already manufactured a Palestine sympathy story ahead of time to explain why they have to pull out and run away like the cowards they are. and your son's and neighbors sons will pay the price with their lives
Quality verification depends on auditing and auditing depends on competent, trusted review and testing. Global transparency is not a substitute for auditing except when discussing absolute rock-bottom standards. Reviewable is significantly better than unreviewed and unreviewable, but that is the lowest possible bar. Unfortunately, software does, as a general rule, have rock-bottom auditing standards, so FOSS does provide meaningful assurance increases in many use cases. But, that is a artifact of the abysmal quality standards rather than any sort of inherent auditing advantage that FOSS provides. That is not to say that global transparency is not valuable for other reasons, but it provides no meaningful quality verification or auditing advantage in serious applications versus local transparency (to the trusted reviewer).
If you want to see how this works in practice, look at literally every other industry.
Thankfully governments are now also funding FOSS work but this really doesn't align with what you're envisioning, I think.
https://dodcio.defense.gov/Open-Source-Software-FAQ/
From where I sit, I find it absurd that everything we use isn't open source. Again, not free, but the code given to purchasers.
You don't need closed source to protect IP, and the proof is in all of these API lawsuits, and copyright law.
I don't want my missiles to not have code I cannot edit, and stepping back from the top secret sphere, tangential I am appalled at how crappy car firmwares are closed source.
In the old days, a country's national transport agencies could look at evey linkage, every rod, every part of a car design.
Now 90% of the design is hidden. And with electric cars, it's even more firmware.
And the idea that OTA updates are a thing for cars. The madness. The absolute madness.
How much do you want to bet that charging firmwares are remotely updatable?
Now imagine that 9am on Monday, every electric car explodes?
Even today, that would mean an immense number of houses on fire. How could the fire department handle it?
And how could it be handled as the fires spread? And what if lots of other infra goes up?
And that's today. What about when 90% of transportationn is electric? Even if not a single house or building burned, or person was hurt, how would you replace all those buses, trucks, delivery vehicles, and cars? As COVID showed us, you cannot ramp up and down overnight.
And what if it happened to all our allies? Would they still sell part to us?
What of all the tractors are electric, and we miss corn and wheat planting season?
My point in this tangent is... no one is even looking at the important bits. And to reiterate, how much do you want to bet charging firmwares are remotely updated? Really, they should be air gapped from the entire rest of the car.
Letting potential hackers have access in this way, is just plain lazy and reckless.
> I don't want my missiles to not have code I cannot edit, and stepping back from the top secret sphere, tangential I am appalled at how crappy car firmwares are closed source.
I read that as that whoever's missiles they are should have full visibility and control of their code; I am assuming that "my missiles" means the government, not that the poster personally owns missiles.
Access does not mean open source. Open source = OSI and what I think GP meant. And I hold the sentiment that government funded development should be open source.
For government work, I care about OSI, not if people can just see and not legally change, contribute, and redistribute.
I think government should fund global goods and want to be precise in my language. So when I say “open source” I specifically mean OSI-licensed stuff.
No matter. One entity doesn't get to unilaterally redefine a century old term. You claim you want precision, well then specify OSI, a subset of open source.
(And yes, the licenses it approves are indeed a subset.)
Using precise language is useful for building software (and in general life).
I like that OSI exists and helps people understand a specific and desired definition for open source.
There’s a lot of thought on this topic by really smart and skilled people. The aspects of open source that I think are most important are captured in OSI.
For me, I don’t care if I can see the source, if it’s not free and usable. I can decompile code and patch for myself without a license. And I can pay for code in escrow.
But what’s important is that I can modify and redistribute. And that I can be part of a virtuous community creating things together.
As we saw with 702, we really have very little leverage left in what our gov does anymore, the threat of being voted out even carries little power, exactly because they're systematically removing humans from every loop they can for the purpose of working unilaterally even when--perhaps especially when--it's against the will of the people. The spooks themselves have literally said exactly as much in their fight against privacy.
To me, the latest 702 is more terrifying than any adversary. When we, who the government should be beholden to, cannot even know the text of the laws which apply to us, it really puts a fine point on how the those who run our government view themselves in relation to its citizenry.
"The liberties of people never were, nor ever will be, secure, when the transactions of their rulers may be concealed from them." -Patrick Henry
Not that it matters, the state still doesn't care. It can't, it's not that kind of being.
I think it's a pretty obvious ipso facto that the more, and more advanced weapons, that are placed into the global battlefield, the less actual security (in life and liberty) we can expect.
The problem isn't that there are adversaries that are willing to sacrifice their people, almost every war is one of attrition after all, it's that leaders that would be adversaries like this are inevitable in a governmental structure that is delegated a monopoly on violence.
Seriously though - the software itself would be separated from secrets architecturally. And because it's open and anyone in the world can contribute it could be superior code than what some private government tech contractor could come up with.
For the equipment use case like jets and missiles, a separate directive component (driver) would likely be necessary anyway, not just for security/privacy but because different nations have different equipment. We use F-16s, M-16s, Autel drones, etc.
This driver/directive component would be the manufacturer's IP - like an nvidia GPU driver. Think of it like we'd all be running/contributing the same OS but because we have different hardware and security needs there are still some private components.
The idea/goal here is that our defense companies would overall benefit from it. Not just because the software is improved, but you can justify funding, build curricula around it - might benefit the defense industry to modernize their tech practices.
If you put all the design up for nuclear weapons but just kept the nuclear codes secret it's great that no one can fire ours, but people could implement the design on their own with different codes.
To use a more realistic example, consider air defence missile systems use to shoot down incoming missiles and drones. The secrets here aren't keys, it's software-driven behaviours including the approach to identify radar tracks as hostile or noise, identify when to commit a missile and how to target it, the code implicitly contains the vulnerabilities where the radar tracking is less effective and more evadable and how the system tries to mitigate this, etc.
When you take away all the secret behaviours, you quickly end up with not much more than just the drivers connecting the hardware to the logic, which isn't a lot of code that's driving the funding. How you set a missile tubes tilt and direction is trivial stuff and is mostly reused from old platforms that were funded decades ago.
Additionally, there is some talk about how this is kind of like security through obscurity. When it comes to things like weapons and other high-tech capabilities, security isn't only security against being owned, it's security against your opponent closing the technological gap. Unfair wars, where you have a significant tech lead on your opponent, means your population bleeds less.
Decreasingly the case, and we might be at the point I can say "poorly architected" if that's the case. A random 19 year old developer in the military with access to the secrets would be a big security hole too.
That's what we should be comparing with regards to open-source vs not open-source - in either case access to weaponry would of course be heavily gated.
Some of the other arguments you make here are no different than conventional arguments against open-source:
> the real code would still be private
Precisely the point, get the crowd to optimize the low-risk parts, build communities around those libraries and frameworks and recruit from it
> opponents might catch up
To remain an industry leader it's actually better to get everyone playing your game rather than trying to compete and stay ahead in a wild west scenario. You want to capture it really, so you can control it and be the leader in it. Open-source is one great way to do that (browser vendors come to mind).
>19 year old developer with access would be a big security hole
It's true, they are. That's why militaries and defence companies go to great lengths to vet their staff and why even within vetted staff, sensitive material is compartmentalised to minimise the risk from any given individual. Even despite that, military secrets are still leaked on an all too regular basis.
What a joke, no they don't. They establish security internally by gating access, not trusting everyone because they've been "pre-vetted".
I'm not saying that just because you have some kind of clearance you will get access to everything, but it's part of the preconditions to your own relevant access.
A very secure codebase is designed in a way that all the sensitive parts are separated from the parts general users (and developers) have access to - it shouldn't be all imbued together such that sensitive parts about missiles are exposed to login APIs etc. as it seems like you were saying.
It may even be lower risk than not to open-source as the public is more likely to find and fix actual security quirks that a private contractor might miss (or could even be paid as a spy to purposely leave vulnerable).
There's also the community/recruitment aspect. AI/LLM companies are cleverly open-sourcing major parts of their work while keeping the only important part that makes them valuable private - it's a win:win as they keep their secrets yet provide for and stimulate a developer community.
I think Linux is as or more secure than windows and it’s open source. There’s tons of sensitive systems that are open source.
It’s a design fallacy that security through obscurity is good.
I like free software a lot, and do not know much about weapon development, but would not the software reveal a lot about capabilities of the weapon platform? The argument to keep the software private might not be motivated just by attempt to hide security holes, but also by desire to hide what the weapon can do, what are the operational limits, etc.
Yet, still obscurity increases security.
Reverse engineering is not trivial and raises the bar.
I disagree and think obscurity decreases security. It just gives the false belief of security.
The point is simple - obscurity as an addition increases security.
Reverse engineering is not trivial and raises the bar.
Edit. Oh, your link supports that
>The obvious disadvantage of this approach is that it doesn't prevent the bugs from being exploited - it only make the meaningful exploitation very hard or even impossible. But if one is concerned also about e.g. DoS attacks, then Security by Obscurity will not prevent them in most cases. The other problem with obfuscating the code is the performance (compiler cannot optimize the code for speed) and maintenance (if we got a crash dump on an "obfuscated" Windows box, we couldn't count on help from the technical support). Finally there is a problem of proving that the whole scheme is correct and that our obfuscator (or e.g. ASLR engine) doesn't introduce bugs to the generated code and that we will not get random crashes later (that we would be most likely unable to debug, as the code will be obfuscated).
- Stuxnet (developed to attack PLCs which control nuclear centrifuges)
- Petya (targeted Microsoft developed systems and software and brought governments and states to their knees. Notably Ukraine)
They likely already do. Between exploiting remote vulnerabilites (a nation-state intrusion will never even be noticed -- they aren't going to encrypt all the files and ask for ransom) and old-fashioned spycraft, it would be really amazing if all the contractors involved on those projects had perfect security.
> Public services are built with public money. So unless there’s a good reason not to do so, the code they’re based should be made available for people to reuse and build on.
> Open source code can be reused by developers working in government, avoiding duplication of work and reducing costs for government as a whole. And publishing source code under an open licence means that you’re less likely to get locked in to working with a single supplier.
Obviously there are exceptions made, but you have to explain why the exception is necessary. An auth library would be an example of where you might not want the code pubic.
They needed some decades to find out. /s
Do you really think the alternatives are more diligent with their defense, 2FA, have spent as much time and effort on security?
I truly don't like Microsoft but this seems like shitting on them but I really don't see the point. Anything they use is going to be a "national security threat." I'd gladly see other vendors stepping up, but it's enormously difficult and we're not really there yet.
The alternative wouldn’t be to use a smattering of open-source stuff, it would be to go with a different cloud provider like AWS or Google.
> Do you really think the alternatives are more diligent […]
That’s what the CISA report suggested.
I’m not shitting on MS, I just don’t want foreign adversaries to be able to pick apart our IT systems like vultures.
Getting everything into AWS doesn't solve anything by itself, either. Then they'd still need to get everything off of windows, office, exchange, AD, etc. etc. Which is a ridiculous amount of work and they'd be fighting bugs and issues for years and years at their scale.
> See also the CISA report about last years breach
The same CISA who just recently got hacked in an even more amateurish way?
https://news.ycombinator.com/item?id=40107675
https://securityintelligence.com/news/cisa-hackers-key-syste...
Point being, there's a lot of infrastructure that kinda doesn't make sense split up. If you want resiliency, build a second system in parallel not make management of an existing one times more expensive.
There is a little bit of wisdom out there: Don't put all your eggs in one basket.
Heterogenous architecture (where you intentionallly mix and match differing parts for increased fault tolerance) has been a thing for decades.
Defence in depth is expensive, infrastructure at scale is expensive, it would be difficult and more expensive to have a bunch of small operators reach the same level considering the initial cost and overhead.
I'm sure though that such equally viable alternatives could be built, but not by splitting existing ones.
Regarding security, at the end of the day, the US government is a huge target for adversaries. You can't outsource your security practices & if MS software is really that much worse they should be fixing their purchasing requirements. The reality though is that whatever software the US government would switch to would become the focus of adversarial research.
Passing an audit isn't having its checkbook balanced. Passing an audit is like your employer delivering a full and accurate account of how many chairs it owns, as well as providing all the necessary paper trail for how all of them were acquired, moved, and discarded.
Doable in a 5-person company, or a 50-person company, next to impossible in a 5,000-person company, actually impossible in a 500,000 person company.
And the DoD has hundreds of thousands of different types of such items to keep track of, all of which were tracked by disparate physical-fingers-counting-stuff accounting systems, that were historically not up to the expected standards. Updating them and reconciling all of them is a long process, which the DoD is working through.
I hate war departments as much as the next liberal, but I also recognize that this is an incredibly difficult ask.
And that the failures are generally not caused by Lt. Billy Bob filching Stinger missiles to sell to some warlord in East Oblastan.
The grift in the DoD comes from contracts for buying more crap that it doesn't need, or for paying good money for contract work that doesn't get done (Iraq, Afghanistan), not from having stuff that was bought 'fall off the back of the truck'. The organization "has* an audit problem, but it's main issue is one of procurement.
Do you think the Pentagon will be able to pass that audit anytime soon (within the decade)?
https://www.military.com/daily-news/2024/02/24/marine-corps-...
I wouldn’t want to be a Microsoft account executive on the US govt contract right now; they’re about to have a massive load of additional requirements.
And if they don’t play ball, possibly antitrust to weaken their stranglehold on being the only real enterprise player.
I’m not saying any of this is the right approach, but it’s a tool in the governments toolbox.
source: knows some idiots
Cheaper just to pay up.
How are you able to conclude that?
Naturally government pay would lag behind even the more mediocre H1Bs.
This has nothing to do with it. Contractors cost notably more, so if the goal was economizing it’d be an obvious step to cut out the middlemen by hiring staff directly.
The problem is that there’s an entire political ideology holding that government is inherently wasteful and its adherents will oppose any attempt to track market salaries because that allows them both to say they’re saving money at the time and later to cite the struggling/failed project as proof that they were right.
Why is it a surprise that employees who are essentially unfirable don't perform well? Aside from a very small percentage where the hiring culture is exceptional, it's almost universal. This is a problem with governments across at all levels in almost all countries.
How many times do we have to match this movie before realizing that just increasing the salaries won't make a difference, and instead will be wasteful.
This is a great example of that political dogma: notice that you’ve accepted as an article of faith the trope that government employees can’t be fired or disciplined or that this is not true of contractors, despite neither of those being true?
If your goal is successful projects, what you’re looking for is accountability and managerial discretion. The managers you think can’t direct civil servants directly aren’t magically more capable of selecting and overseeing contracts, either, and without technical staff they won’t have someone they can turn to for advice who doesn’t have a financial conflict of interest.
Nice strawman. It is harder to fire govt employees than to fire private employees. Disagree?
Just look up at-will employment law that doesn't apply to government entities. Looks like you're the one following political dogma and accepting articles of faith and tropes.
Otherwise why would ignore that a very consequential law is different for private vs govt employees?
> The managers you think can’t direct civil servants directly aren’t magically more capable of selecting and overseeing contracts, either.
Where did I say they cannot? Please stop with the strawmen. They just don't have enough incentive because it's very hard to fire them, unlike managers in the private sector.
Let’s see, so it’s not a straw man when you say government employees are “essentially unfirable” but it is when someone corrects you?
It is harder to fire govt employees than to fire private employees. Disagree?
At-will employment law doesn't apply to government entities.
A very consequential law is different for private vs govt employees.
> The managers you think can’t direct civil servants directly aren’t magically more capable of selecting and overseeing contracts, either.
Never said in my comment they cannot direct civil servants. They just don't have enough incentive because it's very hard to fire them, unlike managers in the private sector.
I mean, were you not attempting to make an argument when you said that government employees were “essentially unfirable”? Your argument just sucked/was factually incorrect and you want to steer away from the “interpersonal” aspect of someone pointing that out.
It’s gauche to point this out on HN but you aren’t engaging in good faith here - wnd when someone else engages you in actual good-faith you fall back to juvenile debate-club attempts to frame them as the wrongdoer.
That’s highly rude and a bad attitude and approach to bring to this community. Act better and take some accountability for your own misbehavior.
Sad that both you and the other poster lost the argument so hard that you have to resort to ad hominems and make it about me instead of the topic. Thats very telling. And also look up the meaning of the word essentially.
A government needs employees right? One way to attract and retain employees is to offer competitive wages, another is to offer a relatively low workload and high job stability. Government worker salaries are easy targets and the people who set them are elected officials, so its not a surprise most governments today lean on job security over wages.
It certainly has something to do with it.
The market rates for engineers was distorted because FAANG had a lot of money to throw around, so therefore hiring staff at government pay rates is quite difficult and is subject to the General Schedule (https://federalnewsnetwork.com/pay-benefits/2023/12/biden-fi...).
Contractors work from bills and projects, where the money is carved out; I agree it’s terribly inefficient.
The other issue is how difficult it is to get into government vs getting in as a contractor.
I tried the former many times without success; the latter? Easier than FAANG or finance.
Politics enters the picture because the pay cap is derived from the salaries for politicians rather than what expertise is valued at on the open market:
https://www.opm.gov/policy-data-oversight/pay-leave/pay-admi...
GS in general cannot grow when the market pay was distorted by both lack of tax funds; and when those driving market pay has a disproportionate amount of wealth to corner the labor market, in order to prevent the hiring of engineers by other industries.
The median population doesn’t have the money to spend.
Focusing on government spending is a distraction.
The ones that have been gaming the tax system for 3 decades?
Those "mediocre H1Bs" work their ass off compared to non-H1Bs because they get laid off/fired and deported on short notice with barely enough or sometimes no time to sell their belongings if they don't perform [1]. Meanwhile it's next to impossible to fire a govt employee for bad performance.
Maybe the solution is to fill the govt with mediocre H1B employees, not pay govt employees even more without accountability if they don't perform.
[1] https://www.timesnownews.com/technology-science/techie-on-h1...
H1Bs wouldn’t be so easy to fire if the skillset were rare or difficult to fill.
Instead, companies are gaming the H1B lottery to create much lower-paid indentured servants while a sliver of the business rakes in 40% to 60% of the spread.
And the Federal government employees are paid at the same level or less.
I don’t find the H1B abuse defensible, but certainly a useful discussion point on relative salaries.
By the same token a good chunk of government employees would be paid higher if their skillset were rare or difficult to fill, and they worked hard. The good ones that work hard switch to the private sector, even if the pay isn't much higher, so that they can get stuff done and not be surrounded by folks just existing while collecting a paycheck.
For software engineers--and the reason contracting is better--is that government schedule pay is far below market. But you see the same problem even in private industries like Lockheed Martin, where the brightest engineers (with security clearances) are really doubling/moonlighting with other jobs and the managers look the other way.
Also: The private sector is extremely inefficient, when it's handling areas of high upfront costs/high risk. Private military companies
If you want to defund government, start by tearing down the TSA which nobody in any party believes is anything but a job-creating sham. Of course, the loudest calls for defunding or non-growth are for deparatments that provide real value--like the FDA or the IRS.
So while you're happy the government is being defunded, most US citizens don't feel like promoting a national security problem for short-term gains.
>...The US government's Bureau of Economic Analysis as of Q3 2023 estimates $10,007.7 billion in annual total government expenditure and $27,610.1 billion annual total GDP which is 36.2%.[1]
https://en.wikipedia.org/wiki/Government_spending_in_the_Uni...
>For the size of the US economy, spending should be significantly higher.
The percentage shot up during the Covid spending and is still slightly above the historical average.
State/federal govts receive 10-20% of every single citizens salary on top of being able to literally create money. They aren't hurting for money.
This is entirely caused by corruption and incompetence.
They're not supposed to (according to for example the Foreign Intelligence Surveillance Act of 1978).
If a US citizen needs to be spied on by the US government, the FBI is supposed to do that.
Semi-autonomous collections systems generally aren't capable of determining citizenship.
Once the data is at rest, much of it is made available to a surprisingly large number of federal and state agencies - and even local agencies via fusion centers, etc.
The system is built for dissemination. Within that, some checks on citizenship could be possible for some data. But even then, once a foreign person is part of the data/comms transaction, all the related data is available - citizenship of the other participants is no barrier.
The government is able to employ tons of smart people that could be making way more money elsewhere.
They might not get the absolute best security people in the world, but they could get good enough - as good as they're getting from MS for a fraction of the price.
Additionally, government salaries aren't terrible when you factor in the pension. Most people want the money now. But if you want financial security in the future - that's a reason a lot of people chose to work for the Fed.
Referring here to contractors as the body of private sector contract organizations that had outsourcing contracts for various support roles, not the individual people. Speaking to working environment.
This varies by agency and field. The older pension system was replaced with a newer model a while back so a prospective federal employee is looking at the combination of effectively a 401k, a pension of 1% top salary per year of service, and social security. That’s not bad but your salary is capped at under $200k so you're not getting anyone with IT skills turning down FAANG positions unless it’s for a cause they support (NASA, the VA, etc. can do that a lot more effectively than Agriculture, etc.). In other professions, of course, that can be pretty different – if you’re an academic who isn’t able to/interested in switching careers to ad-tech, that might be a great job compared to getting in a cage fight for a handful of tenure-track positions.
This matters a lot for security because the defense needs to be everywhere. NASA will never run out of people who want to work on robots because rovers are cool; the IRS needs people who can modernize internal business systems and that’s not only not cool but will get mockery from the more clueless people they know.
The problem is that there isn't much in terms of alternatives, especially not if you prefer to have one software / vendor / tech stack.
- In groupware, there used to be Lotus Notes, but that went down the drain years ago. Thunderbird can do everything Outlook can (i.e. provide an email client, calendar and address book), but there is no official(ly supported) Thunderbird server software suite so there's always a potential for subtle bugs between whatever one chooses for directory, email and calendar backends.
- for AD there's obviously Samba but it, again, lacks a management UI that supports all of its features, so yet another potential for issues.
- the Office suite alternatives are even more of a nightmare, both in terms of usability, stability as well as compatibility with the millions of legacy files originating from Office. Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice. (And I'm not sure stuff like MS Access even has a FOSS counterpart)
- And then, there's all the other stuff that integrates with AD for authentication/authorization. In a lot of cases, it's "either use MS AD or you're on your own when you hit issues".
- finally, Windows itself. Essentially, the US Government would have to sink billions of dollars into ReactOS development to make it compatible enough with mainstream Windows versions to run all the legacy software that people use - and no, WINE alone is not enough, not for anything that deals with hardware directly. And I wouldn't assume it's possible to even hire enough developers that are skilled to develop for WINE/ReactOS and fulfill the project requirements of never having been exposed to Windows source code.
Microsoft has achieved an insane amount of vendor lock-in, even Apple with all its financial and technological might or Valve (who invested a huge amount of money and work into getting WINE feature-rich enough to run a ton of AAA games on their Steam Deck) have been able to even come close. They can provide as shitty a service/software as they want, their audience literally has no other choice.
(Me personally, I'll keep my Windows 7 and 10 VMs alive for as long as I can, but no way in hell I'm ever moving to the ad-ridden, bling-bling flashy pseudo-hipster-UI disaster that is Windows 11)
Also, as someone who actually uses Windows 11 Professional all day, every working day, I honestly haven’t encountered these adverts everyone is always referring to (maybe it’s because I’m not in the US).
As for the UI, initial release didn’t allow ungrouping of Windows on taskbar and context menus hid most things behind an extra click. This was crap, but a major update a while back has resolved these.
I’ve tried Linux, I occasionally need to use it or macOS for development of our cross platform Electron based product, and I would still choose Windows as it really has consistently “just worked” for me since around Windows 2000.
But if macOS or Linux works well for you, I’m happy you’re happy and perhaps you in turn could be happy for people who are happy with Windows (and ideally without thinking they must be deluded or something).
Unfortunately, these days the browser engine market is pretty lopsided so that strategy would need to be paired with something like a requirement to test in Gecko and WebKit because I trust Google’s long-term management even less than Microsoft.
The main problem is the catch-22: there's not much point in developing competitors to Microsoft's stack if businesses aren't going to ever consider them, and government departments won't consider alternatives if they can't get everything from a single vendor in a 100% risk free manner thanks to the "nobody ever got fired for buying Microsoft" and "one throat to choke" mentality you get there.
Governments do this to themselves. The USG doesn't even have to pick Microsoft. They could potentially sign contracts with Apple for workstation hardware and services, that would encourage and feed the alternative ecosystem based around Apple, or they could fund Linux, etc. They don't though. It's just soooo much easier to sign one giant contract, because that minimizes work for them and it's not really important to get value for money if you're in the public sector. Signing a great deal won't get you a big bonus or anything like that, but taking risks can get you blocked from promotions. So, why risk anything?
Depends on the hardware. Stuff like mice, keyboards or game controllers yes, as long as they're supported by the host OS stack (don't get me started on bluez and the clusterfuck that's Linux audio in general). But anything dealing with user-mode USB drivers (looking at you Samsung ODIN or a truckload of webcams) or more exotic hardware is out of luck.
> Governments do this to themselves. The USG doesn't even have to pick Microsoft. They could potentially sign contracts with Apple for workstation hardware and services, that would encourage and feed the alternative ecosystem based around Apple, or they could fund Linux, etc. They don't though.
Apple shot themselves into their own foot here by completely discontinuing their Intel x86 lineup. You can run Windows stuff on the M series SoCs but performance is atrocious (unless it's ARM Windows, but good luck finding software compatible with that oddity, no thanks to Qualcomm here who couldn't be arsed to put out actually usable chips for years).
As for Linux, there is already a huge amount of government funding into Linux because of servers and their support contracts. That's how RH, SuSE and the other commercial Linux distributions are surviving. The problem is the desktop software stuff, here the chicken-egg problem comes into force - and made worse by the fact that unlike Microsoft who can just decree whatever they want and the rest of the world has to accept it, FOSS projects are mostly driven-by-consensus, and if you're a commercial or government entity needing a feature you have to either fork off with all the cost that entails or herding cats and playing petty politics with people from all around the world (who might just block your idea out of principle after finding out you're working on behalf of the government/Monsanto/whoever is problematic these days, on top).
A lot of the software getting hacked isn't even truly Windows-specific stuff to begin with though. VPN appliances, firewalls, they're probably all running out of date Linux distros. Active Directory servers could have some open source competitor too, aren't they mostly Kerberos? And for the rest, ChromeOS could knock it off too with a mix of web apps and RDP.
Yeah, but you'd need to rewrite the whole Windows app for a native Linux or macOS port because WINE to my knowledge can't access devices even if there were a Linux/macOS driver.
> Active Directory servers could have some open source competitor too, aren't they mostly Kerberos?
AD is so much more than just LDAP+Kerberos. You have a fully integrated file store, native and direct integration into Exchange (on-prem and cloud), group policies for users and machines, DNS, DHCP, NTP... one vendor, one tech stack, one management way. Set it up and you're done, and even a novice can set up a minimal working small-business environment in a day.
With Linux, you have to integrate/configure Samba (LDAP, GPO, file storage), Heimdall/MIT Kerberos, postfix/exim (Mail), no idea what one could even use for calendar sync, BIND (DNS), ISC DHCP server, Jitsi (Video calls), Mattermost (Teams chats). Each of these software pieces have their own configuration syntax, nothing integrates automatically on its own, guides are sometimes horribly outdated, there's subtle differences in behavior between distributions, some of these tools have web UIs for management, others are CLI, and all of them look and behave differently.
"Wine includes some limited support for direct hardware access, including running native Windows drivers. This comes with several restrictions, but we still have been able to successfully access some native devices."
"For example, the SteelSeries USB mouse driver offers the ability to modify the mouse's LED settings as well as simply driving the mouse, and in this case Wine is able to pass through the LED settings"
Thunderbird is a lot more reliable than outlook - I use both daily. But yeah there's no 'thunderbird software suite'. I don't think that's really an issue though. Outlook breaks regularly, and our desktop support group had to deploy an 'outlook reset' app because the vendor is unresponsive. Being that unresponsive lessens the attractiveness of a 'software suite'.
>Or hell, even compatibility with old versions of the same app isn't a given in LibreOffice.
If I find MS word can't open a file, Libreoffice always does. I've never had a problem opening a libreoffice writer or MS Word file. I'm no 'power user' though.
The reason businesses and organizations stick to microsoft is because it's what they're used to, not because the apps themselves are better.
As for microsoft as a vendor - they are as obtuse a vendor as I've ever dealt with. Nearly always 100% unresponsive - even with projects involving hundreds of thousands of yearly dollars in licensing fees. I think being unresponsive is some kind of corporate culture thing with them. They're like the McDonalds of IT vendors. You get what's on the menu and that's it. If it doesn't work, well too bad because they know the perceived cost of switching is believed to be too high.
For me, it is. Outlook natively integrates with Teams, Onedrive and the rest of the Office suite. Everything is available at once, with one click, in a reasonably similar UX/UI pattern (okay, outside of the horror that is Teams with direct chats, meeting chats and Teams team chats). Let's just take a Teams meeting invite... in Outlook it directly opens the Teams meeting, in Thunderbird first the web browser pops up, which then opens Teams, and leaves a stray tab behind. It may be just ten seconds, but it's so much more annoying.
and we know that all monopolies require government enforcement to prevent others from competing...
then we know that there's no such thing as a conflict between Microsoft and the us government
the us govt is to serve Microsoft and that's that. any conversation like this about the merits of Microsoft as a market participant are laughable and disingenuous
theres no programmer alive that knows the history of Microsoft that would speak about them as you have just done, as an honest company trying to make a product
actually sick to see how HN has been corrupted
I know the history of MS anti-trust fairly well having grown up during the core part of their monopolist years - now they're part of the familiar oligopaly that's strangling tech.
> and we know that all monopolies require government enforcement to prevent others from competing...
Do we know that? That may be true of government run monopologies, but I'm pretty sure most marketplace monopolies rely on lack of government enforcement. For example, here's some analysis showing how US case law makes the government wary of going after Amazon for driving Quidsi out of business & forcing an acquisition by Amazon: https://cei.org/blog/amazons-private-labels-dont-threaten-co.... Ironically US does enforce rules against dumping from international players which shows that it realizes the harm that such activity can have, it just chooses to allow it for domestic players abusing smaller domestic players.
Could be the opposite. Every provider is an attack vector of the government's network.
The problem is MS fucked up big time by not only losing a key but also that this key is a master key due to an error.
Not to mention they downplayed the problem afterwards and tried to charge customers for the logs needed to identify the attack.
Lax concern and slow turnaround for CVEs, poor update performance that led to much of the world turning off automatic updates, and updates being blocked on non-activated installs, if they weren't just easily broken gave root (literally) to fleets of vulnerable internet connected Windows systems being leveraged against the world at large for roughly a decade until Windows 10 brought (and forced) endpoint security practices into the 21th century.
It's reassuring that their official stance hasn't changed; "This is how big we are, and it'd be a real tragedy if we couldn't afford to stop something like what we caused before from happening again...", instead of being sued into responsibility by the world at large.
(Uh oh, not 5 minutes in and it looks like Microsoft PR is already here to downvote the trouble away.)
“NotPetya” and StuxNet were sophisticated worms/viruses which leveraged flaws in Microsoft products to ultimately bring down infra (ie, Iran nuclear program) or bring down countries (ie, Ukraine suffered attacks to energy sector)
Commenting on your downvotes is off-topic, against the rules, and when done as a snide insult like above: rude. GP might experience better results if they edited their post to remove the snide, insulting comments about downvoters.
Might be very expensive initially, but then they'd have complete control.
The 'outsource everything' 'not allowed to compete with private industry' mentalities are what has the made the government unable to function in a quality manner ... Its virtually impossible for the government to just hire some people to a team to build some shit -- instead they are _required_ to create bids for contractors to bid on and then incredibly formal contract management processes that are just incredibly disfunctional by design ...
It doesn't have to be this way -- its a political result going back to the 'small government' movement which was ultimately about proving that government has to be bad at everything by imposing rules to ensure that result in as many places as possible ...
Much like the consumer decision on the other side of that secrecy barrier, when it was decided the MULTICS was too complex, and you really didn't need all of that complexity (to allow mutli-level security), the easing of hardware requirements, and systems design time all seemed worth it.
Here we are 50 years later, and both decisions have proven to be disastrous. We've swiss-cheesed our infrastructure in a manner that can only be hinted at metaphorically. The closest analogies I know are:
In both cases, everything becomes all or nothing. This is what we have with computers, all or nothing. You have to trust software you run, or you simply can't run it. It's the same with the "License Agreement", nobody actually reads them, and nobody tries to enforce them, yet there they are.It's possible to have secure computing, we had it in the 1980s, but we didn't realize it. It was crude, but effective, because we had systems with no persistent internal storage, and we could write protect the storage we had. Thus it was possible to manage data, and the side effects allowed, in a simple, powerful way by simply not putting data at risk, making copies, and using those instead.
Until we reverse the bad decisions of the 1970s, we're going to keep up the deranged behavior, blaming everything and everyone, and ignoring the one way out of this.
You shouldn't ever have to trust a program you execute to behave in the manner you expect. Your computer should enforce the limits you expect, and the program should have zero ways around those limits.
[1] https://csrc.nist.rip/publications/history/ande72.pdf
[2] https://en.wikipedia.org/wiki/Capability-based_security
[3] https://web.archive.org/web/20120919111301/http://www.albany...
A. It is not metaphorically swiss-cheesed, the swiss-cheese model [1] of software security is officially endorsed. Just stack enough trash and maybe it will be okay is now accepted policy.
B. It was not driven by the NSA. It was actually driven by commercial software vendors like Microsoft and Cisco who could not meet minimum security standards, so demanded the standards be lowered to allow them to make sales. Any time the NSA gets the drivers seat in setting government security procurement standards they almost always push for actual security since the NSA is the government; they are protecting themselves. It is reasonable to distrust them for commercial and external systems, but you can trust their standards for their own and government systems are not meaningfully compromised.
[1] https://en.wikipedia.org/wiki/Swiss_cheese_model
Also, capabilities are irrelevant to the types of hack being discussed on this thread where the vectors are forgotten admin accounts, non-patched software, etc.
The ability, at run time, to pick a file, and give only that file, folder, or set of resources to a piece of software is essential to secure computing. As far as I know, NeXT/macOS/iOS et all don't have that ability.
1. Write a macOS app that has a button which uses the file open window to load a document.
2. Enable the app sandbox by adding the relevant entitlement.
3. Prove to yourself it works by adding some code that directly fopen()s a file in your home directory and watching it fail.
4. Open that file by clicking the button and selecting it in the open dialog. Observe that you can now read it.
Or just go read Apple's documentation that explains their capabilities architecture in detail. Look for docs on sandbox extensions and Mach ports.
The ONLY way that breach got detected was because the State department bought the premium package with extra logging that let them see when mailboxes get opened. It turned out, Microsoft had a signing key that could create access tokens for anything in their cloud, and it was stolen by Storm-0558. (More precisely, the key was only supposed to be useful for a portion of their services, but a bug allowed Storm-0558 to bypass that scope limitation.) And they used that to go read the e-mails of the State department and a bunch of other organizations, and private individuals. There was nothing customers could do to prevent the attack, and apparently no other indication in their logs that it was taking place, besides this category of entry that was gatekept behind a premium subscription package.
Microsoft generated the key in 2016 and discontinued it years prior to the incident, but it was never revoked. Microsoft didn't even bother with key rotations anymore after 2021 because one time they fucked it up and it caused an outage, so they decided to just not do that anymore. Also, Microsoft apparently didn't have any means of detecting the obvious use of a zombie key.
Also, Microsoft still doesn't really know how they got the key. They made a blog post about their theory, representing it as something they were highly confident in based on the evidence. After 6 months of pressure from the government, Microsoft finally updated the post to admit that they had no evidence of critical parts of what they claimed, and several key points in their narrative were factually incorrect.
Then earlier this year, Microsoft got hacked AGAIN because they had an unused-but-active test account with a guessable password and no MFA, and it was authorized for access to e-mail boxes of (at a minimum) numerous members of Microsoft senior leadership.
Microsoft has got serious problems.
edit: I keep futzing with my phrasing. Those wanting a much better account should just read the report, since it has a great deal more nuance and information. https://www.cisa.gov/sites/default/files/2024-04/CSRB_Review...
Key rotation is almost like restoring from backups. It's an absolutely necessary capability and practice.
Other nonsense I've seen: leaking internally signed tokens for external use (front-channel), JWTs being validated without a kid claim in the header - so there's some sketchy coupling going on, skipping audience validation, etc...
Not much surprises me anymore when it comes to this kinda stuff - internally, I suspect most cloud providers operate like "feature factories" and security is treated as a CYA/least-concern thing. Try pushing for proper authz infrastructure inside your company and see what kinda support you'll get.
1. As MS found, revoking old keys is very risky because doing so creates outages. But if you don't do it then changing keys is useless. This isn't a problem specific to Microsoft. Lots of companies have learned this lesson the hard way.
2. It assumes that attackers don't just use stolen keys immediately (e.g. to issue more keys, change passwords, create new accounts etc). In practice they usually do.
3. It assumes that if you change the keys the attackers can't just immediately re-steal the new keys.
So it's only really a useful practice in one very specific scenario: you do something that boots undetected attackers out of your network without realising that's what it did, and the attackers need ongoing access that only that key can provide, and they can't use that key to elevate permissions in a more permanent way like by creating a new account on the system or stealing a user password. Pretty specific scenario.
Unfortunately, key rotation also comes with big downsides. Any software that works with keys has to be built to tolerate a change silently, because now it's a regular occurrence instead of a rare one (where maybe a bit of disruption can be absorbed). That creates complexity and therefore bugs. And because it's a repetitive piece of fiddly and complex work that can break your entire service if you get it wrong it inevitably gets automated, and that in turn means that you end up with a large collection of highly privileged subsystems that have the power to silently change keys in ways admins won't notice because they are expecting it: exactly the sort of thing attackers will immediately focus on.
Overall it's not an obviously winning move. Opportunity cost matters too. Whilst you're setting up all the infrastructure to do this, ironing out the bugs, cleaning up after the outages etc, your competitors might be investing in other kinds of security best practices that are more effective. It's especially useless here because MS don't know how the key was stolen to begin with, so there's no reason to think that if they changed it that would have had any effect. Most likely it could have just been immediately restolen and all the effort would have been theatre.
https://www.theverge.com/2022/3/25/22995144/microsoft-foreig...
https://en.wikipedia.org/wiki/Microsoft_licensing_corruption...
https://www.wsj.com/articles/BL-CJB-17439
What incentive do cloud providers have to add extra security to the government stuff, while not also adding that extra security to the public stuff?
"Oh, yeah, we have more security over in the thing you can't use, but you can trust us with your data anyway."
https://securityintelligence.com/news/cisa-hackers-key-syste...
Apple has same within US schools oll over the country and nobody even cares. Your government will change but your children will be with you for a while.
It's well known among the people who serve that it's a joke.
Source: was IT on a destroyer
A similar claim could obviously be made of Boeing. Just imagine what is happening in their military contracts which we are not allowed to hear about. Looking at the projects which we are allowed to know about, airliners and Boeing's Starliner, clearly Boeing management needs to be put out to pasture.
The core issue is cutting corners, for profit. This is not an issue which is easy to handle in our system. It seems that the best we can do is name and shame. Let's do that at least.
It’s the tiny brick supporting their entire skyscraper.
>The NSA is also tasked with the protection of U.S. communications networks and information systems.
"Hey guys, we should fix this vulnerability ASAP."
"No can do, it has to wait and be stealth-patched indirectly, otherwise we'll lose the advantage of using it ourselves."
The dependency on (pretty much) a single vendor across the entire military organization is such a monumental single point of failure, I can't imagine what level of obtuse bureaucracy led them there. It seems self-inflicted at best; if it wasn't MS it would've been some other corp.
IMO the military (especially given their incredible size and access to resources) should have 100% rolled their own IT solutions from first principles, and built upon that.
1: https://www.youtube.com/watch?v=2zpCOYkdvTQ
If you ACTUALLY want security, you're going to be dealing with computers which are 100x slower than what we're used to with a tiny fraction of the features. For the most part I'm not convinced people actually want secure computer systems, they want computer systems which are productive and they'll just cope with the regular security breaches.
I'm also not saying this based on people reporting vulnerabilities, I'm pointing out the software we're using is fundamentally impossible to secure on merit of its complexity and the tradeoffs people make that neglect security.
Proprietary software is a threat to national security
"Switching vendors" does nothing to solve the core problems here
> According to an early report on the breach, an anonymous source said that the compromised systems were the Infrastructure Protection (IP) Gateway, which houses critical information about the interdependency of U.S. infrastructure, and the Chemical Security Assessment Tool (CSAT), which houses private sector chemical security plans.
> CSAT is an online portal that contains highly sensitive information that determines which facilities are considered high-risk under the Chemical Facility Anti-Terrorism Standards (CFATS).
> CISA declined to confirm or deny which of their systems were taken offline.
https://securityintelligence.com/news/cisa-hackers-key-syste...
> In late February, CISA had already issued a warning that cyber threat actors are exploiting previously identified vulnerabilities in Ivanti Connect Secure and Ivanti Policy Secure gateways. Ivanti Connect Secure is a widely deployed SSL VPN, while Ivanti Policy Secure (IPS) is a network access control (NAC) solution.
> Now, CISA itself has fallen victim to a cyberattack involving Ivanti products.
(This applies to a number of reckless tech companies upon which countries and other companies create dependencies. But MS is one of the most concerning to me.)