> The Pentagon said in the press release that it still needs enterprise-scale cloud capability and announced a new multi-vendor contract known as the Joint Warfighter Cloud Capability. The agency said it plans to solicit proposals from both Amazon and Microsoft for the contract, adding that they are the only cloud service providers that can meet its needs.
If GCP have fedramp high that’s roughly equivalent to AWS govcloud (fedramp high ~= IL4).
AWS and Azure also have IL6 regions (able to hold secret information).
Finally AWS runs a top secret cloud for the intelligence community. I believe Azure announced a similar project but not when it’ll be open for business.
That was an excuse and a convenient cover story. GCP can't even serve the needs of large organizations, they definitely can't service the government. Couple that with the leaked plans to shut the thing down because it's an abject failure (it's a money pit for Google), and why would you want to be on the platform of certain disaster?
I've had demos from GCP sales reps, and the platform is shambolic. It doesn't even work during sales presentations.
GCP has some good bits. They're just running into the same problem everyone had catching Windows: it's really hard to compete from a fresh start with something that's been incrementally improved over years of significant use.
Especially when you're Google (our customers aren't as smart as us) competing with Amazon.
Your example of a "large organization" is the state of Arizona? Australia Post, a tiny regional mail delivery company? What's your definition of "large"?
Also, it's evident they cannot service those customers. Australia Post has a large contract with AWS. Whoops.
I've worked in many large organizations. I've received GCP pitches. I've participated in evaluations of GCP as a technology solution. It's substantially worse than Azure, plus according to their leaked massive losses on the product it'll be shuttered this decade.
>This feels very hyperbolic to me
Their dashboards are typical Angular heavyweight clunk, with confusing error messages, infinite loading spinners and things break constantly. It's been discussed here extensively. [1]
Their actual decent products (BigQuery) are hamstrung by the rest of the failing platform.
The best thing that happened for Windows Server and Exchange is Microsoft started to run them at scale in house. Suddenly things that had been complained about for decades magically started getting fixed :)
A memo went out last year about all federal networks supporting IPv6 only in the coming years. Since Google's cloud doesn't really support IPv6 at all, I don't think they were going to win the contract anyway.
Everyone bar AWS is focusing on multicloud. And VMware's offerings are utter shite and very poor features wise ( where it matters), so I doubt they'll be impacted.
( Their offerings were so bad they were forced to sell their vSphere as a service arm to a low cost hosting provider. Even with the popularity of that dumpster fire in DCs and most companies moving away from DCs they still couldn't capture any market share)
I can understand Google employees' passion to curb the power of military. I myself strongly support keeping government's power in check too. But the hatred towards the military, to the point of actively sabotaging military's effort to improve itself? That I don't understand. I really wish those employees travel back in time to experience the European people's life under Mongolian's reign, or the Aztec's life when Spaniards attacked, or the life of people in Manchurian when Nurgaci's tribe was rising in the early 17th century, or the life of Chinese people merely a hundred years ago when Japanese invaded. Shouldn't it mean something when millions of innocent people were slaughtered in a matter of years, or when a civilization (especially a more advanced one) got destroyed, or when a nation's human rights were stepped on?
P.S., it's worth mentioning that it was the Manchurian who restored slavery in Qing Dynasty. The word Nucai in Chinese or 명사 in Korean, meaning Your Slave, was such an honorary title that for more than 300 years until early 20th century only those who were trusted by the royals could use. Yeah, don't wanna be a slave? Build a good army.
isn't the US Military the equivalent of the "Manchu" here? (The US Military wiped out the natives, took all their land, and brought and enforced slavery on this land?) Didn't the US just annex Hawaii as an official state a few years ago? Isn't all of California land that the military just kinda stole from Mexico?
Then, it means that they are run by people who are not blinded by greed, and care about earth .... Ahh who am I kidding! They will both say yes. Amazon even sued the government for it
Not surprised. Redoing this drops any ability to suspect Trump's tampering, and in the past three or four years, they likely have a wish list of adjustments anyways. Starting over allows them to address both.
In which case, there's probably no conflict because you don't infringe trademarks if you're in totally different markets. Which is why it's OK that there's a Columbia encyclopedia and a Columbia sportswear. Or Fisker scissors and Fisker electric cars.
It's probably challenging to argue the government is infringing Disney's copyright: It's not a remotely related field of industry, it's not for profit, and the US government is not technically able to hold IP rights itself to anything.
"Star Wars" itself was the name of a previous defense program, so, I'm sure this debate was had out decades ago.
"Star Wars" was actually not the name of the program, it was a derogatory nickname coined by Ted Kennedy, akin to calling the ACA "Obamacare." [1] The formal name of the program was "Strategic Defense Initiative," or "SDI."
I work for Platform One and many of us are continually amazed that Disney hasn't dinged us for this yet. I'm pretty sure there have been plans for a while now to get a new logo that isn't such a blatant rip off but I have no idea where they're going.
I mentioned in another comment that George Lucas tried to sue over people calling the SDI "Star Wars" back in 1985. I can't find any evidence now, but I believe he once sued either the Air Force or NRO (can't remember which) over a mission patch with an x-wing on it.
I suspect that the original poster did capitalize it but HN 'helpfully' edits such full word capitalization in titles to simple capitalization. However, what many may not be aware of is that you can go back and fix the title and resubmit it and it will be saved as edited. You have, I believe a 2 hour window in which to do such an edit.
My wife currently works as the test lead for the NRO's next iteration of the Sentient/ECO program and they named the new program Cyberdyne. No relation to the actual Terminator program.
Her last program held a voting pool to pick a name and ended up with "Totally Not a Laser Death Ray Facility" unfortunately coming in 2nd place.
Serious question. How does one go about actually getting a job with the NRO? I’ve heard it doesn’t even have employees, just a consortium of contractors.
Since the basics of Sentient were only declassified in 2019 it’s possible disclosure of the fact there’s a next iteration and it’s name is Cyberdyne would constitute disclosure of classified information?
Yeah, it seems suborbital is really only good for tourism and getting good rocketry practice. At least they're on the path to orbital: New Glenn appears to be late 2022 with some customers booked in 2023.
I will believe literally nothing about the New Glenn until it launches. Not to mention landing the first stage, which was the only part which made this rocket competitive, and is _extremely_ hard to get right on the first try.
Space X will be landing the Starship on the Moon and Bozo will still be flying 3-4 people to the stratosphere. Guy should stick to what he is good at, running a sweat shop and trinket delivery. I put more faith in Boeing as I find Blue Origin more of a C player.
Perhaps, while it is still a tough field, Nowadays there are plenty of successful companies in this space : rocket labs, astra , spaceX , virgin orbit have all launched payloads in the last 10 years then there are new players like Terran .
It is still not easy , but as difficult as say 20 years back.
Technology has progressed quite a bit in 20 years, of course. And, I don't know the answer, but with all of these companies joining in on the space race, did regulations lighten up or was it just the funding cuts to the space program that spurred this?
Regulations are still tight in many areas due to ITAR , the concerns to prevent basaltic missile tech from potentially being leaked to countries without it have not changed.
To an extent NASA's Commercial Crew program and other commercial engagement vision worked in triggering commercial industry
I think it more that technology has changing a lot making it less expensive to attempt building a vehicle. Today, for example Terran is considering building a lot of their rocket with 3D printing, Rocket Labs does that already for their rocket parts. Talent availability with the right skills has become easier in the recent years probably helps.
While I don’t know the specifics of JEDI, it seems problematic that military data is being stored in a civilian center at all, and one that’s likely to be internet connected. Every day practically we see a new breach, and there’s two things in common: a weak point in software, and the internet. Will JEDI be completely air gapped?
The expectations placed on military compliant service is pretty significant, in that it's generally going to be dedicated boxes and networks and significant controls to prevent any easy movement between enterprise or consumer users and government. (Most government cloud customers are already on a pretty separated area from everything else.)
But presumably Internet connectivity is the entire point of why they'd want a cloud contract: The government already has servers in buildings. It sounds like they particularly want the global reach of a cloud service, probably for activities intentionally intended to be on the Internet.
I mean, bear in mind, if you're a state-level actor that engages in cyberattacks, you need to be on the Internet to do them... Not saying that's the purpose of the JEDI contract, just that... there's plenty of scenarios where the military wants Internet-connected things.
In taking a contract like this, I (as a company owner) would be more concerned with the thought that I just added my datacenter to someone's priority target list.
On the other hand, let's be honest... In strategy for a modern war, Microsoft and Amazon's major datacenters are already on someone's priority target list.
Yeah, the thing is too, global actors have already learned that attacking corporate entities is a really good way to screw with our public sector too. And it's a lot easier to wreak havoc by attacking major municipalities than trying to inject code onto an F-35, and arguably, the former is capable of doing more damage than the latter.
You can do both. And I doubt it’s about trying to modify the f-35 design (tho maybe), but more likely simply exfiltrate all the data, possibly with some ability to “rm *” on demand.
There is a reason Amazon and Microsoft keep the locations of their datacenters secret. They are unlabeled buildings. If you drove by it you'd have no idea what it was.
That secrecy really only works for the 90-99% of people they want to keep out who would be showing up at their door for "lookey-loo" purposes.
Every nation-state actor knows where every datacenter is of every major digital player, because they've already implanted spies in those companies and the location maps of those datacenters are not internally secret. It'd be very hard to make them internally secret, since those DCs communicate with the rest of the network and must be controlled in realtime by site reliability engineers.
Pretty much. If you have a datacenter in your local vicinity, it's also not hard to find it: It's the building with far more air conditioning equipment than seems reasonable for a building it's size, with security cameras every two and a half feet.
Once upon a time, telecommunication networks were built point-to-point or otherwise non-publicly switched. I believe the telephone network, for example, is still this way, which includes more than just telephones but also ISDN lines and likely other technologies/use cases as well. There's no reason you couldn't have military facilities on their own network connected to servers in their own cloud, as this AWS instance has shown.
> The AWS Top Secret Region was launched three years ago as the first air-gapped commercial cloud and customers across the U.S. Intelligence Community have made it a resounding success.
AWS has provided air-gapped regions to the US Government for 6+ years, according to that blog post.
Don't know why this is getting downvoted; it's an interesting question, if a simple top-level one. I'm not sure myself why the DoD can't just build its own "cloud" with a $10 billion dollar budget, instead of putting the security of our military apparatus in the hands of some private company.
The same reason it doesn't build it's own "plane" or "ship" or "submarine" or "satellite" or "gun" or "bullet".
The DoD's job is defense, so it's about fighting wars, not about building infrastructure and supplies.
The security of the military apparatus will stay in the DoD. The implementation and operation of the security infrastructure will be delivered by government contract.
> Last year, the Pentagon’s inspector general released a report saying that the award did not appear to be influenced by the White House.
However, the inspector general noted in the 313-page report published in April 2020, that it had limited cooperation from White House officials throughout its review and, as a result, it could not complete its assessment of allegations of ethical misconduct.
Everyone knows the Bezos-run WaPo is Pravda for the Democratic Party. It’s not a huge stretch to think Bezos is being repaid for his 2020 contributions. It really feels like both MS and Amazon should have been kept out of bidding for this out of CoI fears.
WaPo is propaganda for Bezos, but even then he tries to pretend that there is some editorial independence.
WaPo would only be pro-Democrats as long as it benefited Bezos, or if individual journalists leaned that way.
I really doubt there is a quid pro quo of a $10bn contract for what limited sway Bezos has over WaPo.
If anything, the fact that Oracle did not receive the original contract shows how limited political contributions were in affecting the outcome of this contract. Larry Ellison was a big financier of the last administration.
Where does this myth that Bezos is giant Democratic Party supporter come from? Amazon and it’s subsidiaries donate to both major political parties and many members of the Democratic Party are very vocally against Bezos.
I think it takes a fairly narrow, USA_centric, unaware of actual variety and detail of the world outside perspective, to think Washington Post is either "Far Left" (hint: it goes a lot further than WaPo;) or like Pravda.
While I think it does have somewhat progressive slant, especially in more opinion pieces; you only have to go to our mainstream Canadian party to go WAAAAY further left than WaPo - and that's nothing to say of other continents, countries, parties and newspapers.
Why do non-US people feel the need to whine about this all the time? "They wouldn't be far-left in my country!" is completely irrelevant and contributes nothing to the conversation. In the US, which is where both I and WaPo reside, they are far-left.
I'm a US person and I have to say I agree that Washington Post is not far-left. While it's true that the relevant question is "what are the conditions in my country?", it's important to remember that part of the GOP's identity politics has been to paint even moderately left and centrist positions as "far-left".
That said, within the US political climate, WaPo is definitely left of center. Just not far-left.
Maybe because people like you bring up a weird stance that isn’t objectively true snd likely not true to the majority of Americans. WaPo is centerist neoliberal in reality, but Americans likely view it as to the left and liberal. But nothing wild. They aren’t going crazy for Bernie or Ilhan Omar. They certainly are not talking about people to the left of them. Who would be far left.
Your point really breaks down when you would have no way to classify thoughts and media one or two or three steps more to The left of WaPo. Even if you add “radical left” for one spot further to the left. There’s nothing else after that.
Or you’re taking the stance of Fox News and co. who will dishonestly wonder if Biden is a socialist.
Fair enough, let's leave the rest of the world aside:
I am disputing the notion that in USA, Washington Post is "far left".
I can understand that for any given person any given media outlet may feel central, a little, or far to the left or right of their position.
But if we are to create any kind of external scale, with whatever measurement (relative based on how far axis go, or by some thresholds of specific criteria), I would doubt Washington Post fits on the "far left" of that scale.
(in for a penny, in for a pound addendum: I understand that in USA there's also an effort/inclination to label ANYthing "left" as extreme, far left, antifa, socialist, pinkie, commie, fake news, etc; terms that are implicitly meant to be derogatory; but that effort does not need to be engaged with in a rational discourse. To rephrase: Just because Glen Beck or Bill O'Reilly et cetera may call Washington Post "Far Left", doesn't make it so - in those cases it's a label meant to paint & incite and cater to an "in crowd", not be a part of meaningful engagement and honest wide-ranging discussion)
[Addendum to addendum: there was nothing in original post that said "Washington Post is Far Left in USA term "; it was a open claim posted to an international site, many of whom read or access Washington Post as a globally-relevant newspaper, so I don't feel assuming a global perspective is "whining". Or to put it in your preferred terms: Why do some USA people whine where perspective is brought in :-]
If you consider the Washington Post to be far-left, you might need to recalibrate your Overton window. The paper leans slightly left. It's far from the Daily Kos.
> Where does this myth that Bezos is giant Democratic Party supporter come from?
Believe it started with Trump. The Post is a beltway paper. It’s a chronicle for Washington insiders, and they never liked Trump. So the Post had a critical stance towards him, which in turn lead to him branding it as leftist. (Which is false. Its bias is best described as institutionalist and statist, which finds alignment across the political spectrum and parties.)
No, they’ll need a new target to paint as a ‘globalist,’ Bezos doesn’t inspire the same amount of irrational hatred in this crowd for some completely unknown reason.
The reason Soros gets such hate is straight up antisemitism. The dude's Jewish. The "globalist" epithet is a modern reworking of the old "international Jewish banker" stereotype. That particular dogwhistle doesn't apply to Bezos.
He owns the Washington Post and they wrote articles critical of Trump. Trump is a manbaby that can't handle criticism, so he attacked the owner.
All part of the hurr-durr-main-stream-media-is-left-wing-antifa-deep-state bullshit that has infected the US (and to a lesser extent, UK/CA/AU/NZ etc).
You can blame Trump for it. He tainted the contract process because he hates Bezos (who owns WaPo). This was the key point that Amazon focused on in their objection.
edit: re: downvotes - the article states this in the sixth paragraph
Except there was never any evidence that Trump got involved, even after multiple independent reviews. MSFT just made a better proposal and AMZN was able to use the broken protest process to tie things up long enough to make canceling the fastest way forward for the DoD.
"A federal court said on Wednesday that it did not dismiss the possibility that former President Donald J. Trump interfered in the awarding of a military cloud-computing contract worth $10 billion, a decision that could result in the overhaul of a long-running effort to modernize technology at the Defense Department."
They key thing was the court saw enough smoke to keep case alive AND it looked like recently was going to force some disclosure around communications from whitehouse (whitehouse has variously claimed no involvement).
And Trump hasn't dismissed the possibility that he won the 2020 election. At some point you have to produce the evidence. Where are the career DoD procurement people leaking to WaPo that they were pressured?
This was AMZN thinking that all they needed to include in their proposal was market share numbers. MSFT's proposal was more responsive to the DoD's needs.
A couple of points. The whitehouse blocked any disclosures of discussions / pressure even to the inspector general. They also issued notices to staff not to talk to IG. That's a pretty darn good sign that there WAS involvement by the whitehouse.
The DoD is dropping this case because they were asked to reveal some of this information. Instead of disclosing it, they elect to cancel the contract. This was a specific issue in the current case, would DoD have to reveal this type of communication from whitehouse.
Yes - people claim no pressure / no involvement - but it's blindingly obvious that there is a very good chance there was pressure.
I have past experience with govt contracting so have a sense for what the flows look like.
Contract selection put on hold, involvement of a political person (probably esper here). Ban on anyone talking about the potential discussion to internal auditors. Those are all the red flags.
I was responding to this statement - "Except there was never any evidence that Trump got involved, even after multiple independent reviews."
The IG review specifically noted all the ways that Trump / Whitehouse blocked the answer to that question.
The Whitehouse then also got rid of IG Fine in April 2020 - I mean, this guy served 28 years in justice then through bush / clinton / obama terms as an IG.
Amazon also dug up stuff where Trump promised to GET involved.
No one "cleared" trump in this. He stonewalled as long as he could.
I'm sorry but the second these words left his mouth obviously indicating his displeasure with the likelihood of Amazon winning the contract he had influenced the outcome.
“I will be asking them to look at it very closely to see what’s going on because I have had very few things where there’s been such complaining,” Trump said. "Not only complaining from the media — or at least asking questions about it from the media — but complaining from different companies like Microsoft and Oracle and IBM. Great companies are complaining about it. So we’re going to take a look at it. We’ll take a very strong look at it. Thank you very much everybody.”
He head already shown at this point that he was an old boys club type of guy that would go after anyone who disagreed and support anyone who kissed his ass, so anyone having a hand in the selection of the winner of the contract that wanted favor with the president would obviously attempt to influence the outcome to not be Amazon.
He should have kept his mouth shut on such a massive pending contract, but it's pretty obvious by now that keeping his mouth shut isn't something he's physically capable of doing.
Now instead of having started on a crucial service that is needed by our military we're looking at likely further years of delays. All because the president was butthurt over some mean words in a newspaper.
> Except there was never any evidence that Trump got involved, even after multiple independent reviews.
That doesn't matter. We're not talking about criminal proceedings, where anyone has to be proven guilty beyond reasonable doubt. This is, at most, a civil law process. And it's the DoD canceling its own contract, which they are probably allowed to do for whatever reason.
The contract should have gone to Amazon years ago. The fact that two fair evaluations in their favor got overturned (first by Oracle's lobbyists, then by Trump's intervention) was the cheating part.
This ignores the time value of money. What if the couple of years prevents numerous security intrusions by foreign entities? It's not as if MSFT doesn't have the expertise or the manpower to fulfill the contract
Considering the DoD themselves say that the terms of the contract are outdated and they don't need it anymore, I'd trust their decision over random online comments.
The same DoD that has suffered massive leaks over the past 15 years and constantly misplaces funds from its budget? Not all random online comments are created equal
All this does is add friction. I've worked on defense programs that were "shared" between contractors and it's massively stifling on top of the security constraints.
Ben Rich's stories should be considered in context. If you take his stories at face value, then the Skunk Works were significantly more competent AND HONEST than any of their competitors, or the rest of Lockheed!
He found the bullshit tiring because he felt that he didn't deserve to deal with it, because he trusted in his own competency and honesty.
You can see how all the pieces fall together. Somehow, the government has reached a default stance of not really trusting in the honesty (and maybe competence) of their contractors. And frankly, Ben Rich didn't really believe in his competitors either. He just didn't care because he only writes about projects that he's also competing in, so he thinks that his competence will carry him through (which it generally did... not for the F-16 though lol). What about all the competitions that doesn't have the strangely competence and honest vendor?
DoD saw what a gong-show single source could be (look at F-35). They figure multi-vendor, open architecture is the way to mitigate that risk.
The state should procure an entirely open source / public domain cloud stack that would commeditize the "cloud" space allowing any org to run data centers with a standardized interface. The state should then run their own data centers in-house with the stock design rather than outsourcing those operational costs
This would be far better for society. Compute-for-all.
Where would they get the talent and expertise to do that? Whats the most you can make at government job? $170k? How much money do engineers make at AWS? I just had a recruiter call me for architect role that was $500k in compensation a year.
For what? The initial design work and a demo deployment can be outsourced. After the design is proven then cheaper sysadmins can run the thing.
Also, the fact that government jobs are so capped like that is a conspiracy to make the public sector suck, basically. We should fix that too, but I recognize it cannot be done in time, hence allowing that the design work to be outsourced.
The problem with cleaving contractor deliverables from suitability results is that you remove all incentive for quality work.
Or in another words, you're trying to find the one honest person, in the stack of AccgemloitteBMizantPMGataBCS proposals. And everyone else is bidding with the profit expectation of tossing entry-level, over-hyped developers on it, and then crossing their fingers nothing goes wrong.
I am trying to structure the contract to avoid this stuff, by making the deliverables include both the contractors running their own working deployment, and the government also running their own. For the latter, there should be a tech transfer period where the contracter needs to do training and promptly answer questions, and then a government independence period where the contractor is not allowed to intervene at all to prove the knowledge has actually been transferred, and government isn't entering an IT-support protection racket. Only then is full payout given.
Yes, that means finishing the entire project will take quite a bit longer than doing all the dev work. Yes, that also means the project is riskier for the contractor and has less ongoing reward, and so they will need to be paid more initially. But it's worth it in the end.
If there is some gotcha here where the private sector can get out with too much proprietary IP or a weak and dependent customer in the government, do let me know, that is not my intent. I'm trying to envision a transitional project to get out of today's privatized world to one with better US state capacity.
I think many would jump at the opportunity to be paid to pioneer what's known in advance to be highly visible and impactful FOSS project. That's like a manhandle-project-class career legacy for you, at a fraction of the cost for the government.
Someone straight out of college is going to be making close to $50k. $170k would be a very senior person to come in at the top level. I can't imagine building a cloud platform at the government. What type of hardware is the procurement department going to get with the bidding process? You'll also won't able to get unix machine or macs to run as your development machines, it will be windows boxes filled with spyware. I say this as my dad worked as a senior scientist creating software / models to track oil spills and toxic chemical spills. On his last year before retirement, they banned his team from using Macs.
To your second, well, /shrug. A reasonable number of folk might well jump at the chance to have a job like that on their resume, and there's likely a fair amount of possible cross pollination w/ various folk in the DoD/military to bring up bodies and relevant expertise (i.e. security).
Not every co moves at startup speeds. I have seen plenty of large companies move just as slow or slower.
Problems of red tape are not limited to only government ,it is function of most large organizations
Plenty of security engineers are constantly frustrated at the lack of security focus in corporate sector, investments are only made when it affects revenue . Even after all the ransomware companies some times do the bare minimum and buying insurance
Government on the other hand cares about security for different reasons, they take it lot more seriously. They don't sacrifice security in the interest of sales.
I would anyday prefer to work for some org where security actually matters not where it is inconvenience
That's all true, but we should also be willing to pay more in the public sector. Capping government salaries and then paying for consulting is an especially lousy form of privatization.
Government doesn't care about the security of your personal information, hence why they treat their security engineers with contempt by offering them paltry wages and drowning them in red tape.
Government cares about security from foreign threat actors. The loss of data to esponiage is a critical threat.
Most companies I know treat Security engineers similarly and pay not all that much higher. There is world of enterprise IT outside the silicon valley, the pay is not all the much higher. Despite all the recent threats and ransomware attacks security is not treated as it should be.
A former coworker went to work for the USDS. He left a few months later and had nothing positive to say. As a whole, they do not seem in any way to be competitive with the compensation, talent, or innovation of private companies.
You still have to pass a marijuana drug test to work there, for heaven's sake.
You still have to pass a marijuana drug test to work there, for heaven's sake.
I hate to break it to you, but that's normal in the vast majority of companies. So far, I've only lived in two states where it was legal, and in both the laws specifically allowed employers to test for pot. I'm in healthcare, and the company I work for tests everyone on entry, and people randomly after that.
If it's ever legalized at the federal level, expect employer testing to be specifically allowed. Transportation companies, surgeries, heavy industry, and a lot of other companies will filter for this, either because they or their insurance companies don't want the risk.
> I hate to break it to you, but that's normal in the vast majority of companies.
We're talking tech here—the Digital Service—and I've never worked at a company that drug tested their programmers. Not once, since the 90s, during the full-swing War on Drugs.
Anyway, the marijuana test isn't the only antiquated, sclerotic thing about the USDS (and Federal employment in general). But it's certainly a representative example.
If we were serious about fixing this problem, existing federal pay scale constraints would not be a barrier to entry.
Legislation could be passed yesterday that would add arbitrary pay scales for special purposes such as these. Attach an AWS architect salary to a government job and I think you will immediately find the skill gap filled. Bonus points if you put incentive structures in the employee contracts so that the brilliant minds are directly incentivized to deliver, rather than via proxy of their lobbying container organization.
Or how about we just end this culture of legislative micro-management, so that we don't have the problem in the first place?
From there, we can give the agency the budget to do the job, and let someone who's actually close to the situation decide whether it makes more sense to contract it out or build an in-person team. A bunch of legislators who are well-known to be perpetually too busy conducting fundraising lunches with lobbyists to actually read the text of the bills they're crafting and voting on are never going to be making informed decisions on matters at this level of detail.
Legislative oversight of the executive branch is a feature, not a bug. Presidents aren't kings, neither are cabinet members princes.
That said, appropriations is no barrier to hiring the necessary talent. There are plenty of well-run private data centers in the federal government, staffed by both federal employees and government contractors.
Politician micro-management is a real problem in the US, but keep in mind that these restrictions of the bureaucracy are a more second-order effect. They are less about directly giving the politicians more control than keeping the civil service week to the benefit of the contractors, and that outcome continues whether or not the politicians find it worth their time to meddle with any specific project.
Also, it's my understanding that the micro-management is more a problem on the state and local level, like NY state politicians such as Cuomo forcing the MTA to take on debt for these stupid stations rather than improve service.
The current minority party in the Senate, which under the current rules of the chamber is big enough to stop almost all legislation from moving, has no incentive to allow this to happen. In fact they have a strong incentive to oppose it, considering the makeup of their base.
Yup, they do. So it's really important to try to bring some transitional policy about just barely can wedged through that will nonetheless have such a effect that the balance electorial power and/or policy positions of the parties will realign. (In the short term, it may be the former, in the long term in better be the latter.)
Tough cookie!
If you can't do that, well, then you better consider secession or something. I don't want to live out my old age in a "New America, Yukon" military junta rump state because some people sit down, blushing, seeing Rubicons on all sides in 2021.
(The sad irony of course being that crossing Rubicons made military Rubicons then, I know. But it was a lack of meaningful other reforms too. So find a Gracchi metaphor instead, I guess.)
> Legislation could be passed yesterday that would add arbitrary pay scales for special purposes such as these.
Or maybe add real progressive taxation that would get the $500k comps back into normal territory (by, among other things, taxing dividends, share options and the like accordingly).
How many hours you have to work for that ?. How long before you get fired because you didn't kiss someone's ass or can't keep competiting with 27year single male with no other life ?
Compensation is only one aspect of a job. Plenty of people work in defense while private sector pays more, public service motivation, job security, benefits, better work hours, more seniority driven promotions there are a ton of reasons.
Doesn't mean it is the most efficient way to run an org, point is there are enough qualified people who value the other benefits more than just the cash.
Funny you mention 170k Amazon is known to have a fairly strong cap on base salary compensation. Usually at 150-170k range.
I don't know you. I don't know anything about you. I don't mean any disrespect to you, but...
I'll match you against somebody willing to take $150k for the exact same work. I'll match 4 of you against 8 of my hires. I'm willing to bet that my $1.2M team will outperform your $2M team. I'll very happily match 50 of your team against 75 of mine. You could poach 25 of mine who prove to be superstars, and I'll hire more.
That doesn't mean that you are dumb or incompetent or less skilled. I just think that there are some very smart and capable people willing to work for $150k. I think that many of them are as good as any AWS hire. And I think they could be motivated by something besides money to create something as good or better than anything from AWS. Indeed, less money could itself be a strong motivation to outperform.
Thankfully, we don't live in a world where there are only 10 geniuses to go around. The world is much much cooler than that!
I’ll take that bet. 10 150k public sector employees vs 10 250k private sector, its going to be a slaughter in terms of productivity in favor of the private sector. In fact I would take 10 private sector engineers at 250k vs your 20 public sector at 150k and would still win.
The reason is not talent, but process. Governments are bureaucratic by nature, companies are meritocratic by nature. Obviously this is a vast simplification and there are exceptions but I wouldn’t bet on your team being one.
I didn't say my hires would be public sector. They just have to produce something that is freely available to the public sector and meets a specific public sector need.
lol - the larger the company the more bureaucratic they become. I have friends inside the casino industries and they are 10 times worse than I've ever seen government when it comes to gross internal inefficiencies. Makes sense - the government doesn't have to earn its money, it just uses the IRS to confiscate it and casinos don't earn their money either, they just steal it.
I'd take ONE top faang engineer over 8 government software contractors. There's so much tribal knowledge out there that people in the Midwest just don't have, not to mention a can-do attitude.
It blew my mind the first time I worked with a guy from Microsoft on something. We were having issues with some code, and he just popped open the kernel and started actively debugging things that I'd only vaguely even heard about. I feel like I was twice as good at engineering after that experience alone.
I had the same experience from the other direction later on.. a team of smart, hard-working coders had been dealing with stability/perf issues for years on some product. I was able to root cause and straighten all of them out in a couple of weeks, even though this was in an area of software development using a set of tools I'd never touched before.
I'll bet a single FAANG-hardened code wizard could outperform 20 or 30 government coders.
Can you expand on what do you mean by "just popped open the kernel"? I'm genuinely interested in this kind of "tribal knowledge" you talk about. Maybe there is something we (those reading this thread) could learn and make use of.
maybe WPA, which lets you sample stack traces from user-mode all the way through drivers? Or DbgView, which lets you see printk output from kernel mode. Or hell, maybe even Windbg debugging the kernel of a Windows VM over simulated serial (you need a checked build of Windows to get the most bang for your buck there, though.)
the hardest-core thing I've done was step my way through Windows startup into the container subsystem, on a real, physical target machine I had connected to my dev machine over FireWire. I felt like Indiana Jones. (It helped having the source code though!)
I think Sterlind already covered about as much as could fit in a casual comment online.. The basic idea is that you draw a box around your system, then check all the inputs and outputs. If the inputs are good and the outputs are bad, you open the box and draw boxes around what's inside.
Don't be afraid to dig through heap dumps or decode assembly if you have to, although when you hit that point you're probably just going to have to swap out whatever component is misbehaving
> I had the same experience from the other direction later on.. a team of smart, hard-working coders had been dealing with stability/perf issues for years on some product. I was able to root cause and straighten all of them out in a couple of weeks, even though this was in an area of software development using a set of tools I'd never touched before.
I've done this at a company full of ex-Googlers. Some programmers are simply incompetent, no matter where they've worked.
In my experience the good teams have truly exceptional people and the bad teams are filled with people who would be superstars elsewhere. But yeah this isn't proof and hey maybe I'm just saying that because I'm not that good , which is definitely possible.
Yeah that makes sense. I'm sure after 5+ years there you would be far better than the average developer. I am talking from the perspective of people that I've seen go to those companies. Sure they were above average programmers, smart and driven but they weren't superstars or truly exceptional.
Ah yes, that must by why Google's golang has been created because, by the creators own words, Google's hires are not able to pick up things fast enough and that the language was made simple in purpose.
Wasting 500k on an expensive school that pretends you're going to be an engineer, then learning how to reverse a tree on a blackboard so you can be hired at Google or Amazon doesn't make you a code wizard. That's a very expensive code monkey.
> I think that many of them are as good as any AWS hire. And I think they could be motivated by something besides money to create something as good or better than anything from AWS.
If you think people's claims that they aren't motivated by money are anything other than lies they've learned they're expected to tell, perhaps you're just as ignorant as your $150K hires who have no idea that FAANG companies pay their developers $500K or more (which is the only reason you are able to hire them).
Maybe it sounds lame or hokey or whatever but the point of working for the government is not to make money it's to provide your services for a significant discount for the betterment of the population.
Is that how people approach it? Yes some do, and there are fantastic people who are working the government in service to the nation. Certainly not all and almost never for government contractors.
That's the spirit at least. I don't think it's actually working unfortunately because trust in government is so low. There was a time though...
People hate on "design by committee", but the real issue is you have a bunch orgs all trying make the design as close to their shit pile of code that is already written. (Outside of that, I think multiple designers with open minds is a good thing, but I shouldn't digress too much.)
(My coworker made an astute prediction (only semi-jokingly) that eventually all the standardized binary interfaces would eventually transitively refer to all the other standardized binary interfaces as the inevitable conclusion of that phenomenon, and few stingy large orgs properly separating their external interfaces from internals.)
The solution here is to pay a specific contractor to do the design and validate with a demo deployment. Oncely once it is proven to work, and the government can then set up their own in-house (i.e. proove it is real open source by doing the tech transfer) is the full money paid out.
This is the computer equivalent of a drug bounty, basically. It's high time publicly-funded engineering doesn't just deepen private intellectual-property motes.
Yeah but it's pretty convenient for the state security apparatus to have most important web services and corporate infrastructures hosted on the clouds of just two or three vendors with which they have close ties but little public scrutiny.
Custom ASCIs or OS is Cap Ex, and I am saying that part is allowed to be outsourced provided the IP is unencumbered. It's just the operations that must be done in-house.
To avoid some "idiots at the wheel situation", part of the tech transfer exercise will ensure that the deployment isn't too brittle. E.g. the government should be able to make some trivial Kernel modification and redeploy everything without issue. Maybe even also fab and install a slightly modified ASIC too.
> allowing any org to run data centers with a standardized interface. The state should then run their own data centers in-house with the stock design rather than outsourcing those operational costs
I was under the impression that not having to build and operate their own datacenters was the idea behind outsourcing cloud. Since commercial players are already doing it, the government could piggy-back on the economies of scale already being made. Plus, they can decide not to renew their contracts and not be stuck with a datacenter on the balance sheet.
Imagine if people actually though government services could be delivered properly if funded and managed correctly.
Like say, gov.uk, or a large number of public services in Australia (both state and federal). I've dealt with our equivalent of DMV and it was effective, efficient and there wasn't the stupidity of queueing that I've seen in US discussions about the DMV.
The US is one of the few advanced economies where there is a deliberate effort to screw with the government (eg defunding IRS auditing) and then complaints that government can't deliver services, followed by corrupt contracts to private enterprise that deliver worse results for higher costs.
Sure, most governments and bureaucracies aren't the most efficient, by their nature, they have rules and processes imposed on them. But that's more of a function of the "management" than of the services themselves.
It would make much more sense to let Microsoft, Amazon or Google to span a new data center and let the government wrap it entirely with their own monitoring and possibly also a specialized access solution.
That way they leverage all the hard work which these US companies have already made, all their expertise, and are able to keep an eye on all of it.
Microsoft and AWS (not sure about GCP) already provide data centers for the exclusive use of the US government and with specialized monitoring and access.
Microsoft, Amazon or Google are free to open up their IP as needed to fulfill the contract. And they don't even foot the operating cost beyond the initial demo.
Easy money! If they don't mind their mote being filled, that is...
> The state should procure an entirely open source / public domain cloud stack that would commeditize the "cloud" space allowing any org to run data centers with a standardized interface. The state should then run their own data centers in-house with the stock design rather than outsourcing those operational costs
Cloud space is already commoditized through AWS, Azure, GCP, Oracle and other smaller players right? Is this more of a NIH thing?
> This would be far better for society. Compute-for-all.
Can you expand on why/how would this be better for society than what we have available today?
> The state should procure an entirely open source / public domain cloud stack that would commeditize the "cloud" space allowing any org to run data centers with a standardized interface. The state should then run their own data centers in-house with the stock design rather than outsourcing those operational costs
> This would be far better for society. Compute-for-all.
Yeah, but that could also be the perfect being the enemy of the good. IIRC, there have been many, many massive failed military IT projects (e.g. they've spent decades trying and failing to replace legacy accounting systems). If they tried to build their own cloud, a very real possibility is that billions of dollars are spent creating an open source cloud platform that literally no one wants to use.
Also, my employer actually did implement their own private cloud using open source in our own datacenters, but that's been abandoned in favor of AWS and Azure. If the military builds it, it doesn't mean anyone else will come.
I wouldn't call more another massive privatized infrastructure "good" even if it does its job in the the short term. I would rather the government gain autonomy even if it means temporary adjustment pain.
> I wouldn't call more another massive privatized infrastructure "good" even if it does its job in the the short term.
Good in this case means meeting most or all of the project objectives. If they pursue a "perfect" solution, they may not do even that, let alone realize the other social goods you're hoping for.
I acknowledge your goalposts. I personally value my social good goalposts more important than the project objectives, and JEDI as originally envisioned is negative with respective to my goalposts. It is thus net negative to me.
Pithily paraphrased, "Those who would give up essential [self-sufficiency], to purchase a little temporary [efficiency], deserve neither".
> Pithily paraphrased, "Those who would give up essential [self-sufficiency], to purchase a little temporary [efficiency], deserve neither".
From the Pentagon's perspective, they're probably meeting that with either AWS or Azure. They probably look at self-sufficiency as a national, not organizational thing. So long as the thing is built and located in the US and owned and operated by Americans, it means they're not giving up any self-sufficiency.
Certainly. But until we reign in the military as a portion of government spending, big military investments should have direct civilian benefits not civilian costs. https://en.wikipedia.org/wiki/Military_Keynesianism (note the "Keynesianism" is somewhat tongue-in-cheek) does not cut it for the latter.
> The fact that this was kept up in litigation until the technology became archaic is really, really disturbing.
And also, in a way, really really good. It gave them time to know what they actually need and not waste money on something they (incorrectly) thought they needed.
This can also describe almost every software project in history. By my estimation any project delay will incur a redesign which is 50% due to new technology being developed and 50% due to different parties in the design discussion.
Even in Defense contracting land, If you delay a contract by 5 years then nobody from the original proposal will still be around. Had the contract originally gone through there would probably be incremental development to keep it up with the times.
Geesh, I left Microsoft in December, but back then it was all hands on deck, drop everything else, to meet this contract, and I doubt much has changed. I had a feeling it would be a bad end even if JEDI paid off, because so much else was being neglected. Now though....
Not really. If you look at the chart, there was a huge spike in trading the moment the news broke, which then dropped it .4% instantly. That's not normal.
FWIW the stock is now back at what it was before the news broke.
I wonder if this is meant to show a lack of impact on Microsoft's share price? Because I can't imagine any reporter at CNBC thinking that a drop to levels last seen on Friday lunchtime is notable.
I think it’s more likely just something in the style guide that all articles on events impacting public companies note the impact on share price. It’s not a comment on whether it’s news in one direction or the other , it’s just their standard.
This whole thing is a joke. I don't think any of us working as DoD developers give much of a crap what cloud provider you force us to use. But just pick something, anything. Quit switching the ground out from underneath us every couple years. The present state being the way it is, we're forced to be provider-agnostic to effectively everything, which means we need to stand up our own self-hosted services down to running our own Kubernetes engines and deploying our own NFS servers and database servers, and the only managed services we use at all are the ones to deploy the basic network infrastructure and VMs.
Linode can give us servers for probably 1/10th the cost of AWS or Azure if you're not going to actually let us use any managed services.
And people want to know why all defense projects are constantly behind schedule and over budget.
>I don't think any of us working as DoD developers give much of a crap what cloud provider you force us to use
The fundamental problem is that you aren't the one choosing or buying these services. The "people who give a crap" is some mix of political and industrial people with a very light sprinkling of high level program management which are driving the requirements.
So it has nothing - literally nothing - to do with what you as the actual person who is doing the work, want or need to be effective.
This is how the government acquisitions process is written into law unfortunately and I've pushed hard within the government to change that in the past (to some very limited success).
The purchaser / decision maker is not the user / practitioner dichotomy exists to a lesser extent in private commercial sector. It’s part of the problems of enterprise software in general
Yes. Unfortunately, I'm way too aware of this problem. As far as I'm concerned, it's the biggest problem facing government acquisitions, the fact that any provider's "customer" is always an acquisition office and not its actual users. It's even worse for IC enterprise services, because only your acquisition office has any say in what you prioritize for work and they're the only group of people in the government you're authorized to talk to, but then when you release something, any one of potentially hundreds of external stakeholders can all veto you from going into ops, but you can't solicit their feedback during development.
>the only managed services we use at all are the ones to deploy the basic network infrastructure and VMs.
I don't know... that kind of sounds like a good idea to me. Vendor lock-in is serious, and we don't need a 50-year DoD drip-feed going to whatever cloud vendor wins the contest today.
Because it could become a serious issue once the gov starts challenging the big co at something. You most certainly don't want a situation where the government depends on the company it is fighting.
Vendor lock-in makes switching if a better option comes along significantly more difficult.
At the same time, the near-guarantee of the vendor that has the lock means they have far fewer incentives to be responsive to changing needs by the client.
In addition, given the lock-in, whenever 3 or 5 years maintenance agreement contracts near their end, the Vendor has significant leverage to demand more $$$ for pretty much the same level of service.
The overall result is a lower quality product that can eventually put the buyer years behind the current best-of-breed capabilities, all at a significantly higher cost.
Vendor lock-in gives a lot of leverage to the vendor. Migrating away from vendor lock-in after years of development over their platform is a major project (been there, done that, twice).
Using standardised primitives, like the case with Kubernetes, leaves your infrastructure prepared to be migrated to a different cloud provider with much less friction and effort. You will still probably need some shims and/or abstractions for common patterns (caching, persistence, etc.) but if you can decouple completely your platform from the vendor's platform your organisation gets much more leverage, both for negotiation purposes as for regulations, in case there is a need to swap cloud providers your work is more-or-less cut out for you. When you didn't prepare for a cloud agnostic environment you will have years of migration to be done, for services, libraries and so on.
It's a major pain in the ass to be vendor locked-in and a huge cost, in time, opportunities and money.
I should mention that, while I said above I agree about trying to avoid vendor lock-in as much as possible, there is additional consideration when talking about government projects and especially classified projects. Even if your software and infrastruture setup is as vendor-agnostic as you can possibly be, you can only ever possibly migrate to another provider that actually has data centers certified to host classified data and connect to a classified network. Building that data center is at least as much of a bottleneck to a government agency trying to change cloud providers as the actual software migration is.
I don't actually disagree with that. My problem is in structuring these contract awards such that the only companies that can compete for them are those that offer a whole bunch of managed services, and then charge more for basic "turn on a VM" because of it, but then all we get to do is turn on a VM. If all we need is someone to host servers, award a contract to just host servers and be done with it. This is effectively what the NRO's AUE is. That was my wife's old program. But they're forcing everybody to leave that unless they can prove they need some kind of hardware Amazon doesn't provide and use the CIA's C2S instead, even if all you're ever going to use is VMs.
I think past vendor lock in wasn't so much about the technologies involved, but the inability of organizations to transform. If you have the right approach to application development/deployment/management there isn't a way any vendor can lock you in for any significant amount of time. I see more people wasting resources developing to avoid lock in when the applications they are developing won't live long enough to ever worry about having to move off of the systems they are developed on.
I'm not saying we should never be concerned about vendor lock in, but I think the concerns need to be balanced against the costs (and complexity!) of engineering everything to be immune from "vendor lock in".
Agreed. Kubernetes is eating the cloud and those big cloud providers should be scared. The future of these cloud platform providers is as a utility for virtualised hardware, storage and networking, firewall - and that’s about it. The software layer belongs in the kube.
Is it? Seems like it's eating the sunset of services that allow you to scale containers. But I don't see it eating any of the big services that cloud providers like aws care about. Like managed databases, queuing services (sqs), messaging services (ses), etc.
It is. Those can deployed quite easily on k8s these days. Persistence is not trivial, but can be done. Admittedly, at this point I wouldn’t put my databases in k8s, but keep an eye on this space as it’s becoming easier.
If your db is already replicated (cockroachdb, tidb, foundation,etc) running it in k8s is a breeze and can be considerably cheaper and more reliable than managed stuff
They don’t have to be scared, there are tons of corporations that can’t manage any kind of infrastructure effectively and are begging to get to the public cloud.
I anticipate many “we have to get off the cloud” projects in the future for sure, but I think it’s clear that the demand for “public cloud” has not been met by a long shot. There is much more room for the market cap of public cloud to grow.
It seems that the more money are at stake, the more conniving manipulation (propaganda, 'legal moves') will take place.
There has to be a 'behavior law' summarized by somebody, capturing this phenomena.
You are right,
Creating a 'technical' solution to this kind of problem -- is not going to work, unfortunately.
> I don't think any of us working as DoD developers give much of a crap what cloud provider you force us to use.
I thought AWS's Security & Compliance features[1] was the reason why it was selected before for such purposes. Do any of the low-cost alternatives offer similar security and compliance levels?
Honestly, I don't know. Raytheon stood up a very simple cloud platform for the RROC to run on based on Open Stack, and still provides the AUE, which is just bare metal and VMs and no managed services. But it's hard to say if they're cheaper because the charging model is so different. Application providers need to order their equipment, but get someone else to install it and run it for them.
267 comments
[ 2.4 ms ] story [ 278 ms ] threadhttps://www.zdnet.com/article/google-heres-why-were-pulling-...
Unlike AWS and Azure, Google doesn't offer any GovCloud regions, only Fedramp which seems to apply to unclassified data only.
AWS and Azure also have IL6 regions (able to hold secret information).
Finally AWS runs a top secret cloud for the intelligence community. I believe Azure announced a similar project but not when it’ll be open for business.
I've had demos from GCP sales reps, and the platform is shambolic. It doesn't even work during sales presentations.
Especially when you're Google (our customers aren't as smart as us) competing with Amazon.
Not sure where you’ve gotten this information from. GCP serves many “large organizations” including government agencies [0].
> I've had demos from GCP sales reps, and the platform is shambolic. It doesn't even work during sales presentations.
This feels very hyperbolic to me. Either way, demos don’t always work out the way you intend.
[0] https://cloud.google.com/solutions/government
Also, it's evident they cannot service those customers. Australia Post has a large contract with AWS. Whoops.
I've worked in many large organizations. I've received GCP pitches. I've participated in evaluations of GCP as a technology solution. It's substantially worse than Azure, plus according to their leaked massive losses on the product it'll be shuttered this decade.
>This feels very hyperbolic to me
Their dashboards are typical Angular heavyweight clunk, with confusing error messages, infinite loading spinners and things break constantly. It's been discussed here extensively. [1]
Their actual decent products (BigQuery) are hamstrung by the rest of the failing platform.
[1] https://news.ycombinator.com/item?id=25357409
As for anecdata, the only cloud platform I've heard nothing positive is Azure.
Except the most important part of Twitter -- timelines, of course. [1]
> Snapchat
Except the $1 billion contract from AWS, of course. [2]
[1] https://techcrunch.com/2020/12/15/twitter-taps-aws-for-its-l...
[2] https://fortune.com/2017/02/09/snap-inc-signs-big-aws-deal/
Multi cloud means cross cloud redundancy, but there's none of that here.
( Their offerings were so bad they were forced to sell their vSphere as a service arm to a low cost hosting provider. Even with the popularity of that dumpster fire in DCs and most companies moving away from DCs they still couldn't capture any market share)
I guess the DoD is no longer shy about its intentions for US' permanent state of war...
Next up: DoD's name change to DoW?
P.S., it's worth mentioning that it was the Manchurian who restored slavery in Qing Dynasty. The word Nucai in Chinese or 명사 in Korean, meaning Your Slave, was such an honorary title that for more than 300 years until early 20th century only those who were trusted by the royals could use. Yeah, don't wanna be a slave? Build a good army.
I suppose if you consider 1898 a few years ago.
[0] https://en.wikipedia.org/wiki/Newlands_Resolution
"Star Wars" itself was the name of a previous defense program, so, I'm sure this debate was had out decades ago.
1. https://en.wikipedia.org/wiki/Strategic_Defense_Initiative#C...
https://www.copyright.gov/circs/circ33.pdf
You're probably thinking about trademarks, which only protect the use of an identifier when used in the business of a particular trade.
I mentioned in another comment that George Lucas tried to sue over people calling the SDI "Star Wars" back in 1985. I can't find any evidence now, but I believe he once sued either the Air Force or NRO (can't remember which) over a mission patch with an x-wing on it.
Her last program held a voting pool to pick a name and ended up with "Totally Not a Laser Death Ray Facility" unfortunately coming in 2nd place.
*unless he is trying to launch rockets, which somehow he catastrophically sucks at
I admire the faith he demonstrates in his team, by being among the first human lives at risk in a launched Blue Origin vehicle.
But maybe 57 is just the new 27.
It is still not easy , but as difficult as say 20 years back.
To an extent NASA's Commercial Crew program and other commercial engagement vision worked in triggering commercial industry
I think it more that technology has changing a lot making it less expensive to attempt building a vehicle. Today, for example Terran is considering building a lot of their rocket with 3D printing, Rocket Labs does that already for their rocket parts. Talent availability with the right skills has become easier in the recent years probably helps.
But presumably Internet connectivity is the entire point of why they'd want a cloud contract: The government already has servers in buildings. It sounds like they particularly want the global reach of a cloud service, probably for activities intentionally intended to be on the Internet.
I mean, bear in mind, if you're a state-level actor that engages in cyberattacks, you need to be on the Internet to do them... Not saying that's the purpose of the JEDI contract, just that... there's plenty of scenarios where the military wants Internet-connected things.
On the other hand, let's be honest... In strategy for a modern war, Microsoft and Amazon's major datacenters are already on someone's priority target list.
Every nation-state actor knows where every datacenter is of every major digital player, because they've already implanted spies in those companies and the location maps of those datacenters are not internally secret. It'd be very hard to make them internally secret, since those DCs communicate with the rest of the network and must be controlled in realtime by site reliability engineers.
> The AWS Top Secret Region was launched three years ago as the first air-gapped commercial cloud and customers across the U.S. Intelligence Community have made it a resounding success.
AWS has provided air-gapped regions to the US Government for 6+ years, according to that blog post.
The DoD's job is defense, so it's about fighting wars, not about building infrastructure and supplies.
The security of the military apparatus will stay in the DoD. The implementation and operation of the security infrastructure will be delivered by government contract.
Everyone knows the Bezos-run WaPo is Pravda for the Democratic Party. It’s not a huge stretch to think Bezos is being repaid for his 2020 contributions. It really feels like both MS and Amazon should have been kept out of bidding for this out of CoI fears.
Hacker news is basically Gab for nazi programmers.
WaPo would only be pro-Democrats as long as it benefited Bezos, or if individual journalists leaned that way.
I really doubt there is a quid pro quo of a $10bn contract for what limited sway Bezos has over WaPo.
If anything, the fact that Oracle did not receive the original contract shows how limited political contributions were in affecting the outcome of this contract. Larry Ellison was a big financier of the last administration.
While I think it does have somewhat progressive slant, especially in more opinion pieces; you only have to go to our mainstream Canadian party to go WAAAAY further left than WaPo - and that's nothing to say of other continents, countries, parties and newspapers.
Similar to Pravda comparison.
That said, within the US political climate, WaPo is definitely left of center. Just not far-left.
Your point really breaks down when you would have no way to classify thoughts and media one or two or three steps more to The left of WaPo. Even if you add “radical left” for one spot further to the left. There’s nothing else after that.
Or you’re taking the stance of Fox News and co. who will dishonestly wonder if Biden is a socialist.
I am disputing the notion that in USA, Washington Post is "far left".
I can understand that for any given person any given media outlet may feel central, a little, or far to the left or right of their position.
But if we are to create any kind of external scale, with whatever measurement (relative based on how far axis go, or by some thresholds of specific criteria), I would doubt Washington Post fits on the "far left" of that scale.
(in for a penny, in for a pound addendum: I understand that in USA there's also an effort/inclination to label ANYthing "left" as extreme, far left, antifa, socialist, pinkie, commie, fake news, etc; terms that are implicitly meant to be derogatory; but that effort does not need to be engaged with in a rational discourse. To rephrase: Just because Glen Beck or Bill O'Reilly et cetera may call Washington Post "Far Left", doesn't make it so - in those cases it's a label meant to paint & incite and cater to an "in crowd", not be a part of meaningful engagement and honest wide-ranging discussion)
[Addendum to addendum: there was nothing in original post that said "Washington Post is Far Left in USA term "; it was a open claim posted to an international site, many of whom read or access Washington Post as a globally-relevant newspaper, so I don't feel assuming a global perspective is "whining". Or to put it in your preferred terms: Why do some USA people whine where perspective is brought in :-]
In a fit of rage, my center-right sibling once called me a socialist, as if that would hurt my feelings.
They are center-right editorially (in my opinion) and relatively neutral in terms of the two parties.
Both Democrats and Republicans bitch relatively equally about the "bias". Same applies to NYT and others.
"Left" publications would be ones such as Mother Jones.
"Right" would be National Review.
There are academic attempts to classify media on the left/right partisan measure such as https://guides.lib.umich.edu/c.php?g=637508&p=4462444
and even a "patented" bias measurement at:
https://www.allsides.com/media-bias/media-bias-ratings
which is vastly amusing to review outside of the US bubble.
Calling the Economist "left" is just plain wrong. They are the original "liberal" (as in regarding economics, trade etc) newspaper.
Believe it started with Trump. The Post is a beltway paper. It’s a chronicle for Washington insiders, and they never liked Trump. So the Post had a critical stance towards him, which in turn lead to him branding it as leftist. (Which is false. Its bias is best described as institutionalist and statist, which finds alignment across the political spectrum and parties.)
George Soros is 90 and nobody really remembers why he was so “bad”, Bezos is a great replacement.
All part of the hurr-durr-main-stream-media-is-left-wing-antifa-deep-state bullshit that has infected the US (and to a lesser extent, UK/CA/AU/NZ etc).
Disclosure: long MSFT.
edit: re: downvotes - the article states this in the sixth paragraph
https://www.nytimes.com/2021/04/28/technology/trump-jedi-pen...
Can you summarize/quote the relevant parts of the article? What evidence did the court find?
This was AMZN thinking that all they needed to include in their proposal was market share numbers. MSFT's proposal was more responsive to the DoD's needs.
A couple of points. The whitehouse blocked any disclosures of discussions / pressure even to the inspector general. They also issued notices to staff not to talk to IG. That's a pretty darn good sign that there WAS involvement by the whitehouse.
The DoD is dropping this case because they were asked to reveal some of this information. Instead of disclosing it, they elect to cancel the contract. This was a specific issue in the current case, would DoD have to reveal this type of communication from whitehouse.
Yes - people claim no pressure / no involvement - but it's blindingly obvious that there is a very good chance there was pressure.
I have past experience with govt contracting so have a sense for what the flows look like.
Contract selection put on hold, involvement of a political person (probably esper here). Ban on anyone talking about the potential discussion to internal auditors. Those are all the red flags.
What? You're trying to pass off absolutely zero evidence as proof.
The IG review specifically noted all the ways that Trump / Whitehouse blocked the answer to that question.
The Whitehouse then also got rid of IG Fine in April 2020 - I mean, this guy served 28 years in justice then through bush / clinton / obama terms as an IG.
Amazon also dug up stuff where Trump promised to GET involved.
No one "cleared" trump in this. He stonewalled as long as he could.
“I will be asking them to look at it very closely to see what’s going on because I have had very few things where there’s been such complaining,” Trump said. "Not only complaining from the media — or at least asking questions about it from the media — but complaining from different companies like Microsoft and Oracle and IBM. Great companies are complaining about it. So we’re going to take a look at it. We’ll take a very strong look at it. Thank you very much everybody.”
He head already shown at this point that he was an old boys club type of guy that would go after anyone who disagreed and support anyone who kissed his ass, so anyone having a hand in the selection of the winner of the contract that wanted favor with the president would obviously attempt to influence the outcome to not be Amazon.
He should have kept his mouth shut on such a massive pending contract, but it's pretty obvious by now that keeping his mouth shut isn't something he's physically capable of doing.
Now instead of having started on a crucial service that is needed by our military we're looking at likely further years of delays. All because the president was butthurt over some mean words in a newspaper.
That doesn't matter. We're not talking about criminal proceedings, where anyone has to be proven guilty beyond reasonable doubt. This is, at most, a civil law process. And it's the DoD canceling its own contract, which they are probably allowed to do for whatever reason.
Trump administration had asserted a “presidential communications privilege.” when asked if they leaned on the DoD to deselect Amazon.
Pentagon lawyers instructed Defense officials not to talk with the IG about any discussions they may have had with the White House about JEDI.
So basically - you know for SURE that there were communications about this.
"Mr. Johnson, I don't give a damn whether your plane ever gets built. But my forms will be signed and procedures will be followed."
He found the bullshit tiring because he felt that he didn't deserve to deal with it, because he trusted in his own competency and honesty.
You can see how all the pieces fall together. Somehow, the government has reached a default stance of not really trusting in the honesty (and maybe competence) of their contractors. And frankly, Ben Rich didn't really believe in his competitors either. He just didn't care because he only writes about projects that he's also competing in, so he thinks that his competence will carry him through (which it generally did... not for the F-16 though lol). What about all the competitions that doesn't have the strangely competence and honest vendor?
DoD saw what a gong-show single source could be (look at F-35). They figure multi-vendor, open architecture is the way to mitigate that risk.
Edit: DARPA -> Pentagon
This would be far better for society. Compute-for-all.
Also, the fact that government jobs are so capped like that is a conspiracy to make the public sector suck, basically. We should fix that too, but I recognize it cannot be done in time, hence allowing that the design work to be outsourced.
Or in another words, you're trying to find the one honest person, in the stack of AccgemloitteBMizantPMGataBCS proposals. And everyone else is bidding with the profit expectation of tossing entry-level, over-hyped developers on it, and then crossing their fingers nothing goes wrong.
hahaha
-----
I am trying to structure the contract to avoid this stuff, by making the deliverables include both the contractors running their own working deployment, and the government also running their own. For the latter, there should be a tech transfer period where the contracter needs to do training and promptly answer questions, and then a government independence period where the contractor is not allowed to intervene at all to prove the knowledge has actually been transferred, and government isn't entering an IT-support protection racket. Only then is full payout given.
Yes, that means finishing the entire project will take quite a bit longer than doing all the dev work. Yes, that also means the project is riskier for the contractor and has less ongoing reward, and so they will need to be paid more initially. But it's worth it in the end.
If there is some gotcha here where the private sector can get out with too much proprietary IP or a weak and dependent customer in the government, do let me know, that is not my intent. I'm trying to envision a transitional project to get out of today's privatized world to one with better US state capacity.
Windows still dominates most orgs, but never thought I would live to see any significant numbers - but here we are.
To your second, well, /shrug. A reasonable number of folk might well jump at the chance to have a job like that on their resume, and there's likely a fair amount of possible cross pollination w/ various folk in the DoD/military to bring up bodies and relevant expertise (i.e. security).
What security engineer worth his salt is going to accept half the pay to work for the government where things move at a snails pace?
Problems of red tape are not limited to only government ,it is function of most large organizations
Plenty of security engineers are constantly frustrated at the lack of security focus in corporate sector, investments are only made when it affects revenue . Even after all the ransomware companies some times do the bare minimum and buying insurance
Government on the other hand cares about security for different reasons, they take it lot more seriously. They don't sacrifice security in the interest of sales.
I would anyday prefer to work for some org where security actually matters not where it is inconvenience
https://www.nytimes.com/2020/12/17/us/politics/russia-cyber-...
Government doesn't care about the security of your personal information, hence why they treat their security engineers with contempt by offering them paltry wages and drowning them in red tape.
Most companies I know treat Security engineers similarly and pay not all that much higher. There is world of enterprise IT outside the silicon valley, the pay is not all the much higher. Despite all the recent threats and ransomware attacks security is not treated as it should be.
You still have to pass a marijuana drug test to work there, for heaven's sake.
I hate to break it to you, but that's normal in the vast majority of companies. So far, I've only lived in two states where it was legal, and in both the laws specifically allowed employers to test for pot. I'm in healthcare, and the company I work for tests everyone on entry, and people randomly after that.
If it's ever legalized at the federal level, expect employer testing to be specifically allowed. Transportation companies, surgeries, heavy industry, and a lot of other companies will filter for this, either because they or their insurance companies don't want the risk.
We're talking tech here—the Digital Service—and I've never worked at a company that drug tested their programmers. Not once, since the 90s, during the full-swing War on Drugs.
Anyway, the marijuana test isn't the only antiquated, sclerotic thing about the USDS (and Federal employment in general). But it's certainly a representative example.
If we were serious about fixing this problem, existing federal pay scale constraints would not be a barrier to entry.
Legislation could be passed yesterday that would add arbitrary pay scales for special purposes such as these. Attach an AWS architect salary to a government job and I think you will immediately find the skill gap filled. Bonus points if you put incentive structures in the employee contracts so that the brilliant minds are directly incentivized to deliver, rather than via proxy of their lobbying container organization.
From there, we can give the agency the budget to do the job, and let someone who's actually close to the situation decide whether it makes more sense to contract it out or build an in-person team. A bunch of legislators who are well-known to be perpetually too busy conducting fundraising lunches with lobbyists to actually read the text of the bills they're crafting and voting on are never going to be making informed decisions on matters at this level of detail.
That said, appropriations is no barrier to hiring the necessary talent. There are plenty of well-run private data centers in the federal government, staffed by both federal employees and government contractors.
Also, it's my understanding that the micro-management is more a problem on the state and local level, like NY state politicians such as Cuomo forcing the MTA to take on debt for these stupid stations rather than improve service.
Tough cookie!
If you can't do that, well, then you better consider secession or something. I don't want to live out my old age in a "New America, Yukon" military junta rump state because some people sit down, blushing, seeing Rubicons on all sides in 2021.
(The sad irony of course being that crossing Rubicons made military Rubicons then, I know. But it was a lack of meaningful other reforms too. So find a Gracchi metaphor instead, I guess.)
Or maybe add real progressive taxation that would get the $500k comps back into normal territory (by, among other things, taxing dividends, share options and the like accordingly).
Compensation is only one aspect of a job. Plenty of people work in defense while private sector pays more, public service motivation, job security, benefits, better work hours, more seniority driven promotions there are a ton of reasons.
Doesn't mean it is the most efficient way to run an org, point is there are enough qualified people who value the other benefits more than just the cash.
Funny you mention 170k Amazon is known to have a fairly strong cap on base salary compensation. Usually at 150-170k range.
I'll match you against somebody willing to take $150k for the exact same work. I'll match 4 of you against 8 of my hires. I'm willing to bet that my $1.2M team will outperform your $2M team. I'll very happily match 50 of your team against 75 of mine. You could poach 25 of mine who prove to be superstars, and I'll hire more.
That doesn't mean that you are dumb or incompetent or less skilled. I just think that there are some very smart and capable people willing to work for $150k. I think that many of them are as good as any AWS hire. And I think they could be motivated by something besides money to create something as good or better than anything from AWS. Indeed, less money could itself be a strong motivation to outperform.
Thankfully, we don't live in a world where there are only 10 geniuses to go around. The world is much much cooler than that!
The reason is not talent, but process. Governments are bureaucratic by nature, companies are meritocratic by nature. Obviously this is a vast simplification and there are exceptions but I wouldn’t bet on your team being one.
Although I've never worked in public sector so I can't make a cross comparison.
I'd take ONE top faang engineer over 8 government software contractors. There's so much tribal knowledge out there that people in the Midwest just don't have, not to mention a can-do attitude.
It blew my mind the first time I worked with a guy from Microsoft on something. We were having issues with some code, and he just popped open the kernel and started actively debugging things that I'd only vaguely even heard about. I feel like I was twice as good at engineering after that experience alone.
I had the same experience from the other direction later on.. a team of smart, hard-working coders had been dealing with stability/perf issues for years on some product. I was able to root cause and straighten all of them out in a couple of weeks, even though this was in an area of software development using a set of tools I'd never touched before.
I'll bet a single FAANG-hardened code wizard could outperform 20 or 30 government coders.
the hardest-core thing I've done was step my way through Windows startup into the container subsystem, on a real, physical target machine I had connected to my dev machine over FireWire. I felt like Indiana Jones. (It helped having the source code though!)
Don't be afraid to dig through heap dumps or decode assembly if you have to, although when you hit that point you're probably just going to have to swap out whatever component is misbehaving
I've done this at a company full of ex-Googlers. Some programmers are simply incompetent, no matter where they've worked.
Wasting 500k on an expensive school that pretends you're going to be an engineer, then learning how to reverse a tree on a blackboard so you can be hired at Google or Amazon doesn't make you a code wizard. That's a very expensive code monkey.
If you think people's claims that they aren't motivated by money are anything other than lies they've learned they're expected to tell, perhaps you're just as ignorant as your $150K hires who have no idea that FAANG companies pay their developers $500K or more (which is the only reason you are able to hire them).
Is that how people approach it? Yes some do, and there are fantastic people who are working the government in service to the nation. Certainly not all and almost never for government contractors.
That's the spirit at least. I don't think it's actually working unfortunately because trust in government is so low. There was a time though...
(My coworker made an astute prediction (only semi-jokingly) that eventually all the standardized binary interfaces would eventually transitively refer to all the other standardized binary interfaces as the inevitable conclusion of that phenomenon, and few stingy large orgs properly separating their external interfaces from internals.)
The solution here is to pay a specific contractor to do the design and validate with a demo deployment. Oncely once it is proven to work, and the government can then set up their own in-house (i.e. proove it is real open source by doing the tech transfer) is the full money paid out.
This is the computer equivalent of a drug bounty, basically. It's high time publicly-funded engineering doesn't just deepen private intellectual-property motes.
I guess its "cloud.gov" vs "cloud.mil".
https://cloud.gov/docs/technology/iaas/#the-infrastructure-u...
So, no I don't think it really fits the bill because underneath it's using a solution built by a private business.
It'd be like the government designing their own ASICs or making their own OS.
But the part about avoiding vendor lock-in would seem like an appropriate measure.
To avoid some "idiots at the wheel situation", part of the tech transfer exercise will ensure that the deployment isn't too brittle. E.g. the government should be able to make some trivial Kernel modification and redeploy everything without issue. Maybe even also fab and install a slightly modified ASIC too.
Our government is not capable of efficiently building and managing AWS style infra. Or designing new ASICS for consumer products.
Helping with R&D? Strategic planning? Sure. But not doing stuff like that at scale.
I was under the impression that not having to build and operate their own datacenters was the idea behind outsourcing cloud. Since commercial players are already doing it, the government could piggy-back on the economies of scale already being made. Plus, they can decide not to renew their contracts and not be stuck with a datacenter on the balance sheet.
Like say, gov.uk, or a large number of public services in Australia (both state and federal). I've dealt with our equivalent of DMV and it was effective, efficient and there wasn't the stupidity of queueing that I've seen in US discussions about the DMV.
The US is one of the few advanced economies where there is a deliberate effort to screw with the government (eg defunding IRS auditing) and then complaints that government can't deliver services, followed by corrupt contracts to private enterprise that deliver worse results for higher costs.
Sure, most governments and bureaucracies aren't the most efficient, by their nature, they have rules and processes imposed on them. But that's more of a function of the "management" than of the services themselves.
It would make much more sense to let Microsoft, Amazon or Google to span a new data center and let the government wrap it entirely with their own monitoring and possibly also a specialized access solution.
That way they leverage all the hard work which these US companies have already made, all their expertise, and are able to keep an eye on all of it.
Easy money! If they don't mind their mote being filled, that is...
Cloud space is already commoditized through AWS, Azure, GCP, Oracle and other smaller players right? Is this more of a NIH thing?
> This would be far better for society. Compute-for-all.
Can you expand on why/how would this be better for society than what we have available today?
> This would be far better for society. Compute-for-all.
Yeah, but that could also be the perfect being the enemy of the good. IIRC, there have been many, many massive failed military IT projects (e.g. they've spent decades trying and failing to replace legacy accounting systems). If they tried to build their own cloud, a very real possibility is that billions of dollars are spent creating an open source cloud platform that literally no one wants to use.
Also, my employer actually did implement their own private cloud using open source in our own datacenters, but that's been abandoned in favor of AWS and Azure. If the military builds it, it doesn't mean anyone else will come.
Good in this case means meeting most or all of the project objectives. If they pursue a "perfect" solution, they may not do even that, let alone realize the other social goods you're hoping for.
Pithily paraphrased, "Those who would give up essential [self-sufficiency], to purchase a little temporary [efficiency], deserve neither".
From the Pentagon's perspective, they're probably meeting that with either AWS or Azure. They probably look at self-sufficiency as a national, not organizational thing. So long as the thing is built and located in the US and owned and operated by Americans, it means they're not giving up any self-sufficiency.
They’re cancelling it because they don’t need it anymore. They need something like it, but the requirements have become archaic.
The fact that this was kept up in litigation until the technology became archaic is really, really disturbing.
And also, in a way, really really good. It gave them time to know what they actually need and not waste money on something they (incorrectly) thought they needed.
Even in Defense contracting land, If you delay a contract by 5 years then nobody from the original proposal will still be around. Had the contract originally gone through there would probably be incremental development to keep it up with the times.
> Shares of Microsoft were down about 0.4% following the news
So the MSFT share price changed well within the norms of daily volatility. It doesn't mean anything.
FWIW the stock is now back at what it was before the news broke.
Linode can give us servers for probably 1/10th the cost of AWS or Azure if you're not going to actually let us use any managed services.
And people want to know why all defense projects are constantly behind schedule and over budget.
The fundamental problem is that you aren't the one choosing or buying these services. The "people who give a crap" is some mix of political and industrial people with a very light sprinkling of high level program management which are driving the requirements.
So it has nothing - literally nothing - to do with what you as the actual person who is doing the work, want or need to be effective.
This is how the government acquisitions process is written into law unfortunately and I've pushed hard within the government to change that in the past (to some very limited success).
I don't know... that kind of sounds like a good idea to me. Vendor lock-in is serious, and we don't need a 50-year DoD drip-feed going to whatever cloud vendor wins the contest today.
At the same time, the near-guarantee of the vendor that has the lock means they have far fewer incentives to be responsive to changing needs by the client.
In addition, given the lock-in, whenever 3 or 5 years maintenance agreement contracts near their end, the Vendor has significant leverage to demand more $$$ for pretty much the same level of service.
The overall result is a lower quality product that can eventually put the buyer years behind the current best-of-breed capabilities, all at a significantly higher cost.
Using standardised primitives, like the case with Kubernetes, leaves your infrastructure prepared to be migrated to a different cloud provider with much less friction and effort. You will still probably need some shims and/or abstractions for common patterns (caching, persistence, etc.) but if you can decouple completely your platform from the vendor's platform your organisation gets much more leverage, both for negotiation purposes as for regulations, in case there is a need to swap cloud providers your work is more-or-less cut out for you. When you didn't prepare for a cloud agnostic environment you will have years of migration to be done, for services, libraries and so on.
It's a major pain in the ass to be vendor locked-in and a huge cost, in time, opportunities and money.
An example of where I'm coming from: https://www.outsystems.com/blog/posts/vendor-lock-in/
I think past vendor lock in wasn't so much about the technologies involved, but the inability of organizations to transform. If you have the right approach to application development/deployment/management there isn't a way any vendor can lock you in for any significant amount of time. I see more people wasting resources developing to avoid lock in when the applications they are developing won't live long enough to ever worry about having to move off of the systems they are developed on.
I'm not saying we should never be concerned about vendor lock in, but I think the concerns need to be balanced against the costs (and complexity!) of engineering everything to be immune from "vendor lock in".
You are right, Creating a 'technical' solution to this kind of problem -- is not going to work, unfortunately.
I thought AWS's Security & Compliance features[1] was the reason why it was selected before for such purposes. Do any of the low-cost alternatives offer similar security and compliance levels?
[1] https://docs.aws.amazon.com/whitepapers/latest/aws-overview/...
In a DoD facility? pretty hard
Do they not have computers in DoD facilities? Do they never run RHEL? Don't the installer disks have QEMU on them already?