It appears they’ve had access since at least January. I wonder how many customer accounts have been breached? It’s a giant list of targets if so.
It’s also very disappointing this is surfacing via tweets and not a breach notice from Okta, who apparently suspended at least one account used for entry.
Ugh. This is bad if true. Godspeed to incident response, enterprise security, and infrastructure teams. If you're not an Okta customer, you might want to tabletop this anyway.
Tomorrow is our "Armageddon Tabletop Exercise", which is bringing up our business from a complete datacenter failure. I don't envy the blue teams doing it for real.
Not OP, but at a previous company we were given access to a new AWS environment with nothing in it and were timed on how quickly we could get all of our services operational. Fleet teams went first, then networking, then T1 services (identity, platform, etc) and so on
So roughly how long did it take? I participated in table exercises of that sort and couldn’t convincingly get less than two weeks. And not two fun weeks.
I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ticketing system in place to process external client transactions? Probably not, but it'll need it eventually (so maybe that moves to 2nd tier restore process?). My company's RTO is 24hrs to processing new client transactions. Restoring old ones will definitely take longer, but at least new ones can proceed.
If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic.
You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage (https://www.youtube.com/watch?v=30jNsCVLpAE). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.
>"Dead tired people are more likely to make brain dead decisions"
Indeed. In reading the post-mortem on the recent multi-day Roblox outage, it's hard not to imagine some bad decisions were made that only made problems worse and that likely these were on account of people just being fried by lack of sleep:
The PM in charge of it spent probably 6 months to a year planning it with every team prior to our first gameday. The first gameday took 2-5 days to complete entirely, but we made it a bi-annual exercise and eventually got it down to ~12 hours before I left the company
Company was in the middle of an on-prem -> cloud transition. They were responsible for the on-prem hardware as well as provisioning VMs across cloud instances
Sure. We just moved our build configuration documentation out of our on-site Confluence to plaintext markdown so we can have them in the event of a full network breach. Tomorrow's exercise is to discuss how to leverage those docs for an exhaustive rebuild of the company under the assumption that a full data breach has occurred and we can't trust the existing servers/ backup system.
Like I mentioned, it's sort of an "Armageddon", or worst case scenario. It's meant to help us identify issues with our docs, assist with order of operations, and it exposes any new hires to the full scope of all the moving pieces.
We already perform DR exercises, including testing our online backup restores, between production and DR sites, but this tabletop takes it further.
Wow, that's great to hear. I wish more companies would do this. I'm curious if you also made similar provisions for Git as did the Confluence stuff? I would love to see a writeups or blog post on these types of table top exercises as I feel most companies just run on IaaS/PaaS/SaaS and hope for the best.
I imagine the "run it on your own infra" competitors are excited and warming up their cold-call lists. I'll be curious to look at some stocks in the morning.
Honestly: fuck corporate security and leadership who kept pushing for spying on their workers with increasingly intrusive tools. I can’t remember how many times I argued with IT that MITMing SSL was not ok (there are some tools.. crowdstrike maybe? That do this). They made their bed, now they gotta sleep in it.
It's always interesting to hear people think they have rights on corporate networks.
It is a complex issue with entry of nuance I'm too tired to thumb type. I know there are lines that shouldn't be crossed. But it is so naive to think companies shouldn't have an eye on their data.
Do you think banks shouldn't have cameras in their lobbies? Or that tellers can walk in and out of the vault with black bags and no one ask what's inside?
I don't think the OP's point was about rights as an individual on a corporate network, more about trying to backdoor your staff's equipment and the consequences of that.
To use your camera analogy, you better have a good lock on the security room holding the tapes.
MITMing SSL is also Zscaler's MO, and guess what my job is preparing to mandate.
Organizational endpoint security seems to be much more about incident response than incident prevention. That is, it's more focused on providing a nice audit trail that can be used to find and fire/arrest the perp once a breach occurs than it is at preventing breaches from happening. It probably fills a due-diligence tickbox somewhere, allowing the company to say "if a breach occurs (the probability of which is always nonzero) then we have measures in place to find out how, where, and by whom and mitigate the damage" to shareholders and important customers.
Yes, putting every device in Windows AD and installing remote management tools that run at the system level is a recipe for total compromise.
Software diversity is good. Remotely controlled mono-cultures are bad. IT management and security compliance people need to understand this.
I liken the mindset of (only we will use the remote control software to do good things) to Encryption back doors (only law enforcement will use them to catch criminals). Computer scientists call these 'Exceptional Access Systems' and it has been shown (many times) that it is impossible to ensure they will not be abused and used against you.
Keys Under Doormats: Mandating insecurity by requiring government access to all data and communications:
It's not really about alternatives, what would the spyware stop that a simple Daemon that version checks your device won't?
If you're working from home and your device gets owned, and the attacker just sits there slowly slurping data as you go about your business... Chances are you're not going to detect it, but if you did you'd probably notice it's running when it's not supposed to and it's sending network traffic when not actively used.
If someone uses a laptop breach to go look at data you wouldn't normally look at ( or tons of records ) an audit log would give that away.
So really to do security well you have to profile the role. Why are you looking up customer record abc123? Trying to see what your girlfriends doing, or is it because they've called the customer support line?
You basically have to define what does misuse look like and protect for that.
Back on the slow slurp, hardend immutable devices that are regularly updated with limited Auth sessions would likely contain the breach.
Same deal on servers, you harder to stop it, but you monitor for when it happens.
Do people really monitor though? Some, sure, but your average org buys a tool the fails configure it with any sort of context.
In security engineering there's this subculture of individuals that just implicitly trust every vendor that offers a solution because that means they don't have to do their own research on the issue. And if anything goes wrong, just blame the vendor!
I have talked to security engineers that have had the company procure millions of dollars worth of security tools, and yet they don't even know the basics of security.
For God's sake, I had to explain what end-to-end-encryption was to a team of "senior" security engineers the other day. They genuinely had no concept of it. As a security SWE I am so done with these sort of people. Fuck them! Why are they even in security!? Real private data is at risk because of their incompetency!
Many of them the extent of their knowledge is: push a button and run a scan and tell an App or Infra Team: here are your vulnerabilities now fix them. Ask them to dive in a little deeper with you on the vulnerability (Eg past the entry in the spreadsheet they gave you) and they often don’t have the technical ability to do so..
It is rather appalling. I recently listened to a podcast about a police station in a small city that was compromised and all their data ransomed. They had a contract company that handled all their cyber security needs. The "security" company had an RDP port open to the world to an admin account on the server that ran everything for the police station and city. In order to protect the employees webmail, they required that you login from an internal IP and so they gave this admin remote desktop account credentials to employees that needed to check their email remotely. They could login to the server with the admin account and then use the browser on the server to log in to their webmail. The same set of root admin credentials was given to every employee. It's just insanity. And I'm sure that government contractor was hella expensive.
From friends and former colleagues I know at least quite a few companies using tools like Tanium [0]. The Tanium client on the employees' machines (and on all corporate servers, containers, VMs) is still based on python 2.7.
It would be interesting to know Tanium takes to backport security fixes. I mean py2.7 was sunset 2020-01-01 [1].
I have built solutions that are a perfect fit to solve security issues at multiple companies, only to have them turned down because “management says we have to buy a vendor for this”. Incredibly frustrating, especially when they go with a vendor that is in all ways worse than the solution I was providing.
There's a mix of "The Peter Principle", lack of experienced engineers, too expensive to acquire engineers, at play in the IT world. But as far as I recall that was always the case.
> I have talked to security engineers that have had the company procure millions of dollars worth of security tools, and yet they don't even know the basics of security.
When I was in college the whole student network went down. Apparently there was a hack from the student network to the campus police network, so campus IT pulled the plug on any of us having network access. Because those two weren't separated because why would they be?
So I hoofed it to the IT department and I asked to talk to someone who knows a thing or two about networking. The director of IT came out. He sat me down and started telling me about the $300,000 budget he just got approval on. I was thinking, you can't take some of that and stand up a router with packet filtering to isolate the student network from the police network in two different subnets? A FreeBSD box might do in a pinch. (It was the early 2000s and network traffic loads were... different.)
I learned a lesson that day about seniority and the Peter Principle. Climb high enough and your mindset changes, from technical solutions to issues of money and resource allocation. From there, "let's just pay $VENDOR to do it, they'll solve the problem for us" is but a small step.
I worked at a company where the CIO believed everything vendors told him, and nothing any of his employees told him. He also got lots of kickbacks from said vendors and was eventually perp-walked out of the company.
Same company had a head of security who was full-time employed by two different companies at the same time (he had unlimited vacation time for both) without either knowing the other existed.
Best thing was that the company did investment/insurance services and had a banking division.
Bill Demirkapi posted an update later that this earlier assumption was perhaps incorrect [0]. It seems an internet-routable RDP server at rdp.chrisro.fun ("chrisro" is apparently named after the specific Okta employee/contractor whose name is visible in screenshots) is a more likely vector for compromise.
And this explains how the LAPSUS hacker group was able to breach Microsoft, NVIDIA, and countless others. Apparently, they have been inside of OKTA systems for nearly 2 months now.
That seems like an attack vector unrelated to the Okta breach -- if it was possible to attack the MS employee's computer from a compromised app, then whether or not that app was protected by Okta seems immaterial.
Perhaps the app was not compromised, to begin with. Hackers might’ve gotten into the build infrastructure of the app, rewrote it to make it compromised, and then all the users of that app are now compromised.
Lesson: ask your (in this case Microsoft) employees to not use work computers for any personal use.
/me goes to check if my password manager company uses Okta.
Again, this is not related to an Okta hack, employees could be using an app that has nothing to do with Okta and be compromised through this same attack vector.
This is just wild speculation to make up a scenario where Microsoft could be affected by the Okta hack with no evidence to back it up.
That's generally true but the thing about Microsoft is that it's so huge I wouldn't be surprised at them being customers of a lot of SaaS offerings. Could be via an acquired company that is partially integrated? Or some kind of federated login for partners.
This is a reasonable take in my view. When you acquire a company, you don't get to replace their entire tech stack on day 1.
And sometimes there are chains of dependencies that need to be swapped out in sequence.
So from acquisitions alone it would seem reasonable to expect that Microsoft at any point in time has at least a handful of usages of certain SaaS offerings, even if MS themselves offer their own versions of those services.
That said, the surface area of these "unofficial" 3rd party systems is likely to be quite small in comparison to whatever their official tech stack looks like (which does -- I would assume -- centre around Azure AD). I'm not expressing an opinion on whether the GP is correct to make this link...
I've worked with some very large companies where internally they (surprisingly to me) make use of competitors' products even when that company makes their own version of that product. There are all sorts of legitimate reasons why this happens.
It would be confusing to me if Microsoft were utilising Okta themselves. Their own Azure offerings provide everything Okta does (and usually bundled into something you're already buying).
Over the weekend, LAPSUS$ posted a screenshot to its Telegram channel of what appeared to be information taken from an internal developer account for Azure, its cloud computing division.
Acquisitions get authentication replaced fast (well, fast-ish, we got a year to change over). It might be around somewhere, but, AAD is used nearly universally and I've never seen Okta.
The well hidden secret among large tech companies is that persistent threats are quite often discovered after ~6 months active presence going unnoticed. Some persist for years.
Yes, this is very common especially when you're living off the land. You take over the one unmonitored host and then you use the internal tools as if you were a normal employee.
The screenshots are very worrisome. In the pictures below, LAPSUS$ appears to have gotten access to the @Cloudflare tenant with the ability to reset employee passwords
It's not particularly worrisome as shown in the screenshot. It appears the attacker could only send a password reset link or temporary password to the e-mail address of the user account. There is nothing in the screenshot indicating that the attacker could have changed the e-mail address to one controlled by the attacker, thus being able to reset the user's account.
With all those recent news of compromises done by Lapsus$, i really wonder what they're doing "differently" to the rest - do they have an especially large arsenal of exploits or is everything compromised anyways and one just doesn't read about it in the public because no one knows...
One theory is that they are based in South America, and have working knowledge of all the outsourced support that's gone to Brazil, Chile, Argentina, Costa Rica, etc. So they are well positioned for social engineering those workers, how the support systems generally work, etc.
Maybe a stupid question but it seems very foolish to target fedramp companies? I mean why would this not attract the attention of some very effective 3 letter agencies? Granted it’s not technically supply chain, but the recent fed retaliation to darkside, etc. comes to mind.I just feel it’s phenomenally hubristic to brag on social media about penetrating fed ramp certified federal contractors…they are highly likely to be interacting with top secret level info.
If you are operating from a jurisdiction that's already under heavy sanctions or severed diplomatic relations and extradition is unlikely, the laws of another country don't really matter.
I understand that, but I was thinking more in terms of retaliation. The sword cuts both ways. Haven’t the feds been bringing down the hammer on these sorts of groups lately?(by that I mean attacking back)
Yes but equally powerful actors (relatively speaking) are interested in such resources and attacks, and can ensure the same high level of comfort for the attackers, so it's probably a wash.
When they always have a safe haven to run to and lay low in a comfortable way without significant losses (that aren't just cost of business), what good is such a slap?
In my mind I see it like the same as throwing a beer bottle at a robbers car as they drive away. Even if you hit it, at best it's some scratches or maybe a busted tail light, but they're still gone with your goods.
I kind of agree with the parent. Putting top secret info into AWS feels... insane. When I worked for a gov't contractor the top secret info was kept in a faraday caged room.
AWS has had a Top Secret-certified region (AWS Top-Secret-East) since 2014. IIRC, it's physically located on-site at the NSA. They launched a Secret-certified region in 2017, and just launched a second TS region on the West Coast in Dec. 2021.
Sure, but what I was referring to was the fact the attackers claim to specifically be attacking okta customers themselves. I’m saying there is a high overlap between fedramp and companies which somewhere in the organization there will be access to highly sensitive info. Not saying they are in possession of, but have access to for many contracts.
I don't have a twitter account, but I can read everything just fine. Sometime twitter is poping a "login to continue" crap, but by using zapper from uBlock Origin I just shed that crap away and I can continue scrolling.
That seems to depend on how often you click such links, or something else I haven't thought of. I used to have your experience. Now I just get a person's profile pic and a logon prompt. If I uBlock zap that, there's nothing accessible behind it.
Same. My theory is ... well I don't have a good theory on this. I think maybe links to a specific thread are not blocked. Links to a person (to just browse) get blocked.
I should've mention that I also have NoScript and I am using Firefox, if that may of help to you. NoScript has google tag manager, google adsense and a few other stuff related to both google and amazon blocked always.
As for how often I open twitter links, is as often as they make it to HN front page. So something along of one twitter link around every 2 days.
Apart from a guarantee of no such annoying popups, Nitter also always loads significantly faster than Twitter for me; and I much prefer its visual design for actually reading things, makes it much easier to focus on the content.
“Authentication services company Okta Inc said on Tuesday it is investigating a report of a digital breach after hackers posted screenshots of what they claimed were its internal company environment.”
They are simply looking into it. Not offering unwarranted speculation or conjecture. Not providing any information that could assist attackers. And not being unreasonably dismissive, nor downplaying it.
I really don't see the problem with their response. What would you propose in the circumstances?
It's just one of those responses that's not actually a response. It's the only response you can give, and you only give it because you're forced to give a response. It's a noop.
Then I wonder why it is remarkable? The absence of a response may be worth commenting on, but this seems fairly unimpeachable.
It's distinguishable from a noop because some information is imparted, namely that a) they are aware of the issue and b) that they have formed a preliminary view it warrants a response.
If, for example, the screenshots didn't actually look like their internal admin interface (and so were obviously doctored), they would probably say something to that effect in the initial post. The fact that they're merely "looking into it" implies the information they have so far makes them think this at least could be a real hack.
WASHINGTON (Reuters) - Authentication services company Okta Inc said on Tuesday it is investigating a report of a digital breach after hackers posted screenshots of what they claimed were its internal company environment.
A hack at Okta could have potential major consequences because thousands of other companies rely on the San Francisco-based firm to manage access to their own networks and applications.
The screenshots were posted by a group of ransom-seeking hackers known as LAPSUS$ on their Telegram channel late on Monday.
It's a developing story, click back to the article and there are more details now, including the name of the Okta official that made the statement:
WASHINGTON (Reuters) -Authentication services provider Okta Inc is investigating a report of a digital breach, the company said on Tuesday, after hackers posted screenshots showing what they claimed was its internal company environment.
A hack at Okta could have major consequences because thousands of other companies rely on the San Francisco-based firm to manage access to their own networks and applications.
The company was aware of the reports and was investigating, Okta official Chris Hollis said in a brief statement.
"We will provide updates as more information becomes available," he added.
The screenshots were posted by a group of ransom-seeking hackers known as LAPSUS$ on their Telegram channel late on Monday. In an accompanying message, the group said its focus was "ONLY on Okta customers."
Security experts told Reuters the screenshots appeared to be authentic.
"I definitely do believe it is credible," said independent security researcher Bill Demirkapi, citing pictures of what appeared to be Okta's internal tickets and its in-house chat on the Slack messaging app.
Dan Tentler, the founder of cybersecurity consultancy Phobos Group, said he too believed the breach was real and urged Okta customers to be "very vigilant right now."
In an email, Tentler added, "There are timestamps and dates visible in the screenshots indicating January 21st of this year, which suggests they may have had access for two months."
Can confirm. I had just tucked into bed when this tweet got shared with me. Now, in the "is it real or not" stage I'm refreshing constantly, trying to decide if I should go back to bed because folks know how to page me or if it makes more sense to just stay up. If I'm going to get paged in an hour anyway, might as well know what's going on.
Godspeed all the corporate IT and security departments who are about to have a sleepless night. Surprisingly haven't heard a peep from Okta over an hour later. Wonder who is sweating more, their engineers or lawyers..
naïve question: Is there anything regular consumers can/should do in light of this information? Changing passwords etc seems futile if hackers have internal developer access anyway
is there an app that can send bulk gdpr data deletion requests for no-longer-used services?
seems like it'd be possible write an app that scans email history to detect services that have your data, scrape their websites to determine support email address(es), and send a GDPR data deletion request template to each of the selected ones.
I doubt too many people are using Okta as a regular consumer. If your work stuff uses it, the general advice has always been to keep it aggressively partitioned from your personal life. Hacked or not, your IT admin has always had access to everything, and I don't know if I'd trust them with my data any more than Lapsus$.
I didn't see anywhere,for those who have Telegram and want to see the claim, it's here: https://t.me/minsaudebr (Unfortunately web preview without Telegram seems to be not working right now)
And this is why I would ultimately never trust a centralised company with our authentication infrastructure: because something like Okta is an infinitely more attractive target than we are. Their offering is sweet, and I’m always tempted to just give in, but this confirms me in my decision.
+1 to this. Why outsource the most business-critical parts like auth and email to a third party, knowing that they're the biggest target? I know that the most common argument is "they know $service more than we do", but don't really have a counter argument against it.
No, but there are both open source libraries, and complete authorisation servers. If you trust Okta, you can trust these, and embed them in your infrastructure. Auth only gets super hard if you try to be smart and deviate from best practices.
There's also an abundance of libraries for different programming languages to implement authentication or authorisation, lots of them battle-tested by thousands of services. It's not like providing secure services is impossible without Okta.
Could you expand a little? Of course a HN comment is not giving you full instructions on implementing custom authentication infrastructure, but that doesn't mean it's impossible or somehow "wrong" to do so, if you rely on established libraries and patterns.
OAuth2 is a really complicated standard and is complicated to get correct. I mentally compare it with SIP which also has several open-source implementations but still often ends up getting outsourced to others due to the sheer complexity of the thing. Though in OAuth2's case I'm willing to say it's complicated because security is complicated. With SIP I really don't know why lol.
Doing user IP address change detections as part of a heuristic vector for account compromise is sometimes a best practice and sometimes a great way to generate a bunch of useless noise, but I know which is less likely to break UX by integrating such a feature.
I agree with you on not outsourcing authentication etc to a big provider like Okta, but OAuth2/SSO can get ridiculously annoying in large organization/s. If it's one app that you're dealing with, no worries. If it's multiple operating systems, multiple legacy and new apps etc, its a headache. Not that Okta particularly helps with that either though, it's a problem full stop.
Why would I ever need to support arbitrary Oauth2 providers? As long as they're following the standards, a generic Oauth2 client should suffice. If they're misaligned with the standards I wouldn't be using their service.
I can tell you never have, because there's no such thing as a generic OAuth2 implementation. They're all slightly different and incompatible. SAML is what you really want in terms of making things works more generically, but even that is generally a nightmare as you still need mappings.
Because in both auth and email one can make a small screw-up that allows for a company threatening breach. As someone who has performed penetration tests for all of: companies that roll their own SSO, companies that use a third party SSO service, companies that provide third party SSO services, the failure modes for contracting out to a third party are a bit easier to manage if you are a small company, and chances are if your SSO provider is breached they are going to care about bigger fish than you.
For medium to large enterprises, the calculus is harder and there's going to be months of flame war trying to hash that out.
Its expensive and keeping the expertise around and maintaining it was seen as not worth the cost.
The problem is motivation to break it trends up over time as the service gets more and more popular.
In some ways, if Okta can fall basically anyone can. Sometimes there's an advantage in being smaller. One would hope that the actual attack surface on Okta was smaller than the post seems to imply. That there is even any way to get full access to all customers would seem to be a critical failure.
> In some ways, if Okta can fall basically anyone can
This might be true, but for someone to "fall" they must be targeted to begin with.
If "LAPSUS$" is after big companies with significant IP which they can hold ransom for millions, as a small business you'll be safe because they won't even bother attacking you, and may not be willing to put the same amount of effort into breaching your small company's SSO than they would for Okta.
So hire someone who does, or consult an outside company to do the work for you. You can get the expertise without necessarily having it in-house. There are agencies both big and small that can help you, including training your existing staff.
Yes, it's going to cost more upfront, but there are a wide range of options between "rely on a blackbox PaaS/SaaS" and "write and deploy everything yourself".
I meant specifically to do an integration for you on your infrastructure, not to use a centralized PaaS/SaaS like Okta. I'm not in a position to say that Okta never does this for anyone, like key customers, but it is not part of their standard offering.
But you said you want a company with expertise, who has more expertise at building and running an SSO service than a company that's selling that service?
How do you have an SSO system without a centralized service? Whether you buy it as an outsourced SaaS service or run it yourself, it's still SaaS, but now you're on the hook for managing it and keeping it secure.
The parallel to conventional software infrastructure would be: Hire staff, or contract local expertise from reputable businesses, to build and maintain your software (probably using existing technologies rather than writing everything from scratch) on bare-metal or colo, rather than reaching for fully managed PaaS for your entire stack.
Or for comms, run a team- or company-wide chat server rather than relying on Slack/Teams/Discord.
Either you're missing my point (and maybe I didn't express myself super clearly), or you're pitching a false dichotomy.
> who has more expertise at building and running an SSO service than a company that's selling that service
OP illustrates well the consequence of that line of reasoning. I'm certain this is just the beginning as more too-big-to-fail vendors get compromised.
The cloud is just somebody elses' computer, "Move fast and break things" mean things get broken, etc.
Whenever we get subscription services for "smart security" involving cloud-connected cameras, AI-controlled locks and drones, would you also consider that as the one-size-fits all over having security guards or contracting companies like Securitas or your local equivalent?
> now you're on the hook for managing it and keeping it secure
> who has more expertise at building and running an SSO service than a company that's selling that service?
If Okta gets away with this (and the Equifax breach and subsequent outcome suggests they will) it proves that the market doesn't actually care about security, thus a company selling that service has little incentive to actually invest in security.
Can't you say the same thing about a lot of stuff? Why outsource hosting to a third party like AWS, knowing they're the biggest target? I trust AWS a lot more than myself when it comes to securing infrastructure.
I wouldn't throw out the baby with the bathwater. I don't know how publicly it's acknowledged, but most people I know in security have always had a poor view of Okta's security.
On the other hand, I feel fine using GSuite for SSO because we have a much better view of their security.
That said, it sucks that everyone is so fucking bad at security. I maintain that it is not that hard.
> Every single day there’s a zero day practically released
Yeah, people need to stop using memory unsafe languages. They choose not to.
> A single slip up will lead to you being compromised.
Not if you add multiple layers of security. Like sandboxing. Or mTLS. It's not hard to do that.
edit: Let me clarify. Security isn't hard generally, but it's hard individually, because you're drowning under everyone else making it artificially 10000x harder.
Log4j was memory safe. Tho I agree on mTLS and even would like most valuable networks to be connected only via an allow list of safe-ish destinations. It would make things a lot harder, and the logs of denied hosts would also be a nice warning.
Actually, the fact that memory safety bugs are more difficult to exploit seems to have increased the rate of vulnerabilities discovered and exploited, and the fact that these are now often higher-level bugs (think insecure feature design bugs rather than low level implementation bugs) - means that once something is discovered, it can often be exploited in a way that is either much more pervasive or far harder to detect.
So no - safer languages won't stop security from being an issue. Secure design, implementation, configuration, and frequent red-teaming exercises are the only way to reduce your risk, and even then - expect to reduce the rate by some %, but never reach zero.
> Yeah, people need to stop using memory unsafe languages. They choose not to.
Golang and Rust are not magic bullets that makes systems automatically secure.
Flaws in application logic have little to do with language choice.
Also consider the effort and money it takes to rewrite a multi million LOC system with several dependent apps. The new trendy languages introduce breaking changes, switch paradigms, and have less mature ecosystems.
> but most people I know in security have always had a poor view of Okta's security.
What are their usual criticisms? Genuinely curious. I've always hated their UI, their docs, and lack of real support but felt that at least they had security going for them. The last assumption might have been proven wrong tonight.
Also, what would be a reasonable alternative to Okta now that their nearest competitor, Auth0, was acquired by them. Is GSuite a reasonable alternative for SSO with multiple different providers and supporting SAML, etc.? Thanks in advance.
A problem in infosec is information dissemination. Lots of their criticisms are based on backchannel information shared over drinks. It makes it difficult to discuss publicly.
I wouldn't want to make a recommendation, but I'll say that my company uses GSuite.
A typical opsec issue in many companies even to this day is doing ssh shell access to critical systems and doing so with ldap passwords or user-controlled authorised keys and without 2FA and without host key verification. There are trivial browser based local network attacks to gain reverse shell access with such setups. Just because those systems are behind firewall and VPN, people tend to falsely believe they are more secure.
> or user-controlled authorised keys and without 2FA and without host key verification. There are trivial browser based local network attacks to gain reverse shell access with such setups.
I'm struggling to see how that attack works without a copy of the private key.
Any pointers for what to search for to understand this better?
The key thing here is _user controlled_ private keys: this means that those can leak, and if they leak, you're in trouble.
A better solution is to use ephemeral SSH keys generated using an SSH CA. This kind of thing can be implemented with Hashicorp Vault, though I'm sure there are plenty of other solutions out there.
It simplifies key checking on servers too, as they just need to the details of the signer and there's no need to juggle keys in LDAP.
You'll still need a way to deliver those ephemeral ssh keys to a user controlled location, or make them indirectly usable by users (which amounts to the same thing).
Aurora was the major incident. Google has invested heavily in a number of areas with regards to security, such as BeyondCorp, but I haven't talked to anyone from Google sec in a few years so I don't know how things have changed.
It's simple: they pay absurd amounts of money for top talent and let them work. Look: https://www.levels.fyi/company/Google/salaries/Software-Engi... can your company afford to pay 2-300K cash -- not to mention serious stock -- to bread and butter mid level engineers?
Yeah that's a fair point. As much as I badmouth Google in other areas like product longevity, I don't usually laugh at their security related offerings and initiatives.
I lost faith in Okta when we implemented OAG. They don't have an AWS marketplace version so if you want to run it there you have to manually convert their ESX appliance. Once it's up and running you can't login and get a shell. (Well, you can using Systems Manager on AWS...) and installing any agents on the box means support won't help you with anything.
Then their SNMP mib doesn't work properly. So you have a a box that is proxying some of your most critical systems that are so old that to integrate them with MFA requires OAG (think mainframes, old ERP systems etc) and you have to take Okta's word for it that the server is secure and not hacked.
They do thankfully support Syslog for logs, but again you have to take their word for it that you're getting all the logs, because you can't access the system to verify.
Having said all that. OAG solves a very real business problem and it is hard to find a competitor in the market with the level of integration it has to legacy platforms.
A poster further up mentioned something that I think is often ignored.
Security isn't just about overflows and injections, outsider enemies vs insider allies. Any human with privileged access that can be compromised will eventually be compromised as the perceived value of his privilege increases beyond the cost of compromising him. Logging, distribution of privileges, and other such solutions aren't really solutions so much as just a sort of cat and mouse game.
I would claim that impenetrable security is not only hard, but ultimately impossible.
95% of companies on the planet do not have the necessary resources or expertise to run a robust authentication infrastructure, and this one hack (which we really know nothing about at this point) doesn't change that fact.
If you are in the 5%, great! You are probably set already.
If you are not, overreacting isn't going to help anyone. There's no need to start writing user login code tonight.
> If you are not, overreacting isn't going to help anyone. There's no need to start writing user login code tonight.
I agree. That being said, there's no need to write login code yourself. Trust on the existing, tested, proven OSS solutions and libraries available, if you decide to not rely on a provider like Okta.
There's also an abundance of libraries for different programming languages to implement authentication or authorisation, lots of them battle-tested by thousands of services. It's not like providing secure services is impossible without Okta.
Yes but I'm having trouble with the client side.
Especially Cordova or Capacitor.
In a hybrid app with keycloak in the back it's just not working.
I tried the cordova-native keycloak adapter but it's just not working.
Pure Vue or Angular no issues.
Flutter issues because AppAuth is not very well documented and it's even worse for Android Java or Kotlin.
Security or auth should be treated as the president of the 1st class citizens, instead it's a mess.
On the backend everything is fine, but frontend is in a really bad state.
> There's no need to start writing user login code tonight.
I think it's worth pointing out that replacing Okta for most people isn't just doing login +MFA stuff.
Sure, there's that part of it - but a lot of the value in going with Okta in particular is that so many third parties support integrating with them in ways that other Identity providers are just not supported.
Account lifecycle stuff (via SCIM) is often limited to just Okta support for products.
There's also the account lifecycle stuff so that HR doesn't have to tell you when they hire someone, they just add them into the HR system and all the things flow out from that. Similarly when someone leaves, you can have those workflows go and disable/delete their accounts.
It's not just that. The most important part of Okta is the directory integration, not the SSO part. For small companies that have never acquired anyone, it doesn't matter.
It also makes user lifecycle someone else's problem. Very important. As a medium and larger sized company, when you have to tell your customers your homegrown IAM was pwned, you're in for a world of hurt. When you're just in the boat with everyone else who got popped, it's an easily recoverable incident.
If you have any sort of footprint, it's pretty much expected you'd have protective monitoring so you can articulate what these pwned accounts have actually achieved on your infrastructure.
It doesn't really matter who runs your auth service, if the auth service getting pwned results in a reportable attack, you're in for a world of hurt.
The next step is to articulate why this isn't your fault and that you did everything you reasonbly could have done to not get pwned.
Do you have evidence to support you did your due dilligance on your provider?
No, Okta also supports various forms of 2FA/MFA, Active Directory integration, LDAP and SCIM user provisioning. Not to mention the fact that even if there are open source alternatives out there, setting up servers, configuring them correctly and managing/updating them isn't exactly a simple enough operation for the average company admin.
So for example if you want to use AWS SSO tied to GSuite, you have to manually manage users, group membership, etc inside AWS SSO.
Similarly for a bunch of other products.
A couple of companies I've worked for now use GSuite as their primary accounts, and while a ton of things authenticate through GSuite (either as a built-in integration, or as SAML) - when you onboard/offboard users it involves going into a bunch of different systems to create/invite, set up group memberships, and then later deactivate accounts.
We're still considering Okta to solve all that workflow stuff. Yes, even after this event.
I believe Okta can also do custom security flows that GSuite can't.
So, if you're on a trusted device+network, then you might not need a 2FA challenge. Or conversely if you're trying to access a sensitive service - then always require a 2FA challenge every X hours.
yes my company makes us do the 2FA every 9 hours for github (so annoying) via okta, though apparently our gsuite logins via okta are much longer lived.
though that sort of answers a followup Q, it would seem that there would be a market that punts the authentication to gsuite's saml, but automates the user creation and the like on the different services (and deactivation), but one still wouldn't have the "custom flows" as you said.
but if the custom flows aren't that important, I'd rather trust google to handle my authentication than okta, and in my pseudo-okta without authentication scenario, even access to my pseudo-okta would authenticate via google.
> even if there are open source alternatives out there
This reads to me like you are unfamiliar with keycloak, which is a very mature and capable solution with plenty of support for everything you mention other than SCIM.
Hosting it is not overly challenging, running it in a container would suffice for most companies. Updates are in my experience a smooth process.
I'm not claiming it's as easy as an "auth as a service" platform, but as we see, that comes with its own costs.
Keycloak supports 2FA/MFA, Active Directory, LDAP. You might need a plugin for SCIM, but it does a whole load of other wonderful things, and it doesn't cost $2 per user per month.
You will have to find someone to host and patch it, but we shouldn't pretend it's up there with write your own crypto.
Using saas is fine too. Often there cheaper than hiring a person until you need said person anyway, and at that point they're just a single point of failure, and people do underestimate the effort to run a production service well.
But we shouldn't be so anti-intellectual that we pretend that it's the only way to do things, and that well supported foss products don't exist.
Keycloak has nearly all of this built in. You don’t need to be a crack sysadmin to set it up. If all you need is a 2FA-enabled OIDC endpoint federating to AD/LDAP to serve as SSO you can be up and running quite quickly.
So will likely require some changes to Keycloak internals in order to address that reliability issue, which will enable building a reliable interface for your use case.
Yes, you can achieve that specific part with Keycloak, and you may end up owned in a different, but equally awful way: https://www.keycloak.org/2021/12/cve There's nothing wrong with Keycloak, and me pointing at this CVE isn't meant as a gotcha. The point is that just because you're doing it yourself doesn't mean you're going to have better protection. Doing this well is difficult.
Okta is a security company, and due to a well-organized, well-funded attacker, they were able to be breached. If a similar attacker is trying to breach you, you're likely going to get breached.
> If a similar attacker is trying to breach you, you're likely going to get breached.
I think the logic here is accepting that if an attacker is trying to breach you, you’re likely going to get breached. But, if an attacker is trying to breach someone else, you’re still safe(-ish). With a centralized service, you don’t have to be the target to feel the effect.
But are $YOU, as one of the tens of thousand $YOU companies going to be spear headed in a dedicated, not automated way? Maybe, but I would say there is at least one or even two orders of magnitude difference with Okta or other well-known companies.
There's a cost efficiency point where this argument becomes valid.
At small company sizes, where dedicated security team or an admin with sufficient expertise to do proper setup is unaffordable, risk of Okta breach is lower than risk of automated custom setup breach. When you are big enough to deploy self-hosted solution and maintain it for reasonable cost, then yes, I would agree with you, self-hosted setup may reduce risks. When does that happen? When your company has over 1000 people (to break even with your Okta spending) or when your revenue per employee is high enough to spend every year 200-300k€ just on security.
I would think that 200-300k€/yr is low for this. At that budget for security you've maybe got two or three people, a bunch of software, and you're still building out monitoring and alerting systems. You haven't even begun to shift left and you're nowhere near being able to influence the product development lifecycle. You almost certainly don't have a security operations group capable of running a critical service 24/7.
That said, I would think Keycloak would fall under IT in many orgs. So you need a security org mature enough to have trained IT to the point where they can successfully run something this sensitive and security can have confidence in everything around it.
At a guess, that's not before you have 15-20 people in the security org. Probably something like: alerting and incident response (i.e., ops) is 3-5, vendor management 2-3, governance/audit 2-3, security engineering 3-5, security architecture 2-3, management 2-3, privacy 1-2. Run something like this for a year or two and see how the business as a whole shapes up.
As you say, it's a cost question. At what point does running your own IdP become the most valuable thing your company could do with a given chunk of money? One of the perks of a vendor is that the contract comes with liability. How much is that worth to the company?
Makes sense. Basically you are talking about 1.5-3M in annual spending just on salaries. With average IT spending around 3% in retail and manufacturing companies this will mean, that they must have several billions of annual revenue and security will still be a significant part of their IT budget.
For software companies this calculation will be completely different and the threshold can be lower.
If you use anything off the shelf like Keycloak, you're 1 CVE away from being automatically breached too, which is very similar to the situation you'd be in from Okta etc. being breached.
I agree, but it depends on the type of attack. Apparently this LAPSUS$ group uses corruptible employees/contractors to get in. As a small employer with a personal relationship to my employees I have an advantage that I might not want to throw away by delegating security to a company that doesn't have this advantage.
Okta delegated its operations, not its security; it just happens that their operations is your security.
Your company isn't delegating any operations away? Not even support? If they are, what level of access do those people have? Are the internal services they have access to as secure as your perimeter?
Even if they aren't, the attack can come through any internal employee, and your least security conscious person is your weakest link. Again, what do they have access to? Will they be able to use them to move laterally? Spear phishing from an internal employee to another employee is really, really effective.
Your keycloak instance will be found and indexed via automated scans. It will then be attacked minutes after the next CVE in keycloak, Java, Tomcat, or JBoss is disclosed. If you don’t have the 24x7 security team to handle that reality a managed service is likely a better option.
It doesn’t matter if you’re a “needle” if Shodan has a list of all “needles” readily available to attack. Even script kiddies can write for loops.
Proxyshell and similar recent issues have shown “near-instant compromise” to be the current state of affairs. Most instances are attacked within hours or even days before a vulnerability is disclosed publicly and hits the news.
It still raises the bar for the attacker. Now they explicitly need to seek out and target the company. Even if breaking into any single company might now be easier, breaking into all of them is much more difficult than if they used a single point of failure like Okta.
While it feels that way, in my experience, it is very easy to break into 10 poorly secured systems compared to just 1 properly secured and monitored one.
Okta may have messed up here, and there's no doubt they're a large target, but they're still a much harder target to breach compared to anything 99% of companies will be able to set up (unless you're a Fortune 50 company perhaps, and even then - it may not be cost effective even if you can make it work).
Expect to have 50+ engineers and other technical staff to be anywhere nearly as secure with a custom solution. A roll-your-own solution with a small crew supporting it can never compete.
The amount of effort they invested into breaching Okta is probably way more they'd invest in breaching any single company's SSO.
If "LAPSUS$" is after big companies with significant IP they can hold ransom for millions of dollars, it's most likely not even worth their time to find let alone attack your small company's on-prem SSO even if it was a single shell command away, so you'd still be more secure.
> 95% of companies on the planet do not have the necessary resources or expertise to run a robust authentication infrastructure
Wait, what exactly makes this problem so fundamentally hard?
What stops someone from just releasing a software system that solves the authentication infrastructure problem in a general way (e.g. as Kubernetes did for cloud orchestration)?
I don't understand the centralization argument. If anything, software solutions should make systems less centralized, since anyone will be able to run their own auth infrastructure on their own hardware.
If you mean that a single codebase could be an attractive target for attacks, it's still not a problem since the Free Software community loves having a dozen of competing implementations of the same thing just for the sake of it :)
Sorry, I don't mean that the actual system is centralized, as in everybody pings back to the same server or organization, but that it shares a similar vulnerability with having the same codebase running in local hardware.
And this isn't even solved by having standard protocols, open source implementations as you mentioned -- only mitigated. OpenSSL was exploited too.
Economic incentives and strongly held technical opinions like those of paxys occupying staff+ positions at every participant in modern computing. We’ve deluded ourselves as an industry into believing that computers are too hard to build operate for anyone but Marvel’s Avengers of the Bay Area. That lie underpins the entire market capitalization of public cloud and it’s amazing how little clothing that lie actually wears in the end.
Kubernetes doesn’t go after it, either; it complicates your system so exponentially that it drives the same lie. All Kubernetes does is assign a workload to a machine and hook crap up to it. That problem was solved in the 1960s by systems that don’t take multiple FTEs just to operate itself. When you look at Kubernetes in this light there’s a moment of clarity waiting regarding why it exists in the first place, and why it will never in a millennium replace Borg.
In 2010, I joined a social media startup. I was the only operations employee on the team of nine who had ever touched an actual server with my hands. We’re on the third or fourth generation of systems people now who’ve never heard anything except the cloud-native drum and how they’ll never be qualified to run their fingers along the faceplate of the computers running their business. paxys is sharing a derivative of that because it is a strongly held industry belief and a safe language to speak in this market. It just doesn’t make it true, and that’s a powerful realization, but that’s not something most people are ready to hear and truly understand.
You’re asking the absolutely right questions. You’re just asking an industry that gave up on itself so long ago that kids born after UTAH2000 can vote now.
what about having an extra in house level of security? Something that does not require the overhead and maintenance of Identity Management System? Not sure what that would look like though.
Apparently Plaid has 40% penetration of US bank account holders[1], and of course their entire business is storing the username and passwords for all those accounts so they can screen scrape. I struggle to think of a more attractive target than that!
In a B2B setting you get rid of a LOT of paperwork and due diligence by just saying "We use Okta for X, Y, Z." Hell for SOC2, Okta got rid of a shit ton of verifications for us at a critical stage of the startup I was leading, Papa.
Sure you can build your own but at that point you're wasting months for zero customer benefit. You could be building product. I don't know what the right answer is.
There is a lot of conversation here about building your own vs using something 'centralized'. All software comes with the “build vs buy” tradeoff.
Commercial open source companies (COSS) minimise those tradeoffs and give you the best of both worlds. That is - The power of build and the speed of buy.
You can migrate to and from a completely self hosted solution to a managed service as your needs change (for many COSS alternatives)
> something like Okta is an infinitely more attractive target than we are
Agreed. Centralization of attack surface is a major risk.
I understand the allure and have been in situations where even knowing the risks, the path is forced upon us because "nobody is fired for buying IBM".
But yes, a centralized service holding all the marbles is an infinitely more attractive target than lots of disjoint little systems which aren't individually very interesting. The centralized system may be better run (arguable, but at least they have the resources and it is their primary job) but they also face a much higher attack concentration.
A compromise solution is to have both. I've been in one company where okta was used as the primary SSO and for most internal systems the only one. But for the very sensitive resources, one had to be authenticated via okta but also via a homegrown MFA system.
No, that's distinctively different. This is about distributing attack surface, not obscuring authentication infrastructure. Are we really at a point where not relying on external account providers is considered unusual, or a bad practice? I'm not buying it.
> something like Okta is an infinitely more attractive target than we are
In my book that counts as "obscurity". "We are not an attractive target, hence we won't be targeted". You will be targeted, as long as your defence is weak. Only if you manage to set up a comparable defence to a large provider, this argument flies: because all things equal, attackers will attack the more attractive party. But all things are not equal. (Which I'm not certain about, it may very well be that Okta has a really poor defence in place, in which case my argument falls apart, because you'd be able to "make all things equal" much easier. I doubt this though).
> Are we really at a point where not relying on external account providers is considered unusual, or a bad practice?
No. I consider it reasonable to "build your own", but only in certain cases. Things like auth are not your core domain (unless you are an auth-provider, obviously), so all effort and time you spend on building the umpteenth login/auth-flow is not spent on building the stuff that sets you apart. Even if you drag in a standard library and only spend an afternoon: you're still maintaining it, testing it, etc. And with external parties, generally, you'll be following market best practices: fingerprint-login becomes standard? You'll get it almost for free, whereas in the diy-case you'll be designing, testing, building such progressing tech for ever. The economics for building are just wrong.
As are the security: as pointed out above: I highly doubt you'll be able to match a level of defence a large, focused, experienced party can obtain.
But there may very well be cases in which building your own login/auth makes sense and the tradeoffs in economics and security (and features) make sense. Maybe it has to integrate into some legacy; maybe (legal) requirements enforce you to keep it all in-house, etc.
> In my book that counts as "obscurity". "We are not an attractive target, hence we won't be targeted". You will be targeted, as long as your defence is weak.
I find this point to require more subtlety; "we are not as much of an attractive target, so we likely won't be targeted by a sophisticated, human adversary" is more apt for the situation. The attack vector is just not comparable to Okta's scale, which is true for the majority of companies and platforms out there. That doesn't mean security should be taken lightly, just that the magnitude and frequency of bad things happening will be lower.
> Things like auth are not your core domain
That's a good point, but it depends on the situation. I wouldn't recommend spending lots of time on this yourself, but there are standard libraries or OSS solutions available. If you're optimising for reliability, it might be beneficial to keep auth within your own systems. If you're optimising for privacy under GDPR, for example, it might be beneficial to not trust your users' data to an external provider. And so on.
> I wouldn't recommend spending lots of time on this yourself,
Unfortunately, this is in direct opposition to
> That doesn't mean security should be taken lightly
I agree about the subleties that, indeed, make the tradeoff more nuanced. You are less likely to be targeted. But, given that you "wouldn't recommend spending lots of time on this yourself" (my recommandation too), the trade-off balances dangerously into the "attractive target" too.
> but there are standard libraries or OSS solutions available.
Which, unfortunately is hardly different from "a centralized target". Look at, for example, WordPress. I used to have a WordPress hosting company: the constant battering on the entry-doors, by bots, and humans is real. The amount of successful breaches are staggering. I don't host anything WordPress anymore, yet IPs that haven't ever seen WP are hammered by bots trying to breach WordPress admin, XMLRPC, Woocommerce and so on.
By your reasoning, none of the WP sites we hosted back then were "attractive". By that reasoning, the servers I run now, without WP, are even less attractive. Yet they are battered by attackers. 24/7. And breached (not my current servers, though) quite often. Small websites for coaches, mom-pop-shops, the garage around the corner, your local football fanclub, and so on. Breached. Because they were an attractive target, not by being centralized, but because they weren't. Exactly because they use (OSS) libraries, combined with very little resources for defence, they are attractive.
This is a really nasty breach of the supply chain given how many companies nowadays consider IT to be a cost center - even capable engineering orgs end up using Okta because the cost of building and maintaining your own SSO especially throughout an entire business is seen as not worth the cost. This is another situation where the cheaper option (buy-in SSO as a SaaS) can be worse simply because you're actually buying into an enormous attack surface.
I remember being told more often than I can count that it is nest to leave stuff like this to the experts who all day long do nothing else.
Like actually being told that as reason to use Microsoft AD. Or the nobody is better dealing with spam than Google (when I was at a G Suite company). Or never to roll your own security library.
All these were in the same ballpark and in some occasions the latter one was used to drive home the others.
I am not sure what to do/say about it.
I am not (by trade) an IT person but hopefully learned a bit in my time. Trying to understand how all the different tools on my corporate device work (together?) to enhance security is impossible to me. I can't name them but from anti virus, to compliance checking to additional compliance checking by VPN software to endpoint monitoring to remote install capabilities to windows management to remote desktop capabilities and many more.
To me this sounds like too many potential attack vectors waiting to be exploited.
The screenshots on the linked tweet make it look like okta dog foods their own product for access to various services and someone has access to one of their admin accounts. Which is bad, but that could mean “we phished this one person who works at okta” and not “we compromised okta and have unfettered access to their customers/valuable assets”.
The news of the coming days may well prove me wrong, but i am not assuming the worst from this yet. Many companies whether or not they use an idaas do things like login anomalie detecting, and users coming in from weird locations and weird times of day would be sure to set of alarm bells at some of the big targets. Heck, AWS does it for customers with guard duty.
The breached account shown in screenshot belong to a user at a 3rd party outsourcing firm providing support services for Okta. So he is technically not an Okta employee.
It seems strange that such a user would have wide access. It could be that his account was just used to gain further access, or it could be that his account had wide access by mistake. Or the user doesn't actually have that wide access.
There are talks about superuser access. But is that referring to the user's actual privileges or the fact that he has access to the tool called "superuser" shown in the screenshots?
Any word on whether this affects Auth0? The company were bought by Okta last year but I don’t imagine they have consolidated everything yet so I’m hoping our Auth0 tenants are unaffected
In October 2020 I did a deep analysis on commercially supported and/or hosted Kubernetes distributions and platforms (think Rancher, OpenShift, Tanzu, EKS, GKE, etc.) and after doing most of my analysis, a Gartner report found its way to me. It largely matched my results with some skewing for factors I don't consider as important, but, were clearly explained in the report.
So, while I realize this is a small data point, it is one backed by quite a bit of data/analysis. Their report was well reasoned and accurate given the stated priors.
I would say that their credibility is fine so long as you read the reports fully and not simply glance at their magic quadrant figures.
390 comments
[ 28.4 ms ] story [ 6738 ms ] threadIt’s also very disappointing this is surfacing via tweets and not a breach notice from Okta, who apparently suspended at least one account used for entry.
If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic.
You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage (https://www.youtube.com/watch?v=30jNsCVLpAE). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.
>"Dead tired people are more likely to make brain dead decisions"
Indeed. In reading the post-mortem on the recent multi-day Roblox outage, it's hard not to imagine some bad decisions were made that only made problems worse and that likely these were on account of people just being fried by lack of sleep:
https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
Like I mentioned, it's sort of an "Armageddon", or worst case scenario. It's meant to help us identify issues with our docs, assist with order of operations, and it exposes any new hires to the full scope of all the moving pieces.
We already perform DR exercises, including testing our online backup restores, between production and DR sites, but this tabletop takes it further.
Hope that answers your question.
https://twitter.com/BillDemirkapi/status/1506117152461496323
It is a complex issue with entry of nuance I'm too tired to thumb type. I know there are lines that shouldn't be crossed. But it is so naive to think companies shouldn't have an eye on their data.
Do you think banks shouldn't have cameras in their lobbies? Or that tellers can walk in and out of the vault with black bags and no one ask what's inside?
To use your camera analogy, you better have a good lock on the security room holding the tapes.
Organizational endpoint security seems to be much more about incident response than incident prevention. That is, it's more focused on providing a nice audit trail that can be used to find and fire/arrest the perp once a breach occurs than it is at preventing breaches from happening. It probably fills a due-diligence tickbox somewhere, allowing the company to say "if a breach occurs (the probability of which is always nonzero) then we have measures in place to find out how, where, and by whom and mitigate the damage" to shareholders and important customers.
Software diversity is good. Remotely controlled mono-cultures are bad. IT management and security compliance people need to understand this.
I liken the mindset of (only we will use the remote control software to do good things) to Encryption back doors (only law enforcement will use them to catch criminals). Computer scientists call these 'Exceptional Access Systems' and it has been shown (many times) that it is impossible to ensure they will not be abused and used against you.
Keys Under Doormats: Mandating insecurity by requiring government access to all data and communications:
https://dspace.mit.edu/handle/1721.1/97690
What alternatives are there? I honestly don't know
If you're working from home and your device gets owned, and the attacker just sits there slowly slurping data as you go about your business... Chances are you're not going to detect it, but if you did you'd probably notice it's running when it's not supposed to and it's sending network traffic when not actively used.
If someone uses a laptop breach to go look at data you wouldn't normally look at ( or tons of records ) an audit log would give that away.
So really to do security well you have to profile the role. Why are you looking up customer record abc123? Trying to see what your girlfriends doing, or is it because they've called the customer support line?
You basically have to define what does misuse look like and protect for that.
Back on the slow slurp, hardend immutable devices that are regularly updated with limited Auth sessions would likely contain the breach.
Same deal on servers, you harder to stop it, but you monitor for when it happens.
Do people really monitor though? Some, sure, but your average org buys a tool the fails configure it with any sort of context.
Software diversity also means attack surface multiplication, so it's a delicate tradeoff.
I have talked to security engineers that have had the company procure millions of dollars worth of security tools, and yet they don't even know the basics of security.
For God's sake, I had to explain what end-to-end-encryption was to a team of "senior" security engineers the other day. They genuinely had no concept of it. As a security SWE I am so done with these sort of people. Fuck them! Why are they even in security!? Real private data is at risk because of their incompetency!
[0] https://darknetdiaries.com/episode/96/
It would be interesting to know Tanium takes to backport security fixes. I mean py2.7 was sunset 2020-01-01 [1].
[0] https://www.tanium.com/ [1] https://www.python.org/doc/sunset-python-2/
When I was in college the whole student network went down. Apparently there was a hack from the student network to the campus police network, so campus IT pulled the plug on any of us having network access. Because those two weren't separated because why would they be?
So I hoofed it to the IT department and I asked to talk to someone who knows a thing or two about networking. The director of IT came out. He sat me down and started telling me about the $300,000 budget he just got approval on. I was thinking, you can't take some of that and stand up a router with packet filtering to isolate the student network from the police network in two different subnets? A FreeBSD box might do in a pinch. (It was the early 2000s and network traffic loads were... different.)
I learned a lesson that day about seniority and the Peter Principle. Climb high enough and your mindset changes, from technical solutions to issues of money and resource allocation. From there, "let's just pay $VENDOR to do it, they'll solve the problem for us" is but a small step.
Same company had a head of security who was full-time employed by two different companies at the same time (he had unlimited vacation time for both) without either knowing the other existed.
Best thing was that the company did investment/insurance services and had a banking division.
0: https://twitter.com/BillDemirkapi/status/1506123471352438784
[1] https://www.theregister.com/2022/03/21/microsoft_lapsus_brea...
[2] https://www.wired.com/story/lapsus-hacking-group-extortion-n...
[3] https://analyticsindiamag.com/lapsus-hack-leaves-nvidia-in-a...
That other product/service was compromised, which let hackers hack into MS employee’s computer, which gave them access to Azure infrastructure.
Lesson: ask your (in this case Microsoft) employees to not use work computers for any personal use.
/me goes to check if my password manager company uses Okta.
This is just wild speculation to make up a scenario where Microsoft could be affected by the Okta hack with no evidence to back it up.
I was just trying to come up with a threat vector to explain how it might be possible.
And sometimes there are chains of dependencies that need to be swapped out in sequence.
So from acquisitions alone it would seem reasonable to expect that Microsoft at any point in time has at least a handful of usages of certain SaaS offerings, even if MS themselves offer their own versions of those services.
That said, the surface area of these "unofficial" 3rd party systems is likely to be quite small in comparison to whatever their official tech stack looks like (which does -- I would assume -- centre around Azure AD). I'm not expressing an opinion on whether the GP is correct to make this link...
I've worked with some very large companies where internally they (surprisingly to me) make use of competitors' products even when that company makes their own version of that product. There are all sorts of legitimate reasons why this happens.
https://gizmodo.com/microsoft-investigating-potential-lapsus...
Honestly I’m so unimpressed with AWS that I find it hard to say Azure is terrible, because the bar for terrible is set much lower.
[1] https://twitter.com/BillDemirkapi/status/1506109956298317830...
When they always have a safe haven to run to and lay low in a comfortable way without significant losses (that aren't just cost of business), what good is such a slap?
In my mind I see it like the same as throwing a beer bottle at a robbers car as they drive away. Even if you hit it, at best it's some scratches or maybe a busted tail light, but they're still gone with your goods.
Doubt it
Government is big business for AWS.
No secret or top secret info is permitted in the typical Impact Level 4 and Impact Level 5 FedRAMP systems. Okta is only certified at IL4 I think.
AWS & Azure do offer IL6, meaning information processed up to the SECRET classification.
https://nitter.net/_MG_/status/1506109152665382920
As for how often I open twitter links, is as often as they make it to HN front page. So something along of one twitter link around every 2 days.
“Authentication services company Okta Inc said on Tuesday it is investigating a report of a digital breach after hackers posted screenshots of what they claimed were its internal company environment.”
https://www.reuters.com/article/okta-breach-idUSL2N2VP07B
It’s perfectly okay to not say anything prematurely that can cause any confusion; with the employees, customers, and media.
All eyes are on them; it’s better not to screw up whatever little trust they have left.
I really don't see the problem with their response. What would you propose in the circumstances?
It's distinguishable from a noop because some information is imparted, namely that a) they are aware of the issue and b) that they have formed a preliminary view it warrants a response.
I bet a lot of IntoSec departments are stressing right now because of this tweet.
WASHINGTON (Reuters) - Authentication services company Okta Inc said on Tuesday it is investigating a report of a digital breach after hackers posted screenshots of what they claimed were its internal company environment.
A hack at Okta could have potential major consequences because thousands of other companies rely on the San Francisco-based firm to manage access to their own networks and applications.
The screenshots were posted by a group of ransom-seeking hackers known as LAPSUS$ on their Telegram channel late on Monday.
> Authentication services company Okta Inc said on Tuesday
Where? What did they say? To who? The person writing the article?
WASHINGTON (Reuters) -Authentication services provider Okta Inc is investigating a report of a digital breach, the company said on Tuesday, after hackers posted screenshots showing what they claimed was its internal company environment.
A hack at Okta could have major consequences because thousands of other companies rely on the San Francisco-based firm to manage access to their own networks and applications.
The company was aware of the reports and was investigating, Okta official Chris Hollis said in a brief statement.
"We will provide updates as more information becomes available," he added.
The screenshots were posted by a group of ransom-seeking hackers known as LAPSUS$ on their Telegram channel late on Monday. In an accompanying message, the group said its focus was "ONLY on Okta customers."
Security experts told Reuters the screenshots appeared to be authentic.
"I definitely do believe it is credible," said independent security researcher Bill Demirkapi, citing pictures of what appeared to be Okta's internal tickets and its in-house chat on the Slack messaging app.
Dan Tentler, the founder of cybersecurity consultancy Phobos Group, said he too believed the breach was real and urged Okta customers to be "very vigilant right now."
In an email, Tentler added, "There are timestamps and dates visible in the screenshots indicating January 21st of this year, which suggests they may have had access for two months."
The lawyers might be sweating on their next round of bonuses, due to dealing with the fallout. But their jobs are not likely to be on the line.
Why are people Nittering something they can view from source. This is why propaganda is so easy, it's what people want.
seems like it'd be possible write an app that scans email history to detect services that have your data, scrape their websites to determine support email address(es), and send a GDPR data deletion request template to each of the selected ones.
Keycloak, for one: https://www.keycloak.org
I've also heard good things about Hydra: https://www.ory.sh/hydra/
Depending on your needs, an SSO proxy like Vouch works well for internal services: https://github.com/vouch/vouch-proxy
There's also an abundance of libraries for different programming languages to implement authentication or authorisation, lots of them battle-tested by thousands of services. It's not like providing secure services is impossible without Okta.
For medium to large enterprises, the calculus is harder and there's going to be months of flame war trying to hash that out.
The problem is motivation to break it trends up over time as the service gets more and more popular.
In some ways, if Okta can fall basically anyone can. Sometimes there's an advantage in being smaller. One would hope that the actual attack surface on Okta was smaller than the post seems to imply. That there is even any way to get full access to all customers would seem to be a critical failure.
This might be true, but for someone to "fall" they must be targeted to begin with.
If "LAPSUS$" is after big companies with significant IP which they can hold ransom for millions, as a small business you'll be safe because they won't even bother attacking you, and may not be willing to put the same amount of effort into breaching your small company's SSO than they would for Okta.
Yes, it's going to cost more upfront, but there are a wide range of options between "rely on a blackbox PaaS/SaaS" and "write and deploy everything yourself".
Maybe a company that specializes in this kind of thing? Like Okta?
How do you have an SSO system without a centralized service? Whether you buy it as an outsourced SaaS service or run it yourself, it's still SaaS, but now you're on the hook for managing it and keeping it secure.
Or for comms, run a team- or company-wide chat server rather than relying on Slack/Teams/Discord.
Either you're missing my point (and maybe I didn't express myself super clearly), or you're pitching a false dichotomy.
> who has more expertise at building and running an SSO service than a company that's selling that service
OP illustrates well the consequence of that line of reasoning. I'm certain this is just the beginning as more too-big-to-fail vendors get compromised.
The cloud is just somebody elses' computer, "Move fast and break things" mean things get broken, etc.
Whenever we get subscription services for "smart security" involving cloud-connected cameras, AI-controlled locks and drones, would you also consider that as the one-size-fits all over having security guards or contracting companies like Securitas or your local equivalent?
> now you're on the hook for managing it and keeping it secure
You always were. There is no free lunch.
If Okta gets away with this (and the Equifax breach and subsequent outcome suggests they will) it proves that the market doesn't actually care about security, thus a company selling that service has little incentive to actually invest in security.
On the other hand, I feel fine using GSuite for SSO because we have a much better view of their security.
That said, it sucks that everyone is so fucking bad at security. I maintain that it is not that hard.
Add in systems that were built for X and now doing Y. This is hard to get right. A single slip up will lead to you being compromised.
Yeah, people need to stop using memory unsafe languages. They choose not to.
> A single slip up will lead to you being compromised.
Not if you add multiple layers of security. Like sandboxing. Or mTLS. It's not hard to do that.
edit: Let me clarify. Security isn't hard generally, but it's hard individually, because you're drowning under everyone else making it artificially 10000x harder.
Golang and Rust are not magic bullets that makes systems automatically secure.
Flaws in application logic have little to do with language choice.
Also consider the effort and money it takes to rewrite a multi million LOC system with several dependent apps. The new trendy languages introduce breaking changes, switch paradigms, and have less mature ecosystems.
What are their usual criticisms? Genuinely curious. I've always hated their UI, their docs, and lack of real support but felt that at least they had security going for them. The last assumption might have been proven wrong tonight.
Also, what would be a reasonable alternative to Okta now that their nearest competitor, Auth0, was acquired by them. Is GSuite a reasonable alternative for SSO with multiple different providers and supporting SAML, etc.? Thanks in advance.
I wouldn't want to make a recommendation, but I'll say that my company uses GSuite.
I'm struggling to see how that attack works without a copy of the private key.
Any pointers for what to search for to understand this better?
A better solution is to use ephemeral SSH keys generated using an SSH CA. This kind of thing can be implemented with Hashicorp Vault, though I'm sure there are plenty of other solutions out there.
It simplifies key checking on servers too, as they just need to the details of the signer and there's no need to juggle keys in LDAP.
I don't remember Google having any large-scale security incidens.
Have they been better at playing down their security incidents or are they doing something very right that the rest of the industry can learn from?
yes
> that the rest of the industry can learn from
unlikely
It's simple: they pay absurd amounts of money for top talent and let them work. Look: https://www.levels.fyi/company/Google/salaries/Software-Engi... can your company afford to pay 2-300K cash -- not to mention serious stock -- to bread and butter mid level engineers?
Then their SNMP mib doesn't work properly. So you have a a box that is proxying some of your most critical systems that are so old that to integrate them with MFA requires OAG (think mainframes, old ERP systems etc) and you have to take Okta's word for it that the server is secure and not hacked.
They do thankfully support Syslog for logs, but again you have to take their word for it that you're getting all the logs, because you can't access the system to verify.
Having said all that. OAG solves a very real business problem and it is hard to find a competitor in the market with the level of integration it has to legacy platforms.
Security isn't just about overflows and injections, outsider enemies vs insider allies. Any human with privileged access that can be compromised will eventually be compromised as the perceived value of his privilege increases beyond the cost of compromising him. Logging, distribution of privileges, and other such solutions aren't really solutions so much as just a sort of cat and mouse game.
I would claim that impenetrable security is not only hard, but ultimately impossible.
If you are in the 5%, great! You are probably set already.
If you are not, overreacting isn't going to help anyone. There's no need to start writing user login code tonight.
I agree. That being said, there's no need to write login code yourself. Trust on the existing, tested, proven OSS solutions and libraries available, if you decide to not rely on a provider like Okta.
Can you recommend some?
I've also heard good things about Hydra: https://www.ory.sh/hydra/
Depending on your needs, an SSO proxy like Vouch works well for internal services: https://github.com/vouch/vouch-proxy
There's also an abundance of libraries for different programming languages to implement authentication or authorisation, lots of them battle-tested by thousands of services. It's not like providing secure services is impossible without Okta.
I think it's worth pointing out that replacing Okta for most people isn't just doing login +MFA stuff.
Sure, there's that part of it - but a lot of the value in going with Okta in particular is that so many third parties support integrating with them in ways that other Identity providers are just not supported.
Account lifecycle stuff (via SCIM) is often limited to just Okta support for products.
There's also the account lifecycle stuff so that HR doesn't have to tell you when they hire someone, they just add them into the HR system and all the things flow out from that. Similarly when someone leaves, you can have those workflows go and disable/delete their accounts.
In fact, a third-party’s “Okta integration” should work out of the box with Keycloak or any other IdP.
It also makes user lifecycle someone else's problem. Very important. As a medium and larger sized company, when you have to tell your customers your homegrown IAM was pwned, you're in for a world of hurt. When you're just in the boat with everyone else who got popped, it's an easily recoverable incident.
It doesn't really matter who runs your auth service, if the auth service getting pwned results in a reportable attack, you're in for a world of hurt.
The next step is to articulate why this isn't your fault and that you did everything you reasonbly could have done to not get pwned.
Do you have evidence to support you did your due dilligance on your provider?
So for example if you want to use AWS SSO tied to GSuite, you have to manually manage users, group membership, etc inside AWS SSO.
Similarly for a bunch of other products.
A couple of companies I've worked for now use GSuite as their primary accounts, and while a ton of things authenticate through GSuite (either as a built-in integration, or as SAML) - when you onboard/offboard users it involves going into a bunch of different systems to create/invite, set up group memberships, and then later deactivate accounts.
We're still considering Okta to solve all that workflow stuff. Yes, even after this event.
I believe Okta can also do custom security flows that GSuite can't.
So, if you're on a trusted device+network, then you might not need a 2FA challenge. Or conversely if you're trying to access a sensitive service - then always require a 2FA challenge every X hours.
though that sort of answers a followup Q, it would seem that there would be a market that punts the authentication to gsuite's saml, but automates the user creation and the like on the different services (and deactivation), but one still wouldn't have the "custom flows" as you said.
but if the custom flows aren't that important, I'd rather trust google to handle my authentication than okta, and in my pseudo-okta without authentication scenario, even access to my pseudo-okta would authenticate via google.
This reads to me like you are unfamiliar with keycloak, which is a very mature and capable solution with plenty of support for everything you mention other than SCIM.
Hosting it is not overly challenging, running it in a container would suffice for most companies. Updates are in my experience a smooth process.
I'm not claiming it's as easy as an "auth as a service" platform, but as we see, that comes with its own costs.
You will have to find someone to host and patch it, but we shouldn't pretend it's up there with write your own crypto.
Using saas is fine too. Often there cheaper than hiring a person until you need said person anyway, and at that point they're just a single point of failure, and people do underestimate the effort to run a production service well.
But we shouldn't be so anti-intellectual that we pretend that it's the only way to do things, and that well supported foss products don't exist.
[1] 2FA/Time based OTP: https://www.keycloak.org/docs/12.0/server_admin/keycloak-ima...
[2] LDAP/AD (Kerberos) https://dmc.datical.com/z_resources/images/assets/screenshot...
[3] SCIM (third party) https://github.com/Captain-P-Goldfish/scim-for-keycloak
Most of the time I'm talking about SCIM outbound, so provisioning/managing users in some other random service (like Trello, Github, whatever).
That's the major value-add for Okta.
https://stackoverflow.com/questions/58566587/does-keycloak-s...
So will likely require some changes to Keycloak internals in order to address that reliability issue, which will enable building a reliable interface for your use case.
Okta is a security company, and due to a well-organized, well-funded attacker, they were able to be breached. If a similar attacker is trying to breach you, you're likely going to get breached.
I think the logic here is accepting that if an attacker is trying to breach you, you’re likely going to get breached. But, if an attacker is trying to breach someone else, you’re still safe(-ish). With a centralized service, you don’t have to be the target to feel the effect.
At small company sizes, where dedicated security team or an admin with sufficient expertise to do proper setup is unaffordable, risk of Okta breach is lower than risk of automated custom setup breach. When you are big enough to deploy self-hosted solution and maintain it for reasonable cost, then yes, I would agree with you, self-hosted setup may reduce risks. When does that happen? When your company has over 1000 people (to break even with your Okta spending) or when your revenue per employee is high enough to spend every year 200-300k€ just on security.
That said, I would think Keycloak would fall under IT in many orgs. So you need a security org mature enough to have trained IT to the point where they can successfully run something this sensitive and security can have confidence in everything around it.
At a guess, that's not before you have 15-20 people in the security org. Probably something like: alerting and incident response (i.e., ops) is 3-5, vendor management 2-3, governance/audit 2-3, security engineering 3-5, security architecture 2-3, management 2-3, privacy 1-2. Run something like this for a year or two and see how the business as a whole shapes up.
As you say, it's a cost question. At what point does running your own IdP become the most valuable thing your company could do with a given chunk of money? One of the perks of a vendor is that the contract comes with liability. How much is that worth to the company?
Your company isn't delegating any operations away? Not even support? If they are, what level of access do those people have? Are the internal services they have access to as secure as your perimeter?
Even if they aren't, the attack can come through any internal employee, and your least security conscious person is your weakest link. Again, what do they have access to? Will they be able to use them to move laterally? Spear phishing from an internal employee to another employee is really, really effective.
BTW Keycloak.current has tossed JBoss & friends for Quarkus. In any case, it's been a remarkably safe product over the years.
Proxyshell and similar recent issues have shown “near-instant compromise” to be the current state of affairs. Most instances are attacked within hours or even days before a vulnerability is disclosed publicly and hits the news.
If "LAPSUS$" is after big companies with significant IP they can hold ransom for millions of dollars, it's most likely not even worth their time to find let alone attack your small company's on-prem SSO even if it was a single shell command away, so you'd still be more secure.
Wait, what exactly makes this problem so fundamentally hard?
What stops someone from just releasing a software system that solves the authentication infrastructure problem in a general way (e.g. as Kubernetes did for cloud orchestration)?
If you mean that a single codebase could be an attractive target for attacks, it's still not a problem since the Free Software community loves having a dozen of competing implementations of the same thing just for the sake of it :)
And this isn't even solved by having standard protocols, open source implementations as you mentioned -- only mitigated. OpenSSL was exploited too.
Kubernetes doesn’t go after it, either; it complicates your system so exponentially that it drives the same lie. All Kubernetes does is assign a workload to a machine and hook crap up to it. That problem was solved in the 1960s by systems that don’t take multiple FTEs just to operate itself. When you look at Kubernetes in this light there’s a moment of clarity waiting regarding why it exists in the first place, and why it will never in a millennium replace Borg.
In 2010, I joined a social media startup. I was the only operations employee on the team of nine who had ever touched an actual server with my hands. We’re on the third or fourth generation of systems people now who’ve never heard anything except the cloud-native drum and how they’ll never be qualified to run their fingers along the faceplate of the computers running their business. paxys is sharing a derivative of that because it is a strongly held industry belief and a safe language to speak in this market. It just doesn’t make it true, and that’s a powerful realization, but that’s not something most people are ready to hear and truly understand.
You’re asking the absolutely right questions. You’re just asking an industry that gave up on itself so long ago that kids born after UTAH2000 can vote now.
[1] https://www.thediff.co/p/plaid-data-layer-to-payments-layer
In a B2B setting you get rid of a LOT of paperwork and due diligence by just saying "We use Okta for X, Y, Z." Hell for SOC2, Okta got rid of a shit ton of verifications for us at a critical stage of the startup I was leading, Papa.
Sure you can build your own but at that point you're wasting months for zero customer benefit. You could be building product. I don't know what the right answer is.
No, not just that.
Commercial open source companies (COSS) minimise those tradeoffs and give you the best of both worlds. That is - The power of build and the speed of buy.
You can migrate to and from a completely self hosted solution to a managed service as your needs change (for many COSS alternatives)
Agreed. Centralization of attack surface is a major risk.
I understand the allure and have been in situations where even knowing the risks, the path is forced upon us because "nobody is fired for buying IBM".
But yes, a centralized service holding all the marbles is an infinitely more attractive target than lots of disjoint little systems which aren't individually very interesting. The centralized system may be better run (arguable, but at least they have the resources and it is their primary job) but they also face a much higher attack concentration.
A compromise solution is to have both. I've been in one company where okta was used as the primary SSO and for most internal systems the only one. But for the very sensitive resources, one had to be authenticated via okta but also via a homegrown MFA system.
It doesn't work.
In my book that counts as "obscurity". "We are not an attractive target, hence we won't be targeted". You will be targeted, as long as your defence is weak. Only if you manage to set up a comparable defence to a large provider, this argument flies: because all things equal, attackers will attack the more attractive party. But all things are not equal. (Which I'm not certain about, it may very well be that Okta has a really poor defence in place, in which case my argument falls apart, because you'd be able to "make all things equal" much easier. I doubt this though).
> Are we really at a point where not relying on external account providers is considered unusual, or a bad practice?
No. I consider it reasonable to "build your own", but only in certain cases. Things like auth are not your core domain (unless you are an auth-provider, obviously), so all effort and time you spend on building the umpteenth login/auth-flow is not spent on building the stuff that sets you apart. Even if you drag in a standard library and only spend an afternoon: you're still maintaining it, testing it, etc. And with external parties, generally, you'll be following market best practices: fingerprint-login becomes standard? You'll get it almost for free, whereas in the diy-case you'll be designing, testing, building such progressing tech for ever. The economics for building are just wrong.
As are the security: as pointed out above: I highly doubt you'll be able to match a level of defence a large, focused, experienced party can obtain.
But there may very well be cases in which building your own login/auth makes sense and the tradeoffs in economics and security (and features) make sense. Maybe it has to integrate into some legacy; maybe (legal) requirements enforce you to keep it all in-house, etc.
I find this point to require more subtlety; "we are not as much of an attractive target, so we likely won't be targeted by a sophisticated, human adversary" is more apt for the situation. The attack vector is just not comparable to Okta's scale, which is true for the majority of companies and platforms out there. That doesn't mean security should be taken lightly, just that the magnitude and frequency of bad things happening will be lower.
> Things like auth are not your core domain
That's a good point, but it depends on the situation. I wouldn't recommend spending lots of time on this yourself, but there are standard libraries or OSS solutions available. If you're optimising for reliability, it might be beneficial to keep auth within your own systems. If you're optimising for privacy under GDPR, for example, it might be beneficial to not trust your users' data to an external provider. And so on.
Unfortunately, this is in direct opposition to
> That doesn't mean security should be taken lightly
I agree about the subleties that, indeed, make the tradeoff more nuanced. You are less likely to be targeted. But, given that you "wouldn't recommend spending lots of time on this yourself" (my recommandation too), the trade-off balances dangerously into the "attractive target" too.
> but there are standard libraries or OSS solutions available.
Which, unfortunately is hardly different from "a centralized target". Look at, for example, WordPress. I used to have a WordPress hosting company: the constant battering on the entry-doors, by bots, and humans is real. The amount of successful breaches are staggering. I don't host anything WordPress anymore, yet IPs that haven't ever seen WP are hammered by bots trying to breach WordPress admin, XMLRPC, Woocommerce and so on.
By your reasoning, none of the WP sites we hosted back then were "attractive". By that reasoning, the servers I run now, without WP, are even less attractive. Yet they are battered by attackers. 24/7. And breached (not my current servers, though) quite often. Small websites for coaches, mom-pop-shops, the garage around the corner, your local football fanclub, and so on. Breached. Because they were an attractive target, not by being centralized, but because they weren't. Exactly because they use (OSS) libraries, combined with very little resources for defence, they are attractive.
Like actually being told that as reason to use Microsoft AD. Or the nobody is better dealing with spam than Google (when I was at a G Suite company). Or never to roll your own security library.
All these were in the same ballpark and in some occasions the latter one was used to drive home the others.
I am not sure what to do/say about it.
I am not (by trade) an IT person but hopefully learned a bit in my time. Trying to understand how all the different tools on my corporate device work (together?) to enhance security is impossible to me. I can't name them but from anti virus, to compliance checking to additional compliance checking by VPN software to endpoint monitoring to remote install capabilities to windows management to remote desktop capabilities and many more.
To me this sounds like too many potential attack vectors waiting to be exploited.
(I'm aware he's often on here as well, hi eastdakota!)
The news of the coming days may well prove me wrong, but i am not assuming the worst from this yet. Many companies whether or not they use an idaas do things like login anomalie detecting, and users coming in from weird locations and weird times of day would be sure to set of alarm bells at some of the big targets. Heck, AWS does it for customers with guard duty.
It seems strange that such a user would have wide access. It could be that his account was just used to gain further access, or it could be that his account had wide access by mistake. Or the user doesn't actually have that wide access.
There are talks about superuser access. But is that referring to the user's actual privileges or the fact that he has access to the tool called "superuser" shown in the screenshots?
I need more patience.
seems pretty easy to hack a SaaS anyway. just need to phish an engineering employee account. often they have superuser access to debug customer issues
> yes we know the URL has a email address. the account is suspended - we dont care
Not saying this looks good and really not trying to defend Okta, but a user account being terminated isn't proof that Okta knew about it.
Does Gartner have any credibnility other than being a pay to be endorsed shill?
So, while I realize this is a small data point, it is one backed by quite a bit of data/analysis. Their report was well reasoned and accurate given the stated priors.
I would say that their credibility is fine so long as you read the reports fully and not simply glance at their magic quadrant figures.