Why did it take two weeks to recover from this? Aren't there protections in AWS from simply deleting VMs. Wouldn't AWS step in and help a company like Cisco if something horrible went wrong?
For this, in general, the answer is: (1) yes, IAM permissions, and (2) yes, if you configure them, and they often have a cost.
> Wouldn't AWS step in and help a company like Cisco if something horrible went wrong?
If Cisco asked, probably. But some things are inherently non-trivial to recover from once you've allowed them to happen. AWS can help, but they aren't magic.
Set the termination protection flag on critical instances (like databases), then they cannot be deleted programmatically. This works really really well.
> Set the termination protection flag on critical instances (like databases), then they cannot be deleted programmatically.
Well, not accidentally, but the flag itself can be cleared programmatically, so protected instances can definitely still be maliciously deleted programmatically, if the actor has sufficient access.
What’s going on at Cisco where a departed engineer doesn’t have AWS credentials revoked for a whole 4 months? Hell, the person probably would have had access indefinitely if they had not had done something to make to make Cisco aware of the access.
I imagine it was bad communication between the guys handling AWS credentials and IT.
At most companies IT handles logins and credentials for all internal systems. Things like email, vpn, internal sites, and chat. Usually this is all centralized, and when someone leaves IT revokes that user's access to everything.
I'm guessing what happened here was that the engineer got access to AWS manually. Maybe IT wasn't responsible for handling AWS credentials and it was expected that the person who gave him access would revoke it too. Alternatively, the person who gave him might not have gone through the company's regular policies.
Either way, the process in place for handling credentials here clearly didn't work and they got burned in it. Would love to see the internal postmortem here.
Based on the other thread from yesterday, I wouldn’t jump to conclusions and pin the blame 100% to the former engineer. For such a large company, why the hell did they not put in safeguards to prevent former employees from accessing production environments? The vulnerability wasn’t like a week or two... it was 5 whole months.
This demonstrates Cisco is shockingly incompetent. Imagine if, instead of basically causing them a business hassle, this engineer decided to grab the list of sales contacts for companies -- easily available by looking at who salespeople are meeting with -- and sell those?
It's cisco's job to make this impossible in more robust ways than hoping every employee they have is a good person.
Cisco are also liars, because they've almost certainly made commitments in SOC2 or 27001 or other audits that this is impossible via policy and procedures. And yet.
When you login at Cisco WebEx, they warn you of criminal liability. However, if you report a security concern at Cisco they treat you like a criminal. Basically, as an employee at Cisco you take on all the liability but they won't protect you from their own negligence.
I'm guessing that if Cisco has a documented/known security issue they are more liable in case of future lawsuits. If they simply don't know about it then they are less liable. Someone did the math and found that non-liable lawsuits are cheaper than paying to have security issues addressed properly.
16 comments
[ 2.4 ms ] story [ 51.6 ms ] threadFor this, in general, the answer is: (1) yes, IAM permissions, and (2) yes, if you configure them, and they often have a cost.
> Wouldn't AWS step in and help a company like Cisco if something horrible went wrong?
If Cisco asked, probably. But some things are inherently non-trivial to recover from once you've allowed them to happen. AWS can help, but they aren't magic.
Remove the user that is being terminated from the group and 'voila', it is safe.
Well, not accidentally, but the flag itself can be cleared programmatically, so protected instances can definitely still be maliciously deleted programmatically, if the actor has sufficient access.
At most companies IT handles logins and credentials for all internal systems. Things like email, vpn, internal sites, and chat. Usually this is all centralized, and when someone leaves IT revokes that user's access to everything.
I'm guessing what happened here was that the engineer got access to AWS manually. Maybe IT wasn't responsible for handling AWS credentials and it was expected that the person who gave him access would revoke it too. Alternatively, the person who gave him might not have gone through the company's regular policies.
Either way, the process in place for handling credentials here clearly didn't work and they got burned in it. Would love to see the internal postmortem here.
I read this as incompetence on Cisco’s part.
This demonstrates Cisco is shockingly incompetent. Imagine if, instead of basically causing them a business hassle, this engineer decided to grab the list of sales contacts for companies -- easily available by looking at who salespeople are meeting with -- and sell those?
It's cisco's job to make this impossible in more robust ways than hoping every employee they have is a good person.
Cisco are also liars, because they've almost certainly made commitments in SOC2 or 27001 or other audits that this is impossible via policy and procedures. And yet.