Stuff like this is always possible wherever you have fully centralized architecture. I don't know how many massive cloud failures it will take for IT community at large to realize this.
That's not very fair to a company that was one of the pioneers of a lot of the things we consider normal on today's software as a service.
Also incorrect, they have great cloud and devops practices. If anything it's likely this bug's impact would be limited due to how decentralised SFDC operates.
Still a massive fuck-up, I'm interested in seeing if they'll release any more detail on why it happened.
While a little unfair, and Salesforce is a decent product with nice dev tooling (apart from the weird ancient Java ish custom language), but under the hood, it really is just a Oracle database per org.
In fairness they did do long enough ago that they would have massively messed it up if they at least didn't understand something about running software.
SFDC hate is pretty common, maybe because of how big they are. I think that their tech is actually pretty impressive.
Other than being pretty public at this point, this incident could have easily happened in an upgrade/rollout in on-premise settings as well (at least without a good staging environment and test process).
This is just a reminder that no matter how big or successful you are shit like this can always happen. And it’s usually not the fault of a single person, but rather some lack of process/review/control that made it possible in the first place.
I feel terrible for whoever initiated this. I’ve been in that boat and it -really- sucks.
It's hard to make things totally impossible, but hopefully a good post-mortem will identify the systemic issues that led to it happening, fix them, and not throw anyone under the bus.
For anyone reading this who has login access to a production SQL database...:
* Change your account to readonly. Make a new admin account, and put its credentials somewhere hard to get (and audited!).
* Make a directory in git for 'one off sql statements'. Make them all go through code review and have an automated system run them on merge/deploy.
* Enforce style rules with a linter/test, like "UPDATE must have a LIMIT"
* Anything the above process is too burdensome to do should have an API or admin interface built for the purpose.
* Aim to eventually get rid of your readonly account. A leaked customer data dump could kill the company and shouldn't be available to any malware on your machine. You aren't as secure as you think you are.
Fair warning about this: if your transaction isolation level in your database is set to serialized hanging out in mid transaction while you verify your changes will block other calls to the table(s)!
I always recommend that you are wrapping calls in transactions if you are touching prod data. I only left the WHERE clause of my update query off once in my life before I started doing that.
I agree with everything you said except "a customer database leak could kill a company".
Do we have any example of any company going under due to a data breach? Unfortunately, there seem to be a lot of examples of enormous breaches that did essentially nothing. (like Experian)
Well yes, but I would attribute the first to simply doing illegal things and getting caught. The data breach just helped them get caught faster. The second, yes, although privacy was literally their number one product and I wouldn't categorize a leak quite the same as a breach.
> * Enforce style rules with a linter/test, like "UPDATE must have a LIMIT"
As someone who has never had access to a DB with any serious number of users, can you explain this one further? What if you really do want to update every row? Do you just do LIMIT INT_MAX or the like, and just force people to write that so that they always know they're updating the entire table? Or are you saying you should only ever use UPDATE on a known finite (and small?) number of rows?
Updating every row is usually a bad idea anyway. That means every row needs to be rewritten, which involves reading and writing all data in the table. That will take locks which mean old data needs to be kept (in case of a transaction abort), doubling disk usage. Since half of rows are now dead, all other queries to the table will take double the disk access time.
Basically, if your table is over a few gigabytes, updating every row in one query on a production instance is a really bad plan.
So how do you make the change then? Do SQL dialects provide built-in tools to do an UPDATE to a large table in a saner way, or do you end up having to write an application to do so?
I'm sorry for being a SQL noob. Can you elaborate on what is wrong with this, and what would be the way to write it without causing the scale of problem described in the OP?
There are lots of good answers already posted, but if I can make a further suggestion...
It is often worth running a SELECT on the WHERE clause you are about to use for your UPDATE. That way you can make sure only a limited amount of data comes back before you launch something catastrophic.
> In addition to the SQL-standard privilege system available through GRANT, tables can have row security policies that restrict, on a per-user basis, which rows can be returned by normal queries or inserted, updated, or deleted by data modification commands. This feature is also known as Row-Level Security
As a manager, if you see a poor bastard having the worst day of their career because of something similar, it means you have inadequate safeguards in place and you’re not doing your job correctly.
Our product syncs data to Salesforce - we're seeing hit and miss connectivity across our customers' instances. Some API calls are still working, I'm unable to sign in to a developer instance in NA49.
it'd be nice if we could get the hn link pointing to that instead, since it's actually got a bit more details than the reddit post and the reddit post doesn't seem to be getting updated, so it's only going to get more "stale"
Not really what you want with something like this... but the folks holding the keys to a site like that are often never around / fast enough to make those sites helpful.
"The Salesforce Technology team is investigating an issue impacting Salesforce customers who use Pardot, or have used Pardot in the past. The deployment of a database script resulted in granting users broader data access than intended. To protect our customers, we have blocked access to all instances that contain impacted customers until we can complete the removal of the inadvertent permissions in the impacted customer orgs. As a result, customers who were not impacted may experience service disruption. In parallel, we are working to restore the original permissions as quickly as possible. Customers should continue to check Trust for updates."
For a time, users on many instances were able to read/modify data that they shouldn't have been. They got full CRUD access to -all data-. This includes some external users of things like Customer and Partner portals (where functionality and data are made available to external users via Salesforce). When they decided to try to mitigate the issue, they locked down all access and took away CRUD permissions from all users/profiles in Salesforce on those affected instances.
We woke up to a bunch of users unable to do their jobs because they suddenly started receiving "NO ACCESS" errors, effectively. We also haven't been able to modify the profiles and fix the access effectively.
When I saw this headline, my first reaction was that a salesperson in an org could download all customer contact info and immediately go to a competitor and start poaching customers extremely efficiently. How likely is this scenario? What sorts of recourse, legal or otherwise, would the org have? Non-competes are hard to enforce, and I don't know enough about trade secret laws to have a good opinion on this.
It's difficult to say. Exporting data in bulk isn't totally straightforward for your average user, and I'm not sure how long total CRUD access was granted. That being said, given that full access was granted, it's not impossible to imagine someone creating a report and doing a dump of customer data.
Though typically that may fall under NDA and not non-compete. NDAs are a little easier to enforce as far as I understand, but I'm not a lawyer.
It's sort of like if you walked by a bank and saw a bundle of money unattended in the lobby rather than the vault. Technically maybe you could get outside with the bundle, but keeping out of trouble long enough to enjoy the big screen TV you try to buy? ... Don't bet in it.
Nah, it’s more akin to an employ coming in one day and seeing that the vault door is just open, and thinking they might poke around a bit / pocket some bundles of cash...
Except they usually handle most of those bundles of cash day to day anyway.
If I understand it correctly, it's much worse: A customer of Company A could download all internal data of Company A (e.g. all customer info) if Company A is using a Salesforce based support ticket system and the customer had an account in there.
So if a sales person at Company B happens to be a Customer at Company A...
Presumably this should cause most companies in the EU to announce, within 72 hours, a data breach, since it allows any salesforce user to gain any permissions and view/leak/copy/steal any data on any customer in the org.
That will be a lot of companies considering pretty much every big company uses salesforce.
Does it count as a data breach if unauthorized users were allowed to access data, but they didn't actually? ie. server logs showed that nobody actually read anything they weren't supposed to.
Salesforce has a feature called Event Monitoring. Event monitoring does show lots of information, like pages access, reports exported, api endpoints used, etc. but it needs to be purchased. However, Salesforce does track and retain that data for some time so you can decide to purchase it retroactively if you ever want.
> "‘personal data breach’ means a breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data transmitted, stored or otherwise processed;
Based upon this definition just allowing the access is a breach.
Well it seems like a gray area. There's a meaningful difference between "accessing" something and "having access to" something. To me this seems like it would only cover the former.
> Presumably this should cause most companies in the EU to announce, within 72 hours, a data breach, since it allows any salesforce user to gain any permissions and view/leak/copy/steal any data on any customer in the org.
Presumably most users already have an NDA signed? Doesn't that cover PII?
Does this allow a salesman from company A to access customer info from company B (where A and B are both customers of salesforce) or is it just intra-organization?
Still bad if the latter, but catastrophic if the former
Moving off Salesforce is a many month project for a small company and possibly years for a larger company depending on the add-ons and everything.
Doubt they lose any real customers over this but they’re definitely going to be cutting some checks/credits to a ton of people for the next few months but it’ll fade.
95 comments
[ 4.6 ms ] story [ 112 ms ] threadI find it concerning that it was even possible for this to happen, regardless of whether it was intentional.
Also incorrect, they have great cloud and devops practices. If anything it's likely this bug's impact would be limited due to how decentralised SFDC operates.
Still a massive fuck-up, I'm interested in seeing if they'll release any more detail on why it happened.
whaaaaaaaaaaaaaaaaat hahahaha i can't take that seriously; I've used it and it felt like a giant pit of despair
Impossible to run locally.
No debuggers.
Virtually impossible to put an entire org in source control.
No package manager.
More undefined behavior than a C compiler.
they do own heroku
https://www.heroku.com/
SFDC hate is pretty common, maybe because of how big they are. I think that their tech is actually pretty impressive.
Like seriously I would rather lick alcohol soaked razor blades than do a standard annual renewal of a Salesforce contract.
That's the advantage of decentralized architecture. It's a disadvantage too though ...
I feel terrible for whoever initiated this. I’ve been in that boat and it -really- sucks.
"Ah yeah - it got in a bad state somehow, let me fix it manually"
Classic SQL blunder...* Change your account to readonly. Make a new admin account, and put its credentials somewhere hard to get (and audited!).
* Make a directory in git for 'one off sql statements'. Make them all go through code review and have an automated system run them on merge/deploy.
* Enforce style rules with a linter/test, like "UPDATE must have a LIMIT"
* Anything the above process is too burdensome to do should have an API or admin interface built for the purpose.
* Aim to eventually get rid of your readonly account. A leaked customer data dump could kill the company and shouldn't be available to any malware on your machine. You aren't as secure as you think you are.
I always recommend that you are wrapping calls in transactions if you are touching prod data. I only left the WHERE clause of my update query off once in my life before I started doing that.
BEGIN; UPDATE... SELECT for verification ROLLBACK;
And then just change ROLLBACK to COMMIT, after you're sattisfied. There's no need to do it interactively.
Do we have any example of any company going under due to a data breach? Unfortunately, there seem to be a lot of examples of enormous breaches that did essentially nothing. (like Experian)
https://www.theregister.co.uk/2014/06/18/code_spaces_destroy...
MyBizHomepage - once valued at $100m in 2008
https://www.nytimes.com/2012/08/23/business/smallbusiness/st...
but that fact I could find them at all by searching for
https://duckduckgo.com/?q=company+folded+after+breach
is evidence that it is rare
As someone who has never had access to a DB with any serious number of users, can you explain this one further? What if you really do want to update every row? Do you just do LIMIT INT_MAX or the like, and just force people to write that so that they always know they're updating the entire table? Or are you saying you should only ever use UPDATE on a known finite (and small?) number of rows?
Basically, if your table is over a few gigabytes, updating every row in one query on a production instance is a really bad plan.
BEGIN; Statement COMMIT; or ROLLBACK
etc
It should be
It is often worth running a SELECT on the WHERE clause you are about to use for your UPDATE. That way you can make sure only a limited amount of data comes back before you launch something catastrophic.
https://www.postgresql.org/docs/11/ddl-rowsecurity.html
Our product syncs data to Salesforce - we're seeing hit and miss connectivity across our customers' instances. Some API calls are still working, I'm unable to sign in to a developer instance in NA49.
https://status.salesforce.com/
Not really what you want with something like this... but the folks holding the keys to a site like that are often never around / fast enough to make those sites helpful.
We woke up to a bunch of users unable to do their jobs because they suddenly started receiving "NO ACCESS" errors, effectively. We also haven't been able to modify the profiles and fix the access effectively.
It'll be 0 - its more about the possible breach of PHI/HIPAA violations
It's difficult to say. Exporting data in bulk isn't totally straightforward for your average user, and I'm not sure how long total CRUD access was granted. That being said, given that full access was granted, it's not impossible to imagine someone creating a report and doing a dump of customer data.
Though typically that may fall under NDA and not non-compete. NDAs are a little easier to enforce as far as I understand, but I'm not a lawyer.
This issue did not open access to other companies’ data. Just all the data in their own Org.
So if a sales person at Company B happens to be a Customer at Company A...
our SF instance is accessible, but no permissions on login.
just got an update from our admin, no eta
That will be a lot of companies considering pretty much every big company uses salesforce.
It would absolutely be considered a breach if customer data was accessible by non authorized staff, moreso if by partners.
> "‘personal data breach’ means a breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data transmitted, stored or otherwise processed;
Based upon this definition just allowing the access is a breach.
Potentially, everyone that happens to share a instance with an org that has every used Pardot is affected.
Every org on an instance that had an org that used Pardot (huge percentage) had a service outage today.
The security breach was limited to the actual orgs (much smaller percentage) that have actually enabled Pardot in the past.
So, many (I’d guess half) of Salesforce's customers were affected with an outrage, but a much smaller percentage also had a possible data breech.
Presumably most users already have an NDA signed? Doesn't that cover PII?
Still, epic screw-up.
Still bad if the latter, but catastrophic if the former
Moving off Salesforce is a many month project for a small company and possibly years for a larger company depending on the add-ons and everything.
Doubt they lose any real customers over this but they’re definitely going to be cutting some checks/credits to a ton of people for the next few months but it’ll fade.