> We take the privacy of our customers’ data and personal information very seriously
I wish companies would actually mean this and apply it proactively instead of just copy and pasting this in their apology after something like this happens.
> We take the privacy of our customers’ data and personal information very seriously
Yet the first thing I'm greeted with is a cookie consent dialog with "allow all cookies" being the only prominent button.
If I don't want to allow the privacy-invading tracking cookies I need to click a less prominent piece of text called "manage cookies" and avoid clicking on another prominent button called "accept all cookies".
Firstly, this post does not even mention or describe what kind of information was exposed and makes it seem like some inadvertent tracking that's not serious in any way. You can continue with your root cause analysis, but you already know very well about what kind of information was exposed and shared with Facebook. The least you could do is own it and alert people about it.
> We take the privacy of our customers’ data and personal information very seriously...
It's astonishing that there is no apology of any kind on this post. Is that too much to ask for such a grievous violation?
> On March 8, 2021 at 8:39pm Pacific time, a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
It went unnoticed for two weeks, with Facebook gathering information about the customers' files! And yet, no apology!
>a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
"We take privacy very seriously" is a sign that they don't take your privacy seriously at all.
There will never be an apology. A corporate apology is an admission of guilt. An admission of guilt opens it up to lawsuits. Through its litigeous nature, the US has evolved a culture whereby apologies and human decency are antithetical to survival.
Yev from Backblaze here -> I apologize for this mistake. We've updated the post with additional information now that we have it and there's also an apology.
Yes, this is rather disappointing. It's probably also in breach of GDPR assuming backblaze haven't contacted customers directly, informing them about the nature and detail of leaked data (based on your use of backblaze on X, we see that the following filenames and Metadata was incorrectly handed over to a third party (Facebook)...).
Possibly. But Backblaze has an agreement with the user; and they have taken data they need / are justified in handling (filenames, sizes) and leaked them. Google docs wouldn't be liable if I stole your credit-card and put it in a Google doc.
It may be that fb is doing something illegal here too - eg the whole way tracking pixel work seems difficult to square with gdpr - but the tracking pixel/service looks a bit like the drill used to force a lock - another party (like BB) must deploy it. And arguably the purpose is not to "steal" data.
Yev from Backblaze here - with sincere apologies. After we removed the offending code on Sunday night we went into investigation mode and conducted a rca - we wanted to get some kind of blog post out there so that folks could see that we were working and investigating the issue. I will grant you that maybe our initial execution on that was lackluster, but we wanted to make sure we had a better handle on what we were talking about. We've since updated the blog with additional information, you can read those here: https://www.backblaze.com/blog/privacy-update-third-party-tr....
> We are also reviewing applicable third party code on the website.
This is very simple. On all dashboard and similar signed-in pages, there's exactly zero applicable third party code.
If you think otherwise you do not in any way take "the privacy of our customers’ data and personal information very seriously".
As such, you should be much more specific about what you mean with "we removed the offending code".
If you never intended for third party scripts to run on the dashboard page, well then you got some serious explaining to do as to how your deployment process let that slip.
We also subsequently removed Google Tag Manager from the private pages.
This is good. However, the question then becomes how on earth did Google Tag Manager get added to those pages to begin with?
Either you did not consider this a cause for concern, or it slipped past your code review. In either instance, how could I ever trust you with my data?
They literally sent file names and file sizes over to Facebook. That might very well include very personal or confidential data. The fact that they don't even mention this and make it sound like some random unimportant tracking happened without them noticing is completely ridiculous.
Yeah, like file names of say PDF's, files with unique identifiers that give insight to the users personal life, worse still profile names of PC backups etc. etc.
Backblaze is missing the part where you conduct an investigation, request assistance from Facebook, and determine what data was leaked to third parties.
> Our Engineering, Security, and Compliance/Privacy teams—as well as other staff—are continuing to investigate the cause and working on steps to help ensure this doesn’t happen again.
Silo the sales and marketing department away from the real products like the person in that other thread was saying. Sales people can't be trusted with private data.
Security in services is all about liability and risk rather than an absolute guarantee.
I've seen so much anger over this issue that it's left me confused. Questions like how they can ever be trusted now. They could never be trusted. If the information is really that important, then it should be encrypted before being passed to any other service. Companies will screw up, the question is how are you going to be on the hook for it. The great thing about services is that you can pass the blame the service than if you had dealt with it in-house.
That said, this has been an embarrassing display for Backblaze and I hope they redouble their efforts on infosecurity. But mistakes happen. If there's a pattern, then that's a different story.
Even the title suggests that it's just a heads up on some minor functionality. Not really what I would expect from a company that puts "Trusted Storage Cloud" in their <h1> and encountered an incident that completely eradicates that trust.
If you're in the EU and a backblaze user and consider that some PII may have been leaked to 3rd parties, you can contact your local GDPR watchdog (aka national "Data Protection Authority", or DPA).
I've started using rsync.net with the borg client (replacing b2 and rclone) and loving it so far. I'm also surprisingly finding borg/rsync.net is suiting my usecase (personal backups) better than b2 did which is a nice bonus. I am paying more but can't complain as the value I'm getting is still good.
Man, your distrust seems toxic to me. The simplest explanation is they goofed as they admitted, if they were malicious, they'd upload those file names from their server straight to FB's, not through your god damned browser.
And of course they only addressed it after they "got caught", they didn't know about their fuck up before that, if you want to sneer at them, sneer at them for not being careful enough to let this happen, not what you wrote.
I'm not trying to defend them, more like I want to protest against ungrounded casual insulting bashes like yours that seems way too freaking common nowadays. I do think they care about customer privacy, because it affects their income. (Admiteddly they were too casual about it, they were still partnering with Facebook...).
I wish I hadn't put my PII into the filenames of the files I was backing up.
myname_myaddress.txt
4445-3333-2222-1213-0121-354.my.credit.number
NW53342211A.my.national.insurance.number
facebook_password_is_letmein.txt
filename_are_now_storage.exe
I_vote_for_trump.doc
Now this is public I bet there's hundreds of facebook team members sifting through all of that log data to find this PII and manually transcribe it into my facebook account.
You are totally right what info could leak a files named:
ClinicWhereTheyReattachYourJunkWhenItFallsOff_procedureOverview.txt, ClinicWhereTheyReattachYourJunkWhenItFallsOff_invoice.pdf and ClinicWhereTheyReattachYourJunkWhenItFallsOff_WhatWentWrongWithYourJunkBeGratefulYouHadTheChanceToSayGoodbye.rtf ..
It's the old debate on the fact that filenames might be kinda sensitive.
I've been a happy Backblaze B2 user for the past few months, but this and another recent event leaves me feeling like I'll have to abandon their service (or at least mirror everything to S3 just-in-case) soon. Recent event being when GoDaddy suspended their user-facing B2 domain a month ago because of abuse requests that were apparently not handled correctly. That and the fact that they still don't have a user-facing status dashboard to communicate the status of their systems has me worried.
Yev from Backblaze here -> thanks for being a Backblaze customer. We've finished our root cause analysis and have updated that blog post with additional information. We also have moved domain registrars to make sure that behavior doesn't happen again. Sincere apologies for the issues you've experienced.
There are known mechanisms to prevent leaks/exploits of this sort by sending some additional headers[0], on most of the modern browsers. Apart from that, the cookies should be strict for such a service. I logged in to my friends account to find neither implemented.
They better act quick about "ensuring it doesn't happen" or I will distrust them completely.
oof, this blog post is even more cringy than their repeated replies on twitter "...we’ve looked into and verified the issue and have pushed out a fix. We will continue to investigate and will provide updates as we have them."
this is a perfect example of how NOT to handle a PR nightmare.
Oh well, that's it then. I guess I'll only use Backblaze through `rclone` and utilize client-side end-to-end encryption. And I might still not use Backblaze.
Looking at this submission, comments and its ranking here, I’m sure that nobody from Backblaze will comment here or on HN for some more time. This blog post makes it seem like an internal gag order has been imposed. Some hand wavy updates are what we should expect I suppose.
The post title is also dishonest in saying third party tracking and not adding details — this is Backblaze that intentionally and explicitly shared sensitive data with third party tracking. It’s not as if some third party tracked Backblaze’s paying users surreptitiously.
This response is actually helpful for the competition as far as the HN audience is concerned.
Edit: I’m really curious on the extent of coverage of this issue. I searched online and found only mjtsai.com and theregister.com covering this sensitive information leak. Are there any other major tech sites that have covered it so far?
I expect zero http requests to any Facebook servers when I visit anycorp dot com and that’s regardless of whether I’m a paying customer or logged in user.
If any actual data is transferred that’s even worse, but already having a connection to Facebook via a tracking pixel for “audience building” or similar is over the line already.
71 comments
[ 0.19 ms ] story [ 170 ms ] threadI wish companies would actually mean this and apply it proactively instead of just copy and pasting this in their apology after something like this happens.
> a new Facebook campaign was created that started firing a Facebook advertising pixel
I think they actually don't give any figment of a shit of their customers data if they are adding to the Facebook hoover of personal information.
Yet the first thing I'm greeted with is a cookie consent dialog with "allow all cookies" being the only prominent button.
If I don't want to allow the privacy-invading tracking cookies I need to click a less prominent piece of text called "manage cookies" and avoid clicking on another prominent button called "accept all cookies".
They do not give a shit about my privacy.
"Any company announcing that they take their users privacy seriously does not in any way."
So a name "Facebook law of privacy" ?
> We take the privacy of our customers’ data and personal information very seriously...
It's astonishing that there is no apology of any kind on this post. Is that too much to ask for such a grievous violation?
> On March 8, 2021 at 8:39pm Pacific time, a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
It went unnoticed for two weeks, with Facebook gathering information about the customers' files! And yet, no apology!
anyone know if that applies to the real world?
>a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
"We take privacy very seriously" is a sign that they don't take your privacy seriously at all.
It may be that fb is doing something illegal here too - eg the whole way tracking pixel work seems difficult to square with gdpr - but the tracking pixel/service looks a bit like the drill used to force a lock - another party (like BB) must deploy it. And arguably the purpose is not to "steal" data.
This is very simple. On all dashboard and similar signed-in pages, there's exactly zero applicable third party code.
If you think otherwise you do not in any way take "the privacy of our customers’ data and personal information very seriously".
As such, you should be much more specific about what you mean with "we removed the offending code".
If you never intended for third party scripts to run on the dashboard page, well then you got some serious explaining to do as to how your deployment process let that slip.
We also subsequently removed Google Tag Manager from the private pages.
This is good. However, the question then becomes how on earth did Google Tag Manager get added to those pages to begin with?
Either you did not consider this a cause for concern, or it slipped past your code review. In either instance, how could I ever trust you with my data?
That question needs a serious answer.
Why would I trust you this wont happen again?
HN discussion: https://news.ycombinator.com/item?id=26536019
Silo the sales and marketing department away from the real products like the person in that other thread was saying. Sales people can't be trusted with private data.
I've seen so much anger over this issue that it's left me confused. Questions like how they can ever be trusted now. They could never be trusted. If the information is really that important, then it should be encrypted before being passed to any other service. Companies will screw up, the question is how are you going to be on the hook for it. The great thing about services is that you can pass the blame the service than if you had dealt with it in-house.
That said, this has been an embarrassing display for Backblaze and I hope they redouble their efforts on infosecurity. But mistakes happen. If there's a pattern, then that's a different story.
See: https://ec.europa.eu/info/law/law-topic/data-protection/refo...
After reading / comparing prices, for my needs I ended up with:
- OVH Cloud Archive: 0.0023/GB for cold storage + incoming/outgoing prices. https://www.ovhcloud.com/en/public-cloud/prices/#storage
- iDrive: $ 69.50/5TB/year https://www.idrive.com/pricing
- Wasabi: $5.99 / 1TB/month: https://wasabi.com/cloud-storage-pricing/#cost-calc
The smug tone in this joke of a response to the issue proves this.
And of course they only addressed it after they "got caught", they didn't know about their fuck up before that, if you want to sneer at them, sneer at them for not being careful enough to let this happen, not what you wrote.
I'm not trying to defend them, more like I want to protest against ungrounded casual insulting bashes like yours that seems way too freaking common nowadays. I do think they care about customer privacy, because it affects their income. (Admiteddly they were too casual about it, they were still partnering with Facebook...).
That said, this way of doing things gives so much plausible deniability that it must at least occur to people.
We've been burned on so many fronts by so many companies that it really is a learned behaviour to toxically distrust when things like this happen.
It starts with politics. They're never held accountable for lying, evwn brazenly.
And yet, your blog/landing site wants me to opt into sharing my 'personal data' with 20 different 'advertisement partners'. Come on.
Asking for consent means that you don't violate their privacy since they agreed to the data being collected and shared.
Using dark patterns is of course not Ok but also a different topic.
myname_myaddress.txt 4445-3333-2222-1213-0121-354.my.credit.number NW53342211A.my.national.insurance.number facebook_password_is_letmein.txt filename_are_now_storage.exe I_vote_for_trump.doc Now this is public I bet there's hundreds of facebook team members sifting through all of that log data to find this PII and manually transcribe it into my facebook account.
It's the old debate on the fact that filenames might be kinda sensitive.
edit: On further thought, would this have triggered China's firewall?
DemocratVoter Registration.png
"We have added protections to prevent this error occurring again"
Those sentences do - not - appear.
They had time to write, post and update, and I am still not seeing any emails about it
They better act quick about "ensuring it doesn't happen" or I will distrust them completely.
[0]: Content-Security-Policy (setting at least connect-src) and X-XSS-Protection to start with. More here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Co...
this is a perfect example of how NOT to handle a PR nightmare.
A very disappointing article.
The post title is also dishonest in saying third party tracking and not adding details — this is Backblaze that intentionally and explicitly shared sensitive data with third party tracking. It’s not as if some third party tracked Backblaze’s paying users surreptitiously.
This response is actually helpful for the competition as far as the HN audience is concerned.
Edit: I’m really curious on the extent of coverage of this issue. I searched online and found only mjtsai.com and theregister.com covering this sensitive information leak. Are there any other major tech sites that have covered it so far?