rsync.net was posted here on HN a few days ago.
Also, tarsnap is another popular service.
Neither have the special additions that makes Backblaze so popular, but could be popular alternatives.
Keep in mind that their storage boxes only use RAID and do not provide any other kind of redundancy by default. If the machine goes up in flames, your data is gone.
From a quick look at pricing pages it looks like rsync.net is 5x as expensive as backblaze b2, and has a minimum of 400gb per month (it also looks like you might have to preallocate vs pay on demand). And tarsnap is 10x as expensive as rsync.net.
My guess the bulk of that price difference is due to economies of scale.
I don't have huge backup needs (around 500GB, doesn't grow much over time), so I just use duplicity and upload to AWS S3 with the "infrequent access" setting, and have the bucket auto-replicate to another region in another country. Costs me around $12/mo just for the data storage. The AWS calculator tells me if I had to retrieve all that data it would cost under $10 to do so.
Given it's all personal data that I can stand to be without for days if needed, I could probably use Glacier (or even Glacier Deep Archive) and pay less than a third of the cost (or less than a twelfth!), but the absolute dollar amount isn't high enough for me to go through the trouble of changing up my backup scripts.
Sure, if you're running a business and are constantly generating lots of new data that needs to be backed up, that gets more expensive, but I also would expect a business with that much data could easily afford to spend several orders of magnitude more than I do on backups.
I use Linode Object Storage (a S3-like service) with rclone to manage my backups. (my backups are only a few dozens of gb, I don't need something like duplicity at the moment)
So far I'm happy with it, pricing is $5/month per 250gb + 1To egress. And, most importantly, none of the AWS complexity overhead. I have enough of that at work.
One big difference is that with B2 you can't have append-only backups. (or maybe it's very non-obvious from the docs?) That's pretty much why I use rsync.net instead - I can configure a separate key for uploads. Otherwise what's to stop anyone lifting the B2 access key from the host with automatic backups configured and just deleting all old data?
So, looking at some potential things to switch to...
OVH has two interesting products here:
OVH cloud archive works with rsync, costs roughly half as much for storage as blackblaze, roughly the same for egress, but charges for ingress (at the same rate as egress).
OVH object store is s3 compatible (like blackblaze), charges roughly the same for bandwidth, 2x for storage.
Digitalocean has a blob store with the same pricing on bandwidth, 4x the pricing on storage, a minimum spend of $5/month on storage, but the first tb ($10) of bandwidth free.
Before you experience a major problem, you tend to guess what is good enough to mitigate it. Once you've experienced a major problem, you tend to ensure that exact failure mode never happens again.
I would be surprised if OVH had another fire in the future.
The only reason I use Backblaze instead of tarsnap is it is 62 times cheaper for the same amount of storage. tarsnap is dramatically overpriced if you ask me.
I get the feeling the reason for the Tarsnap price is simply a lack of scale and bootstrap nature of it. (i.e. there isn't a VC feeding money into hosting costs to drive the costs down to grow the userbase). The cost seems fair, but noncompetitive against other vendors on most easy to find quantitative measures (Cost, features, SLAs, locations, etc).
However I do think there is value in Tarsnap if security is really that important. Colin is really switched on, and I trust both the service, and if anything happened, he would deal with it quickly and professionally. If I had the kind of profile that meant I needed to protect myself against determined attackers, then tarsnap would be a no-brainer for me.
Backblaze is also bootstrapped, no VC investment except a small initial round. They’re likely investing in the business using the profits thrown off by their unlimited desktop storage offering.
And tarsnap possibly uses the most expensive tier of S3:
> the original version, which can survive the loss of 2 datacenters, not the "reduced redundancy" version which can only survive the loss of a single datacenter
These days, there's not just reduced redundancy, but also infrequent access, which seems better for backups…
First, we only offer SSH / TCP22 so the transit is encrypted.
Second, we have installed, and maintain on the server side tools like 'borg' and 'rclone'. So while you might just 'rsync' or 'sftp' your data to us (in which cases it would not be encrypted on our end) you can also use sophisticated, encrypted backup tools like (borg, duplicity, git-annex, rclone, restic, etc.)
If you choose a tool like that, rsync.net does not hold the encryption keys. The data appears to be random from our viewpoint.
Including the filenames seems to have been unintentional, looks like they were logging analytics to Facebook, probably an even (form submission) but uploaded the form html with contents.
But why they need to submit that to Facebook for paying users I don't understand. The only thing I can think of is excluding active users from advertising... But is that worth the privacy intrusion?
They answered in the Twitter thread: they send data about paying users to Facebook so they can build lookalike audience targeting for new user acquisition. Other major ad platforms (Google, LinkedIn) have similar features.
This doesn’t look like the right data to be sending FB for that, though.
Can someone here translate the PR/marketing speak here for us mere mortals? How does having Facebook tracking on the web front-end of existing and paying users help with lead generation?
That PR/marketing tweet comes across as from someone who doesn't understand how big deal this actually is and why customers won't be comfortable with a FB pixel on their dashboards with data filenames and sizes.
Thanks for the link, I didn't know that was a thing.
Having said that, this sounds like "Hey guess what? We are gonna snoop on you, and profile the hell out of you and leak all of your sensitive data all over the place (filenames can be) all because you are a paying customer"
That's about the worst way to disrespect a paying customer. Is there a way to easily identify companies that does this, I can avoid them?
No, they don't need to snoop on you. Just fire a single pixel saying this is a user that they would like to find more of. FB does the rest based on data in FB, not on Backblaze. The mistake was them just adding it to the online dashboard because that's the easy way to automatically include paying users.
> "easily identify companies that does this"
No. And this can happen for a lot of reasons as explained so even harder to check for.
Within Facebook, you can use the event stream collected by your FB Pixel to both define conversion criteria as well as create audiences and define inclusion/exclusion criteria for that audience. When it comes to tracking on pages behind auth, primarily it's for audience building which can be used for
– Cross-sell/Up-sell campaigns. Build an audience based on usage patterns, and create a create a campaign for a complimentary service or higher tier (say, for example, someone clicks the button for a gated feature they don't have access to).
– Suppression lists. If you don't want your campaigns to target existing users, you can build an audience from pixel data on your authenticated pages and suppress against that.
– Lookalike audiences. After you create an audience in Facebook, you can create a "lookalike audience" from that. So even if you aren't actively doing either of the above, you'd derive value from tracking your "best" customers and using it as a seed list for a lookalike audience.
You're also not limited to using the FB Pixel for any of the above. In addition to a browser-side pixel, FB allows you to upload hashed customer information and use those for conversion tracking and audience building. Which used to be completely transparent to end users, but now you're able to see a list of companies that have uploaded your info to FB in this manner (I can't recall where it's buried in the user settings, off the top of my head).
All of that said, it's entirely likely that Backblaze wasn't intentionally sending any of this data to FB to begin with. An insidious aspect of FB's Pixel is that it automatically attaches listeners to a bunch of stuff on the page such as buttons and sends back interactions and associated metadata[1]. The flag to disable this isn't mentioned in the implementation instructions that are generated upfront, and it's actually a fairly uncommon trait for ad pixels. So a typical implementation tends to leave it on out of ignorance rather than make a deliberate determination on whether to use or disable that functionality.
"An insidious aspect of FB's Pixel is that it automatically attaches listeners to a bunch of stuff on the page such as buttons and sends back interactions and associated metadata[1]. "
Um. Holy crap. Is this common knowledge? I don't get shocked easily these days but... wow
I'm not really sure if it's common knowledge, but I'd say it's less common then it should be. I went ahead and made a top level comment[1] with some info about controlling data disclosure to FB's Pixel in case it's helpful to others.
In the analytics space, you traditionally had to:
– Initialize a tracking object on page load
– Explicitly call methods on that tracking object when you wanted to actually send a hit
This is how it works for Adobe Analytics and Google Analytics historically. Many of the newer analytics providers instantiate auto-listeners, which gave them an edge on the out-of-the-box analytics features. And Google Analytics 4 (the newest release), also does this.
So it's not unheard of for site analytics. And a quick glance at a particular provider's website can usually make it obvious if this is occurring, based on the advertised features.
Ad pixels tend to be different though. You create a conversion event within the ad platform, and you're given a snippet of code to fire when that specific event occurs, which both instantiates the tracking object and calls the tracking method with the conversion event's configuration details.
Facebook's pixel works far more like a modern analytics library than an ad pixel. It vacuums up the hit data from the site and the marketer is able to sort it out after the fact in Facebook's interface and use what they want from it. Marketers working within Facebook can see this is happening because they set up the conversions and audiences against the hit data, but that's "just the way things work" in Facebook so they think nothing of it. Marketers coming from other channels will notice how different it is, but won't realize what's actually happening nor the implications behind it. Devs would realize pretty quickly what's happening after a few minutes exposure to the FB Pixel interface and it'd trigger a red flag for them, but that's marketing's territory and all devs see are the snippets provided for implementation. So the only time most people become aware of it is if marketing has someone technical working directly within FB's interface or if the person tasked with implementation has a reason to dig into Facebook's dev documentation rather than just plop the snippet on the page like they were told.
Some might find this tweet useful for the time being:
> What happens if you resolve http://facebook.com to 127.0.0.1 via hosts-file?
(Or put it into your Pi-Hole DNS Ad-Blocker, or the like.) Does the Backblaze UI still work?
> Answer: Seems to work well. You need to add the main domain and sub domains (www. in this case), btw.
Signing up for Wasabi resulted in a deluge of unwanted marketing spam, all written to be "cheerful and your friend"
Also, huge warning for Wasabi newbies so you are not surprised. This might wreck your wallet.
Your egress is capped at your total amount stored. You cannot store 5TB and have 6TB of downloads against that account. The front page is covered in 'No Charges For Egress' 'no additional charges for egress or API requests'.. etc. This is not written anywhere on the main marketing pages, only behind a small link.
Another important point is file deletion. You are required to pay for multiple billing cycles (months) of files. Be very cautious about this. Using it as a temporary bucket will incur significant costs, if you upload a file, YOU WILL PAY FOR 90 DAYS OF THAT FILE. It is almost certainly not cost effective to store files in Wasabi any less than long term. Deleting a file that you just uploaded will incur a deletion fee equivalent to 3 months of storing that file.
One thing people should bear in mind when looking at Wasabi is that they have a 90 day minimum retention period for their PAYG plans. So if you upload a large file and overwrite it tomorrow, you'll be charged the storage fee for the day plus the storage fee for the remaining 89 days, and then the storage fee for the new file, and so on.
The same goes for AWS IA and Glacier classes and Google's Nearline and Coldline. Unless you're storing files for a long time, always factor in the minimum retention period before estimating costs. It'll prevent any nasty billing surprises -- speaking from experience unfortunately.
Why not just use client side encryption and be done with it? Why would anyone browse their personal files remotely on a web ui? If you care about privacy, take the proper precautions.
There really isn't any competitor to B2 on price, or convenience (e.g. them sending you a NAS device for recovery).
They could maybe salvage goodwill with genuine corporate soul-searching that ends up asserting/reasserting values -- and leads them to focus on providing trustworthy service to their users, and conspicuously away from some "tech" industry norms of selling out one's users.
As a provider of a paid service, it seems like they're in a better position to take the high road than a lot of tech companies are, but they have to decide that's who they are, and be clear they mean it.
Imagine if S3 did this. I don't like Amazon, but at least they are security professionals.
Ad tracking pixels in your object store dashboard is just clownshoes from a security engineering standpoint, over and above the fact that it's a slimy, dickhead move for a paid service.
A dumb mistake that leaks private information to a third party seems like good reason not to trust or use their service. For me, the idealogue, the fact they're sending any tracking data from the dash to Facebook is enough reason.
The thing is, even if it was a dumb mistake, it's one of those mistakes a company like Backblaze can't afford to make.
If they don't pay attention to stuff like this, then why should I trust them with anything at all? This isn't some minor oopsie, this is failing to deliver on their core product[1]:
Top Backblaze B2 Use Case Solutions
Backup & Archive
Store securely to the cloud incl. safeguarding data on VMs, servers, NAS, and computers
Backblaze is not led by dumb people. This was a conscious decision that they made and the got caught. I used to recommend them to people and I never will again.
It's possible to manage the chrome browser centrally via g suite, and push out DoH settings. I use this in conjunction with NextDNS to put all browsers organization-wide onto a single blocklist, and block the common tracker hosts (GA, GTM, Facebook's domains, et c).
I know that many computer systems have to be certified or compliant with a spec to be used (HIPAA). Is there a possibility that such data being sent across the wire to a 3rd party would break such compliance?
I'm a long term B2 customer and left FB a long time ago and block trackers.
The major point to me is: If marketing at Backblaze drives decisions that influence security and tech has either no say in this or is not competent enough, it's not a company for me to trust my data with.
I doubt it’s intentional, but companies should seriously consider if the benefits of integrating things like Facebook Analytics tools outweighs the negatives. It seems like considering their audience they would not use Facebook of all things.
I operate quite a few apps and recently launched a website too which handles a lot of sensitive user data. I decided to make - not having any analytics, trackers and ads the selling points of my apps and sites. Get a lot of positive emails from customers thanking me for that. I was recently even wondering if Google penalizes sites for not having putting their analytics on the sites/apps from appearing on the search results.
I legit don't understand why a paid storage service would put a FB pixel on their dashboard which handles user files. It's a completely foreign concept to me. This seems like a screw up but also erodes a lot of trust which is unfortunate as I had been looking at them for past 2 months actually.
I even made a post just yesterday and another couple weeks ago on how BackBlaze's inability to set a specific file name, file size limit and expiry date on the pre-signed urls is preventing some of us from switching over from S3 to Backblaze for our storage of app data needs. And surprisingly, I wasn't the only one as I got a few people responding with same concern.
> A limitation I ran across when using B2 was that their pre-defined url generation doesn't allow you to set file-size limits nor does it allow you to set the file name in the pre-defined url. It simply gives you a pod url to upload it to. So if you are using b2 for storage for lets say image uploads from browser, some malicious user has the ability to modify the network request with whatever file name or file size they want. Next thing you know, you have a 5gb sized image uploads happening.... This pretty much prevents me from using B2 for now.
> I ran into the same limitation! IIRC, there also wasn't a way to expire a signed upload URL sooner than whatever the default was, which was hours or maybe a day. I had the exact use case you mentioned, too - image uploads bypassing my backend server. I didn't want the generation of a signed url to, say, upload a profile photo, give carte blanche to create a hidden image host when combined with the limitation that you highlighted. All sorts of bad things could come of that. I ended up just going back to S3 - costs more, but still worth it.
Since this is for a site/app which lets users upload data, I am really trying to avoid S3 due to crazy costs. I might look into DigitalOcean's offerings. Anyone have any other recommendations?
> Since this is for a site/app which lets users upload data, I am really trying to avoid S3 due to crazy costs. I might look into DigitalOcean's offerings. Anyone have any other recommendations?
Dumb suggestion: Run it yourself? Minio is easy to use, even in multi-server mode.
That's one of the things I was looking into too actually. Though if I want to go with Minio, I would also want to do that on my own physical servers instead of cloud to completely cut out third parties. Do you have experience in that? I guess I would need a proper dedicated high speed internet for it too for any decent traffic?
Sorry, I've never needed to use it like that (and, full disclosure, I've never run minio in prod either); I can recommend Hetzner dedicated servers because they don't bill for egress but that's it
Did you read anything more than the incredibly poorly worded headline?
The data was harvested by the Facebook pixel as part of their audience building tech for acquisition of new customers. So you are being sold by Facebook, Backblaze just happens to have been quite careless here but they are not "selling your data".
The breach of trust is BB letting third party code on their platform and especially from a particularly untrustworthy third party. That's it, it's egregious and should be taken seriously, that also means discussing it seriously.
This kind of hyperbole is counter-productive, it only makes it easier to ignore your concerns as "crazy overreacting".
I appreciate that their Twitter account is looking into it, but feel like this should be a pretty quick fix. I'm hesitant to even log into my Backblaze account again until it's sorted.
In a sense, that makes life easier for me. One less alternative to consider. The name "Blackblaze" is burned forever. You will never win me as a customer. I write this here, because people in similar positions at similar companies might read this. And I do not think I am alone.
Been a huge fan of Backblaze for years and b2 was my plan for server backup; always seemed like a great underdog company in my eyes- they just lost me. I already canceled my Spotify for a similar —Facebook-related— reason.
Probably because Spotify is using the Facebook SDK in all their apps. As a user you cannot disable it and a few years back the Facebook SDK caused an outage where you couldn't start Spotify (and many other major apps).
backblaze signs BAA agreements with companies storing personally identifiable medical information, I wouldn't believe anyone who told me that this facebook data leak was turned off for those customers; they should immediately be investigated and fined if any such breaches indeed happened.
Full agreement, any interest and goodwill towards the company is now completely gone.
Unfortunately HIPPA is actually specific in this scenario. I highly doubt file sizes have to do with PHI. However if you can show that they are sending the actual data in the backup, that would be a reportable offense.
Disclaimer: I am not a lawyer, this is not legal advice, YMMV.
One possible, slight, softening factor - if it's an accident/bug, I might consider them, if they can explain exactly how it happened and what extreme measures they'll take to ensure that it never happens again. But... their initial answer on that thread is
> Believe that's the Facebook pixel we use for tracking, we've forwarded to our web team for review in case that is not intended behavior.
Just... "in case" it's unintended is not promising.
And this is a smalls subset of filenames that could not only provide PII but also potentially embarrassing or private information that isn't identifying but would be accompanied with files that are personally identifying.
I’ve worked for multiple FAANGs and can assure you that all have considered file names to be PII. Pretty much anything that contains user input should be treated as PII.
Lawfirms often utilize automatic content-derived filenames, up to the maximum OS-supported limit. As a result you'll find all sorts of private information in the filenames within their backups.
I should also mention, those automagic naming schemes use the beginning text in the document. So it's basically the personal details of plaintiff vs. defendant.
I've also seen similar schemes used by doctor offices/hospitals, where you'll see the patient name and ailment. I once had to troubleshoot a backup problem for what turned out to obviously be an OB-GYN. Imagine my horror as I saw long filenames containing patient names and common STDs scroll by in the thousands.
No, that’s neither true nor “technically correct”. Services that the user provides health information to for their own purposes aren’t considered covered entities or their business associates, and thus HIPAA rules don’t apply. This is why Dropbox, and GMail don’t have to be HIPAA compliant.
If you say “Read the law strictly enough” please do...
You might be right, I believed I saw some definition that simply stated that "system receiving or storing PHI" would be required to be HIPAA complaint, regardless of how the data got there.
I'm still wondering, because Backblaze will sign a BAA (their website says so), making them a business associate. I'm not talking about some private person uploading their own documents. My concern is that given that Backblaze will sign a BAA, then some companies must be using Backblaze and potentially storing PHI data there. Yes?
Backblaze then need to follow: "§ 164.312 Technical safeguards. (e)(1) Standard: Transmission security. Implement technical security measures to guard against unauthorized access to electronic protected health information that is being transmitted over an electronic communications network."
Facebook isn't authorized to access this data, but that might be more of a problem for Backblaze, even if Facebook could be required to delete the data.
Yeah, that seems like it would be on Backblaze, but this might not apply here, as 164.312 only applies to PHI, and it's quite likely that Backblaze will only sign a BAA for their B2 service, and not their standard backup service.
Just from looking at my own documents folder, "Stephen Tordoff - ESA Appeal SSCS1.pdf". That would reveal that I have a disability, and that I am (or have) claiming benefits for it. Not everyone would be happy with that being public knowledge, and I'd be less that thrilled if things like it were shared with Facebook for no reason.
I'm in engineering of a financial services. When we built our front-end UI for eKYC our marketing requested for google tag manager / facebook pixel and various other tracking features to be built.
I had to fight hard as an engineer to make sure that it does not happen. We had meetings after meetings, and it took a lot of effort for me to explain the risk of data leakage. I was questioned on my "insecurity" for not "trusting" people. It was not a nice experience. I had to inform them that tracking needs to be dealt with properly, not just lazily install google tag manager because it gives marketing 'flexibility'.
The only thing I knew about 'tag manager' before this was that it was always blocked by NoScript. Your comment made me go look up what it does and now I know that I will never unblock it.
Apparently it lets people drop in random code from a bunch of different analytics platforms, so it's pretty much guaranteed to consist entirely of the sort of stuff I have NoScript enabled to block in the first place.
I've seen GTM take down production multiple times because of marketing shipping random JS with no approvals.
Some random guy in his basement assured someone in marketing they could handle our volume? Chuck their tag in and watch their website get DDoS'ed with millions of requests per minute, which takes out our website because marketing made it fail loudly.
I'm surprised that GTM doesn't handle that. They would have a good idea of how long requests are taking to different domains and limit requests to the slower ones.
I mean, yes, true, but it kind of misses the point: marketing doesn't ask you, they ask up. And we're just the BOFH pinheads who make everything so harrrrrd with our stupid "concerns."
IT can often be "we make someone else's bad idea happen," and that's because we simply lack veto power.
The reason for that is I'm the CTO, responsible for building out the tech / engineering team.
Hiring people is difficult as there is lack of supply of talented people. We hire InfoSec on contract basis not full-time and they don't join such meetings due to the nature of the contract. So all the responsibility fell on me to defend our technical decisions at that point in time.
I'm working on building out the engineering culture / awareness within management now, to ensure these things do not happen, and I don't have to be questioned as to why we cannot install "google tag manager" in our front-end.
It all comes down to creating awareness, and making people understand. Fortunately for me our CEO gets it, he ended up siding with me.
Honest question: how can a financial services company not have an in-house infoSec team?
To me, this is an even more concerning issue. But then, I have no idea how the finance services world works, so maybe this is more common than I think?
FinTech doesn't always mean global mega bank. There's lots of small scale start-ups that fit into the financial services category that wouldn't/couldn't afford full time InfoSec roles.
Outsourced CISO/InfoSec is a valid and reasonable thing for some companies.
I feel like a small scale startup needs internal infosec and audit teams even more. Unlike the incumbents, who are "too big to fail" and therefore are able to get away with blatant insecurity, a startup's in a much more vulnerable position, and any security breach is significantly riskier in terms of corporate longevity.
If I was running a financial services startup, those groups would be near the front of my list in terms of internal hiring.
There are huge repercussions. We are governed by PDPA (our GDPR equivalent) law where penalties are extreme. Thailand takes Data Privacy as seriously as EU.
But again, since it's not always easy to find the right people I end up having to fill in for everything we don't have a team member to execute on.
Worth sharing this thread around your company, maybe?
Could help with convincing people that there are negative marketing consequences to careless and seemingly harmless decisions.
No has nothing to do with tax. Usually contractors have very fixed scope, they focus on doing what is in the contract. Sometimes things that come up ad-hoc like marketing requesting installation of Google tag Manager, is outside the scope of the contract. It would require a lot of giving them context, amending contract etc... It's not necessarily convenient to have to ask them to come in every time there is a problem.
Usually I try to reason with management first. If it can be resolved internally we would not include outside consultants, however if it gets serious beyond something we can handle internally we would ask outside consultant to come in.
LOL, worked at a place and we uploaded all of the inventory for Facebook integration. We charge millions to customers for this data but "we need to show up on Facebook" wins the cake.
That's par for the course for technically orientated roles. I was labelled 'defensive' and denied a pay increase because a business development executive wanted to make our documentation dynamic based on user access, and I pointed out it was difficult to find a solution when users can have over 400 access permissions, which varied depending on country, and our documentation was 900 HTML pages, some of which were equavlent to 200 A4 pages.
If you are the most knowledgeable person then you get blamed for their bullshit fantasies being impossible or unwise (or illegal)
Sure, in the end I asked our head of compliance to join the meeting. I made it very clear to everyone what was at stake, and that I’ve done my duty in raising awareness. If they would like to proceed I hold 0 responsibility. Usually when you do it like that no one wants to put their neck on the line. Our head of compliance take this stuff really seriously as he has to report to central bank, so him and our CEO ended up agreeing to not use GTM.
You really need to present well and be careful about arguments like “millions of websites use GTM”. I did days of research and presented that while yes using GTM on Wordpress sites that hold no sensitive data might be fine however we are a financial services and we collect customer private data. So getting everyone on the same page and presenting alternative ways of solving the problem was critical.
I was questioned on my "insecurity" for not "trusting" people.
"I trust people just fine. I trust people working for outside companies dependent on information gathering to gather information. Google has no fiduciary duty to our clients. Same with Facebook. We DO have a fiduciary duty to our clients and it includes not doing things that may send their confidential information to third parties because SOME of the information used may be useful to our marketing department."
It's scary to think that a company that seem to have a decent policy on privacy / data collection practices at one moment is just one step away from some marketing manager or MBA changing that. It's really hard to gain customer trust once you loose it, and in BackBlaze's case it seem to be for marginal if any monetary benefit.
I think part of the reason is that most of these companies don't value customer trust.
It's a lot more than just the person in marketing not realizing. You can't expect marketing to understand these things.
It means there's either nobody reviewing the privacy implications of marketing decisions, or that somebody who knows better is reviewing these decisions and decided leaking data like this is acceptable.
Both those possibilities make backblaze a non-starter for me now.
Exactly this. If this was possible, nobody said that this is a bad bad idea or did say but got overruled it makes me think this product is no longer safe. I'll give some time for them to publish post mortem and in the meantime will look where to move data.
> You can't expect marketing to understand these things.
I work in marketing ops and I know how difficult it is to get marketing folks to understand or care about how the tracking they use works. There’s a small but growing number of us trying to change behaviour and awareness from the inside out but if marketers fuck up on privacy an example should be made of them.
It's been a problem almost everywhere I've worked. Marketing / business folks just want their analytics, and they don't want to spend any money on it (e.g. they don't want to pay for some privacy-respecting 3rd party service, and they don't want to pay devops to stand up an internal OSS alternative).
Unless you have really strong dev leadership, the trackers will end up in your product.
I'd love to hear from folks who have successfully blocked their business team on this. What tactics did you use?
Seems like a poor decision / dumb mistake. Fix it and move on. I can understand the impetus to need FB ads & lookalike audiences to grow the company -- it serves no one if BackBlaze goes bust. It's obviously wrong to send filenames to FB as well.
If they fix this, goodwill will be burned, but it's not a dealbreaker for using them. If they don't fix it, then yes by all means leave the service.
Just an FYI in case this isn't clear to people. Even though this is a serious problem, it's super unlikely that FB is using the filenames in any way, or aware they were being sent. That part is probably just a (really awful) mistake.
This is the same company that has published iOS apps that have silently escaped the sandbox. I think pathological is probably an appropriate word for them and wouldn't trust anything remotely related to them with a single bit of personal data.
Possibly, but company culture transcends the immediate team. And if your business model is based on data siphoned off every possible channel, the concern is very justified.
Facebook, like other GDPR compliant companies, deletes most data after 90 days by default. My guess is that this data was not intentionally stored, and any place where it was will be deleted manually within a few days as part of remediation for this issue. Even if that doesn't happen, it would automatically disappear after 90 days.
> Facebook, like other GDPR compliant companies, ...
Isn't there a lawsuit or similar going through the EU courts atm about Facebook not really being GDPR compliant? eg They claim they are, but the court case is about them not being.
super unlikely that FB is using the filenames in any way
First, we don't know that. Second, it doesn't matter even if they are not using it in any way. Backblaze shouldn't be sending this data to FB. And lastly, even if FB isn't using it in any way today, how do we know they won't use in future?
Nobody should be having access to any data that they don't absolutely need. It doesn't matter how mundane the data is, which company the data is going to, etc etc.
Like I said, it's definitely a problem regardless.
But this is like saying it that if you misplace your cell phone, it doesn't matter whether you left it in your friend's car or a taxi. Of course it matters whether or not data is being (mis)used.
More so with all the machine learning code being deployed to gather better information about people to thrust more ads on them, it's possible that even those who work at Facebook may probably not know the extent to which all this data is funneled into various systems and attempts are made to extract monetary value from it.
Yes but but if marketing drives their security decisions, where else do they leak? What data do they sell? If they play lose with security, in what areas do they also play lose with your security? With your login email? Payment data? Do they sell your credit history? How do they vet their employees?
It is just lazy engineering. They want lookalikes. Fine. Collect, clean, mask the data in house and send it to FB. Dumpling to a 3rd party data directly from page is irresponsible to say the least as such "Opsies" are bound to happen.
they can’t do that. they don’t have the original data (it is owned by fb) to create the lookalike audience. methinks you don’t understand how these tracking doohickeys work
FB allows a custom audience to be built from identifiers like email, phone, etc. They claim it is hashed before use so they never see the raw data (not sure if they mean hashed client side or server side though).
This will rely on a user's FB account having the same email as used for BB, which could be unlikely in the case a company is paying for it. But it should work well enough for retail targeting.
All they say is they don't learn anything new about your customers. I would take that statement with a lot of caution. For one, they should know that I am a bb customer now. Having a graph of all the companies and products I use is huge PI.
> They claim it is hashed before use so they never see the raw data
If they can match hash data with real data, they can know more than they did before. Depending on what algorithm they use for hashing (no mention of it), they could be using a similarity hash so that will minimize changes if there are minor differences in the dataset.
Let's say I find a profile through comparing hash of email to email in Facebook's database. I can then compare additional information to see if a customer has provided incorrect information to Facebook as a user. Facebook could check if my address is similar to online shopping sites I use and if not, flag the account.
Yeah there is a lot of room to hide in the gaps of what they said.
Still, main point is you only need an identifier and none of the other data Facebook has. Pixels are not required for this as noted in the original comment, they probably have enough in the account details already.
> they don’t have the original data (it is owned by fb) to create the lookalike audience.
they have the original data of their own audience. So they can send it explicitly to FB (instead of FB sucking it from the page's pixel so to speak) for FB to build the lookalike audience which FB would do using the wide FB owned data.
the data they hold on their audience is irrelevant here. eg BB not collecting age and gender, and they don’t have the web history of the user (that fb pixel and thumbs icon enables). the lookalike audience is based on FB profile.
A have a question about creating lookalike audiences, by sending data to FB (either separate or through pixel tracking). Is that data not by definition PII, and so they are likely violating GDPR doing this?
the data sent by a pixel is not PII. it enables lookup of the existing PII. the company putting the pixel on their page isn’t collecting PII and this is outside of GDPR. the accidental transmission of file names doesn’t seem to be PII to me.
Wow they pretty much flushed all user trust and good will down the toilet with this. I had considered using them in the past but I sure won't be considering them now. Even if this were an accident how are there not procedures in place that would have required approval and vetting?
Tag manager is used for managing all of the various tracking and analytics snippets for a site in a central location, so if your ad blocker stops tag manager itself, you often preempt many other things that would run. In this case I’d assume the facebook pixel was installed using tag manager.
538 comments
[ 3.3 ms ] story [ 346 ms ] threadhttps://twitter.com/backblaze/status/1373751015594356739
Not sure if they intended to send file names and sizes to FB, but in any case this doesn't look good. I'm currently looking for alternatives.
My guess the bulk of that price difference is due to economies of scale.
Given it's all personal data that I can stand to be without for days if needed, I could probably use Glacier (or even Glacier Deep Archive) and pay less than a third of the cost (or less than a twelfth!), but the absolute dollar amount isn't high enough for me to go through the trouble of changing up my backup scripts.
Sure, if you're running a business and are constantly generating lots of new data that needs to be backed up, that gets more expensive, but I also would expect a business with that much data could easily afford to spend several orders of magnitude more than I do on backups.
So far I'm happy with it, pricing is $5/month per 250gb + 1To egress. And, most importantly, none of the AWS complexity overhead. I have enough of that at work.
Technically you do have to pre-allocate, but it works like pay-on-demand because:
<paste>
We can increase the size of your filesystem at any time, with no downtime involved - so there is no need to overbuy in anticipation of future usage.
Further, you are granted a +10% grace at all times, so you always have some room to grow.
Finally, the system automatically emails you as you get close to your limit, so there is never an unexpected filling of space.
</paste>
OVH has two interesting products here:
OVH cloud archive works with rsync, costs roughly half as much for storage as blackblaze, roughly the same for egress, but charges for ingress (at the same rate as egress).
OVH object store is s3 compatible (like blackblaze), charges roughly the same for bandwidth, 2x for storage.
https://www.ovhcloud.com/en/public-cloud/prices/#439
Digitalocean has a blob store with the same pricing on bandwidth, 4x the pricing on storage, a minimum spend of $5/month on storage, but the first tb ($10) of bandwidth free.
https://www.digitalocean.com/pricing#spaces-object-storage
I would be surprised if OVH had another fire in the future.
seems this the cheapest of all
However I do think there is value in Tarsnap if security is really that important. Colin is really switched on, and I trust both the service, and if anything happened, he would deal with it quickly and professionally. If I had the kind of profile that meant I needed to protect myself against determined attackers, then tarsnap would be a no-brainer for me.
> the original version, which can survive the loss of 2 datacenters, not the "reduced redundancy" version which can only survive the loss of a single datacenter
These days, there's not just reduced redundancy, but also infrequent access, which seems better for backups…
First, we only offer SSH / TCP22 so the transit is encrypted.
Second, we have installed, and maintain on the server side tools like 'borg' and 'rclone'. So while you might just 'rsync' or 'sftp' your data to us (in which cases it would not be encrypted on our end) you can also use sophisticated, encrypted backup tools like (borg, duplicity, git-annex, rclone, restic, etc.)
If you choose a tool like that, rsync.net does not hold the encryption keys. The data appears to be random from our viewpoint.
But why they need to submit that to Facebook for paying users I don't understand. The only thing I can think of is excluding active users from advertising... But is that worth the privacy intrusion?
This doesn’t look like the right data to be sending FB for that, though.
Good job, backblaze. So many years of work building goodwill with transparency in hard disk reports ruined completely with one stroke.
[1] https://www.facebook.com/business/help/164749007013531?id=40...
Having said that, this sounds like "Hey guess what? We are gonna snoop on you, and profile the hell out of you and leak all of your sensitive data all over the place (filenames can be) all because you are a paying customer"
That's about the worst way to disrespect a paying customer. Is there a way to easily identify companies that does this, I can avoid them?
> "easily identify companies that does this"
No. And this can happen for a lot of reasons as explained so even harder to check for.
– Cross-sell/Up-sell campaigns. Build an audience based on usage patterns, and create a create a campaign for a complimentary service or higher tier (say, for example, someone clicks the button for a gated feature they don't have access to).
– Suppression lists. If you don't want your campaigns to target existing users, you can build an audience from pixel data on your authenticated pages and suppress against that.
– Lookalike audiences. After you create an audience in Facebook, you can create a "lookalike audience" from that. So even if you aren't actively doing either of the above, you'd derive value from tracking your "best" customers and using it as a seed list for a lookalike audience.
You're also not limited to using the FB Pixel for any of the above. In addition to a browser-side pixel, FB allows you to upload hashed customer information and use those for conversion tracking and audience building. Which used to be completely transparent to end users, but now you're able to see a list of companies that have uploaded your info to FB in this manner (I can't recall where it's buried in the user settings, off the top of my head).
All of that said, it's entirely likely that Backblaze wasn't intentionally sending any of this data to FB to begin with. An insidious aspect of FB's Pixel is that it automatically attaches listeners to a bunch of stuff on the page such as buttons and sends back interactions and associated metadata[1]. The flag to disable this isn't mentioned in the implementation instructions that are generated upfront, and it's actually a fairly uncommon trait for ad pixels. So a typical implementation tends to leave it on out of ignorance rather than make a deliberate determination on whether to use or disable that functionality.
[1] https://developers.facebook.com/docs/facebook-pixel/advanced...
Um. Holy crap. Is this common knowledge? I don't get shocked easily these days but... wow
In the analytics space, you traditionally had to: – Initialize a tracking object on page load – Explicitly call methods on that tracking object when you wanted to actually send a hit
This is how it works for Adobe Analytics and Google Analytics historically. Many of the newer analytics providers instantiate auto-listeners, which gave them an edge on the out-of-the-box analytics features. And Google Analytics 4 (the newest release), also does this.
So it's not unheard of for site analytics. And a quick glance at a particular provider's website can usually make it obvious if this is occurring, based on the advertised features.
Ad pixels tend to be different though. You create a conversion event within the ad platform, and you're given a snippet of code to fire when that specific event occurs, which both instantiates the tracking object and calls the tracking method with the conversion event's configuration details.
Facebook's pixel works far more like a modern analytics library than an ad pixel. It vacuums up the hit data from the site and the marketer is able to sort it out after the fact in Facebook's interface and use what they want from it. Marketers working within Facebook can see this is happening because they set up the conversions and audiences against the hit data, but that's "just the way things work" in Facebook so they think nothing of it. Marketers coming from other channels will notice how different it is, but won't realize what's actually happening nor the implications behind it. Devs would realize pretty quickly what's happening after a few minutes exposure to the FB Pixel interface and it'd trigger a red flag for them, but that's marketing's territory and all devs see are the snippets provided for implementation. So the only time most people become aware of it is if marketing has someone technical working directly within FB's interface or if the person tasked with implementation has a reason to dig into Facebook's dev documentation rather than just plop the snippet on the page like they were told.
[1] https://news.ycombinator.com/item?id=26537140
> What happens if you resolve http://facebook.com to 127.0.0.1 via hosts-file? (Or put it into your Pi-Hole DNS Ad-Blocker, or the like.) Does the Backblaze UI still work?
> Answer: Seems to work well. You need to add the main domain and sub domains (www. in this case), btw.
Also, huge warning for Wasabi newbies so you are not surprised. This might wreck your wallet.
Your egress is capped at your total amount stored. You cannot store 5TB and have 6TB of downloads against that account. The front page is covered in 'No Charges For Egress' 'no additional charges for egress or API requests'.. etc. This is not written anywhere on the main marketing pages, only behind a small link.
Another important point is file deletion. You are required to pay for multiple billing cycles (months) of files. Be very cautious about this. Using it as a temporary bucket will incur significant costs, if you upload a file, YOU WILL PAY FOR 90 DAYS OF THAT FILE. It is almost certainly not cost effective to store files in Wasabi any less than long term. Deleting a file that you just uploaded will incur a deletion fee equivalent to 3 months of storing that file.
The same goes for AWS IA and Glacier classes and Google's Nearline and Coldline. Unless you're storing files for a long time, always factor in the minimum retention period before estimating costs. It'll prevent any nasty billing surprises -- speaking from experience unfortunately.
There really isn't any competitor to B2 on price, or convenience (e.g. them sending you a NAS device for recovery).
As a provider of a paid service, it seems like they're in a better position to take the high road than a lot of tech companies are, but they have to decide that's who they are, and be clear they mean it.
Ad tracking pixels in your object store dashboard is just clownshoes from a security engineering standpoint, over and above the fact that it's a slimy, dickhead move for a paid service.
It is just a dumb mistake.
If they don't pay attention to stuff like this, then why should I trust them with anything at all? This isn't some minor oopsie, this is failing to deliver on their core product[1]:
Top Backblaze B2 Use Case Solutions
Backup & Archive
Store securely to the cloud incl. safeguarding data on VMs, servers, NAS, and computers
[1]: https://www.backblaze.com/b2/cloud-storage.html
It's not foolproof, but it would stop this.
https://github.com/truevault/hipaa-compliance-developers-gui... was on HN a week ago. It seemed to jive pretty well with our internal policies at the HIPAA compliant company I work for.
Walmart CEO already did couple of years ago https://www.wsj.com/articles/wal-mart-to-vendors-get-off-ama...
The major point to me is: If marketing at Backblaze drives decisions that influence security and tech has either no say in this or is not competent enough, it's not a company for me to trust my data with.
I legit don't understand why a paid storage service would put a FB pixel on their dashboard which handles user files. It's a completely foreign concept to me. This seems like a screw up but also erodes a lot of trust which is unfortunate as I had been looking at them for past 2 months actually.
I even made a post just yesterday and another couple weeks ago on how BackBlaze's inability to set a specific file name, file size limit and expiry date on the pre-signed urls is preventing some of us from switching over from S3 to Backblaze for our storage of app data needs. And surprisingly, I wasn't the only one as I got a few people responding with same concern.
https://news.ycombinator.com/item?id=26430959
Basically:
> A limitation I ran across when using B2 was that their pre-defined url generation doesn't allow you to set file-size limits nor does it allow you to set the file name in the pre-defined url. It simply gives you a pod url to upload it to. So if you are using b2 for storage for lets say image uploads from browser, some malicious user has the ability to modify the network request with whatever file name or file size they want. Next thing you know, you have a 5gb sized image uploads happening.... This pretty much prevents me from using B2 for now.
> I ran into the same limitation! IIRC, there also wasn't a way to expire a signed upload URL sooner than whatever the default was, which was hours or maybe a day. I had the exact use case you mentioned, too - image uploads bypassing my backend server. I didn't want the generation of a signed url to, say, upload a profile photo, give carte blanche to create a hidden image host when combined with the limitation that you highlighted. All sorts of bad things could come of that. I ended up just going back to S3 - costs more, but still worth it.
Since this is for a site/app which lets users upload data, I am really trying to avoid S3 due to crazy costs. I might look into DigitalOcean's offerings. Anyone have any other recommendations?
Dumb suggestion: Run it yourself? Minio is easy to use, even in multi-server mode.
Even if you're paying for the product, you're probably still the product.
In this particular one you are not.
The data was harvested by the Facebook pixel as part of their audience building tech for acquisition of new customers. So you are being sold by Facebook, Backblaze just happens to have been quite careless here but they are not "selling your data".
The breach of trust is BB letting third party code on their platform and especially from a particularly untrustworthy third party. That's it, it's egregious and should be taken seriously, that also means discussing it seriously.
This kind of hyperbole is counter-productive, it only makes it easier to ignore your concerns as "crazy overreacting".
I appreciate that their Twitter account is looking into it, but feel like this should be a pretty quick fix. I'm hesitant to even log into my Backblaze account again until it's sorted.
So they should be good continuing to use "Backblaze"?
backblaze signs BAA agreements with companies storing personally identifiable medical information, I wouldn't believe anyone who told me that this facebook data leak was turned off for those customers; they should immediately be investigated and fined if any such breaches indeed happened.
Full agreement, any interest and goodwill towards the company is now completely gone.
Disclaimer: I am not a lawyer, this is not legal advice, YMMV.
> Believe that's the Facebook pixel we use for tracking, we've forwarded to our web team for review in case that is not intended behavior.
Just... "in case" it's unintended is not promising.
I'm guessing if you're logged into facebook, now FB can correlate:
1) you use a backup service
2) all the metadata from the file names
Whoops.
This is why my network runs pi-hole / diversion with all tracking blocked network wide.
Names or identifiers are the kinds of things that very often end up in file names.
[0] https://cphs.berkeley.edu/hipaa/hipaa18.html
Rebecca_Range_will_and_final_testament.pdf
NadyaNayme - Resume - [Company Name].pdf
divorce_process.pdf
steps-to-take-after-a-car-accident.pdf
personal_injury_michael_perez_west_coast_trial_lawyers.pdf
embarassing_porno_filename.mp4
And this is a smalls subset of filenames that could not only provide PII but also potentially embarrassing or private information that isn't identifying but would be accompanied with files that are personally identifying.
I've also seen similar schemes used by doctor offices/hospitals, where you'll see the patient name and ailment. I once had to troubleshoot a backup problem for what turned out to obviously be an OB-GYN. Imagine my horror as I saw long filenames containing patient names and common STDs scroll by in the thousands.
That’s taking it to the extreme, but it’s technically correct.
If you say “Read the law strictly enough” please do...
I'm still wondering, because Backblaze will sign a BAA (their website says so), making them a business associate. I'm not talking about some private person uploading their own documents. My concern is that given that Backblaze will sign a BAA, then some companies must be using Backblaze and potentially storing PHI data there. Yes?
Backblaze then need to follow: "§ 164.312 Technical safeguards. (e)(1) Standard: Transmission security. Implement technical security measures to guard against unauthorized access to electronic protected health information that is being transmitted over an electronic communications network."
Facebook isn't authorized to access this data, but that might be more of a problem for Backblaze, even if Facebook could be required to delete the data.
I had to fight hard as an engineer to make sure that it does not happen. We had meetings after meetings, and it took a lot of effort for me to explain the risk of data leakage. I was questioned on my "insecurity" for not "trusting" people. It was not a nice experience. I had to inform them that tracking needs to be dealt with properly, not just lazily install google tag manager because it gives marketing 'flexibility'.
Google Tag Manager is an absolute cancer on web development.
Once it's in, you'll never get rid of it.
Apparently it lets people drop in random code from a bunch of different analytics platforms, so it's pretty much guaranteed to consist entirely of the sort of stuff I have NoScript enabled to block in the first place.
Some random guy in his basement assured someone in marketing they could handle our volume? Chuck their tag in and watch their website get DDoS'ed with millions of requests per minute, which takes out our website because marketing made it fail loudly.
IT can often be "we make someone else's bad idea happen," and that's because we simply lack veto power.
Hiring people is difficult as there is lack of supply of talented people. We hire InfoSec on contract basis not full-time and they don't join such meetings due to the nature of the contract. So all the responsibility fell on me to defend our technical decisions at that point in time.
I'm working on building out the engineering culture / awareness within management now, to ensure these things do not happen, and I don't have to be questioned as to why we cannot install "google tag manager" in our front-end.
It all comes down to creating awareness, and making people understand. Fortunately for me our CEO gets it, he ended up siding with me.
To me, this is an even more concerning issue. But then, I have no idea how the finance services world works, so maybe this is more common than I think?
Outsourced CISO/InfoSec is a valid and reasonable thing for some companies.
If I was running a financial services startup, those groups would be near the front of my list in terms of internal hiring.
But again, since it's not always easy to find the right people I end up having to fill in for everything we don't have a team member to execute on.
Usually I try to reason with management first. If it can be resolved internally we would not include outside consultants, however if it gets serious beyond something we can handle internally we would ask outside consultant to come in.
Personal attacks for doing your job? If you're still there, the stock options had better be enormous!
If you are the most knowledgeable person then you get blamed for their bullshit fantasies being impossible or unwise (or illegal)
You really need to present well and be careful about arguments like “millions of websites use GTM”. I did days of research and presented that while yes using GTM on Wordpress sites that hold no sensitive data might be fine however we are a financial services and we collect customer private data. So getting everyone on the same page and presenting alternative ways of solving the problem was critical.
"I trust people just fine. I trust people working for outside companies dependent on information gathering to gather information. Google has no fiduciary duty to our clients. Same with Facebook. We DO have a fiduciary duty to our clients and it includes not doing things that may send their confidential information to third parties because SOME of the information used may be useful to our marketing department."
It means there's either nobody reviewing the privacy implications of marketing decisions, or that somebody who knows better is reviewing these decisions and decided leaking data like this is acceptable.
Both those possibilities make backblaze a non-starter for me now.
> You can't expect marketing to understand these things.
I work in marketing ops and I know how difficult it is to get marketing folks to understand or care about how the tracking they use works. There’s a small but growing number of us trying to change behaviour and awareness from the inside out but if marketers fuck up on privacy an example should be made of them.
Unless you have really strong dev leadership, the trackers will end up in your product.
I'd love to hear from folks who have successfully blocked their business team on this. What tactics did you use?
If they fix this, goodwill will be burned, but it's not a dealbreaker for using them. If they don't fix it, then yes by all means leave the service.
I can has reference? Sounds genuinely interesting, and that info didn't find its way through the rock I'm apparently under.
You mean other than storing them forever?
Are there audits to check compliance?
FB used 2FA phone numbers for spying, after repeatedly promising that they will be not used in this way.
See also FB promises about WhatsApp during merger.
Is there a truly independent confirmation?
Isn't there a lawsuit or similar going through the EU courts atm about Facebook not really being GDPR compliant? eg They claim they are, but the court case is about them not being.
First, we don't know that. Second, it doesn't matter even if they are not using it in any way. Backblaze shouldn't be sending this data to FB. And lastly, even if FB isn't using it in any way today, how do we know they won't use in future?
Nobody should be having access to any data that they don't absolutely need. It doesn't matter how mundane the data is, which company the data is going to, etc etc.
Like I said, it's definitely a problem regardless.
But this is like saying it that if you misplace your cell phone, it doesn't matter whether you left it in your friend's car or a taxi. Of course it matters whether or not data is being (mis)used.
You need strong legislation and even stronger penalties. The only way to stop pervasive tracking is to financially ruin any company that employs it.
That's extra effort, but the big ones will do it
This will rely on a user's FB account having the same email as used for BB, which could be unlikely in the case a company is paying for it. But it should work well enough for retail targeting.
https://www.facebook.com/business/help/341425252616329?id=24...
> They claim it is hashed before use so they never see the raw data
If they can match hash data with real data, they can know more than they did before. Depending on what algorithm they use for hashing (no mention of it), they could be using a similarity hash so that will minimize changes if there are minor differences in the dataset.
Let's say I find a profile through comparing hash of email to email in Facebook's database. I can then compare additional information to see if a customer has provided incorrect information to Facebook as a user. Facebook could check if my address is similar to online shopping sites I use and if not, flag the account.
Still, main point is you only need an identifier and none of the other data Facebook has. Pixels are not required for this as noted in the original comment, they probably have enough in the account details already.
they have the original data of their own audience. So they can send it explicitly to FB (instead of FB sucking it from the page's pixel so to speak) for FB to build the lookalike audience which FB would do using the wide FB owned data.
If file names are truly sensitive you have to actually be pragmatic in how you choose to render them.