Can HN users share their experience with B2? I wanted to use them as a backup backup solution since their pricing is close to what getting your own leased server would be.
Also, are there ready solutions for B2 and incremental backups of Ubuntu server?
daptin uses rclone to connect to cloud storages. I can add more actions to make a usable dashboard (for a file related utilities server). Please create an issue if someone would be interested in this
duplicati [0] (at least beta 2) can backup to it, however, I use Google cloud storage with it, since B2 is a single datacenter. I like a bit of extra paranoia, even if it is backups. I hear they are spinning up a second one, but not sure how multi-Region storage will work.
According to their FAQ, they're still running primarily out of the single DC [1]. However they did post on the blog that the new DC was online as of June 2017 [2].
I run Backblaze (backup) for Desktop but also have a local NAS that stores copies of critical items. That NAS also serves bulk storage (like old Premier Pro projects) which is backed up to B2. So critical data follows 3-2-1 backups [3]. Desktop, NAS, Backblaze. Less critical bulk data is only 2 copies, NAS & Backblaze B2.
That's a really long way of saying that 1 remote datacenter is enough for me. My data is important to me, I'm a digital packrat, but the likelyhood of something taking out my home (SF Bay Area) and Sacramento (Backblaze DC) ... likely means TEOTWAWKI [4] - and my data will be of very little concern then.
Caveat. I'm just a person. Not a business. If you're a business using B2 for backup... you probably want to worry more about multiple AZ/Region storage.
Yev from Backblaze here -> very good points! We find that some folks even use us along WITH S3 or Azure as not only a different "region" but specifically because we're a different "vendor" - for greater diversification. It's easy enough to fork data over to B2 and with the low costs, it make it an easy sell for IT folks.
Backblaze now has three datacenters spread about (two in Sacramento area 50 miles apart, and one datacenter in Phoenix). However, if you store exactly one file in B2, it will land in one of those three datacenters. Very soon we will allow you to choose which of the three datacenters your one file lands in.
> I like a bit of extra paranoia
So do I, and I would NEVER keep a valuable piece of data in only one datacenter with only one vendor. My advice to my closest friends is you value a piece of data highly, you should have three copies with three separate vendors and technologies. For example, one copy on your local laptop, one copy in Amazon S3, and one copy in Backblaze B2. You might also make sure the Amazon S3 is not hosted in the same building as the Backblaze B2.
Since Backblaze B2 does not share a single line of code with Amazon S3 and no employees and a different billing system, this means a bug or network outage or malicious employee inside Amazon S3 won't affect your copy in Backblaze B2. For bonus points, use two different credit cards to pay for it that expire at different times. If you forget to pay your Amazon S3 bill, they will delete your data for sure.
I'm using B2 with duplicity under Linux. I ran into an issue where the tool couldn't upload its own index files to the backend, but there was a fork that fixed the problem and I expect it will be resolved in the official fork soon.
B2 is great, I've used it roughly since they came out with it. I rolled my own solution for backups with linux, involving NBD and libcurl, but it sounds like there's other available things now.
I'm stoked, they're already the cheapest, now they're even cheaper. I think the only thing left on my "wish list" would be some sort of lower pricing for using it from regions of the internet with lower transfer costs. Like, if there was some VPS provider Backblaze would partner with, then offer B2 at a discounted access rate when used from that provider, akin to how S3 access is cheap when used from EC2.
I switched to B2 after Crashplan Home was discontinued. I backup a couple hundred GB of personal photos/videos mainly, stored on a Ubuntu server. The drives are already protected by SnapRAID, and backups to an external drive, so B2 is my fire/theft/everything-else-failed-at-once protection.
I just use b2 sync, and don't synchronize deletes. It took a few minutes to get set up, and I haven't thought much about it since, other than seeing the bill for a few dollars a month. The cron job should e-mail me if something goes wrong. I've also checked a couple times to ensure the latest files are visible in B2 storage, just to be certain.
I've been using it with the first-party b2sync cli tool. My use case is very simple - refresh the backup, then download the whole thing when the hard drive dies. I was afraid the restore operation would be expensive, but it was cheap enough that I decided to be more liberal with what I upload in the future.
Recently we evaluated to use them to backup 25TB of data. Unfortunately their client sustained bandwidth was about 3 MB/s and even less for app that came with DiskStation. This is on 100 Mbit/line that normally gives that speed sustainably.
Eventually we decided to use tapes. With our data usage just after 2 years the tapes will be cheaper...
I use duplicity + B2 on my Fedora laptop. It works very well. I even proved out the restore when I accidentally borked my .profile file the other day ;)
I tried to evaluate it last month. Their login page does some kind of cross-site cookieing or referering that my privacy settings don't permit. Any time I log in, I get a warning that someone's been trying to brute force my account and it's been locked for 24 hours.
Support was useless. I was repeatedly asked if the IP address I gave them was public or private, when they should've had training or tooling to know at a glance. The help system closes tickets aggressively, ignores replies, and the link to keep a ticket open just leads to a knowledge base. Eventually support agreed to take a curl that reliably reproduces their bug, but I got no tracking number or even a vague promise they'd get back to me when it's fixed.
I concluded that if the product was broken and support was broken, there's probably something upstream in the business that's broken and I am not confident I'd be able to restore data.
> Any time I log in, I get a warning that someone's been trying to brute force my account and it's been locked for 24 hours.
We had a bug where anybody who "cleared their cookies" would get exactly that behavior. It is fixed. I apologize for the inconvenience. I'm assuming you run some sort of cookie blocking software?
If you used stock Chrome browser or stock Safari or stock Microsoft Edge and didn't clear cookies, it always worked.
The great thing about DO storage is that it is S3 API compatible. The not-so-great thing is that DO has serious stability issues. Try deleting 1000 files and the backend just gets stuck for like 5 minutes :/ Make more than 100 calls and you get rate limited. DO does not acknowledge how serious the issues are. Not sure how B2 API performs.
Is this as simple as, "we did market research and we found people wanted to use the service, but didn't want to pay very much?"
Or, was it, "the performance (or some other factor) isn't worth using the service at the current price"
I mean, charging 1/9th of the amount that Amazon does sounds incredible. I'm guessing Amazon has lower costs but it also seems like they're just giving it away now.
>I mean, charging 1/9th of the amount that Amazon does sounds incredible. I'm guessing Amazon has lower costs but it also seems like they're just giving it away now.
Shop around, bandwidth costs nothing. Amazon & Co. simply charge insane prices. Backblaze will charge you $3285 if you download at 1gbit/s for a month straight, you can get a gbit line for 10th of that.
This is really the dirty secret for the big 3 cloud providers. Their egress charges have huge margins. I’m hoping that smaller players like Digital Ocean reaching up, and better competition across the big 3 starts to push egress pricing back into a more reasonable level.
> dirty secret for the big 3 cloud providers. Their egress charges have huge margins.
I don't know about the other providers, but Backblaze buys bandwidth at about 2/10ths of 1 cent per GByte right now, and it drops by maybe 10% per year. We look around hard for inexpensive bandwidth, and then make sure we always have redundant network providers and redundant paths. That's all blended into the 2/10ths of 1 cent per GByte cost basis.
Part of this is complicated by any "spikiness" in network traffic. Network providers sell bandwidth based on things like "95% of peak traffic". So the 2/10ths of 1 cent per GByte is based on an assumption that overall the outbound traffic is perfectly flat. It would be more expensive to Backblaze if "most days" the outbound traffic was 200 Gbits/sec but once every 7 days it spiked to 400 Gbits/sec for 25 hours then dropped again. In that case Backblaze is hit with a bill for 400 Gbits/sec sustained for the entire billing period (a month), but customers did not use it up in a "flat" fashion so it eats into our margin.
Luckily, since Backblaze supports hundreds of thousands of different individual customers, it appears mostly "flat" to us in aggregate. The five week days tend to be about 10% higher network use than the weekends, but other than that it is all super flat and regular (so far).
> Amazon starts at $0.15/GB and drops with more volume.
Which has ALWAYS baffled me about Amazon's pricing. I would have thought they price it the REVERSE where it is low price at first, and rises. For example, start with a "low introductory rate for the first Petabyte" to get small businesses addicted to Amazon services while Amazon still makes a (solid) profit, then when the small business grows into a large business that requires more than a Petabyte of data Amazon would increase the price because: 1) scaling is hard and does cost money, and 2) a company looks into the future and says "heck, if we reach a Petabyte we're making so much money we can AFFORD the higher Amazon rates", and 3) as a company grows quickly they have so many other problems they won't prioritize getting off Amazon, especially if Amazon is scaling well for them.
Related side note: scaling really does cost extra money. For our online backup business, almost no customers have more than 10 million files in any one "backup" (each backup is "per laptop") so as programmers we can use any data structure we like to browse those files, a flat list IS FINE. But for B2, a large customer might have 1 billion files in one "bucket", and so we have to be a HECK of a lot smarter about how we organize that data so that the customer can still sign in with a web browser and view them all. For Backblaze, that meant moving up to a clustered hash database called Cassandra, and deploying (so far) about 50 1U computers, each with two CPUs, each with fast SSDs and a layer of complicated code to manage all of it. So the scaling requirements of the Personal Backup client required 0 extra servers, but at B2 scale it requires an additional $175,000 worth of equipment. Scale costs extra money.
Yev from Backblaze -> There are of course trade-offs to every service, but we realized that we didn't need to be charging that much for downloads (we already lowered it from $0.05/GB to $0.02/GB in April 2017 - but after almost a year of gathering data realized we could have gone further down).
> we realized that we didn't need to be charging that much for downloads
To be totally clear, Backblaze can keep going lower. Backblaze buys bandwidth at about 2/10ths of 1 cent per GByte right now, and it drops by maybe 10% per year. The only reason we don't go lower faster is that we are trying to "ease into" this. Internally we are afraid of hitting a tipping point and attracting too many customers using the outbound bandwidth too quickly. The Backblaze network capacity is currently built out for an extra 500 Gbits/sec or so of outbound bandwidth that we aren't using (ready to sell to customers), and it takes time (a few months) to provision more bandwidth. Sometimes the bandwidth providers have to trench to our datacenter which takes like four months.
So if we drop the price too low too fast and enough customers show up that would require a total of let's say 600 Gbits/sec capacity in less than 30 days, we might not be able to keep them all super happy and we might get a black eye. So we want to "ease into" this to continue to provide top notch service while we figure out how much of a market exists for inexpensive outbound bandwidth. The ALTERNATIVE strategy would be to massively over provision, but if not many customers show up then we have wasted money. Since Backblaze is profitable and not funded by Venture Capital, we do not like wasting money. :-)
Thank you for being so open about the operations at Backblaze. It's really refreshing to see some considerations that go into making decisions like this.
It's very easy for people who have little to no expertise in a domain go on about how X company should do Y and it wouldn't be a big deal, when lots of times it would be.
> refreshing to see some considerations that go into making decisions like this
Most businesses over a certain size figure out their "COGS" (Cost Of Goods Sold) which is what it costs to produce and sell their product if the scale goes through the roof. Then they decide what the "margin" is (the mark up that we get to keep ourselves). And there is your price. Some premium products are based on super high markups/margins, but in competitive areas the margin is usually lower than 50%. The MOST competitive areas (like food in supermarkets like Safeway) get as low as 20% margin. Now, the COGS doesn't include salaries of certain people in the company like the software engineer's salaries to produce the service. The reason those aren't included is they only write the software once, then you can sell it an infinite number of times. So the margin/markup goes into paying the software engineer's salaries. So the margin isn't ALL "profit we keep in a big swimming pool and swim around in".
Programmers like me learn all of this in the first year of running their first startup company, so it isn't a secret. :-) I think the business majors learn in their freshman year of college.
I noticed you use cogent, aside from some peering with Verizon and others, are you sure your transit capacity is available by cogent and he? Had issues at choopa and only peer with them
We have a minimum of two separate bandwidth providers into each datacenter. Depending on which datacenter your data is in, it is some combination of: Cogent, AboveNet (now called "Zayo"), Wave, Century Link, and maybe one or two others??
The reason we always have two separate bandwidth providers is that any one provider is basically guaranteed to be offline at least 5 minutes a year, and there is a 30% chance any one provider will be offline for two hours once a year. But it is highly unlikely TWO providers will be offline at the same time at any point.
We have three separate datacenters, but any one file you store will be in one datacenter. We have two datacenters in the Sacramento region (about 30 miles away from each other) and one datacenter in Phoenix. So depending on which datacenter your file is in, your network path might change to use a different provider. We are also adding brand new datacenters in 2018 in new regions (like Europe) so that will probably mean new and different network providers.
If there is the need for a global CDN and there is some left-over development capacity, just look at local providers for dedicated servers and build your own. The bandwidth prices are often insanely low. For example Hetzner in Germany has 20 TB egress traffic included for their 3€/month cloud servers. The same would cost $1,800 on AWS.
I second that. As I ended up as a sort of DevOps for the family, I recently realized that dedicated servers became so much cheaper than VPS. Of cause with a dedicated server you need to worry about hard disk failures etc, but the price difference is just too big to ignore. Plus on a dedicated server one needs to worry less about Spectre.
I use backblaze for my backups with restic and enjoy it, but their service is just so slow I'm now looking at giving digitalocean a try instead. Transfers both up and down are painfully slow even on a 1Gbit connection.
Have you tried enabling more threads (add -o b2.connections=$number)? It improved enough to make it ok for my (admittedly tiny) backup needs. The process of creating a new snapshot is still really slow though, but I'm not sure that is B2's fault.
I debated switching to Backblaze after CrashPlan's home plan was removed, but they don't appear to have an option to let me send them an external HDD to "seed" the backup, and that's a deal-breaker for me.
I have 4TB of media to backup. My ISP will throttle/penalize me if I download/upload more than 1 TB per month, assuming I do absolutely nothing else with my internet connection. Even assuming I devote 80% of bandwidth to just the backup, it will take me 4-6 months of metered transfers, while paying for their service, before I actually have a "complete" backup with them.
Yev from Backblaze -> We don't currently have a seeding open for Backblaze Online Backup (we do for B2, it's called Fireball, but is quite expensive for just that little bit of data). If you think you can actually upload all 4TBs in one month (might be able to w/ Maximum Throttle + some threading), we've had some of our users contact their ISP and received a "pardon" for a month. Not saying that's the norm, but some folks did manage to get their data caps lifted for a short time!
Interesting question -> We just simply didn't build in that functionality. Right now the only way to get data in to Backblaze Online Backup is to have that data come from the client which resides on the machine. For Backblaze B2, we knew that organizations would have much larger data-sets, so we designed it with that functionality in mind.
Also, at the beginning the "no seeding" thing was kind of a gate to folks who had a lot of data but limited bandwidth. Our theory was that you should never be afraid of uploading your data, you should have the bandwidth to match. That was bout 10 years ago though, so things have changed - but it's still not a priority project for us - especially considering that since Online Backup is a flat-fee, by definition almost anyone that would use the "seeding" feature would not be a break-even customer for us.
1) My 4TB is not really expected to grow much beyond this point. It's the accumulation of a lifetime of hoarding, and the year-to-year addition beyond that will be in gigs, not TBs. So after the first year, it's basically just cold storage. If you guys can profitably store 4TB in cold storage for $50/yr, it should be profitable after the first year.
2) The guys who have enough media and are technical enough to want to seed it are also more than likely going to be the guys who make the backup service recommendations for their friends and families.
That's the website I created for friends, family, and coworkers to read, because I got tired of explaining the same stuff to them every Christmas. I very nearly bought the Crashplan Family plan at one point just to make it easier for them--I'm not glad I didn't. But I guess my point here is that the power-users, as irritating as we are, are going to be the ones who spread your product by word of mouth.
3) People with lots of data who take backing up seriously generally don't like switching vendors unless we have no other choice. I don't want a provider that I feel like I'll "outgrow" when I get too much data and be forced to switch to a more expensive plan or to another vendor. So while a user with 100 GB or less might be with you guys for a year, he will probably move to another vendor if they can do it cheaper, since he doesn't have much data. The hoarders, on the other hand, just want to know you'll keep their stuff safe and won't jack up the price on them at some point, and will otherwise be willing to stick with you forever.
Thanks for your reply. I can certainly investigate that route, but the kinds of ISPs that implement these caps aren't exactly known for their magnanimity to begin with, so I'm not optimistic. Assuming they don't, I would get charged about $10 / 50GB overage, which would work out to 3000 / 50 = $60, which is the yearly cost of your service. So my dilemma is that I'm either paying for about half a year before I get a complete backup set with BB, or I'm effectively paying double the service price just to make sure the data all gets to you guys in a timely manner. For that price, I could just as well just shell out the extra dough to stay with an upgraded Crashplan Small Business account.
I hope you guys will reconsider or re-examine your position on seeded backups. I don't think I am being unreasonable in wanting that to be an option, nor do I think I'm alone in having somewhere north of a terabyte of media--any amateur photographer or video editor probably has a similarly-sized dataset. If unlimited bandwidth were available, this wouldn't be a concern, but, I don't have the luxury of being able to choose a different ISP. And with Chairman Verizon heading the FCC, I don't see that changing any time soon.
Definitely understand that mind-set. We review most of our decisions every year, but it's a tricky problem. Since Backblaze Online Backup is a flat-fee service, by definition most folks who have that much data would not be break-even customers for us, and adding in services that encourage those large customers wouldn't be a great long-term plan. That said, one thing you might be able to do is see how much more expensive an "unlimited" plan with your ISP would be for 1 month (hopefully less than the $60 extra). CrashPlan Small Business is a good service as well, so if you're happy with them and don't mind paying the extra few dollars a month, staying with them, from a bandwidth perspective isn't the worst.
And yea, I'm personally hoping that the ISP situation gets a bit more friendly towards consumers!
Not sure of your ISP, but some offer an unlimited option for an additional fee. For instance, Comcast offers this for an extra $50 per month: https://www.xfinity.com/support/articles/exp-unlimited-data While it sucks to have to pay extra, there is at least a cap (ironically!) on the price.
I'm quite curious. What is more important to you, is it not paying for 6 months while the initial backup is made, or having the initial backup made within 1 month?
Solving the first is definitely within Backblaze's capability if they are prepared to make an exception.
I'm curious to know your use case for backing up that much data.
I have this idea in my head that some people are digital pack rats and want to save every movie and series they pirate off the internet in case the apocalypse happens and they desperately need to re-watch an episode of Friends.
Is it uncompressed video? (Why would you back that up to the cloud?) Is it family media? (How many baby pictures did you take???). I honestly have no idea how someone collects 4 TB of data that is worthy enough to back up.
I'm a hobbyist photographer who gets to shoot significantly about 4-5 weeks per year. In each such week, I typically shoot 2,500 - 3,500 images. My current DSLR camera produces images that are in the range of 50MB each. That means about 125 - 175 GB / week of shooting. I currently have about 1.5TB of image data backed up in the cloud. And I don't shoot video at all.
By comparison to some of the more serious amateur photographers that I often travel with, I don't shoot a lot. They may shoot 10,000 images in a week.
I know a person who has 6 TB of own photo and video. This accumulated through 8 years of documenting various activism-related events. And the photos take a substantial part of that. On a good camera high-quality JPEG is 10MB or more, so just by taking 100 pictures (normal amount if one wants to seriously document an event, not own perception of it) one needs 1GB to store them.
I am certainly a digital pack rat. Luckily, unlike physical pack rats, our data doesn't take up a lot of room, and since storage capacity keeps increasing so rapidly, I have never had to delete anything.
Here are some of the things that I have that add up to my multiple TBs of data I want to keep around.
First, I have every single school assignment from growing up (30+ years ago, so it doesn't take up much space). These are kind of fun to have. I can show my kids the papers I wrote when I was their age.
A lot of old converted 8mm tapes from when I was a kid. I used to be really into making claymation videos, so I have hours of videos that I converted around 15 years ago into digital formats. I could probably reconvert them to a more modern codec that would take up less space, but they are still multiple hours. Currently takes up around 100 gigs.
Every digital photo I have ever taken. Over 20 years, this is a lot of photos. A few hundred gigs there.
Tons of videos of my daughter. A hundred gigs there, growing all the time. Maybe I take too many, but it is really fun looking back and remembering when she was little.
Every programming project I ever created, going back 25 years. Fun to see my progression. Not a ton of space, but many gigs worth.
A ton of old games that don't exist to download. Old shareware, commercial games, etc. I created CD images for the all the ones I had on physical media, so I don't have to lug around the CDs anymore. I like to be able to play games I played years ago every now and then. This takes up a good chunk, and is probably a few hundred gigs.
Ripped DVD movies and TV shows. Bought many over the years, and don't want to have to find and/or rip them again if I lose the data. This is hundreds of gigs.
Countless other minor things (saved games, documents, etc).
I like all of these things. If I have a house fire, I like knowing I wouldn't lose all of that. Since it is so cheap to maintain a backup of it, why not? Yeah, I wouldn't be DEVASTATED if I lost all my old DVD rips, but for 60 bucks a year (and that is the same cost whether I back those up or not), why not keep them backed up?
You're right, the private data that I choose to preserve is objectively worthless. I shall relay your findings to my family.
"Sorry, Mom, I can't help you watch that Mork and Mindy collection on your bedroom-TV Roku. Someone On the Internet informed me that my digital collections are not worthy of keeping, and I dutifully deleted them."
"No, sweetie, we absolutely cannot share your childhood with our daughter and watch my digital rip of Fraggle Rock. Good parents delete their abhorrent digital media and pay HBO Go $15/month for the privilege of viewing it!"
"What are you thinking, keeping those hundreds of CD's and Vinyl's we ripped together on your iPhone, Dad? You can chuck all that crap that we actually own and pay a monthly fee to Spotify for the privilege of streaming it! Unless, of course, the publisher decides to pull their collection off of Spotify, for whatever reason."
"Hey, guys, now that we're all moved out of the house, we need to talk about the Nintendo 64. Mom and Dad bought it for all of us, so I think it's only fair that we keep the physical console at their place so that no one of us will have exclusive access to it. As for the games, I think each of us should be allowed to take the cartridges with us that we individually paid for and/or receives as gifts. So that'll be me taking Super Mario 64, Majora's Mask, Super Smash Bros, Star Fox 64, and NBA: Hangtime; Steve takes Goldeneye, Ocarina of Time, Perfect Dark, and Donkey Kong 64; and Fred takes Mario Party, Pokemon Stadium, Mario Kart, and Banjo Kazooie. Yeah, I know, I do wish there were some kind of legal backup method that might let us all enjoy the games we collectively owned. Oh well. Maybe Nintendo will release an N64 Classic and we can each pay $80 to play the games we already paid for. Assuming the games we own appear on the collection."
I think the idea is that if it is possible / easy to re-obtain the media, then you don't need to back it up (i.e., if it was obtained via bittorrent).
A second classification is data that isn't changing -- you can create a couple backups of that media, store one in a bank safe deposit box, maybe ship another backup to a family member across the country.
Then you have recent acquisitions and ongoing projects -- those are what you would sync daily to an online backup service. So you still have offsite backup of everything, but by splitting up the categories, you end up not needing to sync that much online.
>>I honestly have no idea how someone collects 4 TB of data that is worthy enough to back up.
All data is worthy of Backup. I have about 40TB of Home Storage that is backup;ed on a set of encrypted disks offline. (Total RAW Capacity of all my storage in my home lab last a checked was about 150TB)
I do not go Full 3-2-1 with it, but I do have a duplicate copy of all my data with about 2TB in a Full 3-2-1 Backup, I would say about 750GB to 1TB has 1 copy of the 3-2-1 in a "cloud" vendor of some kind
As I'm in a similar situation with Crashplan I've researched this a bit more (have just under 2TB so was originally just going to pay up for the business plan so cost wasn't to high).
VPS's are probably a no go as they have all moved to SSD's with low sized volumes. Even with Digital Ocean where you can add block storage you are looking at $100 for a week for 4TB + compute and bandwidth. AWS you are looking at $368+ just on data egress.
hetzner.com server auctions look the best option. Current cheapest is €31.93 per month for a Intel Core i7-3770 (CPU-B 9445) with 2x 3 TB drives and 16 GB. Probably simple enough to use lvm and jbod on the drives to give a single 6tb volume. No egress charges but reading they throttle if you go over 20,000GB a month so a non issue.
No setup fees/contract on the auctions although reading I think it's 30 days/1 month notice to cancel so would need to cancel as soon as ordered.
My plan is to probably move to Git Annex https://git-annex.branchable.com/, use my local NAS as the main backup, maintain a copy of media / photos / docs in Backblaze and duplicate files I can't afford to lose to S3 also. Looks simple enough with Git-annex and lets me maintain multiple copies based on cost / importance.
> Are Backblaze planning on offering solutions for europeans?
About 15% of our business comes from Europe today, and we collect VAT tax accurately for all Europeans as required by European law. So if you are willing to encrypt your data in Europe and send it encrypted WITHOUT the encryption keys to store it encrypted in B2 (zero knowledge) in the USA, then Backblaze would love to have your European encrypted data.
If you are asking when Backblaze is going to open a European datacenter where you can guarantee your encrypted data sits on disks inside of Europe, we are hoping to open that datacenter in 2018 (this year).
Backblaze still won't let you label a snapshot. It's so frustrating when I've got say folder A, B, C, and I take a snapshot of each, I have no way to tell which snapshot is which. And god forbid those folders are 100GB in size. This is my only gripe with them.
Plain and simple: you won me. I am hosting (covering the cost out of my own pocket) a very interesting motley collection of pictures of Hungarian history, our first sister site is http://fortepan.us/about/ you can get a feel what it's like and I have been carefully watching for affordable professional backup solutions. Yeah, the unlimited google drive is nice but it's also a hack and they don't really support this at all.
I had my eyes on B2 because the storage costs were within budget but didn't switch our backup to it because if we ever needed to download it, well, what then, it was too expensive.
I will upload our 100k / 2TB or so photos the weekend. Awesome.
Now I am reviewing this... you are now on BunnyCDN level of download pricing. Are you like a CDN or does every European download need to cross the Atlantic?
Yev from Backblaze here -> All data currently has to cross the Atlantic as we have data centers in California and Arizona, but we do hope to add European data centers in the near future!
Not all bandwidth is created equal. Different providers can have widely different levels of performance. Higher-priced bandwidth _can_ also be better bandwidth.
Cloud providers haven't done a great job of explaining this so far, but Google Cloud's Network Service Tiers product is a good attempt at why they charge more for bandwidth, and how you can get it for cheaper: https://cloud.google.com/network-tiers/
That said, cheap (and presumably slower) bandwidth may be the right target for B2, given that it's customers are probably using it for backups and private storage rather than, say, serving a website serving photos to millions of visitors from around the globe.
So comparing AWS or Google transfer costs directly to B2 isn't necessarily a fair comparison, depending on your needs.
I'm about to set up a NAS device at home for personal storage am looking at cloud providers for an offsite backup. I was originally going to send stuff to AWS glacier since my offsite backup is pretty much only going to be for disaster recovery (fire/theft etc).
This is my first time seeing Backblaze and it looks appealing, does any one have any opinions on using it vs glacier/s3?
For context I currently have less than 1TB of data I need to store and I don't expect to upload more than a few GB per month.
Also would I use B2 or the unlimited personal storage?
Cool! I'm using them to store my Time Machine backups from my Synology. The first month I burned through about 3TB of my 1TB with comcast, but that's the only month I went over my 'cap' (they have 2 months' grace period).
The only problem I have is that my Time Machine backups change too much -- for, say, 400MB of data being backed up in time machine, it seems like the deltas (and what is synced to backblaze) is something like 4GB. I'm using the CloudSync functionality from Synology -- I think the issue is with Time Machine, not with CloudSync, though.
What this means is I'd regularly exceed my data cap if I let me CloudSync run all the time. As a workaround, I've scheduled it to only run between midnight and noon on Mondays, which results in syncing less data (apparently the deltas must contain a lot of duplicate data that doesn't have to be synced up to BackBlaze when I only do it a few times a week).
Of course, now my problem is I'm afraid that I'm risking the integrity of my backups because the schedule might cut off my sync in the middle of a run. Since I leave my home computers off during the day, though, there shouldn't be new deltas occuring as I'm syncing to backblaze, so once it catches up (and 12 hours is enough to catch up), there is no more activity.
I strongly recommend Notablespy to everybody right here. Let's do this right. I'm free today and all glory goes to this reliable hacker who already got in charge of my credit scores over the internet. I hired him in last year over my credit report which had been dropping simultaneously and since then we have been working together, as one over and over till the issue on my credit card stopped. He's understanding, cooperative, smart and legit. He fix my scores for me all those years without traces till now, with this i am well pleased. You can equally mail them Notablespy.org@gmail. com And be safe from your deepest thought.
91 comments
[ 2.9 ms ] story [ 70.8 ms ] threadAlso, are there ready solutions for B2 and incremental backups of Ubuntu server?
I really miss a node module for B2 API though.
https://github.com/daptin/daptin
[0] https://www.duplicati.com/
I run Backblaze (backup) for Desktop but also have a local NAS that stores copies of critical items. That NAS also serves bulk storage (like old Premier Pro projects) which is backed up to B2. So critical data follows 3-2-1 backups [3]. Desktop, NAS, Backblaze. Less critical bulk data is only 2 copies, NAS & Backblaze B2.
That's a really long way of saying that 1 remote datacenter is enough for me. My data is important to me, I'm a digital packrat, but the likelyhood of something taking out my home (SF Bay Area) and Sacramento (Backblaze DC) ... likely means TEOTWAWKI [4] - and my data will be of very little concern then.
Caveat. I'm just a person. Not a business. If you're a business using B2 for backup... you probably want to worry more about multiple AZ/Region storage.
[1] https://help.backblaze.com/hc/en-us/articles/217667468-B2-Se... [2] https://www.backblaze.com/blog/data-center-design/ [3] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ [4] https://en.wikipedia.org/wiki/Survivalism#Survivalist_termin...
> since B2 is a single datacenter
Backblaze now has three datacenters spread about (two in Sacramento area 50 miles apart, and one datacenter in Phoenix). However, if you store exactly one file in B2, it will land in one of those three datacenters. Very soon we will allow you to choose which of the three datacenters your one file lands in.
> I like a bit of extra paranoia
So do I, and I would NEVER keep a valuable piece of data in only one datacenter with only one vendor. My advice to my closest friends is you value a piece of data highly, you should have three copies with three separate vendors and technologies. For example, one copy on your local laptop, one copy in Amazon S3, and one copy in Backblaze B2. You might also make sure the Amazon S3 is not hosted in the same building as the Backblaze B2.
Since Backblaze B2 does not share a single line of code with Amazon S3 and no employees and a different billing system, this means a bug or network outage or malicious employee inside Amazon S3 won't affect your copy in Backblaze B2. For bonus points, use two different credit cards to pay for it that expire at different times. If you forget to pay your Amazon S3 bill, they will delete your data for sure.
> ... three separate vendors and technologies ...
> ... does not share a single line of code with ...
> ... use two different credit cards ...
This person legit thinks about failure modes.
Haven't done any file restores yet.
I'm stoked, they're already the cheapest, now they're even cheaper. I think the only thing left on my "wish list" would be some sort of lower pricing for using it from regions of the internet with lower transfer costs. Like, if there was some VPS provider Backblaze would partner with, then offer B2 at a discounted access rate when used from that provider, akin to how S3 access is cheap when used from EC2.
I just use b2 sync, and don't synchronize deletes. It took a few minutes to get set up, and I haven't thought much about it since, other than seeing the bill for a few dollars a month. The cron job should e-mail me if something goes wrong. I've also checked a couple times to ensure the latest files are visible in B2 storage, just to be certain.
Haven't tried a restore yet.
Eventually we decided to use tapes. With our data usage just after 2 years the tapes will be cheaper...
Support was useless. I was repeatedly asked if the IP address I gave them was public or private, when they should've had training or tooling to know at a glance. The help system closes tickets aggressively, ignores replies, and the link to keep a ticket open just leads to a knowledge base. Eventually support agreed to take a curl that reliably reproduces their bug, but I got no tracking number or even a vague promise they'd get back to me when it's fixed.
I concluded that if the product was broken and support was broken, there's probably something upstream in the business that's broken and I am not confident I'd be able to restore data.
> Any time I log in, I get a warning that someone's been trying to brute force my account and it's been locked for 24 hours.
We had a bug where anybody who "cleared their cookies" would get exactly that behavior. It is fixed. I apologize for the inconvenience. I'm assuming you run some sort of cookie blocking software?
If you used stock Chrome browser or stock Safari or stock Microsoft Edge and didn't clear cookies, it always worked.
I went from only affording to backup the most important files with Tarsnap to just backing up / with B2. It's the difference between $1/mo and $50/mo.
Or, was it, "the performance (or some other factor) isn't worth using the service at the current price"
I mean, charging 1/9th of the amount that Amazon does sounds incredible. I'm guessing Amazon has lower costs but it also seems like they're just giving it away now.
Shop around, bandwidth costs nothing. Amazon & Co. simply charge insane prices. Backblaze will charge you $3285 if you download at 1gbit/s for a month straight, you can get a gbit line for 10th of that.
> dirty secret for the big 3 cloud providers. Their egress charges have huge margins.
I don't know about the other providers, but Backblaze buys bandwidth at about 2/10ths of 1 cent per GByte right now, and it drops by maybe 10% per year. We look around hard for inexpensive bandwidth, and then make sure we always have redundant network providers and redundant paths. That's all blended into the 2/10ths of 1 cent per GByte cost basis.
Part of this is complicated by any "spikiness" in network traffic. Network providers sell bandwidth based on things like "95% of peak traffic". So the 2/10ths of 1 cent per GByte is based on an assumption that overall the outbound traffic is perfectly flat. It would be more expensive to Backblaze if "most days" the outbound traffic was 200 Gbits/sec but once every 7 days it spiked to 400 Gbits/sec for 25 hours then dropped again. In that case Backblaze is hit with a bill for 400 Gbits/sec sustained for the entire billing period (a month), but customers did not use it up in a "flat" fashion so it eats into our margin.
Luckily, since Backblaze supports hundreds of thousands of different individual customers, it appears mostly "flat" to us in aggregate. The five week days tend to be about 10% higher network use than the weekends, but other than that it is all super flat and regular (so far).
Amazon, for example, starts at $0.15/GB. It drops down with more volume, but it's obviously gouging to a ridiculous level.
Which has ALWAYS baffled me about Amazon's pricing. I would have thought they price it the REVERSE where it is low price at first, and rises. For example, start with a "low introductory rate for the first Petabyte" to get small businesses addicted to Amazon services while Amazon still makes a (solid) profit, then when the small business grows into a large business that requires more than a Petabyte of data Amazon would increase the price because: 1) scaling is hard and does cost money, and 2) a company looks into the future and says "heck, if we reach a Petabyte we're making so much money we can AFFORD the higher Amazon rates", and 3) as a company grows quickly they have so many other problems they won't prioritize getting off Amazon, especially if Amazon is scaling well for them.
Related side note: scaling really does cost extra money. For our online backup business, almost no customers have more than 10 million files in any one "backup" (each backup is "per laptop") so as programmers we can use any data structure we like to browse those files, a flat list IS FINE. But for B2, a large customer might have 1 billion files in one "bucket", and so we have to be a HECK of a lot smarter about how we organize that data so that the customer can still sign in with a web browser and view them all. For Backblaze, that meant moving up to a clustered hash database called Cassandra, and deploying (so far) about 50 1U computers, each with two CPUs, each with fast SSDs and a layer of complicated code to manage all of it. So the scaling requirements of the Personal Backup client required 0 extra servers, but at B2 scale it requires an additional $175,000 worth of equipment. Scale costs extra money.
https://www.youtube.com/watch?v=-i6wvix6buI
> we realized that we didn't need to be charging that much for downloads
To be totally clear, Backblaze can keep going lower. Backblaze buys bandwidth at about 2/10ths of 1 cent per GByte right now, and it drops by maybe 10% per year. The only reason we don't go lower faster is that we are trying to "ease into" this. Internally we are afraid of hitting a tipping point and attracting too many customers using the outbound bandwidth too quickly. The Backblaze network capacity is currently built out for an extra 500 Gbits/sec or so of outbound bandwidth that we aren't using (ready to sell to customers), and it takes time (a few months) to provision more bandwidth. Sometimes the bandwidth providers have to trench to our datacenter which takes like four months.
So if we drop the price too low too fast and enough customers show up that would require a total of let's say 600 Gbits/sec capacity in less than 30 days, we might not be able to keep them all super happy and we might get a black eye. So we want to "ease into" this to continue to provide top notch service while we figure out how much of a market exists for inexpensive outbound bandwidth. The ALTERNATIVE strategy would be to massively over provision, but if not many customers show up then we have wasted money. Since Backblaze is profitable and not funded by Venture Capital, we do not like wasting money. :-)
It's very easy for people who have little to no expertise in a domain go on about how X company should do Y and it wouldn't be a big deal, when lots of times it would be.
Most businesses over a certain size figure out their "COGS" (Cost Of Goods Sold) which is what it costs to produce and sell their product if the scale goes through the roof. Then they decide what the "margin" is (the mark up that we get to keep ourselves). And there is your price. Some premium products are based on super high markups/margins, but in competitive areas the margin is usually lower than 50%. The MOST competitive areas (like food in supermarkets like Safeway) get as low as 20% margin. Now, the COGS doesn't include salaries of certain people in the company like the software engineer's salaries to produce the service. The reason those aren't included is they only write the software once, then you can sell it an infinite number of times. So the margin/markup goes into paying the software engineer's salaries. So the margin isn't ALL "profit we keep in a big swimming pool and swim around in".
Programmers like me learn all of this in the first year of running their first startup company, so it isn't a secret. :-) I think the business majors learn in their freshman year of college.
The reason we always have two separate bandwidth providers is that any one provider is basically guaranteed to be offline at least 5 minutes a year, and there is a 30% chance any one provider will be offline for two hours once a year. But it is highly unlikely TWO providers will be offline at the same time at any point.
We have three separate datacenters, but any one file you store will be in one datacenter. We have two datacenters in the Sacramento region (about 30 miles away from each other) and one datacenter in Phoenix. So depending on which datacenter your file is in, your network path might change to use a different provider. We are also adding brand new datacenters in 2018 in new regions (like Europe) so that will probably mean new and different network providers.
[1] https://www.backblaze.com/blog/our-secret-data-center/
I have 4TB of media to backup. My ISP will throttle/penalize me if I download/upload more than 1 TB per month, assuming I do absolutely nothing else with my internet connection. Even assuming I devote 80% of bandwidth to just the backup, it will take me 4-6 months of metered transfers, while paying for their service, before I actually have a "complete" backup with them.
EDIT: @admiralasshat: If you want to send me a drive, I'll put it on a GB connection for you until it's backed up.
EDIT: @Yev: Fair point :) Was trying to be helpful.
EDIT -> I can't recommend the above for security reasons :P
EDIT -> Hey, if they're up for it...have at it :P
Also, at the beginning the "no seeding" thing was kind of a gate to folks who had a lot of data but limited bandwidth. Our theory was that you should never be afraid of uploading your data, you should have the bandwidth to match. That was bout 10 years ago though, so things have changed - but it's still not a priority project for us - especially considering that since Online Backup is a flat-fee, by definition almost anyone that would use the "seeding" feature would not be a break-even customer for us.
This would be my sales pitch to your superiors:
1) My 4TB is not really expected to grow much beyond this point. It's the accumulation of a lifetime of hoarding, and the year-to-year addition beyond that will be in gigs, not TBs. So after the first year, it's basically just cold storage. If you guys can profitably store 4TB in cold storage for $50/yr, it should be profitable after the first year.
2) The guys who have enough media and are technical enough to want to seed it are also more than likely going to be the guys who make the backup service recommendations for their friends and families.
Case in point: http://nothingofvalue.org/backup.html
That's the website I created for friends, family, and coworkers to read, because I got tired of explaining the same stuff to them every Christmas. I very nearly bought the Crashplan Family plan at one point just to make it easier for them--I'm not glad I didn't. But I guess my point here is that the power-users, as irritating as we are, are going to be the ones who spread your product by word of mouth.
3) People with lots of data who take backing up seriously generally don't like switching vendors unless we have no other choice. I don't want a provider that I feel like I'll "outgrow" when I get too much data and be forced to switch to a more expensive plan or to another vendor. So while a user with 100 GB or less might be with you guys for a year, he will probably move to another vendor if they can do it cheaper, since he doesn't have much data. The hoarders, on the other hand, just want to know you'll keep their stuff safe and won't jack up the price on them at some point, and will otherwise be willing to stick with you forever.
Just my $0.02 as a potential customer.
Thanks for your reply. I can certainly investigate that route, but the kinds of ISPs that implement these caps aren't exactly known for their magnanimity to begin with, so I'm not optimistic. Assuming they don't, I would get charged about $10 / 50GB overage, which would work out to 3000 / 50 = $60, which is the yearly cost of your service. So my dilemma is that I'm either paying for about half a year before I get a complete backup set with BB, or I'm effectively paying double the service price just to make sure the data all gets to you guys in a timely manner. For that price, I could just as well just shell out the extra dough to stay with an upgraded Crashplan Small Business account.
I hope you guys will reconsider or re-examine your position on seeded backups. I don't think I am being unreasonable in wanting that to be an option, nor do I think I'm alone in having somewhere north of a terabyte of media--any amateur photographer or video editor probably has a similarly-sized dataset. If unlimited bandwidth were available, this wouldn't be a concern, but, I don't have the luxury of being able to choose a different ISP. And with Chairman Verizon heading the FCC, I don't see that changing any time soon.
And yea, I'm personally hoping that the ISP situation gets a bit more friendly towards consumers!
You forgot to multiply by $10, so you're actually looking at $600 for 3 TB of overage
Solving the first is definitely within Backblaze's capability if they are prepared to make an exception.
https://aws.amazon.com/blogs/aws/send-us-that-data/
https://aws.amazon.com/snowball/
I have this idea in my head that some people are digital pack rats and want to save every movie and series they pirate off the internet in case the apocalypse happens and they desperately need to re-watch an episode of Friends.
Is it uncompressed video? (Why would you back that up to the cloud?) Is it family media? (How many baby pictures did you take???). I honestly have no idea how someone collects 4 TB of data that is worthy enough to back up.
By comparison to some of the more serious amateur photographers that I often travel with, I don't shoot a lot. They may shoot 10,000 images in a week.
Here are some of the things that I have that add up to my multiple TBs of data I want to keep around.
First, I have every single school assignment from growing up (30+ years ago, so it doesn't take up much space). These are kind of fun to have. I can show my kids the papers I wrote when I was their age.
A lot of old converted 8mm tapes from when I was a kid. I used to be really into making claymation videos, so I have hours of videos that I converted around 15 years ago into digital formats. I could probably reconvert them to a more modern codec that would take up less space, but they are still multiple hours. Currently takes up around 100 gigs.
Every digital photo I have ever taken. Over 20 years, this is a lot of photos. A few hundred gigs there.
Tons of videos of my daughter. A hundred gigs there, growing all the time. Maybe I take too many, but it is really fun looking back and remembering when she was little.
Every programming project I ever created, going back 25 years. Fun to see my progression. Not a ton of space, but many gigs worth.
A ton of old games that don't exist to download. Old shareware, commercial games, etc. I created CD images for the all the ones I had on physical media, so I don't have to lug around the CDs anymore. I like to be able to play games I played years ago every now and then. This takes up a good chunk, and is probably a few hundred gigs.
Ripped DVD movies and TV shows. Bought many over the years, and don't want to have to find and/or rip them again if I lose the data. This is hundreds of gigs.
Countless other minor things (saved games, documents, etc).
I like all of these things. If I have a house fire, I like knowing I wouldn't lose all of that. Since it is so cheap to maintain a backup of it, why not? Yeah, I wouldn't be DEVASTATED if I lost all my old DVD rips, but for 60 bucks a year (and that is the same cost whether I back those up or not), why not keep them backed up?
"Sorry, Mom, I can't help you watch that Mork and Mindy collection on your bedroom-TV Roku. Someone On the Internet informed me that my digital collections are not worthy of keeping, and I dutifully deleted them."
"No, sweetie, we absolutely cannot share your childhood with our daughter and watch my digital rip of Fraggle Rock. Good parents delete their abhorrent digital media and pay HBO Go $15/month for the privilege of viewing it!"
"What are you thinking, keeping those hundreds of CD's and Vinyl's we ripped together on your iPhone, Dad? You can chuck all that crap that we actually own and pay a monthly fee to Spotify for the privilege of streaming it! Unless, of course, the publisher decides to pull their collection off of Spotify, for whatever reason."
"Hey, guys, now that we're all moved out of the house, we need to talk about the Nintendo 64. Mom and Dad bought it for all of us, so I think it's only fair that we keep the physical console at their place so that no one of us will have exclusive access to it. As for the games, I think each of us should be allowed to take the cartridges with us that we individually paid for and/or receives as gifts. So that'll be me taking Super Mario 64, Majora's Mask, Super Smash Bros, Star Fox 64, and NBA: Hangtime; Steve takes Goldeneye, Ocarina of Time, Perfect Dark, and Donkey Kong 64; and Fred takes Mario Party, Pokemon Stadium, Mario Kart, and Banjo Kazooie. Yeah, I know, I do wish there were some kind of legal backup method that might let us all enjoy the games we collectively owned. Oh well. Maybe Nintendo will release an N64 Classic and we can each pay $80 to play the games we already paid for. Assuming the games we own appear on the collection."
A second classification is data that isn't changing -- you can create a couple backups of that media, store one in a bank safe deposit box, maybe ship another backup to a family member across the country.
Then you have recent acquisitions and ongoing projects -- those are what you would sync daily to an online backup service. So you still have offsite backup of everything, but by splitting up the categories, you end up not needing to sync that much online.
All data is worthy of Backup. I have about 40TB of Home Storage that is backup;ed on a set of encrypted disks offline. (Total RAW Capacity of all my storage in my home lab last a checked was about 150TB)
I do not go Full 3-2-1 with it, but I do have a duplicate copy of all my data with about 2TB in a Full 3-2-1 Backup, I would say about 750GB to 1TB has 1 copy of the 3-2-1 in a "cloud" vendor of some kind
Should be able to get max throughput and cost will be minimal if only have the VPN for the time it takes to seed.
VPS's are probably a no go as they have all moved to SSD's with low sized volumes. Even with Digital Ocean where you can add block storage you are looking at $100 for a week for 4TB + compute and bandwidth. AWS you are looking at $368+ just on data egress.
hetzner.com server auctions look the best option. Current cheapest is €31.93 per month for a Intel Core i7-3770 (CPU-B 9445) with 2x 3 TB drives and 16 GB. Probably simple enough to use lvm and jbod on the drives to give a single 6tb volume. No egress charges but reading they throttle if you go over 20,000GB a month so a non issue.
No setup fees/contract on the auctions although reading I think it's 30 days/1 month notice to cancel so would need to cancel as soon as ordered.
My plan is to probably move to Git Annex https://git-annex.branchable.com/, use my local NAS as the main backup, maintain a copy of media / photos / docs in Backblaze and duplicate files I can't afford to lose to S3 also. Looks simple enough with Git-annex and lets me maintain multiple copies based on cost / importance.
> Are Backblaze planning on offering solutions for europeans?
About 15% of our business comes from Europe today, and we collect VAT tax accurately for all Europeans as required by European law. So if you are willing to encrypt your data in Europe and send it encrypted WITHOUT the encryption keys to store it encrypted in B2 (zero knowledge) in the USA, then Backblaze would love to have your European encrypted data.
If you are asking when Backblaze is going to open a European datacenter where you can guarantee your encrypted data sits on disks inside of Europe, we are hoping to open that datacenter in 2018 (this year).
I had my eyes on B2 because the storage costs were within budget but didn't switch our backup to it because if we ever needed to download it, well, what then, it was too expensive.
I will upload our 100k / 2TB or so photos the weekend. Awesome.
Now I am reviewing this... you are now on BunnyCDN level of download pricing. Are you like a CDN or does every European download need to cross the Atlantic?
Cloud providers haven't done a great job of explaining this so far, but Google Cloud's Network Service Tiers product is a good attempt at why they charge more for bandwidth, and how you can get it for cheaper: https://cloud.google.com/network-tiers/
That said, cheap (and presumably slower) bandwidth may be the right target for B2, given that it's customers are probably using it for backups and private storage rather than, say, serving a website serving photos to millions of visitors from around the globe.
So comparing AWS or Google transfer costs directly to B2 isn't necessarily a fair comparison, depending on your needs.
This is my first time seeing Backblaze and it looks appealing, does any one have any opinions on using it vs glacier/s3?
For context I currently have less than 1TB of data I need to store and I don't expect to upload more than a few GB per month.
Also would I use B2 or the unlimited personal storage?
The only problem I have is that my Time Machine backups change too much -- for, say, 400MB of data being backed up in time machine, it seems like the deltas (and what is synced to backblaze) is something like 4GB. I'm using the CloudSync functionality from Synology -- I think the issue is with Time Machine, not with CloudSync, though.
What this means is I'd regularly exceed my data cap if I let me CloudSync run all the time. As a workaround, I've scheduled it to only run between midnight and noon on Mondays, which results in syncing less data (apparently the deltas must contain a lot of duplicate data that doesn't have to be synced up to BackBlaze when I only do it a few times a week).
Of course, now my problem is I'm afraid that I'm risking the integrity of my backups because the schedule might cut off my sync in the middle of a run. Since I leave my home computers off during the day, though, there shouldn't be new deltas occuring as I'm syncing to backblaze, so once it catches up (and 12 hours is enough to catch up), there is no more activity.