Definitely quite a few companies are attempting something in this space. I think the one that will win will be the one that actually has photographers as founders and understand the need.
Unfortunately it is quite a niche and most photographers are gadget hounds and don't mind having 2-3 drobos (including an off-site one).
If you want to avoid having any backup devices yourself, would you need to upload to two or more Glacier-like services?
I also trust Amazon to stay in business but mistakes can always happen (not just hardware but ID theft, card expiry while you're on a 6 month hike, etc) and unless you're prepared to say "losing these is OK" using one service would be risky.
Offsite tape backup can approach that price, but I don't know of any exact alternatives. It would certainly fulfill the requirement of having two different media types and providers.
The large part of this post for me is about the local thumbnail exports of the RAWs so I actually know what files are what without having to go just by album names, dates et cetera.
Backup is about having several copies. If you delete files after you uploaded them to the cloud, you don't have a backup - with all the problems that come with it (say, if you delete a file by accident there will probably be no way to restore it).
If your files are important to you - you need a back up. If not - just get rid of them in the name of simplicity :)
PS. Also, uploading many gigabytes over a typical ADSL connection is just painfully slow... even local NAS is too slow over WiFi for my taste, and if I have to have a wire - I can just as well connect an external drive.
I only used Glacier in this example but I did mention how it could be tied to additional cloud services. The thing is that not many like cheap long-term Glacier alternatives exist yet. Rest assured when they do, this ideal app would have them in the settings. :)
I wanted the benefit of Glacier plus backup functionality so I went with Arq. It encrypts everything and supports Glacier. Works really, really nicely.
I backup offsite via CrashPlan which keeps files even after I delete them. I sent them a hard drive for my initial upload. Now, I've got about 800M backed up. I haven't yet had to get it all back :)
Yes, PLEASE. We run a wedding photography business, and our numbers are something like this:
- 40-60gigs shot in a weekend, need to offsite the RAWs as soon as possible. In six or seven years we've never needed offsite retrieval. Fine for it to be slow.
- Only need around 100-200g of raws locally for jobs in progress
- Once processed into final jpgs files are approx 8-10x smaller. Offsite these as well, and need access to last 1-2 yrs of jpgs randomly/sparsely w/in 1-2 days of a print order coming in
I definitely think some well thought out service that uses maybe a hybrid of glacier and S3 could be really awesome. Especially if the pain is taken out of getting the bolus 40-50g of photos "uploaded" in the first place.. We'd happily pay for a service that just sends us an external drive and a postage-paid box.. we drop it in the mail after a job and get another drive sent to us right away.
[EDIT: Occurs to me you might wonder what this looks like today. Today we get home, and immediately back up onto an 8-bay Drobo Pro (http://bit.ly/1diZUHD). It's a fancy proprietary RAID-like system. I think ours has 7 1TB Drives in it currently, and it's at about 70% capacity. Then we back up the same jobs to an external hard drive which gets driven offsite (used to be my office, but now that I work at home it's the in-laws place) and copied to a hard drive for permanent offsite storage. Once that happens, the cards from the camera can be cleared. After processing, the final JPGs get copied to the drobo and make it into the sneakernet offsite process as well.]
Your happens today is very similar to mine. I'd say that I need additional offload of the RAID array (drobo in your case), while little is needed to be changed on my processing laptop
> Occurs to me you might wonder what this looks like today
I finally hit a sweet spot of performance, archive accessibility, and offsite safety, by using internal SSD for latest import workspace, direct attached WD Velociraptor (Thunderbolt) for current projects (can reconnect between Mac Mini server and laptops as needed), and 2 direct attached LaCie 4big Quadras (daisy chained via FW800) for archived projects (up to 32 TB).
Interestingly, because these are all direct attached, BackBlaze will happily back them up offsite at ridiculously low cost if that's your thing. Initial backup of several terabytes will take several months, though.
For fast cheap offsite backups, partner with a colleague with similar storage needs who uses the same ISP via the same head end. Just backup to each other. You can enjoy near LAN latencies at maxed out link speeds. This trick will save you both time and headaches.
Finally consider a product like the CyberPower OR2220PFCRT2U (this one also blends nicely in an AV cabinet) to keep an Airport Extreme and Mac Mini along with all that storage running for about 80 minutes of power outage.
The idea of partnering w/ someone using the same ISP is interesting. It's really offsite that's the whole challenge. If it takes three days to get 60g uploaded (setting aside the nuisance of saturating our bandwidth for doing things like making Skype calls), then I effectively have close to no solution.. because during a busy season I have at least one job exposed for half of most weeks. House burns down, and I lose recent work.. which is the most important work since it hasn't been delivered.
Really, I need something that gets a copy of the images away from my home within a day to have something I'd be comfortable with.
Really, I think it makes sense for someone to prepare for the future and assume this will be feasible for upload in the not-crazy future.. unless more hobbyists keep deciding we need exponentially larger RAW files.. It's a crazy race :|
Would it be feasible to simply dupe the drive and find a secure off-site place within driving distance to store it or a storage machine, and then mail drives to glacier (Amazon will import a HDD via mail for around 100USD) as the storage machine fills?
effectively this is already what I do, except that you added a step of sending the data to AZN which doesn't help me any if the data is already offsite. Once I've driven somewhere, we're done.
I worked at a wedding photography/videography place a few years ago and we did a lower tech version of this. The owner had a storage array at the office and at home, and he'd cart an external back and forth daily. This way there were two copies of all work, and three of the most recent: one at the office, one at his home, and the most recent work (that which hasn't been delivered) on the portable drive.
If you're using a newer MacBook Pro I've seen some people swap out the disk drive bay for a 2nd SSD. This doesn't sound like much but that allows them to run a RAID 1 setup on their laptop (1 drive with a mirrored second drive).
I import from cameras to a couple NEXTODIs on the road, then from the NEXTODI to the SSD. I don't delete from the NEXTODIs until images are all the way through to offsite backup. The SSD is not RAID, and worse, the Velociraptor runs in RAID0 for raw (pardon the pun) throughput, but I always have at least two copies on diverse systems.
The idea of a P2P backup solution is interesting, more so if you can hook up strangers into a massive backup network.
Of course, the challenge is to make sure the peers are online when you need the files, and for this reason the network may need 5-10gb of storage for 1gb of data to duplicate the content across multiple users.
Yeah, probly. But I want a company to wrap all of it up in a nice predictable package. Dealing with understanding AWS pricing complexity alone is enough of a service to spawn healthy-margin companies.
Good questions. So today, the way the math works out is we can maybe fit 25 weddings on a 1TB we pick up for 80 bucks.. so that's about three bucks, six to do it twice. We think of it as a part of the cost of a particular wedding. You could charge me 10 or 20 bucks to archive before I'd even do the math. We likely spend more in gas for most weddings. Hell, I think each wedding costs us four dollars in business cards.
If you could get me to _really_ trust your service, I'd consider up-selling permanent archives to our clients. Lots of our clients would pay 50-200 bucks for the re-assurance, I'm sure. I don't know that I'd want the ever-growing liability though (right now we contract for 1yr of backup/online-gallery, but keep everything permanently without committing to doing so).
One time or yearly fee doesn't make a ton of difference to me, but with other services we've definitely been tempted into fixed "Unlimited" pricing so that we don't have to think hard about math, or end up tempted into picking and choosing what we backup. Simpler is better.. Managing a store of assets where some are backed up and some aren't sounds like a really bad accident waiting to happen.
Unfortunately, that math doesn't work out very well. If you were to implement it yourself, those numbers sound great. But to do it as a business, I think the math there is too cut-throat.
4 TB of usable space divided by 60 GB times $20 is $1,365. That's the kind of thing a consumer or prosumer could do on their own, but it would be pretty hard to make a real business out of it...
EXTRA CREDIT: Anyone who considers tackling this and finds themselves interested in the niche of wedding/event photographer should look at combining this service with shopping-cart/gallery solutions that currently exist. The marketplace for those is fucking awful. Feel free to email me, but googling will get you the big players. I just don't wanna bash them by name in a public forum. All well meaning folks I'm sure, but every one of them are terrible flash-based, no-mobile software. Do backup and gallery/print-ordering management correctly and we're a customer at four figures per year.
I appreciate the spirit of your suggestion, but might I suggest that this would be the exact wrong move for someone trying to build the ultimate long-term cold storage for photos?
Simply, they are two radically different businesses with entirely different feature sets. You might very well need both.
It would be great to build the backend and API that supports the ultimate long-term cold storage, and then use it to make the ultimate wedding photography business application.
They're not radically different—one just uses the other. You could easily do both and continue to expand the use of the backend to enable useful applications. It's ridiculous to shun this specific consumer model just because it happens to be only one way you might use the original idea.
Yes, this. Makes crazy amounts of sense. Can be two vendors, I don't care.. but whoever does cold-storage exposing an API and having a business model that supports enough margin for the other business is great, and seems totally feasible.
Most (serious) safes are capable of withstanding a fire or a flood.
I don't have near the volume of pictures that you have (around 1.5 TB) but here's my setup:
- one disk attached to the Lightroom computer, with all images on it; it's local, so it's fast
- one NAS (NAS1) with a 4-disk RAID that backs up this drive (3 TB capacity)
- another NAS (NAS2) with just one 3 TB disk that backs up the other NAS
On NAS2 I rotate disks; disk A is in the safe when disk B is in the NAS.
I think my setup is quite similar to yours, except driving is replaced by "going to the safe in the basement".
If a disaster happen I should have at least the disk in the safe intact, which holds the data since the last rotation; and since backups from NAS1 to NAS2 are quite fast I try to have both disks pretty much in sync (after an important shoot I back up to disk A AND disk B).
Of course I won't know if this setup is good until my house does burn down (I hope I'll never know!) but it does give me peace of mind, which is at least something ;-)
I'm in a similar situation as the OP...an amateur but avid photographer with terabytes of RAW image files.
Currently, I have several multi-terabyte drives that I erratically back up to...I just got a 4TB drive that I'm going to try to backup everything I've ever shot in the last 3 years...
But I do want to move it to online storage...and I think this requires triage. For nearly every photoset, I've quickly gone through them with Lightroom and starred the ones that I kind of like, and then for maybe 1% of them, have taken the time to fully process, label, and upload them (in JPG form) to Flickr.
So for online storage, I think what I'll do is write a batch script that dumps all the unstarred RAW files as JPGs, because they chances that I'll ever need these photos in RAW is very slim, and then upload them. For photos that I've given at least one star and/or taken the time to properly edit and categorize, I'll send up the RAW file along with the processed JPG.
If I have about 3TB of RAW files...and maybe 10% of those are RAW images that weren't disposable...that's 300GB right there. And then the remaining RAW "keep em just in case" files would end up being compressed from 24MB to about 2-3 MB (let's say 15% as an average)...so 400GB.
700GB is still quite a big footprint. However, the biggest problem will be...let's say I give up local storage all together and rely on the cloud...what's the best way to browse any photoset at any arbitrary time? When it's all local, it's trivial to pop open Lightroom and do some quick browsing and filtering. For online storage, I'll have to put a ton of work into proper folder-naming, at the very least
Your last few sentences are exactly why I wrote this post. I want everything stored in the cloud but ALSO have a decent way of browsing them locally. The idea is to just store tiny jpegs (you can set how big they are, but something like 500-1000px wide likely, just for you to get a sense of which photo this is and allow you to remotely delete, retreive, etc)
This is basically what I need for Office Snapshots. I keep all of the original photos submitted so that I can use the full images in the future (if need be) and would love to store them somewhere other than my laptop.
If I could browse locally with smaller images (as you describe) like I currently do with Picasa (galleries, tags, etc) and then have the ability to download the original images when and if I need them, I'd be in heaven.
All of this would be trivial for Apple (or someone creating a plugin) to implement. Apple's flagship photography software aperture already allows the user to work with previews while keeping the mega-huge raws on an external device. Instead of an external device, Aperture should just allow the user to point to a service in the cloud.
Of course, Apple's iCloud is very little like an actual usable cloud for most pro users.
Exactly. What I really want is something that Aperture is aware of. I want to be able to click "Archive" with a Project and have the local copy deleted. Yet, it's still in the catalog/library offsite, so if I want it, I click "Retrieve" and can pull down the set and start fudging the RAWs. Or, if I want to work on just one, I can browse through and work on it as well.
Here's my rant, take it for what it's worth :) I keep 4 types of photos:
1.) Personal: This doesn't mean I don't delete, but I generally tend to keep photos even though they might not be really good photos, especially if I only have a few of that day or person. I know I might enjoy looking at them decades later anyway, a blurry photo is better than nothing.
2.) Archival: like taking a photo of a flat before moving in. These aren't very many, so I tend to keep them no questions asked.
3.) Siblings of Keepers: You could call these "proof I really made the keeper photo" photos. Let's say I come across an animal I never photographed before, and like one of the photos enough to publish; then I'll still keep the others around. Or keeping a wider crop in case you change your mind later, etc.
4.) Keepers: here I try to be as harsh as I can. Photography is about painting with light for the pros, for me it's mostly about selection, that's the part of it I enjoy most. I walk around, and from the thousands of things I see each day, only a few make me take out the camera. Then I select keepers (partly before the images even leave the camera), and even years after I might revisit some keepers again, decide that my standards improved, and delete them.
I don't want to tell others how to pursue their hobby, but I think the web is full of galleries not even the owners look at, and that generally, a lot of people need to learn how to delete. The same goes for blogs and whatnot, the idea that everything has to be written on this indelible roll of toilet paper seems silly.
On one extreme, there is being OCD and deliberating endlessly what to keep and how to rank it, on the other there is just making more stuff to throw behind yourself on a pile you never investigate, because it's too awkward to wade around in. Productivity lies somewhere in the middle I think.
A service would be great ... but you might be able to "roll-your-own" with some basic scripts to handle the conversion (down-sizing) and backup sync - and something like Koken to handle the sharing with friends & family.
For back-ups - just buy a huge disk or set of disks, and keep 'em off-site wherever you have a trustable location (friend, family, work - if allowed and bandwidth isn't an issue) and use CrashPlan's free software to sync a copy to the secondary location. It's free after the initial expense of the drives and gives you encrypted, but physical access to the content in case of an emergency. You could find a friend with the same problem, and just agree to dedicate a certain amount of disk-space to each-other for this purpose.
I think Filosync (www.filosync.com, currently in beta, I'm the founder) would work for this. You could have many projects in your cloud and only sync the one(s) you need to a given computer. We'd just need to add a feature that stores "previews" of the items in the folder so you could browse without downloading.
But it uses S3 in your AWS account, so would be cost-prohibitive for a consumer storing terabytes.
This is exactly what I was thinking about... I want cloud based storage which I am not locked into with a decent UI , way to browse and basic image management (something like ThisLife for the fronted)
I think it's possible to have a either S3 or GDrive based system that has a Angular.js (or some other JS MVC) that can operate without a backend for very simple browsing features and connecting to the S3 via apis.
I still prefer local storage and self-hosted storage, but what I'd like is a good plugin for Lightroom to manage cloud storage and multiple, versioned USB drives (or network locations) for images, from within the Lightroom interface.
(for local storage, once you go beyond individual USB drives, I'd go with a Synology 1813+ NAS ($999 chassis, 8-bay) or an Areca ARC-8050 thunderbolt raid ($1499, 8-bay).)
I've always wanted to work on this, but didn't want to work on the stereotypical nth photo sharing/management startup, especially with other projects taking up my time.
"I open the folder where Lightroom imports are stored and drag these shots to the RAWbox (fake name for this ideal photo app) desktop application."
You shouldn't be touching the directory structure. This tool should be either a plugin for Lightroom/Aperture/Photoshop or an entirely separate application.
I would lean towards separate application. It could be a background agent that detects when new files are added, would work for any known editor or user-defined directory.
We want to take backing up, sharing, and monetizing and make them very simple and enjoyable to do. We've started rolling out slowly already, our goal is to be open to the public within a month or so.
I hope your business model doesn't rely too heavily on the "monetizing" aspect. I know there are lots of photo hosting services with a "sell prints" option, but I get the impression not very many people actually use those in enough volume to cover the hosting costs. I'm imagining a non-pro track that's targeted at the enthusiast who just wants backup and sharing, but won't actually sell any prints. I'll be watching.
This is good to hear. BTW, it might help if you had a little more info on your landing page about the service. For example, I can't tell if this is going to be aimed at true pros who are willing to shell out a decent amount of money, or if I'm going to be able to afford it for my own pro-sumer needs. Having to sign up for a mailing list and wait in order to potentially find out that I'm nowhere near the target audience is frustrating.
Heh, it is unfortunate that there's very little info there. We put it up a while ago and have been so focused on actually building the app that updating the website seemed unimportant. :)
We applied to YC with exactly this idea. We made it to the interview, but were turned down. I wrote about that experience: http://tghw.com/blog/pull-hard
We pulled our bootstraps hard and tried to get it going, but soon realized that there's a pretty big flaw with the idea: most photographers have no idea that they should backup their photos. They know hard drives crash, but they never expect it to happen to them.
This meant that we'd essentially have to become insurance salesmen, convincing people that bad things will happen to them.
Yeah, I'd expect the market for this is more niche but could be high revenue-per customer. As a wedding photography studio we'd pay 100$/mo in a heartbeat if this was done well.
[Edit]
Also note that OP wasn't really just pitching "backup". He wants you to reduce the footprint his images take up locally. I think the messaging to casual and hobbyist users probably ought to play on this. Note Loom's taglines "more room to play". Backup is probably boring for people not under the pressure of liability like pros are.
Would you be willing to open source your code? I am running out of hard disk space as well for my photos and have been thinking of writing coding up something very similar to this idea _when I get time_.
But I would be willing to contribute to an open source project that does this for self-hosting my photos etc. Contact me if interested, email's in my profile.
> For this experience, we spent less than $1,000. Compared to an entrepreneurship class at any college, that's quite cheap!
My undergrad was in USC's entrepreneurship program. One of the points they always hyped on was whether or not you were "selling vitamins or vicodin", so to speak. As you learned through Snaposit, most people when making chit-chat will be supportive of your venture, but you won't know how they really feel until they vote with their wallets.
Most of the professors were older guys who didn't get online startup culture, but one of their favorite tricks was to put up a landing page detailing a concept they were interested in, with a buy button. They'd spend some AdWords to get people to the landing page and measure the conversion rate. If enough people tried to buy to justify the service, they'd go out and build it.
Professional photographers know they need to backup their photos.
It looks like your product targeted consumers, but professionals seem like they'd be a better market. And it should have been priced way, way higher; the more it cost, the more secure it looks. I'd have a hard time trusting my money-making business assets to a $9/month service.
Of course, I'm not a professional photographer nor do I have your experience in trying to build this business, so who knows what my opinion is worth. :-)
The problem with the term 'professional' is that the word is ambiguous - do we mean 'people that live off this' or 'people that know what they're doing'? I have two professional photographer friends, and neither have the slightest clue how to protect their data. They live off it, but never grow out because word of mouth is karma and you can't lose someone's wedding photos. These are people that have a duty to protect other people's memories, and are leaving it to chance. They are everywhere.
Actually, once we started getting into the market, we found that most do not know they need to do regular, offsite backups.
Consider the some examples:
* An older photographer who grew up in the days of film who has just transitioned over to digital.
* A homemaker who is trying to add some much needed income for her family, with a passion but no formal training.
* A well-off software developer who moonlights as a photographer for fun and a little extra money.
Of those, only the third has been exposed, formally, to correct data backup procedures. The others are likely to keep their photos in folders on the desktop of their laptop.
Furthermore, photography is not a very lucrative profession for most photographers. That limits the amount you can charge. Our original pricing was much higher, but we realized that would be problematic, considering the average photographer brings in around $30k per year.
"Professional photographers know they need to backup their photos."
A homemaker or techie moonlighting for extra income aren't really 'professional photographers'. They may take photographs that look professional, but it's not the same thing.
One might say that professional developers use version control, then have counter examples of "well, this web design student doesn't ever uer version control, but she's the only person working on the code, so it doesn't matter". This is not a professional developer, even if she is capable of producing results that look like they were done by a professional.
A professional, imo, is defined as much by their habits and practices as by the output.
Absolutely. The advances in digital photography has opened it up to almost everyone. They're not doing Disney's next add campaign, but they are doing several family sessions per week, and they are bringing in money.
A friend of mine had her wedding photographer lose 6 months of data in a hard drive crash. She should have known better, and may have, but she didn't do proper backups.
I was a professional photographer for a couple of years (as in, paid most of my rent with it until web dev became a more lucrative venture) and assisted some VERY big and experienced photographers.
Everyone I know backed up on-site, using manual methods as and when they remembered. I did actually set up a good automated backup solution for one of them, but there's definitely a limit to their interest. Remember they're making pretty much all their money off commissions, and are only keeping backups out of some sense of pride/responsibility.
Another blow is that, at least here in the UK, we're only starting to get the kind of upstream bandwidth necessary to make this level of data transfer feasible.
So, it's a brilliant idea and, as long as confidence in cloud storage grows, as time goes on the market will grow and grow. I'm just not sure it's quite there yet.
Or maybe your company's problem was this (from one of your blog posts):
> It did this by compressing the photos to full resolution, high quality JPEGs, which saved anywhere from 50%-90% of the size. The downside of this, of course, is that photographers who shoot in RAW would only have a JPEG copy backed up. We figured this was a reasonable compromise. In a catastrophic event, wouldn't a photographer rather have a JPEG copy than nothing at all? It turns out the answer was no. But we didn't ask that question, because we thought the answer was so obvious.
I am no photographer... but from what I can see in this thread, what people are looking for is a solution to store RAW image files. I am sure photographers can find already good solutions to store good-quality JPG copies of their files.
I believe Photry (www.photry.com, currently in beta, I'm the founder) could be a possible solution in the long run. Since we started, a lot has changed on online photo storage market.
With that in mind we've started to look more on how photographers with larger photo volumes could benefit from our service. From the feedback from few photographers we've already included some features (RAW photos with smaller JPG thumbnails, workspaces and personal sharing for example) that will make the day to day workflow a bit easier. Currently our goal is to build something that would fit better for professional photographers (public for clients, ordering prints and possibly integrated payment flows, win/mac client + plugins for your favorite photo software).
Pricing wise we are currently not the cheapest but we have thoughts on how to make this better for high volume photographers. If you're one of them then please send me an email (martin at photry.com) and I would love to talk a bit further what we plan to do and if we would fit your storage needs.
Great post. I appreciate the focus on minimalism and hobbyist photographers. I think it'd be an interesting project to take on. The technical side doesn't sound too difficult, plug into Amazon, lightroom/aperture.
While the post mentions avoiding externals I think a handy feature would be to push to both an external drive, glacier, and then remove the RAWs locally. That way you have two copies, so the data is actually backed up. Trusting all of your photography to amazon not losing data doesn't sound ideal.
I've been looking into Glacier, but the pricing in the case where I might want to retrieve a substantial portion of my data is quite complex. As far as I can tell, the headline $0.01/GB rate for overages they quote is misleading. They don't, as you might expect from that language, calculate ([GB retrieved] - [5% of GB stored]) x $0.01 and bill you that.
Rather, they charge you on a monthly basis, according to your highest hourly retrieval rate from any single hour applied to the whole month (standardized at 720 hours). So if your peak hourly retrieval in a month was 10 GB/hr above your pro-rated free quota for that hour, you're charged for 7200 GB of retrieval overage, or $72, even if you retrieved nowhere near 7 TB of data. For companies with relatively constant retrieval this doesn't matter, but for small users, someone who has a large overage one day will be charged as if they were incurring the same overage continually all month.
To put it concretely, if you store 200GB of photos and then do a bulk retrieval, you aren't charged $1.90 ((200-10) x 0.01). Rather, the xfer is counted as 4 hours, so you have a peak transfer of 50 GB/hr. Your free daily retrieval is allocated to the hours in the day you retrieved data, so in this case 10/4 = 2.5 GB/hr free. So your overage is 47.5 GB/hr. Multiply by 720 and $0.01, and the fee for retrieving 200GB of photos is $342. Plus $22.88 in outgoing AWS bandwidth.
Glacier retrieval pricing has a strange and un-Amazony economy of scale. If you're just backing up your own data you'll be charged a ZOMG huge amount to ever download it. But if you're backing up 100 people you pay nothing to get data back.
Yeah, I wasn't expecting to find it here. This style of pricing isn't entirely unreasonable, and you do find charges based on peak monthly usage elsewhere (it's reminiscent of 95th-percentile bandwidth pricing, for example). But AWS otherwise has pretty straightforward pricing based on micro-accounting of usage, so this seems atypical for them.
I seem to recall a Glacier discussion from a while back suggesting that the recovery fees might be covered by insurance (homeowners, loss of business, etc). If you're primarily concerned about doing a restore after a fire, theft, or natural disaster, it might be worth looking into.
Right, you definitely shouldn't request all 200GB at the same time, because it'd be expensive and also because you probably can't download all 200GB immediately either.
Arq (a backup app I wrote) takes a transfer rate from you (guesses your max transfer rate as a default) and then requests enough to be downloaded in 4 hours at that rate. 4 hours later, it requests another 4 hours' worth while it downloads the first 4 hours' worth, and so on. This mitigates the peak-transfer fee.
This idea is great, but it ignores on very real problem. Namely that you still have not addressed the issue of bandwidth. Take for example your 4th of july example. If I were to do that upload the way you describe, I would already be over my monthly bandwidth allotment with my ISP as would many hobbyists.
The US bandwidth situation is the elephant in the room.
There really is no good solution to automated offsite backup when it takes several days of uploading to handle a single day's worth of data capture. To make any internet-backup service work, you'd first have to pitch the photog on switching from their residential broadband package to something more expensive -- if that's even available short of their leasing office space somewhere.
I guess you could try to build out a reverse-CDN sort of network, with local relay nodes scattered across the various ISP networks, that might achieve faster uploads from the user (and maybe not get counted against bandwidth caps) which then use a fat pipe to send that data to larger regional storage nodes.
My company serves thousands of professional photographers (wedding, commercial, fashion, etc.). What you're describing is definitely an issue for photographers but I think getting them to use a tool like you've described would be an uphill battle.
Right now, most of them use Drobos or similar devices. You'd be asking them to add steps to their workflow which is always tough to do.
Also, I think your estimate of 18 mins to upload 6+GB is an order of magnitude off. We have tools that allow people to upload JPEGs and most customers claim it takes longer than that just for 100s of MBs.
My (wild) guess is that Adobe is going to get in this space too. It only makes sense to have your Lightroom catalog in the cloud. Maybe they wont offer an affordable solution for terabytes of data, but I'm not sure photogs want their images in two different "clouds".
I think the key is to do customer development from full-time photographers who aren't hackers. I'd be happy to send out a survey to some clients if you want do more research.
As a photographer with a couple of terabytes of old raws (I back up to Time Machine locally, and offsite at a friend's house occasionally), I'm very curious as to what Adobe will offer in this regard with LR5.
My Lightroom catalog is backed up to a folder in Dropbox, and this works really well.
You can't really have the current catalog in Dropbox because then each action and edit become really slow.
But if you set up Lightroom to do a backup every time you quit the program, and that backup is in Dropbox, you get the best of both worlds (strong backups + snappy edits).
Lightroom backups are really "backups", as they are named by date+time, so in Dropbox I have a history of everything I ever did in Lightroom, not just the current (last) version.
It's also fast because I suspect Dropbox is able to do some rsync magic and only re-upload new packets; my Lightroom catalog is around 150 MB and the upload to Dropbox is almost instantaneous (on a pretty lousy upstream bandwidth).
Of course this doesn't address the problem of backing up RAWs, but as far as Lightroom is concerned I feel pretty safe.
Just to be clear here... Your Lightroom catalog is very important but is not backing you your Photos. (which I know you said at the end. I am just emphasizing)
Your LR catalog contains your photo edits, metadata and history... but is not your original files. Backing it up is very important but is only part of the story. Putting your Lightroom catalog backups on Dropbox is a great and should be done by everyone. (Again just not your active catalog.)
I'll jump in here and share our current problem and how we are addressing it today. Hopefully it proves useful for whomever tackles this project!
So we are an ecommerce company and we do our own photography for products -- and we do a lot of it. We average about 2-3 photoshoots per month, and each photoshoot can be upwards of 40GB before any processing. We shoot raw images and are starting to experiment more and more with video. After we do the post-processing work and start the video editing we can push 70GB a shoot easily.
Our current workflow is like this:
Shoot tethered to a laptop each day (typically each shoot lasts two days). At the end of the day, we upload the files up to our local NAS (Netgear ReadyNAS, just shy of 3TB) which is redundant.
At the end of the shoot, we load a copy of the entire shoot on the local computers that will be editing so that they don't have to work directly from the NAS. Over the next few weeks while they edit the files, they will push the new files up to the NAS into specific folders.
(This is where it gets hairy)
We also have an offsite NAS that is located in the business owner's house (about 6TB, but this one is not redundant). We used to attempt to upload files each night from the local NAS to an offsite NAS via rsync, but the pipe was just way too small. It would take days to upload a photoshoot, and everyone in the office suffered from slow internet while that happened. Now what we do is bring the offsite NAS onsite every few weeks and do a manual sync! Locally the sync only takes a few hours and then that night it goes back home with the business owner.
Now since the local NAS is less than 3TB, we can't store all of our shoots on there forever. When we get close to full on the local NAS, we grab an external hard drive and archive old shoots (typically 2+ years old) to it, and then store that in our massive safe (on-site though). If we need access to just one or two files, we can retrieve them from the offsite NAS, but if we need access to quite a few of them, we can dig them out of the safe.
So that's our workflow. Probably not bullet proof, but it has been serving us well for the last few years. If someone came up with a better solution, we would be all over it!
It is important to be careful with how you try to use Glacier. I have been looking its pricing; It costs $0.01 per GB per month only if you want to upload your data.
Downloads add significant cost if you want to retrieve more than 0.17% of your data in a day.
Downloading 5% of your data increases your monthly rate to $0.097 per GB.
10% increases to $0.187 per GB
50% increases to $0.907 per GB
100% increases to $1.807 per GB
Downloads are also delayed by 4 hours after you make a request.
if I have 2TB stored and only want to retrieve a single 10GB photo shoot one month (or even just a single photo), it would not be anywhere close to 5% (.5% in fact) of my total amount stored. The key point here is that we are dealing with TBs of photos that are there for safe-keeping and they have already been tinkered with and all photo JPGs needed have been exported and put elsewhere. This is just for RAWs and you likely won't need to access them again, definitely not 5%+ of them like this.
Has anyone used Lightroom 5's new Smart Preview feature? It generates lossy DNGs from your full-sized files that you can still work with while your original sits on an external drive. Once you plug your external drive back in it, it syncs the non-destructive changes back to the RAW file.
185 comments
[ 1.2 ms ] story [ 171 ms ] threadUnfortunately it is quite a niche and most photographers are gadget hounds and don't mind having 2-3 drobos (including an off-site one).
http://www.loom.com/about
I'd love to hear more about this service from some users. I am bumping up against Dropbox limits with my photos as well.
I also trust Amazon to stay in business but mistakes can always happen (not just hardware but ID theft, card expiry while you're on a 6 month hike, etc) and unless you're prepared to say "losing these is OK" using one service would be risky.
http://www.dpbestflow.org/backup/backup-overview
If your files are important to you - you need a back up. If not - just get rid of them in the name of simplicity :)
PS. Also, uploading many gigabytes over a typical ADSL connection is just painfully slow... even local NAS is too slow over WiFi for my taste, and if I have to have a wire - I can just as well connect an external drive.
- 40-60gigs shot in a weekend, need to offsite the RAWs as soon as possible. In six or seven years we've never needed offsite retrieval. Fine for it to be slow.
- Only need around 100-200g of raws locally for jobs in progress
- Once processed into final jpgs files are approx 8-10x smaller. Offsite these as well, and need access to last 1-2 yrs of jpgs randomly/sparsely w/in 1-2 days of a print order coming in
I definitely think some well thought out service that uses maybe a hybrid of glacier and S3 could be really awesome. Especially if the pain is taken out of getting the bolus 40-50g of photos "uploaded" in the first place.. We'd happily pay for a service that just sends us an external drive and a postage-paid box.. we drop it in the mail after a job and get another drive sent to us right away.
[EDIT: Occurs to me you might wonder what this looks like today. Today we get home, and immediately back up onto an 8-bay Drobo Pro (http://bit.ly/1diZUHD). It's a fancy proprietary RAID-like system. I think ours has 7 1TB Drives in it currently, and it's at about 70% capacity. Then we back up the same jobs to an external hard drive which gets driven offsite (used to be my office, but now that I work at home it's the in-laws place) and copied to a hard drive for permanent offsite storage. Once that happens, the cards from the camera can be cleared. After processing, the final JPGs get copied to the drobo and make it into the sneakernet offsite process as well.]
I finally hit a sweet spot of performance, archive accessibility, and offsite safety, by using internal SSD for latest import workspace, direct attached WD Velociraptor (Thunderbolt) for current projects (can reconnect between Mac Mini server and laptops as needed), and 2 direct attached LaCie 4big Quadras (daisy chained via FW800) for archived projects (up to 32 TB).
Interestingly, because these are all direct attached, BackBlaze will happily back them up offsite at ridiculously low cost if that's your thing. Initial backup of several terabytes will take several months, though.
For fast cheap offsite backups, partner with a colleague with similar storage needs who uses the same ISP via the same head end. Just backup to each other. You can enjoy near LAN latencies at maxed out link speeds. This trick will save you both time and headaches.
Finally consider a product like the CyberPower OR2220PFCRT2U (this one also blends nicely in an AV cabinet) to keep an Airport Extreme and Mac Mini along with all that storage running for about 80 minutes of power outage.
Really, I need something that gets a copy of the images away from my home within a day to have something I'd be comfortable with.
Really, I think it makes sense for someone to prepare for the future and assume this will be feasible for upload in the not-crazy future.. unless more hobbyists keep deciding we need exponentially larger RAW files.. It's a crazy race :|
These guys talk about doing that with a RAID 0 setup: http://www.youtube.com/watch?v=cKaXZVHcdGM
The Joplin, Missouri tornado, Moore, Oklahoma tornado, Hurricane Sandy and Katrina.
I don't think you should rely on someone who MERELY uses the same ISP. I think you need to be a thousand miles away from each other.
I also store full "originals" (JPEG only, not RAW, but at least its something) on an unlimited Pro Flickr account, again, until their policy changes.
Of course, the challenge is to make sure the peers are online when you need the files, and for this reason the network may need 5-10gb of storage for 1gb of data to duplicate the content across multiple users.
The difficulty is ensuring enough clients have a copy of the file that it won't disappear, especially with people connecting and disconnecting a lot.
Do you want a one-time fee, or some yearly fee?
If you could get me to _really_ trust your service, I'd consider up-selling permanent archives to our clients. Lots of our clients would pay 50-200 bucks for the re-assurance, I'm sure. I don't know that I'd want the ever-growing liability though (right now we contract for 1yr of backup/online-gallery, but keep everything permanently without committing to doing so).
One time or yearly fee doesn't make a ton of difference to me, but with other services we've definitely been tempted into fixed "Unlimited" pricing so that we don't have to think hard about math, or end up tempted into picking and choosing what we backup. Simpler is better.. Managing a store of assets where some are backed up and some aren't sounds like a really bad accident waiting to happen.
4 TB of usable space divided by 60 GB times $20 is $1,365. That's the kind of thing a consumer or prosumer could do on their own, but it would be pretty hard to make a real business out of it...
EXTRA CREDIT: Anyone who considers tackling this and finds themselves interested in the niche of wedding/event photographer should look at combining this service with shopping-cart/gallery solutions that currently exist. The marketplace for those is fucking awful. Feel free to email me, but googling will get you the big players. I just don't wanna bash them by name in a public forum. All well meaning folks I'm sure, but every one of them are terrible flash-based, no-mobile software. Do backup and gallery/print-ordering management correctly and we're a customer at four figures per year.
Simply, they are two radically different businesses with entirely different feature sets. You might very well need both.
They're not radically different—one just uses the other. You could easily do both and continue to expand the use of the backend to enable useful applications. It's ridiculous to shun this specific consumer model just because it happens to be only one way you might use the original idea.
I don't have near the volume of pictures that you have (around 1.5 TB) but here's my setup: - one disk attached to the Lightroom computer, with all images on it; it's local, so it's fast - one NAS (NAS1) with a 4-disk RAID that backs up this drive (3 TB capacity) - another NAS (NAS2) with just one 3 TB disk that backs up the other NAS
On NAS2 I rotate disks; disk A is in the safe when disk B is in the NAS.
I think my setup is quite similar to yours, except driving is replaced by "going to the safe in the basement".
If a disaster happen I should have at least the disk in the safe intact, which holds the data since the last rotation; and since backups from NAS1 to NAS2 are quite fast I try to have both disks pretty much in sync (after an important shoot I back up to disk A AND disk B).
Of course I won't know if this setup is good until my house does burn down (I hope I'll never know!) but it does give me peace of mind, which is at least something ;-)
Currently, I have several multi-terabyte drives that I erratically back up to...I just got a 4TB drive that I'm going to try to backup everything I've ever shot in the last 3 years...
But I do want to move it to online storage...and I think this requires triage. For nearly every photoset, I've quickly gone through them with Lightroom and starred the ones that I kind of like, and then for maybe 1% of them, have taken the time to fully process, label, and upload them (in JPG form) to Flickr.
So for online storage, I think what I'll do is write a batch script that dumps all the unstarred RAW files as JPGs, because they chances that I'll ever need these photos in RAW is very slim, and then upload them. For photos that I've given at least one star and/or taken the time to properly edit and categorize, I'll send up the RAW file along with the processed JPG.
If I have about 3TB of RAW files...and maybe 10% of those are RAW images that weren't disposable...that's 300GB right there. And then the remaining RAW "keep em just in case" files would end up being compressed from 24MB to about 2-3 MB (let's say 15% as an average)...so 400GB.
700GB is still quite a big footprint. However, the biggest problem will be...let's say I give up local storage all together and rely on the cloud...what's the best way to browse any photoset at any arbitrary time? When it's all local, it's trivial to pop open Lightroom and do some quick browsing and filtering. For online storage, I'll have to put a ton of work into proper folder-naming, at the very least
If I could browse locally with smaller images (as you describe) like I currently do with Picasa (galleries, tags, etc) and then have the ability to download the original images when and if I need them, I'd be in heaven.
Of course, Apple's iCloud is very little like an actual usable cloud for most pro users.
1.) Personal: This doesn't mean I don't delete, but I generally tend to keep photos even though they might not be really good photos, especially if I only have a few of that day or person. I know I might enjoy looking at them decades later anyway, a blurry photo is better than nothing.
2.) Archival: like taking a photo of a flat before moving in. These aren't very many, so I tend to keep them no questions asked.
3.) Siblings of Keepers: You could call these "proof I really made the keeper photo" photos. Let's say I come across an animal I never photographed before, and like one of the photos enough to publish; then I'll still keep the others around. Or keeping a wider crop in case you change your mind later, etc.
4.) Keepers: here I try to be as harsh as I can. Photography is about painting with light for the pros, for me it's mostly about selection, that's the part of it I enjoy most. I walk around, and from the thousands of things I see each day, only a few make me take out the camera. Then I select keepers (partly before the images even leave the camera), and even years after I might revisit some keepers again, decide that my standards improved, and delete them.
I don't want to tell others how to pursue their hobby, but I think the web is full of galleries not even the owners look at, and that generally, a lot of people need to learn how to delete. The same goes for blogs and whatnot, the idea that everything has to be written on this indelible roll of toilet paper seems silly.
On one extreme, there is being OCD and deliberating endlessly what to keep and how to rank it, on the other there is just making more stuff to throw behind yourself on a pile you never investigate, because it's too awkward to wade around in. Productivity lies somewhere in the middle I think.
http://koken.me/
For back-ups - just buy a huge disk or set of disks, and keep 'em off-site wherever you have a trustable location (friend, family, work - if allowed and bandwidth isn't an issue) and use CrashPlan's free software to sync a copy to the secondary location. It's free after the initial expense of the drives and gives you encrypted, but physical access to the content in case of an emergency. You could find a friend with the same problem, and just agree to dedicate a certain amount of disk-space to each-other for this purpose.
I think it's possible to have a either S3 or GDrive based system that has a Angular.js (or some other JS MVC) that can operate without a backend for very simple browsing features and connecting to the S3 via apis.
I think it's doable ...
(for local storage, once you go beyond individual USB drives, I'd go with a Synology 1813+ NAS ($999 chassis, 8-bay) or an Areca ARC-8050 thunderbolt raid ($1499, 8-bay).)
-> https://www.bitcasa.com/
You shouldn't be touching the directory structure. This tool should be either a plugin for Lightroom/Aperture/Photoshop or an entirely separate application.
I would lean towards separate application. It could be a background agent that detects when new files are added, would work for any known editor or user-defined directory.
https://plover.co/
We want to take backing up, sharing, and monetizing and make them very simple and enjoyable to do. We've started rolling out slowly already, our goal is to be open to the public within a month or so.
We pulled our bootstraps hard and tried to get it going, but soon realized that there's a pretty big flaw with the idea: most photographers have no idea that they should backup their photos. They know hard drives crash, but they never expect it to happen to them.
This meant that we'd essentially have to become insurance salesmen, convincing people that bad things will happen to them.
Eventually, we decided to shut it down, which I also wrote about: http://tghw.com/blog/well-that-sucks-what-else-you-got
The old homepage is still up at http://www.snaposit.com/.
[Edit]
Also note that OP wasn't really just pitching "backup". He wants you to reduce the footprint his images take up locally. I think the messaging to casual and hobbyist users probably ought to play on this. Note Loom's taglines "more room to play". Backup is probably boring for people not under the pressure of liability like pros are.
http://fastglacier.com/
My undergrad was in USC's entrepreneurship program. One of the points they always hyped on was whether or not you were "selling vitamins or vicodin", so to speak. As you learned through Snaposit, most people when making chit-chat will be supportive of your venture, but you won't know how they really feel until they vote with their wallets.
Most of the professors were older guys who didn't get online startup culture, but one of their favorite tricks was to put up a landing page detailing a concept they were interested in, with a buy button. They'd spend some AdWords to get people to the landing page and measure the conversion rate. If enough people tried to buy to justify the service, they'd go out and build it.
In fact, that's how the Four Hour Work Week or one of those books was written -- ads first, and when the ads converted, the book was then written.
It looks like your product targeted consumers, but professionals seem like they'd be a better market. And it should have been priced way, way higher; the more it cost, the more secure it looks. I'd have a hard time trusting my money-making business assets to a $9/month service.
Of course, I'm not a professional photographer nor do I have your experience in trying to build this business, so who knows what my opinion is worth. :-)
Consider the some examples:
* An older photographer who grew up in the days of film who has just transitioned over to digital.
* A homemaker who is trying to add some much needed income for her family, with a passion but no formal training.
* A well-off software developer who moonlights as a photographer for fun and a little extra money.
Of those, only the third has been exposed, formally, to correct data backup procedures. The others are likely to keep their photos in folders on the desktop of their laptop.
Furthermore, photography is not a very lucrative profession for most photographers. That limits the amount you can charge. Our original pricing was much higher, but we realized that would be problematic, considering the average photographer brings in around $30k per year.
"Professional photographers know they need to backup their photos."
A homemaker or techie moonlighting for extra income aren't really 'professional photographers'. They may take photographs that look professional, but it's not the same thing.
One might say that professional developers use version control, then have counter examples of "well, this web design student doesn't ever uer version control, but she's the only person working on the code, so it doesn't matter". This is not a professional developer, even if she is capable of producing results that look like they were done by a professional.
A professional, imo, is defined as much by their habits and practices as by the output.
A friend of mine had her wedding photographer lose 6 months of data in a hard drive crash. She should have known better, and may have, but she didn't do proper backups.
What else would you call it, if not professional?
That said, I have to harass her about backups.
Everyone I know backed up on-site, using manual methods as and when they remembered. I did actually set up a good automated backup solution for one of them, but there's definitely a limit to their interest. Remember they're making pretty much all their money off commissions, and are only keeping backups out of some sense of pride/responsibility.
Another blow is that, at least here in the UK, we're only starting to get the kind of upstream bandwidth necessary to make this level of data transfer feasible.
So, it's a brilliant idea and, as long as confidence in cloud storage grows, as time goes on the market will grow and grow. I'm just not sure it's quite there yet.
As a web developer who works on a lot of solo projects, this sentence hurts me.
> It did this by compressing the photos to full resolution, high quality JPEGs, which saved anywhere from 50%-90% of the size. The downside of this, of course, is that photographers who shoot in RAW would only have a JPEG copy backed up. We figured this was a reasonable compromise. In a catastrophic event, wouldn't a photographer rather have a JPEG copy than nothing at all? It turns out the answer was no. But we didn't ask that question, because we thought the answer was so obvious.
I am no photographer... but from what I can see in this thread, what people are looking for is a solution to store RAW image files. I am sure photographers can find already good solutions to store good-quality JPG copies of their files.
With that in mind we've started to look more on how photographers with larger photo volumes could benefit from our service. From the feedback from few photographers we've already included some features (RAW photos with smaller JPG thumbnails, workspaces and personal sharing for example) that will make the day to day workflow a bit easier. Currently our goal is to build something that would fit better for professional photographers (public for clients, ordering prints and possibly integrated payment flows, win/mac client + plugins for your favorite photo software).
Pricing wise we are currently not the cheapest but we have thoughts on how to make this better for high volume photographers. If you're one of them then please send me an email (martin at photry.com) and I would love to talk a bit further what we plan to do and if we would fit your storage needs.
While the post mentions avoiding externals I think a handy feature would be to push to both an external drive, glacier, and then remove the RAWs locally. That way you have two copies, so the data is actually backed up. Trusting all of your photography to amazon not losing data doesn't sound ideal.
Rather, they charge you on a monthly basis, according to your highest hourly retrieval rate from any single hour applied to the whole month (standardized at 720 hours). So if your peak hourly retrieval in a month was 10 GB/hr above your pro-rated free quota for that hour, you're charged for 7200 GB of retrieval overage, or $72, even if you retrieved nowhere near 7 TB of data. For companies with relatively constant retrieval this doesn't matter, but for small users, someone who has a large overage one day will be charged as if they were incurring the same overage continually all month.
To put it concretely, if you store 200GB of photos and then do a bulk retrieval, you aren't charged $1.90 ((200-10) x 0.01). Rather, the xfer is counted as 4 hours, so you have a peak transfer of 50 GB/hr. Your free daily retrieval is allocated to the hours in the day you retrieved data, so in this case 10/4 = 2.5 GB/hr free. So your overage is 47.5 GB/hr. Multiply by 720 and $0.01, and the fee for retrieving 200GB of photos is $342. Plus $22.88 in outgoing AWS bandwidth.
Or at least, I think that's what their pricing page says. Maybe someone else can better interpret this explanation: http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...
Arq (a backup app I wrote) takes a transfer rate from you (guesses your max transfer rate as a default) and then requests enough to be downloaded in 4 hours at that rate. 4 hours later, it requests another 4 hours' worth while it downloads the first 4 hours' worth, and so on. This mitigates the peak-transfer fee.
There really is no good solution to automated offsite backup when it takes several days of uploading to handle a single day's worth of data capture. To make any internet-backup service work, you'd first have to pitch the photog on switching from their residential broadband package to something more expensive -- if that's even available short of their leasing office space somewhere.
I guess you could try to build out a reverse-CDN sort of network, with local relay nodes scattered across the various ISP networks, that might achieve faster uploads from the user (and maybe not get counted against bandwidth caps) which then use a fat pipe to send that data to larger regional storage nodes.
Neither seems particularly plausible at scale.
Right now, most of them use Drobos or similar devices. You'd be asking them to add steps to their workflow which is always tough to do.
Also, I think your estimate of 18 mins to upload 6+GB is an order of magnitude off. We have tools that allow people to upload JPEGs and most customers claim it takes longer than that just for 100s of MBs.
My (wild) guess is that Adobe is going to get in this space too. It only makes sense to have your Lightroom catalog in the cloud. Maybe they wont offer an affordable solution for terabytes of data, but I'm not sure photogs want their images in two different "clouds".
I think the key is to do customer development from full-time photographers who aren't hackers. I'd be happy to send out a survey to some clients if you want do more research.
and I still think it will be hard to convince people to "put these files on Adobe's cloud and these files on this other cloud using X software"
I think you're totally right about Adobe too. Those Smart Previews they added for LR5 - those are not just for space-starved SSD drives...
You can't really have the current catalog in Dropbox because then each action and edit become really slow.
But if you set up Lightroom to do a backup every time you quit the program, and that backup is in Dropbox, you get the best of both worlds (strong backups + snappy edits).
Lightroom backups are really "backups", as they are named by date+time, so in Dropbox I have a history of everything I ever did in Lightroom, not just the current (last) version.
It's also fast because I suspect Dropbox is able to do some rsync magic and only re-upload new packets; my Lightroom catalog is around 150 MB and the upload to Dropbox is almost instantaneous (on a pretty lousy upstream bandwidth).
Of course this doesn't address the problem of backing up RAWs, but as far as Lightroom is concerned I feel pretty safe.
Your LR catalog contains your photo edits, metadata and history... but is not your original files. Backing it up is very important but is only part of the story. Putting your Lightroom catalog backups on Dropbox is a great and should be done by everyone. (Again just not your active catalog.)
So we are an ecommerce company and we do our own photography for products -- and we do a lot of it. We average about 2-3 photoshoots per month, and each photoshoot can be upwards of 40GB before any processing. We shoot raw images and are starting to experiment more and more with video. After we do the post-processing work and start the video editing we can push 70GB a shoot easily.
Our current workflow is like this:
Shoot tethered to a laptop each day (typically each shoot lasts two days). At the end of the day, we upload the files up to our local NAS (Netgear ReadyNAS, just shy of 3TB) which is redundant.
At the end of the shoot, we load a copy of the entire shoot on the local computers that will be editing so that they don't have to work directly from the NAS. Over the next few weeks while they edit the files, they will push the new files up to the NAS into specific folders.
(This is where it gets hairy) We also have an offsite NAS that is located in the business owner's house (about 6TB, but this one is not redundant). We used to attempt to upload files each night from the local NAS to an offsite NAS via rsync, but the pipe was just way too small. It would take days to upload a photoshoot, and everyone in the office suffered from slow internet while that happened. Now what we do is bring the offsite NAS onsite every few weeks and do a manual sync! Locally the sync only takes a few hours and then that night it goes back home with the business owner.
Now since the local NAS is less than 3TB, we can't store all of our shoots on there forever. When we get close to full on the local NAS, we grab an external hard drive and archive old shoots (typically 2+ years old) to it, and then store that in our massive safe (on-site though). If we need access to just one or two files, we can retrieve them from the offsite NAS, but if we need access to quite a few of them, we can dig them out of the safe.
So that's our workflow. Probably not bullet proof, but it has been serving us well for the last few years. If someone came up with a better solution, we would be all over it!
Downloads add significant cost if you want to retrieve more than 0.17% of your data in a day.
Downloading 5% of your data increases your monthly rate to $0.097 per GB. 10% increases to $0.187 per GB 50% increases to $0.907 per GB 100% increases to $1.807 per GB
Downloads are also delayed by 4 hours after you make a request.