As always the problem seems to be the 'Hotel California' issue: you can check out but you can never leave. Once you have massive data in AWS there is no efficient and affordable solution to move that data back out; you are locked in forever subject to whatever future terms Amazon may chose to impose.
It is for backups of which you have another copy. To switch providers you would start shipping new backups to the new provider. Once your confident the provider has all the backups to meet your retention policy, you abandon AWS.
> Once you have massive data in AWS there is no efficient and affordable solution to move that data back out; you are locked in forever subject to whatever future terms Amazon may chose to impose.
You can have your shit packed into a snowball and shipped to you.
Perhaps "once we have imported your data, you can quickly retrieve data from the cloud using the regular S3 APIs and fee structure". Or in other words, "there is no special restriction on data added through Snowball".
Feels a bit like intentional misdirection, but it's more likely a case of sloppy writing.
“Transfer up to 100 PB per Snowmobile, a 45-foot-long ruggedized shipping container pulled by a semi-trailer truck.” - I’m guessing usually used to migrate corporate information into AWS, but can also be used for data export. They advertise Exabytes (using more than one rig I guess). https://aws.amazon.com/snowmobile/
Hmmmm - the subheading of the page says “Migrate or transport exabyte-scale datasets into and out of AWS”, and in the diagram they repeat it. Either AWS is being deceptive, or the FAQ is wrong. Someone who cares that has a corporate AWS account should check or get documentation corrected? More interesting to know how AWS respond…
Q: Can I export data from AWS with Snowmobile? Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary.
Interesting since there is obviously no technical reason the export can’t be done from AWS. ⸘Technically you could export exabytes using tens of thousands of 100TB Snowball Edge devices‽
The snowball “device” is pretty neat:
* 45-foot long High Cube shipping container
* [each snowmobile container] comes with a removable connector rack with up to two kilometers of networking cable that can directly connect to the network backbone in your data center.
* AWS can provide the auxiliary chiller if needed based on the site survey findings [if temperature exceeds 85F/29.4C]
* A fully powered Snowmobile requires ~350KW [sic] Generators can be provided by AWS if needed
* Snowmobile pricing is based on the amount of data stored on the truck per month. Provisioned Snowmobile Capacity: $0.005/GB per month
I wonder if they haven't updated that in a while, it sounds low. You can get 22TB 3.5" hard drives these days. If I take that storage and figure full racks with rackmount NASes I get 30 feet long for 1 row of full-size racks. 45 feet is longer and they probably have room for at least 2 rows. Even accounting for probably needing some controller computers and routers and network interfaces and such, I would think they could at least double that these days.
Surely for hardware being lugged around on the back of a truck, potentially exposed to whatever heat, humidity, vibration, etc - redundancy becomes a vital component of the design. I would expect at least 2-3x replication for every byte written. Depending on how sealed the container is, they may not want to physically extricate failed drives and merely mark them as unavailable for use.
AWS will do anything to get your data into their cloud. I realized this a few years ago when Snowmobile was released: https://aws.amazon.com/snowmobile
"Tape Gateway supports all leading backup applications"
Only huge companies like Amazon can be this dumb. They don't even mention tar or pax, the two most common tape backup applications.
Also, how will this magic Amazon "Tape Gateway" back up petabytes over slow links? There are many data heavy businesses that don't necessarily have tons of Internet bandwidth. Fully saturating a 100 Mbps outgoing connection will only get you 1 TB a day, so what happens when you have a tape's worth of new data daily and an already heavily used Internet connection?
That's not quite how it works. Amazon will send you various storage appliances (including a semi hauling a 45ft shipping container full of hard drives), which you connect to your network, upload your data, and send it back to them. They don't accept pallets of arbitrary media devices.
I started from Tanenbaum’s hurtling station wagon and used my poetic license without worrying too much that the parent commentor would buy a forklift before talking to AWS sales.
You upgrade your WAN circuits to meet your needs, or you suffer the consequences. Simple as that. If you can't upgrade the WAN circuits, you stick to local tape libraries.
AWS doing tapes like this feels weird to me. The physicality of the thing is kind of the point in my mind. Virtualizing the last line of defense seems wrong, but perhaps I could be convinced otherwise.
Why doesn't Iron Mountain or some other competitor offer a service like this?
Iron Mountain does have a service that will pick up physical tape and offload to other storage, but nothing like AWS's virtual tape capabilities.
FWIW, Amazon isn't 'doing tapes' with this product. It's just a compatibility shim laid on top of object storage so that legacy tape backup/archival systems can eliminate physical tape libraries.
Dumb question, and I'm guessing this is a "if you don't know it's not for you" situation, but what is the point of a virtual tape? Isn't the point of a tape that it's not virtual? Or is this more replicating tape software apis (WINE/proton style) so you can get rid of physical tapes (because you no longer care about their physicality) without having to change your backup strategy?
It's the latter. In a large enterprise, the backup configuration can be wildly complicated, a matrix of systems, schedules, slas, etc. Just reconfigure your backup software to use this new virtual tape device and you're on your way.
> Just reconfigure your backup software to use this new virtual tape device and you're on your way
Isn't the whole point of tape that it is a physical thing, and may be taken offsite in a truck / stored in a vault, and all that jazz.
"Just reconfiguring" your backup software sounds like a business might just bypass all that without necessarily realising the consequences.
Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually there are no tapes, it's all the cloud now" -> ... ??
Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually all the tapes where stored at the wrong temperature or humidity and now they are fucked" -> ... ??
> AWS's hard disks are a physical thing that is offsite
Taking tapes to a secure offsite location means they're air-gapped from the source data, so even in a situation where an entire network is compromised, remotely-stored tapes can't be wiped/encrypted.
I'm sure AWS has thought about this issue while designing Tape Gateway, but it's not clear to me how it could know whether a request to retrieve and overwrite a virtual tape was legitimate or not.
The gateway appliance is fairly compatible with legacy backup solutions, which makes it a great drop-in replacement for physical tape library systems. There are certainly better backup methods available these days (though it's hard to beat the durability and cost of LTO tape for long-term archival), but I've seen AWS's virtual tape used as a good stopgap while other backup/recovery solutions are still a ways out an org's infrastructure roadmap.
> though it's hard to beat the durability and cost of LTO tape for long-term archival
I'm not entirely comfortable with the trend towards ever more esoteric and seemingly "all or nothing" technologies.
Tape (and other physical things) may have their downsides, but I think there's something to be said for something that could (theoretically) be forgotten in a closet and read 50 years later.
Likewise, AM radio may not be the best quality, but it seems like you can cover more area with a single installation than any other communication technology, which might come in handy after a serious disaster like a nuclear war (e.g. crank up the nighttime power of some undamaged countryside station running on generators to tell survivors where it's still safe).
It's very unlikely that you'll be able to read modern tape from closet in 50 years. Tape requires very precise temperature and humidity for storage. Otherwise all bets are off.
Only one datapoint; but I’ve seen an FPS/Celerity minicomputer boot off of a Qic-20 tape that’d been stored in a non climate controlled Albuquerque storage unit for 19 years…
Even for a new strategy, using a tape abstraction might simplify the design and/or implementation because backup tools and culture have their roots in tape.
There's a sense in which tape is water in which backups swim.
The VTL was invented because, at the time (2012ish), none of the largest enterprise backup solutions had a good S3 interface and none supported Glacier. After talking with the backup software vendors (Spectrum, Tivoli, Symantec, Commvault, etc) it became clear that adding another backup target wasn't something we (AWS) could get them to prioritize, for perfectly reasonable reasons. We could (and did) apply pressure via our shared customers, even then they estimated it would take years.
The fastest way to enable large enterprise access to S3 and Glacier for backups was to meet them where they were. We did this by virtualizing a tape library.
You did such a great job they still try and steer people this route. I've multiple times mentioned "you know we don't need to emulate tape drives any more"
This is a fantastic example of how broken the patent system is.
This is not an invention it’s a good implementation of a well understood problem.
It is also priced ridiculously - it is at least an order of magnitude more expensive than operating a physical library and remote physical storage that is beyond cyber threat by virtue of it being disconnected.
You did such a good job, that when I worked on Glacier (2013-2016), our interactions with the team running VTL was already pretty minimal. It was one of the least problematical things that interacted with Glacier.
Some of the solutions that external vendors produced were nightmarish and left customers up the creek without a paddle in a disturbing number of situations.
From the perspective of the AWS solution, this is a way of giving on-premises hardware an option versus Iron Mountain or similar for archiving to offsite for DR/BC purposes. Since AWS' storage options are typically pretty insane SLAs, this is acceptable (and certainly not much worse than having old LTO tapes in a storage locker).
VTLs have been around for a long, long time. Initially, they were a way of speeding up tape by writing to a much faster media, and allowing for deduplication on the media to reduce storage costs. This was, of course, in the days of on-prem arrays and storage. Over time, LTO (the dominant tape format, I think we're on LTO-9?) became denser and faster, and the speeds at which you could write to a single drive would outpace the line speeds. Given that, methods were introduced to interleave multiple writers to a single drive/cart, and to leverage the drives speed. One thing you don't want on a tape drive is "shoeshining", this is the effect when the drive has no data to write, but the momentum of the reels in the drive mechanism carry the media forward, past the last written segment, past the write head. The drive must then backup and re index to find where it left off while queuing whatever data is incoming.
Well, as VTLs were built initially to handle speed of writes and dedup, once the tape drives outpaced those issues, VTL became problematic. It was basically taking a randomly writeable media and turning it into a serial storage mechanism. This really wasn't much use, and VTL started to fade. Many solutions were built, one from NetBackup was OpenStorage; this was an API that a storage vendor could write a plugin for and allow NetBackup to write to their storage as intended: multiple writer, multiple access. Allowing for treatment of a deduplicable storage media as a filesystem versus an emulated tape drive. This API approach allowed for plugins to be constructed to talk to any storage, including cloud.
AWS spoke with NetBackup at one point, intending to build a storage mechanism for a very solid solution: allow on-prem customers to export their data to AWS storage, AND to recover that data BACK from AWS storage (think of the data transfer fees you would get out of that!!). When they looked at the options, they decided to build the VTL gateway. I'm not sure of how or where the decision was made, but it seemed a bit odd at the time. As VTLs were fading and the limitations of emulating tape would lead to a caching requirement that might be bad, but with line speeds for interconnect to AWS making the caching requirements even worse. I'm not quire sure how that was done, but I will say, a VTL will always be a stable, solid thing, that tends to simply run and do what was intended; much like a traditional tape drive.
As for some of the other discussion here, about reading old tapes and storage longevity; I've seen tapes kept for decades that were readable, Iron Mountain is still in business for a reason. Note that THEY don't have a solid cloud offering, but really, if you had everyones old data sitting around on tapes, and could charge not only for data delivery but media conversion at DR time, wouldn't you?
And the issue with old tapes isn't so much the storage in that situation, it's the hardware. LTO only reads 2 generations backward (IIRC), and can only write one gen forwards. So, for your LTO-6 drive, it can read LTO-4 to 6, and can write LTO-6 to 7, again if memory serves. So, finding the Apollo code tapes on a format tape where the only possible drives available to read it live in a museum and may or may not actually work, make the fact that they recovered that source code amazing. Not to mention the software needed to be built to interface to hardware that the last person to build a "driver" for likely is sitting in a nice retirement joint in Sarasota.
My first and only experience with tape backups was in 2014 working with the BBB of Chicago. They were using tape backups, and there I learned that almost no company that small should be using tapes. Unless you have one or two people dedicated to handling that plus whatever other backup solutions you have in place, it won't end well.
I hear you. I had to manage a Quantum LTO4 tape library with Bacula back in early 2010s at a small company. Yeah that was a fun experience (not) and I'm glad I don't have to deal with them. Good riddance. Just getting the labeling right and making sure that the tapes were writable was a pain.
But that is Bacula fault isn't it? I used it myself @home and couldn't help thinking that there must be a better way. At work Commvault was used iirc, but others were in charge of that. I evaluated Tivoli SM, but that's more than 15 years ago and I barely remember. TSM and Bacula share some concepts, but the latter appears quite hackish. Not that confidence inspiring.
Interesting. I wonder if tapes got more complicated or if everything else just got easier.
The last time I dealt with tapes was in the 1990s, and it was dead simple. We used tape backups at a company with only 150 people. I was the most junior admin so I had to be the tape monkey, but it was easy. Just run backups regularly, change the tape every couple of days and label it, ship a stack to the offsite storage and request the old stack back, and then do a test restore each time a stack came back.
Everything else was definitely easier by comparison, but the situation was more about the one internal guy responsible for it was also doing all of the normal IT dept functions, so being the tape monkey was at the bottom of the list - for better or worse.
When you're juggling 12 plates, a couple are gonna fall.
> I wonder if tapes got more complicated or if everything else just got easier.
Everything else got easier, bigger, and faster.
Two decades back, you needed tape to cost effectively back up hundreds of gigabytes. These days, pretty much unless you have a spare room with an automatic tape library (robots moving tapes), you're too small for tape to make sense.
Agreed you need someone dedicated a minimum 1/2 to full headcount to run a backup tape backup system - particularly the first six months when you are getting all your routines in place (if can drop down to a minimum 8-10 hour/week time commitment once things are running smoothly - but obvious can scale up to many many headcount if you have a lot of data and a lot of systems being backed up). You purchase a Tape Library with sufficient drives / slots to support your data volume, you decide on tape software, you set up a pickup schedule with Iron Mountain or whoever will pick up (and return) your tapes, set up up your schedule of fulls / diffs rotation. There are three tricky and time consuming elements that require a lot of knowledge and dedicated time - #1 making sure you have a consistent snapshot and backup of the snapshots and RESTORE process for all your systems. Someone needs to test/verify roughly quarterly test restores (very time consuming and annoying but critical). And, the one thing that very few people do, but is also important - doing a “Black Start” - in which your primary backup site (with the robot, and software, and systems) are lost and you have to bootstrap, from scratch everything.
So, yeah - virtual tape library, while obviously having downsides, is massively useful for a small enterprise.
No kidding :)
But the company that had the issue (I left a bit ago) believed in "cheaper the better". Cleaners ? That place would expect you to use soap from the restroom :)
Anyone considering putting giant amounts of data into AWS or most other big clouds should be sure to do the numbers on retrieval. All the big clouds have "roach motel pricing" -- data is free or very cheap to put in, and expensive to get out. Outbound bandwidth costs from AWS are astronomical, so make sure you are not going to need to stream down all that data or move it any time soon or that you have compared that cost to your in-house solution.
> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability.
I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster.
> Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed out that it is impossible to calculate such a remote possibility with any scientific rigor.
There's rigor within the assumptions. The problem is when the assumptions are wrong - most notably around failure independence. Data center fire? Accounted for. Simultaneous terrorist attack / asteroid (as per GP), / nuclear war on all data centers? Maybe not. Where does insider threat fit in on that spectrum? We don't know.
I'm personally fine with the claim as I understand the methodology and assumptions but it has to be interpreted carefully.
Yeah, the most depressing part of my job is when I have to calculate the SLO metrics for our directors. The first pass of stuff we control is always way within SLO. Then we have to subtract stuff where some third party messes up, or like the request doesn't even get to our service and it always puts us in the red.
Yeah but that's also incredibly naive. Inevitably there will be some un-calculated-for shortcuts that make all those copies not as failure-independent as they seem. If nothing else, what are the odds some random Amazon engineer makes a software rollout or hardware upgrade mistake and takes out a whole lot of copies in the process?
Tangent: What's the cheapest way to backup a few TB of personal data these days, pricing based on the premise that I probably will never need to retrieve it (due to local backups as well), but I don't want to pay thousands if I do have to (hundreds would be okay). Glacier Deep Archive?
The only thing I am not sure on is how to quickly restore a large number of files in a bucket from GDA to S3. It obviously can be scripted, but I don't have that handy. I only keep a small number of large, previously encrypted files in it, so I manually restore from the GUI.
(By the way, rclone can transparently encrypt files and filenames client-side!)
Probably. Looks like about $1/TB/month. Although Backblaze personal backup isn't that much more for a few TB--about $6/month "unlimited" paid annually.
edit: it integrates with TrueNAS (formerly FreeNAS) so it's been pretty much set-and-forget, but I can check in and see my stuff through their management interface. I do encrypt my docs before uploading, which TrueNAS also supports.
Have you had any luck with picking and choosing what gets backed up? I have a bunch of smaller personal files I'd love to sync to B2 (photos, backups, etc.) but I also have 7TB+ of linux ISOs that don't really need to be backed up because I can easily download them again. I tried to exclude the folders I don't want to back up, but I haven't had any luck.
It's been a long time since I set it up, but I believe each cloud sync task will sync one directory. So I've got two tasks: one that syncs /mnt/main/media/Photos in PUSH/COPY mode, and one that syncs /mnt/main/Documents in PUSH/SYNC mode.
You should be able to accomplish the same, but you may need to either re-organize your stuff or set up multiple tasks.
edit: for clarification, "/mnt/main/media" contains various other subdirectories of Linux ISOs etc., which is why I push /mnt/main/media/Photos specifically.
I use rsync with Backblaze. I put everything I want sent to BB in /backupfolder/Archive, and everything else straight in /backupfolder.
I have 2 rclone remotes, one pointing to another machine at my house, and 1 pointed to BB.
I have over 3TB backed up to Backblaze (in addition to a local backup) for $70/yr. That's for Mac or PC. If you're on linux, I believe they offer something for that, too.
The pro is that it has a client and, once setup, it just works.
The con is that it's a client running on a single system and the personal pricing plan doesn't support a NAS I believe. I periodically copy my files over from a RAID NAS to a local USB.
I think I'll point to this as one of the examples of how AWS has been scrappy and successful. (Yes, I'll get like 10 flames on this comment.) They made this nice simple API for S3 that has become a de facto standard. They added lots of features and security controls, and I can easily imagine an architect saying "everyone should just adopt S3, it's so easy!" But huge, paying customers were used to tape libraries so they said, damnit, fine we'll pretend to be a tape library even though that is super weird on some level... Meeting users where they are can be a superpower.
When someone is waving money in your face and saying "please just give me the thing I want and I will give you all this money", it's ridiculous not to listen. So many competitors just don't listen.
Sometimes people build things/products/companies/whatever not because of the chase of money, but because they want to put their idea in the world. Maybe it's just a case of someone believing they can build something grander :)
It’s not that weird. Tape libraries are well established and have well vetted highly reliable semantics that are deeply ingrained in many companies way of working. What’s weird is companies trying to sell some completely new way of achieving similar results and being surprised when the universe doesn’t reengineer everything as homage to their brilliance
> What’s weird is companies trying to sell some completely new way of achieving similar results and being surprised when the universe doesn’t reengineer everything as homage to their brilliance
Some abstractions can be proven to be objectively-better solutions, along some axes you care about, for your use-case, such that the existing abstraction you were relying upon is then provably sub-optimal. Switching to the more-optimal abstraction is not then an "homage to the brilliance" of the company that created the abstraction; it's a simple "pay CapEx to lower OpEx in a way that will pay itself back after N years" decision. If you can budget it, you do it.
I'm not saying that every abstraction that the IaaS vendors invent is this kind of better mouse-trap, mind you. You can usually distinguish the ones that are, because they get independently reimplemented and used by people who aren't just aiming to build "cloud-native" software, but instead just really want/need the semantics of that particular abstraction, independent of where it runs.
Object storage is a great example of an abstraction where that is the case. There are many things that do object storage now — both cloud services, and regular deployable software. In fact, even the more-traditional backup vendors, like Backblaze, now also offer object-storage APIs.
I use AWS at the companies I work for because of their amazing customer service.
I'm fairly critical of Amazon's engineering culture and practices but as a customer I have to admit they are head and shoulders above their competition and frankly most SaaS providers as a whole.
They've secured millions of dollars in revenue from me alone just by being so customer friendly.
Yep, that's been my experience as well. AWS (and Amazon as a whole) are bad for the people that work there, but as for their service offerings, it would take GCP and Azure combined to get even close.
I think if you took a poll of AWS workers you'd find that most enjoy it. The culture leads to a vocal minority who REALLY hate it but my time there was some of the most enjoyable I've ever had.
I've worked with some of the smartest people in the tech field, building products from scratch for billion dollar customers and learning every day.
I've always told people that it's probably the best place you can start out of college.
EDIT: This was back in 2016, I'm not sure how much has changed without Bezos and Jassy at helm of AWS
Amazon is a turn and burn employer at its heart. I've watched dozens of friends and acquaintances work there, rarely making it past 2 years, only to leave for less caustic organization thereafter.
Unless your willing to put up with a huge ration of shit on a regular basis, or you make it into management, your time is limited by the up and out mentality of the organization as a whole.
Amazon has created the perfect scenario to spend shitloads of money getting talent up to speed, only to part ways with them shortly thereafter.
Sorry dude, you don't have credibility based on "Unless your willing to put up with a huge ration of shit on a regular basis, or you make it into management,"
Management at AWS arguably puts up with much more shit on a regular basis.
I can't really speak to amazon retail, but in my 7 odd years in aws I found most people left because
* working in infrastructure means doing a lot not quite so exciting things. most people aren't actually excited by ops, and a lot of your dev work will be on ops. its also the case that ops was historically not as well rewarded, though thats been fixed
* every service needs to be held to the highest tier of operational excellence, and so you can't just chabuduo when things aren't working as well as they should. in the worst case, this means throwing bodies to the ops meatgrinder.
* delivery speed is slowwwww. some of this in endemic to the space, but aws at the end of 2021 (when i left :) ) was a lot slower than than when i joined due to so much risk aversion and bureaucracy
* comp wasn't as good compared many tech peers, and comp could very easily not keep pace if you weren't tippy top every year. this is imo the most self-defeating part of amazon culture as a whole because it incentivizes people rated between ~p10-p20 and p50 to leave.
> was a lot slower than than when i joined due to so much risk aversion and bureaucracy
I feel like this is true for every single large company I’ve ever worked for. Things always feel like they’re regressing from a high point. I have no idea why it it.
All of them had free coffee when I joined, and they’ve slowly removed the vending machines and just never replaced them.
The team working on developing integrations for Amazon Payments certainly didn't have diddly squat in their break room when I visited a few years back. Not revenue generating = Get treated terribly at Amazon was a sight to behold in action.
> I can't really speak to amazon retail, but in my 7 odd years in aws I found most people left because
I've worked at Amazon for over 4 years. During this period, I saw around 80% of all team members being promoted to customer, perhaps half of them voluntarily. Amazon provided psychological service to those who felt they needed it, and a couple who quit confided in me they used the service. One guy overtly burned out and never showed up again.
Long work hours were the norm. Plenty of CRs were posted after 10PM. Even SDE3s frequently posted CRs on Sundays. The senior manager was hell bent on proving leadership he could deliver projects in impossible deadlines and not rarely did he warned SDEs that failing to deliver on those promises were firable offenses. This in turn fostered a work environment that led to some backstabbing and less than ethical stunts from team members prone to those and desperate to make a good impression.
It's true that Amazon's salary structure is designed to push out people. It is designed to impose a hefty pay cut after the second year or so. Nevertheless this is not the problem. The problem is the awful corporate culture that's designed to hold SDEs responsible for not delivering things even in impossible scenarios, and the backstabbing culture it sprouts by design.
But I'm sure some people might have a good time working there.
> The senior manager was hell bent on proving leadership he could deliver projects in impossible deadlines and not rarely did he warned SDEs that failing to deliver on those promises were firable offenses. This in turn fostered a work environment that led to some backstabbing and less than ethical stunts from team members prone to those and desperate to make a good impression.
is an extreme situation and one I never saw across ~3 orgs between amazon and aws.
It's weird how it's always the people that talk about friends that worked there but don't actually work there themselves that are usually the only ones to have negative things to tell.
I've been there 5 years, haven't worked more than 40 hours a week ever, and a lot of the people that started at the same time as me are in the same situation. And of those that aren't, they left because they wanted new experiences, not because they didn't like Amazon.
I never had to put up with any "shit", whatever that could mean.
If your prior employer kept you for half a decade before you departed for Amazon, and the employer after Amazon keeps you for years, yet your stint at Amazon lasts under 2 years, that failure reflects on Amazon.
Spending the money to recruit the employee, then pay them for a year or two, only to lose this talent is a great way to light cash on fire. That ain't cheap when your talking about hiring software developers!
> I think if you took a poll of AWS workers you'd find that most enjoy it.
Isn't the average tenure at Amazon less than 3 years? "Most" of the people who you'd ask would be fresh from the interview rounds and with only one or two performance review rounds under their belt.
That's Amazon in general, it's an effective company from the outside because how it treats the people inside. The customer is everything, and other things are secondary. Including work life balance, employee happiness, burn out...
I’m fairly critical of all AWS’s weird limits and the quality of their non-core offerings (CDK is full of bugs and missing resources, half of which are caused by missing shit in CloudFormation).
Well, they do call it "virtual tapes", would be weird (maybe even illegal? False/misleading advertisement?) if they ended up storing it on physical tapes anyways.
> Well, they do call it "virtual tapes", would be weird (maybe even illegal? False/misleading advertisement?) if they ended up storing it on physical tapes anyways.
No, just like it is not weird and definitely not illegal for the things they call “virtual machines” to end up being run on actual machines.
If you purchase a service describing it as "isolated virtual machines" and it ends up just being processes on a computer without any virtualization, wouldn't you call that at least misleading?
I think you're misunderstanding the scenario. What if each 'virtual' machine is actually bare metal i.e.: every EC2 instance is actually a separate computer?
There's still isolation, there's just no virtualisation abstraction.
My knowledge is a decade old, so they very well may be using tape now, but were not originally.
At announcement time Glacier was using super dense racks of hard disks but only powering a subset of them. This led to some tape-like behaviors, like your restore request might take a long time to process because your data was likely stored on disks that weren't powered on and you'd need to wait for them to come online.
HEY! Before you adopt this check the bandwidth egress costs! Seriously.
Your disaster recovery plan will need to budget for extra egress costs if you ever need to restore something big from one of those virtual tapes. AWS charges for egress -- downbound -- bandwidth but they don't charge for ingress -- upbound -- bandwidth.
(And I think it's funny their icons and images look like old school DECTapes.)
It's been a long time since I've heard the word "tape" in my circles (I can't get John Cleese's Institute for Backup Trauma out of my head: https://www.youtube.com/watch?v=q_9wIupr9Hc).
That being said, kudos to Amazon for having a solution for those who aren't in my circles and still use them.
138 comments
[ 2.8 ms ] story [ 209 ms ] threadYou can have your shit packed into a snowball and shipped to you.
https://aws.amazon.com/snowmobile/faqs/ > Snowmobile does not support data export.
https://aws.amazon.com/snowmobile/
Quickly retrieve data from the cloud whenever you need it.
What does it mean?
Feels a bit like intentional misdirection, but it's more likely a case of sloppy writing.
https://docs.aws.amazon.com/snowball/latest/developer-guide/...
> Snowmobile does not support data export.
The snowball “device” is pretty neat:
* 45-foot long High Cube shipping container
* [each snowmobile container] comes with a removable connector rack with up to two kilometers of networking cable that can directly connect to the network backbone in your data center.
* AWS can provide the auxiliary chiller if needed based on the site survey findings [if temperature exceeds 85F/29.4C]
* A fully powered Snowmobile requires ~350KW [sic] Generators can be provided by AWS if needed
* Snowmobile pricing is based on the amount of data stored on the truck per month. Provisioned Snowmobile Capacity: $0.005/GB per month
Only huge companies like Amazon can be this dumb. They don't even mention tar or pax, the two most common tape backup applications.
Also, how will this magic Amazon "Tape Gateway" back up petabytes over slow links? There are many data heavy businesses that don't necessarily have tons of Internet bandwidth. Fully saturating a 100 Mbps outgoing connection will only get you 1 TB a day, so what happens when you have a tape's worth of new data daily and an already heavily used Internet connection?
Easy: it won't. If you have more data than bandwidth, you won't use this service. Not sure why you seem to think otherwise.
Then hauls it all away for ingestion into its cloud.
Or, if it will fit in your station wagon, you can haul your data down the highway to one of Amazon's clouds.
Or put it in a box and ship it Fedex.
https://aws.amazon.com/snow/
I started from Tanenbaum’s hurtling station wagon and used my poetic license without worrying too much that the parent commentor would buy a forklift before talking to AWS sales.
Why doesn't Iron Mountain or some other competitor offer a service like this?
FWIW, Amazon isn't 'doing tapes' with this product. It's just a compatibility shim laid on top of object storage so that legacy tape backup/archival systems can eliminate physical tape libraries.
Isn't the whole point of tape that it is a physical thing, and may be taken offsite in a truck / stored in a vault, and all that jazz.
"Just reconfiguring" your backup software sounds like a business might just bypass all that without necessarily realising the consequences.
Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually there are no tapes, it's all the cloud now" -> ... ??
Taking tapes to a secure offsite location means they're air-gapped from the source data, so even in a situation where an entire network is compromised, remotely-stored tapes can't be wiped/encrypted.
I'm sure AWS has thought about this issue while designing Tape Gateway, but it's not clear to me how it could know whether a request to retrieve and overwrite a virtual tape was legitimate or not.
S3 offers WORM (Write Once Read Many) storage with Glacier and Object Lock. The slow retrieval for Glacier hints that AWS isn't using hard disks.
I'm not entirely comfortable with the trend towards ever more esoteric and seemingly "all or nothing" technologies.
Tape (and other physical things) may have their downsides, but I think there's something to be said for something that could (theoretically) be forgotten in a closet and read 50 years later.
Likewise, AM radio may not be the best quality, but it seems like you can cover more area with a single installation than any other communication technology, which might come in handy after a serious disaster like a nuclear war (e.g. crank up the nighttime power of some undamaged countryside station running on generators to tell survivors where it's still safe).
There's a sense in which tape is water in which backups swim.
The fastest way to enable large enterprise access to S3 and Glacier for backups was to meet them where they were. We did this by virtualizing a tape library.
Background: I'm one of the original inventors - https://image-ppubs.uspto.gov/dirsearch-public/print/downloa.... I am no longer with AWS.
This is a fantastic example of how broken the patent system is.
This is not an invention it’s a good implementation of a well understood problem.
It is also priced ridiculously - it is at least an order of magnitude more expensive than operating a physical library and remote physical storage that is beyond cyber threat by virtue of it being disconnected.
A real backup is offline and offsite.
Some of the solutions that external vendors produced were nightmarish and left customers up the creek without a paddle in a disturbing number of situations.
The last time I dealt with tapes was in the 1990s, and it was dead simple. We used tape backups at a company with only 150 people. I was the most junior admin so I had to be the tape monkey, but it was easy. Just run backups regularly, change the tape every couple of days and label it, ship a stack to the offsite storage and request the old stack back, and then do a test restore each time a stack came back.
The whole thing took maybe two hours a week.
When you're juggling 12 plates, a couple are gonna fall.
Everything else got easier, bigger, and faster.
Two decades back, you needed tape to cost effectively back up hundreds of gigabytes. These days, pretty much unless you have a spare room with an automatic tape library (robots moving tapes), you're too small for tape to make sense.
https://en.wikipedia.org/wiki/Tape_drive#History
https://en.wikipedia.org/wiki/Tape_library
So, yeah - virtual tape library, while obviously having downsides, is massively useful for a small enterprise.
That’s a lot of 9s.
> Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed out that it is impossible to calculate such a remote possibility with any scientific rigor.
https://en.wikipedia.org/wiki/Rogers_Commission_Report
Unless S3 can survive nuclear war or an asteroid hitting the earth, I don’t buy it.
I'm personally fine with the claim as I understand the methodology and assumptions but it has to be interpreted carefully.
You can use rclone to backup to an S3 bucket, and then have the S3 bucket set to "instantly" move files to deep archive.
To retrieve them, you will need to run an extra command to "unarchive" them for some period of time.
I find this easier than trying to interact directly with any Deep Archive API.
It is very cheap, and AWS's durability guarantees are impressive.
Main S3 page, linked to a note about Deep Archive https://rclone.org/s3/#glacier-and-glacier-deep-archive
The only thing I am not sure on is how to quickly restore a large number of files in a bucket from GDA to S3. It obviously can be scripted, but I don't have that handy. I only keep a small number of large, previously encrypted files in it, so I manually restore from the GUI.
(By the way, rclone can transparently encrypt files and filenames client-side!)
https://www.backblaze.com/b2/cloud-storage-pricing.html
edit: it integrates with TrueNAS (formerly FreeNAS) so it's been pretty much set-and-forget, but I can check in and see my stuff through their management interface. I do encrypt my docs before uploading, which TrueNAS also supports.
You should be able to accomplish the same, but you may need to either re-organize your stuff or set up multiple tasks.
edit: for clarification, "/mnt/main/media" contains various other subdirectories of Linux ISOs etc., which is why I push /mnt/main/media/Photos specifically.
The pro is that it has a client and, once setup, it just works.
The con is that it's a client running on a single system and the personal pricing plan doesn't support a NAS I believe. I periodically copy my files over from a RAID NAS to a local USB.
Found it a lot easier than rolling my own AWS Glacier solution.
I'm using CrashPlan for Small Business, so I can backup select NAS shares as well. $10/month.
You can use rclone as one simple tool.
1TB = 3.2€
5TB = 10.9€
10TB = 20.8€
...
https://www.hetzner.com/storage/storage-box
No transfer fees.
Sometimes people build things/products/companies/whatever not because of the chase of money, but because they want to put their idea in the world. Maybe it's just a case of someone believing they can build something grander :)
Some abstractions can be proven to be objectively-better solutions, along some axes you care about, for your use-case, such that the existing abstraction you were relying upon is then provably sub-optimal. Switching to the more-optimal abstraction is not then an "homage to the brilliance" of the company that created the abstraction; it's a simple "pay CapEx to lower OpEx in a way that will pay itself back after N years" decision. If you can budget it, you do it.
I'm not saying that every abstraction that the IaaS vendors invent is this kind of better mouse-trap, mind you. You can usually distinguish the ones that are, because they get independently reimplemented and used by people who aren't just aiming to build "cloud-native" software, but instead just really want/need the semantics of that particular abstraction, independent of where it runs.
Object storage is a great example of an abstraction where that is the case. There are many things that do object storage now — both cloud services, and regular deployable software. In fact, even the more-traditional backup vendors, like Backblaze, now also offer object-storage APIs.
I'm fairly critical of Amazon's engineering culture and practices but as a customer I have to admit they are head and shoulders above their competition and frankly most SaaS providers as a whole.
They've secured millions of dollars in revenue from me alone just by being so customer friendly.
I've worked with some of the smartest people in the tech field, building products from scratch for billion dollar customers and learning every day.
I've always told people that it's probably the best place you can start out of college.
EDIT: This was back in 2016, I'm not sure how much has changed without Bezos and Jassy at helm of AWS
Unless your willing to put up with a huge ration of shit on a regular basis, or you make it into management, your time is limited by the up and out mentality of the organization as a whole.
Amazon has created the perfect scenario to spend shitloads of money getting talent up to speed, only to part ways with them shortly thereafter.
Management at AWS arguably puts up with much more shit on a regular basis.
* working in infrastructure means doing a lot not quite so exciting things. most people aren't actually excited by ops, and a lot of your dev work will be on ops. its also the case that ops was historically not as well rewarded, though thats been fixed
* every service needs to be held to the highest tier of operational excellence, and so you can't just chabuduo when things aren't working as well as they should. in the worst case, this means throwing bodies to the ops meatgrinder.
* delivery speed is slowwwww. some of this in endemic to the space, but aws at the end of 2021 (when i left :) ) was a lot slower than than when i joined due to so much risk aversion and bureaucracy
* comp wasn't as good compared many tech peers, and comp could very easily not keep pace if you weren't tippy top every year. this is imo the most self-defeating part of amazon culture as a whole because it incentivizes people rated between ~p10-p20 and p50 to leave.
I feel like this is true for every single large company I’ve ever worked for. Things always feel like they’re regressing from a high point. I have no idea why it it.
All of them had free coffee when I joined, and they’ve slowly removed the vending machines and just never replaced them.
I've worked at Amazon for over 4 years. During this period, I saw around 80% of all team members being promoted to customer, perhaps half of them voluntarily. Amazon provided psychological service to those who felt they needed it, and a couple who quit confided in me they used the service. One guy overtly burned out and never showed up again.
Long work hours were the norm. Plenty of CRs were posted after 10PM. Even SDE3s frequently posted CRs on Sundays. The senior manager was hell bent on proving leadership he could deliver projects in impossible deadlines and not rarely did he warned SDEs that failing to deliver on those promises were firable offenses. This in turn fostered a work environment that led to some backstabbing and less than ethical stunts from team members prone to those and desperate to make a good impression.
It's true that Amazon's salary structure is designed to push out people. It is designed to impose a hefty pay cut after the second year or so. Nevertheless this is not the problem. The problem is the awful corporate culture that's designed to hold SDEs responsible for not delivering things even in impossible scenarios, and the backstabbing culture it sprouts by design.
But I'm sure some people might have a good time working there.
> The senior manager was hell bent on proving leadership he could deliver projects in impossible deadlines and not rarely did he warned SDEs that failing to deliver on those promises were firable offenses. This in turn fostered a work environment that led to some backstabbing and less than ethical stunts from team members prone to those and desperate to make a good impression.
is an extreme situation and one I never saw across ~3 orgs between amazon and aws.
I've been there 5 years, haven't worked more than 40 hours a week ever, and a lot of the people that started at the same time as me are in the same situation. And of those that aren't, they left because they wanted new experiences, not because they didn't like Amazon.
I never had to put up with any "shit", whatever that could mean.
It can’t be all bad, but it’s clearly not all good either.
It kind of makes sense that the people still there wouldn’t have bad stories to tell. What do you think is the worst part about working for Amazon?
Spending the money to recruit the employee, then pay them for a year or two, only to lose this talent is a great way to light cash on fire. That ain't cheap when your talking about hiring software developers!
It's relatively common that people don't make the cut during their probation period for example. Amazon (and other FAANGs) clearly isn't for everyone.
So I would question anyone that takes the easiest job they can find (their old job) after leaving Amazon in less that a year or two.
Isn't the average tenure at Amazon less than 3 years? "Most" of the people who you'd ask would be fresh from the interview rounds and with only one or two performance review rounds under their belt.
But I agree their support is very responsive.
IIRC, they may not be pretending. It's not public knowledge what storage medium AWS glacier is built on, but many speculate it is actually tape.
Edit: found in another thread here, the inventor linked the patent (https://image-ppubs.uspto.gov/dirsearch-public/print/downloa...) which confirms it's virtual tapes, not actual tapes.
No, just like it is not weird and definitely not illegal for the things they call “virtual machines” to end up being run on actual machines.
There's still isolation, there's just no virtualisation abstraction.
Is that not misleading?
They physical tapes, if any, would be many layers down.
At announcement time Glacier was using super dense racks of hard disks but only powering a subset of them. This led to some tape-like behaviors, like your restore request might take a long time to process because your data was likely stored on disks that weren't powered on and you'd need to wait for them to come online.
We talked about how we want to change our design to copy AWS. Because it's standards, and our users (internal users) know how to use it.
Your disaster recovery plan will need to budget for extra egress costs if you ever need to restore something big from one of those virtual tapes. AWS charges for egress -- downbound -- bandwidth but they don't charge for ingress -- upbound -- bandwidth.
(And I think it's funny their icons and images look like old school DECTapes.)
Closed, works as intended: This is the industry-standard price model for all ransomware.
/light sarcasm
That being said, kudos to Amazon for having a solution for those who aren't in my circles and still use them.