Its essentially saying I'm busy and negligent. Now I'm mad big company didn't come to my rescue.
> scaling our business from 3 to 10 and then 100 people, raising money, editing our film, and planning our release. Oh, and a virus shut down the world, I moved two times, and our business went completely digital.
> and amid 18 hour workdays,
The quantified communication with company provides no details to what was said.
Maybe op was right, maybe they weren't but this sounds like whining without any evidence.
I understand AWS UX is quite bad (I have also personally been bitten by GCP UX) and would be more sympathetic with the OP if this were the cost for 1 or 2 months.
If OP didn't notice it for 8-9 months, I do think it's unfair to blame the entire amount on AWS.
I'm right there with you. This new generation of "medium authors" is really not sticking the landing. Almost every one of these articles could just be renamed to "My Rookie Mistake And How I'm Going To Make It Into A 1,500 Word Blog Post"
> Messages often took a full month to get a response.
Also does not ring true to my experience with AWS unless the author was not paying for any support tier. If they did not have even developer support on the account, then they got exactly what they paid for.
The little animation shows some alert thrown by AWS when the author goes through their process, but it goes by too fast for me to make out the details. I'm not going to watch it 100 times to get it all, but I do see phrases about copies and folder objects that might be relevant to this issue. Perhaps warning about the limits of the action being taken? I can't tell.
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle policy"
"This action applies to all objects within the specified folders"
I'd assume the first one means youll get charged once for standard and once for glacier
It claims that the operation will make a copy instead of transition the objects, which as far as I understood it would have then deleted the original objects and accomplished something relatively similar but with an extra copy step.
Turn on anomaly detection in your billing, it will pay for itself within 3 months. Everyone screws up something in AWS eventually.
We had a run away EFS volume that burnt through $300 in a day, luckily was caught by an alarm. Thats basically $8700 saved if accounting had caught it first.
We're a film company with contractors, and I was raising money as we went. I also said that under slightly different circumstances (some last-minute investment) we could have been bankrupted by this.
> After this experience, I cannot honestly recommend using AWS unless you have a professional sysadmin operate it for you, even if you think you know what you’re doing.
Maybe you should step back and think why people like us put our blood and sweat and it still takes years to learn how to administer infrastructure properly. Stop whining and learn to pay for skills.
Is the comparison of Pastry Chef vs. a Rotisseur closer to your sensibilities?
The comparison the prior comment made was something along the lines of "both work in very similar setting and industry, but each have their own skill set and expertise." Each can probably function in the the others job, but not well and probably at a greater cost.
As an experienced sysadmin who reflexively triple checks for landmines before taking any action, I feel like my days are spent operating software with shitty UIs. It's like I am that person from Star Trek who repeats what the computer says and tells it what to do.
Just because the current status quo of things is this way does not mean it is great. The world would be a better place if more people were empowered to setup computer systems in a safe way.
Out of curiosity, what would market rate for a sysadmin run to order a snowball and sync 24TB of data to it, ensuring that it ends up properly in Deep Glacier?
We had something similar happen with lifecycle rules, moving an entire bucket to glacier: end result was > $200k and AWS weren’t especially helpful on our attempts to refund us.
We were unaware that changes via a rule still count as an api hit per object, despite all happening in the backend. I’m sure we can’t be the only people hit by this.
I almost did that. Someone double checking the plans raised that as an issue and instead we went with a lifecycle at some specific age rule which didn't try moving everything at once.
And yes, it's both fairly well documented and something that can catch you by surprise at the same time.
The reality is that Amazon makes big bucks off the complete opacity of their costs. If you offer a UI where a click costs money, you should also display price.
But that's just my opinion. Maybe others like buying things from a menu without prices.
DigitalOcean emailed me once when I left my single $5/month instance open by mistake. AWS would never do that even for thousands.
I've never worked with a tool that made me this scared to use it. I mean it.
One wrong click and you can blow your budget, crash you entire system, the works. Over the years I've grown used to it but the feeling of uneasiness will never leave me. I get it, AWS is very powerful but it does not give me the comfy feelings of using a product like Render or DigitalOcean, where I pay for what I need and that's the end of it.
> It would be nice to get a hard-cap on AWS costs.
> I don't believe you can "halt everything" if you go over a certain amount though.
You're pretty much on point. Handling a hard cut is simply not feasible for nearly all services. I.e. simply sending E-Mails in the middle of a news letter is not useful. You can have reasonable actions for some services (i.e. disallow new EC2 machines, add the end-of-month-cost for running ones to the bill), but this could break automation in strange ways (i.e. you suddenly could not spin up a large instance temporarily due to the limit). And how would you handle traffic? Disallow peaks? What about API gateway or other usage-based services?
Lastly, AWS is all about scalability. Breaking so much, especially in the scalability space, for such a small feature is simply not worth it for them. So usually they're simply lenient on bills and take the occasional beating (and it's not like there would not be an equal amount of articles on "how AWS caps broke our system [and it did not warn us/the UI was bad/ ...]").
The missing hard cap is the reason I don't have an account there, but I can absolutely understand why they don't have it.
For the budget, you can set alarms about your expenditures. For the rest, I don't think I've ever worked with any piece of infrastructure that couldn't be crashed with one wrong click or keystroke. I've seen it lots of times, at least a few times a year: theret are dozens of systems and countless servers, and sometimes human error bring one down. You roll back the changes, reboot, or whatever action is called for and move on.
What's different is the perception that cloud infrastructure should be significantly easier. But the ways in which it's easier are not in the area of mitigating the impacts of human error.
I dunno... it might be bad UX. I have definitely been annoyed with limitations to bulk edit abilities from S3 console before.
But I've been in this business for a while. I'm a programmer rather than an ops person, but i still sometimes make changes like that. I am responsible for some S3 buckets of stuff whose cost is significant for our budget.
I would never make a change in any software, where a failure for the change to take would cost me a noticeable part of budget, and not do even a cursory check to see if the change took. I mean, I write software, I know how it goes.
Who does that without ever checking to see if it worked? I don't do anything that matters without double-checking to see if it worked like I expected, at least if I'm doing it for the first (or second, or third) time, whether via a console or a script. Cause if you work with computers, you know they don't do what you expected all the damn time.
You're making a new kind of change you have never done before? You try it with a few objects before trying it on your whole bucket. You double-check if it did what you expected. Then you do it to everything. Then you still double-check to make sure it did it as expected to everything. (Even if you "read the manual" as the OP says, it's not about that at all in fact). That's how engineering works. If you don't have time to do it carefully, then sure, your chances of making mistakes or misunderstandings goes up a lot, why would you expect someone else to be responsible for your mistakes from not being careful?
OP here. It's a film company and a lot of those 100 people are artists and contractors. I put a lot of my own money into the company and took out a PPP loan to keep us afloat. So no, we didn't have the money to hire cloud support.
I don't know, maybe there's a legitimate UX issue here, but the author also said it only took them an hour of reading up on Glacier to resolve. That seems a reasonable amount of effort to have put upfront.
It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges.
I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I don't expect them to step in like that for a problem sustained over the course of months.
> It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges.
This. I manage a 15k/month AWS account, and if the expected expenditures of any day are > 15$ off the normal, I try to find out what happened. Especially after changing stuff in S3 and EC2; as with our usage patterns they should have predictable billing.
> I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day.
I've once reserved instances in the wrong region, but (luckily) found this out the same day, and got it refunded the day after. Would have been the equivalent of a year of salary down the drain otherwise.
Okay that sucks and I understand where the author is coming from. There are definitely some rough edges in the AWS interface. I do believe as a large corporation Amazon does have some responsibility to waive some of the charge. In this case the intent was not to store things in S3 and just leave it there (especially if there was no access).
Amazon has traditionally been good about waiving charges for account compromises and things along that nature. However you do have to keep up on it since they bill you every month. If you're running a company part of the responsibility is at least checking in on your financials every so often. Right when you see a charge for $700+ when you expect $50 is a red flag. At that point if he had contacted Amazon I'd be surprised they didn't waive the charge.
I'd be curious if Amazon was willing to waive any part of the charges at all.
In their gif, the big alert right at the top, in the very first bullet point, says it will make a copy of the object. They intentionally scrolled down quickly in the gif to hide this.
Is the argument that this should have been a pop-up that required you to click “ok” instead?
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
> I also review my bills every month, which would have mitigated this issue significantly.
A cursory look would've been enough, really. If this would have been 20$ or so, I would not fault anyone for missing it. But 800$ missing every month should definitely get some attention, especially when these are amounts endangering your ability to stay afloat, as the author seems to argue.
Admittedly the feature that Azure has that AWS doesn't is "kill everything when exceed X dollars". Not so great for prod, but superb for dev.
Please realize that when you hand devs with AWS keys, you're assigning an unlimited liability to your company with whatever accident a dev might do. (Accidents happens, and quite often when experimenting.)
You haven't really explained what you mean. Basically I said the UX was adequate, you said it's not.
I'll elaborate: human error is not all reducible to a UX problem, at least not in any meaningful sense in casting blame at AWS here for causing the user's error. Giving users an explanation of their actions at the top of the page is not particularly bad UX, it seems adequate. If course they could do more, modals that say "here's your estimated costs for the action you're taking" and that might be better. But then again that might not be practical across the range of AWS modules, and a user that if ignores their bills and bank statements for most of a year might easily ignore those notices as well.
If you define every mistake a user makes as a UX problem then you've defined it broadly enough that it loses meaning. I doubt that's you're overall intent, but in claiming the user's failure to read the top of the page was a UX problem you're crossing over into that territory.
I might be more sympathetic if the user wasn't so lacking in awareness that they also didn't read their bills and bank statements for so many months, and didn't take the hour of time upfront that was ultimately necessary to understand part of a massively complex system. No reasonable level of UX design would have fixed all of that.
I do think this is bad UX. The warning message (which basically says "this button doesn't do what it says it does") is commingled with a bunch of other less-important warning messages.
It's also a bit ambiguous-- it seems to be focused on how it will change the settings and last modified date. I could see how one could read that as "this action will create a copy of your files [and then delete the old ones] rather than moving them directly", warning you that your file metadata will be impacted.
There's also the problem that these warnings are everywhere in AWS console so you develop a bit of blindness to them.
So they have a pretty much wrong button name ("Edit Storage Class" --> "Copy to a different storage class") with an ambiguous warning message commingled with a bunch of other warning messages. I'd call that bad UX.
That being said, 8 months is too long for this bill to be considered AWS' fault. and the fact that he's been working really hard on his movie is irrelevant and comes off as whiny
Plus the following. UX is the umbrella term for "User Experience", not User Interface.
Therefore I'm not defining UX as users making mistakes. I'm defining UX as the user's motivations, journey, and actual experience with the site (and possibly extending to their interactions with the support team). This is not an 'overly broad definition', this is THE definition of UX.
If anything, what's happening here is the reverse. People are defending AWS' bad user experience, by narrowing down the definition to the narrow scope of UI-design and safety-nets. However, no amount of UI safety netting will salvage a website with bad UX (or rather, where UX never came into play during the design stage, except at the very late stages when it was only down to designing a UI).
So to reiterate. I'm not blaming AWS for the user's effective failure to read a technical warning message which then caused them harm. I'm blaming AWS for the fact that it even came to that in the first place. That is bad UX.
If it doesn't operate like Dropbox under the hood, don't make it look like Dropbox in your interface.
If an action creates a copy of objects, call it "Copy Objects" or whatever.
I think that presenting an S3 bucket as a set of files that you can select and perform operations on sets up false expectations here. And not showing incomplete multipart uploads in the file listing or having any way to see they are there isn't great.
The definition is overly broad because literally anything the user doesn't like can be blamed on bad UX design. It's too broad to be useful in understanding specific problems. It's like saying anything a person doesn't like in a restaurant was not cooked well.
Also user experience follows pretty heavily from UI design, you cannot separate the two without losing something. But "experience" is also much more subjective: Some might have a bad experience digging through menus while another person might have a good experience, like the way it's all organized. As an example, I found the notification for this issue perfectly acceptable: it let the user know a copy would be made. That is a pretty clear indication that the originals will still exist, and new ones would end up in Glacier. My experience there would be fine.
User experience is unhelpful in this respect because you cannot design your system to satisfy the experience of every user.
There's an entire discipline of UX best practices. Some of it is subjective but some of it more generally agreed upon. I don't think you'd find many UX designers who say AWS follows these, and it's up for debate the degree to which it's intentional.
Yep, but none of that means this particular issue was poor UX. Maybe if this particular issue was a very common problem then I would see things differently, but this story also comes from a person who failed to review billing statements from AWS or their bank account transactions for most of a year. It's simply not a situation I can easily blame on AWS.
Please read the orange triangle warning you scrolled past.
When you did the glacier operation, the GUI was telling you that it was COPYING the files to glacier. Those files also existed elsewhere, in normal S3 space. There's ways to cure this issue at the command line, primarily a list of all attributes per file - and there you can see the respective storage plans.
Secondly, do you have billing alerts set up, at 1.25x per average day's accrual? (30$/mo would be around $1.25/day, as not to email-bomb you). Billing alerts would NOT have solved the issue of improper storage class, but you would have seen a massive spike in costs. It probably would have cost you maybe $100-$200 to solve it early on.
Thirdly, how much of the 5 Pillars of Well Architected Framework are you following? ( https://aws.amazon.com/blogs/apn/the-5-pillars-of-the-aws-we... ) Admittedly some of these are "pay AWS double and get disaster prevention" style rules. However... pillar 5 is all about cost optimization. An ounce of prevention is worth a pound (or $7k) of cure.
Ah that's interesting. In my support conversations with Amazon they never brought up copies of objects as a reason billing could be high, so I didn't look into it. They told me the issue was incomplete multipart uploads, so I just believed that.
My assumption when reading that message was that it would appear to update the storage class, but actually create a copy of objects with a different storage class and delete the old objects. It doesn't make sense that it would just duplicate my objects, and I still can't really imagine anyone wanting that. Why would a box called "Edit storage class" create a copy and then not destroy it? Why not make a dialog box that just...edited the storage class?
So maybe I had a previous version of all my files sitting in regular S3, with the current version sitting in Deep Glacier, and I may have just happened to delete previous versions of objects when I created the lifecycle rule.
To answer your question -- I don't think we're following any of the pillars of well architected frameworks. I'm a filmmaker who knows Javascript and PHP and not system architecture, and I was looking for a cheap way to not lose our movie to the ravages of time. I'm definitely not the only one.
Seems to me that there are no “move” operations in S3, only copy and delete (which mean exactly what they say). So copy still means copy, and if you implement your own “move” you have to make sure it does what you want.
Yep, which is what makes "creates a copy" in this warning seem like "copy and delete". Because just creating a copy and not deleting the original is nothing at all like editing storage class.
Yep, it's right there with a big warning sign saying that a copy will be created. This is something that any IT Pro would notice and read and then hopefully understand the impact.
The author can argue that the UX is bad, but they can't say it's "faulty" or "inaccurate".
Edit...
After fully reading the article, the author even says this: "If an interface doesn’t quite do what it appears to, it should give a warning." The warning is right there in the UI, captured perfectly by his own screengrab.
I suppose not in the Facebook day and age where your next door neighbour's dog getting a new pedicure warrants a big red alert in your news feed, among all other big red alerts.
It says that it will make a copy of the object. However, that was not what caused the charges.
The charges were caused because (a) selecting all does not select incomplete multipart uploads, which are hidden in the interface, and (b) I believe the operation is not recursive.
If it created a copy of the object as described, the charges would go away.
The presence of a warning or documentation does not mean there isn't a problem here. The word 'edit' implies it will apply to the selection, which is different than copying the item to the new storage class. This is akin to 'solving' a user interface problem with training instead of fixing the underlying issue. Amazon failed to properly incorporate human factors in their design.
According to support, the issue was incomplete multipart uploads, not the creation of a copy, though. So I don't think that warning relates to the overcharge.
You are mistaking a technical explanation for good UX.
The warning may have well just said please note that the combothrombobulator may break existing encabulator nodes. If what you need is duckbill-style joints instead, use a thermocoupling-paradigm instillator instead.
When it should have said "THIS MAKES EXTRA COPIES WHICH COST MORE".
I've never used AWS before so you can consider this a "fresh perspective". In defense of AWS, the warning starts with "This action creates a copy of the object". In defense of the author, "Edit storage class" is a pretty misleading name for the operation (how about "Copy with new storage class?"), and perhaps these "warnings" show up under other more routine operations frequently enough --- and unimportantly-enough to cause warning-fatigue.
However, given the nature of cloud services in general (they have a strong $$$ motivation to nickel-and-dime you for every little thing they can) and especially AWS' reputation for doing that, I likely would not have made this mistake. Then again, I'm also someone who carefully reads all available information --- especially whenever it's something I'm paying for.
As far as I understand it, the operation creates a copy and deletes the original, which should succeed in reducing costs. What it doesn't do is delete incomplete multipart uploads and it may not operate recursively.
A lot of people have this mental model that The Cloud is Magic™.
They say things like this with a straight face:
- The cloud is always cheaper than on-prem!
- The cloud has no downtime, unlike on-prem!
Recently, I came across a large government department that had moved a reasonable volume of stuff into Azure. Think 7-figure annual bills.
They didn't like that they were overspending, so they complained to Microsoft, essentially demanding that they "make good" on their promise to make the cloud cheaper by cutting the cost on resources. This would have required an 80% discount, maybe higher. It's an absurd thing to even ask, but ask they did. Microsoft said no.
Meanwhile, I looked at their tenant, and the stuff that goes on there is amateur hour: Daily uncompressed database backups kept forever in premium zone redundant storage. Dozens of misconfigurations like that just burning money.
The worst part of it is that the "Azure Advisor" tab on the portal literally calls this kind of stuff out, recommending fixes worth hundreds of thousands of dollars a year, many with a one button "Quick Fix!" that you can press for an instant discount.
Nope. No. They're not going to bother reading things, or checking the Cost Management portal every now and then. They'd rather complain to their Microsoft rep every few months in a huffy tone.
"You still haven't cut my costs! Why haven't you!? I thought that's what you people did!"
They're busy people and don't have the time to waste cutting half a million dollars of spend here or there with arduous tasks like button pressing.
> A lot of people have this mental model that The Cloud is Magic™.
An consultant specialising in AWS once tried to convince me that moving our large scale computation into the cloud would reduce the environmental impact to nil.
Amazing what wild theories people will come up with when it's hidden behind a curtain.
I use AWS daily. It takes a few days to figure out those warnings are mostly there because some customer found out the hard way what they mean. You ignore them at your own peril.
And seriously:
> This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule.
I hate the AWS Console UI for a bunch of reasons, but short of making you type that sentence before you proceed, I'm not sure what AWS can do here.
Could they calculate and show an estimate of the monthly charges based on current objects in the bucket before and after the change in storage class? OP may still have clicked through w/o heeding, but I can see that as a potential improvement to the warning message where a user would then be forced to understand why it didn't affect pricing in the way they assumed it would.
While I would love to see AWS do this for most things, in this case, it would have resulted in a UI that said something like "Increased monthly cost expected to be $50", and I'm sure they would have read that as "Expected monthly cost change to $50"
But an estimated cost for many things in the console would be a welcome sanity check.
Is S3 marketed towards nontechnical folk like this? The other comments in this thread do a good enough job at dispelling any need for sympathy to the affected party, but my question is how they ended up inside the AWS console in the first place? It looks like the guy had just enough knowledge to shoot himself in the foot.
> Is S3 marketed towards nontechnical folk like this?
Honestly, even as a quite technical person, understanding AWS is not that easy. I've been in the situation of trying to run a very small app with basically zero budget on AWS a few years ago, and the sheer amount of services I needed to involve and things that needed to look for where quite overwhelming. I can easily imagine a non-technical person getting completely lost in the interface.
I "ended up in the AWS console" because S3 was a cheap way to store data so I decided to use their product. I've been a casual web developer, hacked together complex Javascript webapps, built Chrome extensions know my way around the terminal. Enough knowledge to be dangerous but not a professional.
On AWS, I know how to manage IAM roles, set up EC2 instances, etc. etc.
The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
I don't mean to undermine your skills, but by analogy: I know that sailboats are expensive, and I've sailed a few times, but I wouldn't buy a build-your-own-sailboat kit because it's cheaper and I know how to sail.
I agree with you about this, certainly:
> The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
I'm far from the only person who
- has needs that can be met by cloud providers at costs I can afford
- can't afford to pay a sysadmin
- has enough technical skills to hack my way through it
But this is also like saying "You should have hired an expert to tell you that when the sailboat builder kit manual says drill into the back it actually means the front." Regardless of whether someone more experienced at this would know better, this UX is awful, its title is inaccurate, and the whole process around changing storage classes far more confusing than the UX makes it seem.
There are petabytes and petabytes of data from independent filmmakers that are rotting away on drives, and part of the issue is lack of other affordable options. So I get that "shoulda hired a sysadmin" is an easy response but that's not always an option, especially when there's dozens of hard drives or other complexities that make these tasks last weeks or months.
Amazon makes a bajillion dollars a second. They could have made him happy and an evangelist for their customer service at zero impact to them. They just wanted to be dicks about it.
I was an evangelist for the company. I posted regularly about how good of a solution S3 was for filmmakers and was offering to help friends back up their 10-20 terabytes to the cloud.
Not anymore.
And folks, this is why you hire system administrators. We understand how stuff runs, is stored, backup procedures, cost expenditures and savings, and more.
We also know dev work, but it's not our specialty. We also know how to manage projects but we're not project managers. We also know how to handle billing, but we're not accountants.
... But our job encompasses all of them. And we reach out when we don't know. Our job is to make the trains run on schedule, using the average amount of fuel, with nobody stressing, and the customers happy. You get rid of us.. and... $7k of overage is nothing.
Out of curiosity, what would market rate for a sysadmin run to order a snowball and sync 24TB of data to it, ensuring that it ends up properly in Deep Glacier?
Since you mentioned this is for a startup for the film industry, having a permanent sysad on the roles doesn't make sense. (Rates vary from 60k-200k depending on experience, expertise, area).
Instead, I'd look at contract system administration labor. It'll probably set you back 100-150$/hr with minimums of 2h. However, sysads can catch problems, set up alerting, and fix structural issues before they turn into humongous problems.
It would be my recommendation to do more hours with onboarding a consultant and a full audit (not SOX or FedRAMP or the like - but to get you aware of what you're doing even if you weren't aware). From there, a plan can be made to keep you up and running with low/no impacts. And once the hard part's done, should insulate you from any oopses. And the monitoring will help catching them in the act in case there's a new type of oops.
Given that I was pretty knowledgeable about most aspects of backing my data up to S3, it would have been pretty hard to predict where the possibility for error would have been. I can't imagine hiring someone for 2hr to come and check my work, since they really would have had to be there for the entire process.
It sounds like about 2-5 days of work to get someone to set all this up, but perhaps it's more like 1 day at a minimum. So somewhere between $800-4k for that. Later in our project we were running 5 S3 buckets with about 20 stakeholders uploading and downloading, so that could have easily been another 4 days of work.
$800 would have been two years of what we were planning to pay Amazon. $4-8k would have been a major big-ticket item for us (even though, of course, that's the bill we ended up stuck with.)
If I were to do it over again I don't think I'd be able to justify spending on a pro. I would definitely spend on someone to audit our expenses more carefully though.
Ah, reminds me of the time I ran through $15k worth of compute in all of 3 weeks trying to do biological modeling. What was strange was that we were being charged for higher bandwidth even though it was not said to be any more for that same instance even if the usage stayed the same. We switched to Digital Ocean after that and that problem went away and had an all you can eat set of arguably better systems (32 CPUs vs 16) for $640 a month per device. So yeah, really do your homework, keep up with the notifications, and realize that they LOVE people like you! Why do you think cloud companies are so rich? Every startup idea gets put on there and either forgotten or blows up and ends up spending more in services. So yes, keep this in mind moving forward and dont listen to people who swear by AWS for x y of a feature because chances are they dont actually know what's going on behind the hood but have had to he trained it or heard about it in tech interviews if they are a cs person.
116 comments
[ 2.9 ms ] story [ 186 ms ] threadIts essentially saying I'm busy and negligent. Now I'm mad big company didn't come to my rescue.
> scaling our business from 3 to 10 and then 100 people, raising money, editing our film, and planning our release. Oh, and a virus shut down the world, I moved two times, and our business went completely digital.
> and amid 18 hour workdays,
The quantified communication with company provides no details to what was said.
Maybe op was right, maybe they weren't but this sounds like whining without any evidence.
If OP didn't notice it for 8-9 months, I do think it's unfair to blame the entire amount on AWS.
> Messages often took a full month to get a response.
Also does not ring true to my experience with AWS unless the author was not paying for any support tier. If they did not have even developer support on the account, then they got exactly what they paid for.
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
"This action applies to all objects within the specified folders"
I'd assume the first one means youll get charged once for standard and once for glacier
We had a run away EFS volume that burnt through $300 in a day, luckily was caught by an alarm. Thats basically $8700 saved if accounting had caught it first.
This article should have been written in a less bombastic fashion, I felt at the end like it was just an advertisement for a film.
Sorry you didn't like the tone of the article.
Maybe you should step back and think why people like us put our blood and sweat and it still takes years to learn how to administer infrastructure properly. Stop whining and learn to pay for skills.
The comparison the prior comment made was something along the lines of "both work in very similar setting and industry, but each have their own skill set and expertise." Each can probably function in the the others job, but not well and probably at a greater cost.
Just because the current status quo of things is this way does not mean it is great. The world would be a better place if more people were empowered to setup computer systems in a safe way.
We were unaware that changes via a rule still count as an api hit per object, despite all happening in the backend. I’m sure we can’t be the only people hit by this.
And yes, it's both fairly well documented and something that can catch you by surprise at the same time.
But that's just my opinion. Maybe others like buying things from a menu without prices.
DigitalOcean emailed me once when I left my single $5/month instance open by mistake. AWS would never do that even for thousands.
Would it be so hard for them to redesign AWS to tell you what something's going to cost before you do it?
One wrong click and you can blow your budget, crash you entire system, the works. Over the years I've grown used to it but the feeling of uneasiness will never leave me. I get it, AWS is very powerful but it does not give me the comfy feelings of using a product like Render or DigitalOcean, where I pay for what I need and that's the end of it.
This tutorial includes how to setup alerts for a given budget: https://aws.amazon.com/getting-started/hands-on/control-your...
I don't believe you can "halt everything" if you go over a certain amount though. I'd be happy to be proved wrong.
> I don't believe you can "halt everything" if you go over a certain amount though.
You're pretty much on point. Handling a hard cut is simply not feasible for nearly all services. I.e. simply sending E-Mails in the middle of a news letter is not useful. You can have reasonable actions for some services (i.e. disallow new EC2 machines, add the end-of-month-cost for running ones to the bill), but this could break automation in strange ways (i.e. you suddenly could not spin up a large instance temporarily due to the limit). And how would you handle traffic? Disallow peaks? What about API gateway or other usage-based services?
Lastly, AWS is all about scalability. Breaking so much, especially in the scalability space, for such a small feature is simply not worth it for them. So usually they're simply lenient on bills and take the occasional beating (and it's not like there would not be an equal amount of articles on "how AWS caps broke our system [and it did not warn us/the UI was bad/ ...]").
The missing hard cap is the reason I don't have an account there, but I can absolutely understand why they don't have it.
I've never been in a situation where I wanted it decreased, but I guess you could setup an account and ask for a decrease.
If enough people ask for it, they might make it something you can do yourself.
What's different is the perception that cloud infrastructure should be significantly easier. But the ways in which it's easier are not in the area of mitigating the impacts of human error.
But I've been in this business for a while. I'm a programmer rather than an ops person, but i still sometimes make changes like that. I am responsible for some S3 buckets of stuff whose cost is significant for our budget.
I would never make a change in any software, where a failure for the change to take would cost me a noticeable part of budget, and not do even a cursory check to see if the change took. I mean, I write software, I know how it goes.
Who does that without ever checking to see if it worked? I don't do anything that matters without double-checking to see if it worked like I expected, at least if I'm doing it for the first (or second, or third) time, whether via a console or a script. Cause if you work with computers, you know they don't do what you expected all the damn time.
You're making a new kind of change you have never done before? You try it with a few objects before trying it on your whole bucket. You double-check if it did what you expected. Then you do it to everything. Then you still double-check to make sure it did it as expected to everything. (Even if you "read the manual" as the OP says, it's not about that at all in fact). That's how engineering works. If you don't have time to do it carefully, then sure, your chances of making mistakes or misunderstandings goes up a lot, why would you expect someone else to be responsible for your mistakes from not being careful?
While the cloud is just someone else's hardware, you still pay for it like it's your own.
But he waited 9 months and kept paying. That's just too late.
It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges.
I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I don't expect them to step in like that for a problem sustained over the course of months.
This. I manage a 15k/month AWS account, and if the expected expenditures of any day are > 15$ off the normal, I try to find out what happened. Especially after changing stuff in S3 and EC2; as with our usage patterns they should have predictable billing.
> I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day.
I've once reserved instances in the wrong region, but (luckily) found this out the same day, and got it refunded the day after. Would have been the equivalent of a year of salary down the drain otherwise.
Amazon has traditionally been good about waiving charges for account compromises and things along that nature. However you do have to keep up on it since they bill you every month. If you're running a company part of the responsibility is at least checking in on your financials every so often. Right when you see a charge for $700+ when you expect $50 is a red flag. At that point if he had contacted Amazon I'd be surprised they didn't waive the charge.
I'd be curious if Amazon was willing to waive any part of the charges at all.
Is the argument that this should have been a pop-up that required you to click “ok” instead?
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
https://i.imgur.com/Qjiu8F0.png
I also review my bills every month, which would have mitigated this issue significantly.
A cursory look would've been enough, really. If this would have been 20$ or so, I would not fault anyone for missing it. But 800$ missing every month should definitely get some attention, especially when these are amounts endangering your ability to stay afloat, as the author seems to argue.
Admittedly the feature that Azure has that AWS doesn't is "kill everything when exceed X dollars". Not so great for prod, but superb for dev.
Please realize that when you hand devs with AWS keys, you're assigning an unlimited liability to your company with whatever accident a dev might do. (Accidents happens, and quite often when experimenting.)
I'll elaborate: human error is not all reducible to a UX problem, at least not in any meaningful sense in casting blame at AWS here for causing the user's error. Giving users an explanation of their actions at the top of the page is not particularly bad UX, it seems adequate. If course they could do more, modals that say "here's your estimated costs for the action you're taking" and that might be better. But then again that might not be practical across the range of AWS modules, and a user that if ignores their bills and bank statements for most of a year might easily ignore those notices as well.
If you define every mistake a user makes as a UX problem then you've defined it broadly enough that it loses meaning. I doubt that's you're overall intent, but in claiming the user's failure to read the top of the page was a UX problem you're crossing over into that territory.
I might be more sympathetic if the user wasn't so lacking in awareness that they also didn't read their bills and bank statements for so many months, and didn't take the hour of time upfront that was ultimately necessary to understand part of a massively complex system. No reasonable level of UX design would have fixed all of that.
It's also a bit ambiguous-- it seems to be focused on how it will change the settings and last modified date. I could see how one could read that as "this action will create a copy of your files [and then delete the old ones] rather than moving them directly", warning you that your file metadata will be impacted.
There's also the problem that these warnings are everywhere in AWS console so you develop a bit of blindness to them.
So they have a pretty much wrong button name ("Edit Storage Class" --> "Copy to a different storage class") with an ambiguous warning message commingled with a bunch of other warning messages. I'd call that bad UX.
That being said, 8 months is too long for this bill to be considered AWS' fault. and the fact that he's been working really hard on his movie is irrelevant and comes off as whiny
Plus the following. UX is the umbrella term for "User Experience", not User Interface.
Therefore I'm not defining UX as users making mistakes. I'm defining UX as the user's motivations, journey, and actual experience with the site (and possibly extending to their interactions with the support team). This is not an 'overly broad definition', this is THE definition of UX.
If anything, what's happening here is the reverse. People are defending AWS' bad user experience, by narrowing down the definition to the narrow scope of UI-design and safety-nets. However, no amount of UI safety netting will salvage a website with bad UX (or rather, where UX never came into play during the design stage, except at the very late stages when it was only down to designing a UI).
So to reiterate. I'm not blaming AWS for the user's effective failure to read a technical warning message which then caused them harm. I'm blaming AWS for the fact that it even came to that in the first place. That is bad UX.
If it doesn't operate like Dropbox under the hood, don't make it look like Dropbox in your interface.
If an action creates a copy of objects, call it "Copy Objects" or whatever.
I think that presenting an S3 bucket as a set of files that you can select and perform operations on sets up false expectations here. And not showing incomplete multipart uploads in the file listing or having any way to see they are there isn't great.
Also user experience follows pretty heavily from UI design, you cannot separate the two without losing something. But "experience" is also much more subjective: Some might have a bad experience digging through menus while another person might have a good experience, like the way it's all organized. As an example, I found the notification for this issue perfectly acceptable: it let the user know a copy would be made. That is a pretty clear indication that the originals will still exist, and new ones would end up in Glacier. My experience there would be fine.
User experience is unhelpful in this respect because you cannot design your system to satisfy the experience of every user.
This is 100% negligence on their part, not AWS's fault. The cloud isn't magic, you still need to read documentation.
When you did the glacier operation, the GUI was telling you that it was COPYING the files to glacier. Those files also existed elsewhere, in normal S3 space. There's ways to cure this issue at the command line, primarily a list of all attributes per file - and there you can see the respective storage plans.
Secondly, do you have billing alerts set up, at 1.25x per average day's accrual? (30$/mo would be around $1.25/day, as not to email-bomb you). Billing alerts would NOT have solved the issue of improper storage class, but you would have seen a massive spike in costs. It probably would have cost you maybe $100-$200 to solve it early on.
Thirdly, how much of the 5 Pillars of Well Architected Framework are you following? ( https://aws.amazon.com/blogs/apn/the-5-pillars-of-the-aws-we... ) Admittedly some of these are "pay AWS double and get disaster prevention" style rules. However... pillar 5 is all about cost optimization. An ounce of prevention is worth a pound (or $7k) of cure.
My assumption when reading that message was that it would appear to update the storage class, but actually create a copy of objects with a different storage class and delete the old objects. It doesn't make sense that it would just duplicate my objects, and I still can't really imagine anyone wanting that. Why would a box called "Edit storage class" create a copy and then not destroy it? Why not make a dialog box that just...edited the storage class?
So maybe I had a previous version of all my files sitting in regular S3, with the current version sitting in Deep Glacier, and I may have just happened to delete previous versions of objects when I created the lifecycle rule.
To answer your question -- I don't think we're following any of the pillars of well architected frameworks. I'm a filmmaker who knows Javascript and PHP and not system architecture, and I was looking for a cheap way to not lose our movie to the ravages of time. I'm definitely not the only one.
> create a copy of objects with a different storage class and delete the old objects. It doesn't make sense that it would just duplicate my objects
Isn’t that exactly what 'copy' means in pretty much all IT contexts?
It seems that some "move" operations are "copy" operations, which is the context in which I understood this warning:
https://forums.aws.amazon.com/thread.jspa?messageID=455101#j...
The author can argue that the UX is bad, but they can't say it's "faulty" or "inaccurate".
Edit... After fully reading the article, the author even says this: "If an interface doesn’t quite do what it appears to, it should give a warning." The warning is right there in the UI, captured perfectly by his own screengrab.
Which is presumably why the article mentions the below in the second paragraph.
"After this experience, I cannot honestly recommend using AWS unless you have a professional sysadmin operate it for you"
"Sure, what do they say?"
"Who?"
"The errors"
"I don't know, I clicked them away"
A UX that has a high probability of causing harm is accurately described as faulty. I'm with OP here, technical warning-box being present aside.
The charges were caused because (a) selecting all does not select incomplete multipart uploads, which are hidden in the interface, and (b) I believe the operation is not recursive.
If it created a copy of the object as described, the charges would go away.
The warning may have well just said please note that the combothrombobulator may break existing encabulator nodes. If what you need is duckbill-style joints instead, use a thermocoupling-paradigm instillator instead.
When it should have said "THIS MAKES EXTRA COPIES WHICH COST MORE".
However, given the nature of cloud services in general (they have a strong $$$ motivation to nickel-and-dime you for every little thing they can) and especially AWS' reputation for doing that, I likely would not have made this mistake. Then again, I'm also someone who carefully reads all available information --- especially whenever it's something I'm paying for.
They say things like this with a straight face:
- The cloud is always cheaper than on-prem!
- The cloud has no downtime, unlike on-prem!
Recently, I came across a large government department that had moved a reasonable volume of stuff into Azure. Think 7-figure annual bills.
They didn't like that they were overspending, so they complained to Microsoft, essentially demanding that they "make good" on their promise to make the cloud cheaper by cutting the cost on resources. This would have required an 80% discount, maybe higher. It's an absurd thing to even ask, but ask they did. Microsoft said no.
Meanwhile, I looked at their tenant, and the stuff that goes on there is amateur hour: Daily uncompressed database backups kept forever in premium zone redundant storage. Dozens of misconfigurations like that just burning money.
The worst part of it is that the "Azure Advisor" tab on the portal literally calls this kind of stuff out, recommending fixes worth hundreds of thousands of dollars a year, many with a one button "Quick Fix!" that you can press for an instant discount.
Nope. No. They're not going to bother reading things, or checking the Cost Management portal every now and then. They'd rather complain to their Microsoft rep every few months in a huffy tone.
"You still haven't cut my costs! Why haven't you!? I thought that's what you people did!"
They're busy people and don't have the time to waste cutting half a million dollars of spend here or there with arduous tasks like button pressing.
An consultant specialising in AWS once tried to convince me that moving our large scale computation into the cloud would reduce the environmental impact to nil.
Amazing what wild theories people will come up with when it's hidden behind a curtain.
More like "...when it helps Amazon make more $$$."
Which is basically what everyone who has "drunk the Cloud-Aid" will be doing, not only for AWS but other cloud services.
And seriously:
> This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule.
I hate the AWS Console UI for a bunch of reasons, but short of making you type that sentence before you proceed, I'm not sure what AWS can do here.
But an estimated cost for many things in the console would be a welcome sanity check.
Honestly, even as a quite technical person, understanding AWS is not that easy. I've been in the situation of trying to run a very small app with basically zero budget on AWS a few years ago, and the sheer amount of services I needed to involve and things that needed to look for where quite overwhelming. I can easily imagine a non-technical person getting completely lost in the interface.
Now there are 6 different storage classes and loads of options to manage the objects and buckets.
Interestingly the latest storage class(S3 Intelligent-Tiering) was created to minimize the overhead of S3.
On AWS, I know how to manage IAM roles, set up EC2 instances, etc. etc.
The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
I agree with you about this, certainly:
> The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
But this is also like saying "You should have hired an expert to tell you that when the sailboat builder kit manual says drill into the back it actually means the front." Regardless of whether someone more experienced at this would know better, this UX is awful, its title is inaccurate, and the whole process around changing storage classes far more confusing than the UX makes it seem.
There are petabytes and petabytes of data from independent filmmakers that are rotting away on drives, and part of the issue is lack of other affordable options. So I get that "shoulda hired a sysadmin" is an easy response but that's not always an option, especially when there's dozens of hard drives or other complexities that make these tasks last weeks or months.
We also know dev work, but it's not our specialty. We also know how to manage projects but we're not project managers. We also know how to handle billing, but we're not accountants.
... But our job encompasses all of them. And we reach out when we don't know. Our job is to make the trains run on schedule, using the average amount of fuel, with nobody stressing, and the customers happy. You get rid of us.. and... $7k of overage is nothing.
Instead, I'd look at contract system administration labor. It'll probably set you back 100-150$/hr with minimums of 2h. However, sysads can catch problems, set up alerting, and fix structural issues before they turn into humongous problems.
It would be my recommendation to do more hours with onboarding a consultant and a full audit (not SOX or FedRAMP or the like - but to get you aware of what you're doing even if you weren't aware). From there, a plan can be made to keep you up and running with low/no impacts. And once the hard part's done, should insulate you from any oopses. And the monitoring will help catching them in the act in case there's a new type of oops.
I hope I answered you adequately :)
Given that I was pretty knowledgeable about most aspects of backing my data up to S3, it would have been pretty hard to predict where the possibility for error would have been. I can't imagine hiring someone for 2hr to come and check my work, since they really would have had to be there for the entire process.
It sounds like about 2-5 days of work to get someone to set all this up, but perhaps it's more like 1 day at a minimum. So somewhere between $800-4k for that. Later in our project we were running 5 S3 buckets with about 20 stakeholders uploading and downloading, so that could have easily been another 4 days of work.
$800 would have been two years of what we were planning to pay Amazon. $4-8k would have been a major big-ticket item for us (even though, of course, that's the bill we ended up stuck with.)
If I were to do it over again I don't think I'd be able to justify spending on a pro. I would definitely spend on someone to audit our expenses more carefully though.