Console is flickering between "website is unavailable" and being up for my team. This is happening very frequently just now, reliability seems to have taken a hit.
One of our EC2 instances in us-east-1c is unavailable and stuck in "stopping" state after a force stop.
Interestingly enough, EC2 instances in us-east-1b don't seem to be affected.
The console is throwing errors from time to time. As usual no information on AWS status page.
I had the same issue with unavailable, but on an instance in us-east-1b. Finally just got the force stop to go through a minute ago and it's now running and available again.
> Can you expand on that? What feature do you use in east 1 that isn’t everywhere else that it’s your whole implementation?
Your question reads as a strawman. It matters nothing if EC2 is also available in Mumbai or Hong Kong if by default the whole world deploys everything and anything to us-east-1, and us-east-1 alone.
It's not a strawman. There's a huge difference between "AWS is down" and "customers don't know how to use AWS". For the people who use AWS correctly, they only had some degraded service, not downtime.
> It's not a strawman. There's a huge difference between "AWS is down" and "customers don't know how to use AWS".
Deploying a service to a single region is not, nor has it ever been, "customers don't know how to use AWS".
If anything, cargo culting this belief in global deployments being necessary, specially with services that have at most a regional demand, is a telltale sign a customer has no idea about what he is doing and is just mindlessly wasting money and engineering effort in something no one needs.
This blend of bad cargo cult advice sounds like a variant of microservices everywhere.
(Edit: I hope I didn't sound sarcastic. I don't open random console pages and scroll all the way down to check for new features. Some people will have noticed, some won't.)
My Elastic Beanstalk instances are completely unreachable. Seems at the very least ELB is down. Looking @ down detector it looks like this is taking a bunch of sites down with it. As usual AWS status page shows all green.
Where are you located? "X is down" without location is only moderately useful.
I'm having issues with Slack from central EU (Poland) -- can't upload images, or send emoji reactions to post; curiously, text works fine). Wondering if linked
Assuming crates.io is AWS-backed? Getting fun situation where direct dependencies of an application are downloading but then the sub-dependencies aren't.
I wonder if there's an s3 compatible service with similar pricing that can be used as a fallback? Are digital ocean s3 compatible storage accounts's backed by real s3?
afaik there's nothing tying it specifically to GH (where the metatada is), and then the actual code is just in an S3 bucket, so in theory should be reasonably easy [ha!] to just host anywhere. In theory, I mean that's a massive lump of stuff, and surely wherever it gets hosted is going to face exactly the same issues (though if it does become very widely used, then you'd think every major provider that controls infra could easily have a mirror)
The crates.io index is hosted on GitHub, but the application/API is hosted on Heroku (so in the us-east-1 AWS region) and the downloads on S3/CloudFront. And yes crates.io is currently impacted.
Slack seems to have some issues because of that - I'm not sure if anyone is receiving messages, as it became completely silent for the last 15 minutes or so.
Sending and receiving messages works here, but editing them does not, it throws an error. Statuses such as "calling" also do not seem to be updated any longer.
Edit: Restarting Slack does update the edited messages.
> We are experiencing issues with file uploads, message editing, and other services. We're currently investigating the issue and will provide a status update once we have more information.
I fail to understand how a big player like Slack can be impacted this way by a failure in a single AZ in a specific AWS region.
But at least the main feature (sending and displaying messages) is still working.
Similar with GCP. We had a pretty bad outage once where the status page was showing all green. Google informed us that because the actual issue was further down the stack and didn't trigger any internal SLOs the status didn't get an update. It took them hours to acknowledge and fix it.
I think they did eventually but it took us quite a bit of troubleshooting, then creating a P1 ticket, then their investigation in order to get to the bottom of it and getting it fixed. And the status page never got an update, which is the subject I was adding to.
This is a little more broad, beyond just cloud infra providers, but includes some of the kind of data you're looking for (post-mortems for outage events): https://github.com/danluu/post-mortems
You would think that but there always a few contrarian AWS evangelists in the comments going on about the "difficulty" in operating a status page as though it were trying to conjure a N=NP proof.
Like how come down detector can do a superb job of detecting when AWS goes down and AWS can't? Because AWS doesn't want account managers of SLAs asking for credits for the uptime they're paying for but not getting.
Depends on how many customers are ready to move to a different vendor. I suspect most customers are forgiving because either they were also down or half the services they use were down. You don't get fired for hosting in AWS.
Any place I've worked at that managed their own servers (to be fair, the last time I worked at a place like that was 2010) definitely had more protracted downtimes than AWS - it just felt not as bad because we were in control of the situation, but at the end of the day that didn't get us up any faster.
Another side benefit of being with AWS is when you do have an outage, a lot of other people have outages, and so you sort of blend in with the noise. It's not great to be down, but if you're down and also "big service X" who's also an AWS customer is down, it makes your downtime look less like a lack of competence and more like an unavoidable force of nature.
I guess it's extremely dependent on an org to org basis.
I worked at a company that's bread and butter was online services (e-commerce SaaS platform, similar to Netsuite) and we had significantly fewer outages than AWS had.
But we had redundancies built in to most things, I'm not saying it was perfect but it worked.
The major difference might be that almost nobody is willing to spend 20% of what they spend on AWS/GCP to have a self-hosted solution.
The reason "cloud is so expensive" is because they're essentially telling you what the price will be and even if they only spend 40% of that on actual hardware and operations: it's more than most companies would invest in themselves.
This is absurd, of course, but it's absolutely true.
AWS doesn't follow their own advice about hosting multi-regional.
When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it.
Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region
I didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.
There are some services that do have hard US-EAST-1 dependencies. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And there's also the notion of "global endpoints" like https://sts.amazonaws.com... it's not clear why that exists, because it fails when us-east-1 does. It would be better to only have regional endpoints if the "global" ones are region-specific in reality. The endpoint thing is documented, but it's still confusing to people.
The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.
If you're referring to Dec 7, it absolutely did. Metrics went down nearly across the board, which also means most auto-scaling setups were non-functional. Cloudfront metrics didn't properly recover until the next day
Logging in with root credentials was not possible in any region, and even logging in with IAM creds in other regions yielded an intermittently buggy console
and as is usual with us-east-1 outages management API calls were a complete crap shoot regardless of region
True sad fact. I first thought it is a management problem but lately I see it is the tech bros who push for fads in the hopes of staying relevant and not asuming responsability for choices.
Amazing. And as long as "technological progress" sufficiently obscures the impact of such ridiculousness the more such projects will continue to occur.
It's a real shame that the collective world of technology does not properly respect the simple solutions that work.
It is almost funny the dichotomy here. Most technological people "admire" the simplicity, elegance and extensibility of the command line. But tell those same people that the best data store for the solution is a relational database and their nose crinkles up.
Yeah, after getting caught up in the hype for ten years I'm running back to proven tech that is flushed out (Java, Swing-omg it just works, wanting to try Ruby,even PHP is looking good at this point).
Every dependency scrutinized and discarded if possible.
I would probably work for free if someone setup their own on-prem cloud in Tanzu, Open shift, or Ranger and used old school proven frameworks for development.
Working in AWS has been a real shitty experience at these large companies. All the nit picky problems (of which there are thousands) get dumped on devs who are trying to deliver working software.
(Accidentally down-voted, apologies! I would upvote to fix, but can't... Update: fixed)
Agreed. Arguably, not using an existing cloud service is a red flag on any new hires. AWS being the primary, but experience using GCS or Azure are at least viable skills, even if your business is AWS-based.
But the "fad-based-development" meme is not going away any time soon. The incentives in the business are built around it (really! No one want's to work on a boring old relational database solution any more). In the old days it was 4th generation languages, RUP, XML and Function Point Analysis... today it's functional programming, SDKs, big-three cloud PaaS experience or (shudder) block-chain.
I think back to my much younger self, when I thought that technology was something to be mastered to solve real-world problems, and I laugh. Little did I know the real problem to be solved was to figure out how to solve those same-old business problems but with the technology of the season (Kubernetes, GraphQL or ML).
I wonder to what extent this actually becomes less of a problem the more people use AWS. At this point AWS being down just feels like "the internet is down", it's hard for customers to be too mad at any company being down when all their competitors are too.
Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.
This seems like an interesting pendulum swing where the few companies not reliant on AWS could capture significant enough revenue by maintaining uptime during a potential busy season outage.
That depends on them being up the rest of the time. If they have an equal number of outages as AWS, it has the risk to make them look worse (since all attention is on them when they're out).
Maybe they'll just find themselves DDOSed by the sudden influx of visitors? As a small example, i think HN was slightly slower when FB had their outage.
I have a story from only a few years ago where the finance section, and a good portion of management, of Google had no idea how poor their GAE solution was for uptime, until they tried to do business critical work using software that was hosted on GAE.
Right. I've had an excellent experince with Vultr for the last couple years, for about 1/10th the cost of AWS. I use other small VPS providers as well. I run my own small business and I need to keep costs down to stay competitive. I used to use AWS more but the bill always creeps up to inappropriate levels. AWS billing is insulting, oh you forgot to renew your reserved instance? That's going to be double this month. I still use cloudfront, route 53, and a few of the smallest instances for mail servers and asterisk though. It's foolish to go all in with AWS, or with anything really.
"Don't use a self hosted monolithe, it's not reliable! You need a cloud FS with a load balancer under observability and your data in a db that scales horizontally, all orchestrated by kubs."
Meanwhile, I currently have a gig to work on a video service which features a never updated centos 6, an unsupported python 2 blob website, and a push to prod deployment procedure, running a single postgres db serving streaming for 4 millions users a month.
And it's got years of up time, cost 1/100th of AWS, and can be maintained by one dev.
Not saying "cloud is bad", but we got to stop screaming old techs are no good either.
Purely out of interest, I'd like to know more about your streaming architecture. I assume postgres just holds the meta data, and the actual video content is stored elsewhere? What strategies have you employed to scale the streaming part of your service? I imagine 4 million users a month is quite a significant amount of traffic!
1 - For the last 10 years, servers have been beasts. You have a lot of cores, plenty of HD and RAM. Servers are less expensives than devs. Scaling vertically can go VERY far.
2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary fresh data.
3 - if you compile nginx manually, you get to use a lot of plugins that can do stuff super fast, including serving videos. You can script stuff in lua that will just skip the backend completly.
4 - mind your encoding. We carefully chose how we encode videos. The ffmpeg parameters are pretty insane, but the space / quality ratio is amazing, espacially on mobile. It takes a lot of time to experiment with those, nobody share them :)
5 - we offload everything we can to cron tasks or task queues. Including, obviously, encoding, screenshooting, etc.
6 - don't hold data you can't lose. E.G: billing. This way you can have a relaxed attitute toward data. If we ever loose a day of business, users will be in a bad mood for a week, but that won't be the end of the world. We don't need a bullet proof system if bullets can't kill us.
7 - give money to ffmpeg and opencv, because damn those things are fast. And good.
8 - servers are hosted accross 2 providers. This way, if one goes down, or decide to stop doing business with us Google style, we have a second one. Happened recently with leaseweb: they shutdown a whole room without offering an alternative.
E.G: votes.
They don't hit the backend on write. We pile them from nginx to redis, then once a day, we aggregate and store on postgres, which the backends will consumme. We just store each vote on localstorage as well so that the user feels like it's real time when they vote, but in reality it's updated once a day. But votes don't affect the money side of our business, so if we lose them one day, it does not mean death.
P.S: yes, posgres/redis/elasticsearch only hold metadata. Videos are stored on disk. There is no docker images, no mircoservices, FS is ext4. Which means with a lot of RAM, the OS FS cache will have most popular videos already loaded and ready to be streamed. Everything is raid 0, so if we get one disk corrupted, you lose the server. But we upload each videos on severeal servers, so when a disk get corrupted, we just replace the whole server. In fact, anything goes wrong on a server, we replace it. It's not worth it to find the root cause, unless 2 servers die in the same way successively.
The main One Weird Trick that I remember from messing with ffmpeg is that for in-browser viewing, I had to generate a lot more mp4 reference frames than the default. This was because of deficiency in the HTML5 view extensions in browers a few years ago. I don't know if that is still an issue. The extra frames increased bandwidth consumption, but surprisingly, only slightly rather than big bloat.
4:35 AM PST We are investigating increased EC2 launched failures and networking connectivity issues for some instances in a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. Other Availability Zones within the US-EAST-1 Region are not affected by this issue.
Can anyone explain the affiliation of stop.lying.cloud to Amazon? All of the legalese in the header/footer seem to indicate it's actually owned and run by Amazon. If so... why? Why not just... use the real status page?
I mean I'm glad it exists, don't get me wrong. Just weird that they'd have two status pages, one seemingly existing only to sort of 'mock' themselves...
The people who maintain the unofficial site would have, at some point, used their CTRL and C keys followed not immediately, but closely by, their CTRL and V keys.
But that is copyright infringement. You're not allowed to copy some work, modify it, then slap the original copyright on it. This is an illegal website, prone to being taken down by AWS.
Actually, having a satire site taken down over copyright is one of the best ways to extort large amounts of money from the copyright holder, because constitutional attorneys will seep from the floorboards and appear in your shower trying to be an attorney on that case. Satire is extremely protected speech.
Yes mate, Internet Police Officer Jeff Bezos has been dispatched and will take this illegal website down right away.
(Using copyrighted material is permitted under fair use; this website is a parody. I’m not a lawyer but at some level preserving the copyright notice is probably better than claiming it as their own.)
This website is not a parody, and fair use does NOT permit you to retain the original copyright notice on the derived work.
You may say that the original work is copyright of the respective owners and that this is a parody work. But that's not what the site is doing. The footer contains the original, unaltered copyright, creating confusion as to who owns the derived work. Amazon does not own this, nor do they endorse it, so you're not allowed to say it's copyrighted by Amazon.
Nope. Because these laws also protect people who make such websites from the corporations they're commenting on, too. The respect must be mutual, else e.g. OSS has no basis for legal protection, either.
Being concerned for the proper respect of IP laws is something that benefits everyone.
You make more friends defending humans from big companies than you do defending big companies from humans.
Your argument would have a small amount of merit if you acknowledged that the laws DO NOT protect people like they do corporations. That is a hollow ideal, not reality.
I don't really see your point. My original comment was more pointing out that the operators, if not Amazon, could be seen as infringing their rights. They should update their legalese if they want to be truly protected. How is that defending Amazon?
Regardless of your pointed comment, I'm operating in the land of legal objectivity. The law doesn't care about your feelings much.
> Your argument is that IP law should be equally observed by everyone because it protects individuals and corporations alike
That is a weird understanding of what I said, and I don't really think you're arguing in good faith here. There's a lot of bias so I am choosing to not further this conversation.
Let them sue. I see the headline now, "Amazon sues status page website for accurately reporting on outages". Followed by lots of people hosting mirrors to stop.lying.cloud and saying things like "We are all stop.lying.cloud now"
I have a legally sound case to divorce my wife (anyone does, you can divorce for no reason), however she need not worry as that would be colossally stupid on my part. The same goes here.
Fair use is not a right. It’s a defense. When you are sued for copyright infringement, you have to argue that you’re doing it in fair use. It’s not the “get out of jail free” card people think it is.
That's warped view of the world. A corporation can always take you to court and harass the crap out of you, the court will side with your defense because you were right and claimant was wrong, you had the right to do what you did.
Satire is the loophole of Copyright. If you satire ANYTHING you can use their copyrights in the satire. One could safely and legally drive an entire nation's transportation industry thru that loophole.
No, fair use does NOT allow you to retain the original copyright. That would be passing off a derived work as the original copyright holder's work, which could be very damaging. This is a violation of fair use, if it could even be considered that to begin with.
No, it's not a trademark issue. They copied the work verbatim (including code, which is not covered by trademark law, but by copyright law), modified it, and then put the original copyright notice in the legalese. This is copyright infringement.
And consumers are clearly deceived - hence why my original comment asking about it was written and has several upvotes.
The direct copyright is covered by the satire exceptions. If you want to argue that copying the legalese is different, it's going to be on those confusion grounds, which IIUC aren't a copyright concern.
Fair use in the case of satire is not retaining the original copyright, it is referencing the copyright. It is a legal split hair, but it stands in court.
I think you misunderstand. The website in question has "Copyright (c) Amazon, Inc" in the bottom, when in fact the derived work (the site) is not created by Amazon, but by a third party. IANAL but my understanding is that this copyright notice being retained without any clarification of the owner of the derived work can be seen as endorsement, which is an infringement of copyright unless Amazon has expressly permitted such use (which is usually indicated as such, anyway).
It is also clearly not satire. That would not hold up in court, and there are many instances where they have tried that angle and failed.
Amazon has taken down trivial things before. This is a dangerous assumption to make - "it's not important to them so I shouldn't worry about the law". Those are famous last words.
Amazon's own status page sort of lies. So someone probably wget-ed the status page, kept the same html and css and hooked it to their own API to display correct info.
FWIW `lying.cloud` is registered with Namecheap. `amazon.com`/`aws.com`/`amazon.ca` are all registered with Mark Monitor. And I know that AWS uses ghandi behind the scenes for domain reg. Given that, I'd hazard a guess that it's not owned by Amazon. Definitely not a guarantee though.
Apropos of nothing, but a few Christmasses ago the place I worked had a dedicated fibre line that some workmen doing gas line repairs sawed straight through, took out everything; I was just drone worker at the time & it was a beautiful thing
Does your server at home handle similar traffic to that of US-East-1 since you're comparing uptime?
Simiarly, my laptop, if I keep it plugged in the wall, and enable httpd on localhost, will surely have better uptime than any of the top clouds. I'd bet that it'd have 100% uptime if I plugged in a UPS and cared for traffic on my local network only.
Mines had 100% uptime for the past 2 months. I’ve had great value for money using a NUC for personal projects than public cloud subscriptions over the past few years.
That depends on what you mean by AWS. I had production workloads in us-east-1 which haven’t been affected by any of these, and others which had only modest degradation. We had control plane issues but the running services were fine.
Put another way: even if your home ISP has had 100% uptime, are you comfortable saying that was true for all of their customers?
I'm not sure your reply makes sense, seems like a non-sequiter. Individual companies might require thousands of servers and not want then running all the time. This means they either maintain thousands of servers on premises, or they use aws and auto scale. This has nothing to do with aws having other customers, and everything to do with only wanting to pay for and maintain as little as possible while being able to serve your applications.
Most people don't need to handle the traffic of US-East-1. They just need a single, simple, mostly reliable server. But they're often told, "Don't do that. It's too hard, and irresponsible, and what if you get a spike in traffic, and what if you need to add 5 new servers, and security is really hard."
In reality, most people don't need to scale. An occasional spike in traffic is a nuisance, but not the end of the world, and security is not terribly hard, if you keep your servers patched (which is trivial to automate).
I really don't understand why there's so much FUD around running your own stuff.
I think most people on here are coming from the perspective of startups, which scale out of a single server setup pretty quickly. At a bare minimum, most will have dedicated purpose-built servers like Redis or a DB, and often there's separate background workers, or a load balancer with a couple of web servers.
When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first year of being launched), running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount.
> running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount.
What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required.
If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challenge other than maintaining X amount of hardware than just one.
Long term is unknown to the startup , they may fail or pivot .
Buying upfront hardware is not feasible even if I had the cash(which most don't), I don't know if the company would last that long or would be doing things that require x servers .
What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.
This is all assuming that a the startup has skill in setting up and managing physical servers and there is no opportunity costs( delayed features) on doing so, both are not a given.
small companies ( and poor people) typically don't buy low quality stuff or buy into rent seeking business models because they are dumb it is usually because they cannot afford to do long term thinking.
> Buying upfront hardware is not feasible even if I had the cash(which most don't)
You only need one server, slap on a hypervisor and your rocking. Heck you can buy entry servers from dell for budget; upgrade later.
A 10u rack, which is adequate for any small business comes to around $500 a month in LA. 4u would be more than enough for a startup and that's around $200/month.
Is the start-up not going to purchase computer hardware, monitors, television screens for their clients when they sit in the waiting room? Email accounts with Office365, a website, a domain name? If they can fit that in to their budget I am pretty sure they could afford a server and colocation space.
> What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.
But colocation is dynamic. Contracts can be negotiated.
> buy into rent seeking business models
And AWS isn't a rent seeking business model?
Looking at EC2 instances, for 120GBHD, 32 cores "Dedicated" instance, your looking at 679.54 USD for a month.
120GB isn't much especially when the developers start doing their thing.
For $500 you can have so much more, and hardware you actually own and that if the company does not lift off it can then be sold. Is that not the better investment?
You would need to hire a hardware/ops specialist who can buy, spin up, and maintain physical machines. That's a $100K+ job at any reasonable startup (not counting benefits) which is more than our Heroku spend (4-5 year startup). Keep in mind Heroku is way more expensive than just AWS.
This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for. All to solve... a couple days of service-specific downtime a year? People need to chill out with their non-mission critical software. God forbid someone can't access their HR portal for an hour.
> You would need to hire a hardware/ops specialist who can buy, spin up, and maintain physical machines.
No. You rack the server, connect the cable to the switch press the on-switch, let it boot and then operate as you would with any other computer. Need a new server? Get the DCOps remote hands to Rack the server, cable it to the switch and press the on-switch. If you can install Windows 10, you can install Linux.
> This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for.
Hmm, the data is yours; You own the data and that's value to me if I am the customer. And there's no value to the customer if you were to host it in the cloud either, much of a loss retrospectively.
There is no specialized skill you have to pay for, sure if you were going to run a high-density compute mainframe running some specialized OS like AIX then yeah sure. To buy a server, install a OS and plug it in to a switch and navigate it with SSH requires no overqualified anything.
Configuring a server for high traffic, networking, various safety measures against fire/power, backups, connectivity to your external DBs, etc. etc. As a 'feature dev' of almost a decade, I could bungle my way through this and probably leave a bunch of security gaps open. I'm highlighting the fact that it is a specialized skill and a specific part of the stack that feature devs like me and many other early startup engineers forego for other skills.
To your second point: We still own the data for hosted solutions. We have our backups, we have direct access. Barring a catastrophic failure and wiping of AWS, our data is there and ours. The value to our customers for using cloud providers is the time saved on building infrastructure is instead spent on delivering value to the customer in the form of features/bug fixes. And yes I know the argument of "you end up spending more time debugging AWS", but I think you don't need to reach that point if you keep things pretty simple, especially early stage.
I think you're vastly oversimplifying the task. And if I had to guess, it's something you're pretty familiar with so it makes sense that it's easy to you! I'm sure an early stage company would be happy to have you to save a ton of money in their early days on infrastructure costs
> Configuring a server for high traffic, networking, various safety measures against fire/power, backups, connectivity to your external DBs, etc. etc.
All which you have to do on a cloud provider. Fire/Power are normally handled by the DC. You have to have someone with knowledge in the first place to operate that in the cloud and taking an application such as HAProxy is on the same skill level. Especially with vast fields of blogs you can find on the topic.
The cloud brings the instant "power-up" methodology and I would compromise, you could be right. If you want a perfective optimum cloud platform you would need a dedicated "cloud" engineer if you want to ensure security, connectivity etc. etc. But if your going to do that then again you might as well move in-house and hire a system admin, you've still got to operate your companies infrastructure. It's a moot point.
> The value to our customers for using cloud providers is the time saved on building infrastructure is instead spent on delivering value to the customer in the form of features/bug fixes.
I suppose this is a mixed area and what customers value varies. For me I value a service that uses it's own hardware rather than cloud. On the basis that they are willing to put the skill in to operate their own infrastructure rather than.
> I think you're vastly oversimplifying the task.
I don't think so. People seem to think that setting up what you can in the cloud is impossible on bare metal when really it isn't. What did devOp's do before the cloud providers? Amazon,Azure,X have only lassoed FOSS software, constructed a webGUI-admin panel and throw it as a service.
This is not to say Cloud doesn't has a purpose, otherwise it wouldn't exist today.
> it's something you're pretty familiar with so it makes sense that it's easy to you
While true, I won't disagree, I've been working in the SysAdmin field since 2009. But disagree as I started with very little knowledge and gained it through setting up such infrastructures. I've educated a few and those with very little knowledge of servers could setup an infrastructure that companies run in AWS.
I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this.
3 162 days. That's 8 years+ of uptime.
Does it have the traffic of Amazon? No.
Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013).
It only has one port opened though, SSH. And with quite a hardened SSH setup at that.
I installed all the security patches I could install without rebooting it (so, yes, I know, this means I didn't install all the security patches for some required rebooting).
This server is, by now, a statement. It's about how stable Linux can be. It's about how amazingly stable Debian is. It's also about OVH: at times they had part of their datacenter burn (yup), at times they had full racks that had to be moved/disconnected. But somehow my server never got affected. It may have happened that at one point OVH had connectivity issues but my server went down.
I "gave back" many of my servers I didn't need anymore. But this one I keep just because...
I still use it, but only as an additional online/off-site backup where I send encrypted backups. It's not as if it gets zero use: I typically push backups to it daily.
They're only backups, they're encrypted. Even if my server is "owned" by some bad guys, the damage he could do is limited. Never seen anything suspicious on it though.
I like to do "silly" stuff like that. Like that one time I solve LCS35 by computing for about four years on commodity hardware at home.
I think it's about time I start to do some archeology on that server, to see what I can find. Apparently I installed Debian 7 on it in mid-october 2013.
I've created a temporary user account on it, which at times I've handle the password (before resetting it) to people just so they could SSH in and type: "uptime".
I read this as a cautionary tale. Here we have a server that only through the grace of god is still up, and is likely owned up. If it isn't, it's because of how little is going on with it.
At its current use, it's likely not a major issue but imagine if someone saw this uptime and thought to take it as a statement of reliability and built a service on it. I for one, would want that disclosed because this is a disaster waiting to happen. I'd much rather someone disclose that they had a few servers each with no longer than 7 days of uptime because they'd been fully imaged and cycled in that time...
It works both ways: it is also a cautionary tale for those who are prone to believe it's all unreliable cattle that needs constant restart because nothing is stable nor reliable...
5ish years ago it was common knowledge that us-east-1 is generally the worst place to put anything that needs to be reliable. I guess this is still true?
us-east-1 seems to be AWS’s not so well kept little dark secret!
In all seriousness though - even non-regional AWS services seem to have ties to us-east-1 as evidenced by the recent outages. So you might be impacted even if it looks like (on paper at least) you’re not using any services tied to that region.
I don't know about that. It was more like common knowledge that one availability zone in us-east-1 was a problem - you would have to figure out which one it was usually by spinning up instances in all 4 zones (now 6)... and that it was the largest of all regions making it ideal place to put your service if you wanted to be close to other vendors/partners in AWS...
So why are people not migrating out of us-east-1? Operating in ap-southeast, we weren't that affected by the us-east-1 down time, although our system is reasonably static and doesn't make lots of IAM calls (which seems to be a large SPOF from us-east-1).
latency. us-east-1 is positioned very nicely relative to many large businesses in North America and Europe. This gives you pretty good access to a very large percentage of the economies of the world with good latency... while not requiring you to architect your application around multiple regions...
Some “global” systems run in us-east1 even if you’re not hosted there a service you depend on might be.
Notably: cognito, r53 and the default web UI. (You can work around the webui one I’m told, by passing a different domain instead of just console.aws.amazon.com)
645 comments
[ 5.0 ms ] story [ 348 ms ] threadThe console is throwing errors from time to time. As usual no information on AWS status page.
The letters are randomised per AWS account so that instances are spread evenly and biases to certain letters don't lead to biases to certain zones.
The thing is, us-east-1 represents the whole AWS for the majority of us.
Your question reads as a strawman. It matters nothing if EC2 is also available in Mumbai or Hong Kong if by default the whole world deploys everything and anything to us-east-1, and us-east-1 alone.
https://www.reddit.com/r/aws/comments/nztxa5/why_useast1_reg...
Deploying a service to a single region is not, nor has it ever been, "customers don't know how to use AWS".
If anything, cargo culting this belief in global deployments being necessary, specially with services that have at most a regional demand, is a telltale sign a customer has no idea about what he is doing and is just mindlessly wasting money and engineering effort in something no one needs.
This blend of bad cargo cult advice sounds like a variant of microservices everywhere.
https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#...:
(Edit: I hope I didn't sound sarcastic. I don't open random console pages and scroll all the way down to check for new features. Some people will have noticed, some won't.)
I'm having issues with Slack from central EU (Poland) -- can't upload images, or send emoji reactions to post; curiously, text works fine). Wondering if linked
I’m not affiliated with them, and haven’t even really used them other than to explore a bit. They come highly recommended by my acquaintances, though.
Edit: Restarting Slack does update the edited messages.
Edit 15:24 CET: Slack is back up.
- edits failing or working with big lag;
- "Threads" view slow;
- can't emoji-react;
- can't upload images;
- people also say they can't join new channels.
> We are experiencing issues with file uploads, message editing, and other services. We're currently investigating the issue and will provide a status update once we have more information.
> Dec 22, 1:58 PM GMT+1
edit: The question isn't necessarily AWS specific, just any data on amount of downtime per cloud provider on a timeline would be nice.
If you can’t even admit you’re having an issue how can you keep an accurate record?
This is what happens when we've been affected by outages (even without involving support).
There indeed has been an uptick in AWS outages recently. You can see a bit of the history here: https://statusgator.com/services/amazon-web-services
https://en.wikipedia.org/wiki/Potemkin_village
Like how come down detector can do a superb job of detecting when AWS goes down and AWS can't? Because AWS doesn't want account managers of SLAs asking for credits for the uptime they're paying for but not getting.
https://downdetector.co.uk/status/aws-amazon-web-services/
Packaging a way to migrate off AWS could be a unicorn idea.
At least, that's what I understood.
Another side benefit of being with AWS is when you do have an outage, a lot of other people have outages, and so you sort of blend in with the noise. It's not great to be down, but if you're down and also "big service X" who's also an AWS customer is down, it makes your downtime look less like a lack of competence and more like an unavoidable force of nature.
I worked at a company that's bread and butter was online services (e-commerce SaaS platform, similar to Netsuite) and we had significantly fewer outages than AWS had.
But we had redundancies built in to most things, I'm not saying it was perfect but it worked.
The major difference might be that almost nobody is willing to spend 20% of what they spend on AWS/GCP to have a self-hosted solution.
The reason "cloud is so expensive" is because they're essentially telling you what the price will be and even if they only spend 40% of that on actual hardware and operations: it's more than most companies would invest in themselves.
This is absurd, of course, but it's absolutely true.
When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it.
Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region
We didn't take any downtime, but if anything had gone wrong there would have been nothing we could do about it until IAM came back up.
The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.
Logging in with root credentials was not possible in any region, and even logging in with IAM creds in other regions yielded an intermittently buggy console
and as is usual with us-east-1 outages management API calls were a complete crap shoot regardless of region
"Let's move our internal app with 50 users to k8s in the cloud." --true story
It's a real shame that the collective world of technology does not properly respect the simple solutions that work.
It is almost funny the dichotomy here. Most technological people "admire" the simplicity, elegance and extensibility of the command line. But tell those same people that the best data store for the solution is a relational database and their nose crinkles up.
Every dependency scrutinized and discarded if possible.
I would probably work for free if someone setup their own on-prem cloud in Tanzu, Open shift, or Ranger and used old school proven frameworks for development.
Working in AWS has been a real shitty experience at these large companies. All the nit picky problems (of which there are thousands) get dumped on devs who are trying to deliver working software.
Agreed. Arguably, not using an existing cloud service is a red flag on any new hires. AWS being the primary, but experience using GCS or Azure are at least viable skills, even if your business is AWS-based.
But the "fad-based-development" meme is not going away any time soon. The incentives in the business are built around it (really! No one want's to work on a boring old relational database solution any more). In the old days it was 4th generation languages, RUP, XML and Function Point Analysis... today it's functional programming, SDKs, big-three cloud PaaS experience or (shudder) block-chain.
I think back to my much younger self, when I thought that technology was something to be mastered to solve real-world problems, and I laugh. Little did I know the real problem to be solved was to figure out how to solve those same-old business problems but with the technology of the season (Kubernetes, GraphQL or ML).
Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.
Uptime improved rather dramatically after that.
Meanwhile, I currently have a gig to work on a video service which features a never updated centos 6, an unsupported python 2 blob website, and a push to prod deployment procedure, running a single postgres db serving streaming for 4 millions users a month.
And it's got years of up time, cost 1/100th of AWS, and can be maintained by one dev.
Not saying "cloud is bad", but we got to stop screaming old techs are no good either.
2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary fresh data.
3 - if you compile nginx manually, you get to use a lot of plugins that can do stuff super fast, including serving videos. You can script stuff in lua that will just skip the backend completly.
4 - mind your encoding. We carefully chose how we encode videos. The ffmpeg parameters are pretty insane, but the space / quality ratio is amazing, espacially on mobile. It takes a lot of time to experiment with those, nobody share them :)
5 - we offload everything we can to cron tasks or task queues. Including, obviously, encoding, screenshooting, etc.
6 - don't hold data you can't lose. E.G: billing. This way you can have a relaxed attitute toward data. If we ever loose a day of business, users will be in a bad mood for a week, but that won't be the end of the world. We don't need a bullet proof system if bullets can't kill us.
7 - give money to ffmpeg and opencv, because damn those things are fast. And good.
8 - servers are hosted accross 2 providers. This way, if one goes down, or decide to stop doing business with us Google style, we have a second one. Happened recently with leaseweb: they shutdown a whole room without offering an alternative.
E.G: votes.
They don't hit the backend on write. We pile them from nginx to redis, then once a day, we aggregate and store on postgres, which the backends will consumme. We just store each vote on localstorage as well so that the user feels like it's real time when they vote, but in reality it's updated once a day. But votes don't affect the money side of our business, so if we lose them one day, it does not mean death.
P.S: yes, posgres/redis/elasticsearch only hold metadata. Videos are stored on disk. There is no docker images, no mircoservices, FS is ext4. Which means with a lot of RAM, the OS FS cache will have most popular videos already loaded and ready to be streamed. Everything is raid 0, so if we get one disk corrupted, you lose the server. But we upload each videos on severeal servers, so when a disk get corrupted, we just replace the whole server. In fact, anything goes wrong on a server, we replace it. It's not worth it to find the root cause, unless 2 servers die in the same way successively.
Regarding the ffmpeg parameters and formats in general: Do you use newer formats too, like AV1 and the like?
via https://stop.lying.cloud/
I mean I'm glad it exists, don't get me wrong. Just weird that they'd have two status pages, one seemingly existing only to sort of 'mock' themselves...
It's just strange.
(Using copyrighted material is permitted under fair use; this website is a parody. I’m not a lawyer but at some level preserving the copyright notice is probably better than claiming it as their own.)
You may say that the original work is copyright of the respective owners and that this is a parody work. But that's not what the site is doing. The footer contains the original, unaltered copyright, creating confusion as to who owns the derived work. Amazon does not own this, nor do they endorse it, so you're not allowed to say it's copyrighted by Amazon.
Being concerned for the proper respect of IP laws is something that benefits everyone.
Your argument would have a small amount of merit if you acknowledged that the laws DO NOT protect people like they do corporations. That is a hollow ideal, not reality.
Regardless of your pointed comment, I'm operating in the land of legal objectivity. The law doesn't care about your feelings much.
> Nope. Because these laws also protect people who make such websites from the corporations they're commenting on, too.
My response is that your assumption is very obviously wrong: the law does not protect individuals and corporations alike.
That’s all.
That is a weird understanding of what I said, and I don't really think you're arguing in good faith here. There's a lot of bias so I am choosing to not further this conversation.
That's warped view of the world. A corporation can always take you to court and harass the crap out of you, the court will side with your defense because you were right and claimant was wrong, you had the right to do what you did.
And consumers are clearly deceived - hence why my original comment asking about it was written and has several upvotes.
It is also clearly not satire. That would not hold up in court, and there are many instances where they have tried that angle and failed.
Amazon has much bigger legal issues to focus on than some satire.
Speeding? Basically a national past-time at this point.
Misrepresentation, common fraud, and misappropriation? Par for the course in most small businesses.
It's only a crime if someone gives enough of a shit to do something about it; otherwise, it's just life.
Apparently it does some simple transformations of the actual status page, which is why the Amazon copyright stuff is in there.
He also has a decent newsletter and witty commentary, for all things AWS.
[0] https://twitter.com/quinnypig/status/1468331194471178241?s=2...
Simiarly, my laptop, if I keep it plugged in the wall, and enable httpd on localhost, will surely have better uptime than any of the top clouds. I'd bet that it'd have 100% uptime if I plugged in a UPS and cared for traffic on my local network only.
Put another way: even if your home ISP has had 100% uptime, are you comfortable saying that was true for all of their customers?
Better uptime than paying for EC2 on AWS US-East-1.
Obviously this approach isn't scalable but it serves me well.
It's perfectly scalable. Just give everybody their own home server.
Of course it doesn't. Why are you asking antagonistic questions?
I don't think people care that AWS has other customers, they want their workload to work, if it doesn't: then that's a today issue.
Is this not antagonistic? It's pointless to make these statements, so your parent comment pointed it out. Go downvote the first one instead.
In reality, most people don't need to scale. An occasional spike in traffic is a nuisance, but not the end of the world, and security is not terribly hard, if you keep your servers patched (which is trivial to automate).
I really don't understand why there's so much FUD around running your own stuff.
When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first year of being launched), running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount.
What challenges are you thinking of? You buy a full-rack in colocation and then just buy servers/hardware when required.
If a company has the budget for AWS or some other cloud provider then they would have a budget for colocation; which in long term is cheaper. I see no additional challenge other than maintaining X amount of hardware than just one.
Buying upfront hardware is not feasible even if I had the cash(which most don't), I don't know if the company would last that long or would be doing things that require x servers .
What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.
This is all assuming that a the startup has skill in setting up and managing physical servers and there is no opportunity costs( delayed features) on doing so, both are not a given.
small companies ( and poor people) typically don't buy low quality stuff or buy into rent seeking business models because they are dumb it is usually because they cannot afford to do long term thinking.
You only need one server, slap on a hypervisor and your rocking. Heck you can buy entry servers from dell for budget; upgrade later.
A 10u rack, which is adequate for any small business comes to around $500 a month in LA. 4u would be more than enough for a startup and that's around $200/month.
Is the start-up not going to purchase computer hardware, monitors, television screens for their clients when they sit in the waiting room? Email accounts with Office365, a website, a domain name? If they can fit that in to their budget I am pretty sure they could afford a server and colocation space.
> What you are saying is similar to saying may be it is cheaper to buy the building /floor instead of renting space for office. - most small biz cannot afford do that, or expect their business to change (fail/take off) in the time frame ROI would come to take that commitment.
But colocation is dynamic. Contracts can be negotiated.
> buy into rent seeking business models
And AWS isn't a rent seeking business model?
Looking at EC2 instances, for 120GBHD, 32 cores "Dedicated" instance, your looking at 679.54 USD for a month. 120GB isn't much especially when the developers start doing their thing.
For $500 you can have so much more, and hardware you actually own and that if the company does not lift off it can then be sold. Is that not the better investment?
No remarks to lack of skills.
This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for. All to solve... a couple days of service-specific downtime a year? People need to chill out with their non-mission critical software. God forbid someone can't access their HR portal for an hour.
No. You rack the server, connect the cable to the switch press the on-switch, let it boot and then operate as you would with any other computer. Need a new server? Get the DCOps remote hands to Rack the server, cable it to the switch and press the on-switch. If you can install Windows 10, you can install Linux.
> This comment just screams of engineer-only focus. Running servers yourself brings almost no value to the customer and is a specialized skill that you have to pay for.
Hmm, the data is yours; You own the data and that's value to me if I am the customer. And there's no value to the customer if you were to host it in the cloud either, much of a loss retrospectively.
There is no specialized skill you have to pay for, sure if you were going to run a high-density compute mainframe running some specialized OS like AIX then yeah sure. To buy a server, install a OS and plug it in to a switch and navigate it with SSH requires no overqualified anything.
To your second point: We still own the data for hosted solutions. We have our backups, we have direct access. Barring a catastrophic failure and wiping of AWS, our data is there and ours. The value to our customers for using cloud providers is the time saved on building infrastructure is instead spent on delivering value to the customer in the form of features/bug fixes. And yes I know the argument of "you end up spending more time debugging AWS", but I think you don't need to reach that point if you keep things pretty simple, especially early stage.
I think you're vastly oversimplifying the task. And if I had to guess, it's something you're pretty familiar with so it makes sense that it's easy to you! I'm sure an early stage company would be happy to have you to save a ton of money in their early days on infrastructure costs
All which you have to do on a cloud provider. Fire/Power are normally handled by the DC. You have to have someone with knowledge in the first place to operate that in the cloud and taking an application such as HAProxy is on the same skill level. Especially with vast fields of blogs you can find on the topic.
The cloud brings the instant "power-up" methodology and I would compromise, you could be right. If you want a perfective optimum cloud platform you would need a dedicated "cloud" engineer if you want to ensure security, connectivity etc. etc. But if your going to do that then again you might as well move in-house and hire a system admin, you've still got to operate your companies infrastructure. It's a moot point.
> The value to our customers for using cloud providers is the time saved on building infrastructure is instead spent on delivering value to the customer in the form of features/bug fixes.
I suppose this is a mixed area and what customers value varies. For me I value a service that uses it's own hardware rather than cloud. On the basis that they are willing to put the skill in to operate their own infrastructure rather than.
> I think you're vastly oversimplifying the task.
I don't think so. People seem to think that setting up what you can in the cloud is impossible on bare metal when really it isn't. What did devOp's do before the cloud providers? Amazon,Azure,X have only lassoed FOSS software, constructed a webGUI-admin panel and throw it as a service.
This is not to say Cloud doesn't has a purpose, otherwise it wouldn't exist today.
> it's something you're pretty familiar with so it makes sense that it's easy to you
While true, I won't disagree, I've been working in the SysAdmin field since 2009. But disagree as I started with very little knowledge and gained it through setting up such infrastructures. I've educated a few and those with very little knowledge of servers could setup an infrastructure that companies run in AWS.
I have a server at OVH (not affiliated to them) which, at this point, I keep only for fun. It has 3162 days of uptime as I type this.
3 162 days. That's 8 years+ of uptime.
Does it have the traffic of Amazon? No.
Is it secure? Very likely not: it's running an old Debian version (Debian 7, which came out in, well, 2013).
It only has one port opened though, SSH. And with quite a hardened SSH setup at that.
I installed all the security patches I could install without rebooting it (so, yes, I know, this means I didn't install all the security patches for some required rebooting).
This server is, by now, a statement. It's about how stable Linux can be. It's about how amazingly stable Debian is. It's also about OVH: at times they had part of their datacenter burn (yup), at times they had full racks that had to be moved/disconnected. But somehow my server never got affected. It may have happened that at one point OVH had connectivity issues but my server went down.
I "gave back" many of my servers I didn't need anymore. But this one I keep just because...
I still use it, but only as an additional online/off-site backup where I send encrypted backups. It's not as if it gets zero use: I typically push backups to it daily.
They're only backups, they're encrypted. Even if my server is "owned" by some bad guys, the damage he could do is limited. Never seen anything suspicious on it though.
I like to do "silly" stuff like that. Like that one time I solve LCS35 by computing for about four years on commodity hardware at home.
I think it's about time I start to do some archeology on that server, to see what I can find. Apparently I installed Debian 7 on it in mid-october 2013.
I've created a temporary user account on it, which at times I've handle the password (before resetting it) to people just so they could SSH in and type: "uptime".
It is a thing of beauty.
Eight. Years. Of. Uptime.
Awesome! Are you Bernard Fabrot [0]?
[0] https://www.csail.mit.edu/news/programmers-solve-mits-20-yea...
At its current use, it's likely not a major issue but imagine if someone saw this uptime and thought to take it as a statement of reliability and built a service on it. I for one, would want that disclosed because this is a disaster waiting to happen. I'd much rather someone disclose that they had a few servers each with no longer than 7 days of uptime because they'd been fully imaged and cycled in that time...
In all seriousness though - even non-regional AWS services seem to have ties to us-east-1 as evidenced by the recent outages. So you might be impacted even if it looks like (on paper at least) you’re not using any services tied to that region.
Notably: cognito, r53 and the default web UI. (You can work around the webui one I’m told, by passing a different domain instead of just console.aws.amazon.com)