I tried to use ARM instances on AWS, and I discovered a lot of things just don't work. For example, launching an EKS cluster with ARM nodes requires a bunch of hacks, and you can't really do anything useful with it yet. The instances are also not properly supported by Terraform.
I'm excited for ARM because of the pressure it will put on Intel, and also because of the cost savings. However it seems like AWS has a lot of work to do to make it useful.
It won't catch on... Not being able to buy one of these systems also has the issue of platform developers coming up short in terms of adapting tooling for the above list.
A lot more of those things work than you might expect. I was able to get a very large node app working on their ARM instances very quickly... Docker support is also there from as far I know.
What's required for all these to be supported is just developers being able to buy 64-bit ARMv8 processors running a standard 64-bit Linux kernel, not necessarily the exact same processor Amazon has. And the Raspberry Pi 3B is an inexpensive 64-bit ARMv8 processor which can run a standard 64-bit Linux kernel. Sure, it's ARMv8.0 instead of ARMv8.2, but as mentioned in the article software compiled for ARMv8.0 works fine on ARMv8.2.
Okay, I wasn't aware of that... all the same, would be nice to be able to actually buy one to work against. The list of supported platforms and being able to develop (near-locally) are what are really needed.
Haven't really looked into it... it's on my todo list to look at cross-platform builds to target Raspberry Pi devices, will probably setup containerized cross platform builds for at least node, rust and .net core for my own use.
Shame EPYC2 wasn't included, but extremeley impressive numbers.
Also I haven't used Anandtech in a while, but it hasn't changed much at all after their redesign a while back. I'm a fan of how it hasn't changed much. Nice and simple, though a bit more social media stuff that I remember.
Yes, they just published this article that compares two cloud server platforms based on how many times per hour they can run SPEC, which is totally on-brand for them and either a new low in the art of server benchmarking or a wry attempt at self-deprecation.
It's unfortunate to see CPU+motherboard platforms that are entirely unavailable to purchase and own yourself.
There is a real need for people who want to do development/test with on-premises servers, and then migrate the end product to an AWS or similar environment. At least with AMD64 you can reliably do this.
We at Amazon want you to test on Amazon's servers! After all, Jeff Bezos owns every server on the planet.
Pay it no mind that developing software and testing it on AWS is an order of magnitude more expensive than doing things on-premises: don't worry, you're not being ripped off any more than usual! All of those big "unicorn" tech companies you look up to are doing it—everyone from BP to GE to the people who rip you off on concert tickets to the guys that made Fortnite—it's all the rage, nowadays!
That $20,000,000 in venture capital you got could surely be used to set up a dev environment for almost two whole months! Call now and we'll even throw in three whole extra days (throttled by 50%)!﹡
﹡Restrictions may apply. Not valid in the state of California or to companies registered in Delaware. We reserve the right to change the terms of the agreement without prior notice.
> that developing software and testing it on AWS is an order of magnitude more expensive than doing things on-premises
Is there any truth to this? Every time I see this claim, it is actually comparing the price tag for servers in the on-prem case with the price tag for compute on AWS/etc (which includes a whole lot of engineering and data center management costs). It's a pretty meaningless comparison.
My guess is that AWS has the economies of scale to outperform your average private data center on a total-cost-of-ownership basis.
One year of a a1.2xlarge 8vcpu 16gb RAM costs $1782.144 excluding storage. I could build a pretty decent server with that money that would last easily half a decade if not longer.
If all you want is a development machine and electricity is not particularly expensive, you can buy a used 32-core, quad socket Xeon with 128GB of RAM for half that. Dell r910 or similar. It's about a 500W load.
Or you could get an eight core / 32GB ram machine for under $230 on eBay.
Let's be fair, you are actually paying extra for elastic capacity. I.e You expect to have a 2nd Server Ready when ever you need. Not to mention the Cloud is suppose to add a little redundancy, so in case your server fail you could switch on another in an instant.
So in reality I will always times the cost of server ( over X months period ) + colocation / month by 2.5. And Double that to give them Margin for support etc.
That is roughly 5x the cost. And in most cases seems to be what AWS is charging for.
AWS uses the industry standard financial model for leased infrastructure with EC2 instances: all offered pricing structures recover hardware costs at 6 months. After 6 months, your cost basis is OpEx only. The margins on month 7 and beyond are very good.
If you operate at sufficient scales to amortize the fixed overhead of operations (which is not that large of a scale), you can break even running your own server infrastructure within 12 months versus AWS if you know what you are doing. That last part is a big caveat, but not a large hurdle.
There are other considerations but in pure hardware terms, on AWS you've bought the hardware after 6 months of utilization.
Let's do the math. An a1.2xlarge reserved instance with the longest available term (3 years) is $672/year, or $3,360 over 5 years.
A bit of browsing on Geekbench suggests that one a1.4xlarge instance (i.e. two a1.2xlarges) is roughly comparable to a Ryzen 3400G. Adding in rough costs for the other components (motherboard, power supply, case, and 32GB of RAM), a bare-metal system equivalent to an a1.4xlarge looks like it would cost maybe $500-600 to buy outright.
In other words, if all you care about is raw CPU performance, then over long periods of time AWS costs more than 10x as much as bare-metal per unit of performance, even with up-front reserved instances.
Except we are talking about a2’s (m6g) brother, the whole point is that the math changes significantly with these new instances. The a1 was not competitive.
How much would you charge to buy this machine, rent a space to put it in, pay for the utilities and rent related to it, set it up, and consequently maintain it for half a decade? In addition to the principal of purchasing the hardware?
In the real world: UPS,generators, cooling, BIOS/management card patching, 100 gigabit switches, optics, cabling, monitoring, tamper-proof logging, SEIM, cameras, physical security, software licenses. Plus maintenance on all that stuff.
Oh and you need a DR site too, so double it.
And staff up 4x for 24x7 operations and build an orchestration layer yourself.
I’ve done the math in a lot of scenarios, including (especially) staff time. Running your own real data center doesn’t make long-term financial sense unless you are in the mid-hundreds of physical servers.
Colo might make a bit more sense for some situations, but there’s a reason so much is being moved to cloud providers. The people doing it aren’t stupid.
I recently provisioned a "micro-" instance...the smallest possible server available, as I fell hook, line, and sinker for the "FREE 12 months!" promotion that turned out to be totally false.
For 3 months, I was charged $11,$23, and finally $4 for 3 days of the 3rd months, where I QUICKLY cancelled the server...this is OUTRAGEOUSLY expensive for a single processor, 1G RAM, 30GB SSD cloud server!
$37 for some 64 days is totally overpriced. You can get high quality low cost cloud servers for less then $5/month, and if you purchase a full year, half that.
Amazon really disappointed me with their FREE bait-n-switch bullshit marketing, and I will NEVER recommend their low end servers again.
Having definitely gotten out for $0 on the free tier, it sounds like you somehow ended up with other services or on an instance type that was not free tier eligible.
Which isn't an accident, to be fair. AWS pricing is deceptive on purpose, with many cross-dependencies that each have their own price points, exclusions, and caveats.
Only this many cents to use this thing, which, heh, I guess it requires this thing too, and this thing, and this thing ... man, some of these cents are adding up. Oh, a "NAT Gateway" is $30/mo? Well, OK. Keep going for a few weeks and you're staring down a bill for a rather scary number of cents while the CFO's angry rants can be heard half-way across the office.
There are entire consultancies that specialize in decoding AWS bills. Let's not pull leg here.
Imagine a SaaS platform that came with tooling to let you locally emulate the characteristics of different options on their infrastructure.
We don't necessarily care about matching absolute production performance, but could throttle things to get the right ratio between performance of CPU, memory bandwidth, disk IO, etc, in order to explore which code tweaks will have the biggest relative impact on your workload. It would let you do a bunch of your testing and optimization locally, for free.
Could be an edge to distinguish you from the competition, although I'm not sure it's a big enough one to warrant a business case. Maybe it would be attractive to shops whose development costs are high compared to production (i.e. startups and enterprises with lots of ongoing customization), but probably not so much for the established, internet-based businesses where I'm guessing the big money is (like Netflix).
Maybe not to the extent you want, but Azure Stack lets you run many types of cloud infrastructure on site, including production workloads, for a small fee.
Whats wrong with the guys that made Fortnite?
I think their business model is genius.
Make a free game, capable of cross-platform multiplayer across nearly all devices, and let people waste their money on skins and dances that give no competitive edge if they want to.
From the interviews on it, along with the podcast, it looks like they're trying to draw people away from Other People's Computers as a Service, not serve as a way to onboard them into it.
Getting .001% of people away from it is already a market that makes you significantly less dead than Sun is right now. I don't think anyone's that worried!
Ever since cloud compute became an industry meme, I started telling people that the end-game was to get everything into someone else's mainframe and then apply various forms of lock-in that would include proprietary CPU extensions, proprietary services, and so on.
Personal experience, if you hire a sufficient number of BSD and GPL zealots you can accomplish a lot with minimal lock in. You need people who are ideologically committed to not being wholly dependent on AWS and similar.
If i'm remembering the previous ARM announcements correctly, you probably already have an ARM instance somewhere in your stack - they were going to be transitioning a lot of stuff like ELBs over to these processors.
Sensible; if the price/perf ratio is good then AWS would be kinda dumb not to move all their internal stuff to use it. And they control the whole stack and don't expose internals to users for their own services (ELBs, Route 53, probably RDS, etc), so it should be relatively straightforward from an engineering perspective.
Yes they intend to transit all of their Services, as well as SaaS on top of their ARM instances. Although they said this is going to take up to 5 years or more ( usual downplay from Amazon as with everything they do ), I bet they have a much more aggressive schedule.
For those of you considering alternate instance types remember that it it is important to do scientific benchmarks.
I just completed a us-west to us-east migration for a client because after a lengthy benchmark session we learned that the instances we really needed for our specific workload could not be AMD based. I tried a handful of AMD boxes and sent them through the ringer, but the Intel boxes always came out on top. The cost of these is significantly lower in us-east, hence the migration.
Not to knock AMD, but in the presented configuration from AWS the comparative Intel<>AMD instance types vary greatly. There is also no compute-heavy instance type from AMD at this time (AFAIK please correct me).
For reference our workload is network (outside AWS), Compute and RAM heavy. The c5.2xlarge seems to be the sweet spot as far as cost/performance.
So if you’re considering Graviton, make sure to design and run some real benchmarks. For us (in the AMD scenario) things looked real good on paper but after benchmarking the performance was not there.
The compute-focused AMD instances (c5a and c5ad) based on 2nd gen Epyc were announced as launching "soon" back in November, haven't heard anything about them since.
Which is the question I keep throwing but without an answer. The EPYC 2 was previewed in early 2019 and I would not be surprised if All three major cloud provider has had samples since. And it has been 12 months without much action. AMD's projection of their Server Market growth aren't exactly encouraging either.
us-west-2 ec2 pricing is on par with us-east-1 and us-east-2, were you originally in us-west-1 (everything is considerably more expensive in that region)?
I want to buy one... I doubt Amazon will ever sell Graviton chops directly to consumers, but the article says the chip is essentially an underclocked reference design using ARM N1 cores, so other vendors could offer essentially identical chips.
I've had a disdain for SPEC benchmarks for a long time. It was always what workstation makers or Apple (PowerPC Apple) used to show how much faster their chips were than Intel-even though it never really panned out.
Would a more AWS-oriented benchmark suite make more sense? Test Java, run PHP/Ruby/Python/Node, see how many pages you can serve with Apache/NGINX, run queries in MariaDB/Postgres, see how much you can stick through Mongo/Redis...
Intel (and so x86) has had a manufacturing node advantage over most other processor manufacturers for a long time. They do not have that advantage currently. So perhaps it was only a matter of time before another instruction set became competitive.
68 comments
[ 3.0 ms ] story [ 140 ms ] threadI'm excited for ARM because of the pressure it will put on Intel, and also because of the cost savings. However it seems like AWS has a lot of work to do to make it useful.
In AWS there is one extra step to deploy it than an Intel cluster. And that's only because it is in preview.
Haven't really looked into it... it's on my todo list to look at cross-platform builds to target Raspberry Pi devices, will probably setup containerized cross platform builds for at least node, rust and .net core for my own use.
Also I haven't used Anandtech in a while, but it hasn't changed much at all after their redesign a while back. I'm a fan of how it hasn't changed much. Nice and simple, though a bit more social media stuff that I remember.
There is a real need for people who want to do development/test with on-premises servers, and then migrate the end product to an AWS or similar environment. At least with AMD64 you can reliably do this.
Pay it no mind that developing software and testing it on AWS is an order of magnitude more expensive than doing things on-premises: don't worry, you're not being ripped off any more than usual! All of those big "unicorn" tech companies you look up to are doing it—everyone from BP to GE to the people who rip you off on concert tickets to the guys that made Fortnite—it's all the rage, nowadays!
That $20,000,000 in venture capital you got could surely be used to set up a dev environment for almost two whole months! Call now and we'll even throw in three whole extra days (throttled by 50%)!﹡
﹡Restrictions may apply. Not valid in the state of California or to companies registered in Delaware. We reserve the right to change the terms of the agreement without prior notice.
Is there any truth to this? Every time I see this claim, it is actually comparing the price tag for servers in the on-prem case with the price tag for compute on AWS/etc (which includes a whole lot of engineering and data center management costs). It's a pretty meaningless comparison.
My guess is that AWS has the economies of scale to outperform your average private data center on a total-cost-of-ownership basis.
I could double the cores and mem probably too.
Or you could get an eight core / 32GB ram machine for under $230 on eBay.
So in reality I will always times the cost of server ( over X months period ) + colocation / month by 2.5. And Double that to give them Margin for support etc.
That is roughly 5x the cost. And in most cases seems to be what AWS is charging for.
Not saying I agree with their pricing.
If you operate at sufficient scales to amortize the fixed overhead of operations (which is not that large of a scale), you can break even running your own server infrastructure within 12 months versus AWS if you know what you are doing. That last part is a big caveat, but not a large hurdle.
There are other considerations but in pure hardware terms, on AWS you've bought the hardware after 6 months of utilization.
My guess is the advantage evaporates.
A bit of browsing on Geekbench suggests that one a1.4xlarge instance (i.e. two a1.2xlarges) is roughly comparable to a Ryzen 3400G. Adding in rough costs for the other components (motherboard, power supply, case, and 32GB of RAM), a bare-metal system equivalent to an a1.4xlarge looks like it would cost maybe $500-600 to buy outright.
In other words, if all you care about is raw CPU performance, then over long periods of time AWS costs more than 10x as much as bare-metal per unit of performance, even with up-front reserved instances.
Oh and you need a DR site too, so double it.
And staff up 4x for 24x7 operations and build an orchestration layer yourself.
I’ve done the math in a lot of scenarios, including (especially) staff time. Running your own real data center doesn’t make long-term financial sense unless you are in the mid-hundreds of physical servers.
Colo might make a bit more sense for some situations, but there’s a reason so much is being moved to cloud providers. The people doing it aren’t stupid.
For 3 months, I was charged $11,$23, and finally $4 for 3 days of the 3rd months, where I QUICKLY cancelled the server...this is OUTRAGEOUSLY expensive for a single processor, 1G RAM, 30GB SSD cloud server!
$37 for some 64 days is totally overpriced. You can get high quality low cost cloud servers for less then $5/month, and if you purchase a full year, half that.
Amazon really disappointed me with their FREE bait-n-switch bullshit marketing, and I will NEVER recommend their low end servers again.
Only this many cents to use this thing, which, heh, I guess it requires this thing too, and this thing, and this thing ... man, some of these cents are adding up. Oh, a "NAT Gateway" is $30/mo? Well, OK. Keep going for a few weeks and you're staring down a bill for a rather scary number of cents while the CFO's angry rants can be heard half-way across the office.
There are entire consultancies that specialize in decoding AWS bills. Let's not pull leg here.
We don't necessarily care about matching absolute production performance, but could throttle things to get the right ratio between performance of CPU, memory bandwidth, disk IO, etc, in order to explore which code tweaks will have the biggest relative impact on your workload. It would let you do a bunch of your testing and optimization locally, for free.
Could be an edge to distinguish you from the competition, although I'm not sure it's a big enough one to warrant a business case. Maybe it would be attractive to shops whose development costs are high compared to production (i.e. startups and enterprises with lots of ongoing customization), but probably not so much for the established, internet-based businesses where I'm guessing the big money is (like Netflix).
https://azure.microsoft.com/en-us/overview/azure-stack/
Make a free game, capable of cross-platform multiplayer across nearly all devices, and let people waste their money on skins and dances that give no competitive edge if they want to.
Not to mention the environmental impact of doing so.
First the software, then the hardware: This is the ultimate lock-in.
I just completed a us-west to us-east migration for a client because after a lengthy benchmark session we learned that the instances we really needed for our specific workload could not be AMD based. I tried a handful of AMD boxes and sent them through the ringer, but the Intel boxes always came out on top. The cost of these is significantly lower in us-east, hence the migration.
Not to knock AMD, but in the presented configuration from AWS the comparative Intel<>AMD instance types vary greatly. There is also no compute-heavy instance type from AMD at this time (AFAIK please correct me).
For reference our workload is network (outside AWS), Compute and RAM heavy. The c5.2xlarge seems to be the sweet spot as far as cost/performance.
So if you’re considering Graviton, make sure to design and run some real benchmarks. For us (in the AMD scenario) things looked real good on paper but after benchmarking the performance was not there.
us-west-2 ec2 pricing is on par with us-east-1 and us-east-2, were you originally in us-west-1 (everything is considerably more expensive in that region)?
Quite a headline
Would a more AWS-oriented benchmark suite make more sense? Test Java, run PHP/Ruby/Python/Node, see how many pages you can serve with Apache/NGINX, run queries in MariaDB/Postgres, see how much you can stick through Mongo/Redis...
> it means that they also implement the v8.1 ISA which add new instructions such as atomic compare-and-set (CAS).
> you should pay very close attention to make sure your software stack is compiled against Arm v8.1 or higher (N1 is v8.2).
But my understanding is that the new atomics are in an extension, so just compiling to v8.1 may not get the new atomics.
Anyone know what the situation here is? What has to happen to get the compiler to use LSE atomics?