A blog post is coming, so I might have jumped the shark here. An interesting note from a Cloudflare engineer:
Most powerful Xeon is the 28 core Platinum 2180 at $10k RSP and >200W TDP. Due to the nature of Intel turbo boost it almost always operates at TDP under load. Our ARM is the Centriq 2452 at less than $1400 RSP, 46 cores and 120W TDP, that it never hits. Beats Xeon 9/10 workloads.
This is amazing! We're witnessing the beginning of the demise of Intel in the data canter, right in front of our eyes. I had thought this would take decades...
I just hope that thanks to large companies like CloudFlare buying large numbers of ARM servers their prices finally start going down to a reasonable level.
Those are indeed workloads, specific for a webserver and even more specific for CF. Many other workloads that we don't care about that Intel can handily win of course.
We don't have use for most Intel extensions, like AVX512 and barely use any AVX2. Intel is the king of HPC and single threaded performance for a few more years. Although ARM can potentially leapfrog Intel in the HPC space, if someone implements the Salable Vector Instruction extension in 1024 or 2048 bit.
My understanding is that some large parts of the feature set aren't usable anyhow. E.g. there's enough of a ramp up/ramp down power cost for the vector units that you'd never want to compile your binaries with SSE/AVX turned on unless your workload makes near constant use of those units.
It is taking decades at this rate. Or at least a decade.
Not to long ago when Intel was yanking the rug out from under the SPARC and POWER and MIPS and $RANDOM_PROPRIETARY_MAINFRAME_CISC datacenter marketshares.
At $8K savings per server, that is 8M if you have 1000 server at Cloudflare. Of course you have to invest a lot of people into software and testing as well. As well as many other time consuming things that were likely standard on Intel Platform. Still, those wont cost anywhere close to 8M.
To be honest I am very much surprised at how cheap Qualcomm are selling their chip.
This is going to be the start of avalanche. Once the software improves and make a self sustainable ecosystem, everything else will come. I still dont believe Qualcomm could achieve their 5% Server target by 2020 goal. But it is still a good start in longer terms success.
What is more worrying is Intel's growth and Cash Cow is now at risk in the mid term. It will be interesting to see how Intel react to this. There is serious price pressure from AMD EPYC and Qualcomm Centriq. Intel DC 2018 may ship lower unit YoY and lower revenue / unit YoY. PC market is being attacked by AMD, while not at risk, Intel is definitely earning less as they increase core count.
Personally I would have liked to see POWER9 or RISC-V in server space, but both still missing in action.
Encryption is one thing Intel has a lead in, both in single thread, and as a platform. But even there QC is sufficiently fast, and crypto although extensively used actually uses very little CPU. https://blog.cloudflare.com/how-expensive-is-crypto-anyway/
Mid priced xeons: all sorts of motherboards available from eight different Taiwan based OEMs.
AMD: I need some threadripper or server Zen motherboards, again there's 4+ OEMs with many designs.
Both amd and Intel: all sorts of stuff available that meets open compute platform dimensions for motherboards. Or for ordinary 1U servers, or atx size boards that can be mounted in DIY blade enclosures.
ARM: where are the motherboards I can buy right now in small to low quantities, and what are their specs/price? Who makes them? It I want to order 40 or 400 tomorrow... where do I go?
Edit: I just spent ten minutes googling for arm centriq platform motherboards and cannot find a single product I can actually buy. As compared to how easily I could start the process of ordering a hundred 1U xeon based servers in the next half hour.
At a massive scale such as cloudflare, sure you can afford to custom order thousands of motherboards. But for a lot of operators who are not nearly as big as them, the existing economies of scale in the x86-64 platform are a greater advantage. Right now I'd be willing to bet that dual socket current generation xeon motherboards are manufactured in much greater numbers, as much as a ratio of 500:1, compared to ARM server motherboards. Being able to mix and match whitebox parts from Taiwan and China sources to DIY bare metal blade hypervisors is a huge advantage.
Many years ago there was an attempt to get third party manufacturers to start building motherboards for the IBM power4 architecture. Never came to fruition.
Also, when you can get your hands on a ARM motherboard, will you be able to boot mainline Linux or only some ancient kernel version that the vendor has patched with their own binary drivers?
Servers don't need a Mali GPU which is the major bottleneck to support mainline linux.
I'm constantly amazed by people wanting to see Intel die for a minor price decrease. The open source drivers for intel gpus might have not been provided by intel but at the very least they exist. The same cannot be said by ARM. When it comes to proprietary software ARM is at least as evil as intel. By replacing intel with ARM you're going from bad to worse.
Agreed, and Intel actually had a track record of really good open source driver support for important stuff that is not GPU. Example, 11+ years ago when a single 10 gigabit NIC cost $6000, Intel had their own staff write and submit freebsd and Linux kernel drivers for their first gen card. The same has continued up until now if you look at the Linux kernel commits for Intel network interfaces.
I suppose this reality is the reason why ops like Cloudflare make a song and dance about it instead of just quietly putting QC chips in their datacentre: they want the market to flourish so that commodity parts become available, software ecosystem grows, and they start getting the benefits of competition amongst their suppliers.
It is sort of a chicken or egg problem because nobody wants to be the first one to order 5000 bespoke motherboards that might have weird QC and BIOS/firmware issues.
Saw that the Centriq 2434 (40 cores) is listed at RRP $888.
Does anyone know if you can get hold of a Centriq cpu & compatiple motherboard to build a local server or two? I fancy one in our lab rack to play around with.
I'm assuming that this is not possible and that they are currently just for datacentre partners who make a large order. But would be great to hear otherwise.
Not Centriq but I think there are boards with Cavium ARM chips that you can buy as a consumer (can't find link to cite so may be wrong, but may be a useful avenue for your search).
One of the workloads that Cloudflare does is encryption, and RISC-V has been planning a vector extension that can be reconfigured, so that the jump from 128-bit to 256-bit native execution is not an ISA change and doesn't require recompiling.
1. A lot of cloud services are abstracting CPU architecture. E.g. you don't care on which CPU AWS S3 runs.
2. ARM is using less power than x86.
3. ARM license fees are so low comparing to Intel high gross margins. Whole mobile world runs on ARMs and little profit is made on CPUs there comparing to Intel cash cow server chips.
1. Is it? Actually I do not want to recompile all software. Did you try running Docker or Kubernetes with ARM? It's there but there's not a whole lot of software/images you can run.
43 comments
[ 3.0 ms ] story [ 96.0 ms ] threadMost powerful Xeon is the 28 core Platinum 2180 at $10k RSP and >200W TDP. Due to the nature of Intel turbo boost it almost always operates at TDP under load. Our ARM is the Centriq 2452 at less than $1400 RSP, 46 cores and 120W TDP, that it never hits. Beats Xeon 9/10 workloads.
https://twitter.com/thecomp1ler/status/976617883164921857
I just hope that thanks to large companies like CloudFlare buying large numbers of ARM servers their prices finally start going down to a reasonable level.
There is a big chance, that whatever cloudflare is doing on that machine, is massive parallel and simple.
https://news.ycombinator.com/item?id=13925039
Well it took decades...
It is taking decades at this rate. Or at least a decade.
Not to long ago when Intel was yanking the rug out from under the SPARC and POWER and MIPS and $RANDOM_PROPRIETARY_MAINFRAME_CISC datacenter marketshares.
To be honest I am very much surprised at how cheap Qualcomm are selling their chip.
This is going to be the start of avalanche. Once the software improves and make a self sustainable ecosystem, everything else will come. I still dont believe Qualcomm could achieve their 5% Server target by 2020 goal. But it is still a good start in longer terms success.
What is more worrying is Intel's growth and Cash Cow is now at risk in the mid term. It will be interesting to see how Intel react to this. There is serious price pressure from AMD EPYC and Qualcomm Centriq. Intel DC 2018 may ship lower unit YoY and lower revenue / unit YoY. PC market is being attacked by AMD, while not at risk, Intel is definitely earning less as they increase core count.
Personally I would have liked to see POWER9 or RISC-V in server space, but both still missing in action.
I'm really curious at what the blog post will reveal on this.
AMD: I need some threadripper or server Zen motherboards, again there's 4+ OEMs with many designs.
Both amd and Intel: all sorts of stuff available that meets open compute platform dimensions for motherboards. Or for ordinary 1U servers, or atx size boards that can be mounted in DIY blade enclosures.
ARM: where are the motherboards I can buy right now in small to low quantities, and what are their specs/price? Who makes them? It I want to order 40 or 400 tomorrow... where do I go?
Edit: I just spent ten minutes googling for arm centriq platform motherboards and cannot find a single product I can actually buy. As compared to how easily I could start the process of ordering a hundred 1U xeon based servers in the next half hour.
At a massive scale such as cloudflare, sure you can afford to custom order thousands of motherboards. But for a lot of operators who are not nearly as big as them, the existing economies of scale in the x86-64 platform are a greater advantage. Right now I'd be willing to bet that dual socket current generation xeon motherboards are manufactured in much greater numbers, as much as a ratio of 500:1, compared to ARM server motherboards. Being able to mix and match whitebox parts from Taiwan and China sources to DIY bare metal blade hypervisors is a huge advantage.
I'm constantly amazed by people wanting to see Intel die for a minor price decrease. The open source drivers for intel gpus might have not been provided by intel but at the very least they exist. The same cannot be said by ARM. When it comes to proprietary software ARM is at least as evil as intel. By replacing intel with ARM you're going from bad to worse.
Does anyone know if you can get hold of a Centriq cpu & compatiple motherboard to build a local server or two? I fancy one in our lab rack to play around with.
I'm assuming that this is not possible and that they are currently just for datacentre partners who make a large order. But would be great to hear otherwise.
One of the workloads that Cloudflare does is encryption, and RISC-V has been planning a vector extension that can be reconfigured, so that the jump from 128-bit to 256-bit native execution is not an ISA change and doesn't require recompiling.
AMD offers 256 bit vector instructions but they take twice as long because the Zen cores can only process 128 bit at a time.
1. A lot of cloud services are abstracting CPU architecture. E.g. you don't care on which CPU AWS S3 runs.
2. ARM is using less power than x86.
3. ARM license fees are so low comparing to Intel high gross margins. Whole mobile world runs on ARMs and little profit is made on CPUs there comparing to Intel cash cow server chips.
If you use a storage apppiance - be it a local NAS or S3 or B2 - you likely aren’t interested in the cpu running it.
A better comparison would be if you didn’t care what cpu Amazon Lambda ran on. Which, maybe you don’t, if you use it with an interpreted language.