This is quite surprising. I assumed they would get an EPYC server with higher core at similar price (due to the cost of the CPU) and get advantage in performance/dollar. I didn't expect EPYC to beat a similar physical core count intel server running at similar core count let alone by this much margin. Am I missing something here?
I'm saying this as someone who really wants AMD to succeed and push the ecosystem forward. I just want to make sure that I'm reading the results correctly here.
They are effectively getting 35% extra performance 60% of the price that's just beyond impressive.
EPYC is compared to a previous generation Xeon with lower TDP (105w vs 155/170w) and slower RAM (2400 vs 2666). The Xeon server is more expensive probably because it has 4x the amount of RAM.
Edit: actually, this is a dual Xeon machine so its TDP is slightly higher than EPYC's.
> EPYC is compared to a previous generation Xeon with lower TDP ... Edit: actually, this is a dual Xeon machine so its TDP is slightly higher than EPYC's.
"TDP" is not a figure resulting from a standardized process. Each manufacturer has a different (and sometimes changing) definition of it. E.g. for Intel it is a supposed power consumption given a certain load at base clock. Meanwhile with AMD it basically encodes the standard thermal parameters (Tamb, Tj(max), Rthjc).
That's not enough to explain. I highly doubt replacing the 12-core Broadwell Xeon CPUs with a current-generation 12-core Xeon (which would be Skylake) would catch up with the 30-40% performance lead that EPYC displays in their workload.
The Zen microarchitecture of the EPYC is quite impressive and it could just very well be a workload that Zen is particularly good at.
For example the fact EPYC can pack so many cores on a single socket means all the RAM and all its 64MB of last-level cache are "local" to every core (at worst accesses go through the Infinity Fabric.) By comparison, in Intel's case when one of the Xeon cores need to access RAM or cache on the other socket, accesses have to go through the QPI link between the 2 sockets, and it has higher latencies and worse throughput than AMD EPYC's Infinity Fabric.
>They are effectively getting 35% extra performance 60% of the price that's just beyond impressive.
Lets just ignore the price point for now because the memory capacity are different.
But the performance, 24 Core AMD EPYC Zen is ~35% faster than 24 Core Intel Broadwell Xeon under the same frequency?
Broadwell isn't even that old because Intel has only launched Skylake Xeon E5 not long ago. And the different between Skylake and Broadwell IPC aren't that much either.
the max turbo speed are similar, and the Intel platform had the advantage of two CPU using two heatsink, higher TDP, higher heat dissipation more time for turbo.
And most testing has shown Zen to be about Sandybridge IPC performance. Something seems wrong here ( Or may be too good to be true )
Well there seems to be something called nix-shell, but I never heard of it before. It seams pretty useful. The link to plasma tests is provided in the article.
This isn't nix-shell. nix-shell is a tool for spawning ephemeral software environments composed of nix packages, something a little like python's virtualenvs.
Digging a little bit, nixos' tests appear to be written in perl and are executed in a context that has access to a bunch of useful primitives for asserting these sorts of things. https://nixos.org/nixos/manual/#sec-writing-nixos-tests
It's our custom tool. NixOS has testing infrastructure built on top of QEMU, and you can use Perl scripts like this in order to automate aspects of testing inside that environment by talking to the QEMU monitor -- for example, you can not only wait for a window, but click a button in that window, then also wait for a service to start afterwords, for example. (If you were testing GUI functionality).
Tests are written in a mix of Nix and Perl automation scripts like this; the Perl scripts control QEMU, but the Nix code describes what is put into the disk images. It's fairly bespoke and tuned to our own needs. The source code for the Perl driver is here: https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd...
It can even do multi-machine test setups as another comment pointed out, and it's pretty fun for writing functional demonstrations; for example I've been meaning to use these tools to write examples of doing PostgreSQL replication and failover, etc.
The internals of the test framework are pretty heavily tied to Nix and NixOS, so a release independent of Nix(OS) isn't probably going to happen. With that said, the test infrastructure itself is exposed as library functions that any Nix user can use. And even though the test VMs itself will run NixOS, the host machine can be any Linux distro with Nix installed.
I wonder if there were NUMA effects at play since the Intel server was dual-socket and the EPYC was not. It's not entirely surprising that a CPU-heavy workload reading from filesystem cache (i.e. memory) could cause a substantial amount of nonlocal reads that the single-socket server wouldn't have to deal with.
That said, that's just the first thing that popped into my head to explain the result. Either way, the results here are extremely encouraging!
We're now well out of my knowledge base, but your comment certainly makes things interesting. If they have similar NUMA latencies (big if!), I guess maybe the edge would go to whichever chip had higher NUMA bandwidth? Just sort of thinking out loud...
Or perhaps it has absolutely nothing to do with memory speeds, but I can't think of what else would have such a dramatic effect. I'd love to see the same test with a single Xeon.
Unless your buying top of the line Intel parts AMD is going to have more performance at most price points. It's the only logical decision unless you use AVX-512 or your workload is single threaded.
This is comparing far more than Xeon vs. EPYC, not even a single mention of the storage subsystem on these respective machines. It's not like the tasks being timed are 100% CPU-bound...
36 comments
[ 3.0 ms ] story [ 75.1 ms ] threadI'm saying this as someone who really wants AMD to succeed and push the ecosystem forward. I just want to make sure that I'm reading the results correctly here.
They are effectively getting 35% extra performance 60% of the price that's just beyond impressive.
Edit: actually, this is a dual Xeon machine so its TDP is slightly higher than EPYC's.
Comparing TDPs between manufacturers is invalid.
Why? It's all Watts
That's not enough to explain. I highly doubt replacing the 12-core Broadwell Xeon CPUs with a current-generation 12-core Xeon (which would be Skylake) would catch up with the 30-40% performance lead that EPYC displays in their workload.
The Zen microarchitecture of the EPYC is quite impressive and it could just very well be a workload that Zen is particularly good at.
For example the fact EPYC can pack so many cores on a single socket means all the RAM and all its 64MB of last-level cache are "local" to every core (at worst accesses go through the Infinity Fabric.) By comparison, in Intel's case when one of the Xeon cores need to access RAM or cache on the other socket, accesses have to go through the QPI link between the 2 sockets, and it has higher latencies and worse throughput than AMD EPYC's Infinity Fabric.
Lets just ignore the price point for now because the memory capacity are different.
But the performance, 24 Core AMD EPYC Zen is ~35% faster than 24 Core Intel Broadwell Xeon under the same frequency? Broadwell isn't even that old because Intel has only launched Skylake Xeon E5 not long ago. And the different between Skylake and Broadwell IPC aren't that much either. the max turbo speed are similar, and the Intel platform had the advantage of two CPU using two heatsink, higher TDP, higher heat dissipation more time for turbo.
And most testing has shown Zen to be about Sandybridge IPC performance. Something seems wrong here ( Or may be too good to be true )
That's just a hunch though.
So these two servers are pretty similar, I originally thought they were comparing a Amazon m1.xlarge vs EPYC...
Thanks!
https://github.com/NixOS/nixpkgs/blob/49a6964a4250d98644da61...
Digging a little bit, nixos' tests appear to be written in perl and are executed in a context that has access to a bunch of useful primitives for asserting these sorts of things. https://nixos.org/nixos/manual/#sec-writing-nixos-tests
I'll be writing more soon about how these tests work and how neat they are. They really help NixOS quickly release really high quality updates!
In this case, the test makes use of both xdotool and Nix's own testing library.
Tests are written in a mix of Nix and Perl automation scripts like this; the Perl scripts control QEMU, but the Nix code describes what is put into the disk images. It's fairly bespoke and tuned to our own needs. The source code for the Perl driver is here: https://github.com/NixOS/nixpkgs/blob/f148c5c4a1ffa353836bfd...
It can even do multi-machine test setups as another comment pointed out, and it's pretty fun for writing functional demonstrations; for example I've been meaning to use these tools to write examples of doing PostgreSQL replication and failover, etc.
PLEASE consider releasing some form of this tool, or at least documenting the snot out of it... it looks incredible. We'd really use this at work.
- http://open.qa/ - TestKitchen
But I must recommend, indeed, taking a good hard took at NixOS! :)
That said, that's just the first thing that popped into my head to explain the result. Either way, the results here are extremely encouraging!
In NUMA-speak, each group is (probably) a node with neighbouring nodes having a lower distance.
Or perhaps it has absolutely nothing to do with memory speeds, but I can't think of what else would have such a dramatic effect. I'd love to see the same test with a single Xeon.
This seems like a meaningless measurement.