There might be some other cases for less ram. If the rumor that it supports rdimm or lrdimm modules is true then it could enable much easier ecc support since there are not a lot of ecc udimm options out there
Mostly ECC UDIMMs are limited in clock speed, but many can be comfortably overclocked. I presently have four 2666 MHz ECC UDIMMs overclocked to 3200 MHz in my TRX40 workstation (M391A2K43BB1-CTD if anyone is curious).
Saying a $4k USD CPU's price is irrelevant because 128GB DIMMs are expensive is absurd. There is no reason to think everyone who buys one is going to be spending $12k on RAM.
Those are not Threadrippers. IMO AMD misses one market segment where people want workstation CPUs but not so much cores. I would consider 12-16 core Threadripper with 4 memory channels and more PCI-E lanes with adjusted price.
They're not threadrippers but there's not an enormous difference. The equivalent 16-core EPYC has an MSRP that's only 10% more than the desktop chip. The whole stack has similar pricing around core count and performance.
And yeah there is a bit of a gap there. You could always get a low end EPYC but then you miss out on the turbos.
This is commonly referred to as the "HEDT" category, or "High End Desktop"
It's somewhere between the extreme enthusiast market and the workstation market. It's not _truly_ a workstation product as those will instead typically use Xeon or Epyc for things like full official ECC support & other such features. Threadripper muddies this line a bit as it does actually have official ECC support, but motherboard support is not as reliable as you'll find in Epyc boards.
Intel also has products in this category like the i9-10980XE - which is an 18-core product that's essentially the consumer version of a Xeon chip with ECC removed. Intel calls this their X-series line. And then since Intel removes ECC support, you'll again not really find this in workstations.
It's basically "take the server product, remove (some/most) of the server features, jack the power budget to the fucking roof because you can assume water cooling, and allow overclocking" line of products.
> The big news is that the Asia website reported the 3995WX is an offering 8-channel DDR4 memory interface, up to 2 TB of it
This indicates 3995WX supports not only 8ch but also RDIMM/LRDIMM that's not supported on current Threadripper. It should need a new Socket rather than current TRX40 (TRX80 was rumored a year ago). Threadripper gets closer to EPYC.
I expect PCIe lanes limitation still remains for make difference (It's many even limited).
That would be insane. I hope motherboard manufacturers take notice and release an IPMI board for it. We currently run a cluster that has some 3970X's in it and it's super awkward that we couldn't get remote management for them. Asrock has announced availability of one starting august, but it's a little too late for us.
Hm shit, now that you mention it, I didn't even consider X399. The only downside seems to be the lack of support for 3200mhz ddr. In our case that board specifically doesn't have room for the 4th GPU so it's not an option for us.
We've got a weird use case, we're using (black box) software that's sensitive to clock speed, scaling only to about 8 threads until it starts levelling off aggressively (to the point where a 3970X benchmarks as faster than a 3990X). So what we do is we run 4 instances, each with their own GPU and 8 cores assigned to it. I'd be surprised if this makes sense anywhere else (as EPYC has all sorts of other advantages in HPC).
I feel like 64 cores is getting rather close to a tipping point: What workloads are so massively parallel that they can use 64 cores of x86 but can’t use the thousands of CUDA cores on a Quadro card? I’m sure right now there are workloads that just need particular x86 instructions, but that feels like a temporary problem.
Am I wrong about that being a temporary problem (that would feel frustrating)? Are these cores just that much faster than CUDA cores? What else am I missing here?
I often hear people talk about getting CPUs like this for deep learning research, but all the deep learning work I’ve done goes straight to CUDA and lands on my GPU.
One big difference is memory availability; the amount of RAM directly accessible from a 64-core CPU is much larger than that of the thousands of CUDA cores on a Quadro card. Perhaps for many workloads intelligent use of the PCIE bus can make up for that, streaming datasets in/out of the card, but for others, having random access to hundreds of GBs of data may be non-negotiable.
I'd be interested to know what workloads truly require that; most simulation solvers that I can think of that require large amounts of RAM are working on a discretized spatial grid, and the physical laws they're simulating are fairly "local"; e.g. you can process one area then move on to the next, in a nice stencil operation. This lends itself well to both parallelization and a sort of data-locality, so you can stream the necessary pieces of data in to the cores in an orderly fashion. I don't know what kinds of massively parallelizable workloads require much more random access of memory.
The underlying reason is that at around 32GB the JVM internally shifts from 32bit pointers for HEAP management to 64bit. The feature is called compressed object pointers (compressed oops) The increased pointer size uses substantial amounts of RAM, a reasonable estimate is that you need at least ~45GB to not have a loss in total available HEAP. Also, GC cycles get longer the more HEAP you have.
However, some workloads may require more heap to even be possible and there’s pauseless garbage collector implementations able to handle substantial amounts of memory, so it is possible. It’s just not something you should be doing unless you know exactly what you are doing.
While there are no JEP 169 style Value Types, you can still layout your data in a way that significantly reduces the number of pointers. For example using parallel arrays of primitives. It's not as clean, but it is one mitigation.
A "perfect" ES node has 64 gig of memory. 32 to heap and 32 for off heap as well as 8-12 cores for executing queries. Heap is mostly used for internal operations and the inverted index stuff (fielddata), off-heap was used for the columnar data store (docvalues). Most server class machines exceed this nowadays, so we would recommend containerizing multiple instances of ES on a single machine and using the rack awareness feature to make sure shards didn't replicate onto the same physical host. It's been a while since I worked at Elastic, so this info might be out of date by now, but we're still using this formula for our ECE setups where I work now.
The CuckatooN Proof-of-Work puzzle [1] requires 2^N bits of randomly accessed memory for efficient solving which parallelizes very well.
In practice the random access is so slow that an alternative solver using 32 times more memory and mostly sequential access is much faster.
In a single workstation form-factor, once you get enough powerful GPUs, it takes a good amount of CPU power to feed them, even for pure CNNs that GPUs are supposed to be great at. Think data augmentation for training, etc.
> What workloads are so massively parallel that they can use 64 cores of x86 but can’t use the thousands of CUDA cores on a Quadro card?
Thread divergence.
CUDA-cores are linked and take if-statements and for-loops together. This means that if one CUDA core takes a for-loop 1-million times, 31-other CUDA cores will take the 1-million loop with them. (32-cores per NVidia SM). EDIT: CUDA keeps things semantically correct per thread by "throwing away work" (execution mask is disabled, so those 31-other threads effectively execute NOPs), but the cores are wasted.
Worst case scenario, your CUDA cores accomplish 1/32th the work they could do, as they spin idle waiting for the last 1 thread to complete a for loop or its unique combination of if-statements.
Matrix multiplication doesn't have any thread divergence, because all threads loop the same amount of times on all workitems. This means that Tensors / Deep Learning simply ignores the problem, because they're written as matrix multiplication problems.
Avoiding thread-divergence, or mitigating its effects, is possible, but requires advanced programming skill that few have.
--------
Examples:
Chess -- Traditional multithreaded chess algorithms have every thread check out a different branch of the chess search tree. But because each branch has different positions and moves, its very difficult to code it in such a way that all threads are doing useful work.
Web Servers -- If every thread is handling a different request, then every thread probably traverses a different set of if-statements and loops. This is a thread-divergence nightmare.
Databases -- SQL Databases traditionally have each thread running a different, independent SQL query. Like web-servers, its difficult to imagine parallelism from a high-level.
-------
However, elements of Chess, Web-servers, and Databases can probably be efficiently parallelized onto GPUs.
Web-servers -- Text parsing, Regex, and many other problems common to Web-servers have been parallelized on GPUs. The main question for me, is whether or not the PCIe traversal would be worthwhile (lower bandwidth than RAM, high latency). Web-servers tend to be I/O heavy and less compute heavy... probably not dense enough to benefit from GPU compute.
Databases -- Merge-join and Hash-join probably can be parallelized to GPUs.
Figuring out the optimal CPU + GPU team is going to be a research problem over the next 10, maybe 20 years.
You have nailed it; I agree this is the biggest reason why the idea of CPU scaling matching current GPU chips is exciting.
Many other replies speak of the difference between “GPU” and “CPU” without describing it—it’s thread divergence. Processing stream-like data (network streams; compilers and other parsers; human input devices) can branch often and unpredictably using normal algorithms. Think for example how you would tokenize JSON without branching. A single GPU core is quite slow for the general case anyway. They are mostly good at math and bitwise operations. They were able to get this way because of the assumption that there would not be thread divergence for the primary workload, i.e. matrix math on contiguous blocks of memory, which harkens back to their origin as actual Graphics Processing Units.
If thread divergence weren’t such a killer for branching programs, you could probably write anything in CUDA. It’s not very limited; you can write most anything that can be expressed in non-exotic C. A lot the work has to do with schlepping textures around, which is an artifact of the expectation that your code will run at a distance from the messy, branching, unpredictable main program state, so everything (textures, shaders, etc.) should be loaded, processed, and ready to go when it’s frame-buffer time.
GPUs can't run the same code you would have written on a CPU. If you had the choice between an equally performing CPU and GPU you would always use the CPU.
Not really: the CPU costs more and can handle more workloads because it has more complicated control flow.
These days, arithmetic is cheap and control flow is expensive. GPUs and SIMD achieve high throughput by tying a set of arithmetic units together with the same control flow. As soon as you get complicated data-dependent workflows - as soon as you put "if" in your code or a virtual method - the GPU starts getting less efficient.
It's very much a "train vs car" argument: a train can deliver a large number of execution units , provided they're all going to the same place, whereas a car can re-route far more easily.
Fuzzing is unique in that 99% of code probably traverses the same set of if-statements or branches. Whenever code significantly diverges (usually defined by Fuzzers as a significantly different instruction-pointer path of some kind), you've successfully fuzzed a new result.
Fuzzers... strangely enough... are programs I'd expect to run pretty well on GPUs or SIMD. If enough R&D research effort were put into it.
Photogrammetry algorithms that process multiple images independently and then combine the results. Unfortunately they tend to be CPU heavy, and what can be is done with GPU help.
Such workloads are typical in bioinformatics, as the majority of tools do not support running on GPUs. Data parallelization (e.g. run the same tool on every chromosome independently) is a common strategy used in that field.
like all of the legacy quantum chemical and FEM codes or anything else you run in HPC (which are a lot of codes, which are not ported from one day to the other by an army of self-declared ML-pros). Part of them can be GPU-accelerated, but then you can easily eat TBs of memory with the right system and GPUs still have a way to go there.
Surprised databases haven’t been mentioned. Anything stateful is still far easier to scale vertically. Otherwise you have to go multi master and start sharding across smaller instances. A massive chip like this can let you keep a pretty simple master/follower architecture for HA without having to share.
I'm using a 22 core / 44 thread xeon workstation for development and my primary joy using it is being able to run my pytest test suites in parallel. I went from ~45 minutes to run the test suite to about a minute. It is also pretty sweet to be able replicate our three physical servers for bomquote.com, locally.
You can go to Dell, HP, or Lenovo to get a premade Xeon workstation. Getting Epyc premade requires rather more research to find and qualify a vendor, and that vendor may not make it easy to configure and but from the web site with or having to enter a relationship with a sales rep.
I bought the xeon from Ebay at a discount about three years ago, when it was still fairly new. It was some sort of engineering sample and I paid about $3K for it while the list price was at least a multiple higher. We used it for CPU rendering for several years and I've only recently repurposed it for a development workstation, as we moved to software that can utilize GPU rendering. These days I almost certainly buy an AMD Threadripper or Epyc for a development workstation.
I mainly write nodejs code, and now that it supports threads I've been hammering my CPU with all kinds of parallel tasks. One common thing I do is crunch a few thousand 16MB+ JSON files in parallel. Multiple cores saves quite a bit of time parsing huge JSON payloads. Sure, people may crap on me for using Javascript, whatever... It's what I prefer to use, and 64 cores would only help do all sorts of things in parallel, saving me time.
There's all kinds of other things I do that utilize multiple cores.
CPUs are much more general! Unscientifically, on my 2950X/1070Ti workstation, Cycles takes approximately the same amount of time to raytrace a scene on the CPU vs the GPU. I think this is a great example of a highly parallel workload that is challenging for GPUs due to lack of coherence - extreme props to the Cycles team for making it work on GPU at all.
Also, year-on-year performance increases for nVidia GPUs are relatively boring, while Threadrippers routinely double in performance. I have just built a 3970X/RX5700XT workstation, and while I have not benchmarked it yet I expect the CPU to smash the GPU for Cycles raytracing.
Also worth noting that CUDA is a proprietary architecture from a single company - targeting it locks you in, and you need to run proprietary blobs to make it work.
All in all, CUDA works well now but I think it's ultimately a dead-end architecture - hugely parallel CPU architectures will eclipse it.
The people that use CUDA don't care about the underlying architecture, they need results. Which ever manufacturer will deliver those results will get the $.
Raytracing is not an ideal match workload wise for a GPU, too much divergence, but there are plenty of workloads where GPUs will have an edge for along time to come.
It's more than that - NVIDIA invested massively in their software ecosystem and mindshare (e.g. by a huge sponsoring program that saw universities flooded with their hardware).
AMD cards offer comparable compute performance and are sometimes even better. What AMD lacks, though, are ready-to-use software packages like CUDA, cuDNN, cuBLAS, and the like.
NVIDIA basically bought their way into the research community and solidified their strong foothold by keeping CUDA proprietary.
It's fascinating how other companies (Microsoft) get bashed on a regular basis for using the exact same tactics (see Direct3D vs OpenGL on Windows), while people are perfectly fine with NVIDIA doing it...
Because NVIDIA did not do anything illegal - so far. MS is a convicted monopolist that actively went after competitors rather than just to let the market decide. What NVIDIA has done so far is perfectly fine in my book, paying companies not to use competitor hardware, funding third party lawsuits and stealing IP, the browser wars and so on are things that Microsoft has done.
> a huge sponsoring program that saw universities flooded with their hardware
This is really important, and I don't think people quite realise the scale. I was doing a Physics MSc in 2012 and it seemed that almost everybody in computational physics was doing some variant of reimplementing Y with CUDA.
That's not to say it was the wrong tool for many of the jobs – though for some it apparently was, judging by the levels of stress – but I did find it a bit weird how this was an approach to research that pretty much completely ignored vendor lock-in.
I assure you, the affected scientists who have to fight tooth and nail for funding are less than impressed that they are dependent on the mercy of a single vendor.
AMD does have ROCm/HIP, hipDNN and hipBLAS (which mirror their counterparts), but these and the rest of their modern GPGPU stack suffer from a lack of polish, ineffective evangelism and subpar documentation. It's difficult to justify using them unless you work at somewhere like ORNL where folks are building new infrastructure using exclusively AMD GPUs.
They are not "optimized" strictly speaking. Thousands of CUDA cores on them don't do ray tracing, it is done on a separate hardware which just happens to sit on the same chip. Add RTX cores to Threadripper crystal and you will get the same result.
I don't know, the 3970X has nearly 3 times as many cores as your 3900X. You may be right if pitted against an OptiX GPU, but that's a bit unfair since that entire generation's raison d'etre is raytracing-specific hardware - my point concerned GPGPU, with raytracing as an incidental example. nVidia can't redesign their GPUs for every workload that GPUs struggle with.
If you look at the Blender benchmarks you see that the 64 core Threadripper is faster than any GPU. But the price is steep for that CPU (almost $4000). So while CPU's can be faster the question is if it is relevant. Because you still need a GPU next to that Threadripper for some graphics (if you need graphics).
There's quite considerable variation in the device ranking if you look at each scene render time - the GPU is faster on the quicker-to-render (sub 1 min) scenes, but slower on the slower-to-render scenes (1+ min).
I wonder why that is? I see the 3990X is hugely faster than anything at these slow scenes, which is pretty interesting.
I'm not familiar with rendering, but in the deep learning world it is moderately easy to get considerable speed-ups by using multiple GPUs. Is the same possible here?
> What workloads are so massively parallel that they can use 64 cores of x86 but can’t use the thousands of CUDA cores on a Quadro card?
Column stores like SAP HANA. Compressed bitmap indexes leverage SIMD but not floating point matrix math GPUs. The workstation vs server case is narrower though; benchmarking, dev testing, and local MS PowerPivot like analysis on large in-memory datasets.
Two reasons why I can't do my stuff with CUDA even though I've got thousands of work packages that are independent of each other:
* I'm splitting data sets into batches of around 50-300MB, but each individual batch must be processed sequentially. Each batch loops through the data (multiple times), and every single iteration of a loop depends on state from previous iterations.
* Altough each batch essentially does the same thing, it does not operate in lockstep. Lot's of ifs, different execution paths, differently sized sub-problems, etc.
I'd benefit from having 200 CPU cores and enough RAM to fit the batches, but I can't let individual GPU threads run through batches of 50MB, each.
What a miserable hopeless post - "they're going to fuck us and what can we do????".
Answer: quite a lot, if you have even minimal technical understanding (like, enough to use a blocklist or an ad blocker). Are you incapable even of that?
Why would it be difficult to turn off web sockets or turn off multi-threading? Multiple threads already run on single cores all the time, there is no reason a web browser couldn't just put them all on one core.
To be clear, you are saying you don't want something powerful and fundamental to be added because you don't want to learn how to turn a feature off or learn how install an adblocker?
Not only that, but better performance is what pushes you over the edge, not popups, connection slow downs, auto playing videos or tracking?
Any compiler that can properly use multiple cores could potentially benefit. Rust (and Haskell's GHC) uses all 8 cores I have during a sub-period of recompilation, so I'd guess it could as well use more.
At the moment, offline rendering is still massively parallel, but you run into memory limitations when trying to render on the GPU. High end VFX scenes using Renderman or Arnold can push up to 100 GB+ of memory and run for multiple hours on standard 12-24 core machines. I believe some GPU renderers can do "out-of-core" rendering, but not all of them support that yet.
The high end renderers usually split up a large image into "buckets" and then perform path tracing on those regions of an image, and combine the results together. If you have more cores, you can increase the number of simultaneous buckets to evaluate.
To put this in perspective, a lot of the recent news around real-time raytracing is when a game engine does 1-4 samples per pixel and then aggressively denoises (in 2D) the result. That's why you often see splotches or errors. On the other hand, high end VFX renders will end up doing hundreds or even thousands of samples per pixel to get a clean, high quality, physically plausible result.
There’s such a huge difference in the programming paradigm between writing for multi cores on a GPU versus a CPU. It’s really a very different kind of software architecture. I think it’s unfair to compare the two.
And with such a huge CPU core counts, a lot of the very difficult work of organizing an algorithm to run well on a GPU can be avoided because the CPU approaches the same benefits with a more logical architectural design.
Although graphics companies like to talk about "cores" or "processors" to make their GPUs sound powerful, they are really just counting ALUs. If you marketed this chip similarly, you'd end up with something silly like "1024 AVX Core CPU".
A closer analogue of x86 cores is what Nvidia calls a "Streaming Multiprocessor" and AMD calls a "Compute Unit". There's generally only up to 64-80 of those on the highest end GPUs too. However the difference is that each of those cores is able to perform 40-64 SIMD operations at once, compared to x86's puny 8 or 16 or so.
> What workloads are so massively parallel that they can use 64 cores of x86 but can’t use the thousands of CUDA cores on a Quadro card?
Building software, for one. C compilers and python interpreters don't run on a GPU. Lots of stuff doesn't run on a GPU. In fact in practice the only things that run on a GPU are the tiny handful of known subproblems that the industry has collectively decided are "GPU problems".
Like it or not general purpose scalar software is, has always been, and will always remain the standard mechanism by which computing hardware is applied to new problems. Everything else is an optimization around the edges.
GPUs are mainly good at doing matrix multiplication and other math with little branching. Compilers have a lot of if/else statements; training neural networks does not. IIRC (and I may be wrong) the architecture is something like many GPU cores share things like the code cache and branch prediction, so if the different cores have to execute different code paths there is a big performance decrease.
Also GPUs really like predictable memory access patterns so that they can load the data that'll be needed next ahead of time into registers or dedicated caches explicitly. They don't have cache hierarchy features a CPU has to minimize the average disruption from only partially predictable memory accesses and they don't have out of order windows to paper over the occasional low level cache miss.
A really interesting exploration is Co-dfns [1]. This is indeed a compiler that runs on the GPU, but it's also extremely out of the mainstream.
IMHO this is an area ripe for more exploration. If I were working on it, I might look to linking first before compilation, because the basic link task is more similar to what GPUs are good at (advanced stuff such as LTO is a different story, though).
I'm not totally up to speed on modern GPU's - do they have a MMU that handles page faults on non-resident pages like a CPU would?
My understanding is that the GPU memory models are different enough that what an OS traditionally calls "virtual memory" couldn't be implemented in the same way.
Well, the problem with that is that no make system can utilise so much core. Even linux kernel, probably the biggest C project in mainstream use, can not consistently load even 16 cores with mostly handwritten makefiles.
P.S.
I do not say that building software is not CPU parallelisable in principles, I'm saying that regular make systems have trouble handing big number of parallel tasks in practice, even in big, well run projects that use handwritten makefiles.
That's... not true at all. A kernel build will be as parallelizable as you like. I do kernel builds fairly regularly on 56 and 64 core hosts and in fact the build time is quite linear with core count. (The ramdisk packing provided by the distro at "make install" time tends to be what hurts scaling, FWIW).
And my day job involves lots of iterations over the Zephyr test suite, which builds and runs hundreds of individual test apps X dozens of platforms. This is almost a pure build-time problem and scales very linearly indeed.
CPU-parallelizable and GPU-parallelizable are different problems. CPU cores are independent. GPU cores are SIMD--every core does the same operation, but with different data. That works really well for things like matrix multiplication where you need to multiply a million times and then add a million times, because you can have all 3000 CUDA cores multiplying and then all 3000 CUDA cores adding. It doesn't work for compiling software.
It's not that SIMD for GPUs. It'll be 64 or 32 GPU threads together generally sharing the dispatch and control logic. So if you have a 4096 "CUDA core" GPU, it'll be 64 of what would be called cores on the CPU side, each with a 64 lane vector processor.
Bazel utilizes the cores near linearly. It even makes use of remote cores in an efficient manner for compiling c++. 128 thread and 256 thread machines are great if you also have enough ddr4 to let each gcc/clang do their thing.
Something is wrong on your end, you can parallelize the build of typical C and C++ software almost infinitely. The reason why is because most C and C++ software do not have 'happens-before' dependency relationships due to the way translation units work, so basically every translation unit in a module can be compiled 100% independently of any other translation unit. In practice that means you can easily scale up to 1000+ processes with no issue, and in fact, this has been accomplished in combination with AWS Lambda:
Disclaimer: I am not a HW designer, I could very well be wrong.
It is true that there are tasks where threading matters, but still require a CPU rather than a GPU. I wonder however if these tasks do need full SSE/AVX etc. Couldn't these extensions be removed of the CPU cores and instead have the necessary work performed by the GPU?
It would be interesting to produce statistics on how much these extensions are used in these scenario. Imagine how much space and complexity could be saved on a CPU die by making stripped down versions. That space could in turn be used for more cores!
I read a little about the Xeon PHI cpus, which iirc, is a multicore CPU with a very small ISA, but I wonder why x86 makers aren't trying to go in that direction: isn't there plenty of dedicated workloads which would happily run on these (eg, web servers), or is this just a (too) simplistic view?
> It is true that there are tasks where threading matters, but still require a CPU rather than a GPU. I wonder however if these tasks do need full SSE/AVX etc. Couldn't these extensions be removed of the CPU cores and instead have the necessary work performed by the GPU?
SSE/AVX shares an L1 cache that's damn near instantaneous to access for the CPU core. Total L1 bandwidth is on the scale of TB/s.
PCIe -> GPU takes 1-microsecond to 10-microseconds per access, and operates only at 50GB/s (or 1/20th the speed of L1 bandwidths).
------------
Case in point: Memset is very commonly AVX'd to clear out L1 cache and initialize ~1kb to 32kb of data to 0 as quickly as possible.
There's no way for "memset" to move from CPU to GPU unless you feel like obliterating the entire point of L1, L2, and L3 cache. If you moved a "memset" to GPU, it'd operate only at 15GB/s (the speed of PCIe 3.0 x16 lanes), far, far slower than L1 cache AVX-loads/stores.
SIMD units, like SSE and AVX, are highly "local" and have huge advantages.
I think the opposite is where things need to go.
Having a wide SIMD ALU quickly accessible from your CPU core is very useful, especially as it shares the same memory system and a much more flexible programming model that allows you to do everything in a single source.
The programming model is not very flexible at the lowest level: one has to create all the software infrastructure to communicate with the GPU (which boils down to sending commands and receiving response). There are languages (like futhark, julia, or even python), which handle all that boilerplate transparently.
The main problem is, afaik, that there is not enough control about where the code will run in these languages. At some point, one will want to describe all the algorithms using a single language, and somehow describe how the workload will have to be distributed across all the processors, or at least that's what I've been thinking about for a while. Once you have that level of control, the need for a versatile CPU is less clear. Note that nowadays people seems happy with hybrid solutions where the code is scattered across several languages (eg, one for the main program and one for the shaders, or for the client side UI), so my position is maybe not very strong.
HW-wise, is it possible that integrated GPUs are the first steps toward an architecture where CPU and GPU have better interconnections (ie, larger communication bandwidth and smaller latency) to the point where SIMD becomes moot? There is also the SWAR approach, where one doesn't rely on intrinsic SIMD instructions, but instead emulate them (though it's probably not very realistic for floating point computation).
Some other ideas:
- Apple has this neural engine in their latest chips, which is basically dedicated HW for neural networks
- In the wild, people are getting more and more interested in building their custom ASICs to cut software's middle-man cost: for them, the CPU solution is not good enough
- Intel recently introduced a new matrix ops extension in their CPUs: maybe at some point they'll introduce full GPU capabilities directly baked in the CPU? I am a little worried about the resulting ISA.
Anyway, I am not an HW engineer, nor a very good software one. I only have a limited view of the difficulties in writing good, CPU or GPU efficient code. My first post was prompted by remembering the first "large scale" multicores CPUs 15 years ago (specifically the Ultrasparc T1) which wheren't SIMD heavy. The direction naturally shifted as progress was made on SIMD to try to compete with GPUs, when it seems to me that originally CPUs and GPUs were complementary.
I tend to support modular solutions, but I don't know how costly that would be in term of efficiency at the HW level.
This is a point I have been trying to make for a few years now to ever-decreasing amounts of criticism. I think the reality is starting to set in that x86 is actually incredibly powerful, and when you have 64+ parallel copies of that computational facility, you can perform some insane magic. Especially when you start to consider the memory/computation domain you are operating within on CPU when contrasted with a GPU:
With a GPU in the loop, there is an extremely painful hop across PCIe before you can experience any degree of cache coherency. There are protocols, buffering and driver stacks involved. Latency is much higher. You have to plan ahead and think about what memory the GPU needs to mutate vs what memory the CPU needs to mutate. You have to split your application, language, and frameworks across 2 computational domains. How we have tolerated this for so long is beyond me. I think most people who get caught up in it are chasing shiny marketing a lot of the time. This criticism aside, I do think GPUs are still very useful for many applications and users.
With just a CPU, you are looking at L3 as the typical worst case for cache coherency domain. And L3 is getting to be ridiculously large. If your high-performance application's binary image cannot entirely fit within L3 of a modern x86 CPU, you are probably doing something very wrong. The other big advantage with just a CPU is you have 1 cache-coherent memory domain and a single instruction set to answer to. This means you can write 100% of your software in a single language/framework. Well-architected x86 applications can push instructions many times faster than their core clock speed would seem to indicate possible. Using special frameworks that have sympathy with this memory model mean that just a single x86 core can be made to push tens of millions of logical business transactions per second. And then you have 63 other cores to play with. Extremely deep pipelining and OoO execution are what allow for x86 to chew through general purpose computation so easily. Anything with a lot of recursive depth (e.g. raytracing) benefits massively from this style of processor.
I think the next big revolution is developers realizing that they can start porting GPU applications to CPU. Things like raytracing are much better suited for the memory model offered by a pure x86 domain. Sure, a GPU can accelerate some aspects of raytracing, but then you then need to shuttle that information back and forth over a higher latency link, and also fight with 2 completely different technical stacks. Needing to know how to write code for both GPU and CPU is easily the biggest problem with all of this. Especially when you consider how fast GPU APIs move relative to x86. Performance is not the only constraint when dealing with software, especially software as complex as AAA game engines and 3d modeling software. Someone still has to reason with and maintain this stuff over time.
Almost all scientific computing.
In spite of considerable effort, there are relatively few workloads that use GPUs well, and even for codes which do work on GPU, you tend to end up with partial functionality, and a very restrictive region of parameter space where they work efficiently. (And often you run out of memory very quickly - scientific workloads typically run multi-node and share RAM across the nodes via Global Arrays or equivalent. Using multiple terabytes of memory is absolutely normal.) The computation in deep learning is actually quite simple (matrix-vector operations), and still it has taken considerable effort to develop efficient kernels (such as cuDNN).
GPUs are good for split join parallelism. They don't even really have real branching and you mostly have to loop a constant amount of times. Meanwhile any data without dependencies can be dealt with on a CPU core.
Not only that, huge amounts of software could use a lot more concurrency if it was structured differently. Even a web browser (to my knowledge) doesn't split up DOM layout, parsing individual files, decompressing images, decompressing video etc. into separate threads for each file.
> Are these cores just that much faster than CUDA cores? What else am I missing here?
Comparing GPU/CUDA/TPU Cores to CPU Cores is like comparing a riding lawnmower to an automobile. A GPU/CUDA/TPU Core is optimized to handle a different type of workload than a CPU.
Just because your riding lawnmower can cut your grass faster than your car, doesn't mean your lawnmower is the best thing for highway driving.
My understanding of these problems is totally naive, but I wonder if you could mitigate speculative execution vulnerabilities (meltdown, spectre, yes I know the former does not apply to amd) by using the excess cores to speculate enough that a cache hit is no longer meaningful data.
I see this response often and I have to say, "just throw it on the GPU" is phrase only uttered by someone who has never personally attempted to do that. The simplicity of CUDA and `pip install tensorflow-gpu` belie a huge amount of engineering prowess; go look at the source for XLA (https://github.com/tensorflow/tensorflow/tree/master/tensorf...) if you don't believe me. To answer your question, tons of scientific computing tasks parallelize nicely and are a horrible fit for a GPU, because they involve branching, or non-deterministic runtimes, or need a lot of memory bandwidth, or rely on highly optimized legacy numerical code that is written in Fortran, or any one of a number of other reasons. Deep learning is a very narrow use fact and, in fact, GPUs and DL symbiotically co-evolved over the last 15 years to the point where GPU hardware is essentially designed to efficiently train neural nets. (And run games.)
Tasks that involve branching, for one. GPU's are fundamentally different kinds of processors. They are for applying similar transforms to vast amounts of data.
There are other area where big general purpose X64 or ARM cores excel too, like bit twiddling, memory random access, etc., but branching logic is the fundamental one.
In reinforcement learning, you typically need to run many copies of an environment (an Atari emulator for example) to collect experience for your agents. Environment code usually makes more sense on a CPU, because it’s a lot more branchy and stateful.
Think of two agents that run for awhile and end up in different parts of a virtual world, that require very different code to execute. This might be pretty difficult to parallelize on a GPU.
After collecting a bunch of experience, you update whatever function you’re optimizing. This could be something like a neural network that is the brain of your agents. The update step can happen on a GPU.
But then we need to go collect a lot more experience again, so we’re back in CPU land.
Experience collection often dominates overall compute time in reinforcement learning. Which means you want a lot of CPUs.
> What workloads are so massively parallel that they can use 64 cores of x86 but can’t use the thousands of CUDA cores on a Quadro card?
I'm a bit behind on GPU architecture, but presumably workloads which are branch-heavy, lack coherent execution within a warp (see dragontamer's comment), and don't make use of floating-point.
We process, merge and create PDF files. Sometimes large multi-GB PDF files. We try to adjust and optimize them in parallel, but often we also need to merge different files into one for further processing, for example printing or archival. We need a lot of ram and it takes a lot of time. The more CPU, the better.
What they call "cores" in CUDA is different from CPU cores. On GPU "cores" and "threads" refer to the ways in a SIMD ALU. Under this classification a 64 core CPU with AVX-256 like Threadripper is at least 512 cores. So 4K "CUDA cores" translates to 128 "CPU cores" when you are not doing massively parallel computations. 128 cores still is no joke but those are not Zen cores, unfortunately. On top of running off much lower clock (7nm GPUs are at ~2Ghz, 14nm is ~1Ghz) their IPC is nowhere close to that of a CPU, add advanced caches and memory controllers, branch prediction, speculative execution etc and CPU cores are much faster. A GPU will beat any CPU on a SIMD-friendly payload but it won't be any good for serving webpages or compiling code or doing any other task you cannot nicely lay out for a SIMD.
I wonder how much the silicon quality differs on a single Threadripper Chip.
It would really be interesting to see if it's feasible to Overclock the best numa node to a degree that gives this a couple of high single thread performance cores comparable to the 3900X or 3950X while retaining stability overall.
Interesting, is it already feasible to pin tasks to such OCed cores?
It would be interesting to turn the 3990/3995 into a does-it-all chip with best in class single-core performance on certain cores and just lots of a tad slower cores in general.
You can attempt to manually OC higher, but that by & large doesn't work without extreme cooling. Typically instead it's about achieving higher all-core frequencies than the built in all-core turbo, typically via higher voltages & power limits (and of course much better-than-stock cooling)
They all hit around that same 4.5ghz single core turbo mark without any overclocking. And Zen2 by & large doesn't really go much beyond 4.5ghz anyway, so there's not all that much manual overclocking you can do without going sub-ambient cooling anyway.
The only real downside to Threadripper 3rd gen is the price. It's already a pretty killer jack-of-all-trades CPUs otherwise. It's a very competent gaming CPU without any tweaks at all right out of the box. It doesn't at all have the cons of the 1st & 2nd gen Threadrippers, which were actually NUMA and therefore came with huge gaming downsides.
A lot of people question the usefulness of 64+ cores on a workstation CPU. It is true that audio/visual tools like Adobe After Effects stop benefiting from extra cores at around 32 cores. [1]
But if you want to simulate production workloads on your machine say with Apache Spark or Flink, you are in the prime. You couple that CPU with 256+ GB memory, and you can start exploring the bottlenecks in your batch algorithms using as many cores as you want.
I think we will be contending with this sort of barrier for a while on desktop software. 32 cores used to be massively parallel 10 years ago, 15 years ago it was nuts to think that we'd have that in a get-it-from-Newegg offering.
Massively parallel operations take a different focus on your MPI/bus/queue architecture than we've ever needed for desktop software. BUT I think what we've already seen is that in general purpose computing there will always be someone ready to consume resources as they are available.
Both feet don't move at the same time, one takes a step and then the other.
The (bit of) article mentioned 2T, but the thing that was interesting was the 8 channel memory. My current TR4 boards are doing quad, so 2 banks of 4 slots. Sounds like a single set of 8... so wonder if this is a potential BIOS update on the sTR4 or if there is a new chipset on route?
The traces are wired to CPU pins, there is no potential BIOS update to make it 8 channels from CPU pins for only 4 channels. You need different socket with double the number of pins for memory to double the number of channels.
Your 4 channels with 2 slots each have the 4 channels daisy chained to 2 slots, but they are just 4 channels. With 8 channels and 2 slots per channel you can have 16 slots, with 32 GB unbuffered (regular) DIMMs you are limited to 512 GB of RAM, for 2 TB you need registered or load reduced RAM.
It’s unclear to me what eight channel memory actually means, I’m not used to seeing that mentioned in casual architecture overviews. What’s the benefit of that? Is that something that a programmer must explicitly take advantage of, or is that just an automatic part of the CPU?
from the software standpoint, it's invisible to the programmer. it's about bandwidth of the memory bus. when you have single shared memory lane, bandwidth gets split between memory modules. typically(on most of the consumer hardware) there are 2 memory channels.
Nearly every consumer CPU for the last 20 years is dual channel, meaning "2 drive RAID 0". If you've seen things like recommendations to get paired DDR memory sticks, this is why. You only get this RAID-like benefit with multiple RAM sticks (just like RAID 0 of a single drive doesn't do anything). It's also why some consumer products have unexpectedly bad performance for the CPU specs - cheaper laptops may skimp here and just run a single stick of memory instead of 2. Which means half the memory bandwidth.
8 channel then means the CPU can do this up to 8 sticks for 8x the bandwidth instead of the more common 2x.
More channels allow more memory bandwidth, though actual performance gain is marginal unless the data being processed has very strict spatial locality (e.g. fluid dynamics simulations), most other workloads are more constrained by cache.
The real reason for adding more memory channels to a general purpose CPU is actually to increase the maximum amount of memory per node. A few years ago, workstation and server platforms often allowed 3 or more DIMMs per channel with performance declining sharply with memory density. Since then both AMD and Intel have limited each channel to 2 DIMMs and increase the number of memory channels from 4 to 6/8 to make up for the loss.
While more memory channels do enable better performance scaling, it has also made the CPU socket more complex and fragile than ever: The TRX40 socket has 4094 pins and literally requires the CPU to be bolted down using a torque wrench to ensure good contact.
169 comments
[ 3.2 ms ] story [ 199 ms ] threadAMD use this term themselves.
I'd expect closer to $4000 or more.
> is this for the consumer line or server line of products?
Neither or both. Threadripper has high-clocks (like consumer) and high-core counts (like servers). It'd be for "workstations".
And yeah there is a bit of a gap there. You could always get a low end EPYC but then you miss out on the turbos.
It's somewhere between the extreme enthusiast market and the workstation market. It's not _truly_ a workstation product as those will instead typically use Xeon or Epyc for things like full official ECC support & other such features. Threadripper muddies this line a bit as it does actually have official ECC support, but motherboard support is not as reliable as you'll find in Epyc boards.
Intel also has products in this category like the i9-10980XE - which is an 18-core product that's essentially the consumer version of a Xeon chip with ECC removed. Intel calls this their X-series line. And then since Intel removes ECC support, you'll again not really find this in workstations.
It's basically "take the server product, remove (some/most) of the server features, jack the power budget to the fucking roof because you can assume water cooling, and allow overclocking" line of products.
This indicates 3995WX supports not only 8ch but also RDIMM/LRDIMM that's not supported on current Threadripper. It should need a new Socket rather than current TRX40 (TRX80 was rumored a year ago). Threadripper gets closer to EPYC.
I expect PCIe lanes limitation still remains for make difference (It's many even limited).
We've got a weird use case, we're using (black box) software that's sensitive to clock speed, scaling only to about 8 threads until it starts levelling off aggressively (to the point where a 3970X benchmarks as faster than a 3990X). So what we do is we run 4 instances, each with their own GPU and 8 cores assigned to it. I'd be surprised if this makes sense anywhere else (as EPYC has all sorts of other advantages in HPC).
For others, I believe this is the board in question: https://www.asrock.com/asrockrack/general/productdetail.asp?...
Am I wrong about that being a temporary problem (that would feel frustrating)? Are these cores just that much faster than CUDA cores? What else am I missing here?
I often hear people talk about getting CPUs like this for deep learning research, but all the deep learning work I’ve done goes straight to CUDA and lands on my GPU.
I'd be interested to know what workloads truly require that; most simulation solvers that I can think of that require large amounts of RAM are working on a discretized spatial grid, and the physical laws they're simulating are fairly "local"; e.g. you can process one area then move on to the next, in a nice stencil operation. This lends itself well to both parallelization and a sort of data-locality, so you can stream the necessary pieces of data in to the cores in an orderly fashion. I don't know what kinds of massively parallelizable workloads require much more random access of memory.
See https://wiki.openjdk.java.net/display/HotSpot/CompressedOops
[1] https://github.com/tromp/cuckoo
Thread divergence.
CUDA-cores are linked and take if-statements and for-loops together. This means that if one CUDA core takes a for-loop 1-million times, 31-other CUDA cores will take the 1-million loop with them. (32-cores per NVidia SM). EDIT: CUDA keeps things semantically correct per thread by "throwing away work" (execution mask is disabled, so those 31-other threads effectively execute NOPs), but the cores are wasted.
Worst case scenario, your CUDA cores accomplish 1/32th the work they could do, as they spin idle waiting for the last 1 thread to complete a for loop or its unique combination of if-statements.
Matrix multiplication doesn't have any thread divergence, because all threads loop the same amount of times on all workitems. This means that Tensors / Deep Learning simply ignores the problem, because they're written as matrix multiplication problems.
Avoiding thread-divergence, or mitigating its effects, is possible, but requires advanced programming skill that few have.
--------
Examples:
Chess -- Traditional multithreaded chess algorithms have every thread check out a different branch of the chess search tree. But because each branch has different positions and moves, its very difficult to code it in such a way that all threads are doing useful work.
Web Servers -- If every thread is handling a different request, then every thread probably traverses a different set of if-statements and loops. This is a thread-divergence nightmare.
Databases -- SQL Databases traditionally have each thread running a different, independent SQL query. Like web-servers, its difficult to imagine parallelism from a high-level.
-------
However, elements of Chess, Web-servers, and Databases can probably be efficiently parallelized onto GPUs.
Chess -- This github repo demonstrates that enumerating positions can be done GPU-parallel: https://github.com/ankan-ban/perft_gpu
Web-servers -- Text parsing, Regex, and many other problems common to Web-servers have been parallelized on GPUs. The main question for me, is whether or not the PCIe traversal would be worthwhile (lower bandwidth than RAM, high latency). Web-servers tend to be I/O heavy and less compute heavy... probably not dense enough to benefit from GPU compute.
Databases -- Merge-join and Hash-join probably can be parallelized to GPUs.
Figuring out the optimal CPU + GPU team is going to be a research problem over the next 10, maybe 20 years.
Many other replies speak of the difference between “GPU” and “CPU” without describing it—it’s thread divergence. Processing stream-like data (network streams; compilers and other parsers; human input devices) can branch often and unpredictably using normal algorithms. Think for example how you would tokenize JSON without branching. A single GPU core is quite slow for the general case anyway. They are mostly good at math and bitwise operations. They were able to get this way because of the assumption that there would not be thread divergence for the primary workload, i.e. matrix math on contiguous blocks of memory, which harkens back to their origin as actual Graphics Processing Units.
If thread divergence weren’t such a killer for branching programs, you could probably write anything in CUDA. It’s not very limited; you can write most anything that can be expressed in non-exotic C. A lot the work has to do with schlepping textures around, which is an artifact of the expectation that your code will run at a distance from the messy, branching, unpredictable main program state, so everything (textures, shaders, etc.) should be loaded, processed, and ready to go when it’s frame-buffer time.
These days, arithmetic is cheap and control flow is expensive. GPUs and SIMD achieve high throughput by tying a set of arithmetic units together with the same control flow. As soon as you get complicated data-dependent workflows - as soon as you put "if" in your code or a virtual method - the GPU starts getting less efficient.
It's very much a "train vs car" argument: a train can deliver a large number of execution units , provided they're all going to the same place, whereas a car can re-route far more easily.
Its quite possible that someone figures out an x86 emulator on GPUs that fuzzes register values, similar to AVX512 fuzzing. https://gamozolabs.github.io/fuzzing/2018/10/14/vectorized_e...
------
Fuzzing is unique in that 99% of code probably traverses the same set of if-statements or branches. Whenever code significantly diverges (usually defined by Fuzzers as a significantly different instruction-pointer path of some kind), you've successfully fuzzed a new result.
Fuzzers... strangely enough... are programs I'd expect to run pretty well on GPUs or SIMD. If enough R&D research effort were put into it.
Gentoo is actually a usable distro again and most of the packages (even things like mono) install decently fast.
I do not personally, but I work with another team that uses tens of thousands of cores for calculations. Just day to day business.
There's all kinds of other things I do that utilize multiple cores.
I also run a lot of VMs.
Also, year-on-year performance increases for nVidia GPUs are relatively boring, while Threadrippers routinely double in performance. I have just built a 3970X/RX5700XT workstation, and while I have not benchmarked it yet I expect the CPU to smash the GPU for Cycles raytracing.
Also worth noting that CUDA is a proprietary architecture from a single company - targeting it locks you in, and you need to run proprietary blobs to make it work.
All in all, CUDA works well now but I think it's ultimately a dead-end architecture - hugely parallel CPU architectures will eclipse it.
Raytracing is not an ideal match workload wise for a GPU, too much divergence, but there are plenty of workloads where GPUs will have an edge for along time to come.
AMD cards offer comparable compute performance and are sometimes even better. What AMD lacks, though, are ready-to-use software packages like CUDA, cuDNN, cuBLAS, and the like.
NVIDIA basically bought their way into the research community and solidified their strong foothold by keeping CUDA proprietary.
It's fascinating how other companies (Microsoft) get bashed on a regular basis for using the exact same tactics (see Direct3D vs OpenGL on Windows), while people are perfectly fine with NVIDIA doing it...
This is really important, and I don't think people quite realise the scale. I was doing a Physics MSc in 2012 and it seemed that almost everybody in computational physics was doing some variant of reimplementing Y with CUDA.
That's not to say it was the wrong tool for many of the jobs – though for some it apparently was, judging by the levels of stress – but I did find it a bit weird how this was an approach to research that pretty much completely ignored vendor lock-in.
I assure you, the affected scientists who have to fight tooth and nail for funding are less than impressed that they are dependent on the mercy of a single vendor.
On the other hand, wasn't GPU mode added much later than CPU mode? Maybe there has been less development time put into it.
The CPU is never faster than the GPU. CPU + CUDA is way faster, but when I use OptiX (RTX) it is unbelievable how fast it is.
The GPU Raytracer just shines here.
The Blender benchmarks: https://opendata.blender.org/
Maybe that time will come but I am not sure if that will happen soon.
There's quite considerable variation in the device ranking if you look at each scene render time - the GPU is faster on the quicker-to-render (sub 1 min) scenes, but slower on the slower-to-render scenes (1+ min).
I wonder why that is? I see the 3990X is hugely faster than anything at these slow scenes, which is pretty interesting.
I'm not familiar with rendering, but in the deep learning world it is moderately easy to get considerable speed-ups by using multiple GPUs. Is the same possible here?
So it all depends in what is being rendered and how a render core is optimized for it.
Also things like b-tree calculations and other algorithms might influence this.
Column stores like SAP HANA. Compressed bitmap indexes leverage SIMD but not floating point matrix math GPUs. The workstation vs server case is narrower though; benchmarking, dev testing, and local MS PowerPivot like analysis on large in-memory datasets.
* I'm splitting data sets into batches of around 50-300MB, but each individual batch must be processed sequentially. Each batch loops through the data (multiple times), and every single iteration of a loop depends on state from previous iterations.
* Altough each batch essentially does the same thing, it does not operate in lockstep. Lot's of ifs, different execution paths, differently sized sub-problems, etc.
I'd benefit from having 200 CPU cores and enough RAM to fit the batches, but I can't let individual GPU threads run through batches of 50MB, each.
Answer: quite a lot, if you have even minimal technical understanding (like, enough to use a blocklist or an ad blocker). Are you incapable even of that?
If you have any tech chops you can make it what you want. If.
https://github.com/WebAssembly/proposals/issues/14
It's the ads though, and the calibre of people who create ads...
Not only that, but better performance is what pushes you over the edge, not popups, connection slow downs, auto playing videos or tracking?
The high end renderers usually split up a large image into "buckets" and then perform path tracing on those regions of an image, and combine the results together. If you have more cores, you can increase the number of simultaneous buckets to evaluate.
To put this in perspective, a lot of the recent news around real-time raytracing is when a game engine does 1-4 samples per pixel and then aggressively denoises (in 2D) the result. That's why you often see splotches or errors. On the other hand, high end VFX renders will end up doing hundreds or even thousands of samples per pixel to get a clean, high quality, physically plausible result.
You can run on multiple nodes, but that brings its challenges, see Jepsen.
And with such a huge CPU core counts, a lot of the very difficult work of organizing an algorithm to run well on a GPU can be avoided because the CPU approaches the same benefits with a more logical architectural design.
A closer analogue of x86 cores is what Nvidia calls a "Streaming Multiprocessor" and AMD calls a "Compute Unit". There's generally only up to 64-80 of those on the highest end GPUs too. However the difference is that each of those cores is able to perform 40-64 SIMD operations at once, compared to x86's puny 8 or 16 or so.
Building software, for one. C compilers and python interpreters don't run on a GPU. Lots of stuff doesn't run on a GPU. In fact in practice the only things that run on a GPU are the tiny handful of known subproblems that the industry has collectively decided are "GPU problems".
Like it or not general purpose scalar software is, has always been, and will always remain the standard mechanism by which computing hardware is applied to new problems. Everything else is an optimization around the edges.
Edit: another reply to this top level comment: https://news.ycombinator.com/item?id=23788755
pattern like the one below will execute up to 10 times faster if array is sorted due to data locality and branch prediction
IMHO this is an area ripe for more exploration. If I were working on it, I might look to linking first before compilation, because the basic link task is more similar to what GPUs are good at (advanced stuff such as LTO is a different story, though).
[1]: https://github.com/Co-dfns/Co-dfns
This is incorrect.
The other issue is loading that memory - PCIe 4 is still the transfer time bottleneck between GPU and main memory.
My understanding is that the GPU memory models are different enough that what an OS traditionally calls "virtual memory" couldn't be implemented in the same way.
Well, the problem with that is that no make system can utilise so much core. Even linux kernel, probably the biggest C project in mainstream use, can not consistently load even 16 cores with mostly handwritten makefiles.
P.S.
I do not say that building software is not CPU parallelisable in principles, I'm saying that regular make systems have trouble handing big number of parallel tasks in practice, even in big, well run projects that use handwritten makefiles.
And my day job involves lots of iterations over the Zephyr test suite, which builds and runs hundreds of individual test apps X dozens of platforms. This is almost a pure build-time problem and scales very linearly indeed.
https://stanford.edu/~sadjad/gg-paper.pdf
Actual screencast of it in action compiling FFMPEG:
https://asciinema.org/a/257545
(Perhaps the issue is you are bottlenecking on I/O or some other resource?)
It is true that there are tasks where threading matters, but still require a CPU rather than a GPU. I wonder however if these tasks do need full SSE/AVX etc. Couldn't these extensions be removed of the CPU cores and instead have the necessary work performed by the GPU?
It would be interesting to produce statistics on how much these extensions are used in these scenario. Imagine how much space and complexity could be saved on a CPU die by making stripped down versions. That space could in turn be used for more cores!
I read a little about the Xeon PHI cpus, which iirc, is a multicore CPU with a very small ISA, but I wonder why x86 makers aren't trying to go in that direction: isn't there plenty of dedicated workloads which would happily run on these (eg, web servers), or is this just a (too) simplistic view?
SSE/AVX shares an L1 cache that's damn near instantaneous to access for the CPU core. Total L1 bandwidth is on the scale of TB/s.
PCIe -> GPU takes 1-microsecond to 10-microseconds per access, and operates only at 50GB/s (or 1/20th the speed of L1 bandwidths).
------------
Case in point: Memset is very commonly AVX'd to clear out L1 cache and initialize ~1kb to 32kb of data to 0 as quickly as possible.
There's no way for "memset" to move from CPU to GPU unless you feel like obliterating the entire point of L1, L2, and L3 cache. If you moved a "memset" to GPU, it'd operate only at 15GB/s (the speed of PCIe 3.0 x16 lanes), far, far slower than L1 cache AVX-loads/stores.
SIMD units, like SSE and AVX, are highly "local" and have huge advantages.
The main problem is, afaik, that there is not enough control about where the code will run in these languages. At some point, one will want to describe all the algorithms using a single language, and somehow describe how the workload will have to be distributed across all the processors, or at least that's what I've been thinking about for a while. Once you have that level of control, the need for a versatile CPU is less clear. Note that nowadays people seems happy with hybrid solutions where the code is scattered across several languages (eg, one for the main program and one for the shaders, or for the client side UI), so my position is maybe not very strong.
HW-wise, is it possible that integrated GPUs are the first steps toward an architecture where CPU and GPU have better interconnections (ie, larger communication bandwidth and smaller latency) to the point where SIMD becomes moot? There is also the SWAR approach, where one doesn't rely on intrinsic SIMD instructions, but instead emulate them (though it's probably not very realistic for floating point computation).
Some other ideas:
- Apple has this neural engine in their latest chips, which is basically dedicated HW for neural networks
- In the wild, people are getting more and more interested in building their custom ASICs to cut software's middle-man cost: for them, the CPU solution is not good enough
- Intel recently introduced a new matrix ops extension in their CPUs: maybe at some point they'll introduce full GPU capabilities directly baked in the CPU? I am a little worried about the resulting ISA.
Anyway, I am not an HW engineer, nor a very good software one. I only have a limited view of the difficulties in writing good, CPU or GPU efficient code. My first post was prompted by remembering the first "large scale" multicores CPUs 15 years ago (specifically the Ultrasparc T1) which wheren't SIMD heavy. The direction naturally shifted as progress was made on SIMD to try to compete with GPUs, when it seems to me that originally CPUs and GPUs were complementary.
I tend to support modular solutions, but I don't know how costly that would be in term of efficiency at the HW level.
Xeon PHI on the other hand was the first host of AVX-512 instruction set. Sorry.
With a GPU in the loop, there is an extremely painful hop across PCIe before you can experience any degree of cache coherency. There are protocols, buffering and driver stacks involved. Latency is much higher. You have to plan ahead and think about what memory the GPU needs to mutate vs what memory the CPU needs to mutate. You have to split your application, language, and frameworks across 2 computational domains. How we have tolerated this for so long is beyond me. I think most people who get caught up in it are chasing shiny marketing a lot of the time. This criticism aside, I do think GPUs are still very useful for many applications and users.
With just a CPU, you are looking at L3 as the typical worst case for cache coherency domain. And L3 is getting to be ridiculously large. If your high-performance application's binary image cannot entirely fit within L3 of a modern x86 CPU, you are probably doing something very wrong. The other big advantage with just a CPU is you have 1 cache-coherent memory domain and a single instruction set to answer to. This means you can write 100% of your software in a single language/framework. Well-architected x86 applications can push instructions many times faster than their core clock speed would seem to indicate possible. Using special frameworks that have sympathy with this memory model mean that just a single x86 core can be made to push tens of millions of logical business transactions per second. And then you have 63 other cores to play with. Extremely deep pipelining and OoO execution are what allow for x86 to chew through general purpose computation so easily. Anything with a lot of recursive depth (e.g. raytracing) benefits massively from this style of processor.
I think the next big revolution is developers realizing that they can start porting GPU applications to CPU. Things like raytracing are much better suited for the memory model offered by a pure x86 domain. Sure, a GPU can accelerate some aspects of raytracing, but then you then need to shuttle that information back and forth over a higher latency link, and also fight with 2 completely different technical stacks. Needing to know how to write code for both GPU and CPU is easily the biggest problem with all of this. Especially when you consider how fast GPU APIs move relative to x86. Performance is not the only constraint when dealing with software, especially software as complex as AAA game engines and 3d modeling software. Someone still has to reason with and maintain this stuff over time.
The next UK supercomputer is based around AMD's 64 CPU chips for this reason. One concern is that we may now be limited by the shared memory bandwidth. https://www.nextplatform.com/2019/10/18/amd-cpus-will-power-...
Servers serving multiple independent requests, VMs or Docker containers.
In other words, the same thing people were doing with four 16-core servers, just now it takes up less space :)
Not only that, huge amounts of software could use a lot more concurrency if it was structured differently. Even a web browser (to my knowledge) doesn't split up DOM layout, parsing individual files, decompressing images, decompressing video etc. into separate threads for each file.
Comparing GPU/CUDA/TPU Cores to CPU Cores is like comparing a riding lawnmower to an automobile. A GPU/CUDA/TPU Core is optimized to handle a different type of workload than a CPU.
Just because your riding lawnmower can cut your grass faster than your car, doesn't mean your lawnmower is the best thing for highway driving.
There are other area where big general purpose X64 or ARM cores excel too, like bit twiddling, memory random access, etc., but branching logic is the fundamental one.
Think of two agents that run for awhile and end up in different parts of a virtual world, that require very different code to execute. This might be pretty difficult to parallelize on a GPU.
After collecting a bunch of experience, you update whatever function you’re optimizing. This could be something like a neural network that is the brain of your agents. The update step can happen on a GPU.
But then we need to go collect a lot more experience again, so we’re back in CPU land.
Experience collection often dominates overall compute time in reinforcement learning. Which means you want a lot of CPUs.
I'm a bit behind on GPU architecture, but presumably workloads which are branch-heavy, lack coherent execution within a warp (see dragontamer's comment), and don't make use of floating-point.
GPUs are not the equivalent of manycore CPUs.
It would really be interesting to see if it's feasible to Overclock the best numa node to a degree that gives this a couple of high single thread performance cores comparable to the 3900X or 3950X while retaining stability overall.
Do you mean the best ccx? Or best chiplet? There aren't any numa nodes here, and the cache & memory architecture are already the same as a 3900x.
Per-core & per-ccx overclocking does already exist on ryzen though, should work the same on threadripper.
It would be interesting to turn the 3990/3995 into a does-it-all chip with best in class single-core performance on certain cores and just lots of a tad slower cores in general.
These are the cores that can turbo the highest.
You can attempt to manually OC higher, but that by & large doesn't work without extreme cooling. Typically instead it's about achieving higher all-core frequencies than the built in all-core turbo, typically via higher voltages & power limits (and of course much better-than-stock cooling)
Indeed if you look at single-thread cinebench numbers you'll find the 3960x & 3970x right in the middle of the 3700x, 3800x, and 3900x pack: https://www.guru3d.com/articles_pages/amd_ryzen_threadripper...
They all hit around that same 4.5ghz single core turbo mark without any overclocking. And Zen2 by & large doesn't really go much beyond 4.5ghz anyway, so there's not all that much manual overclocking you can do without going sub-ambient cooling anyway.
The only real downside to Threadripper 3rd gen is the price. It's already a pretty killer jack-of-all-trades CPUs otherwise. It's a very competent gaming CPU without any tweaks at all right out of the box. It doesn't at all have the cons of the 1st & 2nd gen Threadrippers, which were actually NUMA and therefore came with huge gaming downsides.
But if you want to simulate production workloads on your machine say with Apache Spark or Flink, you are in the prime. You couple that CPU with 256+ GB memory, and you can start exploring the bottlenecks in your batch algorithms using as many cores as you want.
[1] https://www.pugetsystems.com/labs/articles/After-Effects-CPU...
Massively parallel operations take a different focus on your MPI/bus/queue architecture than we've ever needed for desktop software. BUT I think what we've already seen is that in general purpose computing there will always be someone ready to consume resources as they are available.
Both feet don't move at the same time, one takes a step and then the other.
Your 4 channels with 2 slots each have the 4 channels daisy chained to 2 slots, but they are just 4 channels. With 8 channels and 2 slots per channel you can have 16 slots, with 32 GB unbuffered (regular) DIMMs you are limited to 512 GB of RAM, for 2 TB you need registered or load reduced RAM.
Nearly every consumer CPU for the last 20 years is dual channel, meaning "2 drive RAID 0". If you've seen things like recommendations to get paired DDR memory sticks, this is why. You only get this RAID-like benefit with multiple RAM sticks (just like RAID 0 of a single drive doesn't do anything). It's also why some consumer products have unexpectedly bad performance for the CPU specs - cheaper laptops may skimp here and just run a single stick of memory instead of 2. Which means half the memory bandwidth.
8 channel then means the CPU can do this up to 8 sticks for 8x the bandwidth instead of the more common 2x.
that is like have 32 cores on a normal 2 channel desktop, throughput would often be a limiter there.
The real reason for adding more memory channels to a general purpose CPU is actually to increase the maximum amount of memory per node. A few years ago, workstation and server platforms often allowed 3 or more DIMMs per channel with performance declining sharply with memory density. Since then both AMD and Intel have limited each channel to 2 DIMMs and increase the number of memory channels from 4 to 6/8 to make up for the loss.
While more memory channels do enable better performance scaling, it has also made the CPU socket more complex and fragile than ever: The TRX40 socket has 4094 pins and literally requires the CPU to be bolted down using a torque wrench to ensure good contact.