Is there even an existing replaceable memory standard that would meet the current needs of Apple's "Unified Memory" architecture? I'm not an expert but I'd suspect probably not. The bus probably looks a lot more like VRAM on GPUs, and I've never seen a GPU with replaceable RAM.
CAMM2 could kinda work, but each module is only 128-bit so I think the furthest you could possibly push it is a 512-bit M Max equivalent with CAMM2 modules north, east, west and south of the SOC. There just isn't room to put eight modules right next to the SOC for a 1024-bit bus like the M Ultra.
Framework said that when they built a Strix Halo machine, AMD assigned an engineer to work with them on seeing if there's a way to get CAMM2 memory working with it, and after a bunch of back and forth it was decided that CAMM2 still made the traces too long to maintain proper signal integrity due to the 256 bit interface.
These machines have a 512 bit interface, so presumably even worse.
Entirely possible. Obviously Apple wouldn't have been interested in letting you upgrade the RAM even if it was doable.
I'd love to have more points of comparison available, but Strix Halo is the most analogous chip to an M-series chip on the market right now from a memory point of view, so it's hard to really know anything.
I very much hope CAMM2 or something else can be made to work with a Strix-like setup in the future, but I have my doubts.
My understanding is that works out due to the lower clock speeds of those RAM modules though right?
It's getting that bandwdith by going very wide on very very very many channels, rather than trying to push a gigantic amount of bandwidth through only a few channels.
Yeah, "channels" are just a roundabout way to say "wider bus" and you can't get too much past 128 GB/s of memory bandwidth without leaning heavily into a very wide bus (i.e. more than the "standard" 128 bit we're used to on consumer x86) regardless who's making the chip. Looking at it from the bus width perspective:
- The AI Max+ 395 is a 256 bit bus ("4 channels") of 8000 MHz instead of 128 bits ("2 channels") of 16000 MHz because you can't practically get past 9000 MHz in a consumer device, even if you solder the RAM, at the moment. Max capacity 128 GB.
- 5th Gen Epyc is a 768 bit bus ("12 channels") of 6000 MHz because that lets you use a standard socketed setup. Max capacity 6 TB.
- M3 Ultra is a 1024 bit bus ("16 channels") of "~6266 MHz" as it's 2x the M3 Max (which is 512 bits wide) and we know the final bandwidth is ~800 GB/s. Max capacity 512 GB.
Note: "Channels" is in quotes because the number of bits per channel isn't actually the same per platform (and DDR5 is actually 2x32 bit channels per DIMM instead of 1x64 per DIMM like older DDR... this kind of shit is why just looking at the actual bit width is easier :p).
So really the frequencies aren't that different even though these are completely different products across completely different segments. The overwhelming factor is bus width (channels) and the rest is more or less design choice noise from the perspective of raw performance.
I thought so too when they launched the M1, but I soon got corrected.
The memory bus is the same as for modules, it's just very short. The higher end SoCs have more memory bandwidth because the bus is wider (i.e. more modules in parallel).
You could blame DDR5 (who thought having a speed negotiation that can go over a minute at boot is a good idea?), but I blame the obsession with thin and the ability to overcharge your customers.
> I've never seen a GPU with replaceable RAM
I still have one :) It's an ISA Trident TVGA 8900 that I personally upgraded from 512k VRAM to one full megabyte!!!
_That_, in itself, wouldn't be that difficult, and there are shared-memory setups that do use modular memory. Where you'd really run into trouble is making it _fast_; this is very, very high bandwidth memory.
It is _not_ on die. It's soldered onto the package.
There's a good reason it's soldered, i.e. the wide memory interface and huge bandwidth mean that the extra trace lengths needed for an upgradable RAM slot would screw up the memory timings too much, but there's no need to make false claims like saying it's on-die.
The longer traces are the problem. They want these modules as physically close as possible to the CPU to make the timings work out and maintain signal integrity.
It's the same reason nobody sells GPUs that have user upgradable non-soldered GDDR VRAM modules.
You know that memory can be "easily" de-soldered and soldered at home?
The issue is availability of chips and most likely you have to know which components to change so the new memory is recognised. For instance that could be changing a resistor to different value or bridging certain pads.
This viewpoint is interesting. It is not exactly inaccurate, but it does appear to be missing a point. Soldering in itself is a valuable and useful skill, but I can't say you can just get in and start de-soldering willy-nilly as opposed to opening a box and upgrading ram by plopping stuff in a designated spot.
Do you know that "plopping stuff in a designated spot" can also be out of reach to some people? I know plenty who would give their computer to a tech do to the upgrade for them even if they are shown in person how to do all the steps. Soldering is just one step (albeit fairly big) above that.
But the fact this can be done at home with fairly inexpensive tools, means tech person with reasonable skill could do it, so such upgrade could be accessible in computer/phone repair shop if parts were available to do so.
Soldering is not a barrier - what I am trying to say.
> Is anyone other than a vanishingly small number of hard core hobbiests going to upgrade from an M4 to an M4 Ultra?
I expect that the 2 biggest buyers of M4 Ultra will be people who want to run LLMs locally, and people who want the highest performance machine they can get (professionals), but are wedded to mac-only software.
It is a bit misleading to do that, but in fairness to Apple, almost nobody is upgrading to this from an M4 Mac, so those are probably more useful comparisons.
Yeah VRAM option is good (if it performs well), just sad we'd have to drop 10K to access it tied to a prev gen M3 when they'll likely have M5 by the end of the year.
A server with 512GB of high-bandwidth GPU addressable RAM in a server is probably a six figure expenditure. If memory is your constrain, this is absolutely the server for you.
(sorry, should have specified that the NPU and GPU cores need to access that ram and have reasonable performance). I specified it above, but people didn't read that :-)
There isn't anything particularly high-bandwidth about Apple's DDR5 implementation, either. They just have a lot of channels, which is why I compared it to a 24-channel EPYC system. I agree that their integrated GPU architecture hits a unique design point that you don't get from nvidia, who prefer to ship smaller amounts of very different kinds of memory. Apple's architecture may be more suited to some workloads but it hasn't exactly grabbed the machine learning market.
M3 Ultra has 819GB/s, and a single epyc cpu with 12 channels has 460GB/s. As far as I know, llama.cpp and friends don’t scale across multiple sockets so you can’t use a dual socket Turin system to match the M3 Ultra.
Also, 32GB DDR5 RDIMMS are ~200, so that’s 5K for 24 right there. Then you need 2x CPUs, at ~1K for the cheapest, and you need 2, and then a motherboard that’s another 1K. So for 8K (more, given you need a case, power supply, and cooling!), you get a system with about half the memory bandwidth, much higher power consumption, and very large.
Partial correction, an Epyc CPU with 12 channels has 576 GB/s, i.e. DDR5-6000 x 768 bits. That is 70% of the Apple memory bandwidth, but with possibly much more memory (768 GB in your example).
You do not need 2 CPUs. If however you use 2 CPUs, then the memory bandwidth doubles, to 1152 GB/s, exceeding Apple by 40% in memory bandwidth. The cost of the memory would be about the same, by using 16 GB modules, but the MB would be more expensive and the second CPU would add to the price.
Perhaps this is incorrect now, but I also know with 2x 4090s you don’t get higher tokens per second than 1x 4090 with llama.cpp, just more memory capacity.
(All if this only applies to llama.cpp, I have no experience with other software and how memory bandwidth may scale across sockets)
The memory bandwidth does double, but in order to exploit it the program must be written and executed with care in the memory placement, taking into account NUMA, so that the cores should access mostly memory attached to the closest memory controller and not memory attached to the other socket.
With a badly organized program, the performance can be limited not by the memory bandwidth, which is always exactly double for a dual-socket system, but by the transfers on the inter-socket links.
Moreover, your link is about older Intel Xeon Sapphire Rapids CPUs, with inferior memory interfaces and with more quirks in memory optimization.
about the scaling of llama.cpp and DeepSeek on some dual-socket AMD systems.
While it was rather tricky, after many experiments they have obtained an almost double speed on two sockets, especially on AMD Turin.
However, if you look at the actual benchmark data, that must be much lower than what is really possible, because their test AMD Turin system (named there P1) had only two thirds of the memory channels populated, i.e. performance limited by memory bandwidth could be increased by 50%, and they had 16-core CPUs, so performance limited by computation could be increased around 10 times.
CPUs do not have enough compute typically. You'll be compute bottlenecked before bandwidth if the model is large enough.
Time to first token, context length, and tokens/s are significantly inferior on CPUs when dealing with larger models even if the bandwidth is the same.
One big server CPUs can have a computational capability similar to a mid-range desktop NVIDIA GPU.
When used for ML/AI applications, a consumer GPU has much better performance per dollar.
Nevertheless, when it is desired to use much more memory than in a desktop GPU, a dual-socket server can have higher memory bandwidth than most desktop GPUs, i.e. more than an RTX 4090, and a computational capability that for FP32 could exceed an RTX 4080, but it would be slower for low-precision data where the NVIDIA tensor cores can be used.
True, but I have compared the FP32 used in graphics computations because for that the throughput information is easily available.
Both CPUs (with the BF16 instructions and with the VNNI instructions for INT8 inference) and the GPUs have a higher throughput for lower precision data types than for FP32, but the exact acceleration factors are hard to find.
The Intel server CPUs have the advantage vs. AMD that they also have the AMX matrix instructions, which are intended to compete for inference applications with the NVIDIA tensor cores, but the Intel CPUs are much more expensive for a number of cores big enough to be competitive with GPUs.
The bandwidth difference likely doesn't make a difference though. Benchmarks of Apple Silicon show that the compute bottlenecks far before running out of bandwidth, even when fully loading all CPU cores, the GPU, etc.
CUDA has had managed memory for a long time now. You absolutely can address the entire host memory from your GPU. It will fetch it, if it's needed. Not fast, but addressable.
Ah seems like I remembered the CPU price for a higher tier CPU which can cost the 6k on their own.
Thinking about it you can get a decent 256gb on consumer platforms now too, but the speed will be a bit crap and would need to make sure the platform ully supports ECC UDIMMs
Are the benchmarks worse? Running LLMs in system memory is rather painful. I am having a hard time finding benchmarks for running large models using system memory. Can you point me to some benchmarks you’re referring to?
In a dual-socket EPYC system, the memory bandwidth is higher than in this Apple system by 40% (i.e. 1152 GB/s), and the memory capacity can be many times higher.
Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000.
Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.
Even for computationally-limited programs, the difference between server CPUs and consumer GPUs is not great. One Epyc CPU may have about the same number of FP32 execution units as an RTX 4070, while running at a higher clock frequency (but it lacks the tensor units of an NVIDIA GPU, which can greatly accelerate the execution, where applicable).
Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.
Source on this? CPUs would be very compute constrained.
According to Apple, the GPU of M3 Ultra has 80 graphics cores, which should mean 10240 FP32 execution units, the same like an NVIDIA RTX 4080 Super.
However Apple does not say anything about the GPU clock frequency, which I assume that it is significantly less than that of NVIDIA.
In comparison, a dual-socket AMD Turin can have up to 12288 FP32 execution units, i.e. 20% more than an Apple GPU.
Moreover, the clock frequency of the AMD CPU must be much higher than that of the Apple GPU, so it is likely that the AMD system may be at least twice faster for computing some graphic application than the Apple M3 Ultra GPU.
I do not know what facilities exist in the Apple GPU for accelerating the computations with low-precision data types, like the tensor cores of NVIDIA GPUs.
While for graphic applications big server CPUs are actually less compute constrained than almost all consumer GPUs (except RTX 4090/5090), the GPUs can be faster for ML/AI applications that use low-precision data types, but this is not at all certain for the Apple GPU.
Even if the Apple GPU happens to be faster for some low-precision data type, the difference cannot be great.
However a server that would beat the Apple M3 Ultra GPU computationally would cost much more than $10k, because it would need CPUs with many cores.
If the goal is only to have a system with 50% more memory and 40% more memory bandwidth than the Apple system, that can be done at a $10k price.
While such a system would become compute constrained more often than an Apple GPU, it would still beat it every time when the memory would be the bottleneck.
I have just compared the FP32 computational capabilities, i.e. what is used for graphics, between the Apple M3 Ultra GPU and AMD server CPUs, because these numbers are easily available and they demonstrate the size relationships between them.
Both GPUs and server CPUs have greater throughputs for lower precision data (CPUs have instructions for BF16 and INT8 inference), but the exact acceleration factors are hard to find and it is more difficult to estimate the speeds without access to such systems for running benchmarks.
Anecdotal but it seems like the big EPYC rigs are getting very low tokens per second, and not even consistent. They are strained, as opposed to e.g. M3 Ultra that can likely sustain 40-50 tokens/s based on previous stats.
I'd like to see some proper benchmarking on this though, but it looks like the Apple systems might just be extremely good value if you want to run the large DeepSeek model.
If you're going to overthrow your entire AI workflow to use a different API anyway, surely the AMD Instinct accelerator cards make more sense. They're expensive, but also a lot faster, and you don't need to deal with making your code work on macOS.
It is exactly the opposite. Every computer architecture in production addresses memory in the powers of two.
SI has no business in memory size nomenclature as it is not derived from fundamental physical units. The whole klownbyte change was pushed through by hard drive marketers in 1990s.
> Every computer architecture in production addresses memory in the powers of two.
What does it mean to "address memory in powers of two" ? There are certainly machines with non-power-of-two memory quantities; 96 GiB is common for example.
> The whole klownbyte change was pushed through by hard drive marketers in 1990s.
The metric prefixes based on powers of 10 have been around since the 1790s.
> What does it mean to "address memory in powers of two" ? There are certainly machines with non-power-of-two memory quantities; 96 GiB is common for example.
I challenge you to show me any SKU from any memory manufacturer that has a power of 10 capacity. Or a CPU whose address space is a power of 10. This is an unavoidable artefact of using a binary address bus.
> The metric prefixes based on powers of 10 have been around since the 1790s.
*bibytes are a practical joke played on computer scientists by the salespeople to make it sound like we’re drunk. “Tell us more about your mebibytes, Fred elbows colleague, listen to this”.
If Donald Knuth and Gordon Bell say we use base-2 for RAM, that’s good enough for me.
It's more complicated than that. Data storage sizes are not connected to fundamental physical units, but data transfer rates are. Things get annoying when a 1 MB/s connection cannot transfer a megabyte in a second.
Line discipline rarely has sequences of bytes without any service information (parity, delimiters, preambles etc). So I don't see it as a practical issue.
Reportedly Apple is using its own silicon in data centers to run “Apple Intelligence” and other things like machine translation in safari. I suspect that the initial supply was sent to Apple’s datacenters.
I'm guessing it's more because "Ultra" versions, which "fuse" multiple chips take significant additional engineering work. So we might expect an ultra M4 next year, possibly after non-ultra M5s are released.
No, with limited VRAM you could offload the model partially or split across CPU and GPU. And since CPU has swap, you could run the absolute largest model. It’s just really really slow.
The difference between Deepseek-r1:70b (edit: actually 32b) running on an M4 Pro (48 GB unified RAM, 14 CPU cores, 20 GPU cores) and on an AMD box (64 GB DDR4, 16 core 5950X, RTX 3080 with 10 GB of RAM) is more than a factor of 2.
The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing.
The M4's prompt parsing took significantly longer, but token generation was significantly faster.
Having the memory to the cores that matter makes a big difference.
You're adding detail that's not relevant to anything I said. I was saying this statement:
> VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis.
Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.
The criteria for purchase for anybody trying to use it is "run slowly but acceptably" vs. "run so slow as to be unusable".
My memory is wrong, it was the 32b. I'm running the 70b against a similar prompt and the 5950X is probably going to take over an hour for what the M4 managed in about 7 minutes.
edit: an hour later and the 5950 isn't even done thinking yet. Token generation is generously around 1 token/s.
edit edit: final statistics. M4 Pro managing 4 tokens/s prompt eval, 4.8 tokens/s token generation. 5950X managing 150 tokens/s prompt eval, and 1 token/s generation.
Perceptually I can live with the M4's performance. It's a set prompt, do something else, come back sort of thing. The 5950/RTX3080's is too slow to be even remotely usable with the 70b parameter model.
A beefy GPU which can't hold models in VRAM is of very limited use. You'll see 16 GB of VRAM on gamer Nvidia cards, the RTX 5090 being an exception with 32 GB VRAM. The professional cards have around 96 GB of VRAM.
The thing with these Apple chips is that they have unified memory, where CPU and GPU use the same memory chips, which means that you can load huge models into RAM (no longer VRAM, because that doesn't exist on those devices). And while Apple's integrated GPU isn't as powerful as an Nvidia GPU, it is powerful enough for non-professional workloads and has the huge benefit of access to lots of memory.
LLMs are primarily "memory-bound" rather than "compute-bound" during normal use.
The model weights (billions of parameters) must be loaded into memory before you can use them.
Think of it like this: Even with a very fast chef (powerful CPU/GPU), if your kitchen counter (VRAM) is too small to lay out all the ingredients, cooking becomes inefficient or impossible.
Processing power still matters for speed once everything fits in memory, but it's secondary to having enough VRAM in the first place.
It's pretty close. A 3090 or 4090 has about 1TB/s of memory bandwidth, while the top Apple chips have a bit over 800GB/s. Where you'll see a big difference is in prompt processing. Without the compute power of a pile of GPUs, chewing through long prompts, code, documents etc is going to be slower.
I chose the 3090/4090 because it seems to me that this machine could be a replacement for a workstation or a homelab rig at a similar price point, but not a $100-250k server in a datacenter. It's not really surprising or interesting that the datacenter GPUs are superior.
FWIW I went the route of "bunch of GPUs in a desktop case" because I felt having the compute oomph was worth it.
sure on consumer GPUs but that is not what is constraining the model inference in most actual industry setups. technically even then, you are CPU-GPU memory bandwidth bound more than just GPU memory, although that is maybe splitting hairs
I was able to run and use the DeepSeek distilled 24gb on an M1 Max with 64gb of ram. It wasn't speedy, but it was usable. I imagine the M3/4s are much faster, especially on smaller, more specific models.
Previous model of M2 Ultra had max memory of 192GB. Or 128GB for Pro and some other M3 model, which I think is plenty for even 99.9% of professional task.
They now bump it to 512GB. Along with insane price tag of $9499 for 512GB Mac Studio. I am pretty sure this is some AI Gold rush.
Its not though. For consumer computers somewhere 1k-4k there's nothing better. But for the price of 512gb of RAM you could buy that + a crazy CPU + 2x 5090s by building your own. The market fit is "needs power; needs/wants macOS; has no budget" which is so incredibly niche. But in terms of raw compute output there's absolutely no chance this is providing bang for buck
2x 5090s would only give you 64GB of memory to work with re:LLM workloads, which is what people are talking about in this thread. The 512GB of system RAM you’re referring to would not be useful in this context. Apple’s unified memory architecture is the part you’re missing.
I think the answer is because they can ( there is a market for it ). The benefit to a crazy person like me that with this addition, I might be able to grab 128gb version at a lower price.
With all things semiconductor, low volume = higher cost (and margin).
The people who need the crazy resource can tie it to some need that costs more. You’d spend like $10k running a machine with similar capabilities in AWS in a month.
You have a point; technically they aren't impossible to run if you have enough system RAM (or hell, SSD/HDD space for that mater). But in practice neither running on the CPU, nor on the GPU by constantly paging data in and out of VRAM, is a very attractive option (~10x slowdown at least).
So the only reason the mac is faster is because the RAM is accessible by its GPU, right? Not because the RAM is faster than regular RAM, because AFAIK it isn't far off from workstation RAM speeds.
Every single AI shop on the planet is trying to figure out if there is enough compute or not to make this a reasonable AI path. If the answer is yes, that 10k is a absolute bargain.
Is this actually true? Were people doing this with the 192gb of the M2 Ultra?
I'm curious to learn how AI shops are actually doing model development if anyone has experience there. What I imagined was: Its all in the "cloud" (or, their own infra), and the local machine doesn't matter. If it did matter, the nvidia software stack is too important, especially given that a 512gb M3 Ultra config costs $10,000+.
True. But with Project Digits supposedly around the corner, which supposedly costs $3,000 and supports ConnectX and runs Blackwell; what's the over-under on just buying two of those at about half the price of one maxed M3 Ultra Mac Studio?
Its half that of a max spec Mac Studio, but also half the price and eight times faster memory speed. Realistically which open source LLMs does 512gb over 256gb of memory unlock? My understanding is that the true bleeding edge ones like R1 won't even handle 512gb well, especially with the anemic memory speed.
We really should see what happens when Project Digits is finally released. Also, I would love in NVIDIA decided to get in the CPU/GPU + unified memory space.
I can't imagine the M3 Ultra doing well on a model that loads into ~500G, but they should be a blast on 70b models (well, twice as fast as my M3 Max at least) or even a heavily quantized 400b model.
I agree project digits looks to be the better all-around option for AI researchers, but I still think the Mac is better for people building products with AI
Re memory speed, digits will be at 273GB/s while the Mac Studio is at 819GB/s
Not to mention the Mac has 6 120GB/s thunderbolt 5 ports and can easily be used for video editing, app development, etc.
No AI shop is buying macs to use as a server. Apple should really release some server macOS distribution, maybe even rackable M-series chips. I believe they have one internally.
The higher end NVidia workstation boxes won’t run well on normal 20amp plugs. So you need to move them to a computer room (whoops, ripped those out already) or spend months getting dedicated circuits run to office spaces.
Didn't really think about this before, but that seems to be mainly an issue in Northern / Central America and Japan. In Germany, for example, typical household plugs are 16A at 230V.
To clarify, the circuit is almost always 20A with 15A being used for lighting. However, the outlet itself is almost always 15A because you put multiple outlets on a single circuit. You are going to see very few 20A in outlets (which have a T shaped prong) in residential.
While technically true, the NEMA 5-15R receptacles are rated for use on 20A circuits, and circuits for receptacles are almost always 20A circuits, in modern construction at least. Older builds may not be, of course.
That said, if your load is going to be a continuous load drawing 80% of the rated amperage, it really should be a NEMA 5-20 plug and receptacle, the one where one of the prongs is horizontal instead of vertical. Swapping out the receptacle for one that accepts a NEMA 5-20P plug is like $5.
If you are going to actually run such a load on a 20A circuit with multiple receptacles, you will want to make sure you're not plugging anything substantial into any of the other receptacles on that circuit. A couple LED lights are fine. A microwave or kettle, not so much.
True. But an AI shop doesn't care about that. They get more performance for the money by going for multiple Nvidia GPUs. I have 512 GB ram on my PC too with 8 memory channels, but it's not like it's usable for AI workloads. It's nice to have large amounts of RAM, but increasing the batch size during training isn't going to help when compute is the bottleneck.
No, because there is no CUDA. We have fast and cheap alternatives to NVIDIA, but they do not have CUDA. This is why NVIDIA has 90% margins on its hardware.
Can you do absolutely everything? No. But most models will run or retrain fine now without CUDA. This premise keeps getting recycled from the past, even as that past has grown ever more distant.
CUDA is incredibly important still. It's still an incredible amount of work to get packages working on multiple GPU paradigms, and by default everyone still starts with CUDA.
The example I always give is FFT libraries - if you compare cuFFT to rocFFT. rocFFT only just released support for distributed transforms in December 2024, something you've been able to do since CUDA Toolkit v8.0, released in 2017. It's like this across the whole AMD toolkit, they're so far behind CUDA it's kind of laughable.
CUDA is becoming more critical, not less, every day. Software developed around CUDA is vastly outpacing what other companies produce. And saving a few millions when creating new models doesn't matter; NVIDIA is pretty efficient at scale.
I don't know if you've heard, but NVIDIA is about to add a monthly payment for additional CUDA features and I'm almost certain that many big companies will be happy to pay for them.
> But most models will run or retrain fine now without CUDA.
This is correct for some small startups, not big companies.
The question will be how it will perform. I suspect Deepseek, Llama405B demonstrated the need for larger memory. Right now folks could build an epyc system with that much ram or more to run Deepseek at about 6 tokens/sec for a fraction of that cost. However not everyone is a tinker, so there's a market for this for those that don't want to be bothered. You say "AI Gold rush" like it's a bad thing, it's not.
Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available.
That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use. They really need a 2U rack form for this though.
This hardware is really being held back by the operating system at this point.
>I had read somewhere that the interposer that enabled this for the M1 chips where not available.
With all my love and respect for "Apple rumors" writers; this was always "I read five blogposts about CPU design and now I'm an expert!" territory.
The speculation was based on the M3 Maxes die shots not having the interposer visible, which... implies basically nothing whether that _could have_ been supported in an M3 Ultra configuration; as evidenced by the announcement today.
Baseline M4 doesn't have Thunderbolt 5 either; only the Pro/Max variants do.
The press-release even calls TB5 out:
>Each Thunderbolt 5 port is supported by its own custom-designed controller directly on the chip.
Given that they're doing the same on A-series chips (A18 Pro with 10Gbps USB-C; A18 with USB 2.0); I imagine it's just relatively simple to swap the I/O blocks around and they're doing this for cost and/or product segmentation reasons.
>Which, at this point, why not just use M4 as a base?
I imagine that making those chips is quite a bit more involved than just taking the files for M3 Max, and copy-pasting them twice into a new project.
I imagine it just takes more time to design/verify/produce them; especially given they're not selling very many of them, so they're probably not super-high-priority projects.
TB 5 seems like the sort of thing you could 'slap on' to a beefy enough chip.
Or the sort of thing you put onto a successor when you had your fingers crossed that the spec and hardware would finalize in time for your product launch but the fucking committee went into paralysis again at the last moment and now your product has to ship 4 months before you can put TB 5 hardware on shelves. So you put your TB4 circuitry on a chip that has the bandwidth to handle TB5 and you wait for the sequel.
The world is full of features that didn't make the cutoff for launch date. I believe there's one or two of these publicly known in Apple's history, but it's an old tale.
This is more about "average" end user software, not the type of software that would be running on a machine like this. Yes their applications fell off, but if you're paying for 512gb of RAM apple notes being slow isn't the bottleneck
It'd be an interesting flame war in the comments, if nothing else, go for it! I'm happy to give plenty of concrete evidence why Linux is more suitable for professionals than macOS is in 2025 :)
Omg I despise the fact that there's n competing GUI standards on linux, zero visual consistency.
I love diversity in websites, and apps for that matter, but this isn't diversity, it is the uncanny valley between bespoke graphic design and homogeneity.
Say what you want about SwiftUI, but it makes consistent, good looking apps. Unless something has changed, GTK is a usability disaster.
And that's before I get into how much both X11 and wayland suck equally.
There's so much I miss about Linux, but there's so much I don't
> Lack of focus on quality of software affects all types of workloads, not just consumer-oriented or professional-oriented in isolation.
The apps are developed by different teams. MacOS apps are containerized. Saying macOS's performance is hindered by Notes.app is like saying that Windows is hindered by Paint.exe. Notes.app is just a default[0]
[0]: though, I dislike saying this because I always feel like I need to mention that even Notes links against a hilarious amount of private APIs that could easily be exposed to other developers but... aren't.
No native docker support, no headless management options (enterprise strength), Limited QoS management, lack of robust python support (out of the box), interactive user focused security model.
I feel you on a lot of this! But out of the box Python support? Does anybody actually want that? It’s pretty darn quick & straightforward to get a Python environment up & running on MacOS. Maybe I’m misunderstanding what you mean here.
>it’ll run reliably on other people’s machines a few years from now
That's optimistic. What if the system Python gets upgraded? For some reason, Python libraries tend to be super picky about the Python versions they support (not just Python 2 vs 3).
That's using a Linux VM. The idea people are asking about is native process isolation. Yes you'd have to rebuild Docker containers based on some sort of (small) macOS base layer and Homebrew/Macports, but hey. Being able to even run nodejs or php with its thousands of files natively would be a gamechanger in performance.
Also, it were possible to containerize macos, or even do an unintended vm installation, then it’d be possible for apple to automatically regression test their stuff.
There is no such thing. Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers would you use? And how would you prevent "project collision" (where one Python project bumps into another one and one just stops working)? Example: SSL library differences across projects is a notorious culprit.
Python is garbage and I don't understand why people put up with this crap unless you seriously only run ONE SINGLE Python project at a time and do not care what else silently breaks. Having to run every Python app in its own Docker image (which is the only real solution to this, if you don't want to learn Nix, which you really should, because it is better thanks to determinism... but entails its own set of issues) is not a reasonable compromise.
> at least be able to use Python, but in a very controlled, not-insane way
Thats funny, about 10 years ago I started my career in a startup that had Python business logic running under Erlang (via custom connector) which handled supervision and task distribution, and it looked insane for me at the time.
Even today I think it can be useful but is very hard to maintain, and containers are a good enough way to handle python.
> containers are a good enough way to handle python
I disagree. My take on that is that they are an ugly enough way to handle Python. And, among other problems, don't permit you to easily mess with the code (one of many reasons why this is ugly). Need access to something stateful from the container app? That's another PITA.
This is incoherent to me. Your complaints are about packaging, but the elixir wrapper doesn't deal with that in any way -- it just wraps UV, which you could use without elixir.
What am I missing?
Also, typically when people say things like
> Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers
It means they have been trapped in a cycle of thinking "just one more tool will surely solve my problem", instead of realising that the tools _are_ the problem, and if you just use the official methods (virtualenv and pip from a stock python install), things mostly just work.
I agree. Python certainly had its speedbumps, but it's utterly manageable today and has been for years and years. It seems like people get hung up on there not being 1 official way to do things, but I think that's been great, too: the competition gave us nice things like Poetry and UV. The odds are slim that a Rust tool would've been accepted as the official Python.org-supplied system, but now we have it.
There are reasons to want something more featureful than plain pip. Even without them, pip+virtualenv has been completely usable for, what, 15 years now?
I've seen issues with pip + virtualenv (ssl lib issues, IIRC). I've always used those at minimum and have still run into problems. (I like to download random projects to try them out.) I've also seen issues with python projects silently becoming stale and not working, or python projects walking over other python projects because pip + virtualenv does NOT encompass all Python deps to the metal. This also doesn't mean you can have 2 commandline Python apps available in the same commandline environment, because PATH would have to prefer one or the other at some point.
Here's a question- If you don't touch a project in 1 year, do you expect it to still work, or not? If your answer is the latter, then we simply won't see eye-to-eye on this.
that's not good enough. If I'm in the business of writing Python code, I (ideally) don't want to _also_ be in the business of working around Python design deficiencies. Either solve the problem definitively, or do not try to solve the problem at all, because the middle road just leads to endless headaches for people WHILE ALSO disincentivizing a better solution.
Node has better dependency management than Python- And that's really saying something.
The thing is, most people who are writing python code are not in the business of writing python code. They're students, scientists, people with the word "business" or "analyst" in their title. They have bigger fish to fry than learning a different language ecosystem.
It took 30 years to get them to switch from excel to python. I think it's unrealistic to expect that they're going to switch from python any time soon. So for better or worse, these are problems that we have to solve.
I don't see why it should be so binary. I said it "mostly" just works because there are no packaging systems which do exactly what you want 100% of the time.
I've had plenty of problems with node, for example. You mentioned nix, which is much better, but also comes with tons of hard trade-offs.
If a packaging tool doesn't do what i wanted, but I can understand why, and ultimately the tool is not to blame, that's fine by me. The issues I can think of fit reasonably well within this scope:
- requirement version conflicts: packages are updated by different developers, so sometimes their requirements might not be compatible with each other. That's not pip's fault, and it tells you what the problem is so you can resolve it.
- code that's not compatible with updated packages: this is mainly down to requirement versions which are specified too loosely, and not the fault of pip. If you want to lock dependencies to exact versions (like node does by default) you can do this too (with requirements.txt). It's a bit harsh to blame pip for not doing this for you, it's like blaming npm for not committing your package.lock. It would be better if your average python developer was better at this.
- native library issues: some packages depend on you having specific libraries (and versions thereof) installed, and there's not much that pip can do about that. This is where your "ssl issues" come from. This is pretty common in python because it's used so much as "glue" between native libraries -- all the most fun packages are wrappers around native code. This has got a lot better in the past few years with manylinux wheels (which include native libraries). These require a lot of non-python-specific work to build, so i don't blame pip where they don't exist.
It's not perfect, but it's not a big enough deal to rant about or reject entirely if you would otherwise get a lot of value out of the ecosystem.
Honest question: why do you want this in MacOS? Do you understand what docker does? (it's fundamentally a linux technology, unless you are asking for user namespaces and chroot w/o SIP on MacOS, but that doesn't make sense since the app sandbox exists).
MacOS doesn't have the fundamental ecosystem problems that beget the need for docker.
If the answer is "I want to run docker containers because I have them" then use orbstack or run linux through the virtualization framework (not Docker desktop). It's remarkably fast.
Docker Desktop now offers an option to use the virtualization framework, and works pretty well. But you're still constantly running a VM because "docker is how devs work now right?". I agree with your comment.
> MacOS doesn't have the fundamental ecosystem problems that beget the need for docker.
Anyone wanting to run and manage their own suite of Macs to build multiple massive iOS and Mac apps at scale, for dozens or hundreds or thousands of developers deploying their changes.
xcodebuild is by far the most obvious "needs native for max perf" but there are a few other tools that require macOS. But obviously if you have multiple repos and apps, you might require many different versions of the same tools to build everything.
Sounds like a perfect use case for native containers.
I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive.
I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side.
My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact the ssh server is only available on good days and only after Remote Desktop logged in at least once. Launchd makes you wanna crave systemd. etc, etc.
So, about docker. I would absolutely love to run my app in a containerized environment on a Mac in order to not touch the main OS.
Funny, I ran a bunch of Mac minis in colo for over a decade with no problems. Maybe you have a config problem?
Of course, I had a LOM/KVM and redundant networking etc. They were substantially more reliable than the Dell equipment that I used in my day job for sure.
Software-wise it's much different to an expected behavior. For example, macOS won't let you in over SSH until you log in via Remote Desktop. You'll get "connection closed" immediately.
Or sometimes it will.
And that depends not on the count of connection attempts or anything you can do locally but rather on the boot process somehow. Sometimes it boots in a way that permits ssh, sometimes not. The same computer, the same OS.
Then after you login on screen sharing and log out, macOS will let you in over ssh. For a few days. And then again will force you to login via GUI. Or maybe not. I have no idea what makes it.
I have trouble reading macOS logs or understanding it. It spews a few log messages per second even idle. If you grep ssh these messages contain zero actionable data, like "unsuccessful attempt" or similar.
Another complaint is that launchd reports the same "I/O error" on absolutely all error situations, from syntax error in plist to corrupt binary. Makes development and debugging of launchagents very fun.
What would a containerization environment on MacOS give you that you don't already have? Like concretely - what does containerization mean in the context of a MacOS user space?
In Linux, it means something very specific: a user/mount/pid/network namespace, overlayfs to provide a rootfs, chroot to pivot to the new root to do your work, and port forwarding between the host/guest systems.
On MacOS I don't know what containerization means short of virtualization. But you have virtualization on MacOS already, so why not use that?
On macOS probably I'd like chroot and pid/mount namespaces. I'd like to install OS and dependencies in a container and run my application from there so that it does not interfere with host OS. My app is GPU heavy and has lots of dependencies (OpenCV, LAPACK, armadillo, lots and lots) and I'd like to not pollute the host OS with it.
Also I want to run the latest OS with all security patches on the host while having a stable and known macOS version in a container given how developer-hostile Apple is.
What you want is virtualization, not containerization. And you have this already. Since MacOS doesn't have a stable syscall interface, decoupling the host/guest in a mount namespace and chroot would lead to horrible breakages when the system libraries of your container are out of date with your host OS. So you would have to share the host OS and a big portion of the userspace to begin with.
Or you can package your app as a .app and not worry about it, there's no "pollution" when everything is bundled.
Yeah, seems like on macOS that level of isolation is achievable solely with virtualization unlike in Linux. We were talking about missing things in macOS and containerization is one of them.
I torrent things from two different hosts on my gigabit network. The macos stack literally cannot handle the full bandwidth I have. It fails and the machine needs to be rebooted to fix it. It’s not pretty on the way into this state, either. Other remote connections to the computer are unreliable. On Linux, running the same app in a docker container works perfectly. Transmission is the app.
I haven't had any issue running BiglyBT on my M1 MacBook, granted I don't run it all day every day but everything runs plenty fast for my needs (25-30 MB/s for well-seeded torrents).
I went to Transmission years and years ago because it's just simple. It has all the options if you need them, but no HUUUGE interface with RSS feeds, 10001 stats about your download, categories, tags, etc etc etc.
Transmission is just a small, floating window with your downloads. Click for more. It fits in the macOS vibe. But I'm a person that fully adopted the original macOS "way of working" - kicked the full-screen habit I had in windows and never felt better.
Can I ask, why would you go FROM Transmission to qBittorrent?
>why would you go FROM Transmission to qBittorrent?
In my case: some torrents wouldn't find known-good seeds in Transmission but worked fine in qBittorrent; there's reasonable (but not perfect) support for libtorrent 2.0 in qBittorrent; my download speeds and overall responsiveness is anecdotally better in qBittorrent, and; I make use of some of the nitty gritty settings in qBittorrent.
Well there's a list of good reasons! Thanks for answering. I haven't had any problems with finding seeds, and no need for libtorrent but now I know how to fix that when I do encounter those situations.
The Linux version, in a container no less, handles the entire gigabit bandwidth.
And let's be clear, it wasn't the app that had problems, the Apple Remote Desktop connection to the machine failed when the speeds got above 40MB/s and the network interface stopped working around 80MB/s.
I think Transmission works perfectly fine. I've been using it for 10+ years with no issues at all on Linux.
I forgot to mention this is a Mac mini/Intel (2018).
I get nearly 10Gbps from my NAS to my Mac Studio. It absolutely can handle that bandwidth. It may not handle that specific client well for unrelated reasons.
we’ve heard that claim for the past three years, but every effort by them points to the opposite. don’t get me wrong, I would love for Apple Intelligence to be smart enough on my iPhone and on my Mac, but honestly, the current version is a complete disappointment.
if that were the case, then it would definitely help Apple Intelligence if the iPhone and Mac had higher amounts of RAM, but the base MacBook Pro announced by Apple a while ago had 8 GB of RAM and even the pro versions of the iPhone have 8GB whereas 12, 16, or even higher RAM is very common in android devices which helps users run relatively large language models on their devices
Apple have been putting ML models running on their own silicon into production for far longer than any of their competitors. They publish some of the most innovative ML research
They also own distribution to the wealthiest and most influential people in the world
> Apple’s custom-built UltraFusion packaging technology uses an embedded silicon interposer that connects two M3 Max dies across more than 10,000 signals, providing over 2.5TB/s of low-latency interprocessor bandwidth, and making M3 Ultra appear as a single chip to software.
The comment was that the press had reported that the interposer wasn't available. This obviously uses some form of interposer, so the question is if the press missed it, or Apple has something new.
It sounds like they're using TSMC's new LSI (Local Si Interconnect) technology, which is their version of Intel's EMIB. It's essentially small islands of silicon, just around the inter-chip connections, embedded within the organic substrate. This gives the advantages of silicon interconnect, without the cost and size restrictions of a silicon interposer. It would not be visible from just looking at the package.
I've been buying and using MBP for 6 or 7 years now, and just assumed I could run Linux on one if I wanted to. I just spent a couple of days trying to get a 2018 MBP working with Linux and found out [edit to clarify] that my other ARM MBP basically won't work.
I just want a break from MacOS, I'll be buying a Thinkpad and will probably never come back. This isn't my moaning, I understand it's their market, but if their hardware supported Linux (especially dual booting) or Docker native, I'd probably be buying Apple for the next decade and now I just won't be.
Getting Linux running wasn't difficult. But Mint lost audio (everything else worked), the specialised Mint kernel lost both audio and wifi, and Arch lost both wifi and the onboard keyboard.
I'm sure with tinkering I could eventually get it working, but I'm well past the point of wanting to tinker with hardware and drivers to get Linux working.
Because of the T2 chip it's actually pretty annoying. Mainline kernels I think are still missing keyboard and trackpad support for those models. Plus a host of other issues.
I could have written it clearer. I have both, Intel was the first attempt and when I was struggling to get it up without losing one of wifi, audio and onboard keyboard and read that ARM was worse I gave up. Even the best combination I had (no audio but everything else working) would kill bluetooth after a while if wifi was connected to 2.6. I don't like their hardware enough to fight with it.
Loved my M1 mini, loved my M2 air. I've moved on to 2024 HP Elitebook with an AMD R7 8840U, 1TB replaceable NVME, 32gb of socketed DDR5. 14in laptop with a serviceable enough 1920x1200 matte screen. $800 and a 3 hour drive to the nearest Microcenter. I gave Apple another try (refused apple from 2009-2020 because of the nvidia era issues) and I just can't stomach living off of piles of external drives anymore to make up for their lack luster storage space on the affordable units.
The HP Elitebook was on Ubuntu's list of compatible tested laptops and came in hundreds of dollars less than a Thinkpad. Most of the comparably priced on sale T14's I could find were all crap Intel spec'd ones.
Months in I don't regret it at all and Linux support has been fantastic even for a fairly newer Ryzen chip and not the latest kernel. (I stick to LTS releases of most Distros) Shoving in 4TB of NVME storage and 96GB of DDR5 should I feel the need to upgrade would still put me only around $1300 invested in this machine.
I'm not really moaning about the cost or lack of upgradability. I mean, I don't like it but at least you know what you're getting into. I just always assumed Linux as a backup was an option, and more and more OSX is annoying me (last 2 or 3 days it keeps dropping bluetooth for 30 seconds) and more and more I just find the interface distracting. Plus whether it works with external displays over USB C is a crapshoot.
I'll miss the battery life of the M1 chips, and I'm going to have to re-learn how to type (CTRL instead of ALT, fn rarely being on the left, I use fn+left instead of CTRL A in terminals) but otherwise, I think I'm done.
Surely you're using that thing as a laptop in a minority of cases though, looks like it's basically just specs you bought. That's fine, but if that's all you want then it seems like rather than trying to give a mac a reasonable go of it as opposed to whatever else, you were trying to instead explore a fundamental difference in how you value technology products, which is quite a different battle.
Not at all. Sure when I'm at home its docked, but so far in Linux battery life has been fantastic. Not Apple fantastic sure, but I can get a good 5 hours of heavy use, up to around 8 hours of web browsing and video streaming. I often use it on the road, throw back quake3 lan parties, coffee shop creative sessions.
I just want decent enough power and no thermal throttling if I do have to hammer it. I make music so the extra ram and space for sample libraries is a big benefit and why I had to keep external SSD's around with my Macs.
My Macbook Air needed a usb fan ziptied to the laptop stand to not throttle at times.
>it seems like rather than trying to give a mac a reasonable go of it as opposed to whatever else, you were trying to instead explore a fundamental difference in how you value technology products
I re-evaluate how I feel about technology pretty often and its caused some shifts for sure. My side hobby is ARM/RiscV low power computing and Apple's move to ARM tickled that hyper efficiency side of my brain, but ultimately failed to keep me interested because of all the downsides upgrade/repairability wise.
I think the only laptops you won't find weird issues with linux are from smaller manufacturers dedicated to shipping them like the kde laptop or system76. Every other hardware manufacturer, including those that ship laptops with linux preinstalled, probably have weird hardware incompatibilities because they don't fully customize their SKUs with linux support in mind.
Not that I'm discouraging you from switching or anything. If Linux is what you want/need, there's definitely better laptops to be had than a Macbook for that purpose. It's just that weird incompatibilities and having to fight with the operating system on random issues is, at least in my experience, normal when using a linux laptop. Even my T480 which has overall excellent compatibility isn't trouble-free.
Something like the brightness buttons not working, or sleep being a little erratic is ok. No released wifi drivers, bluetooth issues, and audio and the keyboard not working are not ok. Apple going backwards in terms of supporting Linux is not something I'm ok with.
There are wifi drivers; you just have to install them separately because they use broadcom chips. It's a proprietary blob. The other things do work, but it requires special packages and you'll need an external keyboard while installing. It's a pain to install, for sure, but it's not insurmountably difficult to get it installed.
Apple Silicon chips are arguably more compatible with Asahi Linux [1], but that's largely in thanks to the hard work of Marcan, who's stepped down as project lead from the project [2].
Overall I still think the right choice is to find a laptop better suited for the purpose of running linux on it, just something that requires more careful consideration than people think. Framework laptops, which seem well suited since ideologically it meshes well with linux users, can be a pain to set up as well.
I know there are wifi and keyboard drivers, because the live boots and installers work with them, but then when it comes to installing they're gone. I know it's not insurmountable, and 10 years ago I'd have done it, but I spent a few hours and got sick of it. I agree with you that it's probably better to get another laptop.
> This hardware is really being held back by the operating system at this point.
Apple could either create a 2U rack hardware and support Linux (and I mean Apple supporting it, not hobbysts), or have a build of Darwin headless that could run on that hardware. But in the later case, we probably wouldn't have much software available (though I am sure people would eventually starting porting software to it, there is already MacPorts and Homebrew and I am sure they could be adapted to eventually run in that platform).
But Apple is also not interested in that market, so this will probably never happen.
> But Apple is also not interested in that market, so this will probably never happen.
they're just a tiny company with shareholders who are really tired of never earning back their investments. give 'em a break. I mean they're still so small that they must protect themselves by requiring that macs be used for publishing iPhone and iPad applications.
Not to get in the way of good snark or anything. But.. Apple isn't _requiring_ that everyone uses MacOS on their systems. But you have to bring your own engineering effort to actually make another OS run. And so far Asahi is the only effort that I'm aware of (there were alternatives in the very beginning, but they didn't even get to M2 right?)
> But you have to bring your own engineering effort to actually make another OS run.
I mean, that's usually how it works though. When IBM launched the PS/2, they didn't support anything other than PC-DOS and OS/2, Microsoft had to make MS-DOS work for it (I mean... they did get support from IBM, but not really), the 386BSD and Linux communities brought the engineering effort without IBM's involvement.
When Apple was making Motorola Macs, they may have given Be a little help, but didn't support any other OSes that appeared. Same with PowerPC.
All of the support for alternative OSes has always come from the community, whether that's volunteers or a commercial interest with cash to burn. Why should that change for Apple silicon?
There has to be someone at Apple with a contact at IBM that could make Fedora Apple Remix happen. It may not be on-brand, but this is a prime opportunity to make the competition look worse. File it under Community projects at https://opensource.apple.com/projects
The bargain is the lower price in the UK compared to US, once US sales tax is added. It's not like the pound is strong. It's just cheaper in the UK. And you're right, all Apple products are better value in that UK. I'm not used to any electronics being good value in the UK.
I cannot express how dirt cheap that pricepoint is for what's on offer, especially when you're comparing it to rackmount servers. By the time you've shoehorned in an nVidia GPU and all that RAM, you're easily looking at 5x that MSRP; sure, you get proper redundancy and extendable storage for that added cost, but now you also need redundant UPSes and have local storage to manage instead of centralized SANs or NASes.
For SMBs or Edge deployments where redundancy isn't as critical or budgets aren't as large, this is an incredibly compelling offering...if Apple actually had a competent server OS to layer on top of that hardware, which it does not.
If they did, though...whew, I'd be quaking in my boots if I were the usual Enterprise hardware vendors. That's a damn frightening piece of competition.
From my outsider perspective, it's pretty straightforward why they don't.
In Intel's case, there's ample coverage of the company's lack of direction and complacency on existing hardware, even as their competitors ate away at their moat, year after year. AMD with their EPYC chips taking datacenter share, Apple moving to in-house silicon for their entire product line, Qualcomm and Microsoft partnering with ongoing exploration of ARM solutions. A lack of competency in leadership over that time period has annihilated their lead in an industry they used to single-handedly dictate, and it's unlikely they'll recover that anytime soon. So in a sense, Intel cannot make a similar product, in a timely manner, that competes in this segment.
As for AMD, it's a bit more complicated. They're seeing pleasant success in their CPU lineup, and have all but thrown in the towel on higher-end GPUs. The industry has broadly rallied around CUDA instead of OpenCL or other alternatives, especially in the datacenter, and AMD realizes it's a fool's errand to try and compete directly there when it's a monopoly in practice. Instead of squandering capital to compete, they can just continue succeeding and working on their own moat in the areas they specialize in - mid-range GPUs for work and gaming, CPUs targeting consumers and datacenters, and APUs finding their way into game consoles, handhelds, and other consumer devices or Edge compute systems.
And that's just getting into the specifics of those two companies. The reality is that any vendor who hasn't already unveiled their own chips or accelerators is coming in at what's perceived to be the "top" of the bubble or market. They'd lack the capital or moat to really build themselves up as a proper competitor, and are more likely to just be acquired in the current regulatory environment (or lack thereof) for a quick payout to shareholders. There's a reason why the persistent rumor of Qualcomm purchasing part or whole of Intel just won't die: the x86 market is rather stagnant, churning out mediocre improvements YoY at growing pricepoints, while ARM and RISC chips continue to innovate on modern manufacturing processes and chip designs. The growth is not in x86, but a juggernaut like Qualcomm would be an ideal buyer for a "dying" or "completed" business like Intel's, where the only thing left to do is constantly iterate for diminishing returns.
> By the time you've shoehorned in an nVidia GPU and all that RAM, you're easily looking at 5x that MSRP
That nvidia GPU setup will actually have the compute grunt to make use of the RAM, though, which this M3 Ultra probably realistically doesn't. After all, if the only thing that mattered was RAM then the 2TB you can shove into an Epyc or Xeon would already be dominating the AI industry. But they aren't, because it isn't. It certainly hits at a unique combination of things, but whether or not that's maximally useful for the money is a completely different story.
Had the M3 GPU been much wider, it would be constrained by the memory bandwidth. It might still have an advantage over Nvidia competitors in that it has 512GB accessible to it and will need to push less memory across socket boundaries.
You're forgetting what Apple's been baking into their silicon for (nearly? over?) a decade: the Neural Processing Unit (NPU), now called the "Neural Engine". That's their secret sauce that makes their kit more competitive for endpoint and edge inference than standard x86 CPUs. It's why I can get similarly satisfying performance on my old M1 Pro Macbook Pro with a scant 16GB of memory as I can on my 10900k w/ 64GB RAM and an RTX 3090 under the hood. Just to put these two into context, I ran the latest version of LM Studio with the deepseek-r1-distill-llama-8b model @ Q8_0, both with the exact same prompt and maximally offloaded onto hardware acceleration and memory, with a context window that was entirely empty:
Write me an AWS CloudFormation file that does the following:
* Deploys an Amazon Kubernetes Cluster
* Deploys Busybox in the namespace "Test1", including creating that Namespace
* Deploys a second Busybox in the namespace "Test3", including creating that Namespace
* Creates a PVC for 60GB of storage
The M1Pro laptop with 16GB of Unified Memory:
* 21.28 seconds for "Thinking"
* 0.22s to the first token
* 18.65 tokens/second over 1484 tokens in its responses
* 1m:23s from sending the input to completion of the output
The 10900k CPU, with 64GB of RAM and a full-fat RTX 3090 GPU in it:
* 10.88 seconds for "thinking"
* 0.04s to first token
* 58.02 tokens/second over 1905 tokens in its responses
* 0m:34s from sending the input to completion of the output
Same model, same loader, different architectures and resources. This is why a lot of the AI crowd are on Macs: their chip designs, especially the Neural Engine and GPUs, allow quite competent edge inference while sipping comparative thimbles of energy. It's why if I were all-in on LLMs or leveraged them for work more often (which I intend to, given how I'm currently selling my generalist expertise to potential employers), I'd be seriously eyeballing these little Mac Studios for their local inference capabilities.
Uh.... I must be missing something here, because you're hyping up Apple's NPU only to show it getting absolutely obliterated by the equally old 3090? Your 10900K having 64gb of RAM is also irrelevant here...
You're missing the the bigger picture by getting bogged down in technical details. To an end user, the difference between thirty seconds and ninety seconds is often irrelevant for things like AI, where they expect a delay while it "thinks". When taken in that context, you're now comparing a 14" laptop running off its battery, to a desktop rig gulping down ~500W according to my UPS, for a mere 66% reduction in runtime for a single query at the expense of 5x the power draw.
Sure, the desktop machine performs better, as would a datacenter server jam-packed full of Blackwell GPUs, but that's not what's exciting about Apple's implementation. It's the efficiency of it all, being able to handle modern models on comparatively "weaker" hardware most folks would dismiss outright. That's the point I was trying to make.
We're talking about the m3 ultra here, which is also wall powered and also expensive. Nobody is interested in dropping upwards of $10,000 on a Mac Studio to have "okay" performance just because an unrelated product is battery powered. Similarly saving a few bucks on electricity to triple the time the much, much more expensive engineer time spent waiting on results is foolish
Also Apple isn't unique in having an NPU in a laptop. Fucking everyone does at this point.
It almost feels like you're deliberately missing the forest for the trees, in order to fit some argument that I'm not quite able to sus out here.
The point is that, in terms of practical usage, the M3 Ultra is uniquely competent and highly affordable in a sea of enterprise technology that is decidedly not. I tried to demonstrate why I'm excited about it by pointing out the similar performance of a battery-powered, four-year-old laptop and a quite gargantuan gaming PC that's pulling over 500W from the wall, as an example of what several years of additional refinements and improvements to the architecture was expected to bring.
The point is that it's affordable, more flexible in deployment, and more efficient than similarly-specced datacenter servers specifically designed for inference. For the cost of a single decked-out Dell or HP rackmount server, I can have five of these Mac Studios with M3 Ultra chips - and without the need for substantial cooling, noise isolation, or other datacenter necessities. If the marketing copy is even in the same ballpark as actual performance, that's easily enough inference to serve an office of fifty to a hundred people or more, depending on latency tolerances; if you don't mind "queuing" work (like CurrentCo does with their internal Agents), one of those is likely enough for a hundred users.
That's the excitement. That's the point. It's not the fastest, it's not the cheapest, it's just the most balanced.
Apple defenders have some special sauce reasoning that makes no sense to anyone but them.
Are you a boomer?
I have Apple hardware but it sucks for anything AI, buying it for that purpose is just extremely dumb, just like buying Macs for engineering CADs or things of the sort.
If you are buying Macs and it's not for media production related reasons you are doing something wrong.
> Apple defenders have some special sauce reasoning that makes no sense to anyone but them. Are you a boomer?
I continue to be in awe of the lengths some people will go just to fling insults and shake out some salt. We're, what, ten layers deep? With all the context above, the best you have to contribute to the discussion are baseless accusations and ageist insults?
Your finite time would have been better spent on literally anything else, than actively seeking out a comment just to throw subjective, unsubstantiated shade around. C'mon, be better.
Makes no mistake, it's not an insult. I'm saying that precisely because I have been there.
Apple is the master at creating desire and building narrative in their customers' mind about the many things their devices would allow them to do. It's very aspirational and in practice most of the Macs get used for things that could have been done with a much cheaper option.
It may not be obvious to you but it's somewhat funny seeing you rationalise all kinds of dreams of what this machine could potentially be when in practice the people who would really be working on the kind of stuff you are talking about don't even consider them viable for many good reasons.
It's not that those machines cannot potentially do it, it's just that they don't really fit the goal very well.
A lot like people buying Cybertruck to "haul" stuff when they are a lot more option that are just plain better and make a lot more economic/practical sense.
It's OK to desire the thing and be excited about it but it really doesn't serve anyone to rationalise it so hard, you are lying to yourself as much as everyone else, it's not healthy.
If that was not clear, people working on AI stuff professionally really don't have to deal with a Mac Studio, they have access to better stuff. If you want to get one personally to experiment/toy around it's ok but it's not going to be this amazing thing for AI.
If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo.
The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.
The Asahi maintainer resigned recently. What that means for the future only time will tell. I probably wouldn't want to make a big investment in it right now.
Your wording makes it sound like it was a one-man show. Asahi has a really strong contributor base, new leadership[1], and the backing of Fedora via the Asahi Fedora Remix. While Hector resigning is a loss, I don't think it's a death knell for the project.
M3 support in Asahi is still heavily WIP. I think it doesn't even have display support, Ethernet, or Wifi yet, I think it's only serial over USB . Without any GPU or ANE support, it's not very useful for AI stuff. https://asahilinux.org/docs/M3-Series-Feature-Support/
Hmm, this page links to an out-of-tree ANE module (same as on M1/M2 I believe). No GPU support is a bummer, though.
On the other hand, you can do without display support if you’re only using it as a server. And I think USB Ethernet dongles might work for the time being?
The last I checked, AMD was outperforming Apple perf/dollar on the high end, though they were close on perf/watt for the TDPs where their parts overlapped.
I’d be curious to know if this changes that. It’d take a lot more than doubling cores to take out the very high power AMD parts, but this might squeeze them a bit.
Interestingly, AMD has also been investing heavily in unified RAM. I wonder if they have / plan an SoC that competes 1:1 with this. (Most of the parts I’m referring to are set up for discrete graphics.)
That’s a laptop part, so it makes different tradeoffs.
Somewhere on the internet there is a tdp wattage vs performance x-y plot. There’s a pareto optimal region where all the apple and amd parts live. Apple owns low tdp, AMD owns high tdp. They duke it out in the middle. Intel is nowhere close to the line.
I’d guess someone has made one that includes datacenter ARM, but I’ve never seen it.
True, but these "Ultra" chips do target the same niche as (some) high-TDP chips.
Workstations (like the Mac Studio) have traditionally been a space where "enthusiast"-grade consumer parts (think Threadripper) and actual server parts competed. The owner of a workstation didn't usually care about their machine's TDP; they just cared that it could chew through their workloads as quickly as possible. But, unlike an actual server, workstations didn't need the super-high core count required for multitenant parallelism; and would go idle for long stretches — thus benefitting (though not requiring) more-efficient power management that could drive down baseline TDP.
At least judging by the mounts, they want them to be used that way, even though the CPU might not fit with the de facto industry label for "server-grade".
The rack mount Mac Pro doesn't really make sense for a data center. It's 5U high, which is much too big for a data center. It doesn't have standard server features like redundant power supplies.
The only use case I can think of is for audio workstations, where people have lots of rack mount equipment, so you can have everything including the computer in the rack. But even for that use case it's quite big.
Indeed. The M3 Ultra is in the midrange where they duke it out. Similarly, for its niche, the iPhone CPU is was better than AMD’s low end processors.
Anyway the Apple config in the article costs about 5x more than a comparable low end AMD server with 512GB of ram, but adds an NPU. AMD has NPUs in lower end stuff; not sure about this TDP range.
> You mean server-grade CPUs? Apple doesn't make those.
Right.
It is coming up because we're in a thread about using them as server CPUs. (c.f. "colo", "2U" in OP and OP's child), and the person you're replying to is making the same point you are
For years now, people will comment "these are the best chips, I'd replace all chips with them."
Then someone points out perf/watt is not perf.
Then someone else points out some M-series is much faster than a random CPU.
And someone else points out that the random CPU is not a top performing CPU.
And someone else points out M-series are optimized for perf/watt and it'd suck if it wasn't.
I love my MacBook, the M-series has no competitors in the case it's designed for.
I'd just prefer, at this point, that we can skip long threads rehashing it.
It's a great chip. It's not the fastest, and it's better for that. We want perf/watt in our mobile devices. There's fundamental, well-understood, engineering tradeoffs that imply being great at that necessitates the existence of faster processors.
I think that is much too hand-wavy regarding the performance differences.
Both Passmark and Geekbench are aggregates of a variety of tasks. If you dig into the individual tests that constitute this aggregate score, you will find different platforms perform better, or worse, on certain tests than others. I would wager that, for many applications, only a subset of these tasks are relevant to the performance of the application, yet such benchmark suites distil out all nuance into a single value.
Here is a personal anecdote. I have tried running CASTEP (built from source), a density functional theory calculator, on both an M1 Max MacBook Pro [0], and on a Ryzen 7840HS Lenovo laptop [1]. A cursory glance at those Geekbench results linked might make you expect that the performance is roughly equivalent, but the Ryzen outperforms the Mac by about 4x, a huge difference.
What happens if we try and dig into any particular benchmark to explain this? If you click on any particular benchmark in the Geekbench search lists, you will see they test things like "File Compression", "HTML5 Browser", "Clang". Which of these maps most closely to the sorts of instructions used in CASTEP? Your guess is as good as mine.
If anything, I would say Passmark is quite a bit less abstract about this. Looking at the Mac [2] and Ryzen [3] Passmark results, you can see the Ryzen outperforms the Mac by about 2x on "extended instructions", which appear to involve some matrix math, and also about 2x on "integer math". The Mac, meanwhile, appears to be extremely good at finding prime numbers, at over 3x the speed of the Ryzen. Presumably the Ryzen's balance of instruction performance is more useful for DFT calculations than the Mac's, which perhaps is weaker in areas that might matter for this application, but stronger in areas that might matter for others.
Of course, optimization is likely a component of this. How much effort is put into the OpenBLAS, MPI, etc, implementations on aarch64 darwin vs. x86-64 linux? This is a good question. It is, however, mostly irrelevant to the end consumer, who wishes to consume this software for use in their further research, rather than dig into high-performance computing library optimization.
This is my experience as well. Geekbench heavily favors the type of workload that runs best on Apple hardware (those tends to be general case, most likely to be used by the mass) but in practice if you have complex software to run your experience will not match the bench numbers.
I think PassMark is more honest as well, because it just gives scores for calculation throughput instead of specific tasks. It more closely matches what experience you will get if you have a varied load.
But since it's Apple we are talking about, their users just want to think they have the best and that's all that matters.
Right from the top it's amateurish stuff: their idea of an integer benchmark to measure "raw" CPU throughput (whatever that means) is to make a bunch of random ints and add/subtract/multiply/divide them.
Very few programs do a high volume of either integer multiply or divide. And when they do, they generally aren't doing it on random numbers. This is the kind of thing which gives synthetic benchmarks their highly deserved bad rep. It might be even worse than Dhrystone MIPs, and believe me, in benchmark nerd circles, that is a fucking diss.
If you look up Geekbench's docs, you'll find that it's all about real-world compute tasks. For example, one of the int tests in their suite is to compile a reference program with the Clang compiler. Compilers are a reasonably good litmus test of integer performance; they heavily stress the CPU features most responsible for high integer performance in this day and age. (Branch prediction, memory prefetching, out-of-order execution, speculation, that kind of thing.)
You claimed that PassMark reflects "complex" software, and Geekbench doesn't. However, I would be willing to bet that Clang alone is far more complex than all of PassMark's CPU benchmarks put together, whether you measure by SLOC or program structure.
Note that none of this has anything to do with Mac vs PC. Passmark is simply a bad benchmark that should not be used, period. That said, there are a bunch of warning signs that PassMark's ports to everything outside its native x86 Windows are probably quite sloppy, so it's even less useful for crossplatform comparisons.
Geekbench correlates with SPEC, the industry standard in CPU benchmark and what enterprise companies such as AWS uses to judge a CPU performance. It has .99 correlation.
The M4 Max had great, I would argue the best at time of release, single core results on Geekbench.
That is a different claim from M4 line has the top single thread performance in the world.
I'm curious:
You're signalling both that you understand the fundamental tradeoff ("Apple doesn't make server-grade CPUs") and that you are talking about something else (follow-up with M4 family has top single-thread performance)
What drives that? What's the other thing you're hoping to communicate?
If you are worried that if you leave it at "Apple doesn't make server-grade CPUs", that people will think M4s aren't as great as they are, this is a technical-enough audience, I think we'll understand :) It doesn't come across as denigrating the M-series, but as understanding a fundamental, physically-based, tradeoff.
Maybe it is, maybe not, UNIX and Windows server software have been multithreaded / multi-process for decades, we want tons of threads and processes, not a single one.
What about serviceability? These come with soldered in ssd?
That would be an issue for server use, Its too expensive to throw it away all for a broken ssd.
Nah, in many businesses, everything is on a schedule. For desktop computers, a common cycle is 4 years. For servers, maybe a little longer, but not by much. After that date arrives, it’s liquidate everything and rebuild.
Having things consistently work is much cheaper than down days caused by your ancient equipment. Apple’s SSDs will make it to 5 years no problem - and more likely, 10-15 years.
At my last N jobs, companies built high end server farms and carefully specced all the hardware. Then they looked at SSD specs and said “these are all fine”.
Fast forward 2 years: The $50-$250K machines have a 100% drive failure rate, and some poor bastard has to fly from data center to data center to swap the $60 drive for a $120 one, then re-rack and re-image each machine.
Anyway, soldering a decent SSD to the motherboard board would actually improve reliability at all those places.
What does soldering it to the board have to do with reliability?
If they were soldered onto those systems you talk about, all those would have had to be replaced instead of just having the drive swapped out and re-imaged.
I think the implication was that a soldered SSD doesn't give the customer as much chance to pick the wrong SSD. But it's still possible for the customer to have a different use case in mind than the OEM did when the OEM is picking what SSD to include.
It wouldn't solve other mismatched expectations. For example, the vendor might ship those SSDs only to store firmware-initiated crash dumps. They don't expect them to be used to meet production storage requirements. Maybe to occasionally boot a maintenance system, but that's it. To them, this is kind of obvious because everybody has a SAN anyway. But of course, this is not actually true in practice because customers vary a bit.
If I read this right, the r8g.48xlarge at AMZN [1] has 192 cores and 1536GB which exceeds the M3 Ultra in some metrics.
It reminds me of the 1990s when my old school was using Sun machines based on the 68k series and later SPARC and we were blown away with the toaster-sized HP PA RISC machine that was used for student work for all the CS classes.
Then Linux came out and it was clear the 386 trashed them all in terms of value and as we got the 486 and 586 and further generations, the Intel architecture trashed them in every respect.
The story then was that Intel was making more parts than anybody else so nobody else could afford to keep up the investment.
The same is happening with parts for phones and TSMC's manufacturing dominance -- and today with chiplets you can build up things like the M3 Ultra out of smaller parts.
In fairness, the sun and dec boxes I used back then (up to about 1999) could hold their own against intel machines.
Then, one day, we built a 5 machine amd athlon xp linux cluster for $2000 ($400/machine) that beat all the unix and windows server hardware by at least 10x on $/perf.
It’s nice that we have more than one viable cpu vendor these days, though it seems like there’s only one viable fab company.
For what we needed, five 32 bit address spaces was enough DRAM. The individual CPU parts were way more than 20% as fast, and the 100Mbit switch was good enough.
(The data basically fit in ram, so network transport time to load a machine was bounded by 4GiB / 8MiB / sec = 500 seconds. Also, the hard disks weren’t much faster than network back then.)
The Alpha architecture was 64-bit from the very beginning (though the amount of addressable virtual memory and physical memory depends on the processor implementation).
The EV6 is a bit quirky as it is 43-bit by default, but can use 48-bits when I_CTL<VA_48> or VA_CTL<VA_48> is set. (the distinction of the registers is for each access type, i.e: instruction fetch versus data load/store)
The 21364/EV7 likely has the same characteristics as EV6, but the hardware reference manual seems to have been lost to time...
My understanding is that the VAX from Digital was the mother of all "32-bit" architectures to replace the dead end PDP-11 (had a 64kbyte user space so wasn't really that much better than an Apple ][) and PDP-10/20 (36-bit words were awkward after the 8-bit byte took over the industry) The 68k and 386 protected mode were imitations of the VAX.
Digital struggled with the microprocessor transition because they didn't want to kill their cash cow minicomputers with microcomputer-based replacements. They went with the 64-bit Alpha because they wanted to rule the high end in the CMOS age. And they did, for a little while. But the mass market caught up.
VMS is the only OS (that I know of) that uses all 4 processor privilege modes.
Side note: The 21064 has such bizarre IPR mappings, the read values have lots of bits scrambled around compared to their write counterparts. This is likely a hardware design decision affecting the programmer's model, if I had to guess.
In 1998 I somehow got my hands on a DEC OEM 21164 533mhz board for cheap. PCs were nowhere near the performance of that
at the time. It mounted in a regular PC case. A friend helped me get the power supply working (there was I think one wire to solder somewhere). Equipped with an ASUS SCSI card, an DEC Ethernet card, and an Elsa GLoria Synergy, it was a full machine. I ran Digital Unix at home on my desk on that for quite a few years. Wish I had kept it for old times sake.
One thing I remember about Alpha though was how bad the output from gcc was. Then DEC released a version of their own compilers that was command line compatible with gcc. That changed everything for open source stuff.
It seems Graviton 4 CPUs have 12-channels of DDR5-5600 i.e 540GB/s main memory bandwidth for the CPU to use. M3 Ultra has 64-channels of LPDDR5-6400 i.e. ~800GB/s of memory bandwidth for the CPU or the GPU to use. So the M3 Ultra has way fewer (CPU) cores, but way more memory bandwidth. Depends what you're doing.
Yea ive been thinking about this for a few years. The Mx series’s chip would sell into data centers like crazy if apple went after that market. Especially if they created a server tuned chip. It could probably be their 2nd biggest product line behind the iphone. The performance and efficiency is awesome. I guess it would be meat to see some web serving and database benchmarks to really know.
TSMC couldn’t make enough at the leading node in addition to all the iPhone chips Apple has to sell. There’s a physical thoughput limit. That’s why this isn’t M4.
> The CPUs have zero competition in terms of speed, memory bandwidth.
Maybe not at the same power consumption, but I'm sure mid-range Xeons and EPYCs mop the floor with the M3 Ultra in CPU performance. What the M3 Ultra has that nobody else comes close is a decent GPU near a pool of half a terabyte of RAM.
> Apple does not make server CPUs, they make consumer low W CPUs, it's very different.
This is silly. Given the performance per watt, the M series would be great in a data center. As you all know, electricity for running the servers and cooling for the servers are the two biggest ongoing costs for a data center; the M series requires less power and runs more efficiently than the average Intel or AMD-based server.
> FYI Apple runs Linux in their DC, so no Apple hardware in their own servers.
That's certainly no longer the case. Apple announced their Private Cloud Compute [1] initiative—Apple designed servers running Apple Silicon to support Apple Intelligence functions that can't run on-device.
BTW, Apple just announced a $500 billion investment [2] in US-based manufacturing, including a 250,000 square foot facility to make servers. Yes, these will obviously be for their Private Cloud Compute servers… but it doesn't have to be only for that purpose.
From the press release:
As part of its new U.S. investments, Apple will work with manufacturing partners to begin production of servers in Houston later this year. A 250,000-square-foot server manufacturing facility, slated to open in 2026, will create thousands of jobs.
Previously manufactured outside the U.S., the servers that will soon be assembled in Houston play a key role in powering Apple Intelligence, and are the foundation of Private Cloud Compute, which combines powerful AI processing with the most advanced security architecture ever deployed at scale for AI cloud computing. The servers bring together years of R&D by Apple engineers, and deliver the industry-leading security and performance of Apple silicon to the data center.
Teams at Apple designed the servers to be incredibly energy efficient, reducing the energy demands of Apple data centers — which already run on 100 percent renewable energy. As Apple brings Apple Intelligence to customers across the U.S., it also plans to continue expanding data center capacity in North Carolina, Iowa, Oregon, Arizona, and Nevada.
Hardly. x86 and ARM follow similar security underpinnings and outcomes. ARM has TrustZone, x86 has TEEs. I cannot think of a single attack demonstrated on x86 and not on ARM or vice versa. Could you please cite one?
Sorry you could not cite an attack demonstrated on x86 and not on ARM or vice versa. Well, maybe TikTag (https://github.com/compsec-snu/tiktag) was demonstrated against ARM MTE, since you mention, and Pacman (https://pacmanattack.com/) against PAC. I was hoping for an actual response, no offense.
I think it's interesting everyone that dissented mentioned power consumption.
Our business "only" sees about 1,000-25,000 req/min, our message brokers transmit MAX 25k msg/s. Easily handled by a rack of 10 servers for redundancy.
We are not Google and we don't pretend to be, so we don't care about power, as the difference is a few dollars a month.
> This hardware is really being held back by the operating system at this point.
It really is. Even if they themselves won't bring back their old XServe OS variant, I'd really appreciate it if they at least partnered with a Linux or BSD (good callout, ryao) dev to bring a server OS to the hardware stack. The consumer OS, while still better (to my subjective tastes) than Windows, is increasingly hampered by bloat and cruft that make it untenable for production server workloads, at least to my subjective standards.
A server OS that just treats the underlying hardware like a hypervisor would, making the various components attachable or shareable to VMs and Containers on top, would make these things incredibly valuable in smaller datacenters or Edge use cases. Having an on-prem NPU with that much RAM would be a godsend for local AI acceleration among a shared userbase on the LAN.
You are technically correct (the best kind of correct). I’m just a filthy heathen who lumps the BSDs and Linux distros under “Linux” as an incredibly incorrect catchall for casual discourse.
I’m continually surprised Apple doesn’t just donate something like 0.1% of their software development budget to proton and the asahi projects. It’d give them a big chunk of the gaming and server markets pretty much overnight.
I guess they’re too busy adding dark patterns that re-enable siri and apple intelligence instead.
Sure, but FreeBSD also has a Linux compatability layer. For a company that's given up on the server market so many times, making MacOS compatible with _THE_ server OS makes a lot of sense.
I feel like Apple and Ubiquiti have a missed collaboration opportunity on the latter point, especially with the latter's recent UniFi Express unit. It feels like pairing Ubiquiti's kit with Apple's Homekit could benefit both, by making it easier for Homekit users to create new VLANs specifically for Homekit devices, thereby improving security - with Apple dubbing the term, say, "Secure Device Network" or some marketingspeak to make it easier for average consumers to understand. An AppleTV unit could even act as a limited CloudKey for UniFi devices like Access Points, or UniFi Cameras to connect/integrate as Homekit Cameras.
Don't get me wrong, I wouldn't use that feature (I prefer self-hosting it all myself), but for folks like my family members, it'd be a killer addition to the lineup that makes my life supporting them much easier.
> for folks like my family members, it'd be a killer addition
HomeKit networking existed in Eero briefly. I put that in a lot of casual Apple homes. Seemed like missed oppty for Apple to let Amazon buy Eero, a more "spiritual successor" to the Airports.
Ubiquiti was founded by some Apple employees after they closed their Airport division.
I sincerely doubt they want to go through the trouble of collaborating with a company that was too greedy to keep investing in their network hardware.
Ubiquiti is decently priced, especially for niche hardware, unlike Apple. Fundamentally they diverge on the way to do things...
I've been looking at the potential for Apple to make really interesting LLM hardware. Their unified memory model could be a real game-changer because NVidia really forces market segmentation by limiting memory.
It's worth adding the M3 Ultra has 819GB/s memory bandwidth [1]. For comparison the RTX 5090 is 1800GB/s [2]. That's still less but the M4 Mac Minis have 120-300GB/s and this will limit token throughput so 819GB/s is a vast improvement.
For $9500 you can buy a M3 Ultra Mac Studio with 512GB of unified memory. I think that has massive potential.
The low energy use can be a game changer if you live in a crappy apartment with limited power capacity. I gave up my big GPU box dream because of that.
The PC doesn't have to run Windows either. Strictly speaking, professional applications see MacOS support as an Apple-sanctioned detriment.
> If you've ever used git, svn, or an IDE side by side
I still reach for Windows, even though it's a dogshit OS. I would rather use WSL to write and deploy a single app, as opposed to doing my work in a Linux VM or (god forbid) writing and debugging multiple versions just to support my development runtime. If I'm going to use an ad-encumbered commercial service-slop OS, I might as well pick the one that doesn't actively block my work.
It's also just clearly a powerful and interesting tinkering project, which there are valid arguments for, but this can just chill out on your desk as an elegant general productivity machine. What it wouldn't do that the tinkering project could do is be upgraded, act as a powerful gaming pc, or cause migraines from constant fan noise.
The custom build would work great though, and even moreso in a server room and as well continues to reveal by comparison how excessively Apple prices it's components.
It certainly is held back and that is unfortunate. But if you can run your workloads on this amazing machine, then that's a lot of compute for the buck.
I assume that there's a community of developers focusing on leveraging this hardware instead of complaining about the operating system.
Given that the M1 Ultra and M2 Ultra also exist, I'd expect either straight binning, or two designs that use mostly the same designs for the cores but more of them and a few extra features.
I love Apple but they love to speak in half truths in product launches. Are they saying the M3 Ultra is their first Thunderbolt 5 computer? I don't recall seeing any previous announcements.
I have a 4090 and, out of curiosity, I looked up the FLOPS in comparison with Apple chips.
Nvidia RTX 4090 (Ada Lovelace)
FP32: Approximately 82.6 TFLOPS
FP16: When using its 4th‑generation Tensor Cores in FP16 mode with FP32 accumulation, it can deliver roughly 165.2 TFLOPS (in non‑tensor mode, the FP16 rate is similar to FP32).
FP8: The Ada architecture introduces support for an FP8 format; using this mode (again with FP32 accumulation), the RTX 4090 can achieve roughly 330.3 TFLOPS (or about 660.6 TOPS, depending on how you count operations).
Apple M1 Ultra
(The previous‑generation top‑end Apple chip)
FP32: Around 15.9 TFLOPS (as reported in various benchmarks)
FP16: By similar scaling, FP16 performance would be roughly double that value—approximately 31.8 TFLOPS (again, an estimate based on common patterns in Apple’s GPU designs)
FP8: Like the M3 family, the M1 Ultra does not support a dedicated FP8 precision mode.
So a $2000 Nvidia 4090 gives you about 5x the FLOPS, but with far less high speed RAM (24GB vs. 512GB from Apple in the new M3 Ultra). The RAM bandwidth on the Nvidia card is over 1TBps, compared with 800GBps for Apple Silicon.
Apple is catching up here and I am very keen for them to continue doing so! Anything that knocks Nvidia down a notch is good for humanity.
> Anything that knocks Nvidia down a notch is good for humanity.
I don't love Nvidia a whole lot but I can't understand where this sentinent comes from. Apple abandoned their partnership with Nvidia, tried to support their own CUDA alternative with blackjack and hookers (OpenCL), abandoned that, and began rolling out a proprietary replacement.
CUDA sucks for the average Joe, but Apple abandoned any chance of taking the high road when they cut ties with Khronos. Apple doesn't want better AI infrastructure for humanity; they envy the control Nvidia wields and want it for themselves. Metal versus CUDA is the type of competition where no matter who wins, humanity loses. Bring back OpenCL, then we'll talk about net positives again.
Lots of AI HW is focused on RAM (512GB!). I have a cost-sensitive application that needs speed (300+ TOPS), but only 1GB of RAM. Are there any HW companies focused on that space?
Just buy any gaming card? Even something like the Jetson AGX Orin boasts 275 TOPS (but they add in all kind of different subsystems to reach that number).
The problem with the TOPS is that they add in ~100 TOPS from the "Deep Learning Accelerator" coprocessors, but they have a lot of awkward limitations on what they can do (and software support is terrible). The GPU is an Ampere generation, but there is no strict consumer GPU equivalent.
Like others have said, basically traditional GPUs (RTX 40/50 series in particular, 20/30 series have much weaker tensor cores).
In terms of software, recent NVIDIA and AMD research has focused on fast evaluation of small ~4 layer MLPs using FP8 weights for things like denoising, upscaling, radiance caching, and texture and material BRDF compression/decompression.
NVIDIA has just put out some new graphics API extensions and samples/demos for loading a chunk of neural net weights and performing inference from within a shader.
Too bad it lacks even the streaming mode SVE2 found in M4 cores. If only Apple would provide a full SVE2 implementation to put pressure on ARM to make it non-optional so AArch64 isn't effectively restricted to NEON for SIMD.
They update the Studio to M3 Ultra now, so M4 Ultra can presumably go directly into the Mac Pro at WWDC? Interesting timing. Maybe they'll change the form factor of the Mac Pro, too?
Additionally, I would assume this is a very low-volume product, so it being on N3B isn't a dealbreaker. At the same time, these chips must be very expensive to make, so tying them with luxury-priced RAM makes some kind of sense.
> Maybe they'll change the form factor of the Mac Pro, too?
Either that or kill the Mac Pro altogether, the current iteration is such a half-assed design and blatantly terrible value compared to the Studio that it feels like an end-of-the-road product just meant to tide PCIe users over until they can migrate everything to Thunderbolt.
They recycled a design meant to accommodate multiple beefy GPUs even though GPUs are no longer supported, so most of the cooling and power delivery is vestigial. Plus the PCIe expansion was quietly downgraded, Apple Silicon doesn't have a ton of PCIe lanes so the slots are heavily oversubscribed with PCIe switches.
IMO they had plans for a Mac Pro chip that didn’t work out, so they released the M2 version to let their Mac Pro customers know that they’re still committed to the product in the Apple Silicon era.
Could be. I'm not sure if this current incarnation of the Mac Pro signals a commitment to the product though. Same performance as the Mac Studio but 2-3x the price just to get PCI slots.
I agree. Nonetheless, I agree with Siracusa that the Mac Pro makes sense as a "halo car" in the Mac lineup.
I just find it interesting that you can currently buy a M2 Ultra Mac Pro that is weaker than the Mac Studio (for a comparable config) at a higher price. I guess it "remains a product in their lineup" and we'll hear more about it later.
Additionally: If they wanted to scrap it down the road, why would they do this now?
Isn't the Mac Studio the new trash can? I can't think of how a non-expandable Mac Pro could be meaningfully different to the Studio unless they introduce an even bigger chip above the Ultra.
Indeed, and tbh it really commits even more to the non-expandability that the Trashcan's designers seemed to be going for. After all, the Trashcan at least had replaceable RAM and storage. The Mac Studio has proprietary storage modules for no reason aside from Apple's convenience/profits (and of course the 'integrated' RAM which I'll charitably assume was done for altruistic reasons because of how it's "shared.")
The difference is that today users are accepting modern Macs where they rejected the Trashcan. I think it's because Apple's practices have become more widespread anyway*, and certain parts of the strategy like the RAM thing at least have upsides. That, and the thermals are better because the Trashcan's thermal design was not fit for purpose.
* I was trying to fix a friend's nice Lenovo laptop recently -- it turned out to just have some bad RAM, but when we opened it up we found it was soldered :(
The Mac Pro could exist as a PCIe expansion slot storage case that accepts a logic board from the frequently updated consumer models. Or multiple Mac Studio logic boards all in one case with your expansion cards all working together.
My understanding was that Apple wanted to figure out how to build systems with multi-SOCs to replace the Ultra chips. The way it is currently done means that the Max chips need to be designed around the interconnect. Theoretically speaking, a multi-SOC setup could also scale beyond two chips and into a wider set of products.
I'm not sure if multi-SoC is possible because having 2 GPUs together such that the OS sees it as one big GPU is not very possible if the SoCs are separated.
Interestingly, Apple apparently confirmed to a French website that M4 lacks the interconnect required to make an "Ultra" [0][1], so contrary to what I originally thought, they maybe won't make this after all? I'll take this report with a grain of salt, but apparently it's coming directly from Apple.
Makes it even more puzzling what they are doing with the M2 Mac Pro.
Honestly I don't think we'll see the M4 Ultra at all this year. That they introduced the Studio with an M3 Ultra tells me M4 Ultras are too costly or they don't have capacity to build them.
And anyway, I think the M2 Mac Pro was Apple asking customers "hey, can you do anything interesting with these PCIe slots? because we can't think of anything outside of connectivity expansion really"
RIP Mac Pro unless they redesign Apple Silicon to allow for upgradeable GPUs.
Let's say you want to have the absolute max memory(512GB) to run AI models and let's say that you are O.K. with plugging a drive to archive your model weights then you can get this for a little bit shy of $10K. What a dream machine.
Compared to Nvidia's Project DIGITS which is supposed to cost $3K and be available "soon", you can get a specs matching 128GB & 4TB version of this Mac for about $4700 and the difference would be that you can actually get it in a week and will run macOS(no idea how much performance difference to expect).
I can't wait to see someone testing the full DeepSeek model on this, maybe this would be the first little companion AI device that you can fully own and can do whatever you like with it, hassle-free.
There’s an argument that replaceable pc parts is what you want at that price point, but Apple usually provides multi year durability on their pcs. An Apple ai brick should last awhile.
How many t/s would you expect? I think I feel perfectly fine when its over 50.
Also, people figured a way to run these things in parallel easily. The device is pretty small, I think for someone who wouldn't mind the price tag stacking 2-3 of those wouldn't be that bad.
A back of the napkin calculation: 819GB/s / 37GB/tok = 22 tokens/sec.
Realistically, you’ll have to run quantized to fit inside of the 512GB limit, so it could be more like 22GB of data transfer per token, which would yield 37 tokens per second as the theoretical limit.
It is likely going to be very usable. As other people have pointed out, the Mac Studio is also not the only option at this price point… but it is neat that it is an option.
Not sure why you are being downvoted, we already know the performance numbers due to memory bandwidth constraints on the M4 Max chips, it would apply here as well.
525GB/s to 1000GB/s will double the TPS at best, which is still quite low for large LLMs.
You can chain multiple Mac Studios using exo for inference, you'd "only" need two of these. There's a bottleneck in the RMA speed over TB5, but this may not matter as much for a MoE model.
At 9 grand I would certainly hope that they support the device software wise longer than they supported my 2017 Macbook Air. I see no reason to be forced to cough up 10 grand essentially every 7 years to Apple, that's ridiculous.
> support for more than half a terabyte of unified memory — the most ever in a personal computer
AMD Ryzen Threadripper PRO 3995WX released over four years ago and supports 2TB (64c/128t)
> Take your workstation's performance to the next level with the AMD Ryzen Threadripper PRO 3995WX 2.7 GHz 64-Core sWRX8 Processor. Built using the 7nm Zen Core architecture with the sWRX8 socket, this processor is designed to deliver exceptional performance for professionals such as artists, architects, engineers, and data scientists. Featuring 64 cores and 128 threads with a 2.7 GHz base clock frequency, a 4.2 GHz boost frequency, and 256MB of L3 cache, this processor significantly reduces rendering times for 8K videos, high-resolution photos, and 3D models. The Ryzen Threadripper PRO supports up to 128 PCI Express 4.0 lanes for high-speed throughput to compatible devices. It also supports up to 2TB of eight-channel ECC DDR4 memory at 3200 MHz to help efficiently run and multitask demanding applications.
No the comment misunderstood the difference between CPU memory and unified memory.
This can dedicate 500GB of high bandwidth memory to the GPU. - ~3.5X that of an H200.
> It also supports up to 2TB of eight-channel ECC DDR4 memory at 3200 MHz (sic) to help efficiently run and multitask demanding applications.
8 channels at 3200 MT/s (1600 MHz) is only 204.8 GB/sec; less than a quarter of what the M3 Ultra can do. It's also not GPU-addressable, meaning it's not actually unified memory at all.
I might like Apple again if the SoC could be sold separately and opened up. It would be interesting to see a PC with Asahi or Windows running on Apple’s chips.
When would Apple silicons made natively support for OSes such as Linux? Apple seemlingly reluctant to release detailed technical reference manual for M-series SoCs, which makes running Linux natively on Apple silicon challenging.
Right. Same goes for MacOS and all of it's convenient software services. Apple might stand to sell more units with a more friendlier stance towards Linux, but unless it sells more Apple One subscriptions or increases hardware margins on the Mac, I doubt Cook would consider it.
If you sit around expecting selflessness from Apple you will waste an enormous amount of time, trust me.
As I replied in else where here, I do not run any Apple Services on my Mac hardware. I do on my iDevices though, but that's a different topic. Again, I could be the edge case
But if you're being pedantic, I meant Apple SaaS requiring monthly payments or any other form of using something from Apple where I give them money outside the purchase of their hardware.
If you're talking background services as part of macOS, then you're being intentionally obtuse to the point and you know it
All seven of them. I kid, I have a lot of sympathy for that position, but as a practical matter running Linux VMs on an M4 works great, you even get GPU acceleration.
That’s what’s weird to me too. It’s not like they would lose sales of macOS as it is given away with the hardware. So if someone wants to buy Apple hardware to run Linux, it does not have a negative affect to AAPL
I have Mac hardware and and have spent $0 through the Mac App Store. I do not use iCloud on it either. I do on iDevices though. I must be an edge case though.
All of us on HN are basically edge cases. The main target market of Macs is super dependent on Apple service subscriptions.
Maybe that's why they ship with insultingly-small SSDs by default, so that as people's photo libraries, Desktop and Documents folders fill up, Apple can "fix your problem" for you by selling you the iCloud/Apple One plan to offload most of the stuff to only live in iCloud.
Either they spend the $400 up front to get 2 notches up on the SSD upgrade, to match what a reasonable device would come with, or they spend that $400 $10 a month for the 40 month likely lifetime of the computer. Apple wins either way.
Of course this is the reason.
And this is why Apple has become so bad for the tech enthusiasts, no matter how good the OS/software can be, you have to pay a tax that is way too big because you already have the competence that should allow you to bypass it.
It's like learning about growing vegetables in your garden but then having to pay the seeds for it much more because you actually know how to produce value with them.
The philosophy at Apple has changed from premium tools for professional to luxury device for normies that makes them pay for their incompetence.
While I don't think Apple wants to change course from its services-oriented profit model, surely someone within Apple has run the calculations for a server-oriented M3/M4 device. They're not far behind server CPUs in terms of performance while running a lot cooler AND having accelerated amd64 support, which Ampere lacks.
Whatever the profit margin on an iMac Studio is these days, surely improving non-consumer options becomes profitable at some point if you start selling them by the thousands to data centers.
There are a large number of macOS users that are not app software devs. There's a large base of creative users that couldn't code their way out of a wet paper bag, yet spend lots of money on Mac hardware.
This forum looses track of the world outside this echo chamber
I’m among them, even if creative works aren’t my bread and butter (I’m a dev with a bit of an artistic bent).
That said, attracting creative users also adds value to the platform by creating demand for creative software for macOS, which keeps existing packages for macOS maintained and brings new ones on board every so often.
I'm a mix of both, however, my dev time does not create macOS or iDevice apps. My dev is still focused on creative/media workflows, while I still get work for photo/video. I don't even use Xcode any further than running the CLI command to install the necessary tools to have CLI be useful.
You also lose out on developers. The more macOS users, the more attractive it is to develop for. Supporting Linux would be a loss for the macOS ecosystem, and we all know what that leads to.
> So if someone wants to buy Apple hardware to run Linux, it does not have a negative affect to AAPL
It does. Support costs. How do you prove it's a hardware failure or software? What should they do? Say it "unofficially" supports Linux? People would still try to get support. Eventually they'd have to test it themselves etc.
Apple has already been in this spot. With the TrashCan MacPro, there was an issue with DaVinci Resolve under OS X at the time where the GPU was cause render issues. If you then rebooted into Windows with BootCamp using the exact same hardware and open up the exact same Resolve project with the exact same footage, the render errors disappeared. Apple blamed Resolve. DaVinci blamed GPU drivers. GPU blamed Apple.
I don't think Darwin has been directly distributed in bootable binary format for many years now. And, as far as I know, it has never been made available in that format for Apple silicon.
1,070 comments
[ 3.7 ms ] story [ 489 ms ] threadSoldered?
These machines have a 512 bit interface, so presumably even worse.
I'd love to have more points of comparison available, but Strix Halo is the most analogous chip to an M-series chip on the market right now from a memory point of view, so it's hard to really know anything.
I very much hope CAMM2 or something else can be made to work with a Strix-like setup in the future, but I have my doubts.
It's getting that bandwdith by going very wide on very very very many channels, rather than trying to push a gigantic amount of bandwidth through only a few channels.
- The AI Max+ 395 is a 256 bit bus ("4 channels") of 8000 MHz instead of 128 bits ("2 channels") of 16000 MHz because you can't practically get past 9000 MHz in a consumer device, even if you solder the RAM, at the moment. Max capacity 128 GB.
- 5th Gen Epyc is a 768 bit bus ("12 channels") of 6000 MHz because that lets you use a standard socketed setup. Max capacity 6 TB.
- M3 Ultra is a 1024 bit bus ("16 channels") of "~6266 MHz" as it's 2x the M3 Max (which is 512 bits wide) and we know the final bandwidth is ~800 GB/s. Max capacity 512 GB.
Note: "Channels" is in quotes because the number of bits per channel isn't actually the same per platform (and DDR5 is actually 2x32 bit channels per DIMM instead of 1x64 per DIMM like older DDR... this kind of shit is why just looking at the actual bit width is easier :p).
So really the frequencies aren't that different even though these are completely different products across completely different segments. The overwhelming factor is bus width (channels) and the rest is more or less design choice noise from the perspective of raw performance.
The memory bus is the same as for modules, it's just very short. The higher end SoCs have more memory bandwidth because the bus is wider (i.e. more modules in parallel).
You could blame DDR5 (who thought having a speed negotiation that can go over a minute at boot is a good idea?), but I blame the obsession with thin and the ability to overcharge your customers.
> I've never seen a GPU with replaceable RAM
I still have one :) It's an ISA Trident TVGA 8900 that I personally upgraded from 512k VRAM to one full megabyte!!!
Figure out a way to make it unified without also soldering it, and you'll be a billionaire.
Or are you just grinding a tired, 20-year-old axe.
There's a good reason it's soldered, i.e. the wide memory interface and huge bandwidth mean that the extra trace lengths needed for an upgradable RAM slot would screw up the memory timings too much, but there's no need to make false claims like saying it's on-die.
Existing ones possibly but why not build something that lets you snap-in a BGA package just like we snap in CPUs on full sized PC mainboards?
It's the same reason nobody sells GPUs that have user upgradable non-soldered GDDR VRAM modules.
The issue is availability of chips and most likely you have to know which components to change so the new memory is recognised. For instance that could be changing a resistor to different value or bridging certain pads.
What if both are an issue?
Is anyone other than a vanishingly small number of hard core hobbiests going to upgrade from an M4 to an M4 Ultra?
I expect that the 2 biggest buyers of M4 Ultra will be people who want to run LLMs locally, and people who want the highest performance machine they can get (professionals), but are wedded to mac-only software.
It is reasonable to say many folks in the field prefer to work on mac hardware.
Hard to drop that much cash on an outdated chip.
(sorry, should have specified that the NPU and GPU cores need to access that ram and have reasonable performance). I specified it above, but people didn't read that :-)
Also, 32GB DDR5 RDIMMS are ~200, so that’s 5K for 24 right there. Then you need 2x CPUs, at ~1K for the cheapest, and you need 2, and then a motherboard that’s another 1K. So for 8K (more, given you need a case, power supply, and cooling!), you get a system with about half the memory bandwidth, much higher power consumption, and very large.
You do not need 2 CPUs. If however you use 2 CPUs, then the memory bandwidth doubles, to 1152 GB/s, exceeding Apple by 40% in memory bandwidth. The cost of the memory would be about the same, by using 16 GB modules, but the MB would be more expensive and the second CPU would add to the price.
The memory bandwidth does not double, I believe. See this random issue for a graph that has single/dual socket measurements, there is essentially no difference: https://github.com/abetlen/llama-cpp-python/issues/1098
Perhaps this is incorrect now, but I also know with 2x 4090s you don’t get higher tokens per second than 1x 4090 with llama.cpp, just more memory capacity.
(All if this only applies to llama.cpp, I have no experience with other software and how memory bandwidth may scale across sockets)
With a badly organized program, the performance can be limited not by the memory bandwidth, which is always exactly double for a dual-socket system, but by the transfers on the inter-socket links.
Moreover, your link is about older Intel Xeon Sapphire Rapids CPUs, with inferior memory interfaces and with more quirks in memory optimization.
But where is your data? For llama.cpp? For whatever dual socket CPU system you want. That’s all I am claiming.
https://github.com/ggml-org/llama.cpp/discussions/11733
about the scaling of llama.cpp and DeepSeek on some dual-socket AMD systems.
While it was rather tricky, after many experiments they have obtained an almost double speed on two sockets, especially on AMD Turin.
However, if you look at the actual benchmark data, that must be much lower than what is really possible, because their test AMD Turin system (named there P1) had only two thirds of the memory channels populated, i.e. performance limited by memory bandwidth could be increased by 50%, and they had 16-core CPUs, so performance limited by computation could be increased around 10 times.
A single 192 core Epyc is 11k by itself, so I’d probably go for the simpler integrated M3 ultra solution…
Time to first token, context length, and tokens/s are significantly inferior on CPUs when dealing with larger models even if the bandwidth is the same.
When used for ML/AI applications, a consumer GPU has much better performance per dollar.
Nevertheless, when it is desired to use much more memory than in a desktop GPU, a dual-socket server can have higher memory bandwidth than most desktop GPUs, i.e. more than an RTX 4090, and a computational capability that for FP32 could exceed an RTX 4080, but it would be slower for low-precision data where the NVIDIA tensor cores can be used.
INT8, INT4, FP8 and soon FP4
Both CPUs (with the BF16 instructions and with the VNNI instructions for INT8 inference) and the GPUs have a higher throughput for lower precision data types than for FP32, but the exact acceleration factors are hard to find.
The Intel server CPUs have the advantage vs. AMD that they also have the AMX matrix instructions, which are intended to compete for inference applications with the NVIDIA tensor cores, but the Intel CPUs are much more expensive for a number of cores big enough to be competitive with GPUs.
CUDA has had managed memory for a long time now. You absolutely can address the entire host memory from your GPU. It will fetch it, if it's needed. Not fast, but addressable.
Thinking about it you can get a decent 256gb on consumer platforms now too, but the speed will be a bit crap and would need to make sure the platform ully supports ECC UDIMMs
Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000.
Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.
Even for computationally-limited programs, the difference between server CPUs and consumer GPUs is not great. One Epyc CPU may have about the same number of FP32 execution units as an RTX 4070, while running at a higher clock frequency (but it lacks the tensor units of an NVIDIA GPU, which can greatly accelerate the execution, where applicable).
However Apple does not say anything about the GPU clock frequency, which I assume that it is significantly less than that of NVIDIA.
In comparison, a dual-socket AMD Turin can have up to 12288 FP32 execution units, i.e. 20% more than an Apple GPU.
Moreover, the clock frequency of the AMD CPU must be much higher than that of the Apple GPU, so it is likely that the AMD system may be at least twice faster for computing some graphic application than the Apple M3 Ultra GPU.
I do not know what facilities exist in the Apple GPU for accelerating the computations with low-precision data types, like the tensor cores of NVIDIA GPUs.
While for graphic applications big server CPUs are actually less compute constrained than almost all consumer GPUs (except RTX 4090/5090), the GPUs can be faster for ML/AI applications that use low-precision data types, but this is not at all certain for the Apple GPU.
Even if the Apple GPU happens to be faster for some low-precision data type, the difference cannot be great.
However a server that would beat the Apple M3 Ultra GPU computationally would cost much more than $10k, because it would need CPUs with many cores.
If the goal is only to have a system with 50% more memory and 40% more memory bandwidth than the Apple system, that can be done at a $10k price.
While such a system would become compute constrained more often than an Apple GPU, it would still beat it every time when the memory would be the bottleneck.
I have just compared the FP32 computational capabilities, i.e. what is used for graphics, between the Apple M3 Ultra GPU and AMD server CPUs, because these numbers are easily available and they demonstrate the size relationships between them.
Both GPUs and server CPUs have greater throughputs for lower precision data (CPUs have instructions for BF16 and INT8 inference), but the exact acceleration factors are hard to find and it is more difficult to estimate the speeds without access to such systems for running benchmarks.
I'd like to see some proper benchmarking on this though, but it looks like the Apple systems might just be extremely good value if you want to run the large DeepSeek model.
says who? NVIDIA has essentially entrenched themselves thanks to CUDA
Don't tell my boss! I still get paid.
SI has no business in memory size nomenclature as it is not derived from fundamental physical units. The whole klownbyte change was pushed through by hard drive marketers in 1990s.
What does it mean to "address memory in powers of two" ? There are certainly machines with non-power-of-two memory quantities; 96 GiB is common for example.
> The whole klownbyte change was pushed through by hard drive marketers in 1990s.
The metric prefixes based on powers of 10 have been around since the 1790s.
I challenge you to show me any SKU from any memory manufacturer that has a power of 10 capacity. Or a CPU whose address space is a power of 10. This is an unavoidable artefact of using a binary address bus.
> The metric prefixes based on powers of 10 have been around since the 1790s.
And Babylonians used power of 60, what gives?
If Donald Knuth and Gordon Bell say we use base-2 for RAM, that’s good enough for me.
I wonder if the plan is to only release Ultras for odd number generations.
Do you expect this will be able to handle AI workloads well?
All I’ve heard for the past two years is how important a beefy GPU is. Curious if that holds true here too.
The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing.
The M4's prompt parsing took significantly longer, but token generation was significantly faster.
Having the memory to the cores that matter makes a big difference.
> VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis.
Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.
My memory is wrong, it was the 32b. I'm running the 70b against a similar prompt and the 5950X is probably going to take over an hour for what the M4 managed in about 7 minutes.
edit: an hour later and the 5950 isn't even done thinking yet. Token generation is generously around 1 token/s.
edit edit: final statistics. M4 Pro managing 4 tokens/s prompt eval, 4.8 tokens/s token generation. 5950X managing 150 tokens/s prompt eval, and 1 token/s generation.
Perceptually I can live with the M4's performance. It's a set prompt, do something else, come back sort of thing. The 5950/RTX3080's is too slow to be even remotely usable with the 70b parameter model.
Otherwise, you don't even need a computer. Pen and paper is plenty.
For all practical purposes, VRAM is a limiting factor.
These are unified memory. The M3 Ultra with 512gb has as much VRAM as sixteen 5090.
The thing with these Apple chips is that they have unified memory, where CPU and GPU use the same memory chips, which means that you can load huge models into RAM (no longer VRAM, because that doesn't exist on those devices). And while Apple's integrated GPU isn't as powerful as an Nvidia GPU, it is powerful enough for non-professional workloads and has the huge benefit of access to lots of memory.
https://www.primeline-solutions.com/de/nvidia-h100-nvl-94gb-...
The model weights (billions of parameters) must be loaded into memory before you can use them.
Think of it like this: Even with a very fast chef (powerful CPU/GPU), if your kitchen counter (VRAM) is too small to lay out all the ingredients, cooking becomes inefficient or impossible.
Processing power still matters for speed once everything fits in memory, but it's secondary to having enough VRAM in the first place.
My guess is that these chips could be compute-bound though given how little compute capacity they have.
I agree that compute is likely to become the bottleneck for these new Apple chips, given they only have like ~0.1% the number of flops
FWIW I went the route of "bunch of GPUs in a desktop case" because I felt having the compute oomph was worth it.
Not in case of language models, which are typically bound by memory size rather than bandwidth.
They now bump it to 512GB. Along with insane price tag of $9499 for 512GB Mac Studio. I am pretty sure this is some AI Gold rush.
How much would it cost to get up to 512gb?
In the media composing world they use huge orchestral templates with hundreds and hundreds of tracks with millions of samples loaded into memory.
I don't think anyone commercially offers nearly this much unified memory or NPU/GPUs with anything near 512GB of memory.
The people who need the crazy resource can tie it to some need that costs more. You’d spend like $10k running a machine with similar capabilities in AWS in a month.
I'm curious to learn how AI shops are actually doing model development if anyone has experience there. What I imagined was: Its all in the "cloud" (or, their own infra), and the local machine doesn't matter. If it did matter, the nvidia software stack is too important, especially given that a 512gb M3 Ultra config costs $10,000+.
Where this hardware shines is inference (aka developing products on top of the models themselves)
Its half that of a max spec Mac Studio, but also half the price and eight times faster memory speed. Realistically which open source LLMs does 512gb over 256gb of memory unlock? My understanding is that the true bleeding edge ones like R1 won't even handle 512gb well, especially with the anemic memory speed.
I can't imagine the M3 Ultra doing well on a model that loads into ~500G, but they should be a blast on 70b models (well, twice as fast as my M3 Max at least) or even a heavily quantized 400b model.
Re memory speed, digits will be at 273GB/s while the Mac Studio is at 819GB/s
Not to mention the Mac has 6 120GB/s thunderbolt 5 ports and can easily be used for video editing, app development, etc.
The higher end NVidia workstation boxes won’t run well on normal 20amp plugs. So you need to move them to a computer room (whoops, ripped those out already) or spend months getting dedicated circuits run to office spaces.
https://en.wikipedia.org/wiki/NEMA_connector
That said, if your load is going to be a continuous load drawing 80% of the rated amperage, it really should be a NEMA 5-20 plug and receptacle, the one where one of the prongs is horizontal instead of vertical. Swapping out the receptacle for one that accepts a NEMA 5-20P plug is like $5.
If you are going to actually run such a load on a 20A circuit with multiple receptacles, you will want to make sure you're not plugging anything substantial into any of the other receptacles on that circuit. A couple LED lights are fine. A microwave or kettle, not so much.
This is not true. Standard builds (a majority) still use 15-amp circuits where 20-amp is not required by NEC.
Not exactly though.
This can have 512GB unified memory, 2x M4 Max can only have 128GB total (64GB each).
Can you do absolutely everything? No. But most models will run or retrain fine now without CUDA. This premise keeps getting recycled from the past, even as that past has grown ever more distant.
The example I always give is FFT libraries - if you compare cuFFT to rocFFT. rocFFT only just released support for distributed transforms in December 2024, something you've been able to do since CUDA Toolkit v8.0, released in 2017. It's like this across the whole AMD toolkit, they're so far behind CUDA it's kind of laughable.
I don't know if you've heard, but NVIDIA is about to add a monthly payment for additional CUDA features and I'm almost certain that many big companies will be happy to pay for them.
> But most models will run or retrain fine now without CUDA.
This is correct for some small startups, not big companies.
A 4-bit quantization of Llama-3.1 405b, for example, should fit nicely.
It's still far away from an H100 though.
That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use. They really need a 2U rack form for this though.
This hardware is really being held back by the operating system at this point.
With all my love and respect for "Apple rumors" writers; this was always "I read five blogposts about CPU design and now I'm an expert!" territory.
The speculation was based on the M3 Maxes die shots not having the interposer visible, which... implies basically nothing whether that _could have_ been supported in an M3 Ultra configuration; as evidenced by the announcement today.
No M3 has thunderbolt 5.
This is a new chip with M3 marketing. I’d expect this from Intel, not Apple.
The press-release even calls TB5 out: >Each Thunderbolt 5 port is supported by its own custom-designed controller directly on the chip.
Given that they're doing the same on A-series chips (A18 Pro with 10Gbps USB-C; A18 with USB 2.0); I imagine it's just relatively simple to swap the I/O blocks around and they're doing this for cost and/or product segmentation reasons.
Which, at this point, why not just use M4 as a base?
I imagine that making those chips is quite a bit more involved than just taking the files for M3 Max, and copy-pasting them twice into a new project.
I imagine it just takes more time to design/verify/produce them; especially given they're not selling very many of them, so they're probably not super-high-priority projects.
Or the sort of thing you put onto a successor when you had your fingers crossed that the spec and hardware would finalize in time for your product launch but the fucking committee went into paralysis again at the last moment and now your product has to ship 4 months before you can put TB 5 hardware on shelves. So you put your TB4 circuitry on a chip that has the bandwidth to handle TB5 and you wait for the sequel.
Please elucidate.
^ has a lot of elaborations on this subject
Now try the same with notes on a mac. Notes mangles the punctuation and zsh is not bash.
I love diversity in websites, and apps for that matter, but this isn't diversity, it is the uncanny valley between bespoke graphic design and homogeneity.
Say what you want about SwiftUI, but it makes consistent, good looking apps. Unless something has changed, GTK is a usability disaster.
And that's before I get into how much both X11 and wayland suck equally.
There's so much I miss about Linux, but there's so much I don't
The apps are developed by different teams. MacOS apps are containerized. Saying macOS's performance is hindered by Notes.app is like saying that Windows is hindered by Paint.exe. Notes.app is just a default[0]
[0]: though, I dislike saying this because I always feel like I need to mention that even Notes links against a hilarious amount of private APIs that could easily be exposed to other developers but... aren't.
What would robust python support oob look like?
I avoid writing python, so I’m usually the “other people” in that sentence.
That's optimistic. What if the system Python gets upgraded? For some reason, Python libraries tend to be super picky about the Python versions they support (not just Python 2 vs 3).
2. If you mean MDM, there are several good options. Screen sharing and SSH are build in.
3. In what sense?
4. `uv python install whatever` is infinitely better than upgrading on the OS vendor’s schedule.
5. What does that affect?
That's using a Linux VM. The idea people are asking about is native process isolation. Yes you'd have to rebuild Docker containers based on some sort of (small) macOS base layer and Homebrew/Macports, but hey. Being able to even run nodejs or php with its thousands of files natively would be a gamechanger in performance.
The same way Windows users run them. In a linux VM.
You don't get real on-hardware containerization.
It is definitely odd that Macs have no native container support, though, especially when you learn that Windows does.
Honestly I don't know what XNU/Darwin is good for. It doesn't do anything especially well compared to *BSD, Linux, and NT.
Not disagreeing though.
There is no such thing. Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers would you use? And how would you prevent "project collision" (where one Python project bumps into another one and one just stops working)? Example: SSL library differences across projects is a notorious culprit.
Python is garbage and I don't understand why people put up with this crap unless you seriously only run ONE SINGLE Python project at a time and do not care what else silently breaks. Having to run every Python app in its own Docker image (which is the only real solution to this, if you don't want to learn Nix, which you really should, because it is better thanks to determinism... but entails its own set of issues) is not a reasonable compromise.
Was so glad when the Elixir guys came out with this recently, to at least be able to use Python, but in a very controlled, not-insane way: https://dashbit.co/blog/running-python-in-elixir-its-fine
(Not saying Apple should bundle that, but it's the best current answer to running many different Python projects without using something like Docker)
Thats funny, about 10 years ago I started my career in a startup that had Python business logic running under Erlang (via custom connector) which handled supervision and task distribution, and it looked insane for me at the time.
Even today I think it can be useful but is very hard to maintain, and containers are a good enough way to handle python.
I disagree. My take on that is that they are an ugly enough way to handle Python. And, among other problems, don't permit you to easily mess with the code (one of many reasons why this is ugly). Need access to something stateful from the container app? That's another PITA.
And if you’re genuinely asking, everything’s converging toward uv. If you pick only one, use that and be done with it.
Neither fixed anything. They just make it slightly less painful to deal with python scripts’ constant bitrot.
They also make python uniquely difficult to dockerize.
> They also make python uniquely difficult to dockerize.
Tada, done. No, seriously. That's the whole invocation.Do you actually try new Python projects out with git-clone, or do you just use the same 3 Python projects for years at a time (all regularly)?
That might explain the difference in experiences
What am I missing?
Also, typically when people say things like
> Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers
It means they have been trapped in a cycle of thinking "just one more tool will surely solve my problem", instead of realising that the tools _are_ the problem, and if you just use the official methods (virtualenv and pip from a stock python install), things mostly just work.
There are reasons to want something more featureful than plain pip. Even without them, pip+virtualenv has been completely usable for, what, 15 years now?
Here's a question- If you don't touch a project in 1 year, do you expect it to still work, or not? If your answer is the latter, then we simply won't see eye-to-eye on this.
that's not good enough. If I'm in the business of writing Python code, I (ideally) don't want to _also_ be in the business of working around Python design deficiencies. Either solve the problem definitively, or do not try to solve the problem at all, because the middle road just leads to endless headaches for people WHILE ALSO disincentivizing a better solution.
Node has better dependency management than Python- And that's really saying something.
The thing is, most people who are writing python code are not in the business of writing python code. They're students, scientists, people with the word "business" or "analyst" in their title. They have bigger fish to fry than learning a different language ecosystem.
It took 30 years to get them to switch from excel to python. I think it's unrealistic to expect that they're going to switch from python any time soon. So for better or worse, these are problems that we have to solve.
I've had plenty of problems with node, for example. You mentioned nix, which is much better, but also comes with tons of hard trade-offs.
If a packaging tool doesn't do what i wanted, but I can understand why, and ultimately the tool is not to blame, that's fine by me. The issues I can think of fit reasonably well within this scope:
- requirement version conflicts: packages are updated by different developers, so sometimes their requirements might not be compatible with each other. That's not pip's fault, and it tells you what the problem is so you can resolve it.
- code that's not compatible with updated packages: this is mainly down to requirement versions which are specified too loosely, and not the fault of pip. If you want to lock dependencies to exact versions (like node does by default) you can do this too (with requirements.txt). It's a bit harsh to blame pip for not doing this for you, it's like blaming npm for not committing your package.lock. It would be better if your average python developer was better at this.
- native library issues: some packages depend on you having specific libraries (and versions thereof) installed, and there's not much that pip can do about that. This is where your "ssl issues" come from. This is pretty common in python because it's used so much as "glue" between native libraries -- all the most fun packages are wrappers around native code. This has got a lot better in the past few years with manylinux wheels (which include native libraries). These require a lot of non-python-specific work to build, so i don't blame pip where they don't exist.
It's not perfect, but it's not a big enough deal to rant about or reject entirely if you would otherwise get a lot of value out of the ecosystem.
Honest question: why do you want this in MacOS? Do you understand what docker does? (it's fundamentally a linux technology, unless you are asking for user namespaces and chroot w/o SIP on MacOS, but that doesn't make sense since the app sandbox exists).
MacOS doesn't have the fundamental ecosystem problems that beget the need for docker.
If the answer is "I want to run docker containers because I have them" then use orbstack or run linux through the virtualization framework (not Docker desktop). It's remarkably fast.
Anyone wanting to run and manage their own suite of Macs to build multiple massive iOS and Mac apps at scale, for dozens or hundreds or thousands of developers deploying their changes.
xcodebuild is by far the most obvious "needs native for max perf" but there are a few other tools that require macOS. But obviously if you have multiple repos and apps, you might require many different versions of the same tools to build everything.
Sounds like a perfect use case for native containers.
I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive.
I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side.
My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact the ssh server is only available on good days and only after Remote Desktop logged in at least once. Launchd makes you wanna crave systemd. etc, etc.
So, about docker. I would absolutely love to run my app in a containerized environment on a Mac in order to not touch the main OS.
Of course, I had a LOM/KVM and redundant networking etc. They were substantially more reliable than the Dell equipment that I used in my day job for sure.
Software-wise it's much different to an expected behavior. For example, macOS won't let you in over SSH until you log in via Remote Desktop. You'll get "connection closed" immediately.
Or sometimes it will.
And that depends not on the count of connection attempts or anything you can do locally but rather on the boot process somehow. Sometimes it boots in a way that permits ssh, sometimes not. The same computer, the same OS.
Then after you login on screen sharing and log out, macOS will let you in over ssh. For a few days. And then again will force you to login via GUI. Or maybe not. I have no idea what makes it.
I have trouble reading macOS logs or understanding it. It spews a few log messages per second even idle. If you grep ssh these messages contain zero actionable data, like "unsuccessful attempt" or similar.
Another complaint is that launchd reports the same "I/O error" on absolutely all error situations, from syntax error in plist to corrupt binary. Makes development and debugging of launchagents very fun.
In Linux, it means something very specific: a user/mount/pid/network namespace, overlayfs to provide a rootfs, chroot to pivot to the new root to do your work, and port forwarding between the host/guest systems.
On MacOS I don't know what containerization means short of virtualization. But you have virtualization on MacOS already, so why not use that?
Also I want to run the latest OS with all security patches on the host while having a stable and known macOS version in a container given how developer-hostile Apple is.
Or you can package your app as a .app and not worry about it, there's no "pollution" when everything is bundled.
Former Transmission user here.
I realise you didn't ask, but you might find some improvements in qBittorrent.
Transmission is just a small, floating window with your downloads. Click for more. It fits in the macOS vibe. But I'm a person that fully adopted the original macOS "way of working" - kicked the full-screen habit I had in windows and never felt better.
Can I ask, why would you go FROM Transmission to qBittorrent?
In my case: some torrents wouldn't find known-good seeds in Transmission but worked fine in qBittorrent; there's reasonable (but not perfect) support for libtorrent 2.0 in qBittorrent; my download speeds and overall responsiveness is anecdotally better in qBittorrent, and; I make use of some of the nitty gritty settings in qBittorrent.
And let's be clear, it wasn't the app that had problems, the Apple Remote Desktop connection to the machine failed when the speeds got above 40MB/s and the network interface stopped working around 80MB/s.
I think Transmission works perfectly fine. I've been using it for 10+ years with no issues at all on Linux.
I forgot to mention this is a Mac mini/Intel (2018).
The Apple ecosystem is a walled garden.
what internal AI efforts?
Apple Intelligence is bunkers, and Apple MLX framework remains a hobby project for Apple
https://security.apple.com/documentation/private-cloud-compu...
https://techcrunch.com/2024/12/11/apple-reportedly-developin...
https://techcrunch.com/2025/02/24/apple-commits-500b-to-us-m...
If so, this is hardly a hobby project.
It may not be effective, but there is serious cash behind this.
They are using OpenAI for now but in couple months they will own the full chain of value.
1. Small models running locally with well-established tool interfaces (“app intents”)
2. Large models running in a bespoke cloud that can securely and quickly load all relevant tokens from a device before running inference
No AI lab is even close to what Apple is trying to deliver in the next ~12 months
https://www.bestbuy.com/site/apple-macbook-pro-14-laptop-m3-...
Apple is very good at marketing, a lot less at delivering actual value, especially if it's not about hardware.
It’s their spin of the Google strategy of targeting providjng services to their enterprise GCP customer. I think we’ll see more out of them long term.
They also own distribution to the wealthiest and most influential people in the world
Don’t get lost in recency bias
> Apple’s custom-built UltraFusion packaging technology uses an embedded silicon interposer that connects two M3 Max dies across more than 10,000 signals, providing over 2.5TB/s of low-latency interprocessor bandwidth, and making M3 Ultra appear as a single chip to software.
The comment was that the press had reported that the interposer wasn't available. This obviously uses some form of interposer, so the question is if the press missed it, or Apple has something new.
It sounds like they're using TSMC's new LSI (Local Si Interconnect) technology, which is their version of Intel's EMIB. It's essentially small islands of silicon, just around the inter-chip connections, embedded within the organic substrate. This gives the advantages of silicon interconnect, without the cost and size restrictions of a silicon interposer. It would not be visible from just looking at the package.
https://www.anandtech.com/show/16031/tsmcs-version-of-emib-l...
https://semianalysis.com/2022/01/06/advanced-packaging-part-...
I just want a break from MacOS, I'll be buying a Thinkpad and will probably never come back. This isn't my moaning, I understand it's their market, but if their hardware supported Linux (especially dual booting) or Docker native, I'd probably be buying Apple for the next decade and now I just won't be.
Since the M series of ARM processors didn’t come out until 2020, that would make a lot of sense.
I'm sure with tinkering I could eventually get it working, but I'm well past the point of wanting to tinker with hardware and drivers to get Linux working.
The HP Elitebook was on Ubuntu's list of compatible tested laptops and came in hundreds of dollars less than a Thinkpad. Most of the comparably priced on sale T14's I could find were all crap Intel spec'd ones.
Months in I don't regret it at all and Linux support has been fantastic even for a fairly newer Ryzen chip and not the latest kernel. (I stick to LTS releases of most Distros) Shoving in 4TB of NVME storage and 96GB of DDR5 should I feel the need to upgrade would still put me only around $1300 invested in this machine.
I'll miss the battery life of the M1 chips, and I'm going to have to re-learn how to type (CTRL instead of ALT, fn rarely being on the left, I use fn+left instead of CTRL A in terminals) but otherwise, I think I'm done.
I just want decent enough power and no thermal throttling if I do have to hammer it. I make music so the extra ram and space for sample libraries is a big benefit and why I had to keep external SSD's around with my Macs.
My Macbook Air needed a usb fan ziptied to the laptop stand to not throttle at times.
>it seems like rather than trying to give a mac a reasonable go of it as opposed to whatever else, you were trying to instead explore a fundamental difference in how you value technology products
I re-evaluate how I feel about technology pretty often and its caused some shifts for sure. My side hobby is ARM/RiscV low power computing and Apple's move to ARM tickled that hyper efficiency side of my brain, but ultimately failed to keep me interested because of all the downsides upgrade/repairability wise.
Not that I'm discouraging you from switching or anything. If Linux is what you want/need, there's definitely better laptops to be had than a Macbook for that purpose. It's just that weird incompatibilities and having to fight with the operating system on random issues is, at least in my experience, normal when using a linux laptop. Even my T480 which has overall excellent compatibility isn't trouble-free.
Apple Silicon chips are arguably more compatible with Asahi Linux [1], but that's largely in thanks to the hard work of Marcan, who's stepped down as project lead from the project [2].
Overall I still think the right choice is to find a laptop better suited for the purpose of running linux on it, just something that requires more careful consideration than people think. Framework laptops, which seem well suited since ideologically it meshes well with linux users, can be a pain to set up as well.
[1] https://asahilinux.org/
[2] https://marcan.st/2025/02/resigning-as-asahi-linux-project-l...
Apple could either create a 2U rack hardware and support Linux (and I mean Apple supporting it, not hobbysts), or have a build of Darwin headless that could run on that hardware. But in the later case, we probably wouldn't have much software available (though I am sure people would eventually starting porting software to it, there is already MacPorts and Homebrew and I am sure they could be adapted to eventually run in that platform).
But Apple is also not interested in that market, so this will probably never happen.
they're just a tiny company with shareholders who are really tired of never earning back their investments. give 'em a break. I mean they're still so small that they must protect themselves by requiring that macs be used for publishing iPhone and iPad applications.
I mean, that's usually how it works though. When IBM launched the PS/2, they didn't support anything other than PC-DOS and OS/2, Microsoft had to make MS-DOS work for it (I mean... they did get support from IBM, but not really), the 386BSD and Linux communities brought the engineering effort without IBM's involvement.
When Apple was making Motorola Macs, they may have given Be a little help, but didn't support any other OSes that appeared. Same with PowerPC.
All of the support for alternative OSes has always come from the community, whether that's volunteers or a commercial interest with cash to burn. Why should that change for Apple silicon?
Apple briefly supported a Linux distribution on PowerPC Macs: https://en.wikipedia.org/wiki/MkLinux.
> they're still so small that they must protect themselves by requiring that macs be used for publishing iPhone and iPad applications.
They're not talking about Apple's silicon as a target, but as a development platform.
https://news.ycombinator.com/item?id=43271486
Lowest latency of DDR5-6400 on normal PC starting at 60ns+
Lowest latency of VRAM on GeForce RTX 4090 starting at 14 ns
Lowest latency of Apple M1 Memory starting at 5 ns, its more like L3 cache
And on Apple M chip, this ultrafast memory is available for CPU, GPU and NPU.
https://www.anandtech.com/show/17024/apple-m1-max-performanc... https://chipsandcheese.com/p/microbenchmarking-nvidias-rtx-4...
There is also "parity" in other products like a MacBook Pro from £1,599 / $1,599 or an iPhone 16 from £799 / $799. £9,699 / $9,499 is worse than that!
You know that £9,699 are over $12k, right?
For SMBs or Edge deployments where redundancy isn't as critical or budgets aren't as large, this is an incredibly compelling offering...if Apple actually had a competent server OS to layer on top of that hardware, which it does not.
If they did, though...whew, I'd be quaking in my boots if I were the usual Enterprise hardware vendors. That's a damn frightening piece of competition.
In Intel's case, there's ample coverage of the company's lack of direction and complacency on existing hardware, even as their competitors ate away at their moat, year after year. AMD with their EPYC chips taking datacenter share, Apple moving to in-house silicon for their entire product line, Qualcomm and Microsoft partnering with ongoing exploration of ARM solutions. A lack of competency in leadership over that time period has annihilated their lead in an industry they used to single-handedly dictate, and it's unlikely they'll recover that anytime soon. So in a sense, Intel cannot make a similar product, in a timely manner, that competes in this segment.
As for AMD, it's a bit more complicated. They're seeing pleasant success in their CPU lineup, and have all but thrown in the towel on higher-end GPUs. The industry has broadly rallied around CUDA instead of OpenCL or other alternatives, especially in the datacenter, and AMD realizes it's a fool's errand to try and compete directly there when it's a monopoly in practice. Instead of squandering capital to compete, they can just continue succeeding and working on their own moat in the areas they specialize in - mid-range GPUs for work and gaming, CPUs targeting consumers and datacenters, and APUs finding their way into game consoles, handhelds, and other consumer devices or Edge compute systems.
And that's just getting into the specifics of those two companies. The reality is that any vendor who hasn't already unveiled their own chips or accelerators is coming in at what's perceived to be the "top" of the bubble or market. They'd lack the capital or moat to really build themselves up as a proper competitor, and are more likely to just be acquired in the current regulatory environment (or lack thereof) for a quick payout to shareholders. There's a reason why the persistent rumor of Qualcomm purchasing part or whole of Intel just won't die: the x86 market is rather stagnant, churning out mediocre improvements YoY at growing pricepoints, while ARM and RISC chips continue to innovate on modern manufacturing processes and chip designs. The growth is not in x86, but a juggernaut like Qualcomm would be an ideal buyer for a "dying" or "completed" business like Intel's, where the only thing left to do is constantly iterate for diminishing returns.
That nvidia GPU setup will actually have the compute grunt to make use of the RAM, though, which this M3 Ultra probably realistically doesn't. After all, if the only thing that mattered was RAM then the 2TB you can shove into an Epyc or Xeon would already be dominating the AI industry. But they aren't, because it isn't. It certainly hits at a unique combination of things, but whether or not that's maximally useful for the money is a completely different story.
It all depends on the workload you want to run.
Sure, the desktop machine performs better, as would a datacenter server jam-packed full of Blackwell GPUs, but that's not what's exciting about Apple's implementation. It's the efficiency of it all, being able to handle modern models on comparatively "weaker" hardware most folks would dismiss outright. That's the point I was trying to make.
Also Apple isn't unique in having an NPU in a laptop. Fucking everyone does at this point.
I am thinking that 7 A100's would be the lowest price for that, and that would be $80k with good discounts.
The point is that, in terms of practical usage, the M3 Ultra is uniquely competent and highly affordable in a sea of enterprise technology that is decidedly not. I tried to demonstrate why I'm excited about it by pointing out the similar performance of a battery-powered, four-year-old laptop and a quite gargantuan gaming PC that's pulling over 500W from the wall, as an example of what several years of additional refinements and improvements to the architecture was expected to bring.
The point is that it's affordable, more flexible in deployment, and more efficient than similarly-specced datacenter servers specifically designed for inference. For the cost of a single decked-out Dell or HP rackmount server, I can have five of these Mac Studios with M3 Ultra chips - and without the need for substantial cooling, noise isolation, or other datacenter necessities. If the marketing copy is even in the same ballpark as actual performance, that's easily enough inference to serve an office of fifty to a hundred people or more, depending on latency tolerances; if you don't mind "queuing" work (like CurrentCo does with their internal Agents), one of those is likely enough for a hundred users.
That's the excitement. That's the point. It's not the fastest, it's not the cheapest, it's just the most balanced.
I have Apple hardware but it sucks for anything AI, buying it for that purpose is just extremely dumb, just like buying Macs for engineering CADs or things of the sort.
If you are buying Macs and it's not for media production related reasons you are doing something wrong.
I continue to be in awe of the lengths some people will go just to fling insults and shake out some salt. We're, what, ten layers deep? With all the context above, the best you have to contribute to the discussion are baseless accusations and ageist insults?
Your finite time would have been better spent on literally anything else, than actively seeking out a comment just to throw subjective, unsubstantiated shade around. C'mon, be better.
Apple is the master at creating desire and building narrative in their customers' mind about the many things their devices would allow them to do. It's very aspirational and in practice most of the Macs get used for things that could have been done with a much cheaper option.
It may not be obvious to you but it's somewhat funny seeing you rationalise all kinds of dreams of what this machine could potentially be when in practice the people who would really be working on the kind of stuff you are talking about don't even consider them viable for many good reasons.
It's not that those machines cannot potentially do it, it's just that they don't really fit the goal very well.
A lot like people buying Cybertruck to "haul" stuff when they are a lot more option that are just plain better and make a lot more economic/practical sense.
It's OK to desire the thing and be excited about it but it really doesn't serve anyone to rationalise it so hard, you are lying to yourself as much as everyone else, it's not healthy.
If that was not clear, people working on AI stuff professionally really don't have to deal with a Mac Studio, they have access to better stuff. If you want to get one personally to experiment/toy around it's ok but it's not going to be this amazing thing for AI.
The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.
[1]: https://asahilinux.org/2025/02/passing-the-torch/
My understanding is there are dozens of people working on it.
https://bsky.app/profile/lina.yt
https://github.com/AsahiLina
On the other hand, you can do without display support if you’re only using it as a server. And I think USB Ethernet dongles might work for the time being?
Apple’s whole m.o. is to take FOSS software, repackage it and sell it. They don’t want people using it directly.
I’d be curious to know if this changes that. It’d take a lot more than doubling cores to take out the very high power AMD parts, but this might squeeze them a bit.
Interestingly, AMD has also been investing heavily in unified RAM. I wonder if they have / plan an SoC that competes 1:1 with this. (Most of the parts I’m referring to are set up for discrete graphics.)
Source: https://www.notebookcheck.net/AMD-Ryzen-AI-Max-395-Analysis-...
Cinebench 2024 results.
Somewhere on the internet there is a tdp wattage vs performance x-y plot. There’s a pareto optimal region where all the apple and amd parts live. Apple owns low tdp, AMD owns high tdp. They duke it out in the middle. Intel is nowhere close to the line.
I’d guess someone has made one that includes datacenter ARM, but I’ve never seen it.
Workstations (like the Mac Studio) have traditionally been a space where "enthusiast"-grade consumer parts (think Threadripper) and actual server parts competed. The owner of a workstation didn't usually care about their machine's TDP; they just cared that it could chew through their workloads as quickly as possible. But, unlike an actual server, workstations didn't need the super-high core count required for multitenant parallelism; and would go idle for long stretches — thus benefitting (though not requiring) more-efficient power management that could drive down baseline TDP.
Anyway, I don't think it's comparable really. This thing comes with a fat GPU, NPU, and unified memory. Threadripper is just a CPU.
At least judging by the mounts, they want them to be used that way, even though the CPU might not fit with the de facto industry label for "server-grade".
The only use case I can think of is for audio workstations, where people have lots of rack mount equipment, so you can have everything including the computer in the rack. But even for that use case it's quite big.
Anyway the Apple config in the article costs about 5x more than a comparable low end AMD server with 512GB of ram, but adds an NPU. AMD has NPUs in lower end stuff; not sure about this TDP range.
Right.
It is coming up because we're in a thread about using them as server CPUs. (c.f. "colo", "2U" in OP and OP's child), and the person you're replying to is making the same point you are
For years now, people will comment "these are the best chips, I'd replace all chips with them."
Then someone points out perf/watt is not perf.
Then someone else points out some M-series is much faster than a random CPU.
And someone else points out that the random CPU is not a top performing CPU.
And someone else points out M-series are optimized for perf/watt and it'd suck if it wasn't.
I love my MacBook, the M-series has no competitors in the case it's designed for.
I'd just prefer, at this point, that we can skip long threads rehashing it.
It's a great chip. It's not the fastest, and it's better for that. We want perf/watt in our mobile devices. There's fundamental, well-understood, engineering tradeoffs that imply being great at that necessitates the existence of faster processors.
[1] https://www.cpubenchmark.net/singleThread.html
There are nice things that Apple has, but as you can see there is significant reality warping going on.
Why does it persist?
Both Passmark and Geekbench are aggregates of a variety of tasks. If you dig into the individual tests that constitute this aggregate score, you will find different platforms perform better, or worse, on certain tests than others. I would wager that, for many applications, only a subset of these tasks are relevant to the performance of the application, yet such benchmark suites distil out all nuance into a single value.
Here is a personal anecdote. I have tried running CASTEP (built from source), a density functional theory calculator, on both an M1 Max MacBook Pro [0], and on a Ryzen 7840HS Lenovo laptop [1]. A cursory glance at those Geekbench results linked might make you expect that the performance is roughly equivalent, but the Ryzen outperforms the Mac by about 4x, a huge difference.
What happens if we try and dig into any particular benchmark to explain this? If you click on any particular benchmark in the Geekbench search lists, you will see they test things like "File Compression", "HTML5 Browser", "Clang". Which of these maps most closely to the sorts of instructions used in CASTEP? Your guess is as good as mine.
If anything, I would say Passmark is quite a bit less abstract about this. Looking at the Mac [2] and Ryzen [3] Passmark results, you can see the Ryzen outperforms the Mac by about 2x on "extended instructions", which appear to involve some matrix math, and also about 2x on "integer math". The Mac, meanwhile, appears to be extremely good at finding prime numbers, at over 3x the speed of the Ryzen. Presumably the Ryzen's balance of instruction performance is more useful for DFT calculations than the Mac's, which perhaps is weaker in areas that might matter for this application, but stronger in areas that might matter for others.
Of course, optimization is likely a component of this. How much effort is put into the OpenBLAS, MPI, etc, implementations on aarch64 darwin vs. x86-64 linux? This is a good question. It is, however, mostly irrelevant to the end consumer, who wishes to consume this software for use in their further research, rather than dig into high-performance computing library optimization.
[0] https://browser.geekbench.com/search?q=7840hs
[1] https://browser.geekbench.com/search?q=m1+max
[2] https://www.cpubenchmark.net/cpu.php?cpu=Apple+M1+Max+10+Cor...
[3] https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+7+PRO+784...
I think PassMark is more honest as well, because it just gives scores for calculation throughput instead of specific tasks. It more closely matches what experience you will get if you have a varied load.
But since it's Apple we are talking about, their users just want to think they have the best and that's all that matters.
https://www.cpubenchmark.net/cpu_test_info.html
Right from the top it's amateurish stuff: their idea of an integer benchmark to measure "raw" CPU throughput (whatever that means) is to make a bunch of random ints and add/subtract/multiply/divide them.
Very few programs do a high volume of either integer multiply or divide. And when they do, they generally aren't doing it on random numbers. This is the kind of thing which gives synthetic benchmarks their highly deserved bad rep. It might be even worse than Dhrystone MIPs, and believe me, in benchmark nerd circles, that is a fucking diss.
If you look up Geekbench's docs, you'll find that it's all about real-world compute tasks. For example, one of the int tests in their suite is to compile a reference program with the Clang compiler. Compilers are a reasonably good litmus test of integer performance; they heavily stress the CPU features most responsible for high integer performance in this day and age. (Branch prediction, memory prefetching, out-of-order execution, speculation, that kind of thing.)
You claimed that PassMark reflects "complex" software, and Geekbench doesn't. However, I would be willing to bet that Clang alone is far more complex than all of PassMark's CPU benchmarks put together, whether you measure by SLOC or program structure.
Note that none of this has anything to do with Mac vs PC. Passmark is simply a bad benchmark that should not be used, period. That said, there are a bunch of warning signs that PassMark's ports to everything outside its native x86 Windows are probably quite sloppy, so it's even less useful for crossplatform comparisons.
https://medium.com/silicon-reimagined/performance-delivered-...
Passmark is an outdated benchmark that isn't updated to use ARM instructions.
The M4 Max had great, I would argue the best at time of release, single core results on Geekbench.
That is a different claim from M4 line has the top single thread performance in the world.
I'm curious:
You're signalling both that you understand the fundamental tradeoff ("Apple doesn't make server-grade CPUs") and that you are talking about something else (follow-up with M4 family has top single-thread performance)
What drives that? What's the other thing you're hoping to communicate?
If you are worried that if you leave it at "Apple doesn't make server-grade CPUs", that people will think M4s aren't as great as they are, this is a technical-enough audience, I think we'll understand :) It doesn't come across as denigrating the M-series, but as understanding a fundamental, physically-based, tradeoff.
This?
https://www.videocardbenchmark.net/power_performance.html#sc...
I'm not sure how those benchmarks translate to common real world use cases.
Having things consistently work is much cheaper than down days caused by your ancient equipment. Apple’s SSDs will make it to 5 years no problem - and more likely, 10-15 years.
Fast forward 2 years: The $50-$250K machines have a 100% drive failure rate, and some poor bastard has to fly from data center to data center to swap the $60 drive for a $120 one, then re-rack and re-image each machine.
Anyway, soldering a decent SSD to the motherboard board would actually improve reliability at all those places.
If they were soldered onto those systems you talk about, all those would have had to be replaced instead of just having the drive swapped out and re-imaged.
It reminds me of the 1990s when my old school was using Sun machines based on the 68k series and later SPARC and we were blown away with the toaster-sized HP PA RISC machine that was used for student work for all the CS classes.
Then Linux came out and it was clear the 386 trashed them all in terms of value and as we got the 486 and 586 and further generations, the Intel architecture trashed them in every respect.
The story then was that Intel was making more parts than anybody else so nobody else could afford to keep up the investment.
The same is happening with parts for phones and TSMC's manufacturing dominance -- and today with chiplets you can build up things like the M3 Ultra out of smaller parts.
[1] https://aws.amazon.com/ec2/instance-types/r8g/
Then, one day, we built a 5 machine amd athlon xp linux cluster for $2000 ($400/machine) that beat all the unix and windows server hardware by at least 10x on $/perf.
It’s nice that we have more than one viable cpu vendor these days, though it seems like there’s only one viable fab company.
For what we needed, five 32 bit address spaces was enough DRAM. The individual CPU parts were way more than 20% as fast, and the 100Mbit switch was good enough.
(The data basically fit in ram, so network transport time to load a machine was bounded by 4GiB / 8MiB / sec = 500 seconds. Also, the hard disks weren’t much faster than network back then.)
I think it goes something like:
The EV6 is a bit quirky as it is 43-bit by default, but can use 48-bits when I_CTL<VA_48> or VA_CTL<VA_48> is set. (the distinction of the registers is for each access type, i.e: instruction fetch versus data load/store)The 21364/EV7 likely has the same characteristics as EV6, but the hardware reference manual seems to have been lost to time...
Digital struggled with the microprocessor transition because they didn't want to kill their cash cow minicomputers with microcomputer-based replacements. They went with the 64-bit Alpha because they wanted to rule the high end in the CMOS age. And they did, for a little while. But the mass market caught up.
VMS is the only OS (that I know of) that uses all 4 processor privilege modes.
Side note: The 21064 has such bizarre IPR mappings, the read values have lots of bits scrambled around compared to their write counterparts. This is likely a hardware design decision affecting the programmer's model, if I had to guess.
One thing I remember about Alpha though was how bad the output from gcc was. Then DEC released a version of their own compilers that was command line compatible with gcc. That changed everything for open source stuff.
Maybe not at the same power consumption, but I'm sure mid-range Xeons and EPYCs mop the floor with the M3 Ultra in CPU performance. What the M3 Ultra has that nobody else comes close is a decent GPU near a pool of half a terabyte of RAM.
FYI Apple runs Linux in their DC, so no Apple hardware in their own servers.
This is silly. Given the performance per watt, the M series would be great in a data center. As you all know, electricity for running the servers and cooling for the servers are the two biggest ongoing costs for a data center; the M series requires less power and runs more efficiently than the average Intel or AMD-based server.
> FYI Apple runs Linux in their DC, so no Apple hardware in their own servers.
That's certainly no longer the case. Apple announced their Private Cloud Compute [1] initiative—Apple designed servers running Apple Silicon to support Apple Intelligence functions that can't run on-device.
BTW, Apple just announced a $500 billion investment [2] in US-based manufacturing, including a 250,000 square foot facility to make servers. Yes, these will obviously be for their Private Cloud Compute servers… but it doesn't have to be only for that purpose.
From the press release:
As part of its new U.S. investments, Apple will work with manufacturing partners to begin production of servers in Houston later this year. A 250,000-square-foot server manufacturing facility, slated to open in 2026, will create thousands of jobs.
Previously manufactured outside the U.S., the servers that will soon be assembled in Houston play a key role in powering Apple Intelligence, and are the foundation of Private Cloud Compute, which combines powerful AI processing with the most advanced security architecture ever deployed at scale for AI cloud computing. The servers bring together years of R&D by Apple engineers, and deliver the industry-leading security and performance of Apple silicon to the data center.
Teams at Apple designed the servers to be incredibly energy efficient, reducing the energy demands of Apple data centers — which already run on 100 percent renewable energy. As Apple brings Apple Intelligence to customers across the U.S., it also plans to continue expanding data center capacity in North Carolina, Iowa, Oregon, Arizona, and Nevada.
[1]: https://security.apple.com/blog/private-cloud-compute/
[2]: https://www.apple.com/newsroom/2025/02/apple-will-spend-more...
Security practitioners and academics: https://github.com/enovella/TEE-reversing/ https://media.ccc.de/v/36c3-10859-trustzone-m_eh_breaking_ar... https://www.blackhat.com/docs/us-15/materials/us-15-Shen-Att... https://i.blackhat.com/USA-19/Thursday/us-19-Peterlin-Breaki...
Sorry you could not cite an attack demonstrated on x86 and not on ARM or vice versa. Well, maybe TikTag (https://github.com/compsec-snu/tiktag) was demonstrated against ARM MTE, since you mention, and Pacman (https://pacmanattack.com/) against PAC. I was hoping for an actual response, no offense.
Our business "only" sees about 1,000-25,000 req/min, our message brokers transmit MAX 25k msg/s. Easily handled by a rack of 10 servers for redundancy.
We are not Google and we don't pretend to be, so we don't care about power, as the difference is a few dollars a month.
It really is. Even if they themselves won't bring back their old XServe OS variant, I'd really appreciate it if they at least partnered with a Linux or BSD (good callout, ryao) dev to bring a server OS to the hardware stack. The consumer OS, while still better (to my subjective tastes) than Windows, is increasingly hampered by bloat and cruft that make it untenable for production server workloads, at least to my subjective standards.
A server OS that just treats the underlying hardware like a hypervisor would, making the various components attachable or shareable to VMs and Containers on top, would make these things incredibly valuable in smaller datacenters or Edge use cases. Having an on-prem NPU with that much RAM would be a godsend for local AI acceleration among a shared userbase on the LAN.
I’m continually surprised Apple doesn’t just donate something like 0.1% of their software development budget to proton and the asahi projects. It’d give them a big chunk of the gaming and server markets pretty much overnight.
I guess they’re too busy adding dark patterns that re-enable siri and apple intelligence instead.
Don't get me wrong, I wouldn't use that feature (I prefer self-hosting it all myself), but for folks like my family members, it'd be a killer addition to the lineup that makes my life supporting them much easier.
HomeKit networking existed in Eero briefly. I put that in a lot of casual Apple homes. Seemed like missed oppty for Apple to let Amazon buy Eero, a more "spiritual successor" to the Airports.
Ubiquiti is decently priced, especially for niche hardware, unlike Apple. Fundamentally they diverge on the way to do things...
It's worth adding the M3 Ultra has 819GB/s memory bandwidth [1]. For comparison the RTX 5090 is 1800GB/s [2]. That's still less but the M4 Mac Minis have 120-300GB/s and this will limit token throughput so 819GB/s is a vast improvement.
For $9500 you can buy a M3 Ultra Mac Studio with 512GB of unified memory. I think that has massive potential.
[1]: https://www.apple.com/mac-studio/specs/
[2]: https://www.nvidia.com/en-us/geforce/graphics-cards/50-serie...
https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...
This will run it at 40 tokens/s based on rough calculation. Q4 quant. 37b active parameters.
5x higher price for 10x higher performance.
If you've ever used git, svn, or an IDE side by side on corporate Windows versus Apple I don't know why you would ever go back.
> If you've ever used git, svn, or an IDE side by side
I still reach for Windows, even though it's a dogshit OS. I would rather use WSL to write and deploy a single app, as opposed to doing my work in a Linux VM or (god forbid) writing and debugging multiple versions just to support my development runtime. If I'm going to use an ad-encumbered commercial service-slop OS, I might as well pick the one that doesn't actively block my work.
The custom build would work great though, and even moreso in a server room and as well continues to reveal by comparison how excessively Apple prices it's components.
I assume that there's a community of developers focusing on leveraging this hardware instead of complaining about the operating system.
I love Apple but they love to speak in half truths in product launches. Are they saying the M3 Ultra is their first Thunderbolt 5 computer? I don't recall seeing any previous announcements.
(NB: I've been long on AAPL since $7 a share but I'm also allergic to bullshit)
M1 Max - 24 to 32 GPU cores
M2 Max - 30 to 38 GPU cores
M3 Max - 30 to 40 GPU cores
M4 Max - 32 to 40 GPU cores
I also looked up the announcement dates for the Max and the Ultra variant in each generation.
M1 Max - October 18, 2021
M1 Ultra - March 8, 2022
M2 Max - January 17, 2023
M2 Ultra - June 5, 2023
M3 Max - October 30, 2023
M3 Ultra - March 12, 2025
M4 Max - October 30, 2024
> My guess is that Apple developed this chip for their internal AI efforts
As good a guess as any, given the additional delay between the M3 Max and Ultra being made available to the public.
The theory that the M3 Ultra was being produced, but diverted for internal use makes as much sense as any theory I've seen.
It makes at least as much sense as the "TSMC had difficulty producing enough defect free M3 Max chips" theory.
what's the point of 512GB RAM for LLMs on this Mac Studio if the speed is painfully slow?
it's as if Apple doesn't want to compete with Nvidia... this is really disappointing in a Mac Studio. FYI: M2 Ultra already has 800GB/s bandwidth
NVIDIA RTX 4080: ~717 GB/s
AMD Radeon RX 7900 XTX: ~960 GB/s
AMD Radeon RX 7900 XT: ~800 GB/s
How's that slow exactly ?
You can have 10000000Gb/s and without enough VRAM it's useless.
Nvidia RTX 4090 (Ada Lovelace)
FP32: Approximately 82.6 TFLOPS
FP16: When using its 4th‑generation Tensor Cores in FP16 mode with FP32 accumulation, it can deliver roughly 165.2 TFLOPS (in non‑tensor mode, the FP16 rate is similar to FP32).
FP8: The Ada architecture introduces support for an FP8 format; using this mode (again with FP32 accumulation), the RTX 4090 can achieve roughly 330.3 TFLOPS (or about 660.6 TOPS, depending on how you count operations).
Apple M1 Ultra (The previous‑generation top‑end Apple chip)
FP32: Around 15.9 TFLOPS (as reported in various benchmarks)
FP16: By similar scaling, FP16 performance would be roughly double that value—approximately 31.8 TFLOPS (again, an estimate based on common patterns in Apple’s GPU designs)
FP8: Like the M3 family, the M1 Ultra does not support a dedicated FP8 precision mode.
So a $2000 Nvidia 4090 gives you about 5x the FLOPS, but with far less high speed RAM (24GB vs. 512GB from Apple in the new M3 Ultra). The RAM bandwidth on the Nvidia card is over 1TBps, compared with 800GBps for Apple Silicon.
Apple is catching up here and I am very keen for them to continue doing so! Anything that knocks Nvidia down a notch is good for humanity.
I don't love Nvidia a whole lot but I can't understand where this sentinent comes from. Apple abandoned their partnership with Nvidia, tried to support their own CUDA alternative with blackjack and hookers (OpenCL), abandoned that, and began rolling out a proprietary replacement.
CUDA sucks for the average Joe, but Apple abandoned any chance of taking the high road when they cut ties with Khronos. Apple doesn't want better AI infrastructure for humanity; they envy the control Nvidia wields and want it for themselves. Metal versus CUDA is the type of competition where no matter who wins, humanity loses. Bring back OpenCL, then we'll talk about net positives again.
M3 Max GPU benchmarks around 14 TFLOPs, so the Ultra should score around 28 TFLOPs.
Double the numbers for FP16.
vram is not really the limiting factor for serious actors in this space
"40 tokens/s by my calculations"
to
"40 tokens/s"
to
"41 tokens/s"
Is there a dice involved in "your calculations?"
Doesn't matter. All theorized because no one has publicly tested one.
Why? They have too many M3 chips on stock?
Can you elaborate on how the TOPS value is inflated? What GPU would be the equivalent of the Jetson AGX Orin?
In terms of software, recent NVIDIA and AMD research has focused on fast evaluation of small ~4 layer MLPs using FP8 weights for things like denoising, upscaling, radiance caching, and texture and material BRDF compression/decompression.
NVIDIA has just put out some new graphics API extensions and samples/demos for loading a chunk of neural net weights and performing inference from within a shader.
(Maybe I'm missing something here.)
Additionally, I would assume this is a very low-volume product, so it being on N3B isn't a dealbreaker. At the same time, these chips must be very expensive to make, so tying them with luxury-priced RAM makes some kind of sense.
Either that or kill the Mac Pro altogether, the current iteration is such a half-assed design and blatantly terrible value compared to the Studio that it feels like an end-of-the-road product just meant to tide PCIe users over until they can migrate everything to Thunderbolt.
They recycled a design meant to accommodate multiple beefy GPUs even though GPUs are no longer supported, so most of the cooling and power delivery is vestigial. Plus the PCIe expansion was quietly downgraded, Apple Silicon doesn't have a ton of PCIe lanes so the slots are heavily oversubscribed with PCIe switches.
I just find it interesting that you can currently buy a M2 Ultra Mac Pro that is weaker than the Mac Studio (for a comparable config) at a higher price. I guess it "remains a product in their lineup" and we'll hear more about it later.
Additionally: If they wanted to scrap it down the road, why would they do this now?
Maybe they can bring back the trash can.
Indeed, and tbh it really commits even more to the non-expandability that the Trashcan's designers seemed to be going for. After all, the Trashcan at least had replaceable RAM and storage. The Mac Studio has proprietary storage modules for no reason aside from Apple's convenience/profits (and of course the 'integrated' RAM which I'll charitably assume was done for altruistic reasons because of how it's "shared.")
The difference is that today users are accepting modern Macs where they rejected the Trashcan. I think it's because Apple's practices have become more widespread anyway*, and certain parts of the strategy like the RAM thing at least have upsides. That, and the thermals are better because the Trashcan's thermal design was not fit for purpose.
* I was trying to fix a friend's nice Lenovo laptop recently -- it turned out to just have some bad RAM, but when we opened it up we found it was soldered :(
Makes it even more puzzling what they are doing with the M2 Mac Pro.
[0] https://www.numerama.com/tech/1919213-m4-max-et-m3-ultra-let...
[1] More context on Macrumors: https://www.macrumors.com/2025/03/05/apple-confirms-m4-max-l...
And anyway, I think the M2 Mac Pro was Apple asking customers "hey, can you do anything interesting with these PCIe slots? because we can't think of anything outside of connectivity expansion really"
RIP Mac Pro unless they redesign Apple Silicon to allow for upgradeable GPUs.
Compared to Nvidia's Project DIGITS which is supposed to cost $3K and be available "soon", you can get a specs matching 128GB & 4TB version of this Mac for about $4700 and the difference would be that you can actually get it in a week and will run macOS(no idea how much performance difference to expect).
I can't wait to see someone testing the full DeepSeek model on this, maybe this would be the first little companion AI device that you can fully own and can do whatever you like with it, hassle-free.
at 819 GB per second bandwidth, the experience would be terrible
Also, people figured a way to run these things in parallel easily. The device is pretty small, I think for someone who wouldn't mind the price tag stacking 2-3 of those wouldn't be that bad.
A back of the napkin calculation: 819GB/s / 37GB/tok = 22 tokens/sec.
Realistically, you’ll have to run quantized to fit inside of the 512GB limit, so it could be more like 22GB of data transfer per token, which would yield 37 tokens per second as the theoretical limit.
It is likely going to be very usable. As other people have pointed out, the Mac Studio is also not the only option at this price point… but it is neat that it is an option.
525GB/s to 1000GB/s will double the TPS at best, which is still quite low for large LLMs.
AMD Ryzen Threadripper PRO 3995WX released over four years ago and supports 2TB (64c/128t)
> Take your workstation's performance to the next level with the AMD Ryzen Threadripper PRO 3995WX 2.7 GHz 64-Core sWRX8 Processor. Built using the 7nm Zen Core architecture with the sWRX8 socket, this processor is designed to deliver exceptional performance for professionals such as artists, architects, engineers, and data scientists. Featuring 64 cores and 128 threads with a 2.7 GHz base clock frequency, a 4.2 GHz boost frequency, and 256MB of L3 cache, this processor significantly reduces rendering times for 8K videos, high-resolution photos, and 3D models. The Ryzen Threadripper PRO supports up to 128 PCI Express 4.0 lanes for high-speed throughput to compatible devices. It also supports up to 2TB of eight-channel ECC DDR4 memory at 3200 MHz to help efficiently run and multitask demanding applications.
So unified memory means that the memory is accessible to the GPU and the CPU in a shared pool. AMD does not have that.
[1] https://www.amd.com/en/products/processors/laptop/ryzen/ai-3...
[1] https://www.amd.com/en/products/accelerators/instinct/mi300/...
Its a very specific claim that isnt comparing itself to DIMMs
8 channels at 3200 MT/s (1600 MHz) is only 204.8 GB/sec; less than a quarter of what the M3 Ultra can do. It's also not GPU-addressable, meaning it's not actually unified memory at all.
If you sit around expecting selflessness from Apple you will waste an enormous amount of time, trust me.
Not even OCSP?
But if you're being pedantic, I meant Apple SaaS requiring monthly payments or any other form of using something from Apple where I give them money outside the purchase of their hardware.
If you're talking background services as part of macOS, then you're being intentionally obtuse to the point and you know it
Maybe that's why they ship with insultingly-small SSDs by default, so that as people's photo libraries, Desktop and Documents folders fill up, Apple can "fix your problem" for you by selling you the iCloud/Apple One plan to offload most of the stuff to only live in iCloud.
Either they spend the $400 up front to get 2 notches up on the SSD upgrade, to match what a reasonable device would come with, or they spend that $400 $10 a month for the 40 month likely lifetime of the computer. Apple wins either way.
It's like learning about growing vegetables in your garden but then having to pay the seeds for it much more because you actually know how to produce value with them.
The philosophy at Apple has changed from premium tools for professional to luxury device for normies that makes them pay for their incompetence.
Whatever the profit margin on an iMac Studio is these days, surely improving non-consumer options becomes profitable at some point if you start selling them by the thousands to data centers.
This forum looses track of the world outside this echo chamber
That said, attracting creative users also adds value to the platform by creating demand for creative software for macOS, which keeps existing packages for macOS maintained and brings new ones on board every so often.
It does. Support costs. How do you prove it's a hardware failure or software? What should they do? Say it "unofficially" supports Linux? People would still try to get support. Eventually they'd have to test it themselves etc.
Has been. This is importance. Past tense. Maybe that's the point - they gave up on it acknowledging the extra costs / issues.