Q3_K_S 33b fits on my 16GB laptop + 6GB GPU (so 22GB minus some overhead from the split + linux), but its extremely tight and only works up to ~800 context.
The bloated specs look like typical scam product description from ebay or AliExpress. I can only image low quality finish and bet it will make noises when grabbing more firmly.
Dang, they have a tiny blower fan inside that spins at 20k RPM. That would be annoying spinning at high speeds.
Very impressive they're able to cram that much performance into an Android handheld, but part of me wishes Microsoft would put more research/dev time into Windows on Arm, or Apple allowing you to install their OS onto it.
Android just seems so limiting when you have that much memory. The OS probably can't even take advantage of it all.
I was quite amused hearing how people (probably the same who singed praises) ditched the first gen M1s with 8Gb, because 'not enough memory', merely a year later.
'Unified memory' access on M1 is allegedly almost as fast as CPU cache, and I believe the SSDs are extremely close to the SoC as well. Swap might be faster than some computers' actual RAM.
This is the exact kind of PR bamboozlement I was alluding to - literally none of the above is correct:
M1 memory latency is 100ns, which wasn't really competitive with amd/intel at the time (70 something nano). Any SSD read be an order of magnitude slower than reading from ram - something to the tune of 5gb/s version vs 70gb/s for the SOC ram. For comparison, the intel trashcan mac pro clocked in at 60gb/sec a decade prior.
> This is the kind of PR bamboozlement I was alluding to - literally none of the above is correct
I'm basing my knowledge on discussions with other developers on rwkv.cpp because we were talking about how performance scales with the number of tokens per iteration. Memory speed/bandwidth came up and some things about M1 were said. Sorry about that.
See, this is the thing about M1: I don’t care about your numbers (which I have no doubt are true), I care about my personal experience with the magic cold $999 aluminum slab that runs circles around everything I owned before.
Reading these always feels like winning an F1 race, then being gaslit about how that’s not possible because of inferior cylinder design.
Not Apple PR magic, but true firsthand experience. Objectively, this is the computer that makes me happiest and most productive. It could run on a 6MHz slime mold processor for all I care.
It's not like they ditched them for x86 devices. If people said that 8GB M1s were more capable than their x86 machines, it can be logically consistent with them going for even more power as more M* series chips released.
I distinctly remember the reality distortion field that permeated the discussion ("Apple silicone doesn't need as much memory", "MacOS is more efficient than windows" and similar such idiocies) when those things first came out. Whoever Apple retained for this PR sure managed to bamboozle a lot of gullible people.
I don’t think there’s a reality distortion field. I have a 10 year-old MacBook Pro (the very first Retina model) with 8GB of RAM. I often find myself with VSCode, two dozen Chrome tabs, some terminal windows and Adobe Illustrator (2022) running simultaneously. Somehow, it manages to cope with all of that without an issue. Of course, it doesn’t work as fast as new computers, but the UI remains fluid and responsive. I’ve never had a Windows machine offer that kind of practical longevity.
Only the Air and 13” Pro (which is a glorified Air with a fan) cap at 24 GBs. The 14 and 16” Pro models cap at 96 GB. The 13 inch is a bit of a lame duck product that is likely to be phased out at some point or otherwise be replaced, so while the headline is factually true it is not really indicative of the situation.
I will also say, Apple devices across the board are significantly better at memory management than both Android and Windows. Where on a Windows machine I feel I need 32 GBs, 16 GBs feels adequate on MacOS. And my understanding is the situation is similar between iOS and Android.
Ha no, I'm using 9.98Gb of ram with literally just a single tab browser window open in Mojave. You'll be swapping constantly with anything resembling a normal workflow.
It's memory management, not memory minimalism. I'm always hovering about 12 GB out of my 16 GB MBA, but never have any issue opening a project or opening 10–20 tabs (rookie number) and I have a few apps that are always running. I believe that macOS uses the RAM for caching files, so the reported usage is not really tied to the number of apps running. There are memory-heavy workflows, but these are outliers. Even 8 GB would be fine for most of what I do.
It doesn't - that's exactly why people have workstation with a ton of ram. As soon as you have to hop over the pci bus, you're looking at 1/10th of the speed.
Did you just say Mojave? Because I have extensive experience with Mojave and it does not use swap whatsoever. Instead, you have overcommit and memory compression. That application that the OS says uses 10GB of RAM does not actually.
Pretty simple: software devs have good hardware, and use all the resources they have.
It goes like this: new hardware comes out that is more powerful, people buy it because it feels good and everything is fast. Then new software uses all of it, and the cycle starts again with new hardware.
> software devs have good hardware, and use all the resources they have.
This.
It's not that people make their programs unacceptably slow on purpose. It's that they test their programs on incredibly and unrealistically powerful hardware.
I used to know someone that said they optimized their DOOM port or something like that for cheap old netbooks, and as a result, get hundreds of thousands of FPS on any vaguely modern system. And there's a video series on YouTube where someone is optimizing some voxelized game engine and uses the Intel iGPU - yes, the nearly-useless one you get on most of their Core-series CPUs - to vastly outperform a CPU implementation of a bunch of voxel algorithms anyway, because optimizing for such a weak GPU allows you to focus only on things that GPUs are actually fundamentally good at, as anything else will run unacceptably slowly. Unfortunately I don't remember what it was called.
> It seems like iOS and macOS do more with less ram. Why is that?
AFAIK, Android caches things in terms of the JVM/ART using Zygote, but macOS and iOS are designed to run native code, so they cache the actual dynamic libraries themselves (it's called the "dyld shared cache"). Things like libc, some core frameworks and so on go in there.
Keep in mind this happens across the entire OS, since the Android user experience is implemented largely in terms of high level JVM/ART objects with all of their associated heap overhead, but iOS and macOS and their apps are mainly implemented in C/C++, Swift, Objective-C, which at most use reference counting, and only for objects that need to be reference counted. This is absolutely not the cause of all memory issues but this is sort of the base level foundational stuff of both platforms and has implications for the higher levels.
Even though Android is based on Linux and iOS/macOS are based on Unix and BSD, and those two/three platforms are quite similar conceptually, they have very different userlands.
I get why refcounting could in theory sometimes have less latency than a highly tuned gc. But I don't see how it could possibly be lower memory. Can you explain?
Does anybody know what Android game(s) could actually utilize these sort of specs? Is there some sort of 4K HDR Words With Friends that I’m missing out on?
I love the under-screen front camera. People need to stop adding notches and hole punches. I would prefer no front camera at all, a bezel, or a motorized one like the OnePlus 7 Pro (which I have), but an under-screen camera is absolutely acceptable and cool.
24GB is still less considering the amount of bloatwares these Chinese brands put in their garbage phones and the number of crappy apps run in the background 24*7
47 comments
[ 0.20 ms ] story [ 151 ms ] threadQ3_K_S 33b fits on my 16GB laptop + 6GB GPU (so 22GB minus some overhead from the split + linux), but its extremely tight and only works up to ~800 context.
ZTE did the Axon M, which might have been the first foldable, though clunky.
Nubia did the Z20, which had a screen on the back.
Very impressive they're able to cram that much performance into an Android handheld, but part of me wishes Microsoft would put more research/dev time into Windows on Arm, or Apple allowing you to install their OS onto it.
Android just seems so limiting when you have that much memory. The OS probably can't even take advantage of it all.
This is not the problem. You can find comments here how people literally said what 'it were running better than x86 Macs with 16+GBs'.
ADD: and 'nobody needs more than 8GB, because it works so fine'. Just to clarify.
Idk if it swaps fast or what, and I don't care.
M1 memory latency is 100ns, which wasn't really competitive with amd/intel at the time (70 something nano). Any SSD read be an order of magnitude slower than reading from ram - something to the tune of 5gb/s version vs 70gb/s for the SOC ram. For comparison, the intel trashcan mac pro clocked in at 60gb/sec a decade prior.
Reference:
https://www.anandtech.com/show/16252/mac-mini-apple-m1-teste...
https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-di...
I'm basing my knowledge on discussions with other developers on rwkv.cpp because we were talking about how performance scales with the number of tokens per iteration. Memory speed/bandwidth came up and some things about M1 were said. Sorry about that.
Reading these always feels like winning an F1 race, then being gaslit about how that’s not possible because of inferior cylinder design.
https://en.wikipedia.org/wiki/Priming_(psychology)
I distinctly remember the reality distortion field that permeated the discussion ("Apple silicone doesn't need as much memory", "MacOS is more efficient than windows" and similar such idiocies) when those things first came out. Whoever Apple retained for this PR sure managed to bamboozle a lot of gullible people.
I will also say, Apple devices across the board are significantly better at memory management than both Android and Windows. Where on a Windows machine I feel I need 32 GBs, 16 GBs feels adequate on MacOS. And my understanding is the situation is similar between iOS and Android.
Ha no, I'm using 9.98Gb of ram with literally just a single tab browser window open in Mojave. You'll be swapping constantly with anything resembling a normal workflow.
It goes like this: new hardware comes out that is more powerful, people buy it because it feels good and everything is fast. Then new software uses all of it, and the cycle starts again with new hardware.
This.
It's not that people make their programs unacceptably slow on purpose. It's that they test their programs on incredibly and unrealistically powerful hardware.
I used to know someone that said they optimized their DOOM port or something like that for cheap old netbooks, and as a result, get hundreds of thousands of FPS on any vaguely modern system. And there's a video series on YouTube where someone is optimizing some voxelized game engine and uses the Intel iGPU - yes, the nearly-useless one you get on most of their Core-series CPUs - to vastly outperform a CPU implementation of a bunch of voxel algorithms anyway, because optimizing for such a weak GPU allows you to focus only on things that GPUs are actually fundamentally good at, as anything else will run unacceptably slowly. Unfortunately I don't remember what it was called.
It seems like iOS and macOS do more with less ram. Why is that?
My outdated view on this is that iOS apps are native vs Android that runs JVM apps. Is that even true still?
Still I can imagine that it uses slightly more memory...
AFAIK, Android caches things in terms of the JVM/ART using Zygote, but macOS and iOS are designed to run native code, so they cache the actual dynamic libraries themselves (it's called the "dyld shared cache"). Things like libc, some core frameworks and so on go in there.
Keep in mind this happens across the entire OS, since the Android user experience is implemented largely in terms of high level JVM/ART objects with all of their associated heap overhead, but iOS and macOS and their apps are mainly implemented in C/C++, Swift, Objective-C, which at most use reference counting, and only for objects that need to be reference counted. This is absolutely not the cause of all memory issues but this is sort of the base level foundational stuff of both platforms and has implications for the higher levels.
Even though Android is based on Linux and iOS/macOS are based on Unix and BSD, and those two/three platforms are quite similar conceptually, they have very different userlands.