I reached 3Gbps over Verizon 5G in San Antonio last year and this year i get about 4Mbps over Verizon 5G in Ohio. It’s so bad I disabled it. I did read an article that iPhone 12 (which is what I have) have some kind of radio issue with 5G. Can anyone in here confirm?
Verizon has been real wonky all over Austin. If there's more than a handful of people in the area, bandwidth just goes to the crapper. I'll go from a couple hundred mbps on a good day with no clouds/wind/holding my phone just perfectly right in the right space, but usually get less than 1mbps on their 5g UW.
> It took a minute to summon the picture on the latest and greatest iPhone 14 Pro, uses about 2GiB in-app memory, and requires you to download about 2GiB data to get started. Even though the app itself is rock solid, given these requirements, I would probably call it barely usable.
also
> Even if it took a minute to paint one image, now my Camera Roll is filled with drawings from this app. It is an addictive endeavor. More than that, I am getting better at it. If the face is cropped, now I know how to use the inpainting model to fill it in. If the inpainting model doesn’t do its job, you can always use a paint brush to paint it over and do an image-to-image generation again focused in that area.
Seems very worth a try. I'm downloading the model right now, it's going a bit slow, ~2MB/s.
It would be awesome if it just quickly took a picture from the front camera for that particular request and then just filter it to finish the "in the style of andy warhol"
I have generated some images, I think it only takes less than 1% of the battery for an image, this is already much better than most of the game(for having fun).
It took my battery from 80 to 77% for one generation on the default settings (384^2, 30 iterations). Less than a minute of compute time to complete a generation.
Iphone battery health reports a battery at 100% health. This is an iphone SE3.
Amazing how huge the difference in energy consumption is for the system in standby vs going full throttle.
EDIT: I generated 3 more images; every subsequent generation reduced battery capacity by another 2%. My phone doesn't seem to heat up at all, interestingly.
Late to the party but depending on your charger voltage, it may be. e.g. I can charge my mbp on a cheap usb-usb-c charger but as soon as I use it _too_ much, it will stall, or worse, lower.
Memory compression? I can't find any good resources to read about it, any hints? I'm having trouble imaging how could it possibly work without totally destroying performance.
It is not compressed swap, the compressed data is still in RAM. The OS just compresses inactive memory, with a couple of criteria to define “inactive”.
It doesn't destroy performance for the simple reason that nowadays memory access is slower than pure compute. If you need to use compute to produce some data to be stored in memory, your overall throughput could very well be faster than without compression.
There have been a large amount of innovation on fast compression and fast decompression in recent years. Traditional compression tools like gzip or xz are geared towards higher compression ratio, but memory compression tends to favor speed. Check out those algorithms:
iOS uses memory compression but not swap. iOS devices actually have special CPU instructions to speed up compression of up to page size increments specifically to aid in this model [1]
Memory compression is a generalization of swap, which is only for dynamic memory; files on disk don't need it because you can just read them off the disk.
The problem is that GPUs don't support virtual memory paging, so they can't read files nor decompress nor swap anything unless you write it yourself, which is a lot slower.
Also, ML models (probably) can't be compressed because they already are compressed; learning and compression are the same thing!
Wait. This comment just blew my mind. Does that imply that you might be able to measure the efficiency of a model by it's compressibility? Note, I'm trying to recognize efficient and accurate are not the same. One could imagine evaluating a model on a 2d performance and compression map somehow.
> Also, ML models (probably) can't be compressed because they already are compressed; learning and compression are the same thing!
I feel like they're kind of two sides of the same coin: learning is about putting more information in the same data, while compression is about putting the same information in less data.
I'm wondering if some lossy floating-point compressor (such as zfp) would work.
> I'm wondering if some lossy floating-point compressor (such as zfp) would work.
Well apparently this can work; StableDiffusion comes as 32-bit and 16-bit float versions. I'm kind of surprised they both work, but that's lossy compression.
Sure, but 16-bit float is pretty primitive compression, as it does not exploit any redundancy in the input. zfp groups numbers together in chunks, which means that correlated numbers can be represented more precisely. Its algorithm is described here: https://zfp.readthedocs.io/en/release1.0.0/algorithm.html#lo...
I would like to see if the zfp can be applied to something like Stable Diffusion (or other ML models) and give better results than regular floats at the same size.
In-memory compression means the memory is inherently dirty memory
On Apple platforms if you mmap a read-only file into the process address space, then it is "clean" memory. It is clean because the kernel can drop it at any time because it already exists on disk. You essentially can offload the memory management to the kernel page cache.
The downside is that if you run up to the limit and the "working set" can't fit entirely in memory, then you run into page faults which incur an I/O cost.
The advantage is that the kernel will drop the page cache before it considers killing your process to reclaim memory.
That said, I don't know the typical access patterns for neural network inference, so I don't know how the page faults would effect performance
It is not as useful for this case (inference) because the activations holds long (UNet holds downsampling passes' activations and use that for upsampling) is not that much of a memory (in the range of a few megabytes). If it is for training, it is probably more useful.
This is absolutely incredible. It takes about 45 seconds to generate a whole image on my iPhone SE3 - which is about as fast as my M1 Pro macbook was doing it with the original version!
SE 3rd Gen has 4GiB RAM, therefore the app defaults to 384x384 size. This is about 1/2 computation of your normal run (512x512) and the original version uses PLMS sampler, which defaults to 50 steps, while this one uses the newer DPM++ 2M Karras sampler, that defaults to 30 steps. All in all, your M1 Pro MBP is still 4x of your SE 3rd Gen in raw performance (although my implementation should be faster than PyTorch at about 2x on M1 chips)
For what it's worth, you can decrease resolution and use the sampler mentioned on the pytorch versions. The AUTOMATIC web UI supports this, for instance.
I would also welcome the additional optimizations, however.
Given couple of weeks. Still playing to see what's the optimal UI looks like for such large screen. 16GiB should be able to generate several images to select from at once.
Does it crash upon downloading models, or generating? I haven't tested on all the devices, but 11 Pro seems have 4GiB RAM, and should run with 384x384 resolution (check if that is the selected resolution at top right).
There are reports that iOS is not happy with how I computing SHA256 for downloaded model file by loading them all in memory for Xr (3GiB RAM). If this is happening for other devices, I may need to do streaming hash computing and put up a bugfix.
Yeah, I thought Data(contentsOf:) already do that, but it appears not (tested, indeed allocated all the memory to load the data). Adding `mapIfSafe` in the reading options solved this.
I think that probably is to be expected, unfortunately. The model is 2GB; per the article iOS will kill apps that use 2GB on 4GB RAM devices (and the 13 Mini is 4GB).
It works just fine on my iPhone 12 Mini, which has (basically) the same RAM, so it seems like it's something else. They're basically the same phone in general though, so I would be surprised if it was a hardware issue.
I did have trouble closing the "adjustments" dialog (upper-right button) due to its close button being underneath the status bar, but found that I could just drag the dialog down to the bottom and it closed.
On my iPad mini 5th generation with A12 the download is fast and fine. But with standard settings it first warns “Device capability warning” and then indeed crashes every time.
Is there a way to solve this?
A12 chip should work, no?
This is super cool. I just tried the default prompt on my iPhone 13 with the image size set to 768x512 and using the 3D Model (Redshift v1) and it just crashed the whole phone and restarted. Just like when I get BSOD's at work on my Windows GPU desktop :)
Porting FlashAttention to Metal will be quite hard. Because for performance reasons, they did a lot of shenanigans to respect the memory hierarchy.
Thankfully, you can probably do something slower but more adapted to your memory constraints.
If you relax this need for performance and allow some re-computations, you can write a qkvatt function which takes q,k,v and a buffer to store the resulting attention, and compute without needing any extra memory.
The algorithm is still quadratic in time with respect to the attention horizon (although with a bigger constant (2x or 3x) due to the re computation). But it doesn't need any extra memory allocation which makes it easy to parallelize.
Alternatively you can use an O(attention horizon * number of thread in parallel) (like flash attention) extra memory buffer to avoid the re-computation.
Concerning the backward pass, that's the same thing, you don't need extra memory if you are willing to do some re-computation, or linear in attention horizon to not do re-computation.
One interesting thing to notice in the backward pass, is that it doesn't use the attn of the forward pass, so it doesn't need to be kept preserved (only need to preserve Q,K,V).
One little caveat of the backward pass (which you only need for training) is that it needs atomic_add to be easy to parallelize. This mean, it will be hard on Metal (afaik they don't have atomics for floats though they do have atomics for integer so you can probably use fixed points numbers).
Apple's mobile processors (especially the M1) are waaay faster than a steam deck. Even with the optimizations, it would take like half an hour to run I bet.
CPU offloading doesn't work because Apple has shared memory arch already. The head slicing is similar to https://machinelearning.apple.com/research/neural-engine-tra... I think it is quite practical only if MPSGraph less mysterious about its allocation strategy. It is not the ideal way though. Ideally, FlashAttention / XFormer is the way to go.
the developer is about to have a MASSIVE hosting bill
the download restarts from 0% if the app is sent to the background, as there does not seem to be a download manager. This is especially problematic for the large 1.5gb file.
I am using Cloudflare R2, which doesn't have egress fee and I am getting about 5k Class B operations right now. Unless Cloudflare changes their end of the deal, I think it is OK.
great to hear! please consider introducing a mechanism to suspend the download vs restarting, this may be especially valuable for those with a slower connection. With the traffic growth you'll be seeing chances are Cloudflare's enterprise team will soon be in-touch ;)
Is P2P torrent type of sharing the load possible under AppStore guidelines (be they iOS or Android)? I've honestly never even thought about this being a thing, but with large shared data that doesn't change, why not?
P2P, as in hosted from other people's phones? I think the issue is that people generally wouldn't be happy with P2P data uploads from their phones (compared to P2P on desktop, where internet is cheaper/faster, and battery isn't an issue).
I frequently see such comments about sites not loading fine on iPhone 13/14, while they continue to load just fine on for me on a 4 year old Android device (not pixel / Samsung).
I wonder if it's the hardware or just the blockers that i use. Might be worth trying using blockers to see if it makes the general browsing experience better on Apple devices
183 comments
[ 7.9 ms ] story [ 454 ms ] threadI just used the prompt "A person looking at their phone in amazement" and got a good picture.
Beware that on startup the app downloads almost 2 gig of data.
I reached 3Gbps over Verizon 5G in San Antonio last year and this year i get about 4Mbps over Verizon 5G in Ohio. It’s so bad I disabled it. I did read an article that iPhone 12 (which is what I have) have some kind of radio issue with 5G. Can anyone in here confirm?
Q: How long does a run typically take? 60 seconds?
> It took a minute to summon the picture on the latest and greatest iPhone 14 Pro, uses about 2GiB in-app memory, and requires you to download about 2GiB data to get started. Even though the app itself is rock solid, given these requirements, I would probably call it barely usable.
also
> Even if it took a minute to paint one image, now my Camera Roll is filled with drawings from this app. It is an addictive endeavor. More than that, I am getting better at it. If the face is cropped, now I know how to use the inpainting model to fill it in. If the inpainting model doesn’t do its job, you can always use a paint brush to paint it over and do an image-to-image generation again focused in that area.
Seems very worth a try. I'm downloading the model right now, it's going a bit slow, ~2MB/s.
Iphone battery health reports a battery at 100% health. This is an iphone SE3.
Amazing how huge the difference in energy consumption is for the system in standby vs going full throttle.
EDIT: I generated 3 more images; every subsequent generation reduced battery capacity by another 2%. My phone doesn't seem to heat up at all, interestingly.
I wonder if generating a picture uses so much juice that it even drops the percentage while charging...
EDIT: Generating a picture while charging did not drop the percentage.
There are some performance compressors like Blosch tuned for this:
https://www.blosc.org/pages/blosc-in-depth/
“Faster than memcpy” is the slogan.
It’s something similar to swap - apps do not need to have built in support for it.
There have been a large amount of innovation on fast compression and fast decompression in recent years. Traditional compression tools like gzip or xz are geared towards higher compression ratio, but memory compression tends to favor speed. Check out those algorithms:
* lz4: https://lz4.github.io/lz4/
* Google's snappy: https://github.com/google/snappy
* Facebook's zstd in fast mode: http://facebook.github.io/zstd/#benchmarks
Seems to mesh well with the iOS idea of using a single app at a time and minimizing background processing in apps that you aren't actively using.
In an out of memory situation I think apps just get killed as you suggest.
[1] https://github.com/apple-oss-distributions/xnu/blob/bb611c8f...
The problem is that GPUs don't support virtual memory paging, so they can't read files nor decompress nor swap anything unless you write it yourself, which is a lot slower.
Also, ML models (probably) can't be compressed because they already are compressed; learning and compression are the same thing!
I feel like they're kind of two sides of the same coin: learning is about putting more information in the same data, while compression is about putting the same information in less data.
I'm wondering if some lossy floating-point compressor (such as zfp) would work.
Well apparently this can work; StableDiffusion comes as 32-bit and 16-bit float versions. I'm kind of surprised they both work, but that's lossy compression.
I would like to see if the zfp can be applied to something like Stable Diffusion (or other ML models) and give better results than regular floats at the same size.
On Apple platforms if you mmap a read-only file into the process address space, then it is "clean" memory. It is clean because the kernel can drop it at any time because it already exists on disk. You essentially can offload the memory management to the kernel page cache.
The downside is that if you run up to the limit and the "working set" can't fit entirely in memory, then you run into page faults which incur an I/O cost.
The advantage is that the kernel will drop the page cache before it considers killing your process to reclaim memory.
That said, I don't know the typical access patterns for neural network inference, so I don't know how the page faults would effect performance
It is not as useful for this case (inference) because the activations holds long (UNet holds downsampling passes' activations and use that for upsampling) is not that much of a memory (in the range of a few megabytes). If it is for training, it is probably more useful.
Do you plan to make a macOS version of your app also? Hope you will :)
I would also welcome the additional optimizations, however.
There are reports that iOS is not happy with how I computing SHA256 for downloaded model file by loading them all in memory for Xr (3GiB RAM). If this is happening for other devices, I may need to do streaming hash computing and put up a bugfix.
It transcribed as “image of unicorn poo ping” in tags :(
congratulations to liuliu on the launch!
I did have trouble closing the "adjustments" dialog (upper-right button) due to its close button being underneath the status bar, but found that I could just drag the dialog down to the bottom and it closed.
Porting FlashAttention to Metal will be quite hard. Because for performance reasons, they did a lot of shenanigans to respect the memory hierarchy.
Thankfully, you can probably do something slower but more adapted to your memory constraints.
If you relax this need for performance and allow some re-computations, you can write a qkvatt function which takes q,k,v and a buffer to store the resulting attention, and compute without needing any extra memory.
The algorithm is still quadratic in time with respect to the attention horizon (although with a bigger constant (2x or 3x) due to the re computation). But it doesn't need any extra memory allocation which makes it easy to parallelize.
Alternatively you can use an O(attention horizon * number of thread in parallel) (like flash attention) extra memory buffer to avoid the re-computation.
Concerning the backward pass, that's the same thing, you don't need extra memory if you are willing to do some re-computation, or linear in attention horizon to not do re-computation.
One interesting thing to notice in the backward pass, is that it doesn't use the attn of the forward pass, so it doesn't need to be kept preserved (only need to preserve Q,K,V).
One little caveat of the backward pass (which you only need for training) is that it needs atomic_add to be easy to parallelize. This mean, it will be hard on Metal (afaik they don't have atomics for floats though they do have atomics for integer so you can probably use fixed points numbers).
Love the work, really great job!
You might like to look at the work HuggingFace has been doing (on non-iOS versions). They can run it in under 1GB RAM:
> If is also possible to chain it with attention slicing for minimal memory consumption, running it in as little as < 800mb of GPU vRAM
https://huggingface.co/docs/diffusers/optimization/fp16#offl...
the download restarts from 0% if the app is sent to the background, as there does not seem to be a download manager. This is especially problematic for the large 1.5gb file.
https://developer.apple.com/library/archive/documentation/Fi...
I wonder if it's the hardware or just the blockers that i use. Might be worth trying using blockers to see if it makes the general browsing experience better on Apple devices