By the time we put our paper on arxiv this paper was not out so we were not aware of, but it is similar in some ways. Both this paper and us are reliant on our previous paper https://arxiv.org/abs/2310.04564 and dejavu…
Half the memory was just an example not a sweet spot. With smaller window sizes you can use less memory, but it will come with the cost of loading more from flash. p.s: The window size in the paper is showing how many…
Yes, the goal is at the end to run larger models on the phone as phones have very limited DRAM.
And how does mmap gonna help with the performance when the bottleneck is flash to DRAM bandwidth?
One of the initial ideas was of course doing computation inside flash, but we didn't try to go that path for two reasons: 1. It's not as easy to change the controller, even if you do it was not obvious for me if we need…
By the time we put our paper on arxiv this paper was not out so we were not aware of, but it is similar in some ways. Both this paper and us are reliant on our previous paper https://arxiv.org/abs/2310.04564 and dejavu…
Half the memory was just an example not a sweet spot. With smaller window sizes you can use less memory, but it will come with the cost of loading more from flash. p.s: The window size in the paper is showing how many…
Yes, the goal is at the end to run larger models on the phone as phones have very limited DRAM.
And how does mmap gonna help with the performance when the bottleneck is flash to DRAM bandwidth?
One of the initial ideas was of course doing computation inside flash, but we didn't try to go that path for two reasons: 1. It's not as easy to change the controller, even if you do it was not obvious for me if we need…