HN Explorer

4 comments

[ 3.6 ms ] story [ 22.7 ms ] thread
omeze
Hey all,

After reading the TinyStories paper (using small language models of 1m to 33m parameter size to generate short stories), me and a friend decided to port it as a test to see what performance of tiny LMs look like on mobile devices. The stories it generates aren't great or anything, it's really just a toy tech demo to play with. We used HuggingFace's exporters library to do this.

You can try it out here: https://apps.apple.com/us/app/tinystories-on-device/id645149...

(we are not affiliated with MS research or the authors)

brucethemoose2
If y'all are looking for another iOS port, try a llama 7B or 13B storytelling model running on llama.cpp/koboldcpp or the mlc-llm backend.

7B in particular should be fast and usable on most iOS devices. 13B should work on 8GB of RAM. Both are quite coherent and "smart."

Theres a tragic gap between these optimized implementations and mobile apps, for some reason.

omeze
yea, I was also surprised at the lack of apps. I guess we can take a shot at llama-7b, maybe a base model and a storytelling fine tune. we'll have to look into which compiler is best, I do want to tinker with the mlc toolchain. will post to HN when we get it working!
brucethemoose2
MLC and llama.cpp both have performant metal ports, and mlc has a demo ios app!

For story finetunes, I am out of the loop... You have to dig through HF by date to find them, for instance:

https://huggingface.co/zarakiquemparte/hermesboros-limarp-7b

https://huggingface.co/models?p=1&sort=modified&search=7b

You can also look through what people are hosting on Kobold Horde (a storytelling/rp community) ignoring the old pygmalion/opt models: https://lite.koboldai.net/#

Also, you should at least host an instruct model instead of vanilla llama 7B.