GPT2 is ancient news. There are now local running models which, allegedly, can reach the same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models.
llama.cpp is CPU only but llama runs on GPU using the HuggingFace Transformers library. You can run the llama model which far outpaces GPT-3.5 on a couple of $200 Tesla P40 GPUs at faster speeds than GPT-3.5 Turbo, completely locally.
I don’t know about out of the box, but I’ve been having a great time training my own domain-specific models from scratch and utilizing them locally. It does seem that the more domain specific a task is, far fewer params are needed.
Having fun right now trying to build a model in GDELT, not much luck so far, but I’ve pushed less than 5% of the data through so far.
I’ve also been experimenting on fine-tuning Llama on my personal data archives, which seems promising but it’s pretty expensive to do so. Hoping someone will release a ~13B param model of Llama that they’ve trained with transfer learning from the 65B llama model and other data. FWIW even the 7B llama model running through llama.cpp after being quantized performs (subjectively, but substantially) better than GPT2.
This has been one of my most expensive, but also most rewarding, hobbies thus far.
Really depends. I’ve pretty much almost burned through all the cloud credits I’ve been able to find at this point. Probably spent 20 grand over a year in credits? Never more than a couple hundred of my own money. Still trying to find a better solution. I have some 2080 TIs that I use locally. I tend to prototype architectures locally then rent out some A100s for a few hours to see how it goes, continue training when I can spare the cash.
I do a lot of CPU and sharded training too. I really wish there were better options for hobbyists to play around with this stuff.
Training domain-specific models from scratch is both very difficult and also not possible in some cases. Sometimes we want to generate some niche content where there isn't enough training data to create a domain-specific model. We would have to somehow find a model that can produce the niche content through understanding what we're asking rather than through actually being trained on it. Producing "understanding" is not something you can do at home unless you have a bunch of A100s and have studied machine learning for years.
Totally agree there. Even people that have years of ML under their belt may have a hard time there. And the cost (and scarcity) of those GPUs really are a major barrier to entry.
Its amazing can run Alpaca llama 33B parameters, totally can handle japanese and korean where the earlier ones like 7B parameters could only do english (any other languages was horrible). All able to run on my M1 macbook.
The oobabooga setup feels a lot more mature and has a larger community. Skimming OP's repo, there seems like a lot of fiddling and faffing with JSON to get things running.
Do you know what is the way to find models compatible with oobabooga/text-generation-webui ? I downloaded one with the included script and that worked, but if I try different ones it seems there are so many formats so no idea say how do I search huggingface or google to find the correct format.
I would like to try this new quantized LLAMA versions with the GUI, I can
run them in the CLI on the CPU but llama.cpp uses ggml formats .
Neat, eventually would like to run a purely local LLM Emacs shell https://github.com/xenodium/chatgpt-shell. For now ChatGPT only, but working on making more generic/reusable.
Awesome !! Great … I haven’t gone through it yet but from just what I saw on the GitHub page it would be awesome to have a CLI with standard arguments built into it for everything that can be done through the flask interface. Thanks !!
Someone with more inclination than I at the moment might be able to say something interesting about this being from Nat Friedman (former CEO of GitHub).
Thanks for this! The compare feature is very cool, especially being able to play around with GPT4 settings (I'm still on the waiting list for GPT4 API so having access to this now is fantastic).
This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?
In principle you can use subprocess.run with llama.cpp (especially now that the mmap patch has landed, so model load time is negligible on subsequent runs) and then use stdin and stdout to interact with it. Multiple local sessions should work too. I’ve not tried it out yet... though I was looking for an afternoon project to work on!
Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python/pytorch/CUDA). (And I see that since last week it now has an HTML GUI too.) https://bellard.org/ts_server/
However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap." [1])
Textsynth is awesome, if you don’t want to use a fine tuned version of the models…far as I can tell, there’s no documentation on how to convert a fine tuned version of a model (like alpaca) to work with textsynth.
57 comments
[ 3.0 ms ] story [ 73.6 ms ] threadI'd say they are closer to GOT2.5
[1] https://github.com/ggerganov/llama.cpp
-Emily
Which is pretty cool all things considered. Certainly better than most of us would have expected a short while ago, right?
[0] https://arxiv.org/pdf/2302.13971.pdf
Having fun right now trying to build a model in GDELT, not much luck so far, but I’ve pushed less than 5% of the data through so far.
I’ve also been experimenting on fine-tuning Llama on my personal data archives, which seems promising but it’s pretty expensive to do so. Hoping someone will release a ~13B param model of Llama that they’ve trained with transfer learning from the 65B llama model and other data. FWIW even the 7B llama model running through llama.cpp after being quantized performs (subjectively, but substantially) better than GPT2.
This has been one of my most expensive, but also most rewarding, hobbies thus far.
Do you train exclusively in the cloud?
I do a lot of CPU and sharded training too. I really wish there were better options for hobbyists to play around with this stuff.
-Emily
Eg: "Automatically detects local models in your HuggingFace cache, and lets you install new ones."
Llamma.cpp and HuggingFace models are all local.
Nice stuff all the same.
;-)
...It is a good AGI test - trying the consistency in the "spacial" and temporal development of the construed world.
However, the GPU version is only available commercially. I'd like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp's author wrote "I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap." [1])
EDIT: But if you wish, here's a Python interface to llama.cpp: https://github.com/PotatoSpudowski/fastLLaMa
[1] https://news.ycombinator.com/context?id=35195270
I'm on mobile and can't find it right now though.
[1] https://leanpub.com/langchain
EDIT: GitHub repo https://github.com/mark-watson/langchain-book-examples
"pip install llamacpp" or https://github.com/thomasantony/llamacpp-python
I added the playground to the GUI list here:
to https://github.com/underlines/awesome-marketing-datascience/...