Ask HN: Open-source ChatGPT alternatives?

The impression I've got is that fine tuning large language models is mostly useful for very simple tasks, such as training a spam or categorization filter.

If you're looking to take a model and then e.g. train it on a few thousand additional pages of documentation in order to get it to answer project-specific questions, I've got the impression that fine tuning isn't actually a useful way to achieve that (I'd love to be proven wrong about this).

Instead, people are getting good results with "retrieval augmented generation" - where you first run a search (or an embeddings-based semantic search) against your docs to find relevant snippets, then feed them to the large language model as part of a glued together prompt.

I wrote about my explorations of this technique here - plenty of other people have written about this too: https://simonwillison.net/2023/Jan/13/semantic-search-answer...

But seriously I am asking a very similar question with a focus on LLMs for classification (e.g. "Is this an article about a sports game?"), information extraction, clustering and such. I am not so interested in generation (Which I am assuming you are.), however the GPT style embeddings and those are are useful for the kind of work I do and are interchangeable with BERT-like and other embeddings.

"Good" or "Best" is something you have to define for yourself and the one thing every successful A.I. developer has done is develop a facility for testing if the solution was performing acceptably. With that library you can download a model and start working with the model, again, the successful people all tested at least one model. In the time since your post, a run-of-the-mill Python developer could have made some progress. Learn Python or get a non technical co-founder,

For my kind of tasks I want something that handles bigger documents than ChatGPT and when I go shopping for models I cannot find a high quality very large transformer that has been assembled with a tokenizer and trained hard on language tasks. When I look at the literature it seems the very long transformers like reformer wouldn't perform so well if somebody did try to build an LLM so I wait. I am certain that somebody will upload a better model to huggingface someday -- that's the thought process it takes to get an answer for questions like yours.

If you look though at the process used to make ChatGPT-3 able to converse there is the GPT-style embedding and then a process of dialog generation trained on totally different principles which is the "Reinforcement Learning from Human Feedback"

https://www.assemblyai.com/blog/how-chatgpt-actually-works/

and I think you are not going to get that kind of capability open source in that the training data doesn't exist for it. There are many things you have to do once you have that training data, but I think there are many people able to follow that path now that it has been blazed.

OPs point was most of us can't burn 10k on a internet hobby project, and it costs probably 10x-100x more to train. How would you expect to provide people a free service? People are training it to open source right now, so please be patient.