Ask HN: Open source LLM for commercial use?

does it though? it looks more like it requires a lot of money for compute and a lot of money and data for parameter tuning, but engineering effort seems soso.

except for the compute cost this is perfect application for a distributed open source labeling effort.

Just for my understanding though, are the data sets full of copyrighted material?

I can see a not-too-distant future where initial "base" models (like LLaMA) are released by such entities that do have the resources as they are seen as foundational enablers of the ecosystem (roughly equivalent to the Linux kernel or possibly Torch/Tensorflow/Transformers) where the "real" (differentiating) value from a commercial standpoint is something like 5-10 layers up the stack. The tremendous amount of value afforded by something like a Linux distribution isn't in the kernel, some random library, nginx, docker, etc. When you look hardware up almost everything you see on HN is 90-99% the same code, frameworks, toolkits, etc.

Then, a wide diaspora of commercial, academic, etc interests and other collaborators scratch their own itches and push the needle forward. Some release to the public, some don't but at a certain scale the combined effort easily exceeds the resources available to even a large, well funded entity like OpenAI. I've talked about it before but the last study I could find from 2008 analyzed Fedora 9 and estimated it represented something like $10b in combined dev cost.

There are also such rapid advancements in finetuning models in limited VRAM environments, quantization, applying them to specific use-cases, tooling, etc that the barrier of entry to iterate, build on, and actually use something like LLaMA is no longer 100 A100s (or whatever) and a dedicated large team. If you run apt-get install $SOMETHINGBIG and it grabs dozens of dependencies you're never heard of it starts to drive this point home.

I'm working on a project to be announced/released soon that in the end is something like > 100 python dependencies and other misc enabling packages, frameworks, tools, etc that it ends up being a 12GB docker image. Our "magic", meanwhile, is something like 1k LoC.

The biggest hole in this position is that it could be viewed releasing a model and weights is the equivalent of releasing your application and data itself but back to your original point I don't see the entire world bifurcating into multi-billion dollar startups and "everyone else".

Or maybe I'm just being optimistic :).

> The only reason we have LLAMA is because Meta threw us a bone

IMO, this is pretty inaccurate, you can look at my other post in the thread to see how many other recent and ongoing projects there are. The training data sets (The Pile, The Stack, LAION, etc) are publicly available and have been shown to be able to train very high quality models (and some groups committed to open models like Stability AI and Hugging Face are fairly well capitalized).

Training and fine-tuning costs are both getting better and costs are droping ridiculously fast (fine tunes went from spending thousands, to hundreds, and now to about $10 in the span of weeks). There are new optimizations and techniques being published every day (almost all of it reproducible, most w/ a code repos).

For new foundational models, Cerebras and others now will happily do built-to-order ones for a flat fee, but I suspect all kinds of well-funded EDUs, research labs, corporations, maybe even nation states will continue to train/release new cutting edge models w/ permissive licenses.

One could use chatgpt / gpt4 to create better training material for those models, even if not allowed. In that sense there is an advantage to being second here.

https://huggingface.co/google/flan-t5-xxl

Machine Learning models are made the same way machine learning output is generated.

In other words, the old model is training data to the new model. Just like the pirated torrent site dataset "Books3" Facebook used to train LLaMA is training data.

If Facebook can protect their model under copyright then every publisher in existence sue Facebook into the ground. They can't have it both ways.

I'd love any alternative view points of this.

> You will not use, copy, modify, merge, publish, distribute, reproduce, or create derivative works of the Software, in whole or in part, for any commercial, military, or illegal purposes.

Q: hello, who are you? A: I was twitted. In fact, twitted, or twittered, is one of those tweets that one either never sees or sees right away. Twitter is a website that allows users to post short messages that can be read and retweeted by other users. These messages are called tweets. So, who are you? Oh, hello, who are you? I was twitted. In fact, twitted, or twittered, is one of those tweets that one either never sees or sees right away. Twitter is a website that allows users to post short messages that can be read and retweeted by other

Id be wary of just hacking away without understanding at least the fundamentals of ML + NLP or you'll find yourself lost pretty quick.

I'm a former SWE turned NLP researcher, so i was recently in your position:)

- karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY

- https://towardsdatascience.com/beautifully-illustrated-nlp-m...

- https://dzone.com/articles/a-deep-dive-into-the-transformer-...

- https://peterbloem.nl/blog/transformers

- http://nlp.seas.harvard.edu/2018/04/03/attention.html

- https://lilianweng.github.io/posts/2023-01-27-the-transforme...

- https://blog.quickchat.ai/post/tokens-entropy-question/

- https://dugas.ch/artificial_curiosity/GPT_architecture.html

- https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

- https://d4mucfpksywv.cloudfront.net/better-language-models/l...

- https://arxiv.org/pdf/2005.14165.pdf

- https://arxiv.org/pdf/2303.08774.pdf

- https://arxiv.org/pdf/2303.17564.pdf