I don't expect so, but as LLMs get better more people want to use them in more places. An openly available LLM for commercial use which is easy to integrate in your existing databricks flow could be very tempting. That then leads to increased use of the platform & hours spent computing, so that's better for them.
It also shows how to build and train these things on databricks, so maybe more people will use them to make custom trained LLMs.
My hunch is that OpenAI's window of supremacy will be short. Even if they keep being SOTA, the open sourced models will eat away market underneath them. By the next year they will only be able to sell GPT-4 or 5. At some point the open models will be good enough for 99% of use cases.
Quality is one aspect, running them is another. If I've already got everything setup with them and they work efficiently, they could also offer open source models and let me pay for usage. Both bursty usage and low constant usage benefit from paying per token and having some shared & large infrastructure to use. I don't want to be running a bunch of h100s, I just want my requests processed.
If they're selling gpt-5 and let me pay for LLaMa or whatever is also out then I'll just use them unless pricing is wildly different.
Interesting to see that it's trained on data completely generated by Databricks employees. I wonder how "biased" that makes the data, and how much they spent in terms of lost man hours?
Since databricks has employees all around the world, I expect to see that the data is not biased. But for sure there must be considerable man hours lost. However it shows, how much they are dedicated towards open source contributions.
All datasets are biased, including this specific one. However, we believe it's still very valuable to open source, for a few reasons:
- This dataset is primarily used to train instruction reasoning, not for knowledge. (Keep in mind Dolly and any of the well known models have not been specifically trained for knowledge. They are all just demonstrating instruction reasoning.) The lack of a true open source (available for both research and commercial use) instruction dataset is the primary blocker for making these LLMs available for commercial use.
- We hope this will lead to not just open source innovations in models, but also future training datasets.
- Given the international population of our employee based, it's likely more diverse than datasets created by a small number of human labelers. And it is easier to identify, discuss, and debate dataset bias in the open.
As an academic researcher with a significant interest and time-investment in Transformer-based models, this restores my faith/hope in the trajectory of DL research. Considering it is difficult for academics to catch up to the industry regarding LLMs, seeing a continuation of the OPENness of these research works by a major industry player is a move in the right direction.
I've not done any real benchmarking but the OpenAssistant fine tuning from LAION has been done on it. It worked reasonably well for something local but definitely felt like it wasn't nearly as complete/advanced as any of the ChatGPT stuff. I imagine this Databricks setup is more complete there but I personally wouldn't expect too much more than GPT-3 level performance. That said if this dataset is open (I haven't really looked too much at the article yet) then you could quite easily use it to tune LLaMA just like the stanford alpaca models, which might be a better combo. Though that wouldn't be licensed for commercial use then given the underlying license.
Do we have any quantitative way of benchmarking the quality of these models at all? Like, I don’t care if a model takes one minute per token on my laptop as long as it’s “GPT-4 quality”, and I don’t care if it does 100 tokens per second if it’s straight crap. But every comparison I see people make regarding quality seems to come from “I asked it a couple of my favorite questions and it did… uh, only a little worse than GPT imo”
Perplexity scores are pretty common, which I think involves taking a text corpus like wikipedia and seeing how well the model predicts the next word (token) of it.
The irony is that the English is bad in a lot of the astroturf comments… why don’t they use the LLM to generate them? It would probably seem more genuine than the current comments.
Yes, this is a cool enough achievement on its own. HN is not the best place for these "testimonials". Unless this is some sort of demo of the LLM itself /s
Maybe OP, Data bricks founder, should reply to this comment and answer if he or anyone else in his team paid people to get this post with a not-so-useful-dataset to the top of HN.
I'm not sure what happened but I just sent an email out internally to ask people not to do this. The team might have gotten overly excited by this because they were all part of the creation of the dataset and the model.
WTF? They don't even invest effort in writing good text. This is either some very enthusiastic crowd of workers, who only know twitter and comment-sections, or a very bad marketing-bot. Makes me wonder if something like this often happens here, and I just never see it.
Astroturfing. Likely databricks told their employees to create HN accounts and comment on this thread to get traction (or just had one PR person make a bunch.)
@dang, any chance we can just ban all these accounts? Seems to be pretty cut and dry here.
69 comments
[ 3.9 ms ] story [ 82.9 ms ] threadIt also shows how to build and train these things on databricks, so maybe more people will use them to make custom trained LLMs.
They are siting on gigantic data flywheel of users usages.. (and private access to the best machine that can utilized that data..)
So, IMHO it would be hard to catch with their speed (not in this hype cycle).
If they're selling gpt-5 and let me pay for LLaMa or whatever is also out then I'll just use them unless pricing is wildly different.
Assuming you don't have sensitive data and that you never try anything outside the rules.
I'd be absolutely shocked if they don't launch a version where it's run on more secure setups, particularly as they've got huge Microsoft backing.
All datasets are biased, including this specific one. However, we believe it's still very valuable to open source, for a few reasons:
- This dataset is primarily used to train instruction reasoning, not for knowledge. (Keep in mind Dolly and any of the well known models have not been specifically trained for knowledge. They are all just demonstrating instruction reasoning.) The lack of a true open source (available for both research and commercial use) instruction dataset is the primary blocker for making these LLMs available for commercial use.
- We hope this will lead to not just open source innovations in models, but also future training datasets.
- Given the international population of our employee based, it's likely more diverse than datasets created by a small number of human labelers. And it is easier to identify, discuss, and debate dataset bias in the open.
I'm not sure what happened but I just sent an email out internally to ask people not to do this. The team might have gotten overly excited by this because they were all part of the creation of the dataset and the model.
@dang, any chance we can just ban all these accounts? Seems to be pretty cut and dry here.