it costs 0.001$ per 1K which is slightly cheaper than GPT-3.5-turbo. I have just tested it and it shows extremely worse results on the tasks in my pipelines. Not a game change, unfortunately.
This seems to be the general pattern so far. A particular benchmark shows better or equal performance for a non-openai model, but then someone else tries a different task and it's just not even close.
I think that's really significant. It's arguably a case for fine-tuning models for specific tasks. That's great for teams with ML experience. But for product engineering teams without ML engineers, they can just use a foundational model and get great performance for a low cost.
I think even if it's dishonest, it's still representative of a bullish case for fine-tuning. I'm excited to see people get involved in that to make it easier, because...whew, it's not easy.
I think in a vacuum it's not dishonest, but it is half-baked in a way a lot of these benchmarks are.
Time and time again we see benchmarks like this where 0 effort into actually trying to get anything good out of the model.
Of course you can make extremely specific changes that essentially game the test, so to me a fair way to test this is to establish a realistic token budget across all the models, maybe even adjusted for cost depending on the goals of your testing. Then for each model, use your budget to get the highest quality result possible.
_
You're asking the GPT models for example, in the worst possible way. You've moved detailed instructions into the user prompt, despite their newest updates focusing on steerability via the system. Meanwhile from tinkering LLAMA 2 pretty much doesn't care about the system prompt vs user prompts.
You're also not allowing for any form of chain-of-thought. Giving the models a few hundred tokens to form a conclusion instead of spending those tokens trying to force out a mathematically induced order bias (which I wouldn't expect to do much) would have been much better.
There's also no way that GPT 4 should have struggled to give you a well defined output: All of the models would have probably benefited from a well defined output format in terms of a schema, rather than asking for a single letter response. The model rarely has to produce a single letter answer and will struggle with that. Something like asking for { "output" : "A_IS_MORE_FACTUAL" | "B_IS_MORE_FACTUAL" } would have been better
Overall I think there's no open dishonesty, and obviously there's no objective correct amount of effort to put into the test setup here... but there's a conflict of interest that would have made me want to see more effort put into it. I think there were trivially low hanging fruit ignored here that I'd expect people selling LLMOps to have pick up on.
They didn’t test it on other tasks and yes, it seems there’s no particular reason to believe that the results generalize.
If someone wants to try it, they can, but I think it’s based on some unwarranted assumptions that LLM’s are likely to have balanced performance, just because there are some that do seem to be pretty balanced?
Something to realize is that different models require different prompting styles. You can't prompt non-gpt4 models with GPT4 tuned stylistic ticks and expect similar results.
I've gotten great performance from llama2 derivatives. Out of the box performance is not near GPT4 but it is still very strong in its own right. And, if you are able to break down your problem so precise logit control coupled with guidance from forward or backwards chaining on knowledge graphs is applicable, you can easily exceed gpt4's reasoning ability for your domain. No fine-tuning necessary either.
I've been getting useful things out of LLMs since the days of roberta and raw T5, when Large stood for hundreds of millions of parameters. I am flabbergasted when people say a 7B parameter model is no good for them.
I'm going to butcher this explanation - after you've generated your selection of logits but before you sample from them, you check which ones conform to your schema. If you want the only two options to be "true" or "false", then you take any of the logits that would provide invalid answers and lower their probabilities manually.
Another example is structures like JSON can be validated so when your sample is "{'name':'Carl'" you lower the probability of "{" since that would invalidate the json. In fact the only valid ones you'd likely have left would be ",", " ", and "}"
[Author] Can you cite some examples of this so we're not talking in the void?
I've actually tested Llama 2 for summarization but haven't blogged about that yet, and across multiple domains, Llama 2 is pretty good.
I do see some differences -- Llama 2 doesn't follow instructions as cleanly, and is more verbose.
It's not a plugin replacement -- in the article itself I point it out. GPT 3.5 largely followed instructions to return A or B. Llama 2 didn't so I had to use another LLM to post-process.
I'm not saying you should always use Llama 2 or ChatGPT. It's that for some use cases, you can save a lot of money by using an open source alternative.
[Author] Can you share more details? How does it fail? What type of domain? Perhaps there is some tweaking to prompts that's required? It would help to understand what you're seeing.
We dedicated an entire team of 6 for two months on evaluating LLMs and while Claude 2 was the final choice, we found Llama 2 70b to be absolutely great (for summaries and structured data generation from free text).
We chose Claude because running Llama on an A100 or H100 comes with a baseline cost that doesn’t go to zero when you don’t need it (you could spawn a new instance but right now GPUs are so rare everywhere except for expensive cloud providers that it’s possible you don’t get one).
That said, we found the smaller Llama models to be so hilariously bad we have an internal slack channel where Llama 7b writes jokes and the funny part isn’t the jokes but how utterly stupid and random they are.
This is ignoring OpenAI's margins. We don't know how much GPT-3 and GPT-4 actually cost for them to run but it isn't what they are charging us. For Llama 2 the quoted cost is just compute, but for OpenAI you are also paying for using their software.
[Author] As an end user, you pay the published price of $1 per million tokens for alternatives (such as Anyscale Endpoints -- https://app.endpoints.anyscale.com/landing, but there are others -- Replikate, Fireworks etc). This isn't just the compute price -- there's some margin in there as well.
Per the Llama 2 license, companies with "greater than 700 million monthly active users in the preceding calendar month" must get approval and permission from Meta to use it.
For OpenAI - that is very interesting though. The 700M cap means the big FAANG+ companies that Meta competes with - Google, Apple, Amazon, Netflix, even Microsoft - can't use Llama 2.
But OpenAI has less than 700 million users? So theoretically, if Llama was better than GPT models, OpenAI could replace their GPT engine with Llama :)
Nice idea and all, but I think their methodology was totally unsuited to the task. What they actually assessed was whether an LLM can recognize an accurate summary of a given article, not whether the LLM can produce an accurate summary itself. Here's their description of it:
> We used a 3-way verified hand-labeled set of 373 news report statements and presented one correct and one incorrect summary of each. Each LLM had to decide which statement was the factually correct summary.
The problem with this approach is its assumption that if an LLM can recognize an accurate summary, it'll be able to reliably produce accurate summaries. We know very little about the inner workings of LLMs right now, and what we do know suggests that they work highly counterintuitively, so I think there's no basis to make this assumption.
[Author] I acknowledge this and the article does as well. What I would say is that -- at the very least -- you could apply factual verification as a post process (so stage 1 is create the summary, stage 2 is validate the summary is factually accurate).
Anyscale's business model was completely disrupted by OpenAI. They are trying to shift to provide hosting/fine tuning for open source LLMs, but the model performance will get crushed by Gpt4/newer open AI models. In theory the alternative to openAI models is nice, in reality anyscale is now competing with Azure/AWS/etc to provide model hosting.
Their original compute platform for running arbitrary ml workloads will become obsolete as the industry consolidates around LLMs.
I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way there are a lot of problems that LLMs just aren't suited for.
Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed:
1. a model for sentiment analysis
2. a model for summarization
3. ...other NLP tasks
other tasks:
4. a model for object detection in images
5. a model for face recognition in images
Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis
LLMs come with a higher cost per transaction compared to dedicated models.
imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models.
Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)
The cost will drop like a rock in the coming years. Its always the same pattern with technology. Theres probably 5-20 Billion in capital working on solving this problem right now.
Wouldn't cost savings also translate to simpler models? I anticipate the process of distilling knowledge to simpler models will also become much cheaper and non-technical
To be fair you need to add the cost of new development and estimate when it pays bask. Without that it's not obvious that custom solution is better. For small tasks it's likely not.
I disagree that LLMs are always better for things like classification. But the barrier to entry is so low with them that it's usually worth trying to use an LLM before training some other model.
The whole concept of AGI/super intelligence is that one model can do everything. Thats whats happening with LLMS. If you believe in AI, then you believe that specific models will become obsolete
"Law of Diminishing Returns" .. your case is weakened by the repetitive confidence of the assertions, whether it is true or not remains to be seen via actual practice
Yeah definitely is speculative, but I worked in many NLP research teams in FAANG. Worked on large scale ML systems that support systems you have heard of. All of the work we did is now obsolete (except for cost), and the scientists skill sets were completely eroded.
That's for sure. The main problem is how to put them together. There are several ways. First, tokenizing the media and inserting it in the stream. Second, a media controller which accepts verbal instructions generated by LLM. They can be combined. Interesting variation is controller with (immediate) feedback. It can be anything, for example database access.
> I argue that it will even extend to things like time series analysis
AFAIK transformers have been used for low level robotic control. And LLM for high level have been reported by MS and Google.
[Author] I see it differently. I see that Anyscale built some really great infrastructure for scaling AI that could be applied to many domains. What we've seen is the market shift to LLMs, and we've been able to adapt our tech to new domains.
Our prices are competitive (starting at 25c per million tokens) because of the tech we've built that maximizes GPU utilization. That's what we do better than anyone else.
I still think GPT-4 is best LLM out there. It's just very expensive and you don't need all that horsepower you can save bucketloads of money.
Being in the industry it's actually quite the opposite.
OpenAI had a first mover advantage. ChatGPT has a good interface and GPT4 still leads on many benchmarks. But it continues to get worse and worse and others are catching up with significantly lower parameter counts. Their moat is shrinking very quickly.
Llama2 is an open source meta model, btw. Anyscale's ray makes it much easier to leverage the super fast pace of development in the OSS community (or released by large companies as OSS).
I also work on FAANG NLP research teams. I'm just saying that anyscale is now competing with AWS. Supposedly their moat is shrinking, Google has yet to release a model that competes with GPT-4. Anyone using these models on a daily basis is not impressed with Llama 2, and cannot build useful systems with it, aside from a narrow vertical.
Seeing this right now on the product side. I can't give away untuned GPT3.5, much less sell it. GPT4 is the absolute floor for what I can sell without setting up a pipeline.
> Anyone using these models on a daily basis is not impressed with Llama 2
Being at a FAANG adjacent company working with these models on a daily basis, Llama2 is more than sufficient for a large number of use cases. It ultimately depends what exactly you're looking for. Exposing the LLM directly as a chat interface to users? Then yeah, in many cases it won't feel sufficient. Using it for search, summarization, moderation, etc.? Then it's often more than capable.
FWIW GPT-4 is not one single model as you're aware and it evolves over time. In our experience it has objectively degraded in quality over time.
As for anyscale, ultimately it's more like databricks (and funny enough lots of ex databricks folks there). Anyone can just run hosted spark. Databrick's ecosystem has set them apart and it looks like that's what anyscale is building towards.
"It is not too much of a stretch to conclude that a system that is better at telling factual from non-factual sentences is better at not making them up in the first place – or alternatively could decide through a two stage process if it was being inconsistent."
Stretching aside, how does one follow from the other?
[Author] The problem is: how could you objectively evaluate the factuality in any other way? Summary ranking is an established approach in the literature.
If the system can evaluate the truthfulness of a statement, and it can generate statements, then it can create arbitrary numbers of candidate statements and filter for truthfulness.
[Author] This is what I meant. That even if it doesn't work at a single layer, you could build a "verifier" stage whose approach is similar to this verification process and it can make the decision.
Almost as good as GPT-4? I hear that claim quite often. And then when I test the claim, it falls far far short. I want real competition in this space. But currently, there is none. Except for maybe some very very corner case.
59 comments
[ 3.3 ms ] story [ 126 ms ] threadI think that's really significant. It's arguably a case for fine-tuning models for specific tasks. That's great for teams with ML experience. But for product engineering teams without ML engineers, they can just use a foundational model and get great performance for a low cost.
We've shared the code openly, you can reproduce it yourself if you want.
Time and time again we see benchmarks like this where 0 effort into actually trying to get anything good out of the model.
Of course you can make extremely specific changes that essentially game the test, so to me a fair way to test this is to establish a realistic token budget across all the models, maybe even adjusted for cost depending on the goals of your testing. Then for each model, use your budget to get the highest quality result possible.
_
You're asking the GPT models for example, in the worst possible way. You've moved detailed instructions into the user prompt, despite their newest updates focusing on steerability via the system. Meanwhile from tinkering LLAMA 2 pretty much doesn't care about the system prompt vs user prompts.
You're also not allowing for any form of chain-of-thought. Giving the models a few hundred tokens to form a conclusion instead of spending those tokens trying to force out a mathematically induced order bias (which I wouldn't expect to do much) would have been much better.
There's also no way that GPT 4 should have struggled to give you a well defined output: All of the models would have probably benefited from a well defined output format in terms of a schema, rather than asking for a single letter response. The model rarely has to produce a single letter answer and will struggle with that. Something like asking for { "output" : "A_IS_MORE_FACTUAL" | "B_IS_MORE_FACTUAL" } would have been better
Overall I think there's no open dishonesty, and obviously there's no objective correct amount of effort to put into the test setup here... but there's a conflict of interest that would have made me want to see more effort put into it. I think there were trivially low hanging fruit ignored here that I'd expect people selling LLMOps to have pick up on.
If someone wants to try it, they can, but I think it’s based on some unwarranted assumptions that LLM’s are likely to have balanced performance, just because there are some that do seem to be pretty balanced?
I've gotten great performance from llama2 derivatives. Out of the box performance is not near GPT4 but it is still very strong in its own right. And, if you are able to break down your problem so precise logit control coupled with guidance from forward or backwards chaining on knowledge graphs is applicable, you can easily exceed gpt4's reasoning ability for your domain. No fine-tuning necessary either.
I've been getting useful things out of LLMs since the days of roberta and raw T5, when Large stood for hundreds of millions of parameters. I am flabbergasted when people say a 7B parameter model is no good for them.
What do you mean by this?
https://github.com/guidance-ai/guidance
I'm going to butcher this explanation - after you've generated your selection of logits but before you sample from them, you check which ones conform to your schema. If you want the only two options to be "true" or "false", then you take any of the logits that would provide invalid answers and lower their probabilities manually.
Another example is structures like JSON can be validated so when your sample is "{'name':'Carl'" you lower the probability of "{" since that would invalidate the json. In fact the only valid ones you'd likely have left would be ",", " ", and "}"
I've actually tested Llama 2 for summarization but haven't blogged about that yet, and across multiple domains, Llama 2 is pretty good.
I do see some differences -- Llama 2 doesn't follow instructions as cleanly, and is more verbose.
It's not a plugin replacement -- in the article itself I point it out. GPT 3.5 largely followed instructions to return A or B. Llama 2 didn't so I had to use another LLM to post-process.
I'm not saying you should always use Llama 2 or ChatGPT. It's that for some use cases, you can save a lot of money by using an open source alternative.
We dedicated an entire team of 6 for two months on evaluating LLMs and while Claude 2 was the final choice, we found Llama 2 70b to be absolutely great (for summaries and structured data generation from free text).
We chose Claude because running Llama on an A100 or H100 comes with a baseline cost that doesn’t go to zero when you don’t need it (you could spawn a new instance but right now GPUs are so rare everywhere except for expensive cloud providers that it’s possible you don’t get one).
That said, we found the smaller Llama models to be so hilariously bad we have an internal slack channel where Llama 7b writes jokes and the funny part isn’t the jokes but how utterly stupid and random they are.
this is the age (or year) of token price arbitrage
For OpenAI - that is very interesting though. The 700M cap means the big FAANG+ companies that Meta competes with - Google, Apple, Amazon, Netflix, even Microsoft - can't use Llama 2.
But OpenAI has less than 700 million users? So theoretically, if Llama was better than GPT models, OpenAI could replace their GPT engine with Llama :)
> We used a 3-way verified hand-labeled set of 373 news report statements and presented one correct and one incorrect summary of each. Each LLM had to decide which statement was the factually correct summary.
The problem with this approach is its assumption that if an LLM can recognize an accurate summary, it'll be able to reliably produce accurate summaries. We know very little about the inner workings of LLMs right now, and what we do know suggests that they work highly counterintuitively, so I think there's no basis to make this assumption.
Their original compute platform for running arbitrary ml workloads will become obsolete as the industry consolidates around LLMs.
1. a model for sentiment analysis
2. a model for summarization
3. ...other NLP tasks
other tasks:
4. a model for object detection in images
5. a model for face recognition in images
Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis
imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models.
Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)
The tech consolidator works by: if it reasonably can, it will.
PS5? Gaming machine, 4k disc player, streaming box, etc.
Smartphone? Internet, phone, texting, camera/photos, storage, shopping, identity, gaming, music, video, directions. Soon we'll add highly useful AI bots to the list.
Windows? Pretty much non-stop consolidated software from the industry into itself, across decades.
Also known as eating the ecosystem. GPT & Co. will eat their best plugins.
The point is that a model can do everything, not that it is the most cost effective or best way to do everything.
There are tradeoffs with all tech, and there always will be.
That's for sure. The main problem is how to put them together. There are several ways. First, tokenizing the media and inserting it in the stream. Second, a media controller which accepts verbal instructions generated by LLM. They can be combined. Interesting variation is controller with (immediate) feedback. It can be anything, for example database access.
> I argue that it will even extend to things like time series analysis
AFAIK transformers have been used for low level robotic control. And LLM for high level have been reported by MS and Google.
Ray (https://github.com/ray-project/ray) is available for anyone to use. You can also use Aviary (https://github.com/ray-project/aviary) to serve any of those models yourself.
Our prices are competitive (starting at 25c per million tokens) because of the tech we've built that maximizes GPU utilization. That's what we do better than anyone else.
I still think GPT-4 is best LLM out there. It's just very expensive and you don't need all that horsepower you can save bucketloads of money.
OpenAI had a first mover advantage. ChatGPT has a good interface and GPT4 still leads on many benchmarks. But it continues to get worse and worse and others are catching up with significantly lower parameter counts. Their moat is shrinking very quickly.
Llama2 is an open source meta model, btw. Anyscale's ray makes it much easier to leverage the super fast pace of development in the OSS community (or released by large companies as OSS).
FWIW GPT-4 is not one single model as you're aware and it evolves over time. In our experience it has objectively degraded in quality over time.
As for anyscale, ultimately it's more like databricks (and funny enough lots of ex databricks folks there). Anyone can just run hosted spark. Databrick's ecosystem has set them apart and it looks like that's what anyscale is building towards.
With GPT4 being OpenAI's year ago tech.
Stretching aside, how does one follow from the other?