Sorry, should have clarified. The part of the site that states the model is currently unavailable links to their evaluation policies which is the link used for this submission :)
> *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here].
That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section:
> Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing.
> Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps:
> 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users.
> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it.
> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).
It's possible they temporarily removed it because it broke the whole anonymised part of that. It literally had information about itself in its system prompt and people could deduce it was running on an OAI API because of its API error response formatting. Either OAI collected all the data they need or it'll be back with some changes.
They rate limited it at some point later in the day. The rate limiting applied to all the other OpenAI models also, but none of the non-OpenAI models. Just another data point of validation.
Yeah it's also possible they temporarily took it down because of the flood of users that were slowing down the whole site for everyone. Potential for it to be back but the rate limits are really low, something like 2 queries per user per day
Maybe they detected too many people were trying to extract training data from it. The purpose was clearly hype to squash any llama3 400B hype, I'm quite sure they don't need anonymous votes from users when they can afford other ways. Like someone else said in the thread, it could also be hype management to stop delicate people from being so freaked by novel capabilities on release.
> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it.
> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).
So... make the users do the work for free, but never publicly publish the data? That seems kinda scummy to me.
I was really motivated with the whole public ranking thing, but knowing I'm just doing work for another company with the data never being published sucks the fun out of it.
You get to use unreleased models for free and compare them to others, you are gaining some benefit from this. Why be annoyed about it? They aren't making you do anything, you're choosing to play around with these models in a "gamified evaluation" for whatever reason. If you want to use any of the released models without participating in crowdsourced training then there are tons of options out there to run many of them locally or to pay for private hosting or pay for a chatbot as a service. I don't get what the problem is, they're footing the bill for you after all.
What a single person gains from this is extremely little unless they keep track of every vote they do, and do >100 of votes. The public ranking allows you to not have to do that.
If there was an opt-out for private model voting, I would choose that simply because I don't care to give some corporation extremely useful data for so little pay.
13 comments
[ 2.5 ms ] story [ 42.6 ms ] thread> *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here].
That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section:
> Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing.
> Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps:
> 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users.
> 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it.
> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).
> 4. Remove the model from Arena.
> 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details).
So... make the users do the work for free, but never publicly publish the data? That seems kinda scummy to me.
I was really motivated with the whole public ranking thing, but knowing I'm just doing work for another company with the data never being published sucks the fun out of it.
If there was an opt-out for private model voting, I would choose that simply because I don't care to give some corporation extremely useful data for so little pay.