HN Explorer

Ask HN: How would you know if an AI model has been nerfed?

2 points by gitgud ↗ HN
If consumers are just calling a model in the cloud, what’s actually enforcing that the model is not just a dumbed down cheaper model?

What’s stopping anthropic/openai from just running a cheaper LLM model based on how difficult the question is?

1 comment

[ 2.3 ms ] story [ 24.8 ms ] thread
verdverm
To detect, you need evals

To have guarantees, contracts

When I use the API, I specify the model. They may use a quant'd version, but they aren't going to change the underlying model (caveat that I'm calling the quant and not the same model in this context)