I love love love Unsloth and everything they do, so do not take what I am about to say as criticism of them.
But what's the point? GPT-OSS is regarded as a pretty bad open source model compared to the latest deepseek or qwen releases. Most attempts to use Reinforcement Learning or even any kind of post-training fail in that the data you have is of worse quality and quantity than the data that the model was originally trained on. So you get catastrophic forgetting and a model with lower general IQ than before fine-tuning.
This is true btw even if you use lora or better techniques to supposedly "mitigate" catastrophic forgetting. Even pyreft/reft, which in some cases impact only 0.001% of a models parameters, cause these kind of issues in my experiments.
So why should anyone except AI researchers and the big 4 AI providers care about fine-tuning? The vast majority of people who think they need fine-tuning need good quality RAG/Agentic RAG systems, since they can trivially add or remove data to their model (machine unlearning doesn't work yet), also ground models and objectively makes them more accurate, and fully manipulate and manage how it's used in their prompts context. On top of that, vector DBs/embeddings "easily" scale to billions of records.
Thank you unsloth for the amazing work once again!
The new sleep mode in vLLM is really amazing, and it seems the community hasn’t quite wrapped their heads around how much more accessible this makes RL training.
I’m reading a lot of dismaying posts on this thread, pushing the idea that only big labs should be doing RL. This couldn’t be further from the truth, folks should try it for themselves and see the outcomes!
5 comments
[ 2.8 ms ] story [ 26.1 ms ] threadBut what's the point? GPT-OSS is regarded as a pretty bad open source model compared to the latest deepseek or qwen releases. Most attempts to use Reinforcement Learning or even any kind of post-training fail in that the data you have is of worse quality and quantity than the data that the model was originally trained on. So you get catastrophic forgetting and a model with lower general IQ than before fine-tuning.
This is true btw even if you use lora or better techniques to supposedly "mitigate" catastrophic forgetting. Even pyreft/reft, which in some cases impact only 0.001% of a models parameters, cause these kind of issues in my experiments.
So why should anyone except AI researchers and the big 4 AI providers care about fine-tuning? The vast majority of people who think they need fine-tuning need good quality RAG/Agentic RAG systems, since they can trivially add or remove data to their model (machine unlearning doesn't work yet), also ground models and objectively makes them more accurate, and fully manipulate and manage how it's used in their prompts context. On top of that, vector DBs/embeddings "easily" scale to billions of records.
The new sleep mode in vLLM is really amazing, and it seems the community hasn’t quite wrapped their heads around how much more accessible this makes RL training.
I’m reading a lot of dismaying posts on this thread, pushing the idea that only big labs should be doing RL. This couldn’t be further from the truth, folks should try it for themselves and see the outcomes!
https://github.com/lukehinds/deepfabric/discussions/334