From my experience with finetuning Gemma 12B, I noticed that 4-bit quantization would hurt instruction-following especially when it comes to NLP.
I have LLMs as players in a hidden-information word game. Models would often leak hidden info in their reasoning, so you had to filter what each side sees. Sometimes you have to be aware that these LLMs can be sneaky at…
[dead]
From my experience with finetuning Gemma 12B, I noticed that 4-bit quantization would hurt instruction-following especially when it comes to NLP.
I have LLMs as players in a hidden-information word game. Models would often leak hidden info in their reasoning, so you had to filter what each side sees. Sometimes you have to be aware that these LLMs can be sneaky at…
[dead]
[dead]