Ah but LLM.int8 (eg. as in huggingface transformers) isn't actually int8, it's a mixed precision encoding scheme that is nominally eight bits per parameter. This means custom cuda kernels etc, these kernels could be…
I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL…
Great opportunity to pitch my recent work on a similar solution for gRPC: https://github.com/sillystack/api/blob/main/transport/transp... I'm not going to make any security claims as of now (it's almost certainly…
I recently implemented a very similar service to service gRPC authentication mechanism for Go using EC25519 and the noise protocol framework. https://github.com/sillystack/api/blob/main/transport/transp... The code is…
Ah but LLM.int8 (eg. as in huggingface transformers) isn't actually int8, it's a mixed precision encoding scheme that is nominally eight bits per parameter. This means custom cuda kernels etc, these kernels could be…
I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL…
Great opportunity to pitch my recent work on a similar solution for gRPC: https://github.com/sillystack/api/blob/main/transport/transp... I'm not going to make any security claims as of now (it's almost certainly…
I recently implemented a very similar service to service gRPC authentication mechanism for Go using EC25519 and the noise protocol framework. https://github.com/sillystack/api/blob/main/transport/transp... The code is…