HN Explorer

17 comments

[ 4.3 ms ] story [ 41.0 ms ] thread
visarga
Mamba is the new and hot "linear transformer", could one day replace GPT based LLMs and scale sequence length up to 1M. It uses a clever math trick to parallelize token inference in the input while keeping a constant state size for generating the output.
erichocean
> while keeping a constant state size for generating the output

Isn't Mamba's attention mechanism linear relative to the input? The innovation as I understand it is that attention in Mamba *isn't* quadratic (like transformers).

junipertea
State size is constant (a matrix), attention is linear (a single unidirectional scan over all states)

Both statements are correct

prefer_to_ask
The innovation is that they managed to write a hardware-aware kernel to make it run fast/efficiently on GPUs. The authors of Mamba are the same authors of FlashAttention which was a performance optimization kernel written to reduce IO while computing normal O(N^2) attention. As I understand it, previously SSM models were not as easily parallelizable as the Transformer architecture.
kelseyfrog
Yes and no. There's an dual connection between ssms and convolutional models if certain constraints are met. Training convolutionally and inferring sequentially seeks a compromise between the two sides. I think we're about to find out the degree to which those constraints impact "easily".
wityl
They were previously parallelizable (via fft), but performed poorly on language modeling tasks.

Mamba adds a dependence on the inputs that makes language modeling competitive with transformers, but that prevents using the fft approach. So they switch to a method using parallel prefix scan.

nharada
Has anyone tried this? I’ve seen the same people hyping this up all over /r/ml with multiple posts a week, marketing material, etc. Is this a legit new technique or just cleverly marketed paper?
az226
Kind of weird we haven’t seen Mistral 7B Mamba. Until that’s out it’s untested vaporware.
neuronexmachina
Do you mean include it with the Mixtral "Mixture of Experts" model? I'm not sure Mistral Mamba makes sense, since it's a completely different architecture.
jasonjmcghee
This isn’t a fine tune. It’s not even a transformer like Qwen, Llama, GPT, Gemini, or Mistral. It’s another approach to language models based on RNNs
Solvency
Doesn't the point remain though -- when/where do we demo this?
junipertea
There is Mamba 7B pre trained model on hugging face/ LM studio right now.
intalentive
It’s legit and has been honed and refined for the past few years by the same Stanford PhD students (now graduated and working at CMU & Princeton).

The latest iteration has promising results for language but no one has yet trained and released a big (7B+) model to see how it scales.