HN Explorer

Show HN: Sparse Matrix-Vector Multiplication that works at 30–90% sparsity (github.com)

7 points by vlejd ↗ HN
To get benefits from sparsity, you usually need to have very sparse matrices, impose some structure on the sparsity pattern or have specialized hardware. None of it is the case if you want to rune pruned LLMs on consumer devices. I wanted to see how far can you push it on a GPU and ended up with this. Blog: https://www.grizzlytech.dev/blog/macko-spmv Paper: https://arxiv.org/abs/2511.13061 Code (example with torch): https://github.com/vlejd/macko_spmv

4 comments

[ 3.1 ms ] story [ 17.7 ms ] thread
jjgreen
Interesting approach -- thanks
[dead] fleahunter
[flagged]
vlejd
Interestingly enough, we found that cublas is not that well optimized for some less common consumer GPUs, specifically 3090. We saw that is didn't really achieve it's full potential for a lot of different matrix shapes, probably because of poor tuning. Interestingly enough, out kernel does not have any parameters, and it was able to outperform cublas even in setting where it has no right to do so.

Regarding patterns, we tested mainly random matrices and ones created by Wanda pruning. 2:4 sparsity (commonly used structure) will have same results as random matrix (probably even better). Interestingly enough, block sparsity could have very close to a worst case scenario with our format, because it promotes disproportional long sequences of zeroes.

Regarding other usecases, we are looking into it, but most common ones we found are usually for much smaller sparsity <1%. If you know about some other use case that is in the 30-90 range, let us know.

telmop
Cool method. Pre deep learning there was plenty of interesting research on sparse methods. What do you think we're missing to have more widely used neural+sparse approaches?