Vera: Vector-Based Random Matrix Adaptation

VeRA makes LoRA ~10x more parameter efficient while retaining the same performance.

It's somewhat like a recursive LoRA scheme, where the LoRA A and B matrices are also decomposed using two small trainable vector parameters.

code-master 2y ago ↗

Is the code available anywhere

code-master 2y ago ↗

Highly skeptical paper 1) Authors havent released code 2) They claim its published in ICLR , but i wasnt able to find the paper in openreview https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-... 3) GPT-4 to evaluate performance. It seems to be grading the responses rather arbitrarily and not really considering how well the adapted model has learned the particular behavior it was being tuned for.

This just screams 0% code 0% peer review 100% Hocus Pocus.

Vera: Vector-Based Random Matrix Adaptation (dkopi.github.io)

3 comments