miven
No user record in our sample, but miven has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but miven has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
I'm really glad that these HNet-inspired approaches are getting traction, I'm a big fan of that paper. Though I wonder how much of the gains in this case are actually due to 75% extra parameters compared to the…
Correct me if I'm misinterpreting something in your argument but as I see it Matryoshka embeddings just sort the vector bases of the output space roughly by order of their importance for the task, PCA-style, so when you…
The ARC Prize Foundation ran extensive ablations on HRM for their slew of reasoning tasks and noted that the "hierarchical" part of their architecture is not much more impactful than a vanilla transformer of the same…
As far as I understand the "chunking" of input bytes is learned completely end to end, so it's basically up to the model to figure out how to most efficiently delineate and aggregate the information from the inputs…
According to [0] it's partly due to a key change they introduced in interleaving layers that use standard RoPE positional encodings and layers using what's called NoPE [1], not encoding positions at all and letting the…
That's absolutely correct, KV cache is just an optimization trick, you could run the model without it, that's how encoder-only transformers do it. I guess what I'm trying to convey is that the latent representations…
The key and value vectors are cached, that's kind of the whole point of autoregressive transformer models, the "state" not only survives within the KV cache but, in some sense, grows continuously with each token added,…
I'm not sure I understand what you're trying to say here, information between tokens is propagated through self-attention, and there's an attention block inside each transformer block within the model, that's a whole…
Chain of thought isn't exactly transparent either, you shouldn't fall into the pitfall of believing that the final sequence of tokens thinking about the task is the only processing the model actually performs during…
AFAIK retrieving documents that look like the query is more commonly avoided by using a bi-encoder explicitly trained for retrieval, those generally are conditioned to align embeddings of queries to those of relevant…
In section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models I'm not sure whether it wouldn't be more…
I mean, it's no free lunch, you still need to expend significantly more compute for the QLoRA training compared to any usual PTQ method, be it SpinQuant or any other more conventional quantization approaches.
Is there an intuitive reason why this ends up working this well compared to, say, applying some kind of thresholding to attention activations that are below average for a given head to filter that same attention noise…
Residual connections are also worth mentioning as an extremely ubiquitous adaptation, one will be hard-pressed to find a modern architecture that doesn't use those at least to some extent, to the point where the…
Yeah, it just got updated, here's the new link, they added sections on block-wise quantization for both the rounding-based and LUT-based approach: https://apple.github.io/coremltools/docs-guides/source/opt-p...
Huh, generally whenever I saw the lookup table approach in literature it was also referred to as quantization, guess they wanted to disambiguate the two methods Though I'm not sure how warranted it really is, in both…
> For on-device inference, we use low-bit palletization, a critical optimization technique that achieves the necessary memory, power, and performance requirements. Did they go over the entire text with a thesaurus? I've…
I don't get your point, how is what you're suggesting here different from a few papers we already have on KV cache pruning methods like [1]? [1] https://arxiv.org/abs/2305.15805
The authors mention that Jacobi decoding is equivalent to greedy autoregressive decoding, but in practice don't we often want the sampling temperature to be above zero to avoid repetitions and excessively generic…
Any guesses as to why they bumped the parameter count up from 7B to 8B?
What you describe here sounds a little like the line of work centered around Universal Transformers, which basically process the input embeddings through a single transformer block multiple times with a separate module…
I think the author of this method said it's coming in a month or so
Correct me if I'm wrong but usually when you do normal token by token inference in a transformer you store calculations made in the previous step in a KV cache so you can reuse it instead of calculating it all over…
Now that I think about it, doesn't this "technique" triple the amount of compute and memory per generated token since each model needs to also compute and store the KV values for the two previous tokens it didn't…
It was mentioned to be a 20B in a comparison table in a paper co-written by Microsoft, but they've since claimed that it's just an error, and I mean, they'd need to be sitting on some really impressive distilling…