HN Explorer

27 comments

[ 0.22 ms ] story [ 75.5 ms ] thread
serioussecurity
Wow nature got rolled. Should have stayed closer to their expertise. They were already being hustled by a lot of the applied AI work they were accepting.
[deleted]
(comment deleted)
mrkn1
subword tokenizers were never causal in the first place, so BPE was peeking at future bytes all along! TIL
armcat
There has been extensive research into token-free LLMs, but for some reason we are still operating in a token domain, so there is something to that.

Byte Latent Transformers (BLT): https://arxiv.org/abs/2412.09871

Charformer: https://arxiv.org/abs/2106.12672

gchamonlive
Aren't diffusion models token-free?
armcat
No, they still operate on tokens, it's just the difference in how text is sampled. It's done through iterative denoising and in parallel, across multiple token positions.
LoganDark
Nothing stops diffusion models from operating on byte-level tokens. Larger tokens are more space- and compute-efficient, though
razodactyl
It's because of "chunking" which human minds do as well. If you increase granularity, you increase permutations and required compute to match the same performance of existing systems.

GPT4's tokeniser for example made use of extended tokens for coding structures and with enough training on code was quite ahead of the rest for a while.

The tokeniser and its vocabulary makes a big difference.

andai
Isn't iterative chunking the whole point of the transformer though? Or does it add that much overhead to do it at the byte level?
numeri
What do you mean by iterative chunking? There are Transformer variants that hierarchically build token- or word-level representations from bytes, then reverse it to decide, but they never really took off. Didn't help much compared to a good tokenizer.
Ohentis
No. Most transformer variants work on either a flat stream, or a fixed sized internal state.
JonChesterfield
Weird paper. Models have had tokens for each byte for ages now. They can read and write individual bytes just fine, in addition to also having multibyte tokens.
Philpax
The point is not to have byte tokens: it's to have only byte tokens, so that the usual failures of tokenisation (e.g. the number of Rs in strawberry) can be avoided.
brookst
Would byte tokens really solve how many r’s? It seems like it would require a level of introspective awareness of the tokens being processed that I don’t think transformers have. But I’m not expert enough to go beyond that intuition, so could easily be wrong.
Ohentis
Models generally are capable of counting
brookst
Sure, but they could count bytes without being byte level tokens.

Can models introspect about the tokens that were passed to them?

andai
How can an LLM read individual bytes if all it gets is the tokenized ones? Or does it just get the ones that we didn't know how to tokenize?
mpyne 

    0 -> token_THE
    1 -> token_PART_prefix
    2 -> token_FULLSTOP
    ...
    0x3100 -> token_BYTE_00
    0x3101 -> token_BYTE_01
    0x3102 -> token_BYTE_02
and so on

The tokenizer would have to know to switch over to lexing as individual bytes rather than whatever else it might have been able to tokenize into but that seems like a question of labeling more than anything else.

JonChesterfield
It's written down in the tokenizer. 256 of the tens or hundreds of thousands of tokens correspond to each byte, done.
ricardobeat
[citation needed]

That's not how they work - even if you have those tokens available, the input prompt is tokenized in chunks. If you just read input bytes, there is no tokenizer. There is no existing mainstream architecture doing anything like that.

[deleted]
(comment deleted)
puttycat
I don't understand how this is different from BPE tokenization.