HN Explorer

Tell HN: ChatGPT Can Decode Base64

8 points by Supply5411 ↗ HN
Some engine must hooking into the LLM to answer questions like these. Are there any details about this process?

https://chat.openai.com/share/696def43-be52-4574-ad18-4f8ecf8557b0

EDIT: An even longer, more convincing test https://chat.openai.com/share/417169ed-8256-46cb-a3ed-81ab06a1d70b

27 comments

[ 22.6 ms ] story [ 1916 ms ] thread
Tiberium
That's been widely known for quite some time, there's no engine, it just happens "magically" when a LLM gets big enough (and they sure have a lot of that stuff in their training datasets). GPT-4 is generally very good at this, while Turbo will often make mistakes if you force it to decode Base64 that's some nonsense (like random characters).
Supply5411
Sorry but I have to call bs. How do you explain this? It decodes a huge (guaranteed unique) string perfectly:

https://chat.openai.com/share/417169ed-8256-46cb-a3ed-81ab06...

ipaddr
Too generic to be used as a test. Try changing the comments to include gjdckfkdkxkdkskx34717Q 22371 re-encode and report back
Supply5411
Huh, you're right. It got the random string close, but not exact. I'm still blown away... what in its training set could predict that random string as base64 that closely?

https://chat.openai.com/share/02d616c2-5667-454d-9b72-43dfb1...

moritzwarhier
What about the 1-1 relation between the binary encodings of ASCII-characters and their base64 equivalents? I dunno, but isn't a base64-encoded character sequence still unambiguous / aren't the encoded character sequences formint a prefix-free grammar when concatenated?

Still very impressive.

What's even more interesting are the edge cases as presented above. Now that we consider almost-perfect natural translation a given, we're obsessed with marrying this to basic rule-based logic.

Me too, in every conversation I had with ChatGPT.

Because marrying the two seems almost magical and scary.

Supply5411
There isn't a 1-1 of ascii character to base64 character is the thing. 3 ascii character = 4 base64 characters, so changing one of the input changes 1 or 2 of the output.
moritzwarhier
Fair enough, what I meant by 1:1 wasn't the length, more the prefix-freeness thing.

Trivially, there is a 1:1-correspondence between the set of all possible ASCII strings and the image of that set under Base64 encoding, because otherwise it wouldn't be an encoding anymore :)

This doesn't guarantee that a Base64-encoded string can't be identical to a substring of a different Base64-encoded string. And for arbitrarily short substring length (bit), this doesn't make sense either.

So we rely on some definitions and minimal data structures such as bytes.

What I meant was that AFAIK, the ASCII character set might have an image under Base64 encoding that allows for prefix-free encoding given a minimum read length.

muzani
It's quite inaccurate. It makes frequent mistakes especially in unknown words like names, typos, and swear words. And at some length, it hallucinates the rest of the content.

So it seems to be able to "feel" the answer out there but it's not directly translating it. That's far more fascinating than if it were decoding while hooked to something else.

Supply5411
I'm getting perfect results with very long strings. Can you share an example that fails for you?
muzani
Base64:

VGVsbCBFYS1uYXNpcjogTmFubmkgc2VuZHMgdGhlIGZvbGxvd2luZyBtZXNzYWdlOiAgV2hlbiB5b3UgY2FtZSwgeW91IHNhaWQgdG8gbWUgYXMgZm9sbG93cyA6ICJJIHdpbGwgZ2l2ZSBHaW1pbC1TaW4gKHdoZW4gaGUgY29tZXMpIGZpbmUgcXVhbGl0eSBjb3BwZXIgaW5nb3RzLiIgWW91IGxlZnQgdGhlbiBidXQgeW91IGRpZCBub3QgZG8gd2hhdCB5b3UgcHJvbWlzZWQgbWUuIFlvdSBwdXQgaW5nb3RzIHdoaWNoIHdlcmUgbm90IGdvb2QgYmVmb3JlIG15IG1lc3NlbmdlciAoU2l0LVNpbikgYW5kIHNhaWQ6ICJJZiB5b3Ugd2FudCB0byB0YWtlIHRoZW0sIHRha2UgdGhlbTsgaWYgeW91IGRvIG5vdCB3YW50IHRvIHRha2UgdGhlbSwgZ28gYXdheSEiICBXaGF0IGRvIHlvdSB0YWtlIG1lIGZvciwgdGhhdCB5b3UgdHJlYXQgc29tZWJvZHkgbGlrZSBtZSB3aXRoIHN1Y2ggY29udGVtcHQ/IEkgaGF2ZSBzZW50IGFzIG1lc3NlbmdlcnMgZ2VudGxlbWVuIGxpa2Ugb3Vyc2VsdmVzIHRvIGNvbGxlY3QgdGhlIGJhZyB3aXRoIG15IG1vbmV5IChkZXBvc2l0ZWQgd2l0aCB5b3UpIGJ1dCB5b3UgaGF2ZSB0cmVhdGVkIG1lIHdpdGggY29udGVtcHQgYnkgc2VuZGluZyB0aGVtIGJhY2sgdG8gbWUgZW1wdHktaGFuZGVkIHNldmVyYWwgdGltZXMsIGFuZCB0aGF0IHRocm91Z2ggZW5lbXkgdGVycml0b3J5LiBJcyB0aGVyZSBhbnlvbmUgYW1vbmcgdGhlIG1lcmNoYW50cyB3aG8gdHJhZGUgd2l0aCBUZWxtdW4gd2hvIGhhcyB0cmVhdGVkIG1lIGluIHRoaXMgd2F5PyBZb3UgYWxvbmUgdHJlYXQgbXkgbWVzc2VuZ2VyIHdpdGggY29udGVtcHQhIE9uIGFjY291bnQgb2YgdGhhdCBvbmUgKHRyaWZsaW5nKSBtaW5hIG9mIHNpbHZlciB3aGljaCBJIG93ZSg/KSB5b3UsIHlvdSBmZWVsIGZyZWUgdG8gc3BlYWsgaW4gc3VjaCBhIHdheSwgd2hpbGUgSSBoYXZlIGdpdmVuIHRvIHRoZSBwYWxhY2Ugb24geW91ciBiZWhhbGYgMSwwODAgcG91bmRzIG9mIGNvcHBlciwgYW5kIMWgdW1pLWFidW0gaGFzIGxpa2V3aXNlIGdpdmVuIDEsMDgwIHBvdW5kcyBvZiBjb3BwZXIsIGFwYXJ0IGZyb20gd2hhdCB3ZSBib3RoIGhhdmUgaGFkIHdyaXR0ZW4gb24gYSBzZWFsZWQgdGFibGV0IHRvIGJlIGtlcHQgaW4gdGhlIHRlbXBsZSBvZiBTaGFtYXNoLiAgSG93IGhhdmUgeW91IHRyZWF0ZWQgbWUgZm9yIHRoYXQgY29wcGVyPyBZb3UgaGF2ZSB3aXRoaGVsZCBteSBtb25leSBiYWcgZnJvbSBtZSBpbiBlbmVteSB0ZXJyaXRvcnk7IGl0IGlzIG5vdyB1cCB0byB5b3UgdG8gcmVzdG9yZSAobXkgbW9uZXkpIHRvIG1lIGluIGZ1bGwuICBUYWtlIGNvZ25pemFuY2UgdGhhdCAoZnJvbSBub3cgb24pIEkgd2lsbCBub3QgYWNjZXB0IGhlcmUgYW55IGNvcHBlciBmcm9tIHlvdSB0aGF0IGlzIG5vdCBvZiBmaW5lIHF1YWxpdHkuIEkgc2hhbGwgKGZyb20gbm93IG9uKSBzZWxlY3QgYW5kIHRha2UgdGhlIGluZ290cyBpbmRpdmlkdWFsbHkgaW4gbXkgb3duIHlhcmQsIGFuZCBJIHNoYWxsIGV4ZXJjaXNlIGFnYWluc3QgeW91IG15IHJpZ2h0IG9mIHJlamVjdGlvbiBiZWNhdXNlIHlvdSBoYXZlIHRyZWF0ZWQgbWUgd2l0aCBjb250ZW1wdC4g

ChatGPT's translation (3.5):

The decoded message is: "Tell EA-namir: Nanni sends the following message: When you came, you said to me as follows: 'I will give Gimp-Sin (when he comes) five shekels of silver.' You took your garments, even the garments of your back, just as you had promised, to give the messenger of Gimp-Sin (when he came) five shekels of silver, and then send my messenger. They grasped me, took me away and made me enter for the presence of Bel-Adda (the king)."

Errors: "EA-namir" instead of "Ea-nasir", "Gimp-Sin" instead of "Gimil-Sin", "five shekels of silver" instead of "fine quality copper ingots".

After that it's just hallucination.

muzani
GPT-4:

"Tell Ea-nasir: Nanni sends the following message: When you came, you said to me as follows : "I will give Gimil-Sin (when he comes) fine quality copper ingots." You left then but you did not do what you promised me. You put ingots which were not good before my messenger (Sit-Sin) and said: "If you want to take them, take them; if you do not want to take them, go away!" What do you take me for, that you treat somebody like me with such contempt? I have sent as messengers gentlemen like ourselves to collect the bag with my money (deposited with you) but you have treated me with contempt by sending them back to me empty-handed several times, and that through enemy territory. Is there anyone among the merchants who trade with Telmun who has treated me in this way? You alone treat my messenger with contempt! On account of that one (trifling) mina of silver which I owe(?) you, you feel free to speak in such a way, while I have given to the palace on your behalf 1,080 pounds of copper, and umi-abum has likewise given 1,080 pounds of copper, apart from what we both have had written on a sealed tablet to be kept in the temple of Samas. How have you treated me for that copper? You have withheld my money bag from me in enemy territory; it is now up to you to restore (my money) to me in full. Take cognizance that (from now on) I will not accept here any copper from you that is not of fine quality. I shall (from now on) select and take the ingots individually in my own yard, and I shall exercise against you my right of rejection because you have treated me with contempt."

Errors are significantly smaller:

"umi-abum" instead of "Šumi-abum"

"Samas" instead of "Shamash"

Some extra spacing errors, which other decoders seem to do too, so I take it's an issue with the encoding instead.

Again, seems to get the names wrong but other words correct. It's not a unique string. Actually, I googled the version given by GPT-4 and it matches one given by Reddit.

Supply5411
Thanks, confirmed. Do you have any resources where I can read more about this phenomenon?
Supply5411
GPT 4 gets it right. I understand that LLMs have some very rough emergent properties, like arbitrary arithmetic (but its often incorrect). I don't really understand how its able to base64 decode values it has never seen though. I'm using unique snippets from my own private source code.
Jensson
> I don't really understand how its able to base64 decode values it has never seen though

You are surprised by this but not by its ability to translate languages? You can train a base 64 decoder and encoder neural network trivially, so it is very easy to see how it could have one lurking among all of its many parts give it has likely seen many millions of base64 encoding examples on the internet.

Supply5411
I don't know why this surprises me but translating natural languages doesn't. I think it is the sustained accuracy.
shyn3
Wouldn't it be 100% accurate if it was trained on base64
cpach
Consider this (which is correct, unless I managed to include any brainfarts in my reasoning) – in the mapping between Base64 and ASCII, there are no unknown inputs. There’s no XOR or anything like that going on. It’s a pure mapping of byte values to byte values, just with an alternative representation. So there’s really no secret sauce in there, not even any obfuscation going on. Unlike AES or even a simple Vigenere cipher.
cpach
Perhaps I should have written bit patterns instead of bit values (:
pillefitz
Would be interesting to feed it into some open source LLM that for sure does not use an external decoder and determine whether it still gets it (approximately) right.
usgroup
Just reading the wikipedia page on Base64 (https://en.wikipedia.org/wiki/Base64). Algorithmically it seems to be a 8-bit byte to character mapping. So, if you were to base encode a character string it'd be {characters} -> {bytes} ->[base64] -> {characters}.

I'd presume this prompt would work fine:

Here is a mapping of numbers to letter given as a pairs: (A,1) (B,7) (C,23) (D,14). Use the mapping to convert the following string to numeric representation: ACCDAB.

I'd also presume that the same template could be used to instruct ChatGPT to do Base64 (with some additions for padding).

Yet, I suspect the capability given in the above prompt is not surprising to anyone. So, why is Base64 surprising? What am I missing?

cpach
One can indeed argue that it’s not surprising – depending on your prior understanding of how Base64 works.

But for a person where both Base64 and LLMs are black boxes, then it can indeed be surprising.

To me, the most surprising thing about ChatGPT and other LLMs are the quality (despite all their flaws) and ther general applicability to a broad range of diverse tasks.

Supply5411
>Algorithmically it seems to be a 8-bit byte to character mapping

Base64 is 6-bits, so there is no mapping of a single character to another single character. Changing 1 character in the input can change 1 or 2 characters in the output because each of the 6-bit "characters" are up against each other. So there is no mapping of input character to output character(s), it depends on context.

I'm aware that regular NL translations are a similar problem, but with base64, it's a much more exact and unforgiving problem, and GPT-4 seems to perform at a very high level.

usgroup
Ok I see, but that doesn't strike me as different in kind, is it? I.e. bytes to 6-bit sequences for which there is a mapping table. Now you have a string of 6 symbol sequences. Then divide the string into 8 symbol sequences and map each to a character (again via mapping table). Ignoring padding rules, its two mappings, one followed by the other?
thesimpleone
I would be shocked if it couldn’t decode base64.
lfconsult
Does ChatGPT integrates basics known encoders/decoders aside from LLM (if not, I thought that it was the case)?

Base64 is about encoding not hashing... So decoding it is... natural. The point here is that it's integrated inside ChatGPT? It seems to me that it's an "obvious" "feature" kind of.