Oh that's nifty. Spotting base64 encoded strings is easy enough (and easy enough to test that I give it a shot if I'm even vaguely curious), but I'd never looked at them closely enough to spot patterns.
Something similar pops up if you have to spend a lot of time looking at binary blobs with a hex editor. Certain common character sequences become familiar. This also leads to choosing magic numbers in data formats that decode to easily recognized ASCII strings. I'm sure if I worked with base64 I'd be choosing something that encoded nicely into particular strings for the same purpose.
I discovered this when I created a JWT system for my internship. I got really good at spotting JWTs, or any base64 encoded json payloads in our Kafka streams
I don't really love this. It just feels so wasteful.
JWT does it as well.
Even in this example, they are double base64 encoding strings (the salt).
It's really too bad that there's really nothing quite like json. Everything speaks it and can write it. It'd be nice if something like protobuf was easier to write and read in a schemeless fashion.
The PEM format (that begins with `-----BEGIN [CERTIFICATE|CERTIFICATE REQUEST|PRIVATE KEY|X509 CRL|PUBLIC KEY]-----`) is already Base64 within the body.. the header and footer are ASCII, and shouldn't be encoded[0] (there's no link to the claim so perhaps there's another format similar to PEM?)
You can't spot private keys, unless they start with a repeating text sequence (or use the PEM format with header also encoded).
After staring one time too much at base64-encoded or hex-encoded asn1 I started to believe that scene in the Matrix where operator was looking at raw stream from Matrix at his terminal and was seeing things in it.
Base64 takes 3 bytes x 8 bits = 24 bits, groups that 24 bit-sequence into four parts of 6 bits each, and then converts each to a number between 0-63. If there aren't enough bits (we only have 2 bytes = 16 bits, we need 18 bits), pad them with 0. Of course in reality the last 2 bits would be taken from the 3rd character of the JSON string, which is variable.
The first 6 bits are 011110, which in decimal is 30.
The second 6 bits are 110010, which in decimal is 50.
The last 4 bits are 0010. Pad it with 00 and you get 001000, which is 8.
Mathematically, base64 is such that every block of three characters of raw input will result in four characters of base64'd output.
These blocks can be considered independent of each other. So for example, with the string "Hello world", you can do the following base64 transformations:
* "Hel" -> "SGVs"
* "lo " -> "bG8g"
* "wor" -> "d29y"
* "ld" -> "bGQ="
These encoded blocks can then be concatenated together and you have your final encoded string: "SGVsbG8gd29ybGQ="
(Notice that the last one ends in an equals sign. This is because the input is less than 3 characters, and so in order to produce 4 characters of output, it has to apply padding - part of which is encoded in the third digit as well.)
It's important to note that this is simply a byproduct of the way that base64 works, not actually an intended thing. My understanding is that it's basically like how if you take an ASCII character - which could be considered a base 256 digit - and convert it to hexadecimal (base 16), the resulting hex number will always be two digits long - the same two digits, at that - even if the original was part of a larger string.
In this case, every three base 256 digits will convert to four base 64 digits, in the same way that it would convert to six base 16 digits.
Besides that, I just spent way too much time figuring out this is an encrypted OpenTofu state. It just looked way too much like a terraform state but not entirely. Tells ya what I spend a lot of time with at work.
This is probably another interesting situation in which you cannot read the state, but you can observe changes and growth by observing the ciphertext. It's probably fine, but remains interesting.
The encoded JSON string is going to start with "ey", unless there's whitespace in the first couple characters.
Also, it seem like the really important point is kind of glossed over. Base64 is not a kind of encryption, it's an encoding that anybody can easily decode. Using it to hide secrets in a GitHub repo is a really really dumb thing to do.
Not directly correlated but I know a old guy that can decrypt EBCDIC and credit card positional data format on the fly. And sometimes it was a "feeling" he couldn't explain it properlly but knew exactly the value, name and other data.
It was amazing to see him decode VISA and MASTER transactions on the fly in logs and other places.
32 comments
[ 184 ms ] story [ 2413 ms ] threadJWT does it as well.
Even in this example, they are double base64 encoding strings (the salt).
It's really too bad that there's really nothing quite like json. Everything speaks it and can write it. It'd be nice if something like protobuf was easier to write and read in a schemeless fashion.
The PEM format (that begins with `-----BEGIN [CERTIFICATE|CERTIFICATE REQUEST|PRIVATE KEY|X509 CRL|PUBLIC KEY]-----`) is already Base64 within the body.. the header and footer are ASCII, and shouldn't be encoded[0] (there's no link to the claim so perhaps there's another format similar to PEM?)
You can't spot private keys, unless they start with a repeating text sequence (or use the PEM format with header also encoded).
[0]: https://datatracker.ietf.org/doc/html/rfc7468
It doesn’t even need to be much better than ROT13. Security by obscurity is good for this situation.
$ echo '{"' | base64
Vs
$ echo "{\"" | base64
{" is ASCII 01111011, 00100010
Base64 takes 3 bytes x 8 bits = 24 bits, groups that 24 bit-sequence into four parts of 6 bits each, and then converts each to a number between 0-63. If there aren't enough bits (we only have 2 bytes = 16 bits, we need 18 bits), pad them with 0. Of course in reality the last 2 bits would be taken from the 3rd character of the JSON string, which is variable.
The first 6 bits are 011110, which in decimal is 30.
The second 6 bits are 110010, which in decimal is 50.
The last 4 bits are 0010. Pad it with 00 and you get 001000, which is 8.
Using an encoding table (https://base64.guru/learn/base64-characters), 30 is e, 50 is y and 8 is I. There's your "ey".
Funny how CS people are so incurious now, this blog post touches the surface but didn't get into the explanation.
I know eyJhbG by heart
These blocks can be considered independent of each other. So for example, with the string "Hello world", you can do the following base64 transformations:
* "Hel" -> "SGVs"
* "lo " -> "bG8g"
* "wor" -> "d29y"
* "ld" -> "bGQ="
These encoded blocks can then be concatenated together and you have your final encoded string: "SGVsbG8gd29ybGQ="
(Notice that the last one ends in an equals sign. This is because the input is less than 3 characters, and so in order to produce 4 characters of output, it has to apply padding - part of which is encoded in the third digit as well.)
It's important to note that this is simply a byproduct of the way that base64 works, not actually an intended thing. My understanding is that it's basically like how if you take an ASCII character - which could be considered a base 256 digit - and convert it to hexadecimal (base 16), the resulting hex number will always be two digits long - the same two digits, at that - even if the original was part of a larger string.
In this case, every three base 256 digits will convert to four base 64 digits, in the same way that it would convert to six base 16 digits.
Besides that, I just spent way too much time figuring out this is an encrypted OpenTofu state. It just looked way too much like a terraform state but not entirely. Tells ya what I spend a lot of time with at work.
This is probably another interesting situation in which you cannot read the state, but you can observe changes and growth by observing the ciphertext. It's probably fine, but remains interesting.
1: https://rachelbythebay.com/w/2016/02/21/malloc/
Is this the state of modern understanding of basic primitives?
Also, it seem like the really important point is kind of glossed over. Base64 is not a kind of encryption, it's an encoding that anybody can easily decode. Using it to hide secrets in a GitHub repo is a really really dumb thing to do.
It was amazing to see him decode VISA and MASTER transactions on the fly in logs and other places.