TL;DR: Uppercase and lowercase alphabet plus numbers, which would be 62 characters, except it omits uppercase I (as in India), lowercase l (as in lima), capital O (as in Oscar) and the number 0, as they're all visually similar and sometimes indistinct with other characters.
The Node module base-x (https://www.npmjs.com/package/base-x) can be used to encode and decode these using the alphabet 123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz
> Using standard Base64 in URL requires encoding of '+', '/' and '=' characters into special percent-encoded hexadecimal sequences ('+' becomes '%2B', '/' becomes '%2F' and '=' becomes '%3D'), which makes the string unnecessarily longer.
Apache commons codec has a base 64 encoder that has a url friendly mode where it substitutes those characters. I've used that a few times to work around issues like this.
I recently wrote a base64 encoder/decoder at work. Luckily, I did not need to make it compatible with other base64 encoders/decoders because, as the article points out...
> Base64 has eleven encoding variations that lead to confusion among developers on which variety of Base64 to use.
Each variation has its own niche (URL safe characters, filename safe characters, checksum support, etc). I see Base58 as basically just another variation of Base64 with its own niche (human friendly characters).
There's no need for existing apps to "convert". If their current Base64 implementation is good enough, they'll stick with it. Base58 is just another variation to consider when developing something new.
Base58 has existed for over a decade now, so I wouldn't exactly call it a "new" format at this point. And I've been using it for almost that long, so it's old hat to me.
Satoshi Nakamoto came up with the Base58 encoding in the original Bitcoin whitepaper. It's fair to credit him in some manner even if he didn't have any direct involvement in this most recent standards doc.
It's worth pointing out that several of the concerns mentioned here are specific to the Bitcoin protocol (e.g. SHA256^2 hashing) and aren't relevant to the use of Base58 as a general encoding scheme. The main issues that are non-Bitcoin-specific are: compatibility with QR encoding schemes, difficulty reading out loud mixed case strings, and higher efficiency encoding to Base32 (you can do it all with CPU-friendly bitwise operators).
So if you care about either of those things, then a Base32 encoding may well be good for you. But personally, my use of encodings like these tends to be for GUIDs and such, which I want to encode as a string, take up as little space as possible in URLs, and be maximally compatible with URL parameters and such. Base58 works great for that, and indeed I've been using it for that purpose for almost a decade now. There are some pretty serious efficiency losses from stepping down to Base32 that I don't want to incur (you're almost halving the size of your alphabet, so every byte you want to encode requires nearly another entire bit).
And, crucially, I'm not actually ever even decoding/encoding these strings back to their numeric value because the string itself then becomes the unique identifier (as it's way easier to work with in the fullstack from DB all the way on up through front-end than a byte or GUID type would be), so the encoding efficiency improvements that are salient for Bitcoin when you have thousands of these to decode in each block you want to validate just don't matter.
The critical argument for base32 over base58 in my mind is being able to write the ids as filenames/folder names on macOS without conflicts (APFS is case-insensitive by default).
Wow, how strange. I didn't realize MacOS did that. I don't actually use MacOS for anything development-related though (yay Linux), so it doesn't affect me. We're also not using these IDs in filenames, though I could see how one might want to do so.
...but why are you using base58--which notably focuses on a feature specifically for the thing you are saying you don't care about: ensuring that characters don't look like each other--instead of the nearly-universally-implemented (and much more efficient to execute) "url-safe base64" (which drops the = sign padding and replaces + and / with - and _).
I don't want any non-alphanumeric characters in my identifiers. It visually breaks up identifiers in misleading ways and doesn't always play nice with double-clicking. So I could have gone with a Base62 implementation over a Base58 one, but the latter is more common, and even still, it can be useful to reduce visual confusion because people (at least developers) will be looking at these identifiers, even if they're not typing them in.
These human-readable encoding systems are nifty, but a higher goal is to design secure systems that don't require humans to read or type (or copy and paste) random strings of bytes.
For techies, occasionally eyeballing a hash as an informal check will continue to be useful, but I can't imagine that confusing 1 and l would ever be an issue in that use case.
In a secure system there is always going to be an airgap with a need for human to check what went over the gap before committing the transaction. Nonrandom strings of bytes, such as names, are also spoofable, so not much better in this regard.
That's true -- the parser can just translate any lookalike errors into the canonical character.
But I wish the standard was to keep all digits and drop the lookalike letters. I can't think of a reason for the extra (human) burden of remembering whether it's 0 over O, or I over 1, etc.
Being liberal in what you accept is fine, but being strict in what you emit requires extra mental overhead.
It's nothing cataclysmic (and "on the developer" is a often a reasonable place to add the extra overhead) but in this case it seems like selecting 0 instead of O or o, and 1 instead of I or l, would have been more straightforward.
Selecting "1" (ASCII 49), and "o" (ASCII 111) is inconsistent and feels odd.
EDIT: OK, I can think of one possible justification: with zero disallowed, a leading zero can never be lost. In a short Base58 string, it's not unusual for all characters to be numeric, and some overzealous readers will interpret the whole value as an integer. See also: hex strings, US ZIP codes, ABA routing numbers, etc, vs Microsoft Excel. :)
EDIT2: Nevermind, that's a lame justification. The likelihood of a Base58 string of any useful length containing numeric chars only is very low, since the alphabet would be 17.2% numeric) unlike hex strings (61.5% numeric), or ZIP codes and ABA numbers (100% numeric). Additionally, IIRC Base58 was invented for BTC addresses, which necessarily started (at the time) with a "1" char (now "1", "3", or "bc1").
I think you're missing the fundamental point that the pair 0 O is very easily confused, but o O and o 0 are less easily confused, by virtue of differing height.
I do understand, but that's only true for [0Oo], and does not help with [1Il]. In either case, the risk of human confusion is removed by transliteration in parser code.
But on the parser implementation side, I need to remember that numeric 0 maps to lowercase o, but lowercase l maps to numeric 1.
My thought is that it would be simpler to only remember one rule: numeric chars are canonical, lookalikes are dropped.
Clearly, not a big deal -- you write it once, correctly, and move on (or try to!). That "once" has apparently stuck in my head, and now a few years later I still wonder about the reason for the design choice. :)
Every time someone posts about Base-X encoding I have a little nostalgia for the Base-94 encoding I wrote as probably the first program I wrote from scratch solo (my idea, my design, my implementation), or the funky Base-32 design we did for a dev tool.
We had a new team lead who was a night owl like me. One day everyone left early but he and I were sitting around, and one of us, I forget who, was trying to implement a URL slug format that had the following qualities, ordered by priority:
* does not require URL encoding
* can be dictated or transcribed easily (it was a dev tool, in a time when pasting a URL into chat was not a given)
* as much as possible, could not contain swear words
We nerd-sniped each other into spending a couple of hours coming up with an odd-ball base-32 implementation with no homographs, and also that as long as you have vowels and leetspeak numbers you'd get swear words.
So we started with base 64 minus the symbols, vowels+y, homographs and leetspeak numbers, leaving us somewhere around 40 symbols left, so we just started removing consonants common in swears, starting with epithets and working backward. shtccks4vr is a bad url, but stick c--t or the n-word in a URL and HR is going to definitely have a talk with you then write an angry letter to us. Especially with a selection of choice verbs in the same slug...
I must admit -- I find the comments here which mention concern about encodings containing swear words or naughty words to be ridiculous. Most encoded values are not exposed to users other than long query string values as http session tokens and such. Are we trying to spare the virgin eyes (ROFL) of our fellow engineers who may browse through database rows? Seems like wasted mental cycles to me.
56 comments
[ 3.3 ms ] story [ 120 ms ] threadShameless plug: I have a port of it in Go (https://github.com/eknkc/basex) and .NET (https://github.com/eknkc/basex-dotnet)
https://en.wikipedia.org/wiki/Base64#URL_applications
> Using standard Base64 in URL requires encoding of '+', '/' and '=' characters into special percent-encoded hexadecimal sequences ('+' becomes '%2B', '/' becomes '%2F' and '=' becomes '%3D'), which makes the string unnecessarily longer.
Replaces '+' with '-' and '/' with '_' while omitting the '=' padding.
If you're using Java, it's right in the standard library:
`Base64.getUrlEncoder().withoutPadding().encode(value)`
As an example, this is used in Bitcoin addresses to ensure you don't accidentally type the wrong characters if copying an address by hand.
> Base64 has eleven encoding variations that lead to confusion among developers on which variety of Base64 to use.
Each variation has its own niche (URL safe characters, filename safe characters, checksum support, etc). I see Base58 as basically just another variation of Base64 with its own niche (human friendly characters).
There's no need for existing apps to "convert". If their current Base64 implementation is good enough, they'll stick with it. Base58 is just another variation to consider when developing something new.
https://tools.ietf.org/html/draft-msporny-base58-02
Also of note is that Bitcoin has since moved onto bech32 - the shortcomings of base56 are discussed in the first section:
https://en.bitcoin.it/wiki/BIP_0173
Pieter Wuille’s talk on the design and motivations is great:
https://youtu.be/NqiN9VFE4CU
S. Nakamoto, BitCoin
Huh?
https://github.com/bitcoin/bips/blob/master/bip-0039.mediawi...
There is no mention of Base58 in the whitepaper. http://bitcoin.org/bitcoin.pdf
So if you care about either of those things, then a Base32 encoding may well be good for you. But personally, my use of encodings like these tends to be for GUIDs and such, which I want to encode as a string, take up as little space as possible in URLs, and be maximally compatible with URL parameters and such. Base58 works great for that, and indeed I've been using it for that purpose for almost a decade now. There are some pretty serious efficiency losses from stepping down to Base32 that I don't want to incur (you're almost halving the size of your alphabet, so every byte you want to encode requires nearly another entire bit).
And, crucially, I'm not actually ever even decoding/encoding these strings back to their numeric value because the string itself then becomes the unique identifier (as it's way easier to work with in the fullstack from DB all the way on up through front-end than a byte or GUID type would be), so the encoding efficiency improvements that are salient for Bitcoin when you have thousands of these to decode in each block you want to validate just don't matter.
The option to use the case-sensitive version of HFS+ and now APFS never really caught on.
The part "Test Vectors" seems to be formatted wrongly.
2NEpo7TZRRrLZSi2U is "Hello World!" and the long stuff probably the fox thingy.
Otherwise it is a good idea in my opinion, didn't know it existed and should perform well.
For techies, occasionally eyeballing a hash as an informal check will continue to be useful, but I can't imagine that confusing 1 and l would ever be an issue in that use case.
But I wish the standard was to keep all digits and drop the lookalike letters. I can't think of a reason for the extra (human) burden of remembering whether it's 0 over O, or I over 1, etc.
Being liberal in what you accept is fine, but being strict in what you emit requires extra mental overhead.
It's nothing cataclysmic (and "on the developer" is a often a reasonable place to add the extra overhead) but in this case it seems like selecting 0 instead of O or o, and 1 instead of I or l, would have been more straightforward.
Selecting "1" (ASCII 49), and "o" (ASCII 111) is inconsistent and feels odd.
EDIT: OK, I can think of one possible justification: with zero disallowed, a leading zero can never be lost. In a short Base58 string, it's not unusual for all characters to be numeric, and some overzealous readers will interpret the whole value as an integer. See also: hex strings, US ZIP codes, ABA routing numbers, etc, vs Microsoft Excel. :)
EDIT2: Nevermind, that's a lame justification. The likelihood of a Base58 string of any useful length containing numeric chars only is very low, since the alphabet would be 17.2% numeric) unlike hex strings (61.5% numeric), or ZIP codes and ABA numbers (100% numeric). Additionally, IIRC Base58 was invented for BTC addresses, which necessarily started (at the time) with a "1" char (now "1", "3", or "bc1").
But on the parser implementation side, I need to remember that numeric 0 maps to lowercase o, but lowercase l maps to numeric 1.
My thought is that it would be simpler to only remember one rule: numeric chars are canonical, lookalikes are dropped.
Clearly, not a big deal -- you write it once, correctly, and move on (or try to!). That "once" has apparently stuck in my head, and now a few years later I still wonder about the reason for the design choice. :)
https://duffyjp.github.io/2019/04/12/Base-52.html
[110910, 91980, 6424802, 44, 5910625, 44, 241512485, 4852678, 5696324, 5626611, 97, 43196449936799].map(&:base_52).join(' ')
We had a new team lead who was a night owl like me. One day everyone left early but he and I were sitting around, and one of us, I forget who, was trying to implement a URL slug format that had the following qualities, ordered by priority:
* does not require URL encoding
* can be dictated or transcribed easily (it was a dev tool, in a time when pasting a URL into chat was not a given)
* as much as possible, could not contain swear words
We nerd-sniped each other into spending a couple of hours coming up with an odd-ball base-32 implementation with no homographs, and also that as long as you have vowels and leetspeak numbers you'd get swear words.
So we started with base 64 minus the symbols, vowels+y, homographs and leetspeak numbers, leaving us somewhere around 40 symbols left, so we just started removing consonants common in swears, starting with epithets and working backward. shtccks4vr is a bad url, but stick c--t or the n-word in a URL and HR is going to definitely have a talk with you then write an angry letter to us. Especially with a selection of choice verbs in the same slug...