I think possibly the greatest advantage Elixir has is offering the power of the Erlang/OTP system without weird quirks like Erlang’s string handling. I never have to think about it in Elixir, binary UTF-8 strings are the default and there’s plenty of functions in the standard library for dealing with grapheme clusters and the like.
Always wondered if Elixir loses anything by doing binary strings by default. Is there any problem-domain where it benefits you to have charlists over binary strings? (Lexing, maybe? I know leex expects its input to be a charlist, but I'm not sure whether that's just legacy of being developed before Erlang R5 when binaries were introduced.)
I've done a lot of Erlang, but no Elixir; as long as you can still access charlists, I don't see what you might lose by having a much better default. Changing "foo" to mean something other than a list of codepoints in Erlang would cause a lot of trouble, but Elixir started when assuming utf8 binaries was clearly the right answer.
Having a list of codepoints is kind of useful, but unicode gets pretty complex, some things are multiple codepoints but essentially indivisible (unicode flags, for example don't make much sense with only one code point), so codepoints aren't the useful unit that byte/characters are in 8-bit encodings.
> unicode flags, for example don't make much sense with only one code point
Complete tangent, but:
To me, the Unicode flag codepoints aren't really "combining codepoints with weird behavior" in the sense that e.g. a zero-width joiner is a combining codepoint with weird behavior; so much as they're a special alphabet that colors the text written in it as an ISO country code; where valid country-codes spelled out in that alphabet just happen to be ligature-rendered in a special way by font authors.
It makes just as much sense to have an isolated Regional Indicator Symbol Letter C (U+1F1E8) in text as it does to have an isolated regular C in text. (HN strips out the former, but it does have a visual representation!) In both cases, they're one letter of a "word", and don't make sense to use, but do make sense to mention. You can talk about "countries whose ISO country codes start with U+1F1E8" just like you can talk about "countries whose names start with C." This is unlike e.g. the individual surrogate-pair codepoints in UTF-16, which don't make sense to use nor mention on their own.
And it can also make sense to use the Regional Indicator Symbol Letters to spell out country-codes that don't exist, without the expectation that they'll be rendered as a flag at present; simply because you're adding a layer of semantic meaning to your text for machines (and potentially getting a future flag-symbol for humans as a bonus.) The meaning — "this text is colored to be an ISO country-code" — is still there, whether the text renders a flag or not. It's a forward-compatible encoding, in a way that "individual flag emojis for each country" wouldn't be.
Mind you, I'm on the fence about whether I like the Unicode Committee's choice to implement these codepoints as a special alphabet. On the one hand, I appreciate that these characters need to be their own set of semantically-distinct codepoints in some sense, so that there's no semantic ambiguity of these being interpreted by machine-parsers as "letters in a word", like there would be if these were letters spelled out in the Wingdings font. But on the other hand, would it have really hurt the design of Unicode so much to just come up with a semantic-meaning equivalent to the Variation Selectors, so that we could just use regular letters A+Z, plus a prefix codepoint on each that means "interpreted as an ISO country-code"? Semiotics Selectors, per se?
I had a junior that came from python that did not want to work in elixir because he refused to write unit tests and the fact that double quotes were different from single quotes (and double quotes were default) he couldn't deal with -- joke was on him because python3 made strings a whole lot more complicated!
Yeah the fewer number of quirks is great but, first class tooling like ExUnit and Mix, and otp extensions like Task, DynamicSupervisor, and Registry are pretty hot, if you ask me, not to mention a package registry so good that Erlang is being encouraged to migrate to it...
It’s not surprising that Unicode in Erlang is hard to get right. It’s hard to get right in general. Every time I read something like this that goes deeply into Unicode I come away with some fun new test cases.
LuaJIT is a different type of JIT than the BEAM JIT. LuaJIT is a tracing and optimizing JIT, like HotJava, code is interpreted until it's considered hot, and then is compiled/optimized into native code.
BEAM's JIT is different. It does some optimization, but it applies to all code loaded, and there's no fallback to an interpreter: the interpreter still exists, but you either run BEAM with the interpretter or BEAM with the JIT; there's no mixed mode, previous attempts at a BEAM JIT found that calling between JITed and interpreted BEAM code was very difficult, so this JIT takes a different approach. This approach makes it hard to apply in depth optimizations, because they would tend to delay loading. However, it eliminates the interpreter overhead completely.
As someone else pointed out, there is a proper JIT now, but not too many people are especially concerned about boosting single threaded perf to say Java levels. It just isn't what big users of OTP are concerned with, per se. It would be cool if the JIT get really good, but I'm not holding my breath.
14 comments
[ 2.9 ms ] story [ 40.0 ms ] threadHaving a list of codepoints is kind of useful, but unicode gets pretty complex, some things are multiple codepoints but essentially indivisible (unicode flags, for example don't make much sense with only one code point), so codepoints aren't the useful unit that byte/characters are in 8-bit encodings.
Complete tangent, but:
To me, the Unicode flag codepoints aren't really "combining codepoints with weird behavior" in the sense that e.g. a zero-width joiner is a combining codepoint with weird behavior; so much as they're a special alphabet that colors the text written in it as an ISO country code; where valid country-codes spelled out in that alphabet just happen to be ligature-rendered in a special way by font authors.
It makes just as much sense to have an isolated Regional Indicator Symbol Letter C (U+1F1E8) in text as it does to have an isolated regular C in text. (HN strips out the former, but it does have a visual representation!) In both cases, they're one letter of a "word", and don't make sense to use, but do make sense to mention. You can talk about "countries whose ISO country codes start with U+1F1E8" just like you can talk about "countries whose names start with C." This is unlike e.g. the individual surrogate-pair codepoints in UTF-16, which don't make sense to use nor mention on their own.
And it can also make sense to use the Regional Indicator Symbol Letters to spell out country-codes that don't exist, without the expectation that they'll be rendered as a flag at present; simply because you're adding a layer of semantic meaning to your text for machines (and potentially getting a future flag-symbol for humans as a bonus.) The meaning — "this text is colored to be an ISO country-code" — is still there, whether the text renders a flag or not. It's a forward-compatible encoding, in a way that "individual flag emojis for each country" wouldn't be.
Mind you, I'm on the fence about whether I like the Unicode Committee's choice to implement these codepoints as a special alphabet. On the one hand, I appreciate that these characters need to be their own set of semantically-distinct codepoints in some sense, so that there's no semantic ambiguity of these being interpreted by machine-parsers as "letters in a word", like there would be if these were letters spelled out in the Wingdings font. But on the other hand, would it have really hurt the design of Unicode so much to just come up with a semantic-meaning equivalent to the Variation Selectors, so that we could just use regular letters A+Z, plus a prefix codepoint on each that means "interpreted as an ISO country-code"? Semiotics Selectors, per se?
I had a junior that came from python that did not want to work in elixir because he refused to write unit tests and the fact that double quotes were different from single quotes (and double quotes were default) he couldn't deal with -- joke was on him because python3 made strings a whole lot more complicated!
BEAM's JIT is different. It does some optimization, but it applies to all code loaded, and there's no fallback to an interpreter: the interpreter still exists, but you either run BEAM with the interpretter or BEAM with the JIT; there's no mixed mode, previous attempts at a BEAM JIT found that calling between JITed and interpreted BEAM code was very difficult, so this JIT takes a different approach. This approach makes it hard to apply in depth optimizations, because they would tend to delay loading. However, it eliminates the interpreter overhead completely.