Er, okay. In the tech space, there are often words that refer back to some real or de-facto owner, or strong pattern of historical use. I'm saying this is a case where that isn't true. Making definitive statements about what is, and what isn't a transpiler, is difficult.
It always seems odd to me that there's the separate word "transpiler" that seems to mean compiler that targets JavaScript. I don't really encounter it anywhere else.
I guess I would just use the. word compile. I'm not sure the "similar level of abstraction" distinction always holds a lot of meaning and just feel like splitting hairs to me. But maybe I'm missing something.
I think that was an attempt to exclude traditional compilers from being transpilers. Like 'C' is not at a similar level of abstraction to ASM or a compiled executable.
The first transpilers where used to translate between different assembly dialects to help with porting software between platforms. The term then kind of disappeared until the JavaScript community resurrected it
I agree with the article, there is no meaningful distinction between a compiler and what people typically call a transpiler except if you have a two way translation.
Also „machine code“ is not even the last step anymore, it gets further cached, interpreted compiled to instructions by the hardware. It’s abstraction layers all the way down.
No, transpilers are completely different than compilers.
For a transpiler, you generally need a tokenizer, then a parser to parse the token stream into an AST, which you will then have some set of transformation steps, often transforming into other ASTs for optimization steps or incremental transforms to the final target, before you take the final AST and serialize it out into the target.
Wherease for a compiler, you generally need a tokenizer, then a parser to parse the token stream into an AST, which you will then have some set of transformation steps, often transforming into other ASTs for optimization steps or incremental transforms to the final target, before you take the final AST and serialize it out into the target.
If I was hiring for a compiler position and someone showed up with 10 years of transpiler authorship on their resume, I wouldn't even look at them. Such divergent experience would be useless. It would be like hiring someone for a Javascript backend position when all they had was 10 years of intensive Javascript frontend experience. Madness. You'll go out of business making stupid hiring decisions like that.
This post is oddly adamant about a point that it makes with (what it acknowledges to be) very weak evidence. If the Typescript compiler is really a compiler, then JSweet is surely also a compiler? Both go from a stricter type system to a laxer one, and neither is really reversible.
What really comes through is that the author seems to think that transpiler status is somehow lesser, and the whole nitpicking about compilers/transpilers is just an attempt to make sure Nim lands squarely on the compiler side.
Maybe the conclusion that should be reached is the one the author is hinting towards at the end - which is that transpiler and compiler don't really have a strong distinction, and their usage is muddied enough that saying whether something is "actually" a transpiler or a compiler is mostly pointless. (And then there would be less worry that people "mistakenly" call Nim a transpiler, eh?)
Author here, I specifically tried to pick projects which self-identified as transpilers in order to not accidentally call a project I didn't know enough about a transpiler incorrectly. I don't in any way think that a transpiler is lesser, but a lot of the questions we've gotten about it in the Nim community has been phrased in a way which makes be believe that some people do. Hence why the article is written with that counter-stance.
The conclusion is definitely that these are loose terms that can apply to many different things. But some of the usages are more useful as differentiating terms than others. The very broad definitions of transpilers ends up making most if not all compilers into transpilers which isn't terribly useful, and the in-between ones I've found are so under-defined that the term ends up not meaning anything at all. This was merely my attempt at reaching a definition of transpiler that is not only a useful distinction from a compiler, but also serves as a more concrete term.
It's meant to be used in print, and print has bleed. It's too thin, by far, when it is used on screen (or even modern laser printers): see https://www.levien.com/type/cmr/gain.html
Why not? I'm interested because I mostly hear good things about the way LaTeX output looks, and as an amateur font geek I'd like to understand people's objections to it. Sure, it's off topic, but that happens aplenty on HN no?
It's too thin, when I print it the "e" and the "c" becomes indistinguishable, and even on a 4K screen it looks bad. A font I've found with similar aesthetics but without these issues is Charis Sil.
I mean, to me it is a transpiler because it is a "frontend" to several other languages.
You choose whether you want to emit C, C++, or JS. There's an experimental, fledgling community LLVM frontend for Nim, that is what I would call a "compiler":
> nlvm does not:
- understand C - as a consequence, header, emit and similar pragmas will not work - neither will the fancy importcpp/C++ features
- support all nim compiler flags and features - do file bugs for anything useful that's missing
One of the unique and best things about Nim (IMO) beyond being multi-target, is that you can do things like say "emit this exact C/C++ string", or "import this type from an existing header".
With this LLVM backend, that doesn't work, and it doesn't support JS (which probably isn't a big deal to most people).
That means you can't do stuff like this:
type
VectorIterator {.importcpp: "std::vector<'0>::iterator".} [T] = object
# std::vector<int>::iterator x;
var x: VectorIterator[cint]
With the '--debugger:native' flag, the Nim debugging experience isn't too bad. Not on the level of Java or C#, but you can set breakpoints, step in/over/out of code and inspect or watch variables without having to look at C code. There are some issues with type visualization (to see the contents of a string you have to evaluate something like '(char*)variable->data') but Nim ships with a gdb plugin that should help (I haven't actually tried it).
It could definitely be better, and hopefully some day it will be, but I use Nim in a debugger quite frequently and it usually works out pretty well.
well what does make emitting C different from emitting LLVM IR? neither backends are meant to emit human readable code, the LLVM IR is also a "language", etc...
Hence the "it's pedantic" ending part of my argument.
To me, if the thing that gets emitted is another regular programming language, you've done transpilation.
If the thing that gets emitted is IR, bytecode, object code, essentially anything that's not something you would read/write yourself and is a step further towards being executable, it's "compilation".
I see transpilation as lateral/horizontal movement on the "path to being able to run it" chain, and compilation as vertical movement towards the executable end-goal.
What if it just emitted C as its IR instead? MLton (the SML compiler) uses C as its primary backend (while they also have an x86 one) for portability and ease of implementation. By doing so, it bundles a small runtime needed for SML (garbage collection, closures, and such). Would you also consider that a transpiler? I personally wouldn't because SML to C is a drastic change, especially from the abstraction point of view. C just makes a convenient backend at that point.
But since it also has native backends and C is its only other high level language, would you still consider it a compiler since you specifically mentioned "several other languages"?
I think it's kind of fun to look at this philosophically, since these are broad terms without perfect definitions.
Oh neat! I'm assuming then that C is the default for anything that doesn't have a specific backend for. I believe I heard that they had an x86 backend, but it was a pain to work with (maybe not as fast). Those are probably old concerns though, since their x86 64 backend must be more performant.
Speaking about MLton, I've often heard that it's "special" as it's a " whole-program optimizing compiler", but I was under the impression that it's already what compilers do. Is there a difference between MLton and let's say Clang, GCC or something like that?
GCC will compile one set of files at a time into .o files, and then those are linked together later on, so each "module" (using that term loosely) is compiled separately.
MLton takes in the entire program source, including the standard library, at once. Because of that, there's a ton of information we know, which helps a ton for things like dead code elimination. Because of this, there's a lot of memory required, since we have the entire program's syntax/parse/any other type of tree in memory, and it's not going to be as fast as something like GCC since we're recompiling everything every time in MLton instead of only recompiling the changed modules.
Despite MLton being a bit slower to compile, it's truly an incredible compiler. I had Dr Fluet, the lead maintainer, as my compiler professor in college, and he's a wealth of knowledge. The compiler we wrote in class was in SML and compiled with MLton. The amount of optimizations that occur is insane, it made damn fast executables, and that applies to most big compilers like GCC, Clang, Rustc, etc. The big difference in optimizations is how much information we have at optimization time. Languages like C are a bit looser and thus information may be harder to derive. Rust has a stricter set of rules, so we have more guarantees about a program to base our optimizations around.
Thank you for the explanation. While that explain what makes MLton special, I now wonder why other compilers aren't doing the same. There is probably lots of money to make by optimising some performance critical code, especially C/C++. But you seem to say that MLton is one of a kind. I don't understand that part.
The full program optimization is more applicable in a more formally specified language like SML I believe. Big projects take quite some time to compile, which is definitely a pain while developing.
So I'm out of my depth at this point, but this post from the SE Stack Exchange was interesting in that it highlights that those cross boundary optimizations can be handled at link time. I'm not sure how much is possible compared to MLton. MLton will have more access to type info, I'd assume, but maybe some metadata is stored in a .o file that I'm just not aware of.
I don't know much about this serious level of optimization, but I assume big C compilers get optimizations that are equally, if not more impressive and do just fine.
Yup, the backend IR is shared between JVM, JS (beta) and Native. The new frontend IR, I realize on second thought, will come later[1] with the K2 compiler frontend.
The old Kotlin/JS compiler is definitely closer than the new one to what might be called transpiler.
C is also a frontend to several other languages, which is why you can make portable code with it. Nim is the same. Instead of compiling to x86 for Windows and ARM for Linux, it compiles to C and JS. The idea is the same.
The definition this article puts up is, without references, is: " A transpiler on the other hand is a tool that converts between two languages while keeping the same level of abstraction. A theoretical perfect transpilation is therefore a two-way process where you could go back and forth between languages."
Is the level same-level-of-abstraction really part of the usual meaning of the term?
Here's a blog post by Lindsey Kuper that finds around 5 meanings, with only one of them (and being an unsourced Wikipedia article to boot) converging to the same idea about abstraction leve: http://composition.al/blog/2017/07/30/what-do-people-mean-wh...
I've never thought about it but I suppose it should be, otherwise you'd just have a compiler or decompiler. It's mentioned in the first paragraph of the Wikipedia article [1].
> Is the level same-level-of-abstraction really part of the usual meaning of the term?
No. That's just a definition by the author. The languages might operate on the same or close level of abstraction, but that doesn't mean that all logical constructs of the source and the transpiled target have to map. Especially the authors conclusion that a "transpilation is therefore a two-way process where you could go back and forth between languages" is not a necessary feature of a transpiler.
A transpiler is typically converting from one high-level language to another. Assuming that the level referred to by the phrase "high-level language" is level of abstraction, then yes.
It's all relative, sure, but we can also reasonably assert that Python, Ruby, Java are all operating on the same level of abstraction because we as programmers can largely ignore the details of the computer architecture of the machine they're running on, which isn't possible in assembly. An argument could be made that any compiler that can ignore such details can be called a transpiler.
The details can go very deep. It’s not that you can ignore the details, it’s that you can ignore some but not others, as in every language. Assembly also “largely” ignores the details. After all, assembly is an abstraction over other languages.
But now your “definition” of a transpiler depends on a subjective measure of “sameness”.
For instance, you called Python and Java the same level of abstraction, but in Java a lot of concern is devoted to properly typing numbers, whereas in Python this concern is ignored. If I really didn’t have to worry about the underlying hardware architecture, I would never care about the difference between a 16 bit and 32 bit number. Why aren’t 17 bit numbers a default type in Java? Why are there different sizes at all? Why can’t I just make everything infinity bits so I don’t have to worry about overflows? The answer to these questions have more to do with computer architecture than with Java.
And Java is even worse because now there are two machines: the hardware one and the virtual one, which is hard to always ignore. For some applications you can ignore the GC entirely, for others ignoring the GC will lead to catastrophic failure!
So is Java higher or lower or the same as Python? Depending on the axis measured, probably all three!
This is why people say it’s not so cut and dry. Any time someone puts down a sharp line, it’s easy to blur it. Which usually indicates there is no line at all.
Good points. Poke any abstraction enough and you will find holes in it. However, the question I was originally responding to was asking if same-level-of-abstraction-ness was fundamental to the definition of a transpiler. I would still argue that it is, in the common understanding of the terms. It may be the case that those terms are not particularly useful. Certainly from a technical standpoint, I would agree that it's not useful to try to force a distinction between transpilers and other compilers. If the goal is conveying the basic purpose of a project, maybe the term is of some use.
Compiler / transpiler. The end result is what matters. If it comes with the decent IDE that allows modern source code level debugging then what do I care. If not than I do not care how nice the language itself is. I've had my fun "debugging" machine codes on a long roll of paper and later using print/beep/flashing LED. No more please.
I had the impression we already had that discussion with Babel and TypeScript and the conclusion was, transpilers aren't really different from compilers, and we should simply calling them compilers.
I think, the background was the word transpiler was used by compiler creators as a way of gatekeeping.
> I think, the background was the word transpiler was used by compiler creators as a way of gatekeeping.
I dunno anyone who makes compilers who would draw this distinction (I know a lot of people who make compilers). I think it's more used by language warriors who don't understand there really isn't a distinction.
Best I can tell, it's largely the opposite: "transpiler" is mainly used to sound cool and build hype, traditional compiler folks tend to fall on your side and prefer to call them all "compilers".
(The usual term of disparagement in this context is "preprocessor"!)
Compilers and transpilers are the same. The term "optimizing compiler" exists, which probably mean that some are non-optimizing. TypeScript is a great example of that (although it does optimize a few things depending on the target).
Babel can be used as a macro system for JS, and macros are compilers.
Seems like they’re making up their own definition. I think most people consider a transpiler to be high level => high level
while a compiler would be high level => low level
Nim, JavaScript and C are all high level languages. Do they gain something from being labeled as a compiler?
I guess you get less brownie points for not doing all the steps yourself? Weird to me too. If it works great, it works great, and that's commendable.
Also, I'd classify C as a more intermediate language, rather than a high level one. It doesn't hold your hand as much as things like Python. So technically high -> intermediate would be going down a level, I suppose.
It's really all just playing semantics. I wouldn't mind if they labeled themselves a compiler or a transpiler. Doesn't really matter to me.
I have just as much respect for those who have written a complete transpiler as for those who have built a complete compiler, because, unless the differences between the source and target languages are trivial, much of the pipeline is identical.
If your target is something like C/JS, your translation stage must be fully aware of the idiosyncrasies of the target language. If you are targeting machine code, you will have to include optimization and register allocation stages before generating machine code and the final executable/library.
I'd argue that a compiler is just a subset of transpilers that target direct hardware instructions (CPU/GPU/some niche specific hardware). Every other argument between the two come across, to me, as more of an emotional arbitrariness of what is a "real" compiled language.
Of course, this makes Java, and others, to not actually be compiled. That makes sense to me.
My personal opinion is that it will be better to provide seamless support for C rather than compile to C as has been done by C++ and D. D language even goes further by performing CTFE for the latest ImportC feature.
Check this Pascal to D transcompilation talk in DConf 2019 [1]. It will be great if similarly R language can be transcompiled to D since the effort has been futile so far to write a better and modern compiler for R [2].
74 comments
[ 3.1 ms ] story [ 137 ms ] threadAlso „machine code“ is not even the last step anymore, it gets further cached, interpreted compiled to instructions by the hardware. It’s abstraction layers all the way down.
For a transpiler, you generally need a tokenizer, then a parser to parse the token stream into an AST, which you will then have some set of transformation steps, often transforming into other ASTs for optimization steps or incremental transforms to the final target, before you take the final AST and serialize it out into the target.
Wherease for a compiler, you generally need a tokenizer, then a parser to parse the token stream into an AST, which you will then have some set of transformation steps, often transforming into other ASTs for optimization steps or incremental transforms to the final target, before you take the final AST and serialize it out into the target.
If I was hiring for a compiler position and someone showed up with 10 years of transpiler authorship on their resume, I wouldn't even look at them. Such divergent experience would be useless. It would be like hiring someone for a Javascript backend position when all they had was 10 years of intensive Javascript frontend experience. Madness. You'll go out of business making stupid hiring decisions like that.
I think you'll spend more time and effort hiring than you would've spent training.
What really comes through is that the author seems to think that transpiler status is somehow lesser, and the whole nitpicking about compilers/transpilers is just an attempt to make sure Nim lands squarely on the compiler side.
Maybe the conclusion that should be reached is the one the author is hinting towards at the end - which is that transpiler and compiler don't really have a strong distinction, and their usage is muddied enough that saying whether something is "actually" a transpiler or a compiler is mostly pointless. (And then there would be less worry that people "mistakenly" call Nim a transpiler, eh?)
The conclusion is definitely that these are loose terms that can apply to many different things. But some of the usages are more useful as differentiating terms than others. The very broad definitions of transpilers ends up making most if not all compilers into transpilers which isn't terribly useful, and the in-between ones I've found are so under-defined that the term ends up not meaning anything at all. This was merely my attempt at reaching a definition of transpiler that is not only a useful distinction from a compiler, but also serves as a more concrete term.
You choose whether you want to emit C, C++, or JS. There's an experimental, fledgling community LLVM frontend for Nim, that is what I would call a "compiler":
https://github.com/arnetheduck/nlvm
What Jetbrains calls Kotlin -> JavaScript, is "transpilation"
What Google calls Java -> JavaScript in j2cl, is "transpilation"
But this is all pedantic IMO.
Very nice. Last time I experimented with Nim the lack of this was the single biggest dealbreaker.
With this LLVM backend, that doesn't work, and it doesn't support JS (which probably isn't a big deal to most people).
That means you can't do stuff like this:
https://nim-lang.org/docs/manual.html#implementation-specifi...https://nim-lang.org/docs/manual.html#foreign-function-inter...
https://nim-lang.org/docs/manual.html#implementation-specifi...
It could definitely be better, and hopefully some day it will be, but I use Nim in a debugger quite frequently and it usually works out pretty well.
To me, if the thing that gets emitted is another regular programming language, you've done transpilation.
If the thing that gets emitted is IR, bytecode, object code, essentially anything that's not something you would read/write yourself and is a step further towards being executable, it's "compilation".
I see transpilation as lateral/horizontal movement on the "path to being able to run it" chain, and compilation as vertical movement towards the executable end-goal.
But, again -- "pedantic", and just my $0.02.
But since it also has native backends and C is its only other high level language, would you still consider it a compiler since you specifically mentioned "several other languages"?
I think it's kind of fun to look at this philosophically, since these are broad terms without perfect definitions.
MLton takes in the entire program source, including the standard library, at once. Because of that, there's a ton of information we know, which helps a ton for things like dead code elimination. Because of this, there's a lot of memory required, since we have the entire program's syntax/parse/any other type of tree in memory, and it's not going to be as fast as something like GCC since we're recompiling everything every time in MLton instead of only recompiling the changed modules.
Despite MLton being a bit slower to compile, it's truly an incredible compiler. I had Dr Fluet, the lead maintainer, as my compiler professor in college, and he's a wealth of knowledge. The compiler we wrote in class was in SML and compiled with MLton. The amount of optimizations that occur is insane, it made damn fast executables, and that applies to most big compilers like GCC, Clang, Rustc, etc. The big difference in optimizations is how much information we have at optimization time. Languages like C are a bit looser and thus information may be harder to derive. Rust has a stricter set of rules, so we have more guarantees about a program to base our optimizations around.
So I'm out of my depth at this point, but this post from the SE Stack Exchange was interesting in that it highlights that those cross boundary optimizations can be handled at link time. I'm not sure how much is possible compared to MLton. MLton will have more access to type info, I'd assume, but maybe some metadata is stored in a .o file that I'm just not aware of.
I don't know much about this serious level of optimization, but I assume big C compilers get optimizations that are equally, if not more impressive and do just fine.
https://softwareengineering.stackexchange.com/a/370242
What about the new beta Kotlin/JS compiler that does Kotlin -> frontend IR -> backend IR -> JavaScript?
I guess... "Compilation of Kotlin, then emittance of JS"?
Because I believe this uses the shared "FIR" IR that the JVM/Native backends use right?
This is why the whole argument/topic is so subjective/pedantic haha.
The old Kotlin/JS compiler is definitely closer than the new one to what might be called transpiler.
[1] https://blog.jetbrains.com/kotlin/2021/10/the-road-to-the-k2...
Is the level same-level-of-abstraction really part of the usual meaning of the term?
Here's a blog post by Lindsey Kuper that finds around 5 meanings, with only one of them (and being an unsourced Wikipedia article to boot) converging to the same idea about abstraction leve: http://composition.al/blog/2017/07/30/what-do-people-mean-wh...
1. https://en.wikipedia.org/wiki/Source-to-source_compiler
No. That's just a definition by the author. The languages might operate on the same or close level of abstraction, but that doesn't mean that all logical constructs of the source and the transpiled target have to map. Especially the authors conclusion that a "transpilation is therefore a two-way process where you could go back and forth between languages" is not a necessary feature of a transpiler.
But now your “definition” of a transpiler depends on a subjective measure of “sameness”.
For instance, you called Python and Java the same level of abstraction, but in Java a lot of concern is devoted to properly typing numbers, whereas in Python this concern is ignored. If I really didn’t have to worry about the underlying hardware architecture, I would never care about the difference between a 16 bit and 32 bit number. Why aren’t 17 bit numbers a default type in Java? Why are there different sizes at all? Why can’t I just make everything infinity bits so I don’t have to worry about overflows? The answer to these questions have more to do with computer architecture than with Java.
And Java is even worse because now there are two machines: the hardware one and the virtual one, which is hard to always ignore. For some applications you can ignore the GC entirely, for others ignoring the GC will lead to catastrophic failure!
So is Java higher or lower or the same as Python? Depending on the axis measured, probably all three!
This is why people say it’s not so cut and dry. Any time someone puts down a sharp line, it’s easy to blur it. Which usually indicates there is no line at all.
https://github.com/arnetheduck/nlvm
I think, the background was the word transpiler was used by compiler creators as a way of gatekeeping.
Why do you think that?
I dunno anyone who makes compilers who would draw this distinction (I know a lot of people who make compilers). I think it's more used by language warriors who don't understand there really isn't a distinction.
(The usual term of disparagement in this context is "preprocessor"!)
Babel can be used as a macro system for JS, and macros are compilers.
Nim, JavaScript and C are all high level languages. Do they gain something from being labeled as a compiler?
Also, I'd classify C as a more intermediate language, rather than a high level one. It doesn't hold your hand as much as things like Python. So technically high -> intermediate would be going down a level, I suppose.
It's really all just playing semantics. I wouldn't mind if they labeled themselves a compiler or a transpiler. Doesn't really matter to me.
If your target is something like C/JS, your translation stage must be fully aware of the idiosyncrasies of the target language. If you are targeting machine code, you will have to include optimization and register allocation stages before generating machine code and the final executable/library.
Of course, this makes Java, and others, to not actually be compiled. That makes sense to me.
Check this Pascal to D transcompilation talk in DConf 2019 [1]. It will be great if similarly R language can be transcompiled to D since the effort has been futile so far to write a better and modern compiler for R [2].
https://youtu.be/HvunD0ZJqiA
[1] DConf 2019 - Transcompilation into D:
http://dconf.org/2019/talks/veelo.html
https://youtu.be/HvunD0ZJqiA
[2] Why R? 2020 Keynote - Jan Vitek - How I Learned to Love Failing at Compiling R:
https://www.youtube.com/watch?v=VdD0nHbcyk4