Zig's incremental builds are DEFINITELY a killer feature. In the short term, I could see why you'd make a switch to get it. But, in the medium term, can we really not expect to see this in Rust in the somewhat near future?
I want to go fast, but I don't want to go fast just to shoot my foot off.
If only somehow we could get Rust's safety with all of Zig's features and Go's runtime without GC...
Are compile times that big of a deal? I haven't used Rust a ton, but the few times I have it seemed like the bulk of the compile time was a one off compiling the crates, and then compiling your own code was super fast.
I feel like I'd massively prefer to end up with a binary free of memory exploits than shaving some time off compile.
> remember that for compilers which emit machine code, like roc and rustc, doing memory-unsafe things is a big part of the job
I don't really think that this is true, in the way that it's written.
I think that for the hot binary patching / code reloading features, yes, that is going to need unsafe. But for regular old "producing an executable" compilation? Emitting machine code isn't the part that requires unsafe. The language's runtime is a more likely site to find unsafe.
>ReleaseSafe catches use-after-free errors through runtime checks which panic if the program tries to use freed memory.
I don't know Zig so maybe they know something I don't, but I have seen no evidence that it catches any type of use-after-free including double-free?
While writing a blog post (below) I went through the documentation to figure out the possible runtime memory safety checks Zig can insert. The term "use-after-free" or "UaF" never occurs on that documentation page. Searching for "safety-checked" doesn't yield any related hits either.
Unless maybe they're using the DebugAllocator in release builds? Even that does not reliably surface UaF.
Interesting that OCaml was flexible and expressive enough to be used as a prototype testbed but not chosen as the implementation language, especially given the maturity of both. I would be surprised if Zigs incremental builds could be meaningfully faster than dune's.
Cross compilation is great, but not mentioned in the "why Zig" section. Is memory control that crucial for a compiler?
Rust itself was originally written in OCaml, same with WASM. I'm curious about what milestone gets reached where the maintainers collectively decide to transition away.
I've sort of thought this too. Rust adoption feels a lot like Haskell to me in that it's centered around ideological things that don't really improve the final product and arguably slow development. If your goal is to write a bunch of LLM code there are better languages than rust, and rust development is often regarded as too annoy for anyone not enthusiastic about the ideological improvements.
There are cool ideas in the language but I don't see it enduring over time.
Zig is a pre-1.0 language while Rust is post-1.0. This alone is settles which one to pick for may developers. The library support is probably favours Rust too. Rust build times are much slower than Zig, I get that, but I rarely optimize software for build times.
That’s totally reasonable. But while each minor version has breaking change, a minor version release itself is generally fairly stable and useable. There are solid production software which is already using Zig, like TigerBeetle and Ghostty/libghostty.
I am waiting for (near) 1.0 myself before doing anything major with it. But that’s mainly because I want at least the async IO stuff to be settled for my use cases.
Counter Argument - with Zig it's perfectly viable to make programs using just the standard library. You have json serde, zon (zig object notation) serde, system functions (ie libc/rustix equivalent), random number generators... all without a single dependency.
One thing I wish Rust would improve over time is the builds. Its one of the biggest sources of wasted storage space on all my computers, builds a ton of libraries can take tens of gigs, it adds up very quickly. Not sure what the best solution is, one I found is to set the global build folder so dependencies get reused across projects, but imho it should be an OOTB default behavior whatever the real solution should be.
I don't even know what Zig is but I've seen this topic come up so many times on this site that I'm starting to think the people who are actually doing this are unsure themselves whether it's a good idea or not.
While I'm a rust enthusiast, I do agree that certain languages lend themselves well to particular domains. So a rewrite from Rust to something better suited is fine by me. In fact, while I do work on a rust project, I would not have and still would not recommend it as the choice for that particular project.
That being said, I had to do some double takes while reading this.
I feel that it's a bit weird to compare a rather well tested 7 (?) year old rust implementation with a brand new not yet released less than a year old Zig implementation. Without that context, this looks like a bad comparison for rust, when it is in fact the complete opposite.
The swiftness of the Zig compilere here is insane, and would would very much shift my recommendation of Rust if it got to similar speeds.
That being said, I do find it funny that currently, the compilation speed is actually worse on Zig than Rust, despite Zig (anonymous commenters at least tbf) claiming the opposite for years.
How did you eventually discover the 35 ms figure for Roc? Did you have to temporarily update the codebase to 0.17?
Nothing negative here. I did play around with implementing a scripting language in this DOD-ish, index-based paradigm and yeah, it is neat.
I was thinking that it might be possible to do resumable computation across the network like this (in the context of frontend frameworks "resuming" UIs), but ultimately I have no use for this so just the experience itself was enough.
One note here is that it does tend to break completely if non-pointer-free data is introduced. It seems like it's either all or nothing.
I am not sure, but there might be a bug in their pattern matching example.
What happens if 'verb' is "GET" and 'path' is "/users/1234/posts/1234/extra_path/and/more/"? Will 'post_id' become "extra_path/and/more/"?
I tried running it in the sandbox, and it does indeed seem to buggily result in:
"Post ID: 1234/extra_path/and/more"
I suspect that the reason it is behaving like it is, is due to how it handles characters in the string literal. The example program exploits that only the slashes present in the string literal pattern are matched, to enable matching on 'page' having slashes. But then in the nested 'match', it forgot to account for any possible extra slashes.
Nitpicking end.
I have not read the whole post yet, but the pattern matching not requiring any allocations, seems very nice. The string literal patterns also seem interesting, though I am not completely sold on them, also as per the above possible bug. It seems really clean in some ways, but the specific semantics, I am not fully sure about. Maybe it is excellent, and is so clean and concise that it is overall less bug-prone than alternatives in other programming languages. I do not know.
Irrespective to the technical merits of both language, moving from a stable language to a pre-1.0 one that just lost his most popular open source project is a wild move.
This piece would have been a lot more compelling if they had actually done science on selecting a language for compiler development. From what I can tell, they had an untested hypothesis that a low level systems language is necessary for a high performance compiler https://www.roc-lang.org/faq#self-hosted-compiler and from that concluded that their only
choice besides rust was zig.
I know from experience that this initial assumption is wrong. Compiler performance is dominated by algorithms. The fastes managed languages tend to be at worst within a factor of two for wall time on any given algorithm. Algorithmic differences can be unbounded in their performance gaps. Zig itself is a perfect counterexample to the theory that writing a compiler in a low level systems language will lead to a fast compiler. Roc seems to compile at around 15k lines per second. That is not fast. There were evidently compilers written in ml that did 3k likes per second in 1998 https://flint.cs.yale.edu/cs421/case-for-ml.html
The zig rewrite of roc looks like the author's second compiler. Compiler and language design is a skill like any other and from my vantage point, they appear to have overcommitted to an initial design at the expense of developing their higher level design skills. In my opinion, the best thing they could do for the future of roc is stop working on their current compiler and use it to write a self hosting compiler for a much smaller subset of roc. They should be able to do that in less than 10k lines of code. They might even find that their self hosting compiler is faster than their zig based bootstrap compiler for the self hosted subset of roc. If the self hosting compiler is inadequate. Now they at least have identified a smaller useful subset of roc and can experiment with different compiler implementations in 10k likes of code rather than 300k lines of code. Then they could actually test the theory of whether or not a low level language is necessary to meet whatever arbitrary compiler performance goals they have.
By self hosting, they would also discover what roc features actually matter and they would spend much more time actually writing roc code. The features that are needed to write a self hosted compiler are all features that are generally useful. By improving the self hosted compiler, they also improve downstream programs.
But you can always use the best algorithm no matter what your implementation language is, so it still makes sense to prefer a language that makes it easy to write fast code.
Zig itself is an incredibly fast compiler. And the language and standard library is designed to support writing that compilers. And yeah, that’s all about algorithm and data structures. A large part of why struct-of-arrays is easy to do in Zig is because Andrew wanted it for making the compiler fast. The article also points out that it’s reusing code from the Zig compiler source as well.
Roc may not be super fast now, but that doesn’t mean they haven’t seen ahead and set themselves up for success.
Yeah, I agree there is value in self-hosting as you say. Zig itself is an example of that. But zig is a systems language. If Roc isn’t aiming for that nice it might not be the best choice for writing a compiler in. As an example in the other extreme end of language flavours: Python is still mainly CPython and efforts to make a python interpreter in some variant of python hasn’t been particularly successful.
Does this mean every time you find yourself using lots of “unsafe” Rust blocks, it’s not the right tool for the job? I suspect it’s not that simple, but what are people’s experience?
Tangentially relates, but if any Roc devs are around I'm curious about the use cases for Roc.
It's supposed to be a scripting language right you embed into your C ABI right?
Do you see it competing with WASM for the plugin use case (i.e. a really large Roc platform)? Why would an app author prefer to expose a Roc layer to their app rather than a WASM layer? With a WASM layer, plugin devs can write in any language.
Another use case I've heard from it is as a more app-level language (i.e. a really small Roc platform). Do you see it competing with Gleam for server side http code? Do you see it competing with Elm for client side code?
> If any Roc devs are around I'm curious about the use cases for Roc.
It's a general functional programming language that's interested in the constraints and state control properties but not really in the dogma/traditions. As a specific example, it has a for loop statement that doesn't return anything just because sometimes the algorithm is easier to express imperatively. That said, it really is functional, mutating functions/methods require a `!` suffix and `->` (pure) vs `=>` (not) is distinguished in the type system and enforced. The language is fully decidable so type annotations are optional with the arguable exception of the built-in Serde which needs a concrete type to encode/decode. It's also pretty fast, like in the Go range.
I think it has the best error handling of any language in the ~3 dozen I've tried. It's Rust style in general with `Result` renamed to `Try` but the error side of `Try` is an open tag set and can just aggregate so you get the nice parts of the Rust error experience without the downsides. As an example, a coeffect (effectful input) from an example on my server platform:
book! = |req| {
body : { id : I64 }
body = Req.json_body!(req)?
rows = Sql.query!(req.ctx, db_path, "SELECT id, title, author, year FROM books WHERE id = ?", [Integer(body.id)])?
row = Sql.first(rows) ? |_| NotFound("book ${body.id.to_str()} not found")
book = decode_book(row)?
Ok(book)
}
The full set of errors covers malformed utf8, missing/wrong type for id, db errors, the custom NotFound with message, and missing/changed db columns and these plus all the other errors across the app get rolled up and handled in one spot by the error mapping function which rolls the input errors to 400, a 404 for the NotFound, 500s in general in a big match. I have more compact ways to express this in the platform (sqlx) but those don't show off the error handling as nicely.
All in all, it's pretty much just a nice hosted language for doing things.
> Do you see it competing with WASM for the plugin use case?
It's mostly competing with Lua and friends but the host is a platform and not an embedder so the Roc goes on the outside and produces the binary. Roc is particularly well suited for compiling to wasm because all the effects coming from the host is shared. This is actually one of my primary interests in Roc but I haven't really harassed the Roc team about it because they've been busy with the rewrite and wasm module specs have been WIP.
> Why would an app author prefer to expose a Roc layer to their app rather than a WASM layer?
No need for the relatively large WASM runtime would be one of the first ones but Roc isn't really designed to be embedded. I expect to mainly use Roc for app level code on top of Rust for systems level code. I could write app-level Rust but I like functional programming, GC (refcount) is convenient, the error handling is nice, no annotations are nice, super fast compiles are nice, etc.
> Do you see it competing with Gleam for server side http code? Do you see it competing with Elm for client side code?
Sure. As mentioned, I'm experimenting with a server platform that uses pure handlers plus an effect system. I have a RealWorld implementation and in casual benchmarking on my M1 laptop I get 69k req/s for the article endpoint (serialization bound) and 10k going through the article_list endpoint (sqlite bound, 4 table join). The framework also has full and automatic cache invalidation so if I turn on caching I hit 120-140k req/s on both endpoints with no other code changes.
As for GUI stuff, I'm working on a platform (Clay+Solid2) but I don't see any particular reason it wouldn't work.
>Another use case I've heard from it is as a more app-level language (i.e. a really small Roc platform). Do you see it competing with Gleam for server side http code? Do you see it competing with Elm for client side code?
For sure! I've been using Roc for work exclusively for the past year and a half, writing mostly full-stack web apps, and it's been great. I wrote a small framework/platform (https://github.com/niclas-ahden/joy) which is similar to Elm or Lustre from Gleam.
Previously I used mainly Rust/Haskell/Elm/Ruby/Python and each language has its pros/cons, but for web apps Roc really hits the sweet spot of: runtime performance, compilation speed, strong type system, error handling, and ergonomics. It's definitely worth a try!
52 comments
[ 10.2 ms ] story [ 66.1 ms ] threadI want to go fast, but I don't want to go fast just to shoot my foot off.
If only somehow we could get Rust's safety with all of Zig's features and Go's runtime without GC...
That's what I'm working on building [=
I feel like I'd massively prefer to end up with a binary free of memory exploits than shaving some time off compile.
> remember that for compilers which emit machine code, like roc and rustc, doing memory-unsafe things is a big part of the job
I don't really think that this is true, in the way that it's written.
I think that for the hot binary patching / code reloading features, yes, that is going to need unsafe. But for regular old "producing an executable" compilation? Emitting machine code isn't the part that requires unsafe. The language's runtime is a more likely site to find unsafe.
Well, it's true when it's written badly (eg. the same way as when somebody writes in-line "shellcode" by doing `*(T*)ptr = (T)val;` or similar.
I think there's some history of some compilers that were basically non-trivial to port to other architectures/bit-sizes b.c. of this piggybacking.
I don't know Zig so maybe they know something I don't, but I have seen no evidence that it catches any type of use-after-free including double-free?
While writing a blog post (below) I went through the documentation to figure out the possible runtime memory safety checks Zig can insert. The term "use-after-free" or "UaF" never occurs on that documentation page. Searching for "safety-checked" doesn't yield any related hits either.
Unless maybe they're using the DebugAllocator in release builds? Even that does not reliably surface UaF.
https://landaire.net/memory-safety-by-default-is-non-negotia...
Cross compilation is great, but not mentioned in the "why Zig" section. Is memory control that crucial for a compiler?
Rust itself was originally written in OCaml, same with WASM. I'm curious about what milestone gets reached where the maintainers collectively decide to transition away.
There are cool ideas in the language but I don't see it enduring over time.
I am waiting for (near) 1.0 myself before doing anything major with it. But that’s mainly because I want at least the async IO stuff to be settled for my use cases.
If I want to use allocator debuggers I already have the production ready tools that exist for C and C++ for at least 30 years.
That being said, I had to do some double takes while reading this.
> https://rtfeldman.com/rust-to-zig#memory-safety-post-rewrite
I feel that it's a bit weird to compare a rather well tested 7 (?) year old rust implementation with a brand new not yet released less than a year old Zig implementation. Without that context, this looks like a bad comparison for rust, when it is in fact the complete opposite.
> https://rtfeldman.com/rust-to-zig#build-times
The swiftness of the Zig compilere here is insane, and would would very much shift my recommendation of Rust if it got to similar speeds.
That being said, I do find it funny that currently, the compilation speed is actually worse on Zig than Rust, despite Zig (anonymous commenters at least tbf) claiming the opposite for years.
How did you eventually discover the 35 ms figure for Roc? Did you have to temporarily update the codebase to 0.17?
> https://rtfeldman.com/rust-to-zig#memory-control-zero-parse-...
Nothing negative here. I did play around with implementing a scripting language in this DOD-ish, index-based paradigm and yeah, it is neat.
I was thinking that it might be possible to do resumable computation across the network like this (in the context of frontend frameworks "resuming" UIs), but ultimately I have no use for this so just the experience itself was enough.
One note here is that it does tend to break completely if non-pointer-free data is introduced. It seems like it's either all or nothing.
> https://rtfeldman.com/rust-to-zig#ecosystem-relevance
This is more of an LLVM thing, which is fair, but I find it funny that "LLVM unstable bad" while "Zig unstable whatever".
Overall though, this was an interesting read. And if the folks contributing to roc like zig then more power to them.
Last thing, the link here is broken (points to a TODO):
> Zig's compiler itself is another
I am not sure, but there might be a bug in their pattern matching example.
What happens if 'verb' is "GET" and 'path' is "/users/1234/posts/1234/extra_path/and/more/"? Will 'post_id' become "extra_path/and/more/"?
I tried running it in the sandbox, and it does indeed seem to buggily result in:
"Post ID: 1234/extra_path/and/more"
I suspect that the reason it is behaving like it is, is due to how it handles characters in the string literal. The example program exploits that only the slashes present in the string literal pattern are matched, to enable matching on 'page' having slashes. But then in the nested 'match', it forgot to account for any possible extra slashes.
Nitpicking end.
I have not read the whole post yet, but the pattern matching not requiring any allocations, seems very nice. The string literal patterns also seem interesting, though I am not completely sold on them, also as per the above possible bug. It seems really clean in some ways, but the specific semantics, I am not fully sure about. Maybe it is excellent, and is so clean and concise that it is overall less bug-prone than alternatives in other programming languages. I do not know.
I know from experience that this initial assumption is wrong. Compiler performance is dominated by algorithms. The fastes managed languages tend to be at worst within a factor of two for wall time on any given algorithm. Algorithmic differences can be unbounded in their performance gaps. Zig itself is a perfect counterexample to the theory that writing a compiler in a low level systems language will lead to a fast compiler. Roc seems to compile at around 15k lines per second. That is not fast. There were evidently compilers written in ml that did 3k likes per second in 1998 https://flint.cs.yale.edu/cs421/case-for-ml.html
The zig rewrite of roc looks like the author's second compiler. Compiler and language design is a skill like any other and from my vantage point, they appear to have overcommitted to an initial design at the expense of developing their higher level design skills. In my opinion, the best thing they could do for the future of roc is stop working on their current compiler and use it to write a self hosting compiler for a much smaller subset of roc. They should be able to do that in less than 10k lines of code. They might even find that their self hosting compiler is faster than their zig based bootstrap compiler for the self hosted subset of roc. If the self hosting compiler is inadequate. Now they at least have identified a smaller useful subset of roc and can experiment with different compiler implementations in 10k likes of code rather than 300k lines of code. Then they could actually test the theory of whether or not a low level language is necessary to meet whatever arbitrary compiler performance goals they have.
By self hosting, they would also discover what roc features actually matter and they would spend much more time actually writing roc code. The features that are needed to write a self hosted compiler are all features that are generally useful. By improving the self hosted compiler, they also improve downstream programs.
But you can always use the best algorithm no matter what your implementation language is, so it still makes sense to prefer a language that makes it easy to write fast code.
Zig itself is an incredibly fast compiler. And the language and standard library is designed to support writing that compilers. And yeah, that’s all about algorithm and data structures. A large part of why struct-of-arrays is easy to do in Zig is because Andrew wanted it for making the compiler fast. The article also points out that it’s reusing code from the Zig compiler source as well.
Roc may not be super fast now, but that doesn’t mean they haven’t seen ahead and set themselves up for success.
Yeah, I agree there is value in self-hosting as you say. Zig itself is an example of that. But zig is a systems language. If Roc isn’t aiming for that nice it might not be the best choice for writing a compiler in. As an example in the other extreme end of language flavours: Python is still mainly CPython and efforts to make a python interpreter in some variant of python hasn’t been particularly successful.
I think precious cognitive time should be spent more on the language itself rather than wasting it on rewrites.
It's supposed to be a scripting language right you embed into your C ABI right?
Do you see it competing with WASM for the plugin use case (i.e. a really large Roc platform)? Why would an app author prefer to expose a Roc layer to their app rather than a WASM layer? With a WASM layer, plugin devs can write in any language.
Another use case I've heard from it is as a more app-level language (i.e. a really small Roc platform). Do you see it competing with Gleam for server side http code? Do you see it competing with Elm for client side code?
It's a general functional programming language that's interested in the constraints and state control properties but not really in the dogma/traditions. As a specific example, it has a for loop statement that doesn't return anything just because sometimes the algorithm is easier to express imperatively. That said, it really is functional, mutating functions/methods require a `!` suffix and `->` (pure) vs `=>` (not) is distinguished in the type system and enforced. The language is fully decidable so type annotations are optional with the arguable exception of the built-in Serde which needs a concrete type to encode/decode. It's also pretty fast, like in the Go range.
I think it has the best error handling of any language in the ~3 dozen I've tried. It's Rust style in general with `Result` renamed to `Try` but the error side of `Try` is an open tag set and can just aggregate so you get the nice parts of the Rust error experience without the downsides. As an example, a coeffect (effectful input) from an example on my server platform:
The full set of errors covers malformed utf8, missing/wrong type for id, db errors, the custom NotFound with message, and missing/changed db columns and these plus all the other errors across the app get rolled up and handled in one spot by the error mapping function which rolls the input errors to 400, a 404 for the NotFound, 500s in general in a big match. I have more compact ways to express this in the platform (sqlx) but those don't show off the error handling as nicely.All in all, it's pretty much just a nice hosted language for doing things.
> Do you see it competing with WASM for the plugin use case?
It's mostly competing with Lua and friends but the host is a platform and not an embedder so the Roc goes on the outside and produces the binary. Roc is particularly well suited for compiling to wasm because all the effects coming from the host is shared. This is actually one of my primary interests in Roc but I haven't really harassed the Roc team about it because they've been busy with the rewrite and wasm module specs have been WIP.
> Why would an app author prefer to expose a Roc layer to their app rather than a WASM layer?
No need for the relatively large WASM runtime would be one of the first ones but Roc isn't really designed to be embedded. I expect to mainly use Roc for app level code on top of Rust for systems level code. I could write app-level Rust but I like functional programming, GC (refcount) is convenient, the error handling is nice, no annotations are nice, super fast compiles are nice, etc.
> Do you see it competing with Gleam for server side http code? Do you see it competing with Elm for client side code?
Sure. As mentioned, I'm experimenting with a server platform that uses pure handlers plus an effect system. I have a RealWorld implementation and in casual benchmarking on my M1 laptop I get 69k req/s for the article endpoint (serialization bound) and 10k going through the article_list endpoint (sqlite bound, 4 table join). The framework also has full and automatic cache invalidation so if I turn on caching I hit 120-140k req/s on both endpoints with no other code changes.
As for GUI stuff, I'm working on a platform (Clay+Solid2) but I don't see any particular reason it wouldn't work.
For sure! I've been using Roc for work exclusively for the past year and a half, writing mostly full-stack web apps, and it's been great. I wrote a small framework/platform (https://github.com/niclas-ahden/joy) which is similar to Elm or Lustre from Gleam.
Previously I used mainly Rust/Haskell/Elm/Ruby/Python and each language has its pros/cons, but for web apps Roc really hits the sweet spot of: runtime performance, compilation speed, strong type system, error handling, and ergonomics. It's definitely worth a try!