slightly off topic. Lately, I m seeing so many articles on sourcegraph using golang to meet their needs. I wonder, why did they decide to chose Rust [1] for their syntax highlighting server ?
He chose to use Rust for many of the same reasons that other people mention in this thread: https://news.ycombinator.com/item?id=15324787. I'll ping him in case there are any more details.
Sourcegraph dev here :) At Sourcegraph we've actually tried a lot of different syntax highlighters in the past. Syntax highlighting is actually a _really_ hard problem.
Syntax highlighting for basically all editors today is provided by TextMate language grammers[1] (`.tmLanguage` files) ultimately, which are just giant regex definition files, basically. These are the ubiquitous files for syntax highlighting, and you can almost always find a TextMate grammer for any programming language.
Some 'modern' syntax highlighters, mostly ones designed for use in the web, try not to use these TextMate files -- but in doing so they lose support for TONS of languages, and the languages that they do support are often very buggy / incorrect because the grammers are just simply not that well tested (nobody is testing them in their editor daily).
The annoyance with TextMate grammers is that they are based on Oniguruma Regular Expressions, which means that while they are written in very powerful regex syntax, you cannot really use them with any regex engine built in to most languages today. Almost all editors directly use the original C implementaton of Oniguruma, even editors like Visual Studio Code which are based on web tech primarily.
And even once you have a working Oniguruma regex engine, it is still very non-trivial to use it for highlighting actual code because the TextMate files themselves are pretty complex on their own.
To make matters more interesting, Sublime created their own language grammer format (`.sublime-syntax` files), which is a superset of the TextMate language grammers and supports more features. Sublime can convert any TextMate language grammer into the new .sublime-syntax format.
So when I started seriously looking into syntax highlighting about a year ago, the Pygments syntax highlighter was my first choice and it is pretty good, but I found it to be lacking support for languages we cared about such as TypeScript at the time. Additionally, Pygments was pretty slow when we tried it out. GitHub switched away from it for a similar reason[2]
This was about the time that I ran into the Syntect[3] project. For us, this had huge benefits:
- It was by far the fastest syntax highlighter we found, which meant page load times improved a lot for us.
- Its results were really, really good. I found multiple cases where the output was better than other highlighters such as Pygments.
- The built-in language support, all language Sublime Text 3 supports by default, was great out of the box.
- Because Sublime can convert TextMate grammers to .sublime-text grammers, we ultimately get support for basically all languages.
- It took me _one day_ to write `syntect_server`[4] which highlights arbitrary code for our Go application, which was a spectacular turnaround time given how much time I'd already invested elsewhere in researching all of this.
Overall, we really couldn't be more pleased with Syntect and Rust for modern syntax highlighting. It's a beautiful, fast, high-quality combination.
I don't have hard numbers on me at the moment, but I remember it going from roughly ~2-4s with Highlight.js on some poor file cases down to ~500ms with Syntect. From what I remember, Pygments was better than Highlight.js by a factor of roughly ~2x, but with Syntect still outperforming it significantly in the general cases.
I think Syntect is faster in general just because of the nature of the work (syntax highlighting is basically just a stack machine + running regex 24/7) and Rust being a really performant choice for this in comparison to Python/JS (although I am sure some less idiomatic code could improve the perf of a Python/JS implementation).
Hopefully one day I'll do a proper write-up / blog post on this topic and provide some more hard numbers / stats for everyone. :)
Think it’s just inertia. I remember meeting one of the founders at Gophercon a few years ago, and Sourcegraph was either Go only or it was one of the few languages supported / used. Live-blogging gophercons was a great way to get mindshare and do content marketing, and I think they’ve been doing it ever since.
Sourcegraph CEO here. Yeah, we use mostly Go on the backend, but we interact with most major languages since our product supports many languages for code search/browsing.
I want to credit the many community members who contributed the majority of the liveblog posts this year at GopherCon. This post was contributed by @shurcooL. It is so cool to see the GopherCon liveblogging tradition grow each year!
On a similar note: I have been playing a bit with reagents in ocaml, which could in some way be seen as a generalisation of concurrent ML which is a generalisation of the go concurrency model.
Really neat stuff, even though you rarely need more control than what CML gives you.
I wish they would apply classical concurrency patterns instead of acting every 5-10 years like there would be none and we would need to start to develop them from zero. If you have some experience with distributed systems and take a look at what kubernetes and friends are doing it is really shocking once more. Funny that this presentation is from the go community which is exactly one source of the current starting from zero attempt.
It really does seem like there are a lot of presentations that pretend to do something new when they and most of their audience likely just have a very poor understanding of what has already been done in the past.
If you enjoyed this then there is at least a passable chance you'll be interested in the scala library cats-effect, which just 1.0 and has a nice toolkit of concurrency primitives, which are well documented.
15 comments
[ 5.8 ms ] story [ 46.6 ms ] thread[1] https://www.rust-lang.org/en-US/friends.html
He chose to use Rust for many of the same reasons that other people mention in this thread: https://news.ycombinator.com/item?id=15324787. I'll ping him in case there are any more details.
Syntax highlighting for basically all editors today is provided by TextMate language grammers[1] (`.tmLanguage` files) ultimately, which are just giant regex definition files, basically. These are the ubiquitous files for syntax highlighting, and you can almost always find a TextMate grammer for any programming language.
Some 'modern' syntax highlighters, mostly ones designed for use in the web, try not to use these TextMate files -- but in doing so they lose support for TONS of languages, and the languages that they do support are often very buggy / incorrect because the grammers are just simply not that well tested (nobody is testing them in their editor daily).
The annoyance with TextMate grammers is that they are based on Oniguruma Regular Expressions, which means that while they are written in very powerful regex syntax, you cannot really use them with any regex engine built in to most languages today. Almost all editors directly use the original C implementaton of Oniguruma, even editors like Visual Studio Code which are based on web tech primarily.
And even once you have a working Oniguruma regex engine, it is still very non-trivial to use it for highlighting actual code because the TextMate files themselves are pretty complex on their own.
To make matters more interesting, Sublime created their own language grammer format (`.sublime-syntax` files), which is a superset of the TextMate language grammers and supports more features. Sublime can convert any TextMate language grammer into the new .sublime-syntax format.
So when I started seriously looking into syntax highlighting about a year ago, the Pygments syntax highlighter was my first choice and it is pretty good, but I found it to be lacking support for languages we cared about such as TypeScript at the time. Additionally, Pygments was pretty slow when we tried it out. GitHub switched away from it for a similar reason[2]
This was about the time that I ran into the Syntect[3] project. For us, this had huge benefits:
- It was by far the fastest syntax highlighter we found, which meant page load times improved a lot for us.
- Its results were really, really good. I found multiple cases where the output was better than other highlighters such as Pygments.
- The built-in language support, all language Sublime Text 3 supports by default, was great out of the box.
- Because Sublime can convert TextMate grammers to .sublime-text grammers, we ultimately get support for basically all languages.
- It took me _one day_ to write `syntect_server`[4] which highlights arbitrary code for our Go application, which was a spectacular turnaround time given how much time I'd already invested elsewhere in researching all of this.
Overall, we really couldn't be more pleased with Syntect and Rust for modern syntax highlighting. It's a beautiful, fast, high-quality combination.
[1] https://macromates.com/manual/en/language_grammars
[2] https://github.com/github/linguist/issues/1717#issuecomment-...
[3] https://github.com/trishume/syntect
[4] https://github.com/sourcegraph/syntect_server
How much did it improve load times? Why do you think syntect was faster than what you tried previously?
I think Syntect is faster in general just because of the nature of the work (syntax highlighting is basically just a stack machine + running regex 24/7) and Rust being a really performant choice for this in comparison to Python/JS (although I am sure some less idiomatic code could improve the perf of a Python/JS implementation).
Hopefully one day I'll do a proper write-up / blog post on this topic and provide some more hard numbers / stats for everyone. :)
I want to credit the many community members who contributed the majority of the liveblog posts this year at GopherCon. This post was contributed by @shurcooL. It is so cool to see the GopherCon liveblogging tradition grow each year!
Really neat stuff, even though you rarely need more control than what CML gives you.
I'm still waiting for someone to reinvent QNX.
https://typelevel.org/cats-effect/concurrency/