This is my first time hearing about Raku (the language this tool is apparently written in), and after a quick look at the website[1], I am not sure why I would want to use Raku vs any other language in particular.
According to the guide[2]:
> Raku is a high-level, general-purpose, gradually typed language. Raku is multi-paradigmatic. It supports Procedural, Object Oriented, and Functional programming.
That really doesn't tell me enough. Anyone else have any idea why I might want to use Raku vs one of the many other langs to which those adjectives can be applied?
It has evolved to be much, much more. It is a somewhat pragmatic, powerful, malleable multi-paradigm language that can be (has been) applied to many use cases beyond text processing. I am only vaguely familiar with the language, but I'm often impressed by examples of it's power and expressiveness.
Raku is a Perl descendant so it's goals are the same as Perl's (for the most part). From what I've seen it is supposed to be a terse language for rapid prototyping, it has syntax support for a lot of common idioms (and of course since it came from Perl you can expect some of the best pattern matching support in the world). For me the performance is a dealbreaker, but otherwise it looks pretty good.
Given that, it's 23 years old and done by Larry Wall so it's not exactly yet another fly-by-night hype language, just something that is not in the top 50.
It has grown to be its own thing with some happenstance parenthood through being born from an early attempt at Perl 6, and thus is now only being slightly more related to Perl than e.g Ruby is. Ok I'm pushing it but you get the idea: taking a biological analogy, it has mutated away far enough that it is now considered to be its own strain.
And if that helps clear some doubts, Perl 7 is on the table (if there ever needs to be a Perl 7):
At some point in the future, the PSC may decide that the set of features, taken together, represent a big enough step forward to justify a new baseline for Perl. If that happens, then the version will be bumped to 7.0.
It is helpful for people who were paying attention to the perl community with any passing interest at any time in the past 20 years. It is not helpful for people who weren't. But the former describes a lot of people here.
Absolutely! In the same way that saying the "Dugong Dugon" is a type of fish even though it's technically a Sirenia.
Here, "some swimming creature" is the totality of the objective, we aren't trying to do a crash course in marine zoology or trying to explain what a Sirenia is or what a fish technically means.
Saying it's perl-6 means that it's not say, lisp-like or some kind of prolog or haskell. It's a relatively swift approximation and that's the totality of the objective.
> Regular expressions (Regexes) work well for finding patterns in strings. However, for some tasks, like finding multiple patterns at once, or combining patterns, or testing for patterns that may surround strings regular expressions, alone, are not enough.
> When working with HTML, you could define a grammar to recognize HTML tags, both the opening and closing elements, and the text in between. You could then organize these elements into data structures, such as arrays or hashes.
To me the first adjectives that come to mind for Raku are quirky and clever. I feel it can be extremely expressive at times, with really good happy path. It is at the same time anti-blub but also steers away from academic ivory towers, instead it is very much hacker/power-user oriented.
(disclaimer: I don't actually know Raku that well, but it is very high on my list of languages to learn)
Because it provides one with a lot of flexibility out of the box while not forcing you into a specific programming style and without any additional overhead or costs …
OOP, functional programming, strong/dynamic typing, declarative /imperative - you’ve got it, we got you covered
It's not clear from the README how even to install it, likely just an exploration of writing file searcher in Raku and not a serious attempt at replacing grep/find/ack/ag/rg.
BurntSushi (dev) built a grep replacement, `rg` (ripgrep), in Rust, that tends to run circles in terms of speed around grep itself and other grep replacements. The UX is also pretty nice. I install it everywhere now.
I'm curious (like others here) how `rak` is better than `rg`.
It is "better" in the sense that it uses Raku regexes, which are incredibly powerful, and if that's not enough for you, you can even write code. "rak" is giving you a general-purpose programming language (Raku) that is optimized for text processing, with convenient shortcuts for the most common tasks.
In contrast "rg" only designed to search text files, but it does that really really fast.
It definitely makes sense to have this tool if you're comfortable writing Raku. I'm not sure it makes a whole lot of sense for most people to learn Raku to use this tool, and if you're not going to learn Raku I don't think there's much value add over tools like ripgrep.
in a strange way I both agree and disagree with this:
+ I use raku a lot so it is very natural to embed raku code in a Rak CLI
- I think that there a several benefits of the Rak tool that vanilla grep (or ripgrep) does not offer (eg. embedding code snippets in the expression, unicode, better regex syntax) that would probably make it worthwhile to learn the Rak examples while not having to bother with wider raku
- Support for NFG (Normalization Form Grapheme), which e.g. means that you only need to specify 'é' if you want to look for an 'é', and not have to worry about whether the text you're searching in, consists of the single codepoint 'é', or that it has the decomposed version.
- support for --ignoremark, which means you 'e' will match any accented 'e', such as éëêèęėē.
Ah yeah that's a good one! It does very likely have enormous implications for performance though. I wonder if I should add it as an opt-in feature to ripgrep. Although its support will be inherently limited in some capacity since character classes will always be limited to matching a single codepoint. (i.e., No UTS#18 Level 2 support.)
Have you found folks using these particular Unicode features in practice? I don't think anyone has request it for ripgrep.
> It does very likely have enormous implications for performance though.
Well, but that's only one of the reasons why rak is a lot slower. There's something else going on, but I currently don't have the mindset to investigate this deeply.
Both learning and the ability of tools are gradual though. For doing naive substring search all grep-alike tools works perfectly, and anyone who are willing to spend some effort and learn a bit of tool-specific features can gain a lot more more benefits.
rak is backed by the entirety of raku language and therefore is much easier for some crafting something that's less trivial to generalize in a one-liner. For most of grep-alike tools, their regex engine would be some sort of ceilings of what can be done but for rak, the ceiling is as tall as the raku programming language. That's rak's niche.
It's phenomenal, is what it is. An extremely fast, recursive-by-default, general-purpose `grep` clone in Rust.
Alongside `fzf` and `fd` (a similar project for `find`) `rg` was what finally convinced me to put together a few shell scripts to bootstrap getting all my favorite modern command line tools on a new Ubuntu VM [1]. I'm working on an interactive playground you'll be able to SSH into to try it out yourself without even having to spin up your own, so you can see the difference with your own two eyes, no VM required.
Actually I'm running NixOS on an old ThinkPad I have lying around here. Home Manager works great! But Ubuntu currently has the popularity network effect going for it, so.
At my previous job I often ended up searching raw log files sized around 5-8GB. Regular grep would grind away for 50-60 seconds, while ripgrep returned the same results in 250ms.
I was shocked the first time I used it and have sung it's praises since.
If you can afford it (these products tend to be expensive if they're doing it, and resource hungry if you're doing it, and sometimes both) and you've got logs in files, look at having the logs live in dedicated software.
I have used (at different places) Graylog and Splunk, New Relic is also popular.
Not that ripgrep isn't awesome, but you could have dedicated indexing and a UI that can be shown to non-technical or at least semi-technical people to review logs whereas "Just use ripgrep" really isn't accessible for most users.
Yeah we had ELK for the common logs. This file was a special case that we didn't want indexed and widely accessible because it contained sensitive data. I only occasionally needed to do some digging, so I was fine resorting to ripgrep.
To me, the two big things about ripgrep are (1) it's much more oriented towards the use case of searching a directory hierarchy full of code, eg by default it's recursive and respects .gitignore, it has a --type flag which knows various kinds of source files (eg --type c will only look at .c and .h files), etc, and (2) it's fast - uses all the lastest and greatest algorithms, and makes heavy use of parallelism. It's also just generally good and sensible.
The README seems to make it clear that increased finger exercise is an advantage (steroids are used for increasing muscle mass, after all).
Instead of just typing
rg '\bfoo\b'
you get to type
rak '/ << foo >> /'
and instead of just typing
rg -B2 -A2 foo
or
rg -C2 foo
you get to type
rak foo --before=2 --after=2
(over double the characters!) and instead of just typing
rg 'foo.*bar|bar.*foo'
you get to type
rak '{.contains("foo") && .contains("bar")}'
which has double the characters and also has the advantage of writing each pattern only once.
Okay, I haven't actually tried it (and the README doesn't say how to), and rak does have a bunch of features and capabilities ripgrep doesn't have, but those first examples are not very inspiring after the big initial claim.
[Edited to add] Also, the `\b...\b` form is shorter with rak - as the README also states where it documents this pattern form:
> §string
> If the pattern starts with §, then it indicates that the string should occur as a word (with word-boundaris on both ends) in the item. Basically a shortcut to specifying string --type=words. Any --smartcase, --smartmark, --ignorecase or --ignoremark arguments will be honoured.
That's not correct. ASCII has only 128 codepoints and its maximum value is 127 or 0x7F. Code 167 is not part of ASCII. You're probably thinking of ISO 8859-1.
U+00A7 (codepoint 167) is also the Unicode codepoint for §. Its use on this very web page is via Unicode and the UTF-8 encoding as the bytes \xC2\xA7.
(This is meant as a narrow correction. It doesn't really change the broader point that § is not exactly easy to type on most keyboards. Whether its ASCII or not.)
Sorry, you're right. I guess I meant 8859-1 as opposed to Windows codepages or Unicode-exclusive. My Western bias is showing as I don't know if there are old locales that wouldn't have it that would affect any international users, but I assume not?
rak '{.contains("foo1") && .contains("foo2") && .contains("foo3") && .contains("foo4") && .contains("foo5")}'
My point is that despite their claim of superiority, their examples do not show any. Maybe there's a more efficient rak way to do that chain of .contains? Why not demonstrate that?
There is problem with nice modern Linux tools - they are not available on the generic servers and embedded boxes with you will likely use, or in the image someone else has given you etc. Therefore it is pointless (sorry for the strong term) to learn them, because they simply won't be available for when they are most needed. And learning new tools is not a free process, there are costs, mental and time.
I think you need to caveat the word "pointless" heavily. If 95% of the commands I type are in my local dev environment where I can install a variety of tools, then I don't see how it's pointless to learn different tools for the common case. Otherwise, you're just needlessly limiting yourself to a constrained set of tooling with zero opportunity for any kind of progress or improvement.
Even though I'm the author of ripgrep, believe it or not, I still use grep in those 5% of cases without any kind of fuss.
This is probably a good default position for a random new tool that floats across hacker news, but there are plenty of situations where the benefits of having a tool (even if only in your local environment) outweigh the effort of learning it and getting it.
I reject that heartily. I spend 90% of my shell time on my own machine, and I want it to be comfortable. Here “at home” I use nice editors, and Fish shell, and ripgrep/fzf/etc., because they make my daily life more pleasant and productive. Some remote servers are minimal to the point that they only have Bourne shell and plain vi available, and when I’m working on them, that’s what I use. But darned if I want to handcuff myself to the greatest common denominator in my daily work.
I promise, you can learn both grep and ripgrep at the same time.
This is why I'm also skeptical of the "modern" replacements, like Just for example. The majority of developers already have Make on their system and Make works well enough. When you use Just instead, you're adding friction to the build process for some portion of people who want to work on your code. You have to context switch to go and install it. Brew decides it wants to upgrade everything. Aaaand you've wasted 15 minutes because you went to get a coffee in the meantime.
If you control the servers you’ll be working on, you can install the tooling you’d like. If you don’t, you can lobby for it to be installed.
Some Postgres servers I work on have massive log files, and due to DDOG indexing retention being short, sometimes it’s required to go sleuthing on the host. While LC_ALL=C speeds up grep quite a bit, it’s still an order of magnitude slower than ripgrep.
Luckily, I can add tooling to these servers if there is a valid need, and “massively speeds up searches when speed is of the essence” is very much an accepted case.
Our test servers are upgraded weekly/daily, and embedded devices we are testing are upgraded daily/hourly. It's just not feasible. And embedded devices has an additional constraint for space and/or bureaucracy. I'm still waiting for the feature complete "less" to be added there after like half a year (we have a weird distro with busybox bolted on top). I understand that my case is maybe an outlier, but I've resigned that I only learn what's available in the busybox and basic Debian.
I wish that distributives maintainers would enforce inclusion of the modern tools like Rak in the default install, then they would become the norm across whole industry. Like for example MS includes Powershell regardless of the opinion of people who dislike it. And now you can expect that powershell is available in any modern Win install. In latest Win11 new Terminal is now a permanent feature in every install and you can expect to have it everywhere on Win11. Etc.
Embedded notwithstanding (I admit that’s a special case where most norms go out the window), you could put the tooling and configs into Ansible plays and run them with the upgrades, or use Puppet to push them continuously.
Regarding distro maintainers, specifically Debian, ripgrep is in Bookworm [0].
Very cool! It looks like Rak has a lot of nice features.
I will say that getting this thing built took some digging. Could you please add install/build instructions to the README? As it is, I had to go sleuthing through your CI setup to figure it out. On Archlinux, here's what I did:
> I'm not totally clear on what kind of filtering `rak` does by default
By default, rak search all of the files that look like text, by reading the first 2K (if I remember correctly) and using some algorithm to determine "binaryness".
> I realize features and an expressive language is your main goal
Indeed.
> I wanted to see what it looked like from a perf perspective
The performance times you've given here, coincide with my findings. Which for the "sixteenth.txt" surprises me: a bare loop in Raku reading that file line-by-line finishes in 8 seconds for me. Which would be one order of magnitude less than the search result. Still not in `rg`'s ball park, but speed wasn't a big concern for me. But 8 -> 48 seconds is a pretty big difference, so I guess it's back to the drawing board for me on that one :-)
@burntsushi - thanks for the benchmarks - certainly rg is amazingly fast! I was wondering if somehow rak and rg could be combined to get the benefit of both worlds (eg. use the strong ffi support in raku and rust to interface them) ... but this would only make any sense if you could apply the rg speed boost to raku regex and thus get the syntax benefits too
(sidebar - Larry took to opportunity with perl6/raku to properly reinvent regexes for unicode - there is quite lot to unpack in raku regexes: graphemes management, unicode props as char classes, named regexes, etc.)
soo - I suggest this proposal to anyone looking for a cool rust/unicode project ... extend ripgrep from PCRE/ECMA262 regex 1.0 to embrace the "21st century" regex syntax of raku
To be clear, to do this one would likely need to write an entirely new regex engine. ripgrep's default engine uses finite automata. It's not PCRE2 or ECMA. Many of the things in Raku likely can't easily be implemented in finite automata.
ripgrep does have fairly complete UTS#18 Level 1 support already.
> likely need to write an entirely new regex engine
Indeed.
Also, it should be noted that the Raku Regex engine is not a state machine as such. The regex "slang" in Raku, is basically just another way to write code. Code that uses some state machine primitives underneath, for sure, but still code.
A grammar in Raku is nothing other than a class in which the methods are codegenned using the regex syntax. But a class nonetheless, which can also have attributes and "proper" methods.
This gives the Raku regex engine the flexibility needed to be able to parse Raku source code, and generate executable bytecode from that.
forgive me for assumption that rg does PCRE(2)/ECMA, we had a bit of a chat on raku and UTS#18 here ... https://www.reddit.com/r/rakulang/comments/15ge67k/uts_18_un... ... raku does not attmept to implement that explicitly but (by virtue of being an HLL and having built ins such as Set operators) comes quite close to it at all levels.
And yeah, I figured Raku might not follow UTS#18 to the letter of the law with respect to syntax, but likely has all the concepts available. Which makes sense given what Raku is.
definitely not going to install cursed Raku for this single search lib. perhaps if they had a distributed binary in some byte-code format, but they don't. because forked-Perl-guys were busy figuring how to use all the Unicode and outpace APL in complexity, while others were figuring how to target reasonably large distribution medium (think wasm, .net, and jre if u want).
besides, ripgrep IS much more state of the art than anything else out there. very fast and can fallback to pcre2 for weird lookahead and utf8 (if i remember correctly, sorry). we have to admit that BurntSushi made incredibly good job at it.
ACTUALLY what I need most of the times is qgrep which can grep regexes in 100k files in less than second. is re2 + index essentially and none of its cousins can do similar searches reasonably fast. the only downside - it indexes per-line, so if you wanna grep some fancy multiline-SQL you have to do second pass over the files that match on more general basis
not sure why codeql could not get up to speed with SQL, but I've several times considered actually using multi-passes with tree-sitter in order to dig some SQL neatly hidden in multiline concatenated Java strings. why? well, the usual
"order of things" in many large projects is after all nothing more than respectful mess of entangled enterprise-level orchestration (code) piled years after years. i guess this' why someone at google had to write qgrep to dig the monorepo.
APL is one of the least complex languages I know (simplest being Lisp); there isn’t much syntax to learn and the language doesn’t really have implicit footguns. You just string together function applications and modify some of those functions with adverbs
Let me give an allegory - perhaps not exact, but you'll get my point. Much of the mathematics (either discreet, differential, computation geometry, etc) is actually not so difficult to understand when told right... but incidentally, there were some folks back in the day who decided to write it using the whole greek alphabet and then some cryptic symbols in it (like these https://cloford.com/resources/charcodes/utf-8_mathematical.h...). By doing so they effectively formed some weird cryptic cult of the Mathematicians, which eventually repelled many people away from the discipline, and still do.
Later one, some perhaps not so smart people, figured how to express stuff as 'sum', 'product' in for-loops and suddenly the whole world was writing programs while decrying math as complex. even though... programs are logic are math, right?
I assume this is directed at the author of rak, so I'll answer.
$x ~~ $y is basically syntactic sugar for $y.ACCEPTS($x). Using ~~ involves an additional subroutine call, so there's a short-term efficiency issue (as in programs running more than a second or so the extra call will have been inlined).
But personally, I like the verbosity of $y.ACCEPTS($x) as it reads better for me, having been burnt by ~~ semantics in Perl, which was symmetrical.
97 comments
[ 3.6 ms ] story [ 198 ms ] threadAccording to the guide[2]:
> Raku is a high-level, general-purpose, gradually typed language. Raku is multi-paradigmatic. It supports Procedural, Object Oriented, and Functional programming.
That really doesn't tell me enough. Anyone else have any idea why I might want to use Raku vs one of the many other langs to which those adjectives can be applied?
[1] https://docs.raku.org/
[2] https://raku.guide/#_what_is_raku
Given that, it's 23 years old and done by Larry Wall so it's not exactly yet another fly-by-night hype language, just something that is not in the top 50.
It used to be, but not anymore.
It has grown to be its own thing with some happenstance parenthood through being born from an early attempt at Perl 6, and thus is now only being slightly more related to Perl than e.g Ruby is. Ok I'm pushing it but you get the idea: taking a biological analogy, it has mutated away far enough that it is now considered to be its own strain.
And if that helps clear some doubts, Perl 7 is on the table (if there ever needs to be a Perl 7):
https://blogs.perl.org/users/psc/2022/05/what-happened-to-pe...So Raku cannot be Perl 6!
Given that, nobody got the memo, so calling it "perl 6" is still a very fast way to orient people as to what it's a neighbor of.
Here, "some swimming creature" is the totality of the objective, we aren't trying to do a crash course in marine zoology or trying to explain what a Sirenia is or what a fish technically means.
Saying it's perl-6 means that it's not say, lisp-like or some kind of prolog or haskell. It's a relatively swift approximation and that's the totality of the objective.
C of 40 years ago is nothing like modern c.
Same for basic.
https://docs.raku.org/language/grammar_tutorial
> Regular expressions (Regexes) work well for finding patterns in strings. However, for some tasks, like finding multiple patterns at once, or combining patterns, or testing for patterns that may surround strings regular expressions, alone, are not enough.
> When working with HTML, you could define a grammar to recognize HTML tags, both the opening and closing elements, and the text in between. You could then organize these elements into data structures, such as arrays or hashes.
(disclaimer: I don't actually know Raku that well, but it is very high on my list of languages to learn)
OOP, functional programming, strong/dynamic typing, declarative /imperative - you’ve got it, we got you covered
https://conf.raku.org/talk/174
https://dev.to/lizmat/its-time-to-rak-part-1-30ji
(...I am somewhat disappointed that I don't have a ridiculously buff nose by now but at least it matches the rest of my body).
Thanks, I wouldn't have been able to tell otherwise.
Whait what's that? I'm stuck with ag and {a..z}grep.
I'm curious (like others here) how `rak` is better than `rg`.
https://github.com/BurntSushi/ripgrep
In contrast "rg" only designed to search text files, but it does that really really fast.
I makes sense to have both.
- support for --ignoremark, which means you 'e' will match any accented 'e', such as éëêèęėē.
Have you found folks using these particular Unicode features in practice? I don't think anyone has request it for ripgrep.
Well, but that's only one of the reasons why rak is a lot slower. There's something else going on, but I currently don't have the mindset to investigate this deeply.
rak is backed by the entirety of raku language and therefore is much easier for some crafting something that's less trivial to generalize in a one-liner. For most of grep-alike tools, their regex engine would be some sort of ceilings of what can be done but for rak, the ceiling is as tall as the raku programming language. That's rak's niche.
Alongside `fzf` and `fd` (a similar project for `find`) `rg` was what finally convinced me to put together a few shell scripts to bootstrap getting all my favorite modern command line tools on a new Ubuntu VM [1]. I'm working on an interactive playground you'll be able to SSH into to try it out yourself without even having to spin up your own, so you can see the difference with your own two eyes, no VM required.
[1]: https://github.com/hiAndrewQuinn/shell-bling-ubuntu
At my previous job I often ended up searching raw log files sized around 5-8GB. Regular grep would grind away for 50-60 seconds, while ripgrep returned the same results in 250ms.
I was shocked the first time I used it and have sung it's praises since.
I have used (at different places) Graylog and Splunk, New Relic is also popular.
Not that ripgrep isn't awesome, but you could have dedicated indexing and a UI that can be shown to non-technical or at least semi-technical people to review logs whereas "Just use ripgrep" really isn't accessible for most users.
TBH more frequently I have "argument list too long" problem, than "one 5GB file" problem.
Instead of just typing
you get to type and instead of just typing or you get to type (over double the characters!) and instead of just typing you get to type which has double the characters and also has the advantage of writing each pattern only once.Okay, I haven't actually tried it (and the README doesn't say how to), and rak does have a bunch of features and capabilities ripgrep doesn't have, but those first examples are not very inspiring after the big initial claim.
So
will do what you expect.[Edited to add] Also, the `\b...\b` form is shorter with rak - as the README also states where it documents this pattern form:
> §string
> If the pattern starts with §, then it indicates that the string should occur as a word (with word-boundaris on both ends) in the item. Basically a shortcut to specifying string --type=words. Any --smartcase, --smartmark, --ignorecase or --ignoremark arguments will be honoured.
Typing § is not very easy on most keyboards, and is actually impossible on some keyboard-OS combinations (including mine): https://en.wikipedia.org/wiki/Section_sign#Keyboard_entry
> If the pattern starts with §,
Is it April 1st already?!
I wonder if keyboards in the future will settle on a way to provide access to more characters, than is currently common.
I agree the keyboard thing is really annoying. Maybe the West should investigate the efforts China has put into different typing systems.
U+00A7 (codepoint 167) is also the Unicode codepoint for §. Its use on this very web page is via Unicode and the UTF-8 encoding as the bytes \xC2\xA7.
(This is meant as a narrow correction. It doesn't really change the broader point that § is not exactly easy to type on most keyboards. Whether its ASCII or not.)
https://keyshorts.com/blogs/blog/37615873-how-to-identify-ma...
Do the "contains" example with 5 terms.
This actually shows one of the secret powers of the Raku Programming Language: Junctions https://docs.raku.org/type/Junction
Alternately one could express this as:
$ rak '{.contains: <foo1 foo2 foo3 foo4 foo5>.all}'
Or using whatever currying: https://docs.raku.org/type/Whatever
$ rak '*.contains: <foo1 foo2 foo3 foo4 foo5>.all'
Even though I'm the author of ripgrep, believe it or not, I still use grep in those 5% of cases without any kind of fuss.
I promise, you can learn both grep and ripgrep at the same time.
Some Postgres servers I work on have massive log files, and due to DDOG indexing retention being short, sometimes it’s required to go sleuthing on the host. While LC_ALL=C speeds up grep quite a bit, it’s still an order of magnitude slower than ripgrep.
Luckily, I can add tooling to these servers if there is a valid need, and “massively speeds up searches when speed is of the essence” is very much an accepted case.
I wish that distributives maintainers would enforce inclusion of the modern tools like Rak in the default install, then they would become the norm across whole industry. Like for example MS includes Powershell regardless of the opinion of people who dislike it. And now you can expect that powershell is available in any modern Win install. In latest Win11 new Terminal is now a permanent feature in every install and you can expect to have it everywhere on Win11. Etc.
Regarding distro maintainers, specifically Debian, ripgrep is in Bookworm [0].
[0]: https://packages.debian.org/bookworm/ripgrep
Very cool! It looks like Rak has a lot of nice features.
I will say that getting this thing built took some digging. Could you please add install/build instructions to the README? As it is, I had to go sleuthing through your CI setup to figure it out. On Archlinux, here's what I did:
I realize features and an expressive language is your main goal, but I wanted to see what it looked like from a perf perspective: I'm not totally clear on what kind of filtering `rak` does by default, so I also ran a search on a single plain text file:> I'm not totally clear on what kind of filtering `rak` does by default
By default, rak search all of the files that look like text, by reading the first 2K (if I remember correctly) and using some algorithm to determine "binaryness".
> I realize features and an expressive language is your main goal
Indeed.
> I wanted to see what it looked like from a perf perspective
The performance times you've given here, coincide with my findings. Which for the "sixteenth.txt" surprises me: a bare loop in Raku reading that file line-by-line finishes in 8 seconds for me. Which would be one order of magnitude less than the search result. Still not in `rg`'s ball park, but speed wasn't a big concern for me. But 8 -> 48 seconds is a pretty big difference, so I guess it's back to the drawing board for me on that one :-)
(sidebar - Larry took to opportunity with perl6/raku to properly reinvent regexes for unicode - there is quite lot to unpack in raku regexes: graphemes management, unicode props as char classes, named regexes, etc.)
soo - I suggest this proposal to anyone looking for a cool rust/unicode project ... extend ripgrep from PCRE/ECMA262 regex 1.0 to embrace the "21st century" regex syntax of raku
some initial reading is here: - https://dev.to/bbkr/utf-8-glyphs-and-graphemes-331b - https://docs.raku.org/language/regexes - https://github.com/edumentab/p6-ecma262regex
ripgrep does have fairly complete UTS#18 Level 1 support already.
Indeed.
Also, it should be noted that the Raku Regex engine is not a state machine as such. The regex "slang" in Raku, is basically just another way to write code. Code that uses some state machine primitives underneath, for sure, but still code.
A grammar in Raku is nothing other than a class in which the methods are codegenned using the regex syntax. But a class nonetheless, which can also have attributes and "proper" methods.
This gives the Raku regex engine the flexibility needed to be able to parse Raku source code, and generate executable bytecode from that.
And yeah, I figured Raku might not follow UTS#18 to the letter of the law with respect to syntax, but likely has all the concepts available. Which makes sense given what Raku is.
besides, ripgrep IS much more state of the art than anything else out there. very fast and can fallback to pcre2 for weird lookahead and utf8 (if i remember correctly, sorry). we have to admit that BurntSushi made incredibly good job at it.
ACTUALLY what I need most of the times is qgrep which can grep regexes in 100k files in less than second. is re2 + index essentially and none of its cousins can do similar searches reasonably fast. the only downside - it indexes per-line, so if you wanna grep some fancy multiline-SQL you have to do second pass over the files that match on more general basis
not sure why codeql could not get up to speed with SQL, but I've several times considered actually using multi-passes with tree-sitter in order to dig some SQL neatly hidden in multiline concatenated Java strings. why? well, the usual "order of things" in many large projects is after all nothing more than respectful mess of entangled enterprise-level orchestration (code) piled years after years. i guess this' why someone at google had to write qgrep to dig the monorepo.
Let me give an allegory - perhaps not exact, but you'll get my point. Much of the mathematics (either discreet, differential, computation geometry, etc) is actually not so difficult to understand when told right... but incidentally, there were some folks back in the day who decided to write it using the whole greek alphabet and then some cryptic symbols in it (like these https://cloford.com/resources/charcodes/utf-8_mathematical.h...). By doing so they effectively formed some weird cryptic cult of the Mathematicians, which eventually repelled many people away from the discipline, and still do.
Later one, some perhaps not so smart people, figured how to express stuff as 'sum', 'product' in for-loops and suddenly the whole world was writing programs while decrying math as complex. even though... programs are logic are math, right?
I was wondering why, in your coding style, you always use $y.ACCEPTS($x) instead of $x ~~ $y?
$x ~~ $y is basically syntactic sugar for $y.ACCEPTS($x). Using ~~ involves an additional subroutine call, so there's a short-term efficiency issue (as in programs running more than a second or so the extra call will have been inlined).
But personally, I like the verbosity of $y.ACCEPTS($x) as it reads better for me, having been burnt by ~~ semantics in Perl, which was symmetrical.