My coworker used to teach non-technical people how to code. Students always found regex hard to understand.
Let's be honest, regex can be hard to understand even for technical people. (At least it's hard for me.)
So coworker built this to help anyone understand arbitrary regex commands.
AI disclosure: this was built using Agent Duel, a local coding agent arena system (https://github.com/bottomless/agent-duel). Promotion disclosure: our company is building agent duel. To test it and find bugs we are building passion projects through the system. I nonetheless thought HN would find this interesting.
Really cool. Do you teach people the mechanics, like what the app shows? I wonder if that actually makes it harder. I'm wondering about where the hard bits are, given that it all boils down to just:
"|", which means "or"
".", which means "any symbol"
"?", which means "at most once",
"*", which means "however many times"
"+", which means "at least once"
Maybe encountering it next to PEG scrambles your brain.
Autodidact here. Beyond just the five core quantifiers |.?*+ regex only has capture groups () and three lazy quantifiers
??
*?
+?
The rest is not deep. So while I think it's a worthwhile suggestion not to show people the whole thing, I'm wondering if the whole thing is really that complicated anyway.
Regex has many more features than this, e.g. backreferences. I agree that it's easy to learn enough regex to be dangerous with that subset, but that's not everything regex offers.
Backreferencing inside of a regex (which is a way to find repeat occurrences of a substring) is a very controversial feature that some regex engines intentionally omit, while most others support it only grudgingly. Supporting backreferences opens up a big can of worms: Most regex engines will fall back to a slow, poorly optimised algorithm if they ever encounter backreferences.
Also, it's important for people not to confuse backreferences with capture groups. They look a bit similar, but capture groups are actually fine.
There are hundred different flavors of what "regex" actually means. The basics are often the same but the extended features differ.
The foundation is in theoretical CS with deterministic and nondeterministic finite automata (DFAs and NFAs), and I found it easiest to teach regexes using DFAs since you can then just draw circles and arrows. Regex notation is just a different notation. It's a bit likeif you know piano and then learn another instrument, your brain can always refer back to the piano and it's all much easier.
They are controversial because supporting them raises matching time to O(2^n) in the worst case, while not supporting them enables matching time to be O(n) in the worst case. And a user might not be aware of that danger.
Also, I think it's mistaken to mention NFAs and DFAs the way you just did. Regexes (or NFAs, or DFAs) simply classify exactly the properties of strings which can be checked in constant space.
Also, no regex implementation actually uses DFAs. NFAs yes, but DFAs no. The point of NFAs is that they're easy to execute in constant space, while also making concatenation and Kleene star easy to implement. It seems much harder to implement concatenation and Kleene star using other approaches, like DFA.
I think we are actually on the same page, but it is funny that we express things in exactly the opposite way as well as reach opposite conclusions. :)
It's not about being pedantic, quite the opposite It's intuitive to use circles and arrows as the mental model, instead of the cryptic regex syntax. Both have the same semantics. The quite visual representaton of the finite automata makes the meaning more approachable, not unlike the marble run illustration.
In contrast, bringing "constant space" and "O(n) matching" etc into the discussion is all formally correct (and I'm a big fan of formally correct) but that's where things become pedantic from the common folk's POV and is also where you lose people who just want to understand how something works practically.
Therefore I'm saying, start withthe easy circles-and-arrows intuition and only go to the theoretical background (aka be pedantic) once people want to know more.
(Backrefs can't be expressed with circles-and-arrows which illustrates why they are something special.)
If you really want to get technical: ., ?, and + are also shorthands. The formal definition of a regular expression is very minimal on purpose.
`.` is just `(a|b|c|...)` - List of all literal characters
`x?` is just `(x|)` - "x or empty string"
`x+` is just `xx*` - "x followed by zero or more x characters"
Whether this is exactly how they are implemented in any particular regex engine is an implementation detail, but formally there are very few operations which aren't just syntactic sugar.
You are barely scratching the surface! You forgot about positive and negative look ahead and behind, lookaround, backreferences, recursion, inline engine flags settings and probably some other things I forgotten about. PCRE2 compatible regexes are potentially monstrously complex things.
Regex is conceptually incredibly easy to understand.
But regex is also syntactically designed by Satan himself.
Most programming languages use commands that are semantically understandable, like "for" and "let" and "return" and mostly symbols we already know like "+" and "=". They use newlines and indentation for organization. And they put multiple commands on multiple lines and you can add comments and everything.
Regex is exclusively punctuation, with its own idiosyncratic meanings, all compressed into a single line, zero organization, zero comments, and impossible to remember what each symbol does if you don't use it daily. Not to mention the nightmare of escaping regexes when expressed as strings, on top of escaping regex symbols within the regex itself.
I swear it's high time we come up with a regex replacement that looks like normal Python code or HTML or something. So instead of:
^(\+\d{1,3}-?)?\d{3}-\d{3}-\d{4}$
We would write something like:
<start>
<optional> # Optional country code
"+"
<digit>[1-3]
"-"
</optional>
<digit>[3] # Area code
"-"
<digit>[3]
"-"
<digit>[4]
<end>
Same thing. But one is completely inscrutable to someone who hasn't learned regex, while with the other you can guess pretty easily how it works...
I am not sure if everyone writes unreadable regex because the reputation of regex makes it seem pointless to try, or if being witty one-liners is a critical part of the value proposition of regex
You can enable extended mode for most engines, which allows you to add white space and comments.
Personally, I like regexes, in the same way that I like shell string manipulation: it looks utterly inscrutable, and performs magic. If I write one in code meant for others, I usually use extended mode with each line commented.
I couldn't care less if it looks more like XML or Python or C or whatever. It's just an example.
I just care that things have readable text names rather than inscrutable punctuation and single letters.
Yes, it is a DSL. Just like, you know, Python or C or HTML or CSS. Just because it's a DSL doesn't mean it can't actually use keywords instead of punctuation.
Actually yes? If math equations also wrote a fluent English way of saying the full thing as well as the literal syntax as well every time it would help more people actually understand it.
My "also" referred not to have multiple representations but to also in math have prose instead of equations. I could have expressed it better. I did not think of that interpretation.
As for math, I really do not think that is true. Look at how this beautiful proof is so long in prose.
The point is, if you don't use regex that frequently, you can completely forget that ^ means start and $ means end. And adding those as line comments ("^ # Start of expression") would be dumb.
That's why a well-designed DSL tries to make its syntax as self-explanatory as possible. Using "start" instead of "^", or "digit" instead of "d".
We're not doing equations here, we're doing engineering. Analogies to proofs are not helpful. There is a very good reason why, in coding, we give descriptive variable names such as "accountBalance" rather than "b".
Conciseness is not a virtue in programming. Clarity is. Things should not be longer than they need to be -- but neither should they be shorter. The name of something should not be 100 characters long, but neither should it be 1 character long. A word or two is generally the sweet spot.
I think some languages do have a library-style version of regex. But the function names aren't consistent between the different libraries. Maybe then we need to put on special goggles to make them look the same.
In practice, including a unit test for any nontrivial regex in your application is very helpful. You can demonstrate which sorts of strings are expected to match and, maybe more importantly, which ones are not expected to match. The same goes for testing the capture group contents.
It may feel silly to have a whole unit test for a one-line regex, but that one line can be really dense. If you wrote a function to parse the string without regex, that function would be complex enough to warrant a unit test, after all.
Introducing regressions when tweaking a regex in your application is also a big risk which unit tests can help mitigate.
What you want is something akin to a Raku (perl6) grammar, which was their enhancement to matching they developed to go beyond regular expressions.
grammar IniFile {
token key { \w+ }
token value { <-[\n;]>+ }
token pair { <key> \h* '=' \h* <value> \n+ }
token TOP { <pair>+ }
}
my $text = "user = alice\nrole = admin\n";
my $match = IniFile.parse($text);
if $match {
say "Parsed successfully!";
}
There are various libraries for other languages that try to accomplish something similar or Port the concept, so if this interests you, maybe you can find one. Or in this day and age, maybe you can have some VC (and much of society) subsidize the fairly quick creation of a library for you language of choice through AI.
FWIW, lpeg is nice, of course you'll still have to deal with the terseness, but I don't think there's anything that can be done, since you will have to write an expression.
patt <- country area "-" phone fin
country <- ("+" %digit^-3 "-")?
area <- %digit^3
phone <- %digit^3 "-" %digit^4
fin <- ! .
Not that impressive, but at least it doesn't group captures and groups and string escapes \ with a class labeling symbol.
America's Cup and Formula One races are won at the design stages. We praise the drivers, but what succeeds is the engineering.
If billions of dollars of IPO value were at stake, competing to use animation to convey an understanding to test subjects, this design would be toast. The model is cripplingly realistic, wasting most of its cognitive bandwidth on provoking a "Gee willikers, it really looks like someone built that out of wood!" reaction, leaving us to squint to see the balls or the parts of the machine implementing the logic. There's a particular genius to logo design, that needs to be applied here. If this were instead a cartoon animation, Walt Disney dosed the coffee one day at the studio to see what his animators could do, we'd get a visualization that better used its cognitive bandwidth to establish an icon language with the viewer.
The abacus is the T-shirt for this idea. Like a musical instrument, one can imagine operating an abacus without holding one, and some shopkeepers were able to leave theirs in a drawer after enough practice. Beads, like balls, leverage our brains' ability to manipulate physical reality, but the idea of counting became more vivid as the abacus design grew more streamlined. It not only became easier to build and operate, but it became a more effective "thought worm" implant into our minds.
We make contact with aliens, and in their museums they have an abacus for regular expressions. What does it look like?
This is important to get right, because soon AI will communicate with us using Pixar-quality 3D animation, generated in real time for each conversational turn. Many people still think of AI communication as exchanging "text message" prompts, but for my math research I'm already getting illustrated PDF slide decks as AI's responses to me in single conversational turns. One just has to ask; the form AI communication takes is only limited by our imagination.
Feynman diagrams and the like as rare exceptions, scientific communication is still mired in "telegraph" form, math symbols one reads along lines of text that once lived on physical paper. As future generations adapt to visual symbolic reasoning, the animations AI will use to communicate with us will evolve from representational to abstract art, for a "fatter pipe" pouring symbolic information into our minds. They'll find the "thought worms" that best hack our minds, make us smarter.
I'd love to see what your regex viewer would look like, if you were imagining this future.
41 comments
[ 0.18 ms ] story [ 63.7 ms ] threadLet's be honest, regex can be hard to understand even for technical people. (At least it's hard for me.)
So coworker built this to help anyone understand arbitrary regex commands.
AI disclosure: this was built using Agent Duel, a local coding agent arena system (https://github.com/bottomless/agent-duel). Promotion disclosure: our company is building agent duel. To test it and find bugs we are building passion projects through the system. I nonetheless thought HN would find this interesting.
- using groups "(" .. ")" in replacement "\"i
- character ranges "[" .. "]"
- character classes "[:" .. ":]"
- multiline match
- ignore case
- beginning / end of line / word
Also, it's important for people not to confuse backreferences with capture groups. They look a bit similar, but capture groups are actually fine.
The foundation is in theoretical CS with deterministic and nondeterministic finite automata (DFAs and NFAs), and I found it easiest to teach regexes using DFAs since you can then just draw circles and arrows. Regex notation is just a different notation. It's a bit likeif you know piano and then learn another instrument, your brain can always refer back to the piano and it's all much easier.
Also, I think it's mistaken to mention NFAs and DFAs the way you just did. Regexes (or NFAs, or DFAs) simply classify exactly the properties of strings which can be checked in constant space.
Also, no regex implementation actually uses DFAs. NFAs yes, but DFAs no. The point of NFAs is that they're easy to execute in constant space, while also making concatenation and Kleene star easy to implement. It seems much harder to implement concatenation and Kleene star using other approaches, like DFA.
It's not about being pedantic, quite the opposite It's intuitive to use circles and arrows as the mental model, instead of the cryptic regex syntax. Both have the same semantics. The quite visual representaton of the finite automata makes the meaning more approachable, not unlike the marble run illustration.
In contrast, bringing "constant space" and "O(n) matching" etc into the discussion is all formally correct (and I'm a big fan of formally correct) but that's where things become pedantic from the common folk's POV and is also where you lose people who just want to understand how something works practically.
Therefore I'm saying, start withthe easy circles-and-arrows intuition and only go to the theoretical background (aka be pedantic) once people want to know more.
(Backrefs can't be expressed with circles-and-arrows which illustrates why they are something special.)
If you really want to get technical: ., ?, and + are also shorthands. The formal definition of a regular expression is very minimal on purpose.
`.` is just `(a|b|c|...)` - List of all literal characters
`x?` is just `(x|)` - "x or empty string"
`x+` is just `xx*` - "x followed by zero or more x characters"
Whether this is exactly how they are implemented in any particular regex engine is an implementation detail, but formally there are very few operations which aren't just syntactic sugar.
But regex is also syntactically designed by Satan himself.
Most programming languages use commands that are semantically understandable, like "for" and "let" and "return" and mostly symbols we already know like "+" and "=". They use newlines and indentation for organization. And they put multiple commands on multiple lines and you can add comments and everything.
Regex is exclusively punctuation, with its own idiosyncratic meanings, all compressed into a single line, zero organization, zero comments, and impossible to remember what each symbol does if you don't use it daily. Not to mention the nightmare of escaping regexes when expressed as strings, on top of escaping regex symbols within the regex itself.
I swear it's high time we come up with a regex replacement that looks like normal Python code or HTML or something. So instead of:
We would write something like: Same thing. But one is completely inscrutable to someone who hasn't learned regex, while with the other you can guess pretty easily how it works...Personally, I like regexes, in the same way that I like shell string manipulation: it looks utterly inscrutable, and performs magic. If I write one in code meant for others, I usually use extended mode with each line commented.
Ultimately it is a dsl. Should equations in math also use full words and comments?
No, they should not. Maybe a nice syntax for grammars like raku is a better approach.
I just care that things have readable text names rather than inscrutable punctuation and single letters.
Yes, it is a DSL. Just like, you know, Python or C or HTML or CSS. Just because it's a DSL doesn't mean it can't actually use keywords instead of punctuation.
My "also" referred not to have multiple representations but to also in math have prose instead of equations. I could have expressed it better. I did not think of that interpretation.
As for math, I really do not think that is true. Look at how this beautiful proof is so long in prose.
https://mathcs.clarku.edu/~djoyce/java/elements/bookI/propI4...
Is that better than a^2+b^2 = c^2 ?
https://en.wikipedia.org/wiki/Pythagorean_theorem
Notation helps to be concise. Imagine more complicated things without notation in maths.
That's why a well-designed DSL tries to make its syntax as self-explanatory as possible. Using "start" instead of "^", or "digit" instead of "d".
We're not doing equations here, we're doing engineering. Analogies to proofs are not helpful. There is a very good reason why, in coding, we give descriptive variable names such as "accountBalance" rather than "b".
Conciseness is not a virtue in programming. Clarity is. Things should not be longer than they need to be -- but neither should they be shorter. The name of something should not be 100 characters long, but neither should it be 1 character long. A word or two is generally the sweet spot.
In languages with an expressive typesystem you see a lot more single letter variables.
> We're not doing equations here, we're doing engineering
This sentence is nonsense.
> Conciseness is not a virtue in programming. Clarity is
I see what you are saying, but conciseness is also related to clarity, as I showed with the theorem.
It may feel silly to have a whole unit test for a one-line regex, but that one line can be really dense. If you wrote a function to parse the string without regex, that function would be complex enough to warrant a unit test, after all.
Introducing regressions when tweaking a regex in your application is also a big risk which unit tests can help mitigate.
If billions of dollars of IPO value were at stake, competing to use animation to convey an understanding to test subjects, this design would be toast. The model is cripplingly realistic, wasting most of its cognitive bandwidth on provoking a "Gee willikers, it really looks like someone built that out of wood!" reaction, leaving us to squint to see the balls or the parts of the machine implementing the logic. There's a particular genius to logo design, that needs to be applied here. If this were instead a cartoon animation, Walt Disney dosed the coffee one day at the studio to see what his animators could do, we'd get a visualization that better used its cognitive bandwidth to establish an icon language with the viewer.
The abacus is the T-shirt for this idea. Like a musical instrument, one can imagine operating an abacus without holding one, and some shopkeepers were able to leave theirs in a drawer after enough practice. Beads, like balls, leverage our brains' ability to manipulate physical reality, but the idea of counting became more vivid as the abacus design grew more streamlined. It not only became easier to build and operate, but it became a more effective "thought worm" implant into our minds.
We make contact with aliens, and in their museums they have an abacus for regular expressions. What does it look like?
This is important to get right, because soon AI will communicate with us using Pixar-quality 3D animation, generated in real time for each conversational turn. Many people still think of AI communication as exchanging "text message" prompts, but for my math research I'm already getting illustrated PDF slide decks as AI's responses to me in single conversational turns. One just has to ask; the form AI communication takes is only limited by our imagination.
Feynman diagrams and the like as rare exceptions, scientific communication is still mired in "telegraph" form, math symbols one reads along lines of text that once lived on physical paper. As future generations adapt to visual symbolic reasoning, the animations AI will use to communicate with us will evolve from representational to abstract art, for a "fatter pipe" pouring symbolic information into our minds. They'll find the "thought worms" that best hack our minds, make us smarter.
I'd love to see what your regex viewer would look like, if you were imagining this future.