I don't know why I'd never previously considered regular expressions as being a compile/transpile target. It's pretty obvious from PL theory and makes a ton of sense.
That said, after looking at this syntax, I'm not sure that this is much of an improvement. Maybe I've spent far too much time in Regex land [1], but I know I'd perform much slower in this. It's not particularly beautiful, either. The verbosity doesn't seem clearer.
Variables and comments are great, though. We need to add them in future regexes.
Overall, good idea. I'd like to see more takes on this.
It may be a good way to standardize regexp syntax for users of all levels of expertise.
Every text editor, shell environment, programming language or desktop application seems to use regular expressions with a syntax which is slightly different to all the others, but not different enough to call it with a different name.
This means that a newbie learning regular expressions will be thrown into an environment where it can learn the basic principles, but the rules it learns are not generalisable to all applications (e.g. do I match an arbitrary string with '*', or '.*' ? Can I reuse matched patterns with (1) or with {1}? Etc.)
A new readable and easy-to-learn syntax that is nevertheless portable may work like markdown, as a simple-yet-universal new way to apply regexp that newcomers may learn with confidence and apply everywhere, replacing all the previous slightly incompatible versions.
Matching an arbitrary string with '*' is not a regular expression - it's a shell glob pattern. Different thing entirely. Maybe they should be the same thing, yes.
And although I don't know for sure, reusing a matched pattern within the regex string using (1) seems like it would be very strange, because () are metacharacters for grouping.
They are valid patterns in MS Word and Notepad++ search and replace, though. (Word uses parentheses to create the group then \1 to reuse it in the replacement string, while Notepad uses $1).
That's what I meant about every system using a different syntax for what's basically the same functionality.
Interesting idea but the syntax adds nothing to the readability of regex.
I’d be more impressed with this if it targeted multiple different regex targets, since not all implementations of regex are equal. But in its current state it has all the problems of regex plus now the problems of this new language on top.
Looks a fun personal project though. Hope the developers enjoyed building it.
After learning Perl, nearly every language has been a disappointment when it comes to regexes and text handling in general. I couldn't believe it when Java was released with not only no regex capability in the language syntax, but no library either.
During Sun’s initial Java 1.0 release roadshow I drove to Chicago to attend. At the time, java.util.regex didn’t exist. I don’t know how to identify when it appeared.
Says at the bottom since 1.4 which would be 2002 according to wikipedia but the history of java version numbers seems weird and 1.4 might just be the last time the java.util.regex api changed.
It's a language that compiles to regular expressions, aiming to make regexes easier to write and to maintain. It's compatible with many regex engines and polyfills Unicode support where necessary.
While a language like Raku (formerly known as Perl 6) is unlikely to catch up in the current landscape, it did bring a lot of improvements to regular expressions (link to section that starts to use interesting examples [1]).
Way back somewhere in the 2010s when I was still keeping an eye on Perl 6, I was kind of hoping that all these improvements would make their way into some kind of PCRE v3 that all other tools that already use PCRE would switch to. Would have been nice.
The greatest problem with regexp is not its complexity nor learning curve: it's its apparent complexity & learning curve. That tends to put learners off, but anyone I know who's persisted has found the journey faster than expected. After they're over it, everyone I've talked to thinks pretty highly of the syntax: it's simple (small set of compostable components) & expressive, especially if using labelled groups.
The second greatest problem with Regexp is learning query optimisation: that's definitively complex but this new syntax doesn't even attempt solving that.
Eh, I can manage to write a regexp with a little effort, but reading them has never stopped being painful. So in my anecdotal, sample-size one experience it leans heavily towards write-only language.
The only thing regexp has on the plus side is that it's compact. I'd prefer something that's more readable without knowledge. I know how to use it but I always have to google it. I only use it once per year and so there's no big win in mesmerizing it.
This is 95% of it. After that, it's character classes and lazy matching.
Character classes can often be given as ranges:
[a-z] [A-Z] [0-9] ^ $
You can use the pipe | to combine then.
Lazy matching finds the shortest matching string instead of the longest one. This is given by the symbols:
*? +?
Instead of + and * respectively.
> I only use it once per year
If you use a text editor, any kind of repetitive find+replace activity can be sped up using regexes. Seems strange you only use it once per year.
Some people tell you that you can't parse XML using regexes. THEY'RE WR- right, but you can often do decent one-off jobs on an XML document using regular expressions, as long as you're aware that it's a one-off job that won't work on any other document.
It is not that it is hard to understand the syntax or even write them, it is hard to read and change them. Here is an example I just grabbed off a random webpage:
That is hard to quickly* read and understand what it is. Sure a comment would help, but if someone said it isn't working, I have to work all that out and then fix it.
There is a reason Perl lost to Python. It just seems like we can do better.
You can of course disqualify me in saying I can't possibly have persisted at learning it, if I don't like it. But I am inevitably "the guy who knows regular expressions" in the places I've worked. I do use them, for stuff like fixing tedious one-off data munging jobs without having to write an actual program to do it. But I
* Do it step by step, to avoid writing anything resembling a complex regex, because I've yet to meet anyone who can write a complex regexp without bugs.
> The second greatest problem with Regexp is learning query optimisation: that's definitively complex but this new syntax doesn't even attempt solving that.
Query optimisation is totally unnecessary if you use Rust's or Go's implementation.
I'm a big believer that "premature optimisation" is a bad thing but making a blanket statement that any given implementation is 100% performant for all use-cases is going a bit far.
I doubt whatever you're doing is going to work in Rust or Go.
Say that the length of your string is n, and the length of your regular expression is m. Isn't Regular Expression matching in those languages guaranteed O(m n) time? In plain English, that's only linear time - which you practically can't improve on - even if you tried! It can only do worse if your implementation uses backtracking instead of Finite Automata - which was very common at one time, but is now considered The Wrong Way: https://swtch.com/~rsc/regexp/regexp1.html
There is actually a way to optimise your queries, which is by precompiling them to DFAs, but I don't think that's what you're doing.
> There is actually a way to optimise your queries, which is by precompiling them to DFAs, but I don't think that's what you're doing.
You can do that, but the primary advantage of DFAs in practice is backtracking avoidance, so simply reduction of backtracking in manually constructed queries is more what I was referring to. The missing part of the puzzle here being that not supporting backtracking at all doesn't limit capabilities, so removal rather than reduction is the go/rust approach.
The /x mode (extended regexp) lets you use any amount of whitespace or commenting. I use that mode whenever a regex gets complicated enough to merit an explanation (which is quickly), and also indent groups and whatnot. Examples of where I used this in the past are here:
I feel like this opinion is from people who use regexes only rarely. I use them all the time at work, and you become pretty adept at parsing and understanding even dense ones. Of course using the /x modifier is still recommended.
Disagree, honestly. I really don't like this True Scotsman absolutist statement either. If pomsky was able to guard against ReDos then maybe I'd consider switching. Otherwise, I have no problems reading and understanding regular expressions.
It's such a bad take. I need to check if a string is a dash followed by two numbers. What's better, /^-\d{2}$/ or writing 10 lines of bespoke code to parse it?
> Also regex / variable interpolation should be added to regexes in languages probably
I've often wondered and desired this myself. It feels like the single most useful change one might make, because it enables (albeit very basic) logical decomposition. It comes in handy when a regex works for your use case, but the regex is somewhat long but with many repeated parts. In my experience, that isn't too rare, and the regex readability would likely be helped quite a bit by some kind of decomposition.
You can do that by building up a string and converting that to a regex. Pseudo-code:
let day = "[0-9]{2}"
let month = "jan|feb|…|dec"
let year = "[0-9]{4}"
let regex = Regex(month + " " + day + ", " + year)
Disadvantage is that, if your programming language has automatic syntax check of regular expressions, you probably lose that.
The string concatenations also make things a bit slower, but that almost always needs to be done only once.
Advantage is that this makes testing easier. For example, you can do
let dayRegex = Regex(day)
and then test that regex in isolation. That probably is overkill for this example, but if you change the regex to only accept numbers in [1,31], that can be useful.
If you decide to have your regex to treat 30-day and 31 day months separately, things get more complicated. I wouldn’t go that far, though, and certainly not as far as handling leap years correctly. When regexes get complex, it’s time to go for a proper lexer/parser.
My original comment was talking about adding variables to the regex language itself. String interpolation and combinator APIs and enums and type safety are all beside the point. They surely solve a similar problem as regex variables in certain contexts, but they go about it in a very different way and are specific to the context in which the regex is created.
Perl's been doing this for ages with the qx/.../ operator. It indeed works great for logical composition although Raku's first-class grammars are a much better evolutionary step IMO.
Maybe not a feature eventually, but absolutely during a transitional time, particularly if many of those who would consider such a transition are like me— they "know" regex, have written thousands of them, but even decades later continue to find them annoying, finicky, and hard to debug.
For spec review from business partners I have found "verbal expressions" the most useful flavor. Specifically because to review them you don't need to know regular expressions and the library has ports for most programming languages.
Cognition works in two ways: 1) highly deliberate, slow, patient and 2) spontaneous, quick, reactive. I believe software languages do a great job of leveraging both of these capacities of the minds of programmers.
The first type of thinking is generally not composed in code. It happens staring out the window, or looking at data tables or lists and reading documentation, drawing out the architecture. The second type happens in code composition: programmers know what snippet will accomplish which outcome, and it just comes out when we know it's needed. Boom, we have the thing we imagined during the architecture phase. Maybe we look something up or slow down while writing code, but the composition is fairly reactive/spontaneous. It flows.
So coding goes in this rhythm... thinking openly and patiently and deliberately, followed by a burst of code composition, and then more open/deliberate thought while staring out a window. Again and again. Maybe we reverse some of the quick/spontaneous thinking to rearchitect according to our slow/deliberate thought.
Enter regular expressions. I have never, ever spontaneously composed a regex. I have to be very slow and deliberate. I have to recall the precise definition of every character in the regex as I read or write it. Sometimes I stare at a regex for 2 minutes, look up some operator, stare again for 2 minutes, refactor it. Finally I understand it and can now apply the regex spontaneously to whatever environment I'm working in. It always feels like first principles architecture, never like flow. This is odd, as regex is like a giant shorthand engine.
Maybe I have not yet formed the neurological connections required for it to flow – but more likely this is a function of the language. There must be a syntax that better supports flow-state composition. (I don't think Pomsky accomplishes it.)
The regular formalism is, fundamentally, about composition, and the current syntax doesn't do it justice.
That syntax was devised as a math notation, where sub-patterns were abstracted as one letter variables. You didn't have to deal with complex patterns at all, and it was very readable for that use case.
It was then re-purposed as a write-only language for searching at the CLI (in `ed` and its descendants, then grep).
They gradually graduated into what they are today in general programming languages because they were familiar and worse is better... However, composition got lost in the process.
Swift takes a radically different approach, and gives RegExps a parser combinator syntax. So every sub-expression can be assigned to a variable and reused/tested independently.
When I first encountered the Haskell Parsec library, the beauty of parser combinators hit me and I felt I'd never have to write a regular expression again.
This hasn't been the case because I had to work in other languages where usage of regexes was more idiomatic/commonly understood. Parser combinators are great but most of the time, the problem can easily be solved using a regex and we can't really justify adding yet another library in the dependencies for the rare case it would be better suited.
In addition to all the benefits they can provide in terms or readability, re-usability and abstraction, parser combinators can even compete with regexes in terms of performance. https://pl-rants.net/posts/regexes-and-combinators/
>Enter regular expressions. I have never, ever spontaneously composed a regex. I have to be very slow and deliberate. I have to recall the precise definition of every character in the regex as I read or write it. Sometimes I stare at a regex for 2 minutes, look up some operator, stare again for 2 minutes, refactor it. Finally I understand it and can now apply the regex spontaneously to whatever environment I'm working in. It always feels like first principles architecture, never like flow. This is odd, as regex is like a giant shorthand engine.
I can do simple regexes at pretty close to the speed of thought. e.g. here's one to recognize the "arthurofbabylon 20 minutes ago" bit at the top of HN comments:
I typed that out pretty much all in one go, just had to look up different ways the HN date thing gets formatted. I put it in an online checker and it seemed to work on some examples, just had to fix a mismatched parentheses pair.
Not saying it's completely correct (probably someone will reply with a counterexample or problem with it) but I think it's a demonstration that regex fluency is possible. I do this kind of thing on the command line a lot, interactively to figure stuff out from logs and so on. For production code I'm more circumspect.
But isn't that true of most programs once they reach a certain level of complexity? Especially after they've been optimized? Eventually one relies on tests to ensure your changes don't introduce bugs and unwanted side effects. Sadly, most folks don't bother to write tests for their regexes.
Just to chime on among the naysayers, I just read your regex at "the speed of flow". Made perfect sense to me. (I'd have used non-matching groups, but that's a very minor nit.)
The equivalent code to parse this is much longer and arguably to me not really easier to grok. (Unless of course you regularly avoid regex, in which case it will always remain hard.)
I'd say the hardest thing for beginners is the character class and how it matches EXACTLY ONE LETTER without a modifier. Everything matches exactly one letter without a modifier. It seems intuitive to me, but obviously a lot of folks have trouble with it, so I accept that we are outliers.
Reminds me a bit of folks who steadfastly avoid SQL. You have to change the flow in your brain; there's more than one flavor of flow. For regex, it's the watching for single character match with optional modifier. For SQL, it's thinking in terms of sets, not instructions.
If regexes make you feel stupid, lean into it instead of recoiling from it. That's where the learning happens: immersion into the uncertainty until it becomes familiar.
>Reminds me a bit of folks who steadfastly avoid SQL. You have to change the flow in your brain; there's more than one flavor of flow. For regex, it's the watching for single character match with optional modifier. For SQL, it's thinking in terms of sets, not instructions.
I'm one of those people, SQL just doesn't fit in my brain right :(
Nothing logical ever does. The human brain is a pattern recognition engine. You just haven't had enough time with these particular patterns. It's like spoken language; it just takes time, exposure, and practice.
Just find the right strategy for you to internalize the patterns. Maybe a game?
Named regexes (variables in Pomsky) remind me of Raku [1], which implements an improved flavor of PCRE regexes plus grammars in general as part of the language.
The grammars as part of the language is probably the most interesting thing about Raku to me. Never seen that in another language as a core concept like that.
These tools go back to research by Lauri Karttunen and others at
Xerox Research Center Europe in Grenoble, where an attempt was made
to create highly efficient compilers and runtime libraries for finite-state
transducers, i.e. to move beyond regular expressions to regular RELATIONS.
This not only permits to formalize replacements (regular expressions with an
"output tape"), but also creates reversible automata (input and output roles can be swapped) and leads to a domain specific language that describes transducers in very readable ways, including sub-automata naming, so that it can be useful for formal specification or linguistic rules (phonology, morphology, i.e. word or sound grammar). The latter two projects are open source clones of the former
effort. Once you have used these for a week, you will never want to get back to ugly "ordinary" regexes again.
Matching email addresses with any regular expression is fraught with errors. It can be done, depending on which RFC you are checking against, but in the real world it will eventually cause a problem.
You're better off using an actual parser (which will probably implement a state machine), writing a state machine yourself, or being overly accepting of invalid email addresses and just relying on attempting to deliver an email to the address.
It's probably better than regular expressions. However, is it enough better that it's worth learning yet another syntax?
Well, maybe. What I REALLY like about this one, is that it fully reverses the quoting/escaping assumptions. The default assumption of old regexes that symbols should by default match themselves, I think is regexes' million dollar mistake. The literal matches are the least interesting part of regexes. If you reach for regexes, it's because you want something more complex than literal matches, and the syntax should be about taming that complexity. Even the Unix world sort of conceded that, in reversing the quoting assumption for ?, +, (), [] and | in egrep. The mistake was in stopping there.
I will take a good look at this. I hope they provide good justifications for their choices.
I think the number ranges functionality is pretty handy. In addition, I also like how it supports comments. (However, I do think if the regex grows to needing comments, it is probably too complex.)
Yes and editor search is one of the few reasons to use regex these days. If you're writing something more permanent then you should use a real parser library!
Or use a 21st-century language that has a parser framework in the standard library ;)
Of course you can't always use a 3rd-party library, but IMO it should be your default choice without additional constraints. Much like how a memory-safe garbage-collected language should be your default choice for most things.
Can you be explicit about what you’re saying? What are you talking about exactly? Do you have some examples and arguments as to why it’s better than regexps in your opinion?
Perl regex syntax does what you describe, and has done so since the 1990s. Only alphanumeric characters are guaranteed to match themselves, and the others have a special meaning unless escaped. Some nonalphanumeric characters don't have a special meaning and you can use them either with or without a backslash, but the man page has always warned that it's safer to escape them anyway.
Vim regex is even nicer in this regard because it has four different modes, ranging from "everything is a literal except \\" to "everything is an operator except 0-9A-Za-z". This makes it great for searching code with symbols in it, because you can set it to the "very no-magic" mode and type your symbols as-is, or switch to "very magic" mode for a pattern that uses a lot of control flow.
Still, regex remains a great choice for one-off interactive purposes because it's concise, but for an application something with better composability and a clearer delineation between operators and literals is beneficial. You don't replace all uses of regex with Pomsky, you replace it specifically in "apps" where the pattern is part of the code.
I recently discovered this. It sounds horrible to have four different modes (unless they are exact emulations of existing other implementations, that would be a useful feature).
Annoyingly you can’t set “very magic” only “magic”. So I have written some vimscript to try and ensure when I start a pattern it starts with “\v” though I’d hesitate to call it robust.
It is in Julia, but if you have it installed locally it’s just a few taps away. You can even generate the regex, and use that in Python and just add the ReadableRegex in a comment nearby.
That reads better than regex, but at that point, why not just use parser combinators? At least for me, whenever I want something more complex than a basic regex, I go for a parser combinator library. Maybe that's what's happening under the hood anyways? I don't know Julia well enough to know.
Would be nice if it was possible for the Pomsky playground to show informative modal boxes when you hover over some of the Pomsky style expressions. Kind of like RegExr which remains my favorite tool for quickly writing out a regular expression and seeing what its outcome will be. Very nice to be able to quickly see what some thing does and how it affects your query including documentation in an easily accessible location on the same page.
Not having to navigate to a different page would be a boon for the playground. Very interesting though, I use Regular Expressions for the times when you need to extract information but find and replace functions/methods just aren't enough. Mostly for my scrapers.
It works nicely with Scheme, including for programmatic generation.
(I've done some regular expressions from heck, without the benefit of this, and needed extensive commenting just to keep a few/several levels of nested groupings straight. With S-expressions, that's trivial.)
That doesn't solve the most problematic part of regexing for me: having a different programming language embedded as a string in my other source code. The same happens with SQL in many codebases.
The embedded string is often precluded from static analysis. Fieldnames or names of capture groups may drift apart from the outside code and maybe nobody notices.
What we need are more embedded DSLs where we get static syntax and type checks and hopefully a good integration between the surrounding code and the embedded code.
>problematic part of regexing for me: having a different programming language embedded as a string
While there is nothing stating programming languages need to be Turing-Complete (and there are in fact examples of non-TC programming languages, if only educational), I think calling regular expressions* a "programming language" may be a bit of a stretch.
* Even when its capacities are as extended outside the mathematical definition as modern RegEx is.
edit: specified I meant "non-TC programming languages"
> and there are in fact examples of non-TC languages, if only educational
Sure, SQL is only educational or not programming.
There seems to be a lot of programming going on in the world generating a lot of value which you wouldn't recognize as such. Excel uses a terminating (and therefore not computationally complete) dataflow-based evaluation model but it's in no case programming. Are you sure?
I'd think, telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me. Very weird where you draw the line.
> Sure, SQL is only educational or not programming.
> There seems to be a lot of programming going on in the world generating a lot of value which you wouldn't recognize as such. Excel uses a terminating (and therefore not computationally complete) dataflow-based evaluation model but it's in no case programming. Are you sure?
I think this is not the most charitable reading of your parent. You're taking it as if they definitively rejected SQL and Excel as programming languages and arguing from that, rather than that maybe they didn't think of them. I think it would have been easy to make the case for SQL, Excel, and even (ir)regexes as programming languages without taking a sarcastic tone. For example:
> I'd think, telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me. Very weird where you draw the line.
could easily have asked instead, where do you draw the line?—which would surely be a more interesting discussion.
>Sure, SQL is only educational or not programming.
Last time I checked SQL was proven to be Turing-Complete?
While Excel's formula language wasn't always TC, the program as a whole, I believe, was since the very beginning.
>telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me.
I think by adding "if X then replace" capability we already escaped outside the bounds of Type-3 grammar, i.e. it's not a regular expression anymore (despite being build upon one).
> Last time I checked SQL was proven to be Turing-Complete?
Without recursive CTEs, it isn't. And there's nothing much to prove about that. Finite SQL statements on finite databases always terminate and it's therefore not computationally complete.
I very much think that's a great approach. Though the outside language kinda needs to be static as well for this to be really useful. For SQL I'm impressed with Jooq on Java. The DSL created, barring some shenanigans, doesn't allow you to create a malformed SQL statement that also makes the Java code compile and vice versa. For me, that's the goal.
I had a similar kind of idea for a long time, which I put into action a few weeks ago via a standalone transpiler of Emacs' rx macro to common regexp syntaxes.[0] I ended up getting interrupted and didn't completely finish it, but it generally works, though is probably riddled with edge cases.
The basic idea of rx is to use S-expressions to describe regular expressions, and my elevator pitch would've been to embed rx invocations in shell scripts using $(syntax), the main use case being something like sed invocations.
I still think it's a neat idea, and complex regular expressions tend to be hard to parse for humans.
While I'm not sure about Pomsky specifically, I do think its nice that people explore the language space for regex more. General programming languages have huge variety of styles and syntaxes available, from APL to Haskell and Lisp, whereas regexes are pretty much the same everywhere. It feels like we stuck with the first thing that Kleene, Thompson et al thought of and for 50 years didn't even really try anything else.
156 comments
[ 2.6 ms ] story [ 213 ms ] threadThat said, after looking at this syntax, I'm not sure that this is much of an improvement. Maybe I've spent far too much time in Regex land [1], but I know I'd perform much slower in this. It's not particularly beautiful, either. The verbosity doesn't seem clearer.
Variables and comments are great, though. We need to add them in future regexes.
Overall, good idea. I'd like to see more takes on this.
[1] https://jimbly.github.io/regex-crossword/
Every text editor, shell environment, programming language or desktop application seems to use regular expressions with a syntax which is slightly different to all the others, but not different enough to call it with a different name.
This means that a newbie learning regular expressions will be thrown into an environment where it can learn the basic principles, but the rules it learns are not generalisable to all applications (e.g. do I match an arbitrary string with '*', or '.*' ? Can I reuse matched patterns with (1) or with {1}? Etc.)
A new readable and easy-to-learn syntax that is nevertheless portable may work like markdown, as a simple-yet-universal new way to apply regexp that newcomers may learn with confidence and apply everywhere, replacing all the previous slightly incompatible versions.
And although I don't know for sure, reusing a matched pattern within the regex string using (1) seems like it would be very strange, because () are metacharacters for grouping.
That's what I meant about every system using a different syntax for what's basically the same functionality.
Interesting idea but the syntax adds nothing to the readability of regex.
I’d be more impressed with this if it targeted multiple different regex targets, since not all implementations of regex are equal. But in its current state it has all the problems of regex plus now the problems of this new language on top.
Looks a fun personal project though. Hope the developers enjoyed building it.
pcre has always had java and scala support.
Would you include Ruby in your comparison given that it inherits its regex implementation largely from Perl despite Onigurama's slight differences?
If I know RegEx, why would I use pomsky?
Insignificant whitespace is needed to support it, and as an added bonus it would make it easier to break up patterns across multiple lines.
The syntax changes and added verbosity do not seem great, though. They'd trip me up for sure.
In general, I think I'd like to see a language more like "Regex 2.0", ie. an extension that doesn't depart too far from what we're used to.
+1 to the syntax
Way back somewhere in the 2010s when I was still keeping an eye on Perl 6, I was kind of hoping that all these improvements would make their way into some kind of PCRE v3 that all other tools that already use PCRE would switch to. Would have been nice.
[1] https://docs.raku.org/language/regexes#Alternation:_||
The greatest problem with regexp is not its complexity nor learning curve: it's its apparent complexity & learning curve. That tends to put learners off, but anyone I know who's persisted has found the journey faster than expected. After they're over it, everyone I've talked to thinks pretty highly of the syntax: it's simple (small set of compostable components) & expressive, especially if using labelled groups.
The second greatest problem with Regexp is learning query optimisation: that's definitively complex but this new syntax doesn't even attempt solving that.
This might do better there.
Hello *waves*
The only thing regexp has on the plus side is that it's compact. I'd prefer something that's more readable without knowledge. I know how to use it but I always have to google it. I only use it once per year and so there's no big win in mesmerizing it.
Character classes can often be given as ranges:
You can use the pipe | to combine then.Lazy matching finds the shortest matching string instead of the longest one. This is given by the symbols:
Instead of + and * respectively.> I only use it once per year
If you use a text editor, any kind of repetitive find+replace activity can be sped up using regexes. Seems strange you only use it once per year.
Some people tell you that you can't parse XML using regexes. THEY'RE WR- right, but you can often do decent one-off jobs on an XML document using regular expressions, as long as you're aware that it's a one-off job that won't work on any other document.
^(?=.\d)(?=.[a-z])(?=.[A-Z])(?=.[!@#$^&()_-]).{8,18}$
That is hard to quickly* read and understand what it is. Sure a comment would help, but if someone said it isn't working, I have to work all that out and then fix it.
There is a reason Perl lost to Python. It just seems like we can do better.
I've certainly seen regexes that remind me of compost heaps.
(I agree that regex syntax is fine. I just liked the typo.)
You can of course disqualify me in saying I can't possibly have persisted at learning it, if I don't like it. But I am inevitably "the guy who knows regular expressions" in the places I've worked. I do use them, for stuff like fixing tedious one-off data munging jobs without having to write an actual program to do it. But I
* Do it step by step, to avoid writing anything resembling a complex regex, because I've yet to meet anyone who can write a complex regexp without bugs.
* Use "undo" a lot in that process.
Query optimisation is totally unnecessary if you use Rust's or Go's implementation.
I'm a big believer that "premature optimisation" is a bad thing but making a blanket statement that any given implementation is 100% performant for all use-cases is going a bit far.
Say that the length of your string is n, and the length of your regular expression is m. Isn't Regular Expression matching in those languages guaranteed O(m n) time? In plain English, that's only linear time - which you practically can't improve on - even if you tried! It can only do worse if your implementation uses backtracking instead of Finite Automata - which was very common at one time, but is now considered The Wrong Way: https://swtch.com/~rsc/regexp/regexp1.html
There is actually a way to optimise your queries, which is by precompiling them to DFAs, but I don't think that's what you're doing.
> There is actually a way to optimise your queries, which is by precompiling them to DFAs, but I don't think that's what you're doing.
You can do that, but the primary advantage of DFAs in practice is backtracking avoidance, so simply reduction of backtracking in manually constructed queries is more what I was referring to. The missing part of the puzzle here being that not supporting backtracking at all doesn't limit capabilities, so removal rather than reduction is the go/rust approach.
Complex password validation: https://gist.github.com/pmarreck/4c5f1076498da1a86062
Email header parsing: https://gist.github.com/pmarreck/8476538
An attempt at a JSON validator (yeah, I know): https://gist.github.com/pmarreck/2775709
Remember that commenting is not just for others, but also for "future you"!
It is for the next person that maintains the code.
Im not so sure this solves the actual issue ;)!!!! Jokes aside, regex has its uses.
[:0-255:] could be an option for example to write a number range.
Also regex / variable interpolation should be added to regexes in languages probably :)
> Also regex / variable interpolation should be added to regexes in languages probably
I've often wondered and desired this myself. It feels like the single most useful change one might make, because it enables (albeit very basic) logical decomposition. It comes in handy when a regex works for your use case, but the regex is somewhat long but with many repeated parts. In my experience, that isn't too rare, and the regex readability would likely be helped quite a bit by some kind of decomposition.
The string concatenations also make things a bit slower, but that almost always needs to be done only once.
Advantage is that this makes testing easier. For example, you can do
and then test that regex in isolation. That probably is overkill for this example, but if you change the regex to only accept numbers in [1,31], that can be useful.If you decide to have your regex to treat 30-day and 31 day months separately, things get more complicated. I wouldn’t go that far, though, and certainly not as far as handling leap years correctly. When regexes get complex, it’s time to go for a proper lexer/parser.
Still doesn't change what I said.
My original comment was talking about adding variables to the regex language itself. String interpolation and combinator APIs and enums and type safety are all beside the point. They surely solve a similar problem as regex variables in certain contexts, but they go about it in a very different way and are specific to the context in which the regex is created.
https://github.com/search?l=Rust&p=5&q=%22regex%3A%3Anew%22&...
For the importance of numeric range it was quite easy to find actually a good example:
x0rz3q/advent-of-code-2020
Regex::new(r"^19[2-8][0-9]|199[0-9]|200[0-2]$").unwrap(), ); comparitor.insert("iyr", Regex::new(r"^(201[0-9]|2020)$").unwrap());
There are lots of number parsing.
I would enable both [[:1-12:]] and [[:01-12:]] as options without / with leading zeros.
About the variables:
This file would look much more readable with variables that are reusing other regexes:
https://github.com/spcan/common-regex/blob/3238bc8ee85e0e000...
> If you know RegExp's, the syntax will immediately make sense
Doesn't seem like a feature.
Example: https://github.com/VerbalExpressions/JSVerbalExpressions#tes...
The first type of thinking is generally not composed in code. It happens staring out the window, or looking at data tables or lists and reading documentation, drawing out the architecture. The second type happens in code composition: programmers know what snippet will accomplish which outcome, and it just comes out when we know it's needed. Boom, we have the thing we imagined during the architecture phase. Maybe we look something up or slow down while writing code, but the composition is fairly reactive/spontaneous. It flows.
So coding goes in this rhythm... thinking openly and patiently and deliberately, followed by a burst of code composition, and then more open/deliberate thought while staring out a window. Again and again. Maybe we reverse some of the quick/spontaneous thinking to rearchitect according to our slow/deliberate thought.
Enter regular expressions. I have never, ever spontaneously composed a regex. I have to be very slow and deliberate. I have to recall the precise definition of every character in the regex as I read or write it. Sometimes I stare at a regex for 2 minutes, look up some operator, stare again for 2 minutes, refactor it. Finally I understand it and can now apply the regex spontaneously to whatever environment I'm working in. It always feels like first principles architecture, never like flow. This is odd, as regex is like a giant shorthand engine.
Maybe I have not yet formed the neurological connections required for it to flow – but more likely this is a function of the language. There must be a syntax that better supports flow-state composition. (I don't think Pomsky accomplishes it.)
That syntax was devised as a math notation, where sub-patterns were abstracted as one letter variables. You didn't have to deal with complex patterns at all, and it was very readable for that use case.
It was then re-purposed as a write-only language for searching at the CLI (in `ed` and its descendants, then grep).
They gradually graduated into what they are today in general programming languages because they were familiar and worse is better... However, composition got lost in the process.
Swift takes a radically different approach, and gives RegExps a parser combinator syntax. So every sub-expression can be assigned to a variable and reused/tested independently.
Here's an example (sorry I don't have it at hand in text form): https://pbs.twimg.com/media/FipgjUdUcAAl1xS?format=jpg&name=...
I wrote a lib that does the same in JS (the lib predates the Swift syntax by years). Here's the same example ported to JS: https://flems.io/#0=N4IgtglgJlA2CmIBcBWAzAOgAwCYA0IAzgMYBOA9...
Here's the lib: https://github.com/compose-regexp/compose-regexp.js
This hasn't been the case because I had to work in other languages where usage of regexes was more idiomatic/commonly understood. Parser combinators are great but most of the time, the problem can easily be solved using a regex and we can't really justify adding yet another library in the dependencies for the rare case it would be better suited.
In addition to all the benefits they can provide in terms or readability, re-usability and abstraction, parser combinators can even compete with regexes in terms of performance. https://pl-rants.net/posts/regexes-and-combinators/
I can do simple regexes at pretty close to the speed of thought. e.g. here's one to recognize the "arthurofbabylon 20 minutes ago" bit at the top of HN comments:
I typed that out pretty much all in one go, just had to look up different ways the HN date thing gets formatted. I put it in an online checker and it seemed to work on some examples, just had to fix a mismatched parentheses pair.Not saying it's completely correct (probably someone will reply with a counterexample or problem with it) but I think it's a demonstration that regex fluency is possible. I do this kind of thing on the command line a lot, interactively to figure stuff out from logs and so on. For production code I'm more circumspect.
Trouble comes when one need to update or debug a complex RegExp. The one you posted is not that complex.
The equivalent code to parse this is much longer and arguably to me not really easier to grok. (Unless of course you regularly avoid regex, in which case it will always remain hard.)
I'd say the hardest thing for beginners is the character class and how it matches EXACTLY ONE LETTER without a modifier. Everything matches exactly one letter without a modifier. It seems intuitive to me, but obviously a lot of folks have trouble with it, so I accept that we are outliers.
Reminds me a bit of folks who steadfastly avoid SQL. You have to change the flow in your brain; there's more than one flavor of flow. For regex, it's the watching for single character match with optional modifier. For SQL, it's thinking in terms of sets, not instructions.
If regexes make you feel stupid, lean into it instead of recoiling from it. That's where the learning happens: immersion into the uncertainty until it becomes familiar.
I'm one of those people, SQL just doesn't fit in my brain right :(
Just find the right strategy for you to internalize the patterns. Maybe a game?
https://mystery.knightlab.com/walkthrough.html
https://news.ycombinator.com/item?id=31690878 (242 points | 6 months ago | 189 comments)
[1]: https://docs.raku.org/language/grammars
1. - Xerox xfst
2. - Xerox lexc
3. - Xerox twolc
4. - Mans Hulden's FOMA
Repo: https://fomafst.github.io/
Paper: https://dingo.sbs.arizona.edu/~mhulden/hulden_foma_2009.pdf
Demo: https://dsacl3-2018.github.io/xfst-demo/
Tutorial: https://foma.sourceforge.net/lrec2010/
5. Helsinki HfstXfst:
Homepage: https://github.com/hfst/hfst/wiki/HfstXfst
These tools go back to research by Lauri Karttunen and others at Xerox Research Center Europe in Grenoble, where an attempt was made to create highly efficient compilers and runtime libraries for finite-state transducers, i.e. to move beyond regular expressions to regular RELATIONS. This not only permits to formalize replacements (regular expressions with an "output tape"), but also creates reversible automata (input and output roles can be swapped) and leads to a domain specific language that describes transducers in very readable ways, including sub-automata naming, so that it can be useful for formal specification or linguistic rules (phonology, morphology, i.e. word or sound grammar). The latter two projects are open source clones of the former effort. Once you have used these for a week, you will never want to get back to ugly "ordinary" regexes again.
Books:
(a) https://www.amazon.co.uk/Finite-State-Processing-Synthesis-L...
(b) https://www.amazon.co.uk/Recognition-Algorithms-Finite-State...
(c) https://www.amazon.co.uk/Finite-State-Techniques-Transducers...
(d) https://www.amazon.co.uk/Finite-state-Language-Processing-Sp...
(e) https://www.amazon.co.uk/Finite-State-Morphology-CSLI-Comput...
You're better off using an actual parser (which will probably implement a state machine), writing a state machine yourself, or being overly accepting of invalid email addresses and just relying on attempting to deliver an email to the address.
See http://cubicspot.blogspot.com/2012/06/correct-way-to-validat... for more discussion.
Well, maybe. What I REALLY like about this one, is that it fully reverses the quoting/escaping assumptions. The default assumption of old regexes that symbols should by default match themselves, I think is regexes' million dollar mistake. The literal matches are the least interesting part of regexes. If you reach for regexes, it's because you want something more complex than literal matches, and the syntax should be about taming that complexity. Even the Unix world sort of conceded that, in reversing the quoting assumption for ?, +, (), [] and | in egrep. The mistake was in stopping there.
I will take a good look at this. I hope they provide good justifications for their choices.
Also, learning regex syntax has much better ROI and is a much more transferable skill, than learning a specific parser library’s syntax and bugs.
Just invest 30 minutes once in your life into learning basic regex, for Christ’s sake, it’s not that hard!
Of course you can't always use a 3rd-party library, but IMO it should be your default choice without additional constraints. Much like how a memory-safe garbage-collected language should be your default choice for most things.
It's a quick toggle. Or you can just wrap your literal in quotes.
Still, regex remains a great choice for one-off interactive purposes because it's concise, but for an application something with better composability and a clearer delineation between operators and literals is beneficial. You don't replace all uses of regex with Pomsky, you replace it specifically in "apps" where the pattern is part of the code.
https://github.com/jkrumbiegel/ReadableRegex.jl
It is in Julia, but if you have it installed locally it’s just a few taps away. You can even generate the regex, and use that in Python and just add the ReadableRegex in a comment nearby.
Are we transpiling to regex?
Not having to navigate to a different page would be a boon for the playground. Very interesting though, I use Regular Expressions for the times when you need to extract information but find and replace functions/methods just aren't enough. Mostly for my scrapers.
https://scsh.net/docu/html/man-Z-H-7.html
https://www.ccs.neu.edu/home/shivers/papers/sre.txt
It works nicely with Scheme, including for programmatic generation.
(I've done some regular expressions from heck, without the benefit of this, and needed extensive commenting just to keep a few/several levels of nested groupings straight. With S-expressions, that's trivial.)
The embedded string is often precluded from static analysis. Fieldnames or names of capture groups may drift apart from the outside code and maybe nobody notices.
What we need are more embedded DSLs where we get static syntax and type checks and hopefully a good integration between the surrounding code and the embedded code.
While there is nothing stating programming languages need to be Turing-Complete (and there are in fact examples of non-TC programming languages, if only educational), I think calling regular expressions* a "programming language" may be a bit of a stretch.
* Even when its capacities are as extended outside the mathematical definition as modern RegEx is.
edit: specified I meant "non-TC programming languages"
Sure, SQL is only educational or not programming.
There seems to be a lot of programming going on in the world generating a lot of value which you wouldn't recognize as such. Excel uses a terminating (and therefore not computationally complete) dataflow-based evaluation model but it's in no case programming. Are you sure?
I'd think, telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me. Very weird where you draw the line.
> There seems to be a lot of programming going on in the world generating a lot of value which you wouldn't recognize as such. Excel uses a terminating (and therefore not computationally complete) dataflow-based evaluation model but it's in no case programming. Are you sure?
I think this is not the most charitable reading of your parent. You're taking it as if they definitively rejected SQL and Excel as programming languages and arguing from that, rather than that maybe they didn't think of them. I think it would have been easy to make the case for SQL, Excel, and even (ir)regexes as programming languages without taking a sarcastic tone. For example:
> I'd think, telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me. Very weird where you draw the line.
could easily have asked instead, where do you draw the line?—which would surely be a more interesting discussion.
The commenter made that quite clear. Using a computational model that's not complete can not be called "programming" in any real-world context.
Which is weird. This kinda implies that, whenever I forbid myself to write my Python-Code without While-loops or recursion, I'm not programming.
The traditional definitions of programming are often along the lines of instructing a machine to perform certain tasks.
Last time I checked SQL was proven to be Turing-Complete?
While Excel's formula language wasn't always TC, the program as a whole, I believe, was since the very beginning.
>telling a computer "if you recognize a pattern described as such, replace it with so and so", just regex with replacement, sounds very much like programming to me.
I think by adding "if X then replace" capability we already escaped outside the bounds of Type-3 grammar, i.e. it's not a regular expression anymore (despite being build upon one).
Without recursive CTEs, it isn't. And there's nothing much to prove about that. Finite SQL statements on finite databases always terminate and it's therefore not computationally complete.
A comment further up offered a very promising alternative.
https://github.com/VerbalExpressions/JSVerbalExpressions#tes...
It's a bit verbose, but I don't care anymore, I am too much a veteran to care about my code being sleek, I want it readable and workable.
The basic idea of rx is to use S-expressions to describe regular expressions, and my elevator pitch would've been to embed rx invocations in shell scripts using $(syntax), the main use case being something like sed invocations.
I still think it's a neat idea, and complex regular expressions tend to be hard to parse for humans.
[0]: https://github.com/sulami/rx