The rules aren’t that hard but actually applying it to code and honing it to consistently pull exactly what you want is in my experience the hardest part.
So, you can observe what kind of state machine is produced from any given Regular Expression. You can also use it to merge and such manipulate state machines, or simplify Regular Expressions.
Practice, the more you use them the easier they become. I never studied them but knew when to use them, then just tinkered and iterated until the pattern did what I needed it to. After a while you can mostly just write and read them without much tinkering.
I read through Learning Perl and practiced different regex patterns with simple programs. Programming Perl dove deeper, as did the OReilly Mastering Regular Expressions book.
That and practice. I frequently check them with online regex tools to make sure the regex does what I want before I implement them.
That's my go-to these days, but sometimes I like to see a diagram from this one: https://regexper.com
I've just slowly learnt it by experimenting with it over the past few years. People have mostly mentioned matching, but I use it more for string manipulation.
I'm still not as intermediate a programmer as I'd like to be, so it's great when I need to invert a design decision for example. A similar code structure in multiple places, maybe across multiple files. It also means I don't miss anything, like I would if I did it manually.
For me the biggest hurdle was learning what they were 'for' and that took a long time. The real magic for me was capture groups - I could now suddenly see why you'd have a regex and not just string matching.
Then it was about knowing a situation or a problem when regexes would apply and knowing how to look up the things I needed to solve that problem. Some regex 'phrases' are good for grepping, others for find and replace. Some will help you swap names around, some to reformat phone numbers.
After a while the phrases give way to general understanding and certain things become fluent.
I still only really write short or basic regexes, but I use them all the time in editing text or doing things that are a little bit complicated but actually a short regex just turns it from a hard problem into an easy problem.
We took the academic approach in college, Kleene closures, Chomsky grammars etc. Once the meaning behind the line noise of symbols was clear it become fairly easy to write them.
Mainly in a job that forced me to use them (data mining). There are interactive tools that help you visualize what is going on, they are immensely helpful.
What I do now is I write a comment like this in my IDE:
# a regex that selects <something>
regex =
And then copilot/supermaven auto-completes it. If that doesn't work, I ask GPT-4o/Sonnet. If that doesn't work, I assume that whatever I'm asking for is not really a natural fit for regex and I should accomplish my task in a different way.
In general I try not to use regex in production code. IMO it is an obsolete technology at this point. Most people do not know it well and trying to debug it is a nightmare. May I suggest a simple function or loop that is readable?
One of the things I remember being difficult at the beginning was the subtle differences between implementations, like `^` meaning "beginning of line" in Ruby (and others) but meaning "beginning of string" in JavaScript (and others).
If you're just starting out, it'd be helpful to read about how a regex engine evaluates an expression against a string so that you can understand the "order of operations" and how repeating elements are matched.
Learned regex in the 90's from the Perl documentation, or possibly one of the oreilly perl references. That was a time where printed language references were more convenient than searching the internet. Perl still includes a shell component for accessing it's documentation, that was invaluable in those ancient times. Perl's regex documentation is rather fantastic.
A simple way to test a regex you're building is this website, which offers immediate parsing and documentation of your regex, lets you test it against various inputs, and lets you choose which language's regex parser you are targeting.
First day on the job as tech support at a local dialup ISP/CLEC My manager gave me a regex cheatsheet, because we had a gigantic multi-file spam filter that we would all need to troubleshoot, and some of us were allowed to edit. It was fun.
Cheat sheets are the way to go though, especially because of the different versions. If you do enough with them one day the main stuff will just stick. Once you're fairly productive you will realize you missed a feature or trick that is particularly useful for what you've been doing, and after getting mad at yourself for missing it you will add that into your repertoire. repeat.
Also, don't be afraid to just split/cut and do it in your language of choice instead of regex. Most of the time it doesn't make too much of a difference performance wise. Many times it can be faster and/or more readable. The best approach is often a combination. Nobody likes the wizard that tries to put everything into one regex to rule them all.
Regarding versions, I learned with PCRE, have mostly worked with python, and have hit problems using other various implementations over the years. Though it's never enough of a problem that I can remember what those differences are, I just look it up and move on. Unless it's going to be an ongoing project, in which case I print out a new cheatsheet and hang it up.
I read Mastering Regular Expressions by Jeffrey Friedl 20 years ago when I was in middle school, front to back, and it's probably been the best investment of reading time I've ever made.
If I remember correctly, the local public library was selling books off for ultra cheap for some reason, and I added it to the pile of books my parents bought to fill the empty built-in bookshelves in our new family home (along with a bunch of reader's digest anthologies). I scanned through the first part of it and it seemed super powerful (I was just getting into programming at the time), so it captivated me.
I had read on some tech site, some years ago, that Friedl worked at Yahoo for a while, and IIRC in a role involving a lot of text munging, which would probably have involved a lot of regular expression usage, maybe across the many web properties they had in that period, which included Yahoo Search, Yahoo Mail, Yahoo Groups, Yahoo Finance, and many more.
Found that interesting.
I had bought his book around that time or sometime later, but never read it fully, partly because I used to go cross-eyed from reading the text with all the italics and other highlighting (of the regexes in action) in a small font, which was probably needed to explain regex concepts, but still ...
but after reading this thread, I feel motivated to dive into regex again, at least at the shallow end of the pool, although I have dabbled in it and used it now and then in my work, before now.
I mean, the highlighting was maybe useful, but having it in the small font made it unnecessarily difficult to read. squiggly italics of say, one or two characters in length, are harder to distinguish from non-italic characters, when in quite a small font.
If you have no experience, go through a tutorial to get the general idea.
Learn the rest "on demand" whenever you need it, it's not something to spend a lot of practice time on. Because if you don't use it a lot, you'll forget most of what you learned anyway, and if you do use it a lot, then you don't need to spend dedicated learning time, you'll get good quite quickly.
Start with https://regexone.com/ fun puzzle style interactive tutorial to grasp the basics.
After that it's the matter of either using it with your CLI tools or applying it to problems you are working on.
I learned it in my "Programming Languages" class in university.
And then about 6 months later I had completely forgotten it.
It's one of those things you need to use regularly to keep it in memory. At least that's the case for me.
I tend to shy away from it these days for a lot of cases (ever try to regex validate an email??) but when I do use it I it's honestly just a process of re-learning for about 15 minutes each time.
Develop useful tools that need complex parsing, I gathered data from websites using regex to make automated tools, for example getting flat offers from my city and making a tool to notify me.
Simply try to parse some complex information like movie strings, as an exercise you can try to parse these movie names to produce a result like this.
```
{
"name": "Dawn Of The Planet of The Apes",
"year": "2014",
"resolution": "1080p",
"codec": "h264",
"source": "web-dl",
"audio": "AAC5.1",
"group": "RARBG"
}
```
Piece by piece, googling "how to do X in regex". But that was slow and didn't have a great foundation.
Then I learned Perl and started learning RegEx properly. Now somehow I've turned into one of those wizards I admired in the Stack overflow answers section. It wasn't until I had to teach RegEx to a junior that I realized how far I'd come.
102 comments
[ 3.5 ms ] story [ 92.7 ms ] threadhttps://github.com/qntm/greenery
So, you can observe what kind of state machine is produced from any given Regular Expression. You can also use it to merge and such manipulate state machines, or simplify Regular Expressions.
Quite helpful.
Regex101 is an excellent tool.
That and practice. I frequently check them with online regex tools to make sure the regex does what I want before I implement them.
Super intuitive and great definitions / descriptions of everything.
I've just slowly learnt it by experimenting with it over the past few years. People have mostly mentioned matching, but I use it more for string manipulation.
I'm still not as intermediate a programmer as I'd like to be, so it's great when I need to invert a design decision for example. A similar code structure in multiple places, maybe across multiple files. It also means I don't miss anything, like I would if I did it manually.
Then it was about knowing a situation or a problem when regexes would apply and knowing how to look up the things I needed to solve that problem. Some regex 'phrases' are good for grepping, others for find and replace. Some will help you swap names around, some to reformat phone numbers.
After a while the phrases give way to general understanding and certain things become fluent.
I still only really write short or basic regexes, but I use them all the time in editing text or doing things that are a little bit complicated but actually a short regex just turns it from a hard problem into an easy problem.
Not sure they ever get easier to read though.
In general I try not to use regex in production code. IMO it is an obsolete technology at this point. Most people do not know it well and trying to debug it is a nightmare. May I suggest a simple function or loop that is readable?
One of the things I remember being difficult at the beginning was the subtle differences between implementations, like `^` meaning "beginning of line" in Ruby (and others) but meaning "beginning of string" in JavaScript (and others).
If you're just starting out, it'd be helpful to read about how a regex engine evaluates an expression against a string so that you can understand the "order of operations" and how repeating elements are matched.
`perldoc perlre` from your terminal.
or https://perldoc.perl.org/perlre
A simple way to test a regex you're building is this website, which offers immediate parsing and documentation of your regex, lets you test it against various inputs, and lets you choose which language's regex parser you are targeting.
https://regexr.com/
It's been many years but I remember it as both thorough and easy to understand.
and
https://blog.stevenlevithan.com/
Cheat sheets are the way to go though, especially because of the different versions. If you do enough with them one day the main stuff will just stick. Once you're fairly productive you will realize you missed a feature or trick that is particularly useful for what you've been doing, and after getting mad at yourself for missing it you will add that into your repertoire. repeat.
Also, don't be afraid to just split/cut and do it in your language of choice instead of regex. Most of the time it doesn't make too much of a difference performance wise. Many times it can be faster and/or more readable. The best approach is often a combination. Nobody likes the wizard that tries to put everything into one regex to rule them all.
Regarding versions, I learned with PCRE, have mostly worked with python, and have hit problems using other various implementations over the years. Though it's never enough of a problem that I can remember what those differences are, I just look it up and move on. Unless it's going to be an ongoing project, in which case I print out a new cheatsheet and hang it up.
Regex is one of those things that need a solid base understanding, or you'll always hit things that will baffle and frustrate you as a programmer.
Using split/cut is fine for a small subset of what regex can do, but I often hear that advice from people who also hate regex.
If the goal is to like regex, read 'Mastering Regular Expressions'.
That being said - regex is a superpower.
Found that interesting.
I had bought his book around that time or sometime later, but never read it fully, partly because I used to go cross-eyed from reading the text with all the italics and other highlighting (of the regexes in action) in a small font, which was probably needed to explain regex concepts, but still ...
but after reading this thread, I feel motivated to dive into regex again, at least at the shallow end of the pool, although I have dabbled in it and used it now and then in my work, before now.
Learn the rest "on demand" whenever you need it, it's not something to spend a lot of practice time on. Because if you don't use it a lot, you'll forget most of what you learned anyway, and if you do use it a lot, then you don't need to spend dedicated learning time, you'll get good quite quickly.
And then about 6 months later I had completely forgotten it.
It's one of those things you need to use regularly to keep it in memory. At least that's the case for me.
I tend to shy away from it these days for a lot of cases (ever try to regex validate an email??) but when I do use it I it's honestly just a process of re-learning for about 15 minutes each time.
Simply try to parse some complex information like movie strings, as an exercise you can try to parse these movie names to produce a result like this.
``` { "name": "Dawn Of The Planet of The Apes", "year": "2014", "resolution": "1080p", "codec": "h264", "source": "web-dl", "audio": "AAC5.1", "group": "RARBG" } ```
https://raw.githubusercontent.com/dobladov/video-parser/main...
Then I learned Perl and started learning RegEx properly. Now somehow I've turned into one of those wizards I admired in the Stack overflow answers section. It wasn't until I had to teach RegEx to a junior that I realized how far I'd come.
Later, grepping logs was a pretty similar application that needed and extended those skills.