Only if you're doing a lot of string parsing. Just like learning about algorithms on graphs is a must only if you're working with graphs of a non-trivial size.
I don't agree. Regular expressions come in handy as well during your day to day work: you can save a lot of time if you know how to use them in your editor "search & replace" function, or just when a simple search in a directory gives you too many results and you need to switch to power tools (read: egrep).
I was thinking about this question and it occurred to me. The programmers who probably don't need to know regexes[1] almost certainly already learned them. So I guess that's a yes.
[1] Thinking of embedded systems developers creating code for, say, automotive entertainment systems, or control code for scientific or medical hardware.
[1] Thinking of embedded systems developers creating code for, say, automotive entertainment systems, or control code for scientific or medical hardware.
> Regular expressions are a tool. It happens to be a
> very useful tool, so many people choose to learn how
> to use it. However, there's no "requirement" for
> you to learn how to use this particular tool, any
> more than there is a "requirement" for you to learn
> anything else.
Nails it. I do think most programmers will eventually run across a problem to which the solution
is 'Use Regex', but it's not an absolute "must" like boolean logic.
good point indeed. I spent my first years programming DSP algorythms on embedded systems. Hardly any strings used, let alone there was any need for regular expressions. Learning their ins and outs back then would have been a waste of time, and would have been like a bricklayer having a fork in his toolbag.
Yes, it is, without a doubt. It's one of the most universal tricks of the trade that you'll literally never regret learning, mainly because just about any environment you'll ever work in will by necessity support regexes, and in many, it will be the primary way you interact with text.
But it's also a must that you realize that despite the fact that they're exceptionally useful and widely supported, regexes are a disgusting abomination, one that we should be absolutely mortified to be associated with. It's one of the worst syntaxes to ever be invented, and every one of us should feel the cold stink of UX failure wash over us every time we write a regex. If we ever catch ourselves writing a DSL that in any way, shape, or form resembles regular expressions, we should stop immediately and ask what the fuck is wrong with us and why we're being so opaque and random. Regular expressions are quite literally one of the worst syntaxes to ever be introduced in our field.
I worry a lot when someone doesn't know regular expressions at all. But I worry far more when someone thinks they're beautiful. That person has far too high a tolerance for unintuitive syntax and code, and will cause vastly more damage to my codebase than even the rank amateur that still uses "goto" on a regular basis.
Which is not to say that we shouldn't lean on regexes heavily anyways when they're appropriate - as programmers our primary job description is that we're paid good money to work with shitty interfaces in order to express simple ideas and algorithms.
>It's one of the worst syntaxes to ever be invented, and every one of us should feel the cold stink of UX failure wash over us every time we write a regex.
Okay, so they're ugly and hard to read at first glance. Considering the purpose of a regex, I can't think of another way to implement them that doesn't involve typing more characters needlessly (therefore even making it more hard to comprehend).
How would you design a regular expression syntax more intuitively? Personally, I find beauty and simplicity in regular expressions. Sure, they can grow to hideous atrocities, but you can achieve such disastrous feats with any language/syntax. Maybe you could back up your claim of regexes being a disgusting abomination with, at the very least, anecdotal evidence.
There's characters missing[1] and the tld is too short[2]. And even if that's fixed, we still don't match internationalized addresses or actually validate that the e-mail address exists. You're probably better off with something like...
if "@" in email and "." in email.split("@")[1]:
send_verification(email)
...but you should probably also check for common misspellings like "gmial.com" etc.
There's nothing wrong with regex syntax. But there _is_ something wrong with the formatting of your example: it's not readable. Perhaps you _should_ write (assuming Java-syntax):
(?x: #standard token is an uppercase letter, digit, dot, or hyphen
\b
[A-Z0-9._%-]+ #1 or more of standard token, underscore, or percent sign
@ #at-sign
[A-Z0-9.-]+ #1 or more standard tokens
\. #dot
[A-Z]{2,4} #2 to 4 uppercase letters
\b
)
We can easily optimize for readability with regex syntax.
You can say that about pretty much anything if you aren't familiar with the syntax. That looks pretty readable to me - certainly far more portable and readable than equivalent code.
[It's also perfectly obvious that if this is an attempt to match email addresses that it's not a very good one - but I don't know the context where it's supposed to be used, it might be good enough for whatever the author intended].
IMO, the main problem is that the syntax is too terse. The syntax is taken wholesale from the algebraic notation used in mathematics and dropped in unmodified. Essentially, regex code ('cause regexes really are code) is not skimmable.
The greatest syntactical atrocity in regexes is that they don't have the `x` modifier (in Perl parlance) on by default. This means that you can't use whitespace to chunk code into meaningful bits, nor can you comment it to easily document what does what or explain a particularly hairy section to handle some weird edge case. This means that regexes degenerate a lot faster than ordinary code in terms of readability.
Actually, regex syntax isn't primarily a Perl invention. Regexes as software tools go back to early Unix text processing tools (ed and grep, according to Wikipedia), and Perl took the syntax from those tools.
It is true that Perl reformed the syntax in important ways (to the better, if you ask me), and later on extended it a lot, but it's certainly not a Perl invention.
Hmm - you know I actually think it's not correct to look at regex as an interface - even though we use it as one. It's really more accurate to look at it as a grammar (type 3 if I remember correctly).
Anything that comes out of the whole chomskian hierarchy stuff isn't going to look intuitive. But the point is that it is a particular, very rigorously defined system of representation. And various systems of representation are always more or less intuitively accessible - and come with a whole set of trade offs around what they can represent vs their ease of use etc...
These things just are - written into the laws of the world. They are discovered - not invented. We pick them up and use them as we would rocks left lying around. We find better ones when we can and fashion better ones when we can too...
> Anything that comes out of the whole chomskian hierarchy stuff isn't going to look intuitive.
Why not? Type-0 (recursively enumerable) languages are equivalent to Turing machines, and we've managed to invent some pretty good syntaxes for that. The main problem with regexes really is the syntax. Regular languages are a lot easier to understand (IMO) if you look at the left/right-linear grammars that define the same language as the regex.
Regex syntax as a representation is very close to the FSA used for matching, and that's not necessarily the representation best suited for human consumption.
> These things just are - written into the laws of the world. They are discovered - not invented.
This applies only to the theoretical computer science regexes. The practical regexes are very different in this aspect.
Practical regexes are neither discovered nor invented - they are constructed. The theoretical regexes are just the basis but on top of that there's a lot of features added. Some of them are just syntactic sugar for theoretical regexes but others actually make the language non-regular.
Groups that match what previous named groups have matched are definitely in this category.
I think that regular expressions have a good syntax for most typical (small) uses of regular expressions. Maybe the choice of special characters isn't ideal, and it might be nice to have 'English' versions of more special characters (similar to e.g. [[:digit:]] in POSIX), but for small regexes the (mostly) one-to-one mapping between characters in the pattern and characters in the string is a very nice and intuitive syntax.
I think the real problem is that we lack (or don't learn) good tools to bridge the gap between regular expressions and 'custom parser'. We're reluctant to refactor from '1 line of just-starting-to-be-horrible regex' to tens or hundreds of lines (depending on language and libraries) to do it 'properly', and so we end up stretching regular expressions beyond the point where they make life easier.
Perl has Parse::RecDescent (and probably several others), which is pretty close to the right thing, and clearly it's very doable in a lot of languages - anyone got any suggestions in other languages?
I could imagine a Jquery-like DSL where something like /^[A-Z]+[0-9]{2}$/ could be expressed like Match(str).BeginsWith().BigCaseAlpha().AtLeastOne().Numeric().FixedLength(2).EndsWith()
Of course, you would also have to be able to nest these for more advanced matching...
Is there something like this already in existence? :)
I taught my friend how to use regular expressions for search and replace. He's found them invaluable and he's NOT a developer. He uses then to clean up and filter lists of keywords from Adwords.
I reach for refex frequently but almost never as something I add to the code I'm writing. They're just amazing for filtering and bulk editing text.
Yes! They're not that hard and there's only so much syntax to them (vs. an actual programming language). I essentially learned from this site by just entering in random strings and trying to match them. http://rubular.com/
Amen to that. For reasons I can only guess at, I'm the "regex guy" at work. When people have a regex question they come to me. And the interesting thing is that the answers are not hard, well within the 20% area. I've found that most people don't care to learn how to use character classes, or even what they're for. For example, to match the single letter 'i', I've seen people write:
[i]
Many don't know the difference nor implications of .* ? vs .*
And on and on. It amazes me that these people actually code for a living.
I regularly forget regex intricacies, so now have an app called "regexbuddy" for Windows. Very very useful, has a killer help file, has the ability to adjust and test regexes for many languages.
I find one the hardest aspects of using a regex is all the subtle differences between languages/environments. I use them about four times a year but I feel like I have go read a mini tutorial each time. I know the concepts but I can't remember the special char for whitespace or digits in the particular language I'm working in.
Also the java implementation is so bad. Having to encode it as a string with its own escaping rules is not easy. Then the 3 line api usage is a real hassle.
Regex is a good example for a bad interface. When usually half of the input needs to be escaped something must have gone fundamentally wrong. I regularly forget regular expressions and look them up again when I really need them.
At least is good to know what can be solved with regular expressions, then if you need to solve something you can always look it up in a book or use some software for it (I use RegexBuddy).
I think it's essential to know of their existence, but not necessarily to know their syntax inside-out. Basically, enough enough that you're not writing looping and parsing code yourself, using a regex where applicable.
There's no "a must." But I don't consider a programmer to be a good one if he doesn't know regexps.
If you ask "do you personally really need regexps" I'll tell you: don't learn them. As you're asking that question at all, I understand that you're not interested to learn and that you are looking for an excuse not to learn, so do something that interests you.
I think they're a valuable as a simple tool e.g. using :s/^#// in vim.
However, I try to avoid using them in my code unless they improve readability. Using re.VERBOSE can help in Python.
If you find a regex online, you should definitely reference it in your code, to help provide background understanding, such as validating a UK Post Code.
While knowledge how to use regular expression is invaluable, I also recommend learning how they actually work under the hood. It really gives a good lesson when regular expressions are applicable, and when they're not. From my experience, while many programmers are apt in tools like regexps or grammar->parser generators, they very rarely know how it actually works, which results in people trying to parse HTML with regexps or similar things. It is also a good starting point to some very interesting theoretical stuff like the theory of computations.
Also, you see people doing simplistic string comparisons using regexes. Which is ok sometimes but is an easy target if your system has performance issues.
The problem is that regexp is often misused as things like html parsers and e-mail validators. Learning regexp syntax without learning when and how to use it, makes you a worse programmer not a better one.
I think every programmer should have an understanding of what Regular Expressions are, how they can be used, and where to find a cheat sheet or reference (like www.regular-expressions.info).
It helps to know the basic syntax by memory, but you can just as easily look it up if you understand how they work.
You can get by as a programmer without knowing regexes. You can also get by without knowing about pointers. Heck, maybe you can even get by without knowing SQL or OOP.
But you will be limited. You will run into a lot of situations where your ignorance will create friction and limitations. Something as simple as editing an nginx configuration file, for example.
You should learn regexes, you should be able to use a regex as necessary and be able to understand regexes you come across, you should understand when they are most useful, and when they are not. You should learn their limits and your limits in using them as well. When used appropriately they can be a potent addition to your technical knowledge. When they should be used but are avoided the result is typically a massive inflation of the effort to solve a simple problem. And when used incorrectly they can result in an inflation of intractable complexity (much like any technology).
I think every programmer should know them, but should try to avoid using them unless they really are the best solution to your problem. Too many people reach for them too soon, leaving hard to understand and hard to maintain code that could have been better expressed some other way.
59 comments
[ 3.3 ms ] story [ 77.4 ms ] threadI would say it's a must to take your coding skills to the next level.
[1] Thinking of embedded systems developers creating code for, say, automotive entertainment systems, or control code for scientific or medical hardware.
Don't they have log files to read?
But the point is I can't imagine someone getting to that point without previously having learned it.
[1] http://programmers.stackexchange.com/questions/133968/is-it-...
But it's also a must that you realize that despite the fact that they're exceptionally useful and widely supported, regexes are a disgusting abomination, one that we should be absolutely mortified to be associated with. It's one of the worst syntaxes to ever be invented, and every one of us should feel the cold stink of UX failure wash over us every time we write a regex. If we ever catch ourselves writing a DSL that in any way, shape, or form resembles regular expressions, we should stop immediately and ask what the fuck is wrong with us and why we're being so opaque and random. Regular expressions are quite literally one of the worst syntaxes to ever be introduced in our field.
I worry a lot when someone doesn't know regular expressions at all. But I worry far more when someone thinks they're beautiful. That person has far too high a tolerance for unintuitive syntax and code, and will cause vastly more damage to my codebase than even the rank amateur that still uses "goto" on a regular basis.
Which is not to say that we shouldn't lean on regexes heavily anyways when they're appropriate - as programmers our primary job description is that we're paid good money to work with shitty interfaces in order to express simple ideas and algorithms.
Okay, so they're ugly and hard to read at first glance. Considering the purpose of a regex, I can't think of another way to implement them that doesn't involve typing more characters needlessly (therefore even making it more hard to comprehend).
[1] http://en.wikipedia.org/wiki/Email_address#Syntax [2] http://en.wikipedia.org/wiki/List_of_Internet_top-level_doma...
http://ex-parrot.com/~pdw/Mail-RFC822-Address.html
[It's also perfectly obvious that if this is an attempt to match email addresses that it's not a very good one - but I don't know the context where it's supposed to be used, it might be good enough for whatever the author intended].
The greatest syntactical atrocity in regexes is that they don't have the `x` modifier (in Perl parlance) on by default. This means that you can't use whitespace to chunk code into meaningful bits, nor can you comment it to easily document what does what or explain a particularly hairy section to handle some weird edge case. This means that regexes degenerate a lot faster than ordinary code in terms of readability.
Edit: Misplaced close paren.
I always wonder what regexes would look like if they were derived from Python instead.
It is true that Perl reformed the syntax in important ways (to the better, if you ask me), and later on extended it a lot, but it's certainly not a Perl invention.
Anything that comes out of the whole chomskian hierarchy stuff isn't going to look intuitive. But the point is that it is a particular, very rigorously defined system of representation. And various systems of representation are always more or less intuitively accessible - and come with a whole set of trade offs around what they can represent vs their ease of use etc...
These things just are - written into the laws of the world. They are discovered - not invented. We pick them up and use them as we would rocks left lying around. We find better ones when we can and fashion better ones when we can too...
Why not? Type-0 (recursively enumerable) languages are equivalent to Turing machines, and we've managed to invent some pretty good syntaxes for that. The main problem with regexes really is the syntax. Regular languages are a lot easier to understand (IMO) if you look at the left/right-linear grammars that define the same language as the regex.
Regex syntax as a representation is very close to the FSA used for matching, and that's not necessarily the representation best suited for human consumption.
This applies only to the theoretical computer science regexes. The practical regexes are very different in this aspect.
Practical regexes are neither discovered nor invented - they are constructed. The theoretical regexes are just the basis but on top of that there's a lot of features added. Some of them are just syntactic sugar for theoretical regexes but others actually make the language non-regular.
Groups that match what previous named groups have matched are definitely in this category.
I think the real problem is that we lack (or don't learn) good tools to bridge the gap between regular expressions and 'custom parser'. We're reluctant to refactor from '1 line of just-starting-to-be-horrible regex' to tens or hundreds of lines (depending on language and libraries) to do it 'properly', and so we end up stretching regular expressions beyond the point where they make life easier.
Perl has Parse::RecDescent (and probably several others), which is pretty close to the right thing, and clearly it's very doable in a lot of languages - anyone got any suggestions in other languages?
Of course, you would also have to be able to nest these for more advanced matching...
Is there something like this already in existence? :)
I reach for refex frequently but almost never as something I add to the code I'm writing. They're just amazing for filtering and bulk editing text.
And on and on. It amazes me that these people actually code for a living.
Surprisingly, here: http://www.regexbuddy.com/
The help file alone will make you want to buy it.
If you ask "do you personally really need regexps" I'll tell you: don't learn them. As you're asking that question at all, I understand that you're not interested to learn and that you are looking for an excuse not to learn, so do something that interests you.
However, I try to avoid using them in my code unless they improve readability. Using re.VERBOSE can help in Python.
If you find a regex online, you should definitely reference it in your code, to help provide background understanding, such as validating a UK Post Code.
It helps to know the basic syntax by memory, but you can just as easily look it up if you understand how they work.
But you will be limited. You will run into a lot of situations where your ignorance will create friction and limitations. Something as simple as editing an nginx configuration file, for example.
You should learn regexes, you should be able to use a regex as necessary and be able to understand regexes you come across, you should understand when they are most useful, and when they are not. You should learn their limits and your limits in using them as well. When used appropriately they can be a potent addition to your technical knowledge. When they should be used but are avoided the result is typically a massive inflation of the effort to solve a simple problem. And when used incorrectly they can result in an inflation of intractable complexity (much like any technology).
Also if you are familiar with this quote:
Read Jeffrey Friedl's research into the quote - http://regex.info/blog/2006-09-15/247