Alternatively, don't validate and then use the original. Instead, pull out the acceptable input and use that.
Even better, compare that to the original and fail validation if they're not identical, but that requires maintaining a higher level of paranoia than may be reasonable to expect.
Heh. I wrote up my comment, and then thought "hey, I bet that's what that 'Parse don't validate' article meant, the one I never quite got around to reading." So I pulled it up — great article! — but then didn't post the link because it uses the type system to record the results of the parse. Whereas here, you'd probably parse from a string into another string.
But philosophically I agree, that's exactly the relevant advice.
parsing from a string to a string runs the risk of erroneously assigning the original value to the new string. Which kinda defeats the whole parsing, not validating.
What would work is having a small object holding a readonly string which parses the original on creation, then becomes immutable.
EDIT: this is documented behavior for Ruby. What other languages call multiline mode is the default; you're supposed to use \A and \Z instead. They do have an `/m` but it only affects the interpretation of `.`
Yeah, my takeaway from this was more "the dangers of Ruby" rather than "the dangers of single line regular expressions" (:
I think the simplest fix would be to use "\Z" rather than "$", which means "match end of input" rather than "end of line." This is also Perl-compatible. So weird that the "$" default meaning is different in Ruby.
I guess one could argue that Ruby's way is better since "$" has a fixed meaning, rather than being context-dependent.
> Ruby seems to be in multiline mode all the time?
Ruby does have a "/m" for multiline mode, but it just makes "." match newline, rather than changing the meaning of "$", it seems.
The potential trouble with $ (even in single-line mode) is that it matches the end of a string BOTH with AND without a newline at the end. If you're using it to ensure the string has no newline before doing something with it, this can lead to trouble.
$ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo") else "no")'
yes
$ python3 -c 'import re; print("yes" if re.search(r"^foo$", "foo\n") else "no")'
yes
$ python3 -c 'import re; print("yes" if re.search(r"\Afoo\Z", "foo") else "no")'
yes
$ python3 -c 'import re; print("yes" if re.search(r"\Afoo\Z", "foo\n") else "no")'
no
Even if the newline is not problematic, using \A and \Z makes your intentions clearer to the reader, especially if you add re.X and place comments into the pattern.
Asides:
1. Based on syntax, you appear to be testing with python2.
2. With python, re.match is implicitly anchored to the start, so the ^ is redundant. Use re.search or omit the ^.
Correct me if I'm wrong, but if you extract a capture group (^foo$), you would get "foo" without the "\n", right?
If so, it is not "matching the end of a string" at all. Just end of line. That's exactly as expected in single-line mode, so it's good. May mismatch your expectations in multi-line mode though.
That's right. It all depends on what you're doing with the input string after the match. The point is to be aware of the nuance and to communicate that clearly in the code in cases where it matters.
False. "$" does NOT mean end-of-string in Perl, Python, PHP, Ruby, Java, or .NET. In particular, a trailing newline (at least) is accepted in those languages.
A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go.
I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned.
If anyone knows of vulnerabilities caused by thus, let me know.
`$` does mean end of input in Java, unless you explicitly ask for multiline mode. In the latter case it means `(?=$|\n)` if also in Unix-lines mode, and the horrible `(?=$|(?<!\r)\n|[\r\u0085\u2028\u2029])` otherwise.
I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1].
Edit: also quoting from [2]:
> By default, the regular expressions ^ and $ ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence. If MULTILINE mode is activated then ^ matches at the beginning of input and after any line terminator except at the end of input. When in MULTILINE mode $ matches just before a line terminator or the end of the input sequence.
OK it seems they changed the doc since. In the docs for JDK 21 we read instead [1]:
> If MULTILINE mode is not activated, the regular expression ^ ignores line terminators and only matches at the beginning of the entire input sequence. The regular expression $ matches at the end of the entire input sequence, but also matches just before the last line terminator if this is not followed by any other input character. Other line terminators are ignored, including the last one if it is followed by other input characters.
Interesting that a trailing newline is accepted. Not as bad as what's in the post, at least. Definitely worth breaking out which languages do which of those, though! Python, for instance, only accepts a trailing newline but not additional chars beyond that.
I don't think Java should be in your first list, though? Pattern.matches("^foo$", "foo\n") returns false.
Which version of Java (JDK) are you using? Which implementation?
If that's true, then I fear the answer for Java may vary. The O'Reilly book on Regular Expressions, and the JDK documentation for version 21, say clearly that $ permits an optional \n at the end. The Java 8 documentation is murky, and maybe Java 8 is different.
I'm not sure which version of PHP had the behavior you describe, or whether it misbehaves under more specific conditions, but preg_match() is one of the more commonly-used regex functions, all of which share the same engine. The behavior here seems to be "correct" for at least the last 5 years, for varying interpretations of "correct".
edit: https://3v4l.org/N4o8D suggests that the behavior here is identical for all versions of PHP from 4.3 to 8.3.6.
So, you are technically correct when you say PHP accepts a trailing newline, but it doesn't mean it refutes the comment and the context we are discussing.
Which all makes sense, as by default PHP doesn't operate in multiline mode. So, by default, PHP is not going to fall prey to the same problem being discussed here. In addition, the first \n would be apart of the first line it's on, so including it as a part of the string would make sense. More to the point, in this context, $ does mean end of the string in PHP. You can prove otherwise by getting the 2nd and 3rd example above to output a 1 instead of a 0 without going into multiline mode.
That is clear proof that "$" does NOT just match the end of the string; it also accepts an extra newline at the end of the string. In PHP you need to use \z if you want to match the end of the string, or use the "D" flag when using "$".
That definition of "$" is often reasonable when you read files a line-at-a-time from a file, which is why Perl changed its definition. However, PHP is often used for server-side web applications. In this case, you are often NOT reading a line-at-a-time from a file. In such cases, allowing an extra newline at the end could be disastrous. The MediaWiki code (written in PHP) deals with this by adding the "D" flag when it uses "$", but I'm not sure it always uses it, and I doubt all PHP programs use this flag when they should.
You want \Z in Python, and \z in most other languages, to match on end of string. But in some languages $ really does match end of string. As always, you must check your docs.
This was interesting and new to me, but as other commenters indicate, part of the problem is that we're trying to find the bad thing rather than trying to verify it is the good thing
There's a related concept of "failing open vs failing closed" (fail open: fire exit, fail closed: ranch gate)
In Jurassic park (amazing book/film to understand system failures), when the power goes out, the fence is functionally an open gate
In this case, we shouldn't assume that we can enumerate all possible bad strings (even with a regex)
I don't think this is a good example, because the regex does just that: it doesn't try to filter out bad input, it specifically only accepts known good input. If the regex did what it was meant to do, only allowing strings composed of ascii letters and numbers, and space, than the code would have not been exploitable.
Still seems like that is broken. Shouldn't they be escaping whatever control characters? Like if your user wanted to highlight "Now 75% off". Seems like it is reasonable to want to allow that.
Yeah but the real bug is the trying to roll-your-own, instead of using a `ERB.escape_tainted_input` method or somesuch. Either that method doesn't exist, which seems like major mis-feature, or the author didn't know about it, or didn't want to use it.
I pretty much always consider regex expressions as the wrong solution. They're notoriously hard to get right.
There's a whole lot of faulty expressions out there for validating email addresses. I prefer to do less validation and let it fail. If the email address is wrong, whatever service you're using for sending emails will just reject it. If you really do need to validate email addresses, use something somebody else wrote that does it properly.
Here, you have good advice: "I ... consider regex expressions as the wrong solution. They're notoriously hard to get right."
However, the conclusion of "use something somebody else wrote that does it properly", while valid, is asking a lot. As regex is hard to get right, don't assume the code you find on the web or book or via some other means works correctly.
My rule is if I didn't write it and can't wrap my head around the code to convince myself it is the right solution, I don't use it. And as I think others have written, there are some interactive online tests for regex expressions that can help.
I think the average developer has a better chance of finding a robust, battle tested library to do what they need than cooking up some regex of their own. Preferably, the library does not use regex at all and checks data more intelligently.
Sometimes valid email addresses will be rejected as invalid, and sometimes invalid email addresses are still successfully delivered. Validation guarantees nothing, and at most it should be a UI cue.
Regex works very well for what it was originally designed: describing/validating regular languages. It can work ok if your language is simple and almost regular. They work very badly for validating non-regulars languages, even when extensions are added Perl-style to support that. And, unfortunately, most structured formats you might care to valdiate are in fact not regular languages at all.
Email addresses in particular are surprisingly complicated and far from being regular languages. I don't know how commonly real servers support the full feature set, but even if they just support non-ascii names they quickly become a pain.
132 comments
[ 4.3 ms ] story [ 199 ms ] threadThe bigger problem here is executing user input.
But I guess this is why Python has so many ways of matching a pattern against a string (match, find, findall, I think - they are hard to remember)
Even better, compare that to the original and fail validation if they're not identical, but that requires maintaining a higher level of paranoia than may be reasonable to expect.
Parse don't validate https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
But philosophically I agree, that's exactly the relevant advice.
What would work is having a small object holding a readonly string which parses the original on creation, then becomes immutable.
Ruby seems to be in multiline mode all the time?
EDIT: this is documented behavior for Ruby. What other languages call multiline mode is the default; you're supposed to use \A and \Z instead. They do have an `/m` but it only affects the interpretation of `.`https://docs.ruby-lang.org/en/master/Regexp.html#class-Regex...
I think the simplest fix would be to use "\Z" rather than "$", which means "match end of input" rather than "end of line." This is also Perl-compatible. So weird that the "$" default meaning is different in Ruby.
I guess one could argue that Ruby's way is better since "$" has a fixed meaning, rather than being context-dependent.
> Ruby seems to be in multiline mode all the time?
Ruby does have a "/m" for multiline mode, but it just makes "." match newline, rather than changing the meaning of "$", it seems.
[1] https://ruby-doc.org/3.2.2/Regexp.html#class-Regexp-label-An...
[2] https://perldoc.perl.org/perlre#Metacharacters
if !m=/^[a-z0-9 ]+$/match(str) return "Bad Input" end str=m[0]
Asides:
1. Based on syntax, you appear to be testing with python2.
2. With python, re.match is implicitly anchored to the start, so the ^ is redundant. Use re.search or omit the ^.
If so, it is not "matching the end of a string" at all. Just end of line. That's exactly as expected in single-line mode, so it's good. May mismatch your expectations in multi-line mode though.
A $ does mean end-of-string in Javascript, POSIX, Rust (if using its usual package), and Go.
I'm working with the OpenSSF best practices working group to create some guidance on this stuff. It's a very common misconception. Stay tuned.
If anyone knows of vulnerabilities caused by thus, let me know.
I wrote a compiler from Java regex to JavaScript RegExp, in which you'll find that particular compilation scheme [1].
Edit: also quoting from [2]:
> By default, the regular expressions ^ and $ ignore line terminators and only match at the beginning and the end, respectively, of the entire input sequence. If MULTILINE mode is activated then ^ matches at the beginning of input and after any line terminator except at the end of input. When in MULTILINE mode $ matches just before a line terminator or the end of the input sequence.
[1] https://github.com/scala-js/scala-js/blob/eb160f1ef113794999...
[2] https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pa...
> If MULTILINE mode is not activated, the regular expression ^ ignores line terminators and only matches at the beginning of the entire input sequence. The regular expression $ matches at the end of the entire input sequence, but also matches just before the last line terminator if this is not followed by any other input character. Other line terminators are ignored, including the last one if it is followed by other input characters.
Looks like I have some code to fix.
[1] https://docs.oracle.com/en%2Fjava%2Fjavase%2F21%2Fdocs%2Fapi...
I don't think Java should be in your first list, though? Pattern.matches("^foo$", "foo\n") returns false.
If that's true, then I fear the answer for Java may vary. The O'Reilly book on Regular Expressions, and the JDK documentation for version 21, say clearly that $ permits an optional \n at the end. The Java 8 documentation is murky, and maybe Java 8 is different.
edit: https://3v4l.org/N4o8D suggests that the behavior here is identical for all versions of PHP from 4.3 to 8.3.6.
This is easily demonstrated with an example.
versus versus Which all makes sense, as by default PHP doesn't operate in multiline mode. So, by default, PHP is not going to fall prey to the same problem being discussed here. In addition, the first \n would be apart of the first line it's on, so including it as a part of the string would make sense. More to the point, in this context, $ does mean end of the string in PHP. You can prove otherwise by getting the 2nd and 3rd example above to output a 1 instead of a 0 without going into multiline mode.In PHP, the following is considered true:
> var_dump(preg_match("/^[a-z0-9 ]+\$/", "hello\n"));
That is clear proof that "$" does NOT just match the end of the string; it also accepts an extra newline at the end of the string. In PHP you need to use \z if you want to match the end of the string, or use the "D" flag when using "$".
That definition of "$" is often reasonable when you read files a line-at-a-time from a file, which is why Perl changed its definition. However, PHP is often used for server-side web applications. In this case, you are often NOT reading a line-at-a-time from a file. In such cases, allowing an extra newline at the end could be disastrous. The MediaWiki code (written in PHP) deals with this by adding the "D" flag when it uses "$", but I'm not sure it always uses it, and I doubt all PHP programs use this flag when they should.
(\Z allows a trailing newline, \z does not)
There's a related concept of "failing open vs failing closed" (fail open: fire exit, fail closed: ranch gate)
In Jurassic park (amazing book/film to understand system failures), when the power goes out, the fence is functionally an open gate
In this case, we shouldn't assume that we can enumerate all possible bad strings (even with a regex)
There's a whole lot of faulty expressions out there for validating email addresses. I prefer to do less validation and let it fail. If the email address is wrong, whatever service you're using for sending emails will just reject it. If you really do need to validate email addresses, use something somebody else wrote that does it properly.
If you're working with some exotic format for which there isn't already an open source library, do what this guy says: parse it, don't try to validate it with regex: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
However, the conclusion of "use something somebody else wrote that does it properly", while valid, is asking a lot. As regex is hard to get right, don't assume the code you find on the web or book or via some other means works correctly.
My rule is if I didn't write it and can't wrap my head around the code to convince myself it is the right solution, I don't use it. And as I think others have written, there are some interactive online tests for regex expressions that can help.
Email addresses in particular are surprisingly complicated and far from being regular languages. I don't know how commonly real servers support the full feature set, but even if they just support non-ascii names they quickly become a pain.