Regex is the only real sensible way to validate strings until something better is found. Even if you just wrote code to do it manually, you'd really just be writing a verbose and poorly implemented finite state machine that globbed symbols together, which in the end, would just be inferior to writing a well tested Regex string.
Regex can be easier to read if you have something do a graphical expansion for you. Otherwise, it's write-once, read-never.
That's just silly. Depending on what's in the string, writing a parser might be much better than a regex. Lots of parser libraries already out there, too.
I can agree with this sentiment, there's also nothing wrong with writing a parser that breaks the input up into pieces and validates those individual parts using specialized regular expressions.
True, though it's EBNF with a bunch of explanatory text and annotations.
What would be interesting, but I can't find with some googling: Has someone implemented a parser-generator based on the spec? The ideal would be that the parser specification looks a lot like the RFC, since then you'd have more confidence it was actually correct (and it'd be easier to maintain for future changes).
There was a long post on the topic of 'regex-vs-parser' for email on reddit a while back. I hope I'm pointing to the correct person, but he wrote a parser in Haskell that validates against RFC5322: http://hackage.haskell.org/package/email-validate
> Regex is the only real sensible way to validate strings until something better is found.
There are plenty of ‘better’ (in the sense of ‘more powerful’) string-validation techniques. For example, lots of grammars are expressed in BNF; the languages that can be so expressed are (if I remember my Chomsky hierarchy correctly) the context-free grammars, a strictly larger class than the regular languages. The extra power comes from the fact that they have the expressive power of a finite-state automaton augmented by an (infinite) stack. (It's fair to argue that it's not ‘really’ infinite, since a computer's memory is finite; but, in that sense, real-life computers will never be Turing complete.)
(Of course, common ‘regular-expression’ libraries aren't actually regular any more, because of added features like capture groups. I don't know if they recognise all CFG's, though; I suspect not.)
Given this, why would we use regular expressions? Well, by intentionally sacrificing power, we can achieve faster matching (http://swtch.com/~rsc/regexp/regexp1.html) and, probably, lower memory usage. Sometimes this trade-off is worth it, even if it means that the match must be somewhat fuzzy; but sometimes one needs a precise match, and regexes just aren't up to the job.
> Given this, why would we use regular expressions?
If I can get a way with a simple regexp I will use it. If it starts getting complicated I would use something else.(e.g. top down parser or a combination)
> If I can get a way with a simple regexp I will use it. If it starts getting complicated I would use something else.(e.g. top down parser or a combination)
If I remember correctly my theory of languages course a parser (implemented in a turing machine) is more powerful than a reggex (which is a finite state machine).
Merely looking up MX records is something you do for a _domain_ and it's no different than the DNS requests your computer makes when you go to a website.
practically speaking, most respectable email services will have an MX record. i tested it on about 30,000 emails & a simple MX lookup caught about 40 gmial's and only 1 false positive.
What's the deal with using a port number in an e-mail address? I can't imagine many systems supporting that. Anyone got any more info on it that doesn't involve me digging through 101 pages of RFCs? :-)
I don't think that's valid. I think they misread the RFC. First of all, an IP address (address-literal) is supposed to be enclosed in [square brackets]. The colon is only allowed to designate address family, as in "foo@[IPv6:XXXXXX]".
At the risk of sounding flip, not supporting punycode sounds like a feature. Getting internationalization right is clearly important, but punycode as a means of doing so? It's one of the many things that make me weep for my industry.
In fact, I often wonder if punycode is a prank that got out of hand.
UTF-8, on the other hand, would have been excellent for this purpose.
True. It doesn't. But as far as I know, there is no deep technical reason why it can't. The dots in domain names are '\0', and a few characters ('@', in particular) need to remain reserved, but other than that, what stops UTF-8 from being used?
Nothing is finished, nothing is permanent, and nothing is perfect.
In particular, one of the evaluation tests used here is wrong: it requires failure-to-match on a TLD with a digit in it:
numbersInTLD@domain.c0m
In fact, IDN TLDs will have digits in them. An internet-draft is in the works to replace RFC1123's IDN-unfriendly implication that digits in TLDs are illegal:
A nice try at pedantry, which I would usually respect, but in the domain of internet standards with which I familiar, you are wrong.
The existing specs are in conflict, with the more recent ones (such as IDN) allowing digits. Internet authorities, including ICANN, have enabled domains with digits in TLDs; software which is far more foundational than any web-app's email validation regex has been updated.
Registration of names in some of these digited-TLDs has begun; you can visit these TLDs with your browser; your users can have functioning email addresses on these TLDs.
If your app rejects such email addresses because of slavish compliance with imprecise language in a 31-year-old RFC, you'd be the one violating prevailing standards, which are a function of more than just formal IETF RFCs.
That the Internet-Draft I referenced may soon become an RFC is just cleaning up loose ends on a change that's already happened. This final step isn't even strictly necessary for the de facto standard to have changed by consensus among practitioners. Plenty of vibrant well-understood standards never reach RFC status, nor pass through any formal standards body. The standard is ultimately what people do, not what someone once-upon-a-time decreed.
I've accepted that it's best to treat people like grown-ups and if there's '@' and '.' and it's retyped then it passes. Someone can easily submit a fake name or phone number or street address, and e-mail's no different.
If they get it wrong, intentionally or not, then they don't get their receipt, confirmation, validation link, etc. and I believe in most cases the incentive is there for them to get it right.
In the rare case where there's some incentive to circumvent the system and this has some measurable impact on a site, then more validation may be warranted. Otherwise, why worry about it?
Along those lines, I've settled on the following overly permissive regex: /^[^\s@]+@[^\s@]+\.[^\s@]{2,}$/ -- it makes sure it looks something like an email address (a@b.cd)
interestingly i count 19 TLD's with mx records. but judging from your earlier post you know as well as i do (i use <single character>@w.tf for many purposes) that there are pretty much no web forms on the planet that would take such an address. tis a shame, but such addresses while pretty nifty are also frequently unusable. not to mention trying to explain to someone that that is really your email address and if it doesn't work they should talk to their ISP/mail client author/etc...well, suffice it to say they won't be getting in touch with you that way anyhow. (be that a blessing or curse, it's just reality)
RFC 5321 section 2.3.5 specifically prohibits TLDs from receiving email. Other RFCs back that up, actually including RFC5322 (not directly, but in wire format).
That said, we see some TLDs run MX's. I think at least some portion of them sell the mail received there to spammers. Seriously.
You obviously know a lot more on this topic... but where exactly does it say in that section that TLDs are prohibited from receiving emails?
From my reading, it seems to suggest the opposite, e.g. "In the case of a top-level domain used by itself in an email address, a single string is used without any dots."
I've re-read that section many times and forgive me if I missed it — it's late! =)
People have been posting conflicting responses throughout this thread suggesting that TLDs can/cannot receive emails, and I'd really like to know which should be valid... thanks!
You'd need something more along the lines of a spell checking algorithm to do that sanely. Don't abuse regular expressions for things they're not good at.
Yeah, that python script (from StackOverflow London - http://norvig.com/spell-correct.html) that learned what was correct from a large number of words would work. Analyse your DB of users' email address url (after @) that have registered fully and provide a "Did you mean.." type response if a new users email url was not found and something close (within 2 alterations) to it was.
True, but you have to balance that against the small but nonzero number of people put off by an extra text field. Plus, I would find email repetition more annoying if I didn't always do Cmd-A/Cmd-C/Tab/Cmd-V, and in this case the repeated field won't catch any errors.
The fact that you know the shortcuts for select all, copy, and paste puts you in the top percentile of users. Most people don't even know that's possible, and certainly not with keyboard shortcuts.
(The point being that for most users the faster approach that requires less thinking is to type it twice. Sometimes I do things the "slow" or "long" way when coding because it doesn't require a mental shift from the task at hand.)
That's true. My main point was the first one. The second point was mainly to illustrate that annoyance isn't hypothetical; I'm annoyed by an extra text box.
I agree they are annoying. I guess I'm just less annoyed than you are. Cmd-a Cmd-c Tab Cmd-v has a very negligible impact on the amount of keystrokes I make in a day. Typing my address again doesn't even make a real difference now that I think about it, 3 seconds or so.
Have you ever abandoned a sign-up because of the e-mail confirmation?
The point being that for most users the faster approach that requires less thinking is to type it twice.
Spoken like a touch-typist with a short email address. I feel confident in claiming that annabelle.t.johnson@woodandplaster.co.uk will be looking for that copypaste button. It's not like copypaste is a new technological innovation - it's practically the most used feature of personal computing, after backspace.
To let users revisit their email addresses without adding a second field for copy-pasting, Russ Unger of UserGlue and Jonathan Knoll found a nice solution:
My routine for dealing with a second e-mail field is shift+tab, ctrl+a, ctrl+c, tab, ctrl-v. Usually auto-fill takes care of the first fields anyway, never of the second one. It doesn't help me in any way.
Most forms asking for an email address use the same field name (or one of a limited number of field names), so the browser remembers what you typed last time, and it'll suggest it to you in a pop-up for autocompletion. All it takes is to type one letter, press down, and enter.
In which you are assuming you will receive a proper bounce, which is not warranted. Checking bounces is something you can do in addition to checking whether the user didn't screw up his email address.
My thoughts exactly. There are so many websites that tell me my valid email address is invalid that it's not even funny. These people then have to deal with phonecalls and lost business, because their form won't even submit without a valid email address (and why should I change if they're the ones that suck).
BTW, the email address that doesn't work is "jon-whatever@jrock.us". The .us confuses people and the - confuses people. WTF!?
I've seen code written in India that disallowed any email address containing one or more digits. This was on a site accepting email addresses from the general public, as well.
The have a lot more problems to deal with if they have to deal with all the people that include a stray space in the address.
There are regexes out there that catch all valid addresses that people reasonably use (including those with dashes and ending in .us) and the fact that all kinds of incompetent developers use a homebrewn regex is no reason to lower the best case scenario to "just don't validate". Just do it right.
Your recommendation is almost as wrong as the regexp in the article, as both will reject the perfectly valid postmaster@ai address (check it out, "dig -t mx ai" will return a result, so that address must exist). The only thing you can be sure about in a non-local email address is that it contains at least one @ sign. More are allowed (for source routing), but hopefully no server is still configured to honor that, so you might get away with requiring exactly one @. Everything else is evil.
Oh, and don't forget to make sure that one component in your spam^W email processing chain correctly encodes unicode charaters in the domain part into punycode.
There are more reasons to check for an email than simply validation. Perhaps you want to detect emails in a field and let people click/tap on them to send that person an email? Don't want to let people send email off to foo@bar.baz now, do we?
There are many other uses for this regex than registration systems. We have at least two situations where a staff member would type in a customer mail address, and some get it wrong. Confirming won't help, and getting it wrong costs lots of time to fix. Using a better regex costs very little.
If they get it wrong, intentionally or not, then they
don't get their receipt, confirmation, validation link,
etc. and I believe in most cases the incentive is there
for them to get it right.
Well, I hope you have a large enough support team, because with any reasonable amount of customer growth, you'll soon be swamped with support emails that say "I didn't get my ..., where is it? You suck!". Validating the email address (which includes checking for common typos in domains) is a service that reduces frustration and the amount of customer support needed.
Even if they've typed it correctly it still can be blocked for various reasons.
You will still need a support mechanism (doesn't have to get all the way to a person) to sort out undelivered verification mails. It's just how it goes.
Still, some rudimentary validation can prevent cases of someone entering the wrong value (e.g., just their username) or an incomplete value (e.g., scott.trudeau@gmail ) -- which I see with some regularity ~10k signups/month. If this lowers support email volume by even just 10%, that's not insignificant.
You assume an email regex is only used for validating addresses. It can also be used to parse text for email addresses (for formatting or mining purposes maybe).
It does not look like any of these can detect or were tested against quoted-local-part addresses. As I understand it, the local part can be quoted to allow illegal characters to be used, e.g. "John Doe"@example.com
I fully understand that these are not in common use, but they are part of the RFC and may be in use somewhere.
His test also considers the % sign invalid, yet I've both sent and received email that required the % for the mail to be routed/relayed correctly (think "in care of" or c/o).
Granted, that was 17 years ago, but who's to say it's not in use somewhere?
It thinks baddomain@300.0.0.1 is valid, and thinks decimaldomain@2130706433 (localhost) is not valid.
It also thought user@3ffe:1900:4545:3:200:f8ff:fe21:67cf as invalid.
I never understood the purposes of the "perfectly valid by the RFC" email regex. You may be able to with 100% accuracy say that something should be an email address, but you'll never be able to tell if it's a valid account on the server, or if the server even exists.
As a typical optimizer, I was always trying to reduce my source code a few more bytes or speed up my processes a few nanaseconds here and there. I was so proud of myself. Until I tried to maintain my own slickness.
There's a fine line between clever and practical. Why do I have a gut feeling that this approach is way over that line?
So many web services do this wrong, that it isn't even worth doing it right: no one is going to complain that your service doesn't accept their "wacky! quoted"@email.address
Sometimes, you aren't validating a whole string, you are searching for email addresses in a sea of text, or an arbitrarily delimited, user-entered list of contacts.
Support usernames with alphanumerics, dashes, underscores, periods, and plus signs; Require a single @; Support domains with alphanumerics, dashes, and at least one period. Screw anyone with something more complex than that. Done deal.
That is part of Lepl - http://www.acooke.org/lepl/ - and although it's implemented in a recursive decent parser, much is compiled to regular expressions for efficiency. So you get the best of all worlds: regexp efficiency; parser accuracy; standards based.
Then they DDoS the login form with a long password and email, and between the md5 and the retarded regex, your cpu is choking. You still need to limit action by ip and add some kind of a captcha system. People can just invite themselves to gmail a million times. Email verification is not so useful anymore.
I wrote an article "Validating Email Address in Web Forms - The Hazards of Complexity" that discusses several comprehensive email validation libraries and common problems with complex methods of validation.
AFAIK a naked IP address as domain like IPInsteadOfDomain@127.0.0.1 doesn't work (at least in some mail servers like Postfix they are bounced with a "bad address syntax error). The address has to be like IPInsteadOfDomain@[127.0.0.1] and I don't see this case covered.
117 comments
[ 2.9 ms ] story [ 152 ms ] threadRegex can be easier to read if you have something do a graphical expansion for you. Otherwise, it's write-once, read-never.
What would be interesting, but I can't find with some googling: Has someone implemented a parser-generator based on the spec? The ideal would be that the parser specification looks a lot like the RFC, since then you'd have more confidence it was actually correct (and it'd be easier to maintain for future changes).
While you do that, I'll do something that doesn't involve tweezers and code.
There are plenty of ‘better’ (in the sense of ‘more powerful’) string-validation techniques. For example, lots of grammars are expressed in BNF; the languages that can be so expressed are (if I remember my Chomsky hierarchy correctly) the context-free grammars, a strictly larger class than the regular languages. The extra power comes from the fact that they have the expressive power of a finite-state automaton augmented by an (infinite) stack. (It's fair to argue that it's not ‘really’ infinite, since a computer's memory is finite; but, in that sense, real-life computers will never be Turing complete.)
(Of course, common ‘regular-expression’ libraries aren't actually regular any more, because of added features like capture groups. I don't know if they recognise all CFG's, though; I suspect not.)
Given this, why would we use regular expressions? Well, by intentionally sacrificing power, we can achieve faster matching (http://swtch.com/~rsc/regexp/regexp1.html) and, probably, lower memory usage. Sometimes this trade-off is worth it, even if it means that the match must be somewhat fuzzy; but sometimes one needs a precise match, and regexes just aren't up to the job.
Isn't that what I said?
http://lua-users.org/lists/lua-l/2009-10/msg00817.html
Possibly this approach is too slow to use by itself without a regex. Also maybe there are other problems with this method I'm not aware of?
The best way to validate an email remains to be a test email apparently.
Merely looking up MX records is something you do for a _domain_ and it's no different than the DNS requests your computer makes when you go to a website.
http://tools.ietf.org/html/rfc5321#section-5.1
Good luck trying to register on any site with it though :)
Users of http://موقع.وزارة-الاتصالات.مصر won't be pleased :)
In fact, I often wonder if punycode is a prank that got out of hand.
UTF-8, on the other hand, would have been excellent for this purpose.
In particular, one of the evaluation tests used here is wrong: it requires failure-to-match on a TLD with a digit in it:
numbersInTLD@domain.c0m
In fact, IDN TLDs will have digits in them. An internet-draft is in the works to replace RFC1123's IDN-unfriendly implication that digits in TLDs are illegal:
http://tools.ietf.org/html/draft-liman-tld-names-02
The existing specs are in conflict, with the more recent ones (such as IDN) allowing digits. Internet authorities, including ICANN, have enabled domains with digits in TLDs; software which is far more foundational than any web-app's email validation regex has been updated.
Registration of names in some of these digited-TLDs has begun; you can visit these TLDs with your browser; your users can have functioning email addresses on these TLDs.
If your app rejects such email addresses because of slavish compliance with imprecise language in a 31-year-old RFC, you'd be the one violating prevailing standards, which are a function of more than just formal IETF RFCs.
That the Internet-Draft I referenced may soon become an RFC is just cleaning up loose ends on a change that's already happened. This final step isn't even strictly necessary for the de facto standard to have changed by consensus among practitioners. Plenty of vibrant well-understood standards never reach RFC status, nor pass through any formal standards body. The standard is ultimately what people do, not what someone once-upon-a-time decreed.
If they get it wrong, intentionally or not, then they don't get their receipt, confirmation, validation link, etc. and I believe in most cases the incentive is there for them to get it right.
In the rare case where there's some incentive to circumvent the system and this has some measurable impact on a site, then more validation may be warranted. Otherwise, why worry about it?
Also: HTML5. ;)
http://tools.ietf.org/html/rfc5321#section-2.3.5
RFC 5321 section 2.3.5 specifically prohibits TLDs from receiving email. Other RFCs back that up, actually including RFC5322 (not directly, but in wire format).
That said, we see some TLDs run MX's. I think at least some portion of them sell the mail received there to spammers. Seriously.
You obviously know a lot more on this topic... but where exactly does it say in that section that TLDs are prohibited from receiving emails?
From my reading, it seems to suggest the opposite, e.g. "In the case of a top-level domain used by itself in an email address, a single string is used without any dots."
I've re-read that section many times and forgive me if I missed it — it's late! =)
People have been posting conflicting responses throughout this thread suggesting that TLDs can/cannot receive emails, and I'd really like to know which should be valid... thanks!
"@"@example.com
?
Retyping only makes sense for password field, which is obfuscated and doesn't allow copy&paste.
Multiply that by say, 130,000 people, and you are dealing with 325 people who don't receive their download, etc. and are not happy!
I think what would be really awesome is a regex that catches these common typos and warns the user immediately.
(The point being that for most users the faster approach that requires less thinking is to type it twice. Sometimes I do things the "slow" or "long" way when coding because it doesn't require a mental shift from the task at hand.)
Have you ever abandoned a sign-up because of the e-mail confirmation?
Spoken like a touch-typist with a short email address. I feel confident in claiming that annabelle.t.johnson@woodandplaster.co.uk will be looking for that copypaste button. It's not like copypaste is a new technological innovation - it's practically the most used feature of personal computing, after backspace.
http://blog.mozilla.com/faaborg/2010/03/23/visualizing-usage...
Display the email address on the submit button: http://www.userglue.com/blog/2009/09/09/solving-the-repeat-e...
Also try out the live examples: http://infinityplusone.com/experiments/email-repeat/version5
Regexes take care of the syntax. The semantics still have to be checked by a human.
Not a lot of chance of mistyping, there;
1) Send an email to the address.
2) If it succeeds, the address was valid.
BTW, the email address that doesn't work is "jon-whatever@jrock.us". The .us confuses people and the - confuses people. WTF!?
There are regexes out there that catch all valid addresses that people reasonably use (including those with dashes and ending in .us) and the fact that all kinds of incompetent developers use a homebrewn regex is no reason to lower the best case scenario to "just don't validate". Just do it right.
Oh, and don't forget to make sure that one component in your spam^W email processing chain correctly encodes unicode charaters in the domain part into punycode.
You will still need a support mechanism (doesn't have to get all the way to a person) to sort out undelivered verification mails. It's just how it goes.
I fully understand that these are not in common use, but they are part of the RFC and may be in use somewhere.
Granted, that was 17 years ago, but who's to say it's not in use somewhere?
But we know a valid email address when we see it, right?
--
Edit: Nicely done, HTML5.
There's a fine line between clever and practical. Why do I have a gut feeling that this approach is way over that line?
I don't get this preoccupation with making sure addresses look valid. The ONLY way to validate an email address is to send it a message.
http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html
Sometimes, you aren't validating a whole string, you are searching for email addresses in a sea of text, or an arbitrarily delimited, user-entered list of contacts.
Support usernames with alphanumerics, dashes, underscores, periods, and plus signs; Require a single @; Support domains with alphanumerics, dashes, and at least one period. Screw anyone with something more complex than that. Done deal.
It's 6,598 bytes long.
That is part of Lepl - http://www.acooke.org/lepl/ - and although it's implemented in a recursive decent parser, much is compiled to regular expressions for efficiency. So you get the best of all worlds: regexp efficiency; parser accuracy; standards based.
A blog post on the compilation to regexps is here - http://www.acooke.org/cute/LEPLOptimi0.html
http://www.messagingnews.com/onmessage/ben-gross/validating-...
(I know.)