For me, the site rendered dark gray text on a dark gray background and is a chore to read as-is. Outline.com fixed my issue with it: https://outline.com/YSYgsp
I got curious and looked back in archive.org to this page's initial release in 2014. The text background started out as good old reliable background-color: #EEEEEE, which was later replaced with background: url("http://a.yu8.us/bg-tile-parch.gif")
...because what could possibly go wrong? From the latest comment at the end of the page, the author would like you to know that the outcome is your problem, because you're using the wrong browser:
June 20, 2021 - 15:02
Subject: RE: Undoing whatever is hiding this page.
Hi Allen, try a different browser. There's no strange shading on the page, your browser is deciding to display it in a weird way. Regards, -Rex
Most likely using the HTTPS Everywhere addon. That website is not available via HTTP, and the user must visit the page first to accept the 'risk' of using the http version.
Firefox also defaults to HTTPS by default nowadays. Lots of content blockers block third party content too. Regardless, if literally anything goes wrong with the third party dependency that the article's contrast depends on, the best case scenario here is that the text falls back on the body's background.
Interestingly, the author also appears to control yu8.us
Breaking one's own content by https-ing one site but not another is a great example of why to not prop up a website's basic legibility on a third party dependency, even if it's one you own and control.
Yes, they web author made the mistake of defining the <article> background-color: #EEEEEE within a min-width 960px media query. If the background image fails to load in wider window, there's still a readable contrast between text and background but on a phone or other narrow screen, the dark background color set on the <body> is what's behind the article text.
"Please don't complain about website formatting, back-button breakage, and similar annoyances. They're too common to be interesting. Exception: when the author is present. Then friendly feedback might be helpful."
(It's not that the annoyances aren't annoying, it's that they're so common that they lead to repetitive offtopicness that compounds into more boring threads.)
The ? syntax group has to be the most unmemorable of the bunch. I've used it maybe over 1,000 times or so and I still have to look up ?: Or ?! ?< or whatever else.
I used to have a laminated sheet on my wall at an office because it was so terribly bad.
I dunno, the "logic" solution seems like the obvious one to me; if your boss really has that much trouble with propositional logic that they don't immediately see why it works, well, that's what code comments are for.
(...the trick is still cool, though; I can imagine other situations where it would be more useful. However it does seem like it potentially depends on the particular regex engine being used, in contrast to the author's claim about it being totally portable; yes, it'll compile on anything, but will it work?)
How could it not work. I've regularly relied on order or matching, and never found an environment that didn't test left-to-right for the `|` operator in regex.
I'm talking about regex. Regex libraries in practical use do not use NFA. I'm talking about actual code that's written using normal languages. I'm familiar with the difference between "regular expressions" as in "regular languages".
Lex/Flex, wich I think we can agree is used by "actual code that's written using normal languages" use DFAs, both inside rules and between rules, and they do not try '|' cases left to right (They probably could have if they wanted since there is a REJECT action that already force them to store the list of all the rules/texts that were matched):
Regexp for tokenization does work. This entire essay boils down to the fact that you can always postprocess matches and in this case that corresponds to tossing unwanted tokens out.
I've seen several regexs in various code reviews that are used to validate user input but do so in an exponential manner that can be exploited for simple DOS attacks.
It's more a question of which ones can't be. There are some really nasty and not very obvious gotchas here; https://regular-expressions.mobi/catastrophic.html has a good dive into how, for example, backtracking combines with incautious regex design to produce exponential behavior in the length of input.
I don't have a hard and fast rule of my own about regex complexity, but I do have a strong intuition over what's now ca. 25 years of working with regexes dating back to initial exposure in Perl 5 as a high schooler. That intuition boils down more or less to the idea that, when a regex grows too complex to comprehend at a glance, it's time to start thinking hard about replacing it with a proper parser, especially if it's operating over (as yet) imperfectly sanitized user input.
Sure, it's maybe a little more work up front, at least until you get good at writing small fast parsers - which doesn't take long, in my experience at least; formal training might make it easier still, but I've rarely felt the lack. In exchange for that small investment, you gain reliability and maintainability benefits throughout the lifetime of the code. Much of that comes from the simple source of no longer having to re-comprehend the hairball of punctuation that is any complex regex, before being able to modify it at all - something at which I was actually really good, as recently as a decade or so ago. The expertise has since expired through disuse, and that's given me no cause for regret; the thing about being a regex expert is that it's a really good skill for writing unreadable and subtly dangerous code, and not a skill good for much of anything else. Unreadable and subtly dangerous code was fine when I was a kid doing my own solo projects for fun, where the worst that'd happen is I might have to hit ^C. As an engineer on a team of engineers building software for production, it's not even something I would want to be good at doing.
> That intuition boils down more or less to the idea that, when a regex grows too complex to comprehend at a glance, it's time to start thinking hard about replacing it with a proper parser
You can get some surprisingly complex yet readable regexes in Perl by using qr//x[1] and decomposing the pieces into smaller qr//s that are then interpolated into the final pattern, along with proper inline comments in the regexes themselves.
I don't see anything about qr//x that makes regexes built this way less vulnerable to the kind of exponential backtracking problem under discussion here.
I do see a great opportunity to, by assuming interpolated qr// substrings have the locality the syntax falsely suggests, inadvertently create exactly that kind of mishap with it being minimally no easier, and potentially actually more difficult, to notice.
Write your code however you like, of course, including concatenating strings and passing the result to 'eval'. The last time I dealt with more Perl than a shell one-liner was around 2012, and that the language encourages this kind of thing is one of the reasons I'm glad of that.
Given that I write my code with a text editor that does nothing but concatenate strings that I input and then I pass it to a compiler or an interpreter, all of the code I write is concatenating strings and passing it to 'eval'.
And I use proper decomposition to keep it cognitively manageable. It's pretty clear that reasoning about composition is beyond you, but trust me that given two procedures that both do not have an undesirable property, one can rest assured that simple composition will not introduce that undesirable property.
Many things are beyond me. Perhaps it's to my good fortune that the generally low utility of gratuitous personal insults is not among them. Certainly the next technical discussion I see improved by such behavior will be the first.
Well then, in the interest of amity let me suggest that it would be to your good fortune to work on your self-awareness. But, should you prefer not to, then by all means, you do you.
Ooooh or worse, I once caught someone's "email matching" RegEx code during a code review that was opening the door for some nasty SQL Injection or XSS attacks (kind of like validating if the text field contained a valid email.. but not if it was ONLY a valid email).
The problem with RegEx is its "obscurity". However Maybe someone could write a nice testing tool that would throw millions of known exploits into each regex it finds in your code to see if it is vulnerable.
It's semantics, not syntax. It's one of the simplest and oldest search engines in the history of computing; obviously a more complex engine can provide more robust semantics.
The more general tip is that a single regex isn't the only tool you have. You don't have to get your final product one one step. Almost every "disaster" regex comes from someone trying to do too much at once.
One other solution would have been to run the regex twice, once to pick up all instances of Tarzan, and a second on the results of the first to filter out all instances of "Tarzan".
A big source of trying to do too much is environments that offer easy regex-based transformations defined as a pair of regex and a single replacement string (that may contain references to matching groups) and make other transformations hard ("while find + rest"). When you have the option to provide a "process match" closure instead of the replacement string the lure of putting too much into a single regex almost collapses.
I’m pretty decent with regex, but I often break complex regexes down into multiple steps for better clarity and easier debugging. Sure, you can use extremely clever one-liners, but the next maintainer of your code may hunt you down and murder you on the spot for wasting weeks of their time.
And, as always, the "next maintainer" is most likely your future self. So conversely, it's really nice to give your future self the gift of clarity and con-conversely look back and say "Thanks past me!"
to your point (and how to fix it in a way that seemingly nobody does) - you can make complicated regular expressions pretty simple by using named groups, and ignoring pattern whitespace because they allow you to logically separate different components and specify intent. Nobody would ask a fellow dev to debug javascript where it is all on a single line and every variable name is a1, a2, etc. Except people do it all the time with regex. Its insane. Hell, you don't even get the blessing of a1, a2, a3. It is all unnamed. Insanity.
This is also often the problem with disaster SQL queries. I've seen some monsters that got hopelessly tangled up in their own JOIN constraints, trying to fetch all the data in one roundtrip because OMG LATENCY but then having to do full table scans over large-ish tables instead. Rewriting it as three small indexed queries reduced the runtime from 40 minutes (!) to less than a second.
Don't do too much in one operation whether it's regexes, SQL queries or OOP classes!
I’ve used the pattern several times of “select these SQL objects into a cursor, then iterate over the cursor to assign/revoke/check permissions on the objects”.
It’s still in a single batch of SQL (stored procedure in our case, so no additional network roundtrips), but the code is vastly clearer to read/maintain this way.
They're clearer in the sense that it makes it easier/possible to do things like:
While maintaining/changing the SQL, comment in/out select-statements-as-printf-debugging, and comment in/out actual execution of the statements themselves.
These cursors would often contain [identifying object reference], [category of statement], [text of SQL statement to execute]. You would write a select statement to populate the cursor, then loop over the cursor to run all the statements in the order you wanted (drops, then user/role creates, then grants, or whatever the situation called for).
It's not about logical clarity, but practical maintainability given the (overall weak) state of tooling for database queries. Is it a bastardization of SQL to do something that "should be" done in another scripting language? Maybe, but there's a lot of power in giving the DBAs tooling that works exclusively in a language and environment that's familiar for them rather than splitting it across SQL and python/tcl/ruby/whatever. Not nearly every competent [relational] DBA is competent across multiple languages. Every competent [relational] DBA is competent in SQL.
Is it even possible to use set-based SQL to call EXEC SQL EXECUTE IMMEDIATE or sp_executesql on each statement in a set?
Caveats apply. A regular expression isn't just a way of saving yourself a few lines of explicit string manipulation. It's describing a state machine that does these text operations efficiently (in some programming environments, that state machine will get optimized and compiled down to metal prior to first use).
If you're matching a couple short strings, sure, don't bother overthinking the regex. If you're matching a lot of them, and/or they're long, then the extra time spent on making a single regex work will be worth it. The regex will work smarter than your hand-rolled code, and it also won't waste memory returning partial results.
Also: in my experience, almost every "disaster" regex comes from people not bothering to document and test what they write.
I got the feeling that a lot of those »I have to do this in a single regex« questions come from places where a single regex is basically the only API you have available. Something like form input validation where the framework provides a handy regex it uses for validation, but doesn't expose actual validation callbacks or events do to the same in code without having to redo everything around it. It's only a hunch, but when I have the opportunity to use code to validate a string I probably wouldn't assume that code to be a mandatory one-liner, even as a beginner developer.
This is why any attempts to make plain http sites throw up scare warnings is a horrible idea. The internet is littered with old websites that contain a wealth of knowledge and deserve to remain accessible.
Just make browsers for into “read only” mode where input cannot be accepted on non-secure pages. But don’t wall them out!
... is interesting. But since it returns the match in a submatch I would say the \K approach is better:
(?:not_this.*?)*\Kbut_this
Because usually when you try hard to accomplish something with a regex, you do not have the luxury to say "And then please disregard the match and look at the submatch instead".
That doesn't work. `(?:"Tarzan".*?)*\KTarzan` should behave identically without `\K`, and it will match `"Tarzan" "Tarzan"` because the ungreedy quantifier ? still allows backtracking (it just changes the search order). You want the possessive quantifier + instead; `not_this|(but_this)` is equivalent because regexp engines will not look back into once matched string.
A bit off topic, but
the commented version was much clearer, than the version with separate function.
(full sentences are very good at explaining things)
I'm not sure if any regex library exposes this, but since regular languages are closed over compliment and intersection you could theoretically do something like match("....string..", regex("Tarzan") - regex("\"Tarzan\"")), where the - operation is shorthand for intersection with the compliment. Does anyone know if any regex libraries expose these sorts of operations on the regular expression/underlying DFA?
Not exactly that but take a look at https://github.com/mtrencseni/rxe ("literate regex"). I found this on HN and recall the comment thread being good but I can't find it now.
Unfortunately (or perhaps fortunately), “regexes” as commonly implemented in programming languages are only loosely related to regular expressions from automata theory. With all their extensions, they can recognize much, much more than just regular languages, and I don’t think they’re closed under complement (though I’m not sure). However, most regex engines have a feature called negative lookahead assertions, (?!do not match), which would almost work in the way you suggest.
You have to be careful about inputs like this though: “Inside a string”Tarzan”Again inside a string”
Yeah, a DFA that recognizes a regular language can easily be implemented with O(n) worst case behavior.
My attitude is generally that one should use regexes for matching regular languages and if one needs a stack or even Turing completeness then handle that in code around the regex.
Wouldn't that end up just being the same as 'regex(Tarzan)'? Those regexes can't match the same thing, they can only overlap.
What you want is something like all matches of regex("Tarzan") not contained in a match for regex("\"Tarzan\""), which is a bit trickier. That would require something like:
Intersection with the complement will not work here.
Because the idea is to match Tarzan, but only if it is not preceded and followed by a quote.
Regex intersection and complement do not perform look-behind or trailing context.
Live demo:
This is the TXR Lisp interactive listener of TXR 265.
Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet.
TXR may be used in areas that are not necessarily well ventilated.
1> [#/Tarzan&~"Tarzan"/ "Jane shouted, \"Tarzan\""]
"Tarzan"
2> [#/Tarzan&~"Tarzan"/ "Jane shouted, \"Tarzan!\""]
"Tarzan"
The &~"Tarzan" makes absolutely no difference. The reason is that Tarzan matches exactly one string. The complement ~"Tarzan" matches a whole countable infinity of strings, and one of those is Tarzan. The intersection of that infinity and Tarzan is therefore Tarzan.
Intersection with complement is useful like this:
Search for a three-character substring that is not cat:
My biggest grief with regexp is that it is just a compact code disguised as something else. It is relatively common that you want to scan a string but action codes intermixed. There is a way to do that with regexp (Perl (?{...}) etc. or PCRE callouts), but it is always awkward to put a code to a regexp. As a result we typically end up with either a complex code that really should have used a regexp but couldn't, or a contorted regexp barring the understanding. The essay suggests `(*SKIP)(*FAIL)` at the end, which is another evidence that a code and a regexp don't mix well so a regexp-only solution is somehow considered worthy.
OK that's pretty clever (I certainly never thought of putting a capturing group inside only one side of an "or")...
...but it doesn't seem particularly useful? It probably won't work in most cases where this is just part of a larger expression. You're usually using capturing groups in a particular way for a good reason, and this would mess that up.
In contrast, the lookbehind+lookahead way is the "proper" and intuitive way to write it, and works as part of any larger expression.
So... +100 points for cleverness, but don't actually use this please. :)
Not GP, but I'd go a very simple and verbose way, maybe that's what they meant to. Match:
(.)Tarzan(.)
Then in an additional line of code assert
(Group 1 == Group 2) ≠ "
This shifts the logic out of regex and into the surrounding programming language context. That's arguably better, but the resulting regex is extremely dull and unclever.
Yeah, that's more or less what I meant. Write a regex (plus line of code) to make sure `Tarzan` appears. Then write another regex and line of code to make sure `"Tarzan"` doesn't appear.
Maybe at this point you aren't using regex even. Nice, you solved two problems.
(I do appreciate regex and even use them a lot. But, I use them enough to avoid them as much as possible.)
I mean, I guess if nobody on your team understands regexes.
But generally, once you decide to use a regex in the first place, you might as well put as much regular everyday logic as you can in it. Otherwise you might as well look for "Tarzan" with a dumb string search.
Lookbehinds and lookaheads aren't rocket science. And you can always leave a comment about what they're doing if you're worried other team members won't grok the syntax.
> Lookbehinds and lookaheads aren't rocket science.
Lookbehinds and lookaheads (especially negative lookbehinds) are rocket science.
What is "rocket science?" "Rocket science" is the feeling you get in math class where the instructor explains a proof to you in the clearest possible terms and you just don't get it. You have to listen to the explanation multiple times, preferably in a few different ways, and then you have to sleep on it, and then you get it, maybe.
But "rocket science" isn't just hard to understand. It's a hard problem where the consequences for failure are catastrophic. When you fail at rocket science, a multi-million dollar rocket explodes.
Anyone who's ever tried to teach lookbehinds to a newbie has seen it: you explain how lookbehinds work, and then ask the newbie to create a regex with negative lookbehind, to demonstrate mastery. I've done it a few times, and they never get it right, ever.
At best, they flub the syntax, but even once they get over that, they usually write the worst possible regex: a regex that works correctly on desired inputs but does the wrong thing on the input the regex is designed to reject.
This is a notorious problem with writing regexes, but it's way worse for negative lookbehind, because it's asserting that something isn't there, rather than querying for something that is there.
When I see a regex with negative lookbehind during code review, I ask for unit tests, not just comments. Reliably, regexes get even more complex when unit tests are added, because it's just so damn hard to write a correct regex with negative lookbeind.
I've never used the "trick" from TFA before, but it already sounds way easier to use than negative lookbehinds, and I'm curious to try it.
I agree on unit tests for non-trivial regexes as a general rule, but respectfully disagree on lookaheads and lookbehinds.
Things like greedy vs. non-greedy matching, matching newlines or not, handling Unicode correctly, inserting a capturing group when you actually needed a non-capturing group, making sure your regex works if it matches the start or end of a string, escaping characters -- those can be tricky.
On the other hand, lookaheads and lookbehinds are conceptually extremely straightforward, you just need a cheatsheet to remember the syntax is all.
I think dumb, brute force, simple approaches like this are underrated. Writing elegant, pithy code that pleases you aesthetically is nice but writing code that's explicit and obvious and can be maintained by the new kid is often more pragmatic.
This trick may be thought of as a simplification of the systematic approach to parsing stuff, that is the lexer-parser division of responsibilities.
The lexer uses regexes but only for splitting the input stream of characters into tokens. Identifiers, integers, operators, strings, keywords, opening brackets and whatnot - each type of token is defined by a regex. This part is hopefully deterministic and simple, although the lexer matches regexes for all kinds of tokens at once, which is why lexer generators are often used to generate lexers.
The heavy lifting is done by the actual parser which tries to combine the tokens into something that makes sense from the point of the grammar.
So in this trick the sub-regexes between |'s define the tokens (the lexer part) while the group mechanism selects the single token that we want to keep (a very very simple parser).
It took me over 15 years until I started to willingly use RegExp, but now I can't live without it. It's like the curse of knowledge, once you learn something you'll loose all empathy and assume everyone else knows it too. It still surprises me though, I've had bug like my regex matching terminal color sequences messing up the data if it was colored.
It feels like something that was more discovered than invented, something that would exist even if nobody knew of its existence. I get the same feeling when listening to Pharrell Williams' Happy.
Very long build up to what is definitely a neat trick, although without SKIP FAIL, it might cause explosive growth in the memory usage as it allocated space for the results you don’t need (unless you use a streaming regex option).
Speaking of lengthy: this site breaks the iOS Safari scroll bar! It just disappears altogether (even when scrolling up or down to make it show, like you have to these days to please the UX designers in Palo Alto).
Doc Brown spoke to me: "You're just not thinking fourth dimensionally!" Like Marty, I have a real problem with that. But don't you see: [b]anana matches banana but it doesn't match 'grep [b]anana' as a raw string. And so I get only the process I wanted!
the simple things in life elude me. holy cow. i learned the "tag a grep -v grep" at the end and never looked into refining it. but just flipping the greps? nope, not once did that ever occur to me. thanks
That's a valid concern, but in this case it won't cause problems -- grep exits with 0 if a line was selected; in the case of `grep -v grep` that means there was a line without "grep", which is what we want.
(Also @thewakalix made a good suggestion to reverse the greps.)
pgrep is great, but note that you can still encounter this problem if you run pgrep in parallel -- it will never match its own process, but it will match other pgrep processes.
So for example if you have a script that uses `pgrep -f banana` to search for a "banana" process, and you run that script twice in parallel, pgrep might see the other pgrep process and think "banana" is running even though it isn't.
What systems don’t have grep? Off the top of my head, anything vaguely POSIX compliant would have it, and anything descended from 4th Edition Unix would also have it, which seems like it would cover everything.
Maybe someone deleted grep by accident? Or maybe you accidentally traveled in time to the 70's and need to use a nearby PDP-11 to calculate a way home? Lots of possibilities.
Toolchains for compiling systems from source is another answer, e.g., NetBSD's toolchain has sed, but not grep.
A third answer is install media. For example, NetBSD install kernels have ramdisks with sed but not grep.
A fourth answer is personal, customised systems. I create small systems that run from RAM. I run these on small computers with limited resources. When one of these computers first boots up, it may not have a "full" set of userland programs. It may not have grep. I am not inclined to use the limited space available to include grep at such an early stage if I can get by with sed.
The regex `[b]anana` matches the string "banana". But it does not match the literal string `[b]anana` --- to match that, you'd need something like `\[b\]anana`. The literal string `[b]anana` is what shows up in the process table for grep, and so it doesn't match.
199 comments
[ 6.2 ms ] story [ 252 ms ] threadLookbehind
Lookahead
Advanced handling of tags
Replace before matching
the best regex trick ever:
"Tarzan"|(Tarzan)
The whole site contains useful regex advice
https://itnext.io/a-wild-way-to-check-if-a-number-is-prime-u...
The main trick for me was you first have to convert the number to unary, which was done outside of the regex.
...because what could possibly go wrong? From the latest comment at the end of the page, the author would like you to know that the outcome is your problem, because you're using the wrong browser:
June 20, 2021 - 15:02
Subject: RE: Undoing whatever is hiding this page.
Hi Allen, try a different browser. There's no strange shading on the page, your browser is deciding to display it in a weird way. Regards, -Rex
Interestingly, the author also appears to control yu8.us
Breaking one's own content by https-ing one site but not another is a great example of why to not prop up a website's basic legibility on a third party dependency, even if it's one you own and control.
> Page copy protected against web site content infringement by Copyscape
(It's not that the annoyances aren't annoying, it's that they're so common that they lead to repetitive offtopicness that compounds into more boring threads.)
https://news.ycombinator.com/newsguidelines.html
I used to have a laminated sheet on my wall at an office because it was so terribly bad.
Named capture group: (?<name>foo)
Non-capturing group: (?:foo)
Lookahead: (?=foo)
For negative lookahead, change = to !: (?!foo)
For lookbehind, add <: (?<=foo)
For negative lookbehind, change = to !: (?<!foo)
(Not from memory, had to look everything up...)
right, great list but I'll forget it all by tomorrow.
(...the trick is still cool, though; I can imagine other situations where it would be more useful. However it does seem like it potentially depends on the particular regex engine being used, in contrast to the author's claim about it being totally portable; yes, it'll compile on anything, but will it work?)
regex is not regular expressions - if using NFA to match then you're matching all alternates simultaneously.
Russ Cox has good pictures explaining idea in 'Regular Expression Search Algorithms' section of <https://swtch.com/~rsc/regexp/regexp1.html>
a|ab {cout << "matched ab" << std::endl; } b { cout << "matched b" << std::endl; }
if provided with "ab", will match the first rule with "ab", and not the first with "a" then the second with "b".
But DFAs can implement the preference-order semantics found in backtracking regex engines too. Russ Cox's articles show how to do that.
(Just adding some additional info to your point.)
This style of writing is just obnoxious.
Parsing the tokenization result of a Dyck language still requires as context-free grammar.
It's not a badge of honor or a great trick to try that with regular expressions. It is using the wrong tool for the job.
was to convince programmers it didn't exist?
I don't have a hard and fast rule of my own about regex complexity, but I do have a strong intuition over what's now ca. 25 years of working with regexes dating back to initial exposure in Perl 5 as a high schooler. That intuition boils down more or less to the idea that, when a regex grows too complex to comprehend at a glance, it's time to start thinking hard about replacing it with a proper parser, especially if it's operating over (as yet) imperfectly sanitized user input.
Sure, it's maybe a little more work up front, at least until you get good at writing small fast parsers - which doesn't take long, in my experience at least; formal training might make it easier still, but I've rarely felt the lack. In exchange for that small investment, you gain reliability and maintainability benefits throughout the lifetime of the code. Much of that comes from the simple source of no longer having to re-comprehend the hairball of punctuation that is any complex regex, before being able to modify it at all - something at which I was actually really good, as recently as a decade or so ago. The expertise has since expired through disuse, and that's given me no cause for regret; the thing about being a regex expert is that it's a really good skill for writing unreadable and subtly dangerous code, and not a skill good for much of anything else. Unreadable and subtly dangerous code was fine when I was a kid doing my own solo projects for fun, where the worst that'd happen is I might have to hit ^C. As an engineer on a team of engineers building software for production, it's not even something I would want to be good at doing.
You can get some surprisingly complex yet readable regexes in Perl by using qr//x[1] and decomposing the pieces into smaller qr//s that are then interpolated into the final pattern, along with proper inline comments in the regexes themselves.
[1] https://perldoc.perl.org/perlre#/x-and-/xx
Regexes are code.
Therefore, decomposition makes complex regexes both feel and actually be easier to reason about.
I do see a great opportunity to, by assuming interpolated qr// substrings have the locality the syntax falsely suggests, inadvertently create exactly that kind of mishap with it being minimally no easier, and potentially actually more difficult, to notice.
Write your code however you like, of course, including concatenating strings and passing the result to 'eval'. The last time I dealt with more Perl than a shell one-liner was around 2012, and that the language encourages this kind of thing is one of the reasons I'm glad of that.
And I use proper decomposition to keep it cognitively manageable. It's pretty clear that reasoning about composition is beyond you, but trust me that given two procedures that both do not have an undesirable property, one can rest assured that simple composition will not introduce that undesirable property.
The problem with RegEx is its "obscurity". However Maybe someone could write a nice testing tool that would throw millions of known exploits into each regex it finds in your code to see if it is vulnerable.
- context sensitive matching
- matching with multi-char-exclusions
(regex is happy the most, when it's used to match "regular language" things)
So to search for words without any vowels just 'grep [^aeiou]'?
One other solution would have been to run the regex twice, once to pick up all instances of Tarzan, and a second on the results of the first to filter out all instances of "Tarzan".
Some rare people can figure out:
\d{1,2}[-/]\d{1,2}[-/](\d{4}|\d{2})
but a dummy can figure out this:
(?<month> \d{1,2} ) [-/] (?<day> \d{1,2} ) [-/] (?<year> \d{4} | \d{2} )
Don't do too much in one operation whether it's regexes, SQL queries or OOP classes!
It’s still in a single batch of SQL (stored procedure in our case, so no additional network roundtrips), but the code is vastly clearer to read/maintain this way.
While maintaining/changing the SQL, comment in/out select-statements-as-printf-debugging, and comment in/out actual execution of the statements themselves.
These cursors would often contain [identifying object reference], [category of statement], [text of SQL statement to execute]. You would write a select statement to populate the cursor, then loop over the cursor to run all the statements in the order you wanted (drops, then user/role creates, then grants, or whatever the situation called for).
It's not about logical clarity, but practical maintainability given the (overall weak) state of tooling for database queries. Is it a bastardization of SQL to do something that "should be" done in another scripting language? Maybe, but there's a lot of power in giving the DBAs tooling that works exclusively in a language and environment that's familiar for them rather than splitting it across SQL and python/tcl/ruby/whatever. Not nearly every competent [relational] DBA is competent across multiple languages. Every competent [relational] DBA is competent in SQL.
Is it even possible to use set-based SQL to call EXEC SQL EXECUTE IMMEDIATE or sp_executesql on each statement in a set?
If you're matching a couple short strings, sure, don't bother overthinking the regex. If you're matching a lot of them, and/or they're long, then the extra time spent on making a single regex work will be worth it. The regex will work smarter than your hand-rolled code, and it also won't waste memory returning partial results.
Also: in my experience, almost every "disaster" regex comes from people not bothering to document and test what they write.
Good trick though.
Just make browsers for into “read only” mode where input cannot be accepted on non-secure pages. But don’t wall them out!
Now that I try it, it indeed does not work.
Maybe the author reads this and can look at it.
sadly, this trick still requires a code comment to explain. Python example:
which in practice means it probably deserves a function: I don't recommend monkeypatching re, i.e. re.search_but_exclude = ...You have to be careful about inputs like this though: “Inside a string”Tarzan”Again inside a string”
My attitude is generally that one should use regexes for matching regular languages and if one needs a stack or even Turing completeness then handle that in code around the regex.
What you want is something like all matches of regex("Tarzan") not contained in a match for regex("\"Tarzan\""), which is a bit trickier. That would require something like:
regex("Tarzan") - all-substrings(regex("\"Tarzan\""))
and I'm not sure regular languages are closed over the "all-substrings" operation. Actually I'm pretty sure they aren't.
I’ll take that as a complement.
Because the idea is to match Tarzan, but only if it is not preceded and followed by a quote.
Regex intersection and complement do not perform look-behind or trailing context.
Live demo:
The &~"Tarzan" makes absolutely no difference. The reason is that Tarzan matches exactly one string. The complement ~"Tarzan" matches a whole countable infinity of strings, and one of those is Tarzan. The intersection of that infinity and Tarzan is therefore Tarzan.Intersection with complement is useful like this:
Search for a three-character substring that is not cat:
OK that's pretty clever (I certainly never thought of putting a capturing group inside only one side of an "or")...
...but it doesn't seem particularly useful? It probably won't work in most cases where this is just part of a larger expression. You're usually using capturing groups in a particular way for a good reason, and this would mess that up.
In contrast, the lookbehind+lookahead way is the "proper" and intuitive way to write it, and works as part of any larger expression.
So... +100 points for cleverness, but don't actually use this please. :)
I would say, the "proper" way is to have a separate line of code validating what's not there :)
Maybe at this point you aren't using regex even. Nice, you solved two problems.
(I do appreciate regex and even use them a lot. But, I use them enough to avoid them as much as possible.)
But generally, once you decide to use a regex in the first place, you might as well put as much regular everyday logic as you can in it. Otherwise you might as well look for "Tarzan" with a dumb string search.
Lookbehinds and lookaheads aren't rocket science. And you can always leave a comment about what they're doing if you're worried other team members won't grok the syntax.
Lookbehinds and lookaheads (especially negative lookbehinds) are rocket science.
What is "rocket science?" "Rocket science" is the feeling you get in math class where the instructor explains a proof to you in the clearest possible terms and you just don't get it. You have to listen to the explanation multiple times, preferably in a few different ways, and then you have to sleep on it, and then you get it, maybe.
But "rocket science" isn't just hard to understand. It's a hard problem where the consequences for failure are catastrophic. When you fail at rocket science, a multi-million dollar rocket explodes.
Anyone who's ever tried to teach lookbehinds to a newbie has seen it: you explain how lookbehinds work, and then ask the newbie to create a regex with negative lookbehind, to demonstrate mastery. I've done it a few times, and they never get it right, ever.
At best, they flub the syntax, but even once they get over that, they usually write the worst possible regex: a regex that works correctly on desired inputs but does the wrong thing on the input the regex is designed to reject.
This is a notorious problem with writing regexes, but it's way worse for negative lookbehind, because it's asserting that something isn't there, rather than querying for something that is there.
When I see a regex with negative lookbehind during code review, I ask for unit tests, not just comments. Reliably, regexes get even more complex when unit tests are added, because it's just so damn hard to write a correct regex with negative lookbeind.
I've never used the "trick" from TFA before, but it already sounds way easier to use than negative lookbehinds, and I'm curious to try it.
Things like greedy vs. non-greedy matching, matching newlines or not, handling Unicode correctly, inserting a capturing group when you actually needed a non-capturing group, making sure your regex works if it matches the start or end of a string, escaping characters -- those can be tricky.
On the other hand, lookaheads and lookbehinds are conceptually extremely straightforward, you just need a cheatsheet to remember the syntax is all.
Yes, this was sort of the idea as well (also see sibling response). I'd just as soon have 2 lines of code rather than a regex.
If anybody on your team doesn't understand regexes, you mean.
Save the clever stuff for where it's needed.
The lexer uses regexes but only for splitting the input stream of characters into tokens. Identifiers, integers, operators, strings, keywords, opening brackets and whatnot - each type of token is defined by a regex. This part is hopefully deterministic and simple, although the lexer matches regexes for all kinds of tokens at once, which is why lexer generators are often used to generate lexers.
The heavy lifting is done by the actual parser which tries to combine the tokens into something that makes sense from the point of the grammar.
So in this trick the sub-regexes between |'s define the tokens (the lexer part) while the group mechanism selects the single token that we want to keep (a very very simple parser).
It's almost like nature, many simple rules coming together to make extremely clever and fairly complex ideas
Speaking of lengthy: this site breaks the iOS Safari scroll bar! It just disappears altogether (even when scrolling up or down to make it show, like you have to these days to please the UX designers in Palo Alto).
I have a process table and I want to grep it for the phrase "banana":
ps auxww | grep banana
root 87 Jun21 0:26.78 /System/Library/CoreServices/FruitProcessor --core=banana
mikec 456 450PM 0:00.00 grep banana
Argh! It also greps for the grep for banana! Annoying!
Well, I'm sure there's pgrep or some clever thing, but my coworker showed me this and it took me a few minutes to realize how it works:
ps auxww | grep [b]anana
root 87 Jun21 0:26.78 /System/Library/CoreServices/FruitProcessor --core=banana
Doc Brown spoke to me: "You're just not thinking fourth dimensionally!" Like Marty, I have a real problem with that. But don't you see: [b]anana matches banana but it doesn't match 'grep [b]anana' as a raw string. And so I get only the process I wanted!
(Also @thewakalix made a good suggestion to reverse the greps.)
the reversing the greps will be my new default behavior
ps auxwww | grep banana | grep -v grep && echo ${PIPESTATUS[1]}
type of thing
Never thought of that. Nice.
So for example if you have a script that uses `pgrep -f banana` to search for a "banana" process, and you run that script twice in parallel, pgrep might see the other pgrep process and think "banana" is running even though it isn't.
I was bitten by this :)
Escaping works under zsh. My preferred method is single quotes:
But nice regex though
Edit : someone already posted that solution https://news.ycombinator.com/item?id=27777901
Emedded systems is one answer.
Toolchains for compiling systems from source is another answer, e.g., NetBSD's toolchain has sed, but not grep.
A third answer is install media. For example, NetBSD install kernels have ramdisks with sed but not grep.
A fourth answer is personal, customised systems. I create small systems that run from RAM. I run these on small computers with limited resources. When one of these computers first boots up, it may not have a "full" set of userland programs. It may not have grep. I am not inclined to use the limited space available to include grep at such an early stage if I can get by with sed.
Hope this answers your question.
It does, thanks for the detailed response.
edit: saw someone else posted this as well. should have known