This could be useful for writing lexers. Although I typically need to interleave some "decoding" with proper lexing (like decoding escape sequences in strings, universal character names in identifiers etc.), which would make it harder, when you want to do it in a single pass.
This comment[1] from ianlancetaylor on Github appears to provide reasons as to why the standard Go implementation of regex is less than optimal.
As you say, in a language like Python, the regexp engine is implemented in C. So you are comparing a somewhat tuned Go implementation with a highly tuned C implementation. You also need to consider the characteristics of the engine. Go has chosen to follow the re2 path (not surprising, since Russ Cox is a major author of both Go and re2). re2 has much better performance characteristics than some other regexp engines, in that it never has an exponential slowdown, but that comes at a cost for other regexps. Even then, the C++ re2 implementation has many optimizations that are not in the Go implementation.
regexp2go author here: yes, mostly it's the combination of the regexp package not aiming at top performance and therefore omitting some optimizations, the implicit regexp interpreter overhead, and the go compiler not aiming at top performance.
On the avoiding exponential slowdown front, regexp2go and other re2-inspired implementations (including e.g. rust's) all have that as a goal.
3 comments
[ 2.5 ms ] story [ 13.5 ms ] threadAs you say, in a language like Python, the regexp engine is implemented in C. So you are comparing a somewhat tuned Go implementation with a highly tuned C implementation. You also need to consider the characteristics of the engine. Go has chosen to follow the re2 path (not surprising, since Russ Cox is a major author of both Go and re2). re2 has much better performance characteristics than some other regexp engines, in that it never has an exponential slowdown, but that comes at a cost for other regexps. Even then, the C++ re2 implementation has many optimizations that are not in the Go implementation.
[1] https://github.com/golang/go/issues/19629#issuecomment-28815...
On the avoiding exponential slowdown front, regexp2go and other re2-inspired implementations (including e.g. rust's) all have that as a goal.