HN Explorer

Why is this regex so slow? Am I missing something obvious?

1 points by Mystic_09 ↗ HN
I’ve been trying to optimize a script that processes a huge text file, but my regex seems to be the bottleneck. Here’s the snippet:

python Copy code import re

pattern = re.compile(r"(?:https?|ftp)://[^\s/$.?#].[^\s]*") with open("large_file.txt", "r") as f: matches = [pattern.search(line) for line in f] It works, but it’s painfully slow on large datasets. I tried using find all instead of search, but it didn’t help much.

2 comments

[ 3.4 ms ] story [ 17.3 ms ] thread
zahlman
(HN doesn't support full Markdown. Please see https://news.ycombinator.com/formatdoc for details.)

Can you quantify "painfully slow", "large dataset" etc.?

Have you tried running through with a simpler regex as a baseline? What about just reading the file in the first place?

Does this "huge text file" have excessively long lines, or just a lot of them?

Terr_
One thing that stands out is that you are basically duplicating the entire file into memory, which might be a problem if your files are bigger than your free RAM.

This is because pattern.search(subject) returns a Match object which contains the original line, and you're keeping all of them in the resulting list.

It should be easy to check if that's the issue by monitoring system memory usage, or by testing with a version that does the search but immediately discards the result.