HN Explorer

Ask HN: Why is editing a large file still slow in 2017?

7 points by pfarnsworth ↗ HN
I've amazed that in 2017, with 16-32GB machines, it's still incredibly slow to open and edit a 2 GB text file. I'm opening up a text file in Sublime or vi, and it's extremely slow and my laptop is going crazy with CPU usage. Is there a particular reason why this is still a "hard" problem today?

3 comments

[ 5.8 ms ] story [ 21.9 ms ] thread
xyzzy123
Because very few people want to do that. Many more people want syntax highlighting, automatic backups, vcs support and a kitchen sink full of plugins.

If supporting feature x that nearly no-one wants makes popular features a, b, c much harder to implement then it generally makes engineering sense to forgo x.

smt88
UltraEdit opens large files incredibly quickly and makes it easy to work with them. I haven't used any other GUI editors that do it so smoothly.
cauterized
It's slower than it used to be. Too many layers of abstraction and not enough optimization a in our text editors.

They're built to open 100KB source files and be easy for developers to maintain. Which, granted, covers 98% of use. There was a time when handling files that size required optimizations that might detract from maintainability, but hardware is fast enough now that for files measurable in KB, it makes more sense to optimize the code for the developer rather than for performance.

Version 3 of my favorite text editor could open a 400MB log file in 2-3 seconds on a circa-2002 computer and search or scroll through it like silk. Version 4 beach-balls for 5 minutes for a 40MB file on circa-2012 hardware.

That said, I might ask why you're trying to open a 2GB text file in an editor. Most files that size are either media or computer-generated text like log files.

With the latter what you're generally trying to do is read and filter them, not write. The most sensible way to do that might be to use grep to filter the lines you need into a new, smaller file. If you do this frequently, consider an ELK stack (elasticsearch, logstash, kibana) for ease of use.

If you just need to read the one file and you're not yet sure what you want out of it, I recommend less (the command line tool) to scroll and search. It may take a minute or two to open, but should be performant after that.