HN Explorer

29 comments

[ 4.8 ms ] story [ 1418 ms ] thread
hmottestad
A good comparison. I couldn’t see that the author has created an issue on the golang GitHub. I assume that this would be considered a bug and that the golang community would be open to a fix.
rini17
Or not, because it simplifies code, no differences between platforms. And if syscall overhead once per each GB is really a problem you should probably try mmap or some other IO style.
hmottestad
The author saw performance issues with this restriction, which is why they wrote the post.
xyzzy123
I would be curious to see the numbers on that.
hmottestad
You should probably ask the author then. There are a bunch of links to various socials on the bottom of the post.
technicolor
Hello, author here. Just to clarify, I wasn't saying that this has any perf impact per se. I was trying to see why syscall showed up quite a bit in the profile, when reading a large file, no matter what buffer size I pass in. Rather than spend time finishing the actual work, I spent way too much time trying to understand the reason for the limit.
hmottestad
Mmap seems specific to POSIX according to this post[1], so it's not a portable way of using memory mapped files. Since there are already differences between platform capabilities in golang there really shouldn't need to be an artificial limit to how large of a chunk you can read in one operation. In 20 years I'm sure 1 GiB will be considered small.

1: https://stackoverflow.com/questions/52589949/undefined-sysca...

xyzzy123
Interesting factoid, but I can't see how this would lead to any real world performance impacts.

I am also baffled as to why you would want 2gb reads for wc. System calls are not that expensive and at some point touching so many pages has gotta hurt.

aydio
FYI, factoid has a different meaning: https://en.wikipedia.org/wiki/Factoid
deaddodo
> Since the term's invention in 1973, it has become used to describe a brief or trivial item of news or information.

In the very first paragraph, in fact.

DemocracyFTW2
Äckschully no: A factoid is either an invented or assumed statement presented as a fact, or a true but brief or trivial item of news or information. which is exacktly what this is. Innit.
technicolor
Author here. I wasn't suggesting it would have any perf impact. Just that it was an interesting change set.
xyzzy123
Makes sense, I guess I misunderstood the premise of the article. It's a good dive.
jsnell
I feel I'm missing something here, but it feels like the intro is not only suggesting there was a performance impact, but very explicitly stating there was:

> profile of a sample word count program I was writing, which showed the program was spending way too much time in the syscall module. That in this context can only mean one thing: way too many read syscalls were getting called.

I find it hard to believe that the profile would look any different with 1 vs 2 syscalls per 2GB chunk. The syscall overhead is going to be insignificant compared to actually copying the data. The program is going to be spending a lot of time doing syscalls no matter how many there are, because the individual syscalls will just start to take more time as you increase the size of the buffer.

Edit:

Compare:

  strace -e read -T perl -MFcntl -e 'sysopen FD, "foo", Fcntl::O_RDONLY; while (sysread FD, $buf, 1*1024*1024*1024) {} '

  read(3, ..., 1073741824) = 1073741824 <0.455901>
  read(3, ..., 1073741824) = 1073741824 <0.219711>
  read(3, ..., 1073741824) = 1073741824 <0.213923>
  read(3, ..., 1073741824) = 1073741824 <0.211783>
Vs.

  strace -e read -T perl -MFcntl -e 'sysopen FD, "foo", Fcntl::O_RDONLY; while (sysread FD, $buf, 2*1024*1024*1024) {} '

  read(3, ..., 2147483648) = 2147479552 <0.921789>
  read(3, ..., 2147483648) = 2147479552 <0.487007>
  read(3, ..., 2147483648) = 8192 <0.000031>
markoman
I tried your strace commands on my linux desktop and I don't get anything like your results. I get a few lines of 'read(3, "\177ELF\2\1\1...' type stuff. Are you using bash (as I am) or some other shell? Could there be some escaping that I'm missing? Thanks.
jsnell
It's reading from a file named foo, which in this case was 4GB large and created with:

  dd if=/dev/zero of=foo bs=1024 count=$((1024*1024*4))
You probably don't have a file of that name in the current working directory?
cm2187
Isn't GiB = 10^9, GB = 1024^3? Some people seem to use them interchangeably.
progbits
The other way around, author uses them correctly.

k/M/G/... come from SI which is based on 10^3. Ki/Mi/Gi/... are for 1024.

pkage
Other way around, 10^9 bytes is a gigabyte (GB) and 1024^3 is a gibibyte (GiB).
none_to_remain
I have yet to hear anyone say "kibi" or "gibi" that wasn't mocking the term.

I believe the virtuous answer is that bytes are measured in powers of two, & hard drive manufacturers are dishonest. (Bits are measured in powers of ten)

DaSHacka
> hard drive manufacturers are dishonest. (Bits are measured in powers of ten)

You'd have to add ISPs to that too, as network links are typically measured in decimal increments.

syene
I say “mibi” and “gibi” all the time, because my system denotes them as such and it sounds much cuter
DemocracyFTW2
Yeah sure and it's completely irrelevant in this context and also the other way around
PaulKeeble
In the past when I have tested buffer sizes with reads I can't say I have noticed much performance difference past really quite small buffers like 16k. I am sure there is some CPU usage cost associated with more calls but its not much and its just not worth using so much memory for streaming purposes.
silisili
This seems like an article that could have just been summed up and rewritten as the title. It's a conscious choice and consistent. I have no idea what use case would need to use 2GB per read call, if anything the limit seems excessive.
neonsunset
Does not reproduce :)

    // Assuming 2GiB length
    using var file = File.OpenRead("verylargefile.txt");

    var read = 0;
    var iterations = 0;
    var buffer = new byte[Array.MaxLength].AsMemory();

    while ((read = await file.ReadAsync(buffer)) > 0)
    {
        buffer = buffer[read..];
        iterations++;
    }

    Console.WriteLine($"Read {Array.MaxLength} in {iterations} call(s).");