While the default behavior of this library is to map the entire file into address space (and also the default assumption that people have regarding mmap), you can map specific portions of a file into memory to avoid virtual address exhaustion. It's just far less convenient to do so, though typically still faster than the alternatives.
I misread the comment. The file fits in address space? As sibling comment suggests, address space is v. big and it's very much a corner case (or very specific use case) for most people.
All x86-64 CPUs support 48 bit address space, with 57 bits in the newest ones [1]. It's unlikely that address space exhaustion would be a problem for the vast majority of use cases.
According to the developers of Mongo at least, it's not worth considering. You can't store more than 2GB of data on a 32-bit machine because they map the entire datastore into memory.
For people doing memory mapping in Go, I would strongly advise calling debug.SetPanicOnFault() and setting up a panic handler. Without it, your program will simply crash in case of I/O failures, file truncation, etc..
Here's some code I wrote some time ago that does exactly this:
Be aware that using mmap instead of read/write might mess up Go’s scheduling. The runtime has special handling of other blocking system calls which tried to minimize the impact of those on other running goroutines. With directly accessing memory locations mapped via mmap you won’t get this benefit.
11 comments
[ 5.2 ms ] story [ 46.5 ms ] thread> One of the main problems a database storage engine has to solve is how to deal with data in disk that is bigger than the available memory.
[1] - https://en.wikipedia.org/wiki/Intel_5-level_paging
Here's some code I wrote some time ago that does exactly this:
https://github.com/buildbarn/bb-storage/blob/c346ca331930f1b...