What's most annoying with gperf and similar tools is that they aren't really suited to applications where the set of keys is known at runtime during initialization.
I wrote a python library that would build a perfect hash at run-time, it was basically the stupidest way you could do it - it would shell out to gperf to build the library, compile it to a shared library, then link the entry points in (I think with ctypes? I can't remember).
It was just for fun, but in the end even ignoring the startup costs of all of that, the default performance of python's dictionary was better.
Maybe this is one of those situations where compile time code execution wins out. Instead of needing one solution for runtime and one for a priori knowledge, you just run the runtime code during the build process and Bob's your uncle.
I've written code that, during initialization, after all keys have been collected, essentially called gperf to create the lookup function, compiled it, and then dynamically loaded it, so that the (long-running) program would be able to use it.
Uhh, there is nothing at all "academic" about pthash, (or phast, or ptrhash), apart from the fact that they also described their ideas in papers. All of those tools work on massive sets of data, support massively parallel construction, and support external memory construction. They are all well-engineered libraries and not "academic" in any derogatory sense.
I believe there are some high concurrency hash tables out there where the data structure contains two tables, and so each get results in a constant number of fetches, but the count is almost always greater than one. But if it avoids concurrency issues that ends up being acceptable.
I remember Cliff Click presenting one that was lockless and if I'm recalling correctly, where capacity growth operations could happen in parallel with reads. And possibly writes.
There’s an explanation of how to implement a perfect hash generator in “Hacker’s Delight” that you can read to take a known set of keys and produce a consistent perfect hash at runtime.
It’s a very worthwhile thing to keep in your back pocket if you’re going to deal with hashes whose keys are 100% knowable at runtime but not before.
11 comments
[ 2.8 ms ] story [ 31.4 ms ] threadThough honestly this post really needed some numbers and benchmarks.
It was just for fun, but in the end even ignoring the startup costs of all of that, the default performance of python's dictionary was better.
https://github.com/rurban/gperf/tree/autotools or some other branch. Forgot which.
https://github.com/rurban/cmph/tree/compressed_o (lotsa improvements)
https://github.com/rurban/pthash (compile to static C++)
I've also extended nbperf for those features
I remember Cliff Click presenting one that was lockless and if I'm recalling correctly, where capacity growth operations could happen in parallel with reads. And possibly writes.
It’s a very worthwhile thing to keep in your back pocket if you’re going to deal with hashes whose keys are 100% knowable at runtime but not before.