They should throw the "Simple Example" through a code formatter. Would look more professional. Other than that, this is a nice project, congrats to the author.
A change in the last ten years is that it now makes a lot of sense, due to the development of nuanced algorithms, to optimize your hash infrastructure for the details of the specific application. This has been a positive development generally when it comes to efficient software.
I am also wondering whether ahash is faster. ahash is developed with rust. We need a head to head comparsion. However, it is not available in SMHasher package.
They use the wyhash Rust crate, so if wyhash itself was updated doing a head to head comparison would boil down to updating the wyhash crate and rerunning ahash's benchmark suite.
wyhash crate is not wyhash itself. It depends on the skill of the translator and the wyhash version. wyhash has been improved significantly version by version. Also ahash should submit a PR to smhasher and play with ~100 other hash functions there.
Okay, I've quickly added the new wyhash to ahash's benchmark suite (the original C version converted to Rust with c2rust so that it can be inlined by the compiler) and reran the benchmarks; here are the results on my machine:
> It may be the fastest rust hash, but certainly not faster than other fast hashes. More like 2x slower.
If it's so much slower then most likely something is wrong.
1) Did you enable the AES instruction when compiling? (IIRC it's disabled by default)
2) When compiling the benchmark did you have cross-language inlining enabled? (IIRC you need to compile your C++ code with a specific version of Clang to get it to inline between Rust and C++)
Based on the comment in the source, the hash table doesn't handle hash collisions. And furthermore, I don't see the code which benchmarks this versus other hash tables, just the claim that it is 2-3x faster. Specifically, comparisons for different tables sizes using the same hash function would be good.
The API this presents is not really inspiring either.
I feel like you’d want something a bit safer than “we don’t store the keys and just rely on the hash to be really good” [1], putting “please do not use this for serious tasks” in a comment embedded in the header file isn’t a clear enough warning.
It’s not clear to me that that probability of collision assumptions hold. It’s basically assuming that the hashing is perfect and distributes any inputs to the full 64-bit space with uniform probability. That’s the usual hash map / randomized algorithm hope, but does BigCrush or similar avalanche testing really prove that? (Presumably not, otherwise there wouldn’t be image attacks for things like md5).
I agree that not storing keys is riskier, and it's not a risk that's mitigated even if it is a perfect uniform distribution because you can still have collisions. Maybe I don't understand the reasons for that design but personally I don't think it's a good design for a hash table. But even so the hash underlying it is a very good hash.
Big crush and smhasher give a good indication of uniform distribution, but nothing can guarantee no collisions because you're always going to have collisions it's always a possibility. Even if you have a perfect permutation of the 64 bit space, the minute you go beyond 64 bits of keys you're going to collide within 64 bits of hash.
By all the tests they run md5 is a much poorer hash than many others. But it's a or it was a cryptographic hash. It's different.
We're getting close. There's way more RAM than that in the world, and a quick search turned up a single machine from a few years ago that was only off by 160x or so.
Even without relying on the pigeonhole principle though you still have the birthday paradox. As a ballpark estimate, if you have N keys then you'd expect a 50/50 chance of at least one collision with sqrt(N) hashes -- 2^32 for this problem.
I do a similar thing with constant keys. Check the result with a simple memcmp/strcmp of the key for false positives. But only usable for constant perfect hashes.
64 bit is not good enough for that claim. 128 would be good, 256 perfect for production use. Regardless of the hash function quality. wyhash is a very good hash, the best and fastest portable one.
> “we don’t store the keys and just rely on the hash to be really good”
I think that's fine if you actually use a really good hash (e.g. a 128-bit cryptographic PRF). But I wouldn't be comfortable with hashes shorter than that or which aren't crypto quality.
People already make such assumptions when assuming uniqueness v4 UUIDs.
> It’s not clear to me that that probability of collision assumptions hold. It’s basically assuming that the hashing is perfect and distributes any inputs to the full 64-bit space with uniform probability.
You can get this guarantee by using a random hash function if you don't support insertion and are fine with using a relatively larger amounts of memory.
The meow 0.4 was faster at short keys, but failed at the smhasher "LongNeighborTest" [1]. However, doubling the AES rounds makes it pass that test. Two rounds is enough for full diffusion in AES [2]. I recently looked at computing 4 Meow keys per hash function [3], and found the speedup to be almost 2x in a microbench. That puts it in rare territory for hash speed.
[2] Section 5.4 of Introduction to Cryptography by Trappe and Washington -- It can be shown that two rounds are sufficient to obtain full diffusion, namely, each of the 128 output bits depends on each of the 128 input bits.
I had an assignment in college many years ago that was partially graded on performance. It amounted to basically a hash table with a bunch of lookups. To optimize mine I had it not store the keys, fortunately I was lucky and the test data they used didn’t cause any collisions. As a result mine was the fastest in the class :)
With respect to wyrand, it seems that all prime numbers are not created equal. I implemented wyrand() using the two primes just below 2^64. The upper 53 bits of each 64-bit random deviate was used to generate a uniform [0,1) floating point deviate. The expected value of the sum of the uniform floating point deviates is 0.5 * #deviates. When using the two primes numbers above, the resulting value was 0.6430236 * #deviates - indicating significant bias in the random deviates generated by wyhash for those particular prime numbers.
41 comments
[ 2.8 ms ] story [ 91.7 ms ] threadA what now? https://github.com/wangyi-fudan/wyhash/blob/master/wyhash.h#...
https://github.com/tkaitchuck/aHash/blob/master/compare/read...
Did anything change?
https://github.com/tkaitchuck/aHash/tree/master/smhasher
There are also ahash's own benchmarks here:
https://github.com/tkaitchuck/aHash/blob/master/compare/test...
They use the wyhash Rust crate, so if wyhash itself was updated doing a head to head comparison would boil down to updating the wyhash crate and rerunning ahash's benchmark suite.
Very interesting is his claim to create wyhash collisions at will. Even with bad keys, not bad seeds!
It may be the fastest rust hash, but certainly not faster than other fast hashes. More like 2x slower.
xxh3, t1ha0, wyhash are all much faster on the 2 machines I tested it on, an old 7 years old Intel i5-2300, and a new Ryzen 3200U.
If it's so much slower then most likely something is wrong.
1) Did you enable the AES instruction when compiling? (IIRC it's disabled by default) 2) When compiling the benchmark did you have cross-language inlining enabled? (IIRC you need to compile your C++ code with a specific version of Clang to get it to inline between Rust and C++)
https://news.ycombinator.com/item?id=19357895
The API this presents is not really inspiring either.
It’s not clear to me that that probability of collision assumptions hold. It’s basically assuming that the hashing is perfect and distributes any inputs to the full 64-bit space with uniform probability. That’s the usual hash map / randomized algorithm hope, but does BigCrush or similar avalanche testing really prove that? (Presumably not, otherwise there wouldn’t be image attacks for things like md5).
[1] https://github.com/wangyi-fudan/wyhash/blob/d2a305811972f391...
Big crush and smhasher give a good indication of uniform distribution, but nothing can guarantee no collisions because you're always going to have collisions it's always a possibility. Even if you have a perfect permutation of the 64 bit space, the minute you go beyond 64 bits of keys you're going to collide within 64 bits of hash.
By all the tests they run md5 is a much poorer hash than many others. But it's a or it was a cryptographic hash. It's different.
Even without relying on the pigeonhole principle though you still have the birthday paradox. As a ballpark estimate, if you have N keys then you'd expect a 50/50 chance of at least one collision with sqrt(N) hashes -- 2^32 for this problem.
64 bit is not good enough for that claim. 128 would be good, 256 perfect for production use. Regardless of the hash function quality. wyhash is a very good hash, the best and fastest portable one.
I think that's fine if you actually use a really good hash (e.g. a 128-bit cryptographic PRF). But I wouldn't be comfortable with hashes shorter than that or which aren't crypto quality.
People already make such assumptions when assuming uniqueness v4 UUIDs.
You can get this guarantee by using a random hash function if you don't support insertion and are fine with using a relatively larger amounts of memory.
And there are a bunch of other good suggestions here in the comments looking at around the same 50-60gb/sec speed
[1] https://github.com/injinj/smhasher/
[2] Section 5.4 of Introduction to Cryptography by Trappe and Washington -- It can be shown that two rounds are sufficient to obtain full diffusion, namely, each of the 128 output bits depends on each of the 128 input bits.
[3] https://github.com/raitechnology/raikv/blob/3ce2b23e0d9853fe...