├── README.md ├── plot128.gnuplot ├── plot128.svg ├── plot128lat.gnuplot ├── plot128lat.svg ├── plot64.gnuplot ├── plot64.svg ├── plot64lat.gnuplot ├── plot64lat.svg └── src ├── .gitignore ├── AESHash.h ├── AESTest.cpp ├── AvalancheTest.cpp ├── AvalancheTest.h ├── Birthday.cpp ├── Birthday.h ├── Bitslice.cpp ├── Bitvec.cpp ├── Bitvec.h ├── CMakeLists.txt ├── City.cpp ├── City.h ├── CityTest.cpp ├── DifferentialTest.cpp ├── DifferentialTest.h ├── Hashes.cpp ├── Hashes.h ├── KeysetTest.cpp ├── KeysetTest.h ├── LongNeighborTest.cpp ├── LongNeighborTest.h ├── LongNeighborTest.md ├── MeowTest.cpp ├── MurmurHash1.cpp ├── MurmurHash1.h ├── MurmurHash2.cpp ├── MurmurHash2.h ├── MurmurHash3.cpp ├── MurmurHash3.h ├── PMurHash.c ├── PMurHash.h ├── Platform.cpp ├── Platform.h ├── Random.cpp ├── Random.h ├── RiskyHash.h ├── RiskyTest.cpp ├── SpeedTest.cpp ├── SpeedTest.h ├── Spooky.cpp ├── Spooky.h ├── SpookyTest.cpp ├── SpookyV2.cpp ├── SpookyV2.h ├── Stats.cpp ├── Stats.h ├── SuperFastHash.cpp ├── Types.cpp ├── Types.h ├── Xxh3Test.cpp ├── crc.cpp ├── lookup3.cpp ├── main.cpp ├── md5.cpp ├── meow_hash.h ├── pstdint.h ├── sha1.cpp ├── sha1.h ├── xxh3.h ├── xxhash.c └── xxhash.h /README.md: -------------------------------------------------------------------------------- 1 | This is smhasher with a couple of hashes I added. 2 | 3 | 1. [AESTest.cpp](src/AESTest.cpp) and [AESHash.h](src/AESHash.h) are AES based hashes, these 4 | require AES and AVX support on x64. 5 | 6 | 2. [RiskyTest.cpp](src/RiskyTest.cpp) and [RiskyHash.h](src/RiskyHash.h) are hashes derived from 7 | [facil.io](https://github.com/boazsegev/facil.io) located in [fio.h](https://github.com/boazsegev/facil.io/blob/master/lib/facil/fio.h). 8 | 9 | 3. [Xxh3Test.cpp](src/Xxh3Test.cpp) and [xxh3.h](src/xxh3.h), [xxhash.c](src/xxhash.c), [xxhash.h](src/xxhash.h) are 10 | hashes from [Cyan4973](https://github.com/Cyan4973/xxHash) (updated July 2020). 11 | 12 | 4. [MeowTest.cpp](src/MeowTest.cpp) and [meow_hash.h](src/meow_hash.h) are 13 | hashes from [cmuratori](https://github.com/injinj/meow_hash) (added 2nd round July 2020). 14 | 15 | With MeowHash, which is AES based, I've found that a SIMD hashing can 16 | accerlerate the hash throughput. Computing 4 hashes at a time can almost 17 | double the hash rate, expecially when they keys are the same size. These 18 | functions in this file, 19 | [key_hash.cpp](https://github.com/raitechnology/raikv/blob/3ce2b23e0d9853fe4babcd9127d81faa1ebdfe86/src/key_hash.c#L1690) 20 | is where the code lives, and 21 | [hash_test.cpp](https://github.com/raitechnology/raikv/blob/3ce2b23e0d9853fe4babcd9127d81faa1ebdfe86/test/hash_test.cpp#L277) 22 | is an example of the usage (Mar 10 2021). 23 | 24 | I merged the LongNeighborTest from [hmakholm](https://github.com/hmakholm/smhasher): 25 | 26 | > # Fork information 27 | > 28 | > This fork of SMhasher is the home of a new test to discover 29 | > hash collisions with small Hamming distance; see [LongNeighborTest](src/LongNeighborTest.md). 30 | > 31 | > The plan is to contribute this upstream (but given that the upstream repo has 32 | > a numer of open pull requests, yet no commits since the "woo, we're on 33 | > Github" commit three years ago, holding one's breath on that is probably not 34 | > to be recommended). 35 | 36 | There are two variations of the small hash value speed test. The first is a 37 | just calling hash() in a loop. The second is calling hash() and feeding the 38 | result into hash() seed (this is also used in 39 | [rurban](https://github.com/rurban/smhasher)). This causes the result of the 40 | hash() to be needed immediately, causing the pipeline to flush. I've dubbed 41 | this a latency test, since this is probably how the hashes will be used in a 42 | program: a hash is usually used immediately after computing it (for a 43 | hashtable lookup, for example). The latency test adds a few CPU cycles to each 44 | hash function, depending on how long it takes for the pipeline to flush. This 45 | is variable for each hash, some have longer latency than others. 46 | 47 | Here are graphs for the 64 bit and 128 bit hashes as run on a Ryzen 3970X 48 | using these gnuplot scripts: [plot64.gnuplot](plot64.gnuplot) 49 | and [plot128.gnuplot](plot128.gnuplot). 50 | 51 | And these are graphs for the 64 bit and 128 bit latency as run on a Ryzen 3970X 52 | using these gnuplot scripts: [plot64lat.gnuplot](plot64lat.gnuplot) 53 | and [plot128lat.gnuplot](plot128lat.gnuplot). 54 | 55 | The cmake was changed to use the gcc flag '-march=native'. 56 | 57 | To use the plot scripts, load them in gnuplot like this (I'm using Fedora 32): 58 | 59 | ```console 60 | [5875]; git clone https://github.com/injinj/smhasher 61 | [5876]; cd smhasher 62 | [5877]; cmake src 63 | [5878]; make 64 | [5879]; gnuplot-qt 65 | 66 | G N U P L O T 67 | Version 5.0 patchlevel 6 last modified 2017-03-18 68 | 69 | Copyright (C) 1986-1993, 1998, 2004, 2007-2017 70 | Thomas Williams, Colin Kelley and many others 71 | 72 | gnuplot home: http://www.gnuplot.info 73 | faq, bugs, etc: type "help FAQ" 74 | immediate help: type "help" (plot window: hit 'h') 75 | 76 | Terminal type set to 'qt' 77 | gnuplot> load "plot64.gnuplot" 78 | donothing64 79 | City64 80 | Spooky64 81 | AES64 82 | Risky64 83 | Murmur2B 84 | Murmur2C 85 | ``` 86 | 87 | ![plot64](plot64.svg) 88 | 89 | ![plot128](plot128.svg) 90 | 91 | ![plot64lat](plot64lat.svg) 92 | 93 | ![plot128lat](plot128lat.svg) 94 | 95 | Original SMHasher Readme: 96 | 97 | ## [SMHasher](https://github.com/aappleby/smhasher/wiki) is a test suite designed to test the distribution, collision, and performance properties of non-cryptographic hash functions. 98 | 99 | This is the home for the [MurmurHash](https://github.com/aappleby/smhasher/tree/master/src) family of hash functions along with the [SMHasher](https://github.com/aappleby/smhasher/tree/master/src) test suite used to verify them. SMHasher is released under the MIT license. All MurmurHash versions are public domain software, and the author disclaims all copyright to their code. 100 | 101 | SMHasher is a test suite designed to test the distribution, collision, and performance properties of non-cryptographic hash functions - it aims to be the [DieHarder](http://www.phy.duke.edu/~rgb/General/dieharder.php) of hash testing, and does a pretty good job of finding flaws with a number of popular hashes. 102 | 103 | The SMHasher suite also includes [MurmurHash3](https://github.com/aappleby/smhasher/blob/master/src/MurmurHash3.cpp), which is the latest version in the series of MurmurHash functions - the new version is faster, more robust, and its variants can produce 32- and 128-bit hash values efficiently on both x86 and x64 platforms. 104 | 105 | 106 | ## Updates 107 | 108 | ### 1/8/2016 109 | 110 | Woo, we're on Github! I've been putting off the migration for a few, uh, years or so, but hopefully Github won't be shutting down any time soon so I don't have to move things again. MurmurHash is still used all over the place, SMHasher is still the defacto hash function test suite, and there are all sorts of interesting new hash functions out there that improve bulk hashing speed and use new shiny hardware instructions for AES and whatnot. Interesting times. :) 111 | 112 | I've copied the few wiki pages from code.google.com/p/smhasher to this wiki, though I haven't reformatted them to Markdown yet. The MurmurHash page on Wikipedia should also be linking here now as well. Feel free to send me pull requests. 113 | -------------------------------------------------------------------------------- /plot128.gnuplot: -------------------------------------------------------------------------------- 1 | #!/usr/bin/gnuplot 2 | 3 | !echo donothing128 4 | !taskset -c 1 ./SMHasher donothing128 -s -n > donothing128.txt 5 | !tail -1 donothing128.txt | awk '{ printf "%s %s donothing128\n", $1, $2 }' > donothing128label.txt 6 | !echo City128 7 | !taskset -c 1 ./SMHasher City128 -s -n > City128.txt 8 | !tail -1 City128.txt | awk '{ printf "%s %s City128\n", $1, $2 }' > City128label.txt 9 | !echo Spooky128 10 | !taskset -c 1 ./SMHasher Spooky128 -s -n > Spooky128.txt 11 | !tail -1 Spooky128.txt | awk '{ printf "%s %s Spooky128\n", $1, $2 }' > Spooky128label.txt 12 | !echo AES128 13 | !taskset -c 1 ./SMHasher AES128 -s -n > AES128.txt 14 | !tail -1 AES128.txt | awk '{ printf "%s %s AES128\n", $1, $2 }' > AES128label.txt 15 | !echo Risky128 16 | !taskset -c 1 ./SMHasher Risky128 -s -n > Risky128.txt 17 | !tail -1 Risky128.txt | awk '{ printf "%s %s Risky128\n", $1, $2 }' > Risky128label.txt 18 | !echo XXH3128 19 | !taskset -c 1 ./SMHasher XXH3128 -s -n > XXH3128.txt 20 | !tail -1 XXH3128.txt | awk '{ printf "%s %s XXH3128\n", $1, $2 }' > XXH3128label.txt 21 | !echo Meow128 22 | !taskset -c 1 ./SMHasher Meow128 -s -n > Meow128.txt 23 | !tail -1 Meow128.txt | awk '{ printf "%s %s Meow128\n", $1, $2 }' > Meow128label.txt 24 | !echo Murmur3C 25 | !taskset -c 1 ./SMHasher Murmur3C -s -n > Murmur3C.txt 26 | !tail -1 Murmur3C.txt | awk '{ printf "%s %s Murmur3C\n", $1, $2 }' > Murmur3Clabel.txt 27 | !echo Murmur3F 28 | !taskset -c 1 ./SMHasher Murmur3F -s -n > Murmur3F.txt 29 | !tail -1 Murmur3F.txt | awk '{ printf "%s %s Murmur3F\n", $1, $2 }' > Murmur3Flabel.txt 30 | 31 | set title "SMHasher 128-bit hashes" 32 | set ylabel "CPU cycles / hash" 33 | set xlabel "Key length bytes" 34 | unset key 35 | set grid 36 | 37 | plot "donothing128.txt" with linespoints, \ 38 | "donothing128label.txt" with labels offset +7,0, \ 39 | "City128.txt" with linespoints, \ 40 | "City128label.txt" with labels offset +5,0, \ 41 | "Spooky128.txt" with linespoints, \ 42 | "Spooky128label.txt" with labels offset +6,0, \ 43 | "AES128.txt" with linespoints, \ 44 | "AES128label.txt" with labels offset +5,0, \ 45 | "Risky128.txt" with linespoints, \ 46 | "Risky128label.txt" with labels offset +5,0, \ 47 | "XXH3128.txt" with linespoints, \ 48 | "XXH3128label.txt" with labels offset +5,0, \ 49 | "Meow128.txt" with linespoints, \ 50 | "Meow128label.txt" with labels offset +5,0, \ 51 | "Murmur3C.txt" with linespoints, \ 52 | "Murmur3Clabel.txt" with labels offset +6,0, \ 53 | "Murmur3F.txt" with linespoints, \ 54 | "Murmur3Flabel.txt" with labels offset +6,0 55 | -------------------------------------------------------------------------------- /plot128lat.gnuplot: -------------------------------------------------------------------------------- 1 | #!/usr/bin/gnuplot 2 | 3 | !echo donothing128 4 | !taskset -c 1 ./SMHasher donothing128 -S -n > donothing128.txt 5 | !tail -1 donothing128.txt | awk '{ printf "%s %s donothing128\n", $1, $2 }' > donothing128label.txt 6 | !echo City128 7 | !taskset -c 1 ./SMHasher City128 -S -n > City128.txt 8 | !tail -1 City128.txt | awk '{ printf "%s %s City128\n", $1, $2 }' > City128label.txt 9 | !echo Spooky128 10 | !taskset -c 1 ./SMHasher Spooky128 -S -n > Spooky128.txt 11 | !tail -1 Spooky128.txt | awk '{ printf "%s %s Spooky128\n", $1, $2 }' > Spooky128label.txt 12 | !echo AES128 13 | !taskset -c 1 ./SMHasher AES128 -S -n > AES128.txt 14 | !tail -1 AES128.txt | awk '{ printf "%s %s AES128\n", $1, $2 }' > AES128label.txt 15 | !echo Risky128 16 | !taskset -c 1 ./SMHasher Risky128 -S -n > Risky128.txt 17 | !tail -1 Risky128.txt | awk '{ printf "%s %s Risky128\n", $1, $2 }' > Risky128label.txt 18 | !echo XXH3128 19 | !taskset -c 1 ./SMHasher XXH3128 -S -n > XXH3128.txt 20 | !tail -1 XXH3128.txt | awk '{ printf "%s %s XXH3128\n", $1, $2 }' > XXH3128label.txt 21 | !echo Meow128 22 | !taskset -c 1 ./SMHasher Meow128 -S -n > Meow128.txt 23 | !tail -1 Meow128.txt | awk '{ printf "%s %s Meow128\n", $1, $2 }' > Meow128label.txt 24 | !echo Murmur3C 25 | !taskset -c 1 ./SMHasher Murmur3C -S -n > Murmur3C.txt 26 | !tail -1 Murmur3C.txt | awk '{ printf "%s %s Murmur3C\n", $1, $2 }' > Murmur3Clabel.txt 27 | !echo Murmur3F 28 | !taskset -c 1 ./SMHasher Murmur3F -S -n > Murmur3F.txt 29 | !tail -1 Murmur3F.txt | awk '{ printf "%s %s Murmur3F\n", $1, $2 }' > Murmur3Flabel.txt 30 | 31 | set title "SMHasher 128-bit latency" 32 | set ylabel "CPU cycles / hash" 33 | set xlabel "Key length bytes" 34 | unset key 35 | set grid 36 | 37 | plot "donothing128.txt" with linespoints, \ 38 | "donothing128label.txt" with labels offset +7,0, \ 39 | "City128.txt" with linespoints, \ 40 | "City128label.txt" with labels offset +5,0, \ 41 | "Spooky128.txt" with linespoints, \ 42 | "Spooky128label.txt" with labels offset +6,0, \ 43 | "AES128.txt" with linespoints, \ 44 | "AES128label.txt" with labels offset +5,0, \ 45 | "Risky128.txt" with linespoints, \ 46 | "Risky128label.txt" with labels offset +5,-1, \ 47 | "XXH3128.txt" with linespoints, \ 48 | "XXH3128label.txt" with labels offset +5,0, \ 49 | "Meow128.txt" with linespoints, \ 50 | "Meow128label.txt" with labels offset +5,0, \ 51 | "Murmur3C.txt" with linespoints, \ 52 | "Murmur3Clabel.txt" with labels offset +6,0, \ 53 | "Murmur3F.txt" with linespoints, \ 54 | "Murmur3Flabel.txt" with labels offset +6,0 55 | -------------------------------------------------------------------------------- /plot64.gnuplot: -------------------------------------------------------------------------------- 1 | #!/usr/bin/gnuplot 2 | 3 | !echo donothing64 4 | !taskset -c 1 ./SMHasher donothing64 -s -n > donothing64.txt 5 | !tail -1 donothing64.txt | awk '{ printf "%s %s donothing64\n", $1, $2 }' > donothing64label.txt 6 | !echo City64 7 | !taskset -c 1 ./SMHasher City64 -s -n > City64.txt 8 | !tail -1 City64.txt | awk '{ printf "%s %s City64\n", $1, $2 }' > City64label.txt 9 | !echo Spooky64 10 | !taskset -c 1 ./SMHasher Spooky64 -s -n > Spooky64.txt 11 | !tail -1 Spooky64.txt | awk '{ printf "%s %s Spooky64\n", $1, $2 }' > Spooky64label.txt 12 | !echo AES64 13 | !taskset -c 1 ./SMHasher AES64 -s -n > AES64.txt 14 | !tail -1 AES64.txt | awk '{ printf "%s %s AES64\n", $1, $2 }' > AES64label.txt 15 | !echo Risky64 16 | !taskset -c 1 ./SMHasher Risky64 -s -n > Risky64.txt 17 | !tail -1 Risky64.txt | awk '{ printf "%s %s Risky64\n", $1, $2 }' > Risky64label.txt 18 | !echo XXH364 19 | !taskset -c 1 ./SMHasher XXH364 -s -n > XXH364.txt 20 | !tail -1 XXH364.txt | awk '{ printf "%s %s XXH364\n", $1, $2 }' > XXH364label.txt 21 | !echo Meow64 22 | !taskset -c 1 ./SMHasher Meow64 -s -n > Meow64.txt 23 | !tail -1 Meow64.txt | awk '{ printf "%s %s Meow64\n", $1, $2 }' > Meow64label.txt 24 | !echo Murmur2B 25 | !taskset -c 1 ./SMHasher Murmur2B -s -n > Murmur2B.txt 26 | !tail -1 Murmur2B.txt | awk '{ printf "%s %s Murmur2B\n", $1, $2 }' > Murmur2Blabel.txt 27 | !echo Murmur2C 28 | !taskset -c 1 ./SMHasher Murmur2C -s -n > Murmur2C.txt 29 | !tail -1 Murmur2C.txt | awk '{ printf "%s %s Murmur2C\n", $1, $2 }' > Murmur2Clabel.txt 30 | 31 | set title "SMHasher 64-bit hashes" 32 | set ylabel "CPU cycles / hash" 33 | set xlabel "Key length bytes" 34 | unset key 35 | set grid 36 | 37 | plot "donothing64.txt" with linespoints, \ 38 | "donothing64label.txt" with labels offset +8,0, \ 39 | "City64.txt" with linespoints, \ 40 | "City64label.txt" with labels offset +5,0, \ 41 | "Spooky64.txt" with linespoints, \ 42 | "Spooky64label.txt" with labels offset +6,0, \ 43 | "AES64.txt" with linespoints, \ 44 | "AES64label.txt" with labels offset +4,0, \ 45 | "Risky64.txt" with linespoints, \ 46 | "Risky64label.txt" with labels offset +5,0, \ 47 | "XXH364.txt" with linespoints, \ 48 | "XXH364label.txt" with labels offset +5,0, \ 49 | "Meow64.txt" with linespoints, \ 50 | "Meow64label.txt" with labels offset +5,+1, \ 51 | "Murmur2B.txt" with linespoints, \ 52 | "Murmur2Blabel.txt" with labels offset +6,0, \ 53 | "Murmur2C.txt" with linespoints, \ 54 | "Murmur2Clabel.txt" with labels offset +6,0 55 | -------------------------------------------------------------------------------- /plot64lat.gnuplot: -------------------------------------------------------------------------------- 1 | #!/usr/bin/gnuplot 2 | 3 | !echo donothing64 4 | !taskset -c 1 ./SMHasher donothing64 -S -n > donothing64.txt 5 | !tail -1 donothing64.txt | awk '{ printf "%s %s donothing64\n", $1, $2 }' > donothing64label.txt 6 | !echo City64 7 | !taskset -c 1 ./SMHasher City64 -S -n > City64.txt 8 | !tail -1 City64.txt | awk '{ printf "%s %s City64\n", $1, $2 }' > City64label.txt 9 | !echo Spooky64 10 | !taskset -c 1 ./SMHasher Spooky64 -S -n > Spooky64.txt 11 | !tail -1 Spooky64.txt | awk '{ printf "%s %s Spooky64\n", $1, $2 }' > Spooky64label.txt 12 | !echo AES64 13 | !taskset -c 1 ./SMHasher AES64 -S -n > AES64.txt 14 | !tail -1 AES64.txt | awk '{ printf "%s %s AES64\n", $1, $2 }' > AES64label.txt 15 | !echo Risky64 16 | !taskset -c 1 ./SMHasher Risky64 -S -n > Risky64.txt 17 | !tail -1 Risky64.txt | awk '{ printf "%s %s Risky64\n", $1, $2 }' > Risky64label.txt 18 | !echo XXH364 19 | !taskset -c 1 ./SMHasher XXH364 -S -n > XXH364.txt 20 | !tail -1 XXH364.txt | awk '{ printf "%s %s XXH364\n", $1, $2 }' > XXH364label.txt 21 | !echo Meow64 22 | !taskset -c 1 ./SMHasher Meow64 -S -n > Meow64.txt 23 | !tail -1 Meow64.txt | awk '{ printf "%s %s Meow64\n", $1, $2 }' > Meow64label.txt 24 | !echo Murmur2B 25 | !taskset -c 1 ./SMHasher Murmur2B -S -n > Murmur2B.txt 26 | !tail -1 Murmur2B.txt | awk '{ printf "%s %s Murmur2B\n", $1, $2 }' > Murmur2Blabel.txt 27 | !echo Murmur2C 28 | !taskset -c 1 ./SMHasher Murmur2C -S -n > Murmur2C.txt 29 | !tail -1 Murmur2C.txt | awk '{ printf "%s %s Murmur2C\n", $1, $2 }' > Murmur2Clabel.txt 30 | 31 | set title "SMHasher 64-bit latency" 32 | set ylabel "CPU cycles / hash" 33 | set xlabel "Key length bytes" 34 | unset key 35 | set grid 36 | 37 | plot "donothing64.txt" with linespoints, \ 38 | "donothing64label.txt" with labels offset +8,0, \ 39 | "City64.txt" with linespoints, \ 40 | "City64label.txt" with labels offset +5,0, \ 41 | "Spooky64.txt" with linespoints, \ 42 | "Spooky64label.txt" with labels offset +6,0, \ 43 | "AES64.txt" with linespoints, \ 44 | "AES64label.txt" with labels offset +4,0, \ 45 | "Risky64.txt" with linespoints, \ 46 | "Risky64label.txt" with labels offset +5,0, \ 47 | "XXH364.txt" with linespoints, \ 48 | "XXH364label.txt" with labels offset +5,0, \ 49 | "Meow64.txt" with linespoints, \ 50 | "Meow64label.txt" with labels offset +5,0, \ 51 | "Murmur2B.txt" with linespoints, \ 52 | "Murmur2Blabel.txt" with labels offset +6,0, \ 53 | "Murmur2C.txt" with linespoints, \ 54 | "Murmur2Clabel.txt" with labels offset +6,0 55 | -------------------------------------------------------------------------------- /src/.gitignore: -------------------------------------------------------------------------------- 1 | SMHasher 2 | SMHasher.exe 3 | CMakeCache.txt 4 | CMakeFiles 5 | Makefile 6 | cmake_install.cmake 7 | *.a 8 | *.o 9 | *~ 10 | 11 | -------------------------------------------------------------------------------- /src/AESHash.h: -------------------------------------------------------------------------------- 1 | #pragma GCC push_options 2 | #pragma GCC target("aes") 3 | #pragma GCC target("avx") 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | void 11 | AESHash128( const void *p, size_t sz, uint64_t *x1, uint64_t *x2 ) 12 | { 13 | __m128i data[ 8 ], hash, seed = _mm_set_epi64x( *x1 + sz, *x2 ); 14 | size_t i = 0; 15 | 16 | const uint8_t *e = &((const uint8_t *) p)[ sz ]; 17 | const uint8_t *b = (const uint8_t *) p; 18 | hash = seed; 19 | if ( b != e ) { 20 | do { 21 | uint64_t lo = 0, hi = 0; 22 | switch ( e - b ) { 23 | default: 24 | hi = *(uint64_t *) (void *) &b[ 8 ]; 25 | lo = *(uint64_t *) (void *) &b[ 0 ]; 26 | break; 27 | case 15: hi |= ( (uint64_t) b[ 14 ] ) << 56; /* FALLTHRU */ 28 | case 14: lo |= ( (uint64_t) b[ 13 ] ) << 48; /* FALLTHRU */ 29 | case 13: hi |= ( (uint64_t) b[ 12 ] ) << 48; /* FALLTHRU */ 30 | case 12: 31 | lo |= ( (uint64_t) *(const uint32_t *) (void *) &b[ 8 ] ) << 16; 32 | hi |= ( (uint64_t) *(const uint32_t *) (void *) &b[ 4 ] ) << 16; 33 | lo |= (uint64_t) *(const uint16_t *) (void *) &b[ 2 ]; 34 | hi |= (uint64_t) *(const uint16_t *) (void *) &b[ 0 ]; 35 | break; 36 | case 11: hi |= ( (uint64_t) b[ 10 ] ) << 40; /* FALLTHRU */ 37 | case 10: lo |= ( (uint64_t) b[ 9 ] ) << 32; /* FALLTHRU */ 38 | case 9: hi |= ( (uint64_t) b[ 8 ] ) << 32; /* FALLTHRU */ 39 | case 8: 40 | lo |= (uint64_t) *(const uint32_t *) (void *) &b[ 4 ]; 41 | hi |= (uint64_t) *(const uint32_t *) (void *) &b[ 0 ]; 42 | break; 43 | case 7: hi |= ( (uint64_t) b[ 6 ] ) << 24; /* FALLTHRU */ 44 | case 6: lo |= ( (uint64_t) b[ 5 ] ) << 16; /* FALLTHRU */ 45 | case 5: hi |= ( (uint64_t) b[ 4 ] ) << 16; /* FALLTHRU */ 46 | case 4: 47 | lo |= (uint64_t) *(const uint16_t *) (void *) &b[ 2 ]; 48 | hi |= (uint64_t) *(const uint16_t *) (void *) &b[ 0 ]; 49 | break; 50 | case 3: hi |= ( (uint64_t) b[ 2 ] ) << 8; /* FALLTHRU */ 51 | case 2: lo |= ( (uint64_t) b[ 1 ] ); /* FALLTHRU */ 52 | case 1: hi |= ( (uint64_t) b[ 0 ] ); 53 | break; 54 | } 55 | data[ i ] = _mm_xor_si128( _mm_set_epi64x( hi, lo ), seed ); 56 | if ( ++i == 8 ) { 57 | while ( i > 1 ) { 58 | data[ 0 ] = _mm_aesenc_si128( data[ 0 ], data[ --i ] ); 59 | data[ 0 ] = _mm_aesenc_si128( data[ 0 ], data[ i ] ); 60 | } 61 | data[ 0 ] = _mm_aesenc_si128( data[ 0 ], seed ); 62 | hash = _mm_aesenc_si128( hash, data[ 0 ] ); 63 | i = 0; 64 | } 65 | b += 16; 66 | } while ( b < e ); 67 | while ( i > 1 ) { 68 | data[ 0 ] = _mm_aesenc_si128( data[ 0 ], data[ --i ] ); 69 | data[ 0 ] = _mm_aesenc_si128( data[ 0 ], data[ i ] ); 70 | } 71 | } 72 | else { 73 | data[ 0 ] = seed; 74 | } 75 | hash = _mm_aesenc_si128( hash, data[ 0 ] ); 76 | hash = _mm_aesenc_si128( hash, seed ); 77 | hash = _mm_aesenc_si128( hash, seed ); 78 | hash = _mm_aesenc_si128( hash, seed ); 79 | *x1 = _mm_extract_epi64( hash, 0 ); 80 | *x2 = _mm_extract_epi64( hash, 1 ); 81 | } 82 | 83 | uint64_t 84 | AESHash64( const void *p, size_t sz, uint64_t seed ) 85 | { 86 | uint64_t h1 = seed, h2 = seed; 87 | AESHash128( p, sz, &h1, &h2 ); 88 | return h1; 89 | } 90 | 91 | -------------------------------------------------------------------------------- /src/AESTest.cpp: -------------------------------------------------------------------------------- 1 | #include "AESHash.h" 2 | 3 | void AESHash128_test(const void *key, int len, uint32_t seed, void *out) { 4 | uint64_t h1 = seed, h2 = seed; 5 | AESHash128(key, len, &h1, &h2); 6 | ((uint64_t*)out)[0] = h1; 7 | ((uint64_t*)out)[1] = h2; 8 | } 9 | 10 | void AESHash64_test(const void *key, int len, uint32_t seed, void *out) { 11 | uint64_t out2[ 2 ]; 12 | AESHash128_test((const char *)key,len,seed, out2); 13 | ((uint64_t*)out)[0] = out2[0]; 14 | } 15 | -------------------------------------------------------------------------------- /src/AvalancheTest.cpp: -------------------------------------------------------------------------------- 1 | #include "AvalancheTest.h" 2 | 3 | //----------------------------------------------------------------------------- 4 | 5 | void PrintAvalancheDiagram ( int x, int y, int reps, double scale, int * bins ) 6 | { 7 | const char * symbols = ".123456789X"; 8 | 9 | for(int i = 0; i < y; i++) 10 | { 11 | printf("["); 12 | for(int j = 0; j < x; j++) 13 | { 14 | int k = (y - i) -1; 15 | 16 | int bin = bins[k + (j*y)]; 17 | 18 | double b = double(bin) / double(reps); 19 | b = fabs(b*2 - 1); 20 | 21 | b *= scale; 22 | 23 | int s = (int)floor(b*10); 24 | 25 | if(s > 10) s = 10; 26 | if(s < 0) s = 0; 27 | 28 | printf("%c",symbols[s]); 29 | } 30 | 31 | printf("]\n"); 32 | } 33 | } 34 | 35 | //---------------------------------------------------------------------------- 36 | 37 | double maxBias ( std::vector & counts, int reps ) 38 | { 39 | double worst = 0; 40 | 41 | for(int i = 0; i < (int)counts.size(); i++) 42 | { 43 | double c = double(counts[i]) / double(reps); 44 | 45 | double d = fabs(c * 2 - 1); 46 | 47 | if(d > worst) 48 | { 49 | worst = d; 50 | } 51 | } 52 | 53 | return worst; 54 | } 55 | 56 | //----------------------------------------------------------------------------- 57 | -------------------------------------------------------------------------------- /src/AvalancheTest.h: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // Flipping a single bit of a key should cause an "avalanche" of changes in 3 | // the hash function's output. Ideally, each output bits should flip 50% of 4 | // the time - if the probability of an output bit flipping is not 50%, that bit 5 | // is "biased". Too much bias means that patterns applied to the input will 6 | // cause "echoes" of the patterns in the output, which in turn can cause the 7 | // hash function to fail to create an even, random distribution of hash values. 8 | 9 | 10 | #pragma once 11 | 12 | #include "Types.h" 13 | #include "Random.h" 14 | 15 | #include 16 | #include 17 | #include 18 | 19 | // Avalanche fails if a bit is biased by more than 1% 20 | 21 | #define AVALANCHE_FAIL 0.01 22 | 23 | double maxBias ( std::vector & counts, int reps ); 24 | 25 | //----------------------------------------------------------------------------- 26 | 27 | template < typename keytype, typename hashtype > 28 | void calcBias ( pfHash hash, std::vector & counts, int reps, Rand & r ) 29 | { 30 | const int keybytes = sizeof(keytype); 31 | const int hashbytes = sizeof(hashtype); 32 | 33 | const int keybits = keybytes * 8; 34 | const int hashbits = hashbytes * 8; 35 | 36 | keytype K; 37 | hashtype A,B; 38 | 39 | for(int irep = 0; irep < reps; irep++) 40 | { 41 | if(irep % (reps/10) == 0) printf("."); 42 | 43 | r.rand_p(&K,keybytes); 44 | 45 | hash(&K,keybytes,0,&A); 46 | 47 | int * cursor = &counts[0]; 48 | 49 | for(int iBit = 0; iBit < keybits; iBit++) 50 | { 51 | flipbit(&K,keybytes,iBit); 52 | hash(&K,keybytes,0,&B); 53 | flipbit(&K,keybytes,iBit); 54 | 55 | for(int iOut = 0; iOut < hashbits; iOut++) 56 | { 57 | int bitA = getbit(&A,hashbytes,iOut); 58 | int bitB = getbit(&B,hashbytes,iOut); 59 | 60 | (*cursor++) += (bitA ^ bitB); 61 | } 62 | } 63 | } 64 | } 65 | 66 | //----------------------------------------------------------------------------- 67 | 68 | template < typename keytype, typename hashtype > 69 | bool AvalancheTest ( pfHash hash, const int reps ) 70 | { 71 | Rand r(48273); 72 | 73 | const int keybytes = sizeof(keytype); 74 | const int hashbytes = sizeof(hashtype); 75 | 76 | const int keybits = keybytes * 8; 77 | const int hashbits = hashbytes * 8; 78 | 79 | printf("Testing %3d-bit keys -> %3d-bit hashes, %8d reps",keybits,hashbits,reps); 80 | 81 | //---------- 82 | 83 | std::vector bins(keybits*hashbits,0); 84 | 85 | calcBias(hash,bins,reps,r); 86 | 87 | //---------- 88 | 89 | bool result = true; 90 | 91 | double b = maxBias(bins,reps); 92 | 93 | printf(" worst bias is %f%%",b * 100.0); 94 | 95 | if(b > AVALANCHE_FAIL) 96 | { 97 | printf(" !!!!! "); 98 | result = false; 99 | } 100 | 101 | printf("\n"); 102 | 103 | return result; 104 | } 105 | 106 | //---------------------------------------------------------------------------- 107 | // Tests the Bit Independence Criteron. Stricter than Avalanche, but slow and 108 | // not really all that useful. 109 | 110 | template< typename keytype, typename hashtype > 111 | void BicTest ( pfHash hash, const int keybit, const int reps, double & maxBias, int & maxA, int & maxB, bool verbose ) 112 | { 113 | Rand r(11938); 114 | 115 | const int keybytes = sizeof(keytype); 116 | const int hashbytes = sizeof(hashtype); 117 | const int hashbits = hashbytes * 8; 118 | 119 | std::vector bins(hashbits*hashbits*4,0); 120 | 121 | keytype key; 122 | hashtype h1,h2; 123 | 124 | for(int irep = 0; irep < reps; irep++) 125 | { 126 | if(verbose) 127 | { 128 | if(irep % (reps/10) == 0) printf("."); 129 | } 130 | 131 | r.rand_p(&key,keybytes); 132 | hash(&key,keybytes,0,&h1); 133 | 134 | flipbit(key,keybit); 135 | hash(&key,keybytes,0,&h2); 136 | 137 | hashtype d = h1 ^ h2; 138 | 139 | for(int out1 = 0; out1 < hashbits; out1++) 140 | for(int out2 = 0; out2 < hashbits; out2++) 141 | { 142 | if(out1 == out2) continue; 143 | 144 | uint32_t b = getbit(d,out1) | (getbit(d,out2) << 1); 145 | 146 | bins[(out1 * hashbits + out2) * 4 + b]++; 147 | } 148 | } 149 | 150 | if(verbose) printf("\n"); 151 | 152 | maxBias = 0; 153 | 154 | for(int out1 = 0; out1 < hashbits; out1++) 155 | { 156 | for(int out2 = 0; out2 < hashbits; out2++) 157 | { 158 | if(out1 == out2) 159 | { 160 | if(verbose) printf("\\"); 161 | continue; 162 | } 163 | 164 | double bias = 0; 165 | 166 | for(int b = 0; b < 4; b++) 167 | { 168 | double b2 = double(bins[(out1 * hashbits + out2) * 4 + b]) / double(reps / 2); 169 | b2 = fabs(b2 * 2 - 1); 170 | 171 | if(b2 > bias) bias = b2; 172 | } 173 | 174 | if(bias > maxBias) 175 | { 176 | maxBias = bias; 177 | maxA = out1; 178 | maxB = out2; 179 | } 180 | 181 | if(verbose) 182 | { 183 | if (bias < 0.01) printf("."); 184 | else if(bias < 0.05) printf("o"); 185 | else if(bias < 0.33) printf("O"); 186 | else printf("X"); 187 | } 188 | } 189 | 190 | if(verbose) printf("\n"); 191 | } 192 | } 193 | 194 | //---------- 195 | 196 | template< typename keytype, typename hashtype > 197 | bool BicTest ( pfHash hash, const int reps ) 198 | { 199 | const int keybytes = sizeof(keytype); 200 | const int keybits = keybytes * 8; 201 | 202 | double maxBias = 0; 203 | int maxK = 0; 204 | int maxA = 0; 205 | int maxB = 0; 206 | 207 | for(int i = 0; i < keybits; i++) 208 | { 209 | if(i % (keybits/10) == 0) printf("."); 210 | 211 | double bias; 212 | int a,b; 213 | 214 | BicTest(hash,i,reps,bias,a,b,true); 215 | 216 | if(bias > maxBias) 217 | { 218 | maxBias = bias; 219 | maxK = i; 220 | maxA = a; 221 | maxB = b; 222 | } 223 | } 224 | 225 | printf("Max bias %f - (%3d : %3d,%3d)\n",maxBias,maxK,maxA,maxB); 226 | 227 | // Bit independence is harder to pass than avalanche, so we're a bit more lax here. 228 | 229 | bool result = (maxBias < 0.05); 230 | 231 | return result; 232 | } 233 | 234 | //----------------------------------------------------------------------------- 235 | // BIC test variant - store all intermediate data in a table, draw diagram 236 | // afterwards (much faster) 237 | 238 | template< typename keytype, typename hashtype > 239 | void BicTest3 ( pfHash hash, const int reps, bool verbose = true ) 240 | { 241 | const int keybytes = sizeof(keytype); 242 | const int keybits = keybytes * 8; 243 | const int hashbytes = sizeof(hashtype); 244 | const int hashbits = hashbytes * 8; 245 | const int pagesize = hashbits*hashbits*4; 246 | 247 | Rand r(11938); 248 | 249 | double maxBias = 0; 250 | int maxK = 0; 251 | int maxA = 0; 252 | int maxB = 0; 253 | 254 | keytype key; 255 | hashtype h1,h2; 256 | 257 | std::vector bins(keybits*pagesize,0); 258 | 259 | for(int keybit = 0; keybit < keybits; keybit++) 260 | { 261 | if(keybit % (keybits/10) == 0) printf("."); 262 | 263 | int * page = &bins[keybit*pagesize]; 264 | 265 | for(int irep = 0; irep < reps; irep++) 266 | { 267 | r.rand_p(&key,keybytes); 268 | hash(&key,keybytes,0,&h1); 269 | flipbit(key,keybit); 270 | hash(&key,keybytes,0,&h2); 271 | 272 | hashtype d = h1 ^ h2; 273 | 274 | for(int out1 = 0; out1 < hashbits-1; out1++) 275 | for(int out2 = out1+1; out2 < hashbits; out2++) 276 | { 277 | int * b = &page[(out1*hashbits+out2)*4]; 278 | 279 | uint32_t x = getbit(d,out1) | (getbit(d,out2) << 1); 280 | 281 | b[x]++; 282 | } 283 | } 284 | } 285 | 286 | printf("\n"); 287 | 288 | for(int out1 = 0; out1 < hashbits-1; out1++) 289 | { 290 | for(int out2 = out1+1; out2 < hashbits; out2++) 291 | { 292 | if(verbose) printf("(%3d,%3d) - ",out1,out2); 293 | 294 | for(int keybit = 0; keybit < keybits; keybit++) 295 | { 296 | int * page = &bins[keybit*pagesize]; 297 | int * bins = &page[(out1*hashbits+out2)*4]; 298 | 299 | double bias = 0; 300 | 301 | for(int b = 0; b < 4; b++) 302 | { 303 | double b2 = double(bins[b]) / double(reps / 2); 304 | b2 = fabs(b2 * 2 - 1); 305 | 306 | if(b2 > bias) bias = b2; 307 | } 308 | 309 | if(bias > maxBias) 310 | { 311 | maxBias = bias; 312 | maxK = keybit; 313 | maxA = out1; 314 | maxB = out2; 315 | } 316 | 317 | if(verbose) 318 | { 319 | if (bias < 0.01) printf("."); 320 | else if(bias < 0.05) printf("o"); 321 | else if(bias < 0.33) printf("O"); 322 | else printf("X"); 323 | } 324 | } 325 | 326 | // Finished keybit 327 | 328 | if(verbose) printf("\n"); 329 | } 330 | 331 | if(verbose) 332 | { 333 | for(int i = 0; i < keybits+12; i++) printf("-"); 334 | printf("\n"); 335 | } 336 | } 337 | 338 | printf("Max bias %f - (%3d : %3d,%3d)\n",maxBias,maxK,maxA,maxB); 339 | } 340 | 341 | 342 | //----------------------------------------------------------------------------- 343 | // BIC test variant - iterate over output bits, then key bits. No temp storage, 344 | // but slooooow 345 | 346 | template< typename keytype, typename hashtype > 347 | void BicTest2 ( pfHash hash, const int reps, bool verbose = true ) 348 | { 349 | const int keybytes = sizeof(keytype); 350 | const int keybits = keybytes * 8; 351 | const int hashbytes = sizeof(hashtype); 352 | const int hashbits = hashbytes * 8; 353 | 354 | Rand r(11938); 355 | 356 | double maxBias = 0; 357 | int maxK = 0; 358 | int maxA = 0; 359 | int maxB = 0; 360 | 361 | keytype key; 362 | hashtype h1,h2; 363 | 364 | for(int out1 = 0; out1 < hashbits-1; out1++) 365 | for(int out2 = out1+1; out2 < hashbits; out2++) 366 | { 367 | if(verbose) printf("(%3d,%3d) - ",out1,out2); 368 | 369 | for(int keybit = 0; keybit < keybits; keybit++) 370 | { 371 | int bins[4] = { 0, 0, 0, 0 }; 372 | 373 | for(int irep = 0; irep < reps; irep++) 374 | { 375 | r.rand_p(&key,keybytes); 376 | hash(&key,keybytes,0,&h1); 377 | flipbit(key,keybit); 378 | hash(&key,keybytes,0,&h2); 379 | 380 | hashtype d = h1 ^ h2; 381 | 382 | uint32_t b = getbit(d,out1) | (getbit(d,out2) << 1); 383 | 384 | bins[b]++; 385 | } 386 | 387 | double bias = 0; 388 | 389 | for(int b = 0; b < 4; b++) 390 | { 391 | double b2 = double(bins[b]) / double(reps / 2); 392 | b2 = fabs(b2 * 2 - 1); 393 | 394 | if(b2 > bias) bias = b2; 395 | } 396 | 397 | if(bias > maxBias) 398 | { 399 | maxBias = bias; 400 | maxK = keybit; 401 | maxA = out1; 402 | maxB = out2; 403 | } 404 | 405 | if(verbose) 406 | { 407 | if (bias < 0.05) printf("."); 408 | else if(bias < 0.10) printf("o"); 409 | else if(bias < 0.50) printf("O"); 410 | else printf("X"); 411 | } 412 | } 413 | 414 | // Finished keybit 415 | 416 | if(verbose) printf("\n"); 417 | } 418 | 419 | printf("Max bias %f - (%3d : %3d,%3d)\n",maxBias,maxK,maxA,maxB); 420 | } 421 | 422 | //----------------------------------------------------------------------------- 423 | -------------------------------------------------------------------------------- /src/Birthday.cpp: -------------------------------------------------------------------------------- 1 | #include "Birthday.h" 2 | #include 3 | 4 | double Birthday::collisionProbability(unsigned hashbits, double tests, double extras) { 5 | double hashes = exp2((double)hashbits); 6 | 7 | // First compute the probability of avoiding a collision among 8 | // the tests. The formula is 9 | // p = h! / (h-t)! / h^t 10 | // which through Stirling's approximation and a page or so of 11 | // algebra works out as 12 | // log(p) = -log(1-t/h)(h-t+1/2) - t 13 | // but we need to be careful about numeric effects. 14 | double frac = tests / hashes; 15 | double mlog; // becomes -log(1-frac) 16 | double logsuccess; // becomes log(p) 17 | if( tests >= hashes || frac >= 1.0 ) { 18 | // Pigeonhole principle says there will always be a collision 19 | return 1.0; 20 | } else if( tests < 2.5 ) { 21 | mlog = -log1p(-frac); 22 | if( tests < 1.5 ) 23 | logsuccess = 0; 24 | else 25 | logsuccess = -mlog; 26 | } else if( frac > 1e-8 ) { 27 | // In this range the main numeric risk is that mlog*h is nearly 28 | // canceled by t, so we need to be sure to subtract those before 29 | // addint the other terms in. 30 | // In principle the log1p function loses some precision if frac 31 | // is close to 1, but that would be lost anyway because the end 32 | // result rounds up to 1.0 no matter what. 33 | mlog = -log1p(-frac); 34 | logsuccess = (mlog-frac) * hashes - mlog * (tests - 0.5); 35 | } else { 36 | // Here is frac too small for the difference mlog-frac to be visible 37 | // compared to mlog. Instead of log1p we will manually approximate 38 | // log(1+x)-x ~~ -x^2/2 39 | double diff = frac*frac*0.5 ; 40 | mlog = frac + diff; 41 | logsuccess = diff * hashes - mlog * (tests - 0.5); 42 | // The threshold of 1e-8 was chosen as the largest order of 43 | // magnitude before a jump in the value between the strategies 44 | // begins to be visible in the tests below. (The small-frac 45 | // algorithm generally gives smoother output, so we'll want to 46 | // use it until the higher-order terms in the logarithm series 47 | // begin to be visible). 48 | } 49 | 50 | // Now multiply with the probability of the additional tries succeeding: 51 | // p = (1-b/h)^m 52 | logsuccess -= mlog * extras ; 53 | 54 | // Since we want the probability of failure, compute 1 - e^logsucces 55 | return -expm1(logsuccess); 56 | } 57 | 58 | // -------------------------------------------------------------------------- 59 | 60 | #ifdef INCLUDE_TEST_MAIN 61 | #include 62 | 63 | static void checkContinuity(int bits, double threshold, double span) { 64 | double hashes = exp2(bits); 65 | double prev = 0; 66 | double prevdelta = 0; 67 | double step = 2*span / 17 ; 68 | for( double frac = threshold-span ; frac <= threshold+span ; frac += step ) { 69 | double tests = ceil(hashes*frac); 70 | frac = tests/hashes ; 71 | double prob = Birthday::collisionProbability(bits, tests, 0); 72 | double delta = prob-prev; 73 | printf("%d bits, p=%.9g for %g tests, frac %g (+ %g + %g)\n", 74 | bits, prob, tests, frac, delta, delta-prevdelta); 75 | prev = prob; 76 | prevdelta = delta; 77 | } 78 | printf("\n"); 79 | } 80 | 81 | int main(int argc, const char **argv) { 82 | checkContinuity(3, 0.5, 0.5); 83 | checkContinuity(6, 0.5, 0.1); 84 | checkContinuity(7, 0.5, 0.06); 85 | checkContinuity(8, 0.5, 0.03); 86 | 87 | checkContinuity(128, 1e-29, 0.5e-29); 88 | 89 | checkContinuity(48, 1e-8, 0.0005e-8); 90 | checkContinuity(56, 1e-8, 0.0005e-8); 91 | checkContinuity(64, 1e-8, 0.0005e-8); 92 | return 0; 93 | } 94 | #endif 95 | -------------------------------------------------------------------------------- /src/Birthday.h: -------------------------------------------------------------------------------- 1 | // Assume a set of `tests` random hashes of `hashbytes` bits each are 2 | // generated, as well as a set of `extras`. What is the probability 3 | // that there is at least one collision between two test hashes or 4 | // between a test hash and an extra hash? (Collisions between two 5 | // extras do not count). 6 | 7 | class Birthday { 8 | public: 9 | static double collisionProbability(unsigned hashbits, double tests, double extras); 10 | }; 11 | -------------------------------------------------------------------------------- /src/Bitslice.cpp: -------------------------------------------------------------------------------- 1 | #include "Bitvec.h" 2 | #include 3 | #include 4 | 5 | // handle xnor 6 | 7 | typedef std::vector slice; 8 | typedef std::vector slice_vec; 9 | 10 | int countbits ( slice & v ) 11 | { 12 | int c = 0; 13 | 14 | for(size_t i = 0; i < v.size(); i++) 15 | { 16 | int d = countbits(v[i]); 17 | 18 | c += d; 19 | } 20 | 21 | return c; 22 | } 23 | 24 | int countxor ( slice & a, slice & b ) 25 | { 26 | assert(a.size() == b.size()); 27 | 28 | int c = 0; 29 | 30 | for(size_t i = 0; i < a.size(); i++) 31 | { 32 | int d = countbits(a[i] ^ b[i]); 33 | 34 | c += d; 35 | } 36 | 37 | return c; 38 | } 39 | 40 | void xoreq ( slice & a, slice & b ) 41 | { 42 | assert(a.size() == b.size()); 43 | 44 | for(size_t i = 0; i < a.size(); i++) 45 | { 46 | a[i] ^= b[i]; 47 | } 48 | } 49 | 50 | //----------------------------------------------------------------------------- 51 | // Bitslice a hash set 52 | 53 | template< typename hashtype > 54 | void Bitslice ( std::vector & hashes, slice_vec & slices ) 55 | { 56 | const int hashbytes = sizeof(hashtype); 57 | const int hashbits = hashbytes * 8; 58 | const int slicelen = ((int)hashes.size() + 31) / 32; 59 | 60 | slices.clear(); 61 | slices.resize(hashbits); 62 | 63 | for(int i = 0; i < (int)slices.size(); i++) 64 | { 65 | slices[i].resize(slicelen,0); 66 | } 67 | 68 | for(int j = 0; j < hashbits; j++) 69 | { 70 | void * sliceblob = &(slices[j][0]); 71 | 72 | for(int i = 0; i < (int)hashes.size(); i++) 73 | { 74 | int b = getbit(hashes[i],j); 75 | 76 | setbit(sliceblob,slicelen*4,i,b); 77 | } 78 | } 79 | } 80 | 81 | void FactorSlices ( slice_vec & slices ) 82 | { 83 | std::vector counts(slices.size(),0); 84 | 85 | for(size_t i = 0; i < slices.size(); i++) 86 | { 87 | counts[i] = countbits(slices[i]); 88 | } 89 | 90 | bool changed = true; 91 | 92 | while(changed) 93 | { 94 | int bestA = -1; 95 | int bestB = -1; 96 | 97 | for(int j = 0; j < (int)slices.size()-1; j++) 98 | { 99 | for(int i = j+1; i < (int)slices.size(); i++) 100 | { 101 | int d = countxor(slices[i],slices[j]); 102 | 103 | if((d < counts[i]) && (d < counts[j])) 104 | { 105 | if(counts[i] < counts[j]) 106 | { 107 | bestA = j; 108 | bestB = i; 109 | } 110 | } 111 | else if(d < counts[i]) 112 | { 113 | //bestA = 114 | } 115 | } 116 | } 117 | } 118 | } 119 | 120 | 121 | void foo ( void ) 122 | { 123 | slice a; 124 | slice_vec b; 125 | 126 | Bitslice(a,b); 127 | } -------------------------------------------------------------------------------- /src/Bitvec.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include "Platform.h" 4 | 5 | #include 6 | 7 | //----------------------------------------------------------------------------- 8 | 9 | void printbits ( const void * blob, int len ); 10 | void printhex32 ( const void * blob, int len ); 11 | void printbytes ( const void * blob, int len ); 12 | void printbytes2 ( const void * blob, int len ); 13 | 14 | uint32_t popcount ( uint32_t v ); 15 | uint32_t parity ( uint32_t v ); 16 | 17 | uint32_t getbit ( const void * blob, int len, uint32_t bit ); 18 | uint32_t getbit_wrap ( const void * blob, int len, uint32_t bit ); 19 | 20 | void setbit ( void * blob, int len, uint32_t bit ); 21 | void setbit ( void * blob, int len, uint32_t bit, uint32_t val ); 22 | 23 | void clearbit ( void * blob, int len, uint32_t bit ); 24 | 25 | void flipbit ( void * blob, int len, uint32_t bit ); 26 | 27 | int countbits ( uint32_t v ); 28 | int countbits ( std::vector & v ); 29 | 30 | int countbits ( const void * blob, int len ); 31 | 32 | void invert ( std::vector & v ); 33 | 34 | //---------- 35 | 36 | template< typename T > 37 | inline uint32_t getbit ( T & blob, uint32_t bit ) 38 | { 39 | return getbit(&blob,sizeof(blob),bit); 40 | } 41 | 42 | template<> inline uint32_t getbit ( uint32_t & blob, uint32_t bit ) { return (blob >> (bit & 31)) & 1; } 43 | template<> inline uint32_t getbit ( uint64_t & blob, uint32_t bit ) { return (blob >> (bit & 63)) & 1; } 44 | 45 | //---------- 46 | 47 | template< typename T > 48 | inline void setbit ( T & blob, uint32_t bit ) 49 | { 50 | return setbit(&blob,sizeof(blob),bit); 51 | } 52 | 53 | template<> inline void setbit ( uint32_t & blob, uint32_t bit ) { blob |= uint32_t(1) << (bit & 31); } 54 | template<> inline void setbit ( uint64_t & blob, uint32_t bit ) { blob |= uint64_t(1) << (bit & 63); } 55 | 56 | //---------- 57 | 58 | template< typename T > 59 | inline void flipbit ( T & blob, uint32_t bit ) 60 | { 61 | flipbit(&blob,sizeof(blob),bit); 62 | } 63 | 64 | template<> inline void flipbit ( uint32_t & blob, uint32_t bit ) { bit &= 31; blob ^= (uint32_t(1) << bit); } 65 | template<> inline void flipbit ( uint64_t & blob, uint32_t bit ) { bit &= 63; blob ^= (uint64_t(1) << bit); } 66 | 67 | //----------------------------------------------------------------------------- 68 | // Left and right shift of blobs. The shift(N) versions work on chunks of N 69 | // bits at a time (faster) 70 | 71 | void lshift1 ( void * blob, int len, int c ); 72 | void lshift8 ( void * blob, int len, int c ); 73 | void lshift32 ( void * blob, int len, int c ); 74 | 75 | void rshift1 ( void * blob, int len, int c ); 76 | void rshift8 ( void * blob, int len, int c ); 77 | void rshift32 ( void * blob, int len, int c ); 78 | 79 | inline void lshift ( void * blob, int len, int c ) 80 | { 81 | if((len & 3) == 0) 82 | { 83 | lshift32(blob,len,c); 84 | } 85 | else 86 | { 87 | lshift8(blob,len,c); 88 | } 89 | } 90 | 91 | inline void rshift ( void * blob, int len, int c ) 92 | { 93 | if((len & 3) == 0) 94 | { 95 | rshift32(blob,len,c); 96 | } 97 | else 98 | { 99 | rshift8(blob,len,c); 100 | } 101 | } 102 | 103 | template < typename T > 104 | inline void lshift ( T & blob, int c ) 105 | { 106 | if((sizeof(T) & 3) == 0) 107 | { 108 | lshift32(&blob,sizeof(T),c); 109 | } 110 | else 111 | { 112 | lshift8(&blob,sizeof(T),c); 113 | } 114 | } 115 | 116 | template < typename T > 117 | inline void rshift ( T & blob, int c ) 118 | { 119 | if((sizeof(T) & 3) == 0) 120 | { 121 | lshift32(&blob,sizeof(T),c); 122 | } 123 | else 124 | { 125 | lshift8(&blob,sizeof(T),c); 126 | } 127 | } 128 | 129 | template<> inline void lshift ( uint32_t & blob, int c ) { blob <<= c; } 130 | template<> inline void lshift ( uint64_t & blob, int c ) { blob <<= c; } 131 | template<> inline void rshift ( uint32_t & blob, int c ) { blob >>= c; } 132 | template<> inline void rshift ( uint64_t & blob, int c ) { blob >>= c; } 133 | 134 | //----------------------------------------------------------------------------- 135 | // Left and right rotate of blobs. The rot(N) versions work on chunks of N 136 | // bits at a time (faster) 137 | 138 | void lrot1 ( void * blob, int len, int c ); 139 | void lrot8 ( void * blob, int len, int c ); 140 | void lrot32 ( void * blob, int len, int c ); 141 | 142 | void rrot1 ( void * blob, int len, int c ); 143 | void rrot8 ( void * blob, int len, int c ); 144 | void rrot32 ( void * blob, int len, int c ); 145 | 146 | inline void lrot ( void * blob, int len, int c ) 147 | { 148 | if((len & 3) == 0) 149 | { 150 | return lrot32(blob,len,c); 151 | } 152 | else 153 | { 154 | return lrot8(blob,len,c); 155 | } 156 | } 157 | 158 | inline void rrot ( void * blob, int len, int c ) 159 | { 160 | if((len & 3) == 0) 161 | { 162 | return rrot32(blob,len,c); 163 | } 164 | else 165 | { 166 | return rrot8(blob,len,c); 167 | } 168 | } 169 | 170 | template < typename T > 171 | inline void lrot ( T & blob, int c ) 172 | { 173 | if((sizeof(T) & 3) == 0) 174 | { 175 | return lrot32(&blob,sizeof(T),c); 176 | } 177 | else 178 | { 179 | return lrot8(&blob,sizeof(T),c); 180 | } 181 | } 182 | 183 | template < typename T > 184 | inline void rrot ( T & blob, int c ) 185 | { 186 | if((sizeof(T) & 3) == 0) 187 | { 188 | return rrot32(&blob,sizeof(T),c); 189 | } 190 | else 191 | { 192 | return rrot8(&blob,sizeof(T),c); 193 | } 194 | } 195 | 196 | template<> inline void lrot ( uint32_t & blob, int c ) { blob = ROTL32(blob,c); } 197 | template<> inline void lrot ( uint64_t & blob, int c ) { blob = ROTL64(blob,c); } 198 | template<> inline void rrot ( uint32_t & blob, int c ) { blob = ROTR32(blob,c); } 199 | template<> inline void rrot ( uint64_t & blob, int c ) { blob = ROTR64(blob,c); } 200 | 201 | //----------------------------------------------------------------------------- 202 | // Bit-windowing functions - select some N-bit subset of the input blob 203 | 204 | uint32_t window1 ( void * blob, int len, int start, int count ); 205 | uint32_t window8 ( void * blob, int len, int start, int count ); 206 | uint32_t window32 ( void * blob, int len, int start, int count ); 207 | 208 | inline uint32_t window ( void * blob, int len, int start, int count ) 209 | { 210 | if(len & 3) 211 | { 212 | return window8(blob,len,start,count); 213 | } 214 | else 215 | { 216 | return window32(blob,len,start,count); 217 | } 218 | } 219 | 220 | template < typename T > 221 | inline uint32_t window ( T & blob, int start, int count ) 222 | { 223 | if((sizeof(T) & 3) == 0) 224 | { 225 | return window32(&blob,sizeof(T),start,count); 226 | } 227 | else 228 | { 229 | return window8(&blob,sizeof(T),start,count); 230 | } 231 | } 232 | 233 | template<> 234 | inline uint32_t window ( uint32_t & blob, int start, int count ) 235 | { 236 | return ROTR32(blob,start) & ((1< 240 | inline uint32_t window ( uint64_t & blob, int start, int count ) 241 | { 242 | return (uint32_t)ROTR64(blob,start) & ((1< // for size_t. 47 | #include 48 | 49 | // Microsoft Visual Studio may not have stdint.h. 50 | #if defined(_MSC_VER) && (_MSC_VER < 1600) 51 | typedef unsigned char uint8_t; 52 | typedef unsigned int uint32_t; 53 | typedef unsigned __int64 uint64_t; 54 | #else // defined(_MSC_VER) 55 | #include 56 | #endif // !defined(_MSC_VER) 57 | 58 | typedef uint8_t uint8; 59 | typedef uint32_t uint32; 60 | typedef uint64_t uint64; 61 | typedef std::pair uint128; 62 | 63 | inline uint64 Uint128Low64(const uint128& x) { return x.first; } 64 | inline uint64 Uint128High64(const uint128& x) { return x.second; } 65 | 66 | // Hash function for a byte array. 67 | uint64 CityHash64(const char *buf, size_t len); 68 | 69 | // Hash function for a byte array. For convenience, a 64-bit seed is also 70 | // hashed into the result. 71 | uint64 CityHash64WithSeed(const char *buf, size_t len, uint64 seed); 72 | 73 | // Hash function for a byte array. For convenience, two seeds are also 74 | // hashed into the result. 75 | uint64 CityHash64WithSeeds(const char *buf, size_t len, 76 | uint64 seed0, uint64 seed1); 77 | 78 | // Hash function for a byte array. 79 | uint128 CityHash128(const char *s, size_t len); 80 | 81 | // Hash function for a byte array. For convenience, a 128-bit seed is also 82 | // hashed into the result. 83 | uint128 CityHash128WithSeed(const char *s, size_t len, uint128 seed); 84 | 85 | // Hash 128 input bits down to 64 bits of output. 86 | // This is intended to be a reasonably good hash function. 87 | inline uint64 Hash128to64(const uint128& x) { 88 | // Murmur-inspired hashing. 89 | const uint64 kMul = 0x9ddfea08eb382d69ULL; 90 | uint64 a = (Uint128Low64(x) ^ Uint128High64(x)) * kMul; 91 | a ^= (a >> 47); 92 | uint64 b = (Uint128High64(x) ^ a) * kMul; 93 | b ^= (b >> 47); 94 | b *= kMul; 95 | return b; 96 | } 97 | 98 | // Conditionally include declarations for versions of City that require SSE4.2 99 | // instructions to be available. 100 | #if defined(__SSE4_2__) && defined(__x86_64__) 101 | 102 | // Hash function for a byte array. 103 | uint128 CityHashCrc128(const char *s, size_t len); 104 | 105 | // Hash function for a byte array. For convenience, a 128-bit seed is also 106 | // hashed into the result. 107 | uint128 CityHashCrc128WithSeed(const char *s, size_t len, uint128 seed); 108 | 109 | // Hash function for a byte array. Sets result[0] ... result[3]. 110 | void CityHashCrc256(const char *s, size_t len, uint64 *result); 111 | 112 | #endif // __SSE4_2__ 113 | 114 | #endif // CITY_HASH_H_ 115 | -------------------------------------------------------------------------------- /src/CityTest.cpp: -------------------------------------------------------------------------------- 1 | #include "City.h" 2 | 3 | void CityHash64_test ( const void * key, int len, uint32_t seed, void * out ) 4 | { 5 | *(uint64*)out = CityHash64WithSeed((const char *)key,len,seed); 6 | } 7 | 8 | void CityHash128_test ( const void * key, int len, uint32_t seed, void * out ) 9 | { 10 | uint128 s(0,0); 11 | 12 | s.first = seed; 13 | 14 | *(uint128*)out = CityHash128WithSeed((const char*)key,len,s); 15 | } 16 | -------------------------------------------------------------------------------- /src/DifferentialTest.cpp: -------------------------------------------------------------------------------- 1 | #include "DifferentialTest.h" 2 | 3 | //---------------------------------------------------------------------------- 4 | -------------------------------------------------------------------------------- /src/DifferentialTest.h: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // Differential collision & distribution tests - generate a bunch of random keys, 3 | // see what happens to the hash value when we flip a few bits of the key. 4 | 5 | #pragma once 6 | 7 | #include "Types.h" 8 | #include "Stats.h" // for chooseUpToK 9 | #include "KeysetTest.h" // for SparseKeygenRecurse 10 | #include "Random.h" 11 | 12 | #include 13 | #include 14 | #include 15 | 16 | //----------------------------------------------------------------------------- 17 | // Sort through the differentials, ignoring collisions that only occured once 18 | // (these could be false positives). If we find collisions of 3 or more, the 19 | // differential test fails. 20 | 21 | template < class keytype > 22 | bool ProcessDifferentials ( std::vector & diffs, int reps, bool dumpCollisions ) 23 | { 24 | std::sort(diffs.begin(), diffs.end()); 25 | 26 | int count = 1; 27 | int ignore = 0; 28 | 29 | bool result = true; 30 | 31 | if(diffs.size()) 32 | { 33 | keytype kp = diffs[0]; 34 | 35 | for(int i = 1; i < (int)diffs.size(); i++) 36 | { 37 | if(diffs[i] == kp) 38 | { 39 | count++; 40 | continue; 41 | } 42 | else 43 | { 44 | if(count > 1) 45 | { 46 | result = false; 47 | 48 | double pct = 100 * (double(count) / double(reps)); 49 | 50 | if(dumpCollisions) 51 | { 52 | printbits((unsigned char*)&kp,sizeof(kp)); 53 | printf(" - %4.2f%%\n", pct ); 54 | } 55 | } 56 | else 57 | { 58 | ignore++; 59 | } 60 | 61 | kp = diffs[i]; 62 | count = 1; 63 | } 64 | } 65 | 66 | if(count > 1) 67 | { 68 | double pct = 100 * (double(count) / double(reps)); 69 | 70 | if(dumpCollisions) 71 | { 72 | printbits((unsigned char*)&kp,sizeof(kp)); 73 | printf(" - %4.2f%%\n", pct ); 74 | } 75 | } 76 | else 77 | { 78 | ignore++; 79 | } 80 | } 81 | 82 | printf("%d total collisions, of which %d single collisions were ignored",(int)diffs.size(),ignore); 83 | 84 | if(result == false) 85 | { 86 | printf(" !!!!! "); 87 | } 88 | 89 | printf("\n"); 90 | printf("\n"); 91 | 92 | return result; 93 | } 94 | 95 | //----------------------------------------------------------------------------- 96 | // Check all possible keybits-choose-N differentials for collisions, report 97 | // ones that occur significantly more often than expected. 98 | 99 | // Random collisions can happen with probability 1 in 2^32 - if we do more than 100 | // 2^32 tests, we'll probably see some spurious random collisions, so don't report 101 | // them. 102 | 103 | template < typename keytype, typename hashtype > 104 | void DiffTestRecurse ( pfHash hash, keytype & k1, keytype & k2, hashtype & h1, hashtype & h2, int start, int bitsleft, std::vector & diffs ) 105 | { 106 | const int bits = sizeof(keytype)*8; 107 | 108 | for(int i = start; i < bits; i++) 109 | { 110 | flipbit(&k2,sizeof(k2),i); 111 | bitsleft--; 112 | 113 | hash(&k2,sizeof(k2),0,&h2); 114 | 115 | if(h1 == h2) 116 | { 117 | diffs.push_back(k1 ^ k2); 118 | } 119 | 120 | if(bitsleft) 121 | { 122 | DiffTestRecurse(hash,k1,k2,h1,h2,i+1,bitsleft,diffs); 123 | } 124 | 125 | flipbit(&k2,sizeof(k2),i); 126 | bitsleft++; 127 | } 128 | } 129 | 130 | //---------- 131 | 132 | template < typename keytype, typename hashtype > 133 | bool DiffTest ( pfHash hash, int diffbits, int reps, bool dumpCollisions ) 134 | { 135 | const int keybits = sizeof(keytype) * 8; 136 | const int hashbits = sizeof(hashtype) * 8; 137 | 138 | double diffcount = chooseUpToK(keybits,diffbits); 139 | double testcount = (diffcount * double(reps)); 140 | double expected = testcount / pow(2.0,double(hashbits)); 141 | 142 | Rand r(100); 143 | 144 | std::vector diffs; 145 | 146 | keytype k1,k2; 147 | hashtype h1,h2; 148 | 149 | printf("Testing %0.f up-to-%d-bit differentials in %d-bit keys -> %d bit hashes.\n",diffcount,diffbits,keybits,hashbits); 150 | printf("%d reps, %0.f total tests, expecting %2.2f random collisions",reps,testcount,expected); 151 | 152 | for(int i = 0; i < reps; i++) 153 | { 154 | if(i % (reps/10) == 0) printf("."); 155 | 156 | r.rand_p(&k1,sizeof(keytype)); 157 | k2 = k1; 158 | 159 | hash(&k1,sizeof(k1),0,(uint32_t*)&h1); 160 | 161 | DiffTestRecurse(hash,k1,k2,h1,h2,0,diffbits,diffs); 162 | } 163 | printf("\n"); 164 | 165 | bool result = true; 166 | 167 | result &= ProcessDifferentials(diffs,reps,dumpCollisions); 168 | 169 | return result; 170 | } 171 | 172 | //----------------------------------------------------------------------------- 173 | // Differential distribution test - for each N-bit input differential, generate 174 | // a large set of differential key pairs, hash them, and test the output 175 | // differentials using our distribution test code. 176 | 177 | // This is a very hard test to pass - even if the hash values are well-distributed, 178 | // the differences between hash values may not be. It's also not entirely relevant 179 | // for testing hash functions, but it's still interesting. 180 | 181 | // This test is a _lot_ of work, as it's essentially a full keyset test for 182 | // each of a potentially huge number of input differentials. To speed things 183 | // along, we do only a few distribution tests per keyset instead of the full 184 | // grid. 185 | 186 | // #TODO - put diagram drawing back on 187 | 188 | template < typename keytype, typename hashtype > 189 | void DiffDistTest ( pfHash hash, const int diffbits, int trials, double & worst, double & avg ) 190 | { 191 | std::vector keys(trials); 192 | std::vector A(trials),B(trials); 193 | 194 | for(int i = 0; i < trials; i++) 195 | { 196 | rand_p(&keys[i],sizeof(keytype)); 197 | 198 | hash(&keys[i],sizeof(keytype),0,(uint32_t*)&A[i]); 199 | } 200 | 201 | //---------- 202 | 203 | std::vector diffs; 204 | 205 | keytype temp(0); 206 | 207 | SparseKeygenRecurse(0,diffbits,true,temp,diffs); 208 | 209 | //---------- 210 | 211 | worst = 0; 212 | avg = 0; 213 | 214 | hashtype h2; 215 | 216 | for(size_t j = 0; j < diffs.size(); j++) 217 | { 218 | keytype & d = diffs[j]; 219 | 220 | for(int i = 0; i < trials; i++) 221 | { 222 | keytype k2 = keys[i] ^ d; 223 | 224 | hash(&k2,sizeof(k2),0,&h2); 225 | 226 | B[i] = A[i] ^ h2; 227 | } 228 | 229 | double dworst,davg; 230 | 231 | TestDistributionFast(B,dworst,davg); 232 | 233 | avg += davg; 234 | worst = (dworst > worst) ? dworst : worst; 235 | } 236 | 237 | avg /= double(diffs.size()); 238 | } 239 | 240 | //----------------------------------------------------------------------------- 241 | // Simpler differential-distribution test - for all 1-bit differentials, 242 | // generate random key pairs and run full distribution/collision tests on the 243 | // hash differentials 244 | 245 | template < typename keytype, typename hashtype > 246 | bool DiffDistTest2 ( pfHash hash, bool drawDiagram ) 247 | { 248 | Rand r(857374); 249 | 250 | int keybits = sizeof(keytype) * 8; 251 | const int keycount = 256*256*32; 252 | keytype k; 253 | 254 | std::vector hashes(keycount); 255 | hashtype h1,h2; 256 | 257 | bool result = true; 258 | 259 | for(int keybit = 0; keybit < keybits; keybit++) 260 | { 261 | printf("Testing bit %d\n",keybit); 262 | 263 | for(int i = 0; i < keycount; i++) 264 | { 265 | r.rand_p(&k,sizeof(keytype)); 266 | 267 | hash(&k,sizeof(keytype),0,&h1); 268 | flipbit(&k,sizeof(keytype),keybit); 269 | hash(&k,sizeof(keytype),0,&h2); 270 | 271 | hashes[i] = h1 ^ h2; 272 | } 273 | 274 | result &= TestHashList(hashes,true,true,drawDiagram); 275 | printf("\n"); 276 | } 277 | 278 | return result; 279 | } 280 | 281 | //---------------------------------------------------------------------------- 282 | -------------------------------------------------------------------------------- /src/Hashes.cpp: -------------------------------------------------------------------------------- 1 | #include "Hashes.h" 2 | 3 | #include "Random.h" 4 | 5 | 6 | #include 7 | //#include 8 | #include 9 | //#include 10 | //#include 11 | 12 | //---------------------------------------------------------------------------- 13 | // fake / bad hashes 14 | 15 | void BadHash ( const void * key, int len, uint32_t seed, void * out ) 16 | { 17 | uint32_t h = seed; 18 | 19 | const uint8_t * data = (const uint8_t*)key; 20 | 21 | for(int i = 0; i < len; i++) 22 | { 23 | h ^= h >> 3; 24 | h ^= h << 5; 25 | h ^= data[i]; 26 | } 27 | 28 | *(uint32_t*)out = h; 29 | } 30 | 31 | void sumhash ( const void * key, int len, uint32_t seed, void * out ) 32 | { 33 | uint32_t h = seed; 34 | 35 | const uint8_t * data = (const uint8_t*)key; 36 | 37 | for(int i = 0; i < len; i++) 38 | { 39 | h += data[i]; 40 | } 41 | 42 | *(uint32_t*)out = h; 43 | } 44 | 45 | void sumhash32 ( const void * key, int len, uint32_t seed, void * out ) 46 | { 47 | uint32_t h = seed; 48 | 49 | const uint32_t * data = (const uint32_t*)key; 50 | 51 | for(int i = 0; i < len/4; i++) 52 | { 53 | h += data[i]; 54 | } 55 | 56 | *(uint32_t*)out = h; 57 | } 58 | 59 | void DoNothingHash ( const void *, int, uint32_t, void * ) 60 | { 61 | } 62 | 63 | //----------------------------------------------------------------------------- 64 | // One-byte-at-a-time hash based on Murmur's mix 65 | 66 | uint32_t MurmurOAAT ( const void * key, int len, uint32_t seed ) 67 | { 68 | const uint8_t * data = (const uint8_t*)key; 69 | 70 | uint32_t h = seed; 71 | 72 | for(int i = 0; i < len; i++) 73 | { 74 | h ^= data[i]; 75 | h *= 0x5bd1e995; 76 | h ^= h >> 15; 77 | } 78 | 79 | return h; 80 | } 81 | 82 | void MurmurOAAT_test ( const void * key, int len, uint32_t seed, void * out ) 83 | { 84 | *(uint32_t*)out = MurmurOAAT(key,len,seed); 85 | } 86 | 87 | //---------------------------------------------------------------------------- 88 | 89 | void FNV ( const void * key, int len, uint32_t seed, void * out ) 90 | { 91 | unsigned int h = seed; 92 | 93 | const uint8_t * data = (const uint8_t*)key; 94 | 95 | h ^= BIG_CONSTANT(2166136261); 96 | 97 | for(int i = 0; i < len; i++) 98 | { 99 | h ^= data[i]; 100 | h *= 16777619; 101 | } 102 | 103 | *(uint32_t*)out = h; 104 | } 105 | 106 | //----------------------------------------------------------------------------- 107 | 108 | uint32_t x17 ( const void * key, int len, uint32_t h ) 109 | { 110 | const uint8_t * data = (const uint8_t*)key; 111 | 112 | for(int i = 0; i < len; ++i) 113 | { 114 | h = 17 * h + (data[i] - ' '); 115 | } 116 | 117 | return h ^ (h >> 16); 118 | } 119 | 120 | //----------------------------------------------------------------------------- 121 | 122 | void Bernstein ( const void * key, int len, uint32_t seed, void * out ) 123 | { 124 | const uint8_t * data = (const uint8_t*)key; 125 | 126 | for(int i = 0; i < len; ++i) 127 | { 128 | seed = 33 * seed + data[i]; 129 | } 130 | 131 | *(uint32_t*)out = seed; 132 | } 133 | 134 | //----------------------------------------------------------------------------- 135 | // Crap8 hash from http://www.team5150.com/~andrew/noncryptohashzoo/Crap8.html 136 | 137 | uint32_t Crap8( const uint8_t *key, uint32_t len, uint32_t seed ) { 138 | #define c8fold( a, b, y, z ) { p = (uint32_t)(a) * (uint64_t)(b); y ^= (uint32_t)p; z ^= (uint32_t)(p >> 32); } 139 | #define c8mix( in ) { h *= m; c8fold( in, m, k, h ); } 140 | 141 | const uint32_t m = 0x83d2e73b, n = 0x97e1cc59, *key4 = (const uint32_t *)key; 142 | uint32_t h = len + seed, k = n + len; 143 | uint64_t p; 144 | 145 | while ( len >= 8 ) { c8mix(key4[0]) c8mix(key4[1]) key4 += 2; len -= 8; } 146 | if ( len >= 4 ) { c8mix(key4[0]) key4 += 1; len -= 4; } 147 | if ( len ) { c8mix( key4[0] & ( ( 1 << ( len * 8 ) ) - 1 ) ) } 148 | c8fold( h ^ k, n, k, k ) 149 | return k; 150 | } 151 | 152 | void Crap8_test ( const void * key, int len, uint32_t seed, void * out ) 153 | { 154 | *(uint32_t*)out = Crap8((const uint8_t*)key,len,seed); 155 | } 156 | -------------------------------------------------------------------------------- /src/Hashes.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include "Types.h" 4 | 5 | #include "MurmurHash1.h" 6 | #include "MurmurHash2.h" 7 | #include "MurmurHash3.h" 8 | 9 | //---------- 10 | // These are _not_ hash functions (even though people tend to use crc32 as one...) 11 | 12 | void sumhash ( const void * key, int len, uint32_t seed, void * out ); 13 | void sumhash32 ( const void * key, int len, uint32_t seed, void * out ); 14 | 15 | void DoNothingHash ( const void * key, int len, uint32_t seed, void * out ); 16 | void crc32 ( const void * key, int len, uint32_t seed, void * out ); 17 | 18 | void randhash_32 ( const void * key, int len, uint32_t seed, void * out ); 19 | void randhash_64 ( const void * key, int len, uint32_t seed, void * out ); 20 | void randhash_128 ( const void * key, int len, uint32_t seed, void * out ); 21 | 22 | //---------- 23 | // Cryptographic hashes 24 | 25 | void md5_32 ( const void * key, int len, uint32_t seed, void * out ); 26 | void sha1_32a ( const void * key, int len, uint32_t seed, void * out ); 27 | 28 | //---------- 29 | // General purpose hashes 30 | 31 | void FNV ( const void * key, int len, uint32_t seed, void * out ); 32 | void Bernstein ( const void * key, int len, uint32_t seed, void * out ); 33 | void SuperFastHash ( const void * key, int len, uint32_t seed, void * out ); 34 | void lookup3_test ( const void * key, int len, uint32_t seed, void * out ); 35 | void MurmurOAAT_test ( const void * key, int len, uint32_t seed, void * out ); 36 | void Crap8_test ( const void * key, int len, uint32_t seed, void * out ); 37 | void CityHash128_test ( const void * key, int len, uint32_t seed, void * out ); 38 | void CityHash64_test ( const void * key, int len, uint32_t seed, void * out ); 39 | 40 | //---------- 41 | // Spooky 42 | 43 | void SpookyHash32_test ( const void * key, int len, uint32_t seed, void * out ); 44 | void SpookyHash64_test ( const void * key, int len, uint32_t seed, void * out ); 45 | void SpookyHash128_test ( const void * key, int len, uint32_t seed, void * out ); 46 | void SpookyHash64V2_test ( const void * key, int len, uint32_t seed, void * out ); 47 | void SpookyHash128V2_test ( const void * key, int len, uint32_t seed, void * out ); 48 | 49 | //---------- 50 | // AES 51 | 52 | void AESHash128_test ( const void * key, int len, uint32_t seed, void * out ); 53 | void AESHash64_test ( const void * key, int len, uint32_t seed, void * out ); 54 | 55 | //---------- 56 | // Risky 57 | 58 | void RiskyHash128_test ( const void * key, int len, uint32_t seed, void * out ); 59 | void RiskyHash64_test ( const void * key, int len, uint32_t seed, void * out ); 60 | 61 | //---------- 62 | // Xxh3 63 | 64 | void XXH3128_test ( const void * key, int len, uint32_t seed, void * out ); 65 | void XXH364_test ( const void * key, int len, uint32_t seed, void * out ); 66 | 67 | //---------- 68 | // Meow 69 | 70 | void Meow128_test ( const void * key, int len, uint32_t seed, void * out ); 71 | void Meow64_test ( const void * key, int len, uint32_t seed, void * out ); 72 | 73 | uint32_t MurmurOAAT ( const void * key, int len, uint32_t seed ); 74 | 75 | //---------- 76 | // MurmurHash2 77 | 78 | void MurmurHash2_test ( const void * key, int len, uint32_t seed, void * out ); 79 | void MurmurHash2A_test ( const void * key, int len, uint32_t seed, void * out ); 80 | 81 | //----------------------------------------------------------------------------- 82 | // Test harnesses for Murmur1/2 83 | 84 | inline void MurmurHash1_test ( const void * key, int len, uint32_t seed, void * out ) 85 | { 86 | *(uint32_t*)out = MurmurHash1(key,len,seed); 87 | } 88 | 89 | inline void MurmurHash2_test ( const void * key, int len, uint32_t seed, void * out ) 90 | { 91 | *(uint32_t*)out = MurmurHash2(key,len,seed); 92 | } 93 | 94 | inline void MurmurHash2A_test ( const void * key, int len, uint32_t seed, void * out ) 95 | { 96 | *(uint32_t*)out = MurmurHash2A(key,len,seed); 97 | } 98 | 99 | inline void MurmurHash64A_test ( const void * key, int len, uint32_t seed, void * out ) 100 | { 101 | *(uint64_t*)out = MurmurHash64A(key,len,seed); 102 | } 103 | 104 | inline void MurmurHash64B_test ( const void * key, int len, uint32_t seed, void * out ) 105 | { 106 | *(uint64_t*)out = MurmurHash64B(key,len,seed); 107 | } 108 | -------------------------------------------------------------------------------- /src/KeysetTest.cpp: -------------------------------------------------------------------------------- 1 | #include "KeysetTest.h" 2 | 3 | #include "Platform.h" 4 | #include "Random.h" 5 | 6 | #include 7 | #include 8 | 9 | //----------------------------------------------------------------------------- 10 | // This should hopefully be a thorough and uambiguous test of whether a hash 11 | // is correctly implemented on a given platform 12 | 13 | bool VerificationTest ( pfHash hash, const int hashbits, uint32_t expected, bool verbose ) 14 | { 15 | const int hashbytes = hashbits / 8; 16 | 17 | uint8_t * key = new uint8_t[256]; 18 | uint8_t * hashes = new uint8_t[hashbytes * 256]; 19 | uint8_t * final = new uint8_t[hashbytes]; 20 | 21 | memset(key,0,256); 22 | memset(hashes,0,hashbytes*256); 23 | memset(final,0,hashbytes); 24 | 25 | // Hash keys of the form {0}, {0,1}, {0,1,2}... up to N=255,using 256-N as 26 | // the seed 27 | 28 | for(int i = 0; i < 256; i++) 29 | { 30 | key[i] = (uint8_t)i; 31 | 32 | hash(key,i,256-i,&hashes[i*hashbytes]); 33 | } 34 | 35 | // Then hash the result array 36 | 37 | hash(hashes,hashbytes*256,0,final); 38 | 39 | // The first four bytes of that hash, interpreted as a little-endian integer, is our 40 | // verification value 41 | 42 | uint32_t verification = (final[0] << 0) | (final[1] << 8) | (final[2] << 16) | (final[3] << 24); 43 | 44 | delete [] key; 45 | delete [] hashes; 46 | delete [] final; 47 | 48 | //---------- 49 | 50 | if(expected != verification) 51 | { 52 | if(verbose) printf("Verification value 0x%08X : Failed! (Expected 0x%08x)\n",verification,expected); 53 | return false; 54 | } 55 | else 56 | { 57 | if(verbose) printf("Verification value 0x%08X : Passed!\n",verification); 58 | return true; 59 | } 60 | } 61 | 62 | //---------------------------------------------------------------------------- 63 | // Basic sanity checks - 64 | 65 | // A hash function should not be reading outside the bounds of the key. 66 | 67 | // Flipping a bit of a key should, with overwhelmingly high probability, 68 | // result in a different hash. 69 | 70 | // Hashing the same key twice should always produce the same result. 71 | 72 | // The memory alignment of the key should not affect the hash result. 73 | 74 | bool SanityTest ( pfHash hash, const int hashbits ) 75 | { 76 | printf("Running sanity check 1"); 77 | 78 | Rand r(883741); 79 | 80 | bool result = true; 81 | 82 | const int hashbytes = hashbits/8; 83 | const int reps = 10; 84 | const int keymax = 256; 85 | const int pad = 16; 86 | const int buflen = keymax + pad*3; 87 | 88 | uint8_t * buffer1 = new uint8_t[buflen]; 89 | uint8_t * buffer2 = new uint8_t[buflen]; 90 | 91 | uint8_t * hash1 = new uint8_t[hashbytes]; 92 | uint8_t * hash2 = new uint8_t[hashbytes]; 93 | 94 | //---------- 95 | 96 | for(int irep = 0; irep < reps; irep++) 97 | { 98 | if(irep % (reps/10) == 0) printf("."); 99 | 100 | for(int len = 4; len <= keymax; len++) 101 | { 102 | for(int offset = pad; offset < pad*2; offset++) 103 | { 104 | uint8_t * key1 = &buffer1[pad]; 105 | uint8_t * key2 = &buffer2[pad+offset]; 106 | 107 | r.rand_p(buffer1,buflen); 108 | r.rand_p(buffer2,buflen); 109 | 110 | memcpy(key2,key1,len); 111 | 112 | hash(key1,len,0,hash1); 113 | 114 | for(int bit = 0; bit < (len * 8); bit++) 115 | { 116 | // Flip a bit, hash the key -> we should get a different result. 117 | 118 | flipbit(key2,len,bit); 119 | hash(key2,len,0,hash2); 120 | 121 | if(memcmp(hash1,hash2,hashbytes) == 0) 122 | { 123 | result = false; 124 | } 125 | 126 | // Flip it back, hash again -> we should get the original result. 127 | 128 | flipbit(key2,len,bit); 129 | hash(key2,len,0,hash2); 130 | 131 | if(memcmp(hash1,hash2,hashbytes) != 0) 132 | { 133 | result = false; 134 | } 135 | } 136 | } 137 | } 138 | } 139 | 140 | if(result == false) 141 | { 142 | printf("*********FAIL*********\n"); 143 | } 144 | else 145 | { 146 | printf("PASS\n"); 147 | } 148 | 149 | delete [] buffer1; 150 | delete [] buffer2; 151 | 152 | delete [] hash1; 153 | delete [] hash2; 154 | 155 | return result; 156 | } 157 | 158 | //---------------------------------------------------------------------------- 159 | // Appending zero bytes to a key should always cause it to produce a different 160 | // hash value 161 | 162 | void AppendedZeroesTest ( pfHash hash, const int hashbits ) 163 | { 164 | printf("Running sanity check 2"); 165 | 166 | Rand r(173994); 167 | 168 | const int hashbytes = hashbits/8; 169 | 170 | for(int rep = 0; rep < 100; rep++) 171 | { 172 | if(rep % 10 == 0) printf("."); 173 | 174 | unsigned char key[256]; 175 | 176 | memset(key,0,sizeof(key)); 177 | 178 | r.rand_p(key,32); 179 | 180 | uint32_t h1[16]; 181 | uint32_t h2[16]; 182 | 183 | memset(h1,0,hashbytes); 184 | memset(h2,0,hashbytes); 185 | 186 | for(int i = 0; i < 32; i++) 187 | { 188 | hash(key,32+i,0,h1); 189 | 190 | if(memcmp(h1,h2,hashbytes) == 0) 191 | { 192 | printf("\n*********FAIL*********\n"); 193 | return; 194 | } 195 | 196 | memcpy(h2,h1,hashbytes); 197 | } 198 | } 199 | 200 | printf("PASS\n"); 201 | } 202 | 203 | //----------------------------------------------------------------------------- 204 | // Generate all keys of up to N bytes containing two non-zero bytes 205 | 206 | void TwoBytesKeygen ( int maxlen, KeyCallback & c ) 207 | { 208 | //---------- 209 | // Compute # of keys 210 | 211 | int keycount = 0; 212 | 213 | for(int i = 2; i <= maxlen; i++) keycount += (int)chooseK(i,2); 214 | 215 | keycount *= 255*255; 216 | 217 | for(int i = 2; i <= maxlen; i++) keycount += i*255; 218 | 219 | printf("Keyset 'TwoBytes' - up-to-%d-byte keys, %d total keys\n",maxlen, keycount); 220 | 221 | c.reserve(keycount); 222 | 223 | //---------- 224 | // Add all keys with one non-zero byte 225 | 226 | uint8_t key[256]; 227 | 228 | memset(key,0,256); 229 | 230 | for(int keylen = 2; keylen <= maxlen; keylen++) 231 | for(int byteA = 0; byteA < keylen; byteA++) 232 | { 233 | for(int valA = 1; valA <= 255; valA++) 234 | { 235 | key[byteA] = (uint8_t)valA; 236 | 237 | c(key,keylen); 238 | } 239 | 240 | key[byteA] = 0; 241 | } 242 | 243 | //---------- 244 | // Add all keys with two non-zero bytes 245 | 246 | for(int keylen = 2; keylen <= maxlen; keylen++) 247 | for(int byteA = 0; byteA < keylen-1; byteA++) 248 | for(int byteB = byteA+1; byteB < keylen; byteB++) 249 | { 250 | for(int valA = 1; valA <= 255; valA++) 251 | { 252 | key[byteA] = (uint8_t)valA; 253 | 254 | for(int valB = 1; valB <= 255; valB++) 255 | { 256 | key[byteB] = (uint8_t)valB; 257 | c(key,keylen); 258 | } 259 | 260 | key[byteB] = 0; 261 | } 262 | 263 | key[byteA] = 0; 264 | } 265 | } 266 | 267 | //----------------------------------------------------------------------------- 268 | 269 | template< typename hashtype > 270 | void DumpCollisionMap ( CollisionMap & cmap ) 271 | { 272 | typedef CollisionMap cmap_t; 273 | 274 | for(typename cmap_t::iterator it = cmap.begin(); it != cmap.end(); ++it) 275 | { 276 | const hashtype & hash = (*it).first; 277 | 278 | printf("Hash - "); 279 | printbytes(&hash,sizeof(hashtype)); 280 | printf("\n"); 281 | 282 | std::vector & keys = (*it).second; 283 | 284 | for(int i = 0; i < (int)keys.size(); i++) 285 | { 286 | ByteVec & key = keys[i]; 287 | 288 | printf("Key - "); 289 | printbytes(&key[0],(int)key.size()); 290 | printf("\n"); 291 | } 292 | printf("\n"); 293 | } 294 | 295 | } 296 | 297 | // test code 298 | 299 | void ReportCollisions ( pfHash hash ) 300 | { 301 | printf("Hashing keyset\n"); 302 | 303 | std::vector hashes; 304 | 305 | HashCallback c(hash,hashes); 306 | 307 | TwoBytesKeygen(20,c); 308 | 309 | printf("%d hashes\n",(int)hashes.size()); 310 | 311 | printf("Finding collisions\n"); 312 | 313 | HashSet collisions; 314 | 315 | FindCollisions(hashes,collisions,1000); 316 | 317 | printf("%d collisions\n",(int)collisions.size()); 318 | 319 | printf("Mapping collisions\n"); 320 | 321 | CollisionMap cmap; 322 | 323 | CollisionCallback c2(hash,collisions,cmap); 324 | 325 | TwoBytesKeygen(20,c2); 326 | 327 | printf("Dumping collisions\n"); 328 | 329 | DumpCollisionMap(cmap); 330 | } 331 | -------------------------------------------------------------------------------- /src/LongNeighborTest.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #include "Types.h" 3 | 4 | // return true on success, false on failure 5 | bool testLongNeighbors(pfHash hashFunc, int hashbits); 6 | -------------------------------------------------------------------------------- /src/LongNeighborTest.md: -------------------------------------------------------------------------------- 1 | # LongNeighborTest 2 | 3 | This test was developed by [Henning Makholm](http://henning.makholm.net) 4 | in January 2019, while working at [Semmle Ltd](https://www.semmle.com). 5 | 6 | As of this writing Semmle has not yet officially allowed it to be released 7 | under a permissive license, so beware of possible legal trouble if you 8 | use it. (I'm going to seek permission once the work is done!) 9 | 10 | ## Overview 11 | 12 | This test uses a meet-in-the middle approach to look for hash collisions 13 | between messages with low Hamming distances. 14 | 15 | Developing the test was prompted by the discovery that 16 | [SpookyHash V2](http://burtleburtle.net/bob/hash/spooky.html) 17 | has lots of 3-bit collisions due to insufficient diffusion between 18 | the processing of the last full input block and the partial block 19 | at the end of the input. The goal of the test is to discover similar 20 | trouble in other hashes. 21 | 22 | Because Spooky has an unusually large block size of 96 bytes (=768 bits), 23 | and it uses a completely different algorithm for messages shorter than 2 24 | blocks, this problem only shows up for messages of length 281 to 287 bytes, 25 | and then again from 377 to 383 bytes, etc. Notably, messages of any "nice" 26 | length are not affectes, which means that we need to test for all possible 27 | message lenghts. 28 | 29 | ## Method 30 | 31 | We generate a lot of "base" messages of lengths between 10 and 300 bytes. 32 | For each length there is one base message consisting entirely of zero bits, 33 | one that consists entirely of ones, and three random base messages. 34 | Each of these base messages are processed separately. 35 | 36 | With the default parameters we derive about 3 million variants of the 37 | base message; then we look for hash collisions between those variants. 38 | There's a variant for each 1-bit flip anywhere in the mssage, one for 39 | each combination of 2 bit-flips within the last 2048 bits of the message, 40 | and one for each combination of 3 flips in the last 160 bits of the message. 41 | There are also some variants that consist of appending some zero bytes 42 | to the message and then flipping 2 bits. (Appending zeroes often has the 43 | same effect as bit flips, given how many hashes include a length counter 44 | in their final padding). 45 | 46 | If we find any collision between variants of a base message, that base 47 | is declared "bad". At the end we compare the number of bad bases found with 48 | the number we would expect with an ideal ("truly random") hash function. 49 | 50 | ## Interpreting the results 51 | 52 | The test tries to adapt its reporting to odd hash sizes. SMHasher 53 | only supports sizes that are powers of twos, which leads to three 54 | rather distinct reporting regimes: 55 | 56 | ### 128-bit hashes or longer 57 | 58 | An 128-bit hash is disqualified by discovering even a single bad base, 59 | since it would be astronomically implausible to find one for an ideal 60 | hash. 61 | 62 | The collision is displayed as soon as it is found, but the test continues 63 | for some time afterwards, in the hope of finding even more striking 64 | collisions (that is, one with fewer bits or the bits closer together) 65 | with slightly longer bases. 66 | 67 | ### 64-bit hashes 68 | 69 | With the default search parameters we would expect to find a bad base 70 | for about one in 8000 truly-random 64-bit hash functions tried, so a 71 | single bad base is not an absolute condemnation. If the hash has a 72 | _systematic_ tendency towards small-distance collisions it would generally 73 | show up as finding more than one bad base, in which case it fails the test. 74 | 75 | If any bad base is found, the "most striking" collision among those found 76 | will be printed at the end of the test. Even if there is only a single 77 | example (and the hash therefore passes), it is probably worthwhile to 78 | investigate how it managed to collide, and see if there is a discernible 79 | reason that can be fixed somehow. 80 | 81 | Collisions with particularly simple deltas may still cause a 64-bit hash 82 | to fail with only a single bad base. (This is based on the "surprise score" 83 | described below). 84 | 85 | ### 32-bit hashes 86 | 87 | With so few bits, the usual battery of variants would exceed the birthday 88 | bound for _every_ base. This would invalidate the methodology of comparing 89 | the number of bad and good bases. 90 | 91 | The test will automatically reduce how many 2- and 3-bit variants it tries, 92 | such as to keep below the birthday bound. This means, in effect, that 93 | collisions of Hamming distance 3 or more are expected if one of more of 94 | their bits are far from the _end_ of the message. 95 | 96 | There are still enough variants that even with an ideal hash, a few hundred 97 | bad bases is expected. If the _actual_ number of bad bases is more than 1.3 98 | times the theoretical valie, the hash fails the test. 99 | 100 | The 1.3 factor is more or less pulled out of a hat. It causes the 101 | `MurmurOAAT` and `Murmur2A` hashes to fail the test just barely -- 102 | they seem to have particular problems with inputs that are mostly 103 | zeroes. The newer version `Murmur3A` passes the test nicely. 104 | 105 | ## Surprise score 106 | 107 | The test computes a "surprise score" for each collision it finds, mostly 108 | so it can choose the "most surprising" of the collisions to print out 109 | at the end of the test. 110 | 111 | The score is computed as `(1/p)-1` where `p` is a somewhat sloppily computed 112 | probability that an ideal hash function would have a collision for 113 | _any prefix of this particular base message_ and either this particular 114 | delta or a simpler one. (A simpler delta is roughly one with the same 115 | number of bits flipped, but where those bits are closer to the end of 116 | the message.) 117 | 118 | **Warning:** 119 | Because of the selection bias inherent in displaying the collision with 120 | the _highest_ surprise score, the `p` **should not** be interpreted as 121 | anything like how likely it ought to be for a hash function to have a 122 | collision as bad as this. Completely fine 32-bit hashes will routinely 123 | find a collision with a surprise score in the hundreds. 124 | 125 | ### Horrible collisions 126 | 127 | That being said, if the test sees a surprise score of 1012 of 128 | more, it is considered a "horrible collision". These are ones we don't 129 | ever expect to come across for any good hash function. (Note that an 130 | "indistinguishably random" hash function will _have_ bases that have 131 | horrible collisions, but those bases ought to be so rare that we never 132 | expect to come across one during any realistic test run). 133 | 134 | A horrible collision will be printed out immediately when it is found 135 | (unless an even more horrible one has already been printed), and will in 136 | and of itself cause the hash to fail the test. 137 | 138 | A collision with a 128-bit hash is always horrible. (Or more precisely, 139 | the test would not reach a delta that could produce a non-horrible score 140 | until after hashing several hundred terabytes of data). 141 | 142 | It is impossible for a collision with a 32-bit hash to be horrible 143 | -- even if the hash of the one-byte messages `00` and `01` coincide, 144 | the score will be less than 1010. 145 | 146 | 64-bit collisions may land on either side of the threshold. 147 | 148 | ## Tweaking the search parameters 149 | 150 | The throroughness of the search is controlled by hard-coded constants 151 | at the top of `LongNeighborTest.cpp`. 152 | 153 | If you know the input block size of the hash you're testing you can save 154 | time by decreasing `MAXMSGBYTES` so it corresponds to just a few (say, three) 155 | input blocks. 156 | 157 | The three `MAXRANGE_xDELTA` constants control how many _bits_ from the 158 | end of the message will be considered for constructing 1-, 2- or 3-bit 159 | deltas. 160 | 161 | The default value for `MAXRANGE_1DELTA` is effectively infinity. 162 | 163 | `MAXRANGE_2DELTA` defaults to 2048 bits, which should be "large enough 164 | for everyone". 165 | 166 | `MAXRANGE_3DELTA` is 160 bits by default, which makes for roughly as many 167 | 3-bit deltas as 2-bit deltas. Increase it at your own risk; the cost scales 168 | _cubically_. Setting it to 512 will let you search for arbitrary 6-bit 169 | deltas in a 64-byte message, but the test will take most of a day unless 170 | you scale `MAXMSGBYTES` back. More than 1024 will (a) take forever and 171 | (b) require you to increase the size of `PackedDelta::codeword_t`. 172 | 173 | You can set `STORE_3BIT_HASHES` to `false` to conserve memory at the expense 174 | of not discovering collisions between two 3-bit variants. This is generally 175 | only useful if you are on a 32-bit machine, or if you have jacked 176 | `MAXRANGE_3DELTA` up way high and have the patience enough to compute a 177 | trillion hashes but not the RAM to store them. 178 | 179 | If you have even more time to spare you can increase `BASES_PER_LENGTH` 180 | to try more random base messages for each message lenght. But beware that 181 | this may sometimes mask problems, such as the particular problems Murmur2A 182 | has with all-zeroes bases. 183 | 184 | ## Detaching from SMHasher 185 | 186 | At the time of this writing, the code is only weakly coupled to SMHasher. 187 | You can grab `LongNeighborTest.{cpp,h}` and `Birthday.{cpp.h}` (which 188 | contains a helper function for computing very small birthday-collision 189 | probabilities in a numerically stable way), and be on your way. 190 | 191 | The main code depends on SMHasher's `Random.h` for creating base messages, 192 | but it should be easy to substitute another PRNG. 193 | 194 | ## So ... what about SpookyHash anyway? 195 | 196 | I recommend you use version 1 rather than version 2. It is marginally 197 | slower but does not appear to have any problems with small-delta collisions. 198 | 199 | (At your option you may use the change to _short_ hashes from version 2; 200 | that seems to be harmless and possibly improve the hash quality). 201 | -------------------------------------------------------------------------------- /src/MeowTest.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include "meow_hash.h" 6 | 7 | void Meow128_test(const void *key, int len, uint32_t seed, void *out) { 8 | /*meow_hash h = MeowHash_Accelerated( seed, len, const_cast( key ) ); 9 | memcpy( out, &h, 128 / 8 );*/ 10 | uint64_t h[ 2 ] = { seed, seed }; 11 | hash_meow128( key, len, &h[ 0 ], &h[ 1 ] ); 12 | memcpy( out, h, 128 / 8 ); 13 | } 14 | 15 | void Meow64_test(const void *key, int len, uint32_t seed, void *out) { 16 | /*meow_hash h = MeowHash_Accelerated( seed, len, const_cast( key ) ); 17 | memcpy( out, &h, 64 / 8 );*/ 18 | uint64_t h1 = seed, h2 = seed; 19 | hash_meow128( key, len, &h1, &h2 ); 20 | memcpy( out, &h1, 64 / 8 ); 21 | } 22 | -------------------------------------------------------------------------------- /src/MurmurHash1.cpp: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // MurmurHash was written by Austin Appleby, and is placed in the public 3 | // domain. The author hereby disclaims copyright to this source code. 4 | 5 | // Note - This code makes a few assumptions about how your machine behaves - 6 | 7 | // 1. We can read a 4-byte value from any address without crashing 8 | // 2. sizeof(int) == 4 9 | 10 | // And it has a few limitations - 11 | 12 | // 1. It will not work incrementally. 13 | // 2. It will not produce the same results on little-endian and big-endian 14 | // machines. 15 | 16 | #include "MurmurHash1.h" 17 | 18 | //----------------------------------------------------------------------------- 19 | 20 | uint32_t MurmurHash1 ( const void * key, int len, uint32_t seed ) 21 | { 22 | const unsigned int m = 0xc6a4a793; 23 | 24 | const int r = 16; 25 | 26 | unsigned int h = seed ^ (len * m); 27 | 28 | //---------- 29 | 30 | const unsigned char * data = (const unsigned char *)key; 31 | 32 | while(len >= 4) 33 | { 34 | unsigned int k = *(unsigned int *)data; 35 | 36 | h += k; 37 | h *= m; 38 | h ^= h >> 16; 39 | 40 | data += 4; 41 | len -= 4; 42 | } 43 | 44 | //---------- 45 | 46 | switch(len) 47 | { 48 | case 3: 49 | h += data[2] << 16; 50 | case 2: 51 | h += data[1] << 8; 52 | case 1: 53 | h += data[0]; 54 | h *= m; 55 | h ^= h >> r; 56 | }; 57 | 58 | //---------- 59 | 60 | h *= m; 61 | h ^= h >> 10; 62 | h *= m; 63 | h ^= h >> 17; 64 | 65 | return h; 66 | } 67 | 68 | //----------------------------------------------------------------------------- 69 | // MurmurHash1Aligned, by Austin Appleby 70 | 71 | // Same algorithm as MurmurHash1, but only does aligned reads - should be safer 72 | // on certain platforms. 73 | 74 | // Performance should be equal to or better than the simple version. 75 | 76 | unsigned int MurmurHash1Aligned ( const void * key, int len, unsigned int seed ) 77 | { 78 | const unsigned int m = 0xc6a4a793; 79 | const int r = 16; 80 | 81 | const unsigned char * data = (const unsigned char *)key; 82 | 83 | unsigned int h = seed ^ (len * m); 84 | 85 | int align = (uint64_t)data & 3; 86 | 87 | if(align && (len >= 4)) 88 | { 89 | // Pre-load the temp registers 90 | 91 | unsigned int t = 0, d = 0; 92 | 93 | switch(align) 94 | { 95 | case 1: t |= data[2] << 16; 96 | case 2: t |= data[1] << 8; 97 | case 3: t |= data[0]; 98 | } 99 | 100 | t <<= (8 * align); 101 | 102 | data += 4-align; 103 | len -= 4-align; 104 | 105 | int sl = 8 * (4-align); 106 | int sr = 8 * align; 107 | 108 | // Mix 109 | 110 | while(len >= 4) 111 | { 112 | d = *(unsigned int *)data; 113 | t = (t >> sr) | (d << sl); 114 | h += t; 115 | h *= m; 116 | h ^= h >> r; 117 | t = d; 118 | 119 | data += 4; 120 | len -= 4; 121 | } 122 | 123 | // Handle leftover data in temp registers 124 | 125 | int pack = len < align ? len : align; 126 | 127 | d = 0; 128 | 129 | switch(pack) 130 | { 131 | case 3: d |= data[2] << 16; 132 | case 2: d |= data[1] << 8; 133 | case 1: d |= data[0]; 134 | case 0: h += (t >> sr) | (d << sl); 135 | h *= m; 136 | h ^= h >> r; 137 | } 138 | 139 | data += pack; 140 | len -= pack; 141 | } 142 | else 143 | { 144 | while(len >= 4) 145 | { 146 | h += *(unsigned int *)data; 147 | h *= m; 148 | h ^= h >> r; 149 | 150 | data += 4; 151 | len -= 4; 152 | } 153 | } 154 | 155 | //---------- 156 | // Handle tail bytes 157 | 158 | switch(len) 159 | { 160 | case 3: h += data[2] << 16; 161 | case 2: h += data[1] << 8; 162 | case 1: h += data[0]; 163 | h *= m; 164 | h ^= h >> r; 165 | }; 166 | 167 | h *= m; 168 | h ^= h >> 10; 169 | h *= m; 170 | h ^= h >> 17; 171 | 172 | return h; 173 | } 174 | 175 | -------------------------------------------------------------------------------- /src/MurmurHash1.h: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // MurmurHash1 was written by Austin Appleby, and is placed in the public 3 | // domain. The author hereby disclaims copyright to this source code. 4 | 5 | #ifndef _MURMURHASH1_H_ 6 | #define _MURMURHASH1_H_ 7 | 8 | //----------------------------------------------------------------------------- 9 | // Platform-specific functions and macros 10 | 11 | // Microsoft Visual Studio 12 | 13 | #if defined(_MSC_VER) && (_MSC_VER < 1600) 14 | 15 | typedef unsigned char uint8_t; 16 | typedef unsigned int uint32_t; 17 | typedef unsigned __int64 uint64_t; 18 | 19 | // Other compilers 20 | 21 | #else // defined(_MSC_VER) 22 | 23 | #include 24 | 25 | #endif // !defined(_MSC_VER) 26 | 27 | //----------------------------------------------------------------------------- 28 | 29 | uint32_t MurmurHash1 ( const void * key, int len, uint32_t seed ); 30 | uint32_t MurmurHash1Aligned ( const void * key, int len, uint32_t seed ); 31 | 32 | //----------------------------------------------------------------------------- 33 | 34 | #endif // _MURMURHASH1_H_ 35 | -------------------------------------------------------------------------------- /src/MurmurHash2.h: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // MurmurHash2 was written by Austin Appleby, and is placed in the public 3 | // domain. The author hereby disclaims copyright to this source code. 4 | 5 | #ifndef _MURMURHASH2_H_ 6 | #define _MURMURHASH2_H_ 7 | 8 | //----------------------------------------------------------------------------- 9 | // Platform-specific functions and macros 10 | 11 | // Microsoft Visual Studio 12 | 13 | #if defined(_MSC_VER) && (_MSC_VER < 1600) 14 | 15 | typedef unsigned char uint8_t; 16 | typedef unsigned int uint32_t; 17 | typedef unsigned __int64 uint64_t; 18 | 19 | // Other compilers 20 | 21 | #else // defined(_MSC_VER) 22 | 23 | #include 24 | 25 | #endif // !defined(_MSC_VER) 26 | 27 | //----------------------------------------------------------------------------- 28 | 29 | uint32_t MurmurHash2 ( const void * key, int len, uint32_t seed ); 30 | uint64_t MurmurHash64A ( const void * key, int len, uint64_t seed ); 31 | uint64_t MurmurHash64B ( const void * key, int len, uint64_t seed ); 32 | uint32_t MurmurHash2A ( const void * key, int len, uint32_t seed ); 33 | uint32_t MurmurHashNeutral2 ( const void * key, int len, uint32_t seed ); 34 | uint32_t MurmurHashAligned2 ( const void * key, int len, uint32_t seed ); 35 | 36 | //----------------------------------------------------------------------------- 37 | 38 | #endif // _MURMURHASH2_H_ 39 | 40 | -------------------------------------------------------------------------------- /src/MurmurHash3.cpp: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // MurmurHash3 was written by Austin Appleby, and is placed in the public 3 | // domain. The author hereby disclaims copyright to this source code. 4 | 5 | // Note - The x86 and x64 versions do _not_ produce the same results, as the 6 | // algorithms are optimized for their respective platforms. You can still 7 | // compile and run any of them on any platform, but your performance with the 8 | // non-native version will be less than optimal. 9 | 10 | #include "MurmurHash3.h" 11 | 12 | //----------------------------------------------------------------------------- 13 | // Platform-specific functions and macros 14 | 15 | // Microsoft Visual Studio 16 | 17 | #if defined(_MSC_VER) 18 | 19 | #define FORCE_INLINE __forceinline 20 | 21 | #include 22 | 23 | #define ROTL32(x,y) _rotl(x,y) 24 | #define ROTL64(x,y) _rotl64(x,y) 25 | 26 | #define BIG_CONSTANT(x) (x) 27 | 28 | // Other compilers 29 | 30 | #else // defined(_MSC_VER) 31 | 32 | #define FORCE_INLINE inline __attribute__((always_inline)) 33 | 34 | inline uint32_t rotl32 ( uint32_t x, int8_t r ) 35 | { 36 | return (x << r) | (x >> (32 - r)); 37 | } 38 | 39 | inline uint64_t rotl64 ( uint64_t x, int8_t r ) 40 | { 41 | return (x << r) | (x >> (64 - r)); 42 | } 43 | 44 | #define ROTL32(x,y) rotl32(x,y) 45 | #define ROTL64(x,y) rotl64(x,y) 46 | 47 | #define BIG_CONSTANT(x) (x##LLU) 48 | 49 | #endif // !defined(_MSC_VER) 50 | 51 | //----------------------------------------------------------------------------- 52 | // Block read - if your platform needs to do endian-swapping or can only 53 | // handle aligned reads, do the conversion here 54 | 55 | FORCE_INLINE uint32_t getblock32 ( const uint32_t * p, int i ) 56 | { 57 | return p[i]; 58 | } 59 | 60 | FORCE_INLINE uint64_t getblock64 ( const uint64_t * p, int i ) 61 | { 62 | return p[i]; 63 | } 64 | 65 | //----------------------------------------------------------------------------- 66 | // Finalization mix - force all bits of a hash block to avalanche 67 | 68 | FORCE_INLINE uint32_t fmix32 ( uint32_t h ) 69 | { 70 | h ^= h >> 16; 71 | h *= 0x85ebca6b; 72 | h ^= h >> 13; 73 | h *= 0xc2b2ae35; 74 | h ^= h >> 16; 75 | 76 | return h; 77 | } 78 | 79 | //---------- 80 | 81 | FORCE_INLINE uint64_t fmix64 ( uint64_t k ) 82 | { 83 | k ^= k >> 33; 84 | k *= BIG_CONSTANT(0xff51afd7ed558ccd); 85 | k ^= k >> 33; 86 | k *= BIG_CONSTANT(0xc4ceb9fe1a85ec53); 87 | k ^= k >> 33; 88 | 89 | return k; 90 | } 91 | 92 | //----------------------------------------------------------------------------- 93 | 94 | void MurmurHash3_x86_32 ( const void * key, int len, 95 | uint32_t seed, void * out ) 96 | { 97 | const uint8_t * data = (const uint8_t*)key; 98 | const int nblocks = len / 4; 99 | 100 | uint32_t h1 = seed; 101 | 102 | const uint32_t c1 = 0xcc9e2d51; 103 | const uint32_t c2 = 0x1b873593; 104 | 105 | //---------- 106 | // body 107 | 108 | const uint32_t * blocks = (const uint32_t *)(data + nblocks*4); 109 | 110 | for(int i = -nblocks; i; i++) 111 | { 112 | uint32_t k1 = getblock32(blocks,i); 113 | 114 | k1 *= c1; 115 | k1 = ROTL32(k1,15); 116 | k1 *= c2; 117 | 118 | h1 ^= k1; 119 | h1 = ROTL32(h1,13); 120 | h1 = h1*5+0xe6546b64; 121 | } 122 | 123 | //---------- 124 | // tail 125 | 126 | const uint8_t * tail = (const uint8_t*)(data + nblocks*4); 127 | 128 | uint32_t k1 = 0; 129 | 130 | switch(len & 3) 131 | { 132 | case 3: k1 ^= tail[2] << 16; 133 | case 2: k1 ^= tail[1] << 8; 134 | case 1: k1 ^= tail[0]; 135 | k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1; 136 | }; 137 | 138 | //---------- 139 | // finalization 140 | 141 | h1 ^= len; 142 | 143 | h1 = fmix32(h1); 144 | 145 | *(uint32_t*)out = h1; 146 | } 147 | 148 | //----------------------------------------------------------------------------- 149 | 150 | void MurmurHash3_x86_128 ( const void * key, const int len, 151 | uint32_t seed, void * out ) 152 | { 153 | const uint8_t * data = (const uint8_t*)key; 154 | const int nblocks = len / 16; 155 | 156 | uint32_t h1 = seed; 157 | uint32_t h2 = seed; 158 | uint32_t h3 = seed; 159 | uint32_t h4 = seed; 160 | 161 | const uint32_t c1 = 0x239b961b; 162 | const uint32_t c2 = 0xab0e9789; 163 | const uint32_t c3 = 0x38b34ae5; 164 | const uint32_t c4 = 0xa1e38b93; 165 | 166 | //---------- 167 | // body 168 | 169 | const uint32_t * blocks = (const uint32_t *)(data + nblocks*16); 170 | 171 | for(int i = -nblocks; i; i++) 172 | { 173 | uint32_t k1 = getblock32(blocks,i*4+0); 174 | uint32_t k2 = getblock32(blocks,i*4+1); 175 | uint32_t k3 = getblock32(blocks,i*4+2); 176 | uint32_t k4 = getblock32(blocks,i*4+3); 177 | 178 | k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1; 179 | 180 | h1 = ROTL32(h1,19); h1 += h2; h1 = h1*5+0x561ccd1b; 181 | 182 | k2 *= c2; k2 = ROTL32(k2,16); k2 *= c3; h2 ^= k2; 183 | 184 | h2 = ROTL32(h2,17); h2 += h3; h2 = h2*5+0x0bcaa747; 185 | 186 | k3 *= c3; k3 = ROTL32(k3,17); k3 *= c4; h3 ^= k3; 187 | 188 | h3 = ROTL32(h3,15); h3 += h4; h3 = h3*5+0x96cd1c35; 189 | 190 | k4 *= c4; k4 = ROTL32(k4,18); k4 *= c1; h4 ^= k4; 191 | 192 | h4 = ROTL32(h4,13); h4 += h1; h4 = h4*5+0x32ac3b17; 193 | } 194 | 195 | //---------- 196 | // tail 197 | 198 | const uint8_t * tail = (const uint8_t*)(data + nblocks*16); 199 | 200 | uint32_t k1 = 0; 201 | uint32_t k2 = 0; 202 | uint32_t k3 = 0; 203 | uint32_t k4 = 0; 204 | 205 | switch(len & 15) 206 | { 207 | case 15: k4 ^= tail[14] << 16; 208 | case 14: k4 ^= tail[13] << 8; 209 | case 13: k4 ^= tail[12] << 0; 210 | k4 *= c4; k4 = ROTL32(k4,18); k4 *= c1; h4 ^= k4; 211 | 212 | case 12: k3 ^= tail[11] << 24; 213 | case 11: k3 ^= tail[10] << 16; 214 | case 10: k3 ^= tail[ 9] << 8; 215 | case 9: k3 ^= tail[ 8] << 0; 216 | k3 *= c3; k3 = ROTL32(k3,17); k3 *= c4; h3 ^= k3; 217 | 218 | case 8: k2 ^= tail[ 7] << 24; 219 | case 7: k2 ^= tail[ 6] << 16; 220 | case 6: k2 ^= tail[ 5] << 8; 221 | case 5: k2 ^= tail[ 4] << 0; 222 | k2 *= c2; k2 = ROTL32(k2,16); k2 *= c3; h2 ^= k2; 223 | 224 | case 4: k1 ^= tail[ 3] << 24; 225 | case 3: k1 ^= tail[ 2] << 16; 226 | case 2: k1 ^= tail[ 1] << 8; 227 | case 1: k1 ^= tail[ 0] << 0; 228 | k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1; 229 | }; 230 | 231 | //---------- 232 | // finalization 233 | 234 | h1 ^= len; h2 ^= len; h3 ^= len; h4 ^= len; 235 | 236 | h1 += h2; h1 += h3; h1 += h4; 237 | h2 += h1; h3 += h1; h4 += h1; 238 | 239 | h1 = fmix32(h1); 240 | h2 = fmix32(h2); 241 | h3 = fmix32(h3); 242 | h4 = fmix32(h4); 243 | 244 | h1 += h2; h1 += h3; h1 += h4; 245 | h2 += h1; h3 += h1; h4 += h1; 246 | 247 | ((uint32_t*)out)[0] = h1; 248 | ((uint32_t*)out)[1] = h2; 249 | ((uint32_t*)out)[2] = h3; 250 | ((uint32_t*)out)[3] = h4; 251 | } 252 | 253 | //----------------------------------------------------------------------------- 254 | 255 | void MurmurHash3_x64_128 ( const void * key, const int len, 256 | const uint32_t seed, void * out ) 257 | { 258 | const uint8_t * data = (const uint8_t*)key; 259 | const int nblocks = len / 16; 260 | 261 | uint64_t h1 = seed; 262 | uint64_t h2 = seed; 263 | 264 | const uint64_t c1 = BIG_CONSTANT(0x87c37b91114253d5); 265 | const uint64_t c2 = BIG_CONSTANT(0x4cf5ad432745937f); 266 | 267 | //---------- 268 | // body 269 | 270 | const uint64_t * blocks = (const uint64_t *)(data); 271 | 272 | for(int i = 0; i < nblocks; i++) 273 | { 274 | uint64_t k1 = getblock64(blocks,i*2+0); 275 | uint64_t k2 = getblock64(blocks,i*2+1); 276 | 277 | k1 *= c1; k1 = ROTL64(k1,31); k1 *= c2; h1 ^= k1; 278 | 279 | h1 = ROTL64(h1,27); h1 += h2; h1 = h1*5+0x52dce729; 280 | 281 | k2 *= c2; k2 = ROTL64(k2,33); k2 *= c1; h2 ^= k2; 282 | 283 | h2 = ROTL64(h2,31); h2 += h1; h2 = h2*5+0x38495ab5; 284 | } 285 | 286 | //---------- 287 | // tail 288 | 289 | const uint8_t * tail = (const uint8_t*)(data + nblocks*16); 290 | 291 | uint64_t k1 = 0; 292 | uint64_t k2 = 0; 293 | 294 | switch(len & 15) 295 | { 296 | case 15: k2 ^= ((uint64_t)tail[14]) << 48; 297 | case 14: k2 ^= ((uint64_t)tail[13]) << 40; 298 | case 13: k2 ^= ((uint64_t)tail[12]) << 32; 299 | case 12: k2 ^= ((uint64_t)tail[11]) << 24; 300 | case 11: k2 ^= ((uint64_t)tail[10]) << 16; 301 | case 10: k2 ^= ((uint64_t)tail[ 9]) << 8; 302 | case 9: k2 ^= ((uint64_t)tail[ 8]) << 0; 303 | k2 *= c2; k2 = ROTL64(k2,33); k2 *= c1; h2 ^= k2; 304 | 305 | case 8: k1 ^= ((uint64_t)tail[ 7]) << 56; 306 | case 7: k1 ^= ((uint64_t)tail[ 6]) << 48; 307 | case 6: k1 ^= ((uint64_t)tail[ 5]) << 40; 308 | case 5: k1 ^= ((uint64_t)tail[ 4]) << 32; 309 | case 4: k1 ^= ((uint64_t)tail[ 3]) << 24; 310 | case 3: k1 ^= ((uint64_t)tail[ 2]) << 16; 311 | case 2: k1 ^= ((uint64_t)tail[ 1]) << 8; 312 | case 1: k1 ^= ((uint64_t)tail[ 0]) << 0; 313 | k1 *= c1; k1 = ROTL64(k1,31); k1 *= c2; h1 ^= k1; 314 | }; 315 | 316 | //---------- 317 | // finalization 318 | 319 | h1 ^= len; h2 ^= len; 320 | 321 | h1 += h2; 322 | h2 += h1; 323 | 324 | h1 = fmix64(h1); 325 | h2 = fmix64(h2); 326 | 327 | h1 += h2; 328 | h2 += h1; 329 | 330 | ((uint64_t*)out)[0] = h1; 331 | ((uint64_t*)out)[1] = h2; 332 | } 333 | 334 | //----------------------------------------------------------------------------- 335 | 336 | -------------------------------------------------------------------------------- /src/MurmurHash3.h: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // MurmurHash3 was written by Austin Appleby, and is placed in the public 3 | // domain. The author hereby disclaims copyright to this source code. 4 | 5 | #ifndef _MURMURHASH3_H_ 6 | #define _MURMURHASH3_H_ 7 | 8 | //----------------------------------------------------------------------------- 9 | // Platform-specific functions and macros 10 | 11 | // Microsoft Visual Studio 12 | 13 | #if defined(_MSC_VER) && (_MSC_VER < 1600) 14 | 15 | typedef unsigned char uint8_t; 16 | typedef unsigned int uint32_t; 17 | typedef unsigned __int64 uint64_t; 18 | 19 | // Other compilers 20 | 21 | #else // defined(_MSC_VER) 22 | 23 | #include 24 | 25 | #endif // !defined(_MSC_VER) 26 | 27 | //----------------------------------------------------------------------------- 28 | 29 | void MurmurHash3_x86_32 ( const void * key, int len, uint32_t seed, void * out ); 30 | 31 | void MurmurHash3_x86_128 ( const void * key, int len, uint32_t seed, void * out ); 32 | 33 | void MurmurHash3_x64_128 ( const void * key, int len, uint32_t seed, void * out ); 34 | 35 | //----------------------------------------------------------------------------- 36 | 37 | #endif // _MURMURHASH3_H_ 38 | -------------------------------------------------------------------------------- /src/PMurHash.c: -------------------------------------------------------------------------------- 1 | /*----------------------------------------------------------------------------- 2 | * MurmurHash3 was written by Austin Appleby, and is placed in the public 3 | * domain. 4 | * 5 | * This implementation was written by Shane Day, and is also public domain. 6 | * 7 | * This is a portable ANSI C implementation of MurmurHash3_x86_32 (Murmur3A) 8 | * with support for progressive processing. 9 | */ 10 | 11 | /*----------------------------------------------------------------------------- 12 | 13 | If you want to understand the MurmurHash algorithm you would be much better 14 | off reading the original source. Just point your browser at: 15 | http://code.google.com/p/smhasher/source/browse/trunk/MurmurHash3.cpp 16 | 17 | 18 | What this version provides? 19 | 20 | 1. Progressive data feeding. Useful when the entire payload to be hashed 21 | does not fit in memory or when the data is streamed through the application. 22 | Also useful when hashing a number of strings with a common prefix. A partial 23 | hash of a prefix string can be generated and reused for each suffix string. 24 | 25 | 2. Portability. Plain old C so that it should compile on any old compiler. 26 | Both CPU endian and access-alignment neutral, but avoiding inefficient code 27 | when possible depending on CPU capabilities. 28 | 29 | 3. Drop in. I personally like nice self contained public domain code, making it 30 | easy to pilfer without loads of refactoring to work properly in the existing 31 | application code & makefile structure and mucking around with licence files. 32 | Just copy PMurHash.h and PMurHash.c and you're ready to go. 33 | 34 | 35 | How does it work? 36 | 37 | We can only process entire 32 bit chunks of input, except for the very end 38 | that may be shorter. So along with the partial hash we need to give back to 39 | the caller a carry containing up to 3 bytes that we were unable to process. 40 | This carry also needs to record the number of bytes the carry holds. I use 41 | the low 2 bits as a count (0..3) and the carry bytes are shifted into the 42 | high byte in stream order. 43 | 44 | To handle endianess I simply use a macro that reads a uint32_t and define 45 | that macro to be a direct read on little endian machines, a read and swap 46 | on big endian machines, or a byte-by-byte read if the endianess is unknown. 47 | 48 | -----------------------------------------------------------------------------*/ 49 | 50 | 51 | #include "PMurHash.h" 52 | 53 | /* I used ugly type names in the header to avoid potential conflicts with 54 | * application or system typedefs & defines. Since I'm not including any more 55 | * headers below here I can rename these so that the code reads like C99 */ 56 | #undef uint32_t 57 | #define uint32_t MH_UINT32 58 | #undef uint8_t 59 | #define uint8_t MH_UINT8 60 | 61 | /* MSVC warnings we choose to ignore */ 62 | #if defined(_MSC_VER) 63 | #pragma warning(disable: 4127) /* conditional expression is constant */ 64 | #endif 65 | 66 | /*----------------------------------------------------------------------------- 67 | * Endianess, misalignment capabilities and util macros 68 | * 69 | * The following 3 macros are defined in this section. The other macros defined 70 | * are only needed to help derive these 3. 71 | * 72 | * READ_UINT32(x) Read a little endian unsigned 32-bit int 73 | * UNALIGNED_SAFE Defined if READ_UINT32 works on non-word boundaries 74 | * ROTL32(x,r) Rotate x left by r bits 75 | */ 76 | 77 | /* Convention is to define __BYTE_ORDER == to one of these values */ 78 | #if !defined(__BIG_ENDIAN) 79 | #define __BIG_ENDIAN 4321 80 | #endif 81 | #if !defined(__LITTLE_ENDIAN) 82 | #define __LITTLE_ENDIAN 1234 83 | #endif 84 | 85 | /* I386 */ 86 | #if defined(_M_IX86) || defined(__i386__) || defined(__i386) || defined(i386) 87 | #define __BYTE_ORDER __LITTLE_ENDIAN 88 | #define UNALIGNED_SAFE 89 | #endif 90 | 91 | /* gcc 'may' define __LITTLE_ENDIAN__ or __BIG_ENDIAN__ to 1 (Note the trailing __), 92 | * or even _LITTLE_ENDIAN or _BIG_ENDIAN (Note the single _ prefix) */ 93 | #if !defined(__BYTE_ORDER) 94 | #if defined(__LITTLE_ENDIAN__) && __LITTLE_ENDIAN__==1 || defined(_LITTLE_ENDIAN) && _LITTLE_ENDIAN==1 95 | #define __BYTE_ORDER __LITTLE_ENDIAN 96 | #elif defined(__BIG_ENDIAN__) && __BIG_ENDIAN__==1 || defined(_BIG_ENDIAN) && _BIG_ENDIAN==1 97 | #define __BYTE_ORDER __BIG_ENDIAN 98 | #endif 99 | #endif 100 | 101 | /* gcc (usually) defines xEL/EB macros for ARM and MIPS endianess */ 102 | #if !defined(__BYTE_ORDER) 103 | #if defined(__ARMEL__) || defined(__MIPSEL__) 104 | #define __BYTE_ORDER __LITTLE_ENDIAN 105 | #endif 106 | #if defined(__ARMEB__) || defined(__MIPSEB__) 107 | #define __BYTE_ORDER __BIG_ENDIAN 108 | #endif 109 | #endif 110 | 111 | /* Now find best way we can to READ_UINT32 */ 112 | #if __BYTE_ORDER==__LITTLE_ENDIAN 113 | /* CPU endian matches murmurhash algorithm, so read 32-bit word directly */ 114 | #define READ_UINT32(ptr) (*((uint32_t*)(ptr))) 115 | #elif __BYTE_ORDER==__BIG_ENDIAN 116 | /* TODO: Add additional cases below where a compiler provided bswap32 is available */ 117 | #if defined(__GNUC__) && (__GNUC__>4 || (__GNUC__==4 && __GNUC_MINOR__>=3)) 118 | #define READ_UINT32(ptr) (__builtin_bswap32(*((uint32_t*)(ptr)))) 119 | #else 120 | /* Without a known fast bswap32 we're just as well off doing this */ 121 | #define READ_UINT32(ptr) (ptr[0]|ptr[1]<<8|ptr[2]<<16|ptr[3]<<24) 122 | #define UNALIGNED_SAFE 123 | #endif 124 | #else 125 | /* Unknown endianess so last resort is to read individual bytes */ 126 | #define READ_UINT32(ptr) (ptr[0]|ptr[1]<<8|ptr[2]<<16|ptr[3]<<24) 127 | 128 | /* Since we're not doing word-reads we can skip the messing about with realignment */ 129 | #define UNALIGNED_SAFE 130 | #endif 131 | 132 | /* Find best way to ROTL32 */ 133 | #if defined(_MSC_VER) 134 | #include /* Microsoft put _rotl declaration in here */ 135 | #define ROTL32(x,r) _rotl(x,r) 136 | #else 137 | /* gcc recognises this code and generates a rotate instruction for CPUs with one */ 138 | #define ROTL32(x,r) (((uint32_t)x << r) | ((uint32_t)x >> (32 - r))) 139 | #endif 140 | 141 | 142 | /*----------------------------------------------------------------------------- 143 | * Core murmurhash algorithm macros */ 144 | 145 | #define C1 (0xcc9e2d51) 146 | #define C2 (0x1b873593) 147 | 148 | /* This is the main processing body of the algorithm. It operates 149 | * on each full 32-bits of input. */ 150 | #define DOBLOCK(h1, k1) do{ \ 151 | k1 *= C1; \ 152 | k1 = ROTL32(k1,15); \ 153 | k1 *= C2; \ 154 | \ 155 | h1 ^= k1; \ 156 | h1 = ROTL32(h1,13); \ 157 | h1 = h1*5+0xe6546b64; \ 158 | }while(0) 159 | 160 | 161 | /* Append unaligned bytes to carry, forcing hash churn if we have 4 bytes */ 162 | /* cnt=bytes to process, h1=name of h1 var, c=carry, n=bytes in c, ptr/len=payload */ 163 | #define DOBYTES(cnt, h1, c, n, ptr, len) do{ \ 164 | int _i = cnt; \ 165 | while(_i--) { \ 166 | c = c>>8 | *ptr++<<24; \ 167 | n++; len--; \ 168 | if(n==4) { \ 169 | DOBLOCK(h1, c); \ 170 | n = 0; \ 171 | } \ 172 | } }while(0) 173 | 174 | /*---------------------------------------------------------------------------*/ 175 | 176 | /* Main hashing function. Initialise carry to 0 and h1 to 0 or an initial seed 177 | * if wanted. Both ph1 and pcarry are required arguments. */ 178 | void PMurHash32_Process(uint32_t *ph1, uint32_t *pcarry, const void *key, int len) 179 | { 180 | uint32_t h1 = *ph1; 181 | uint32_t c = *pcarry; 182 | 183 | const uint8_t *ptr = (uint8_t*)key; 184 | const uint8_t *end; 185 | 186 | /* Extract carry count from low 2 bits of c value */ 187 | int n = c & 3; 188 | 189 | #if defined(UNALIGNED_SAFE) 190 | /* This CPU handles unaligned word access */ 191 | 192 | /* Consume any carry bytes */ 193 | int i = (4-n) & 3; 194 | if(i && i <= len) { 195 | DOBYTES(i, h1, c, n, ptr, len); 196 | } 197 | 198 | /* Process 32-bit chunks */ 199 | end = ptr + len/4*4; 200 | for( ; ptr < end ; ptr+=4) { 201 | uint32_t k1 = READ_UINT32(ptr); 202 | DOBLOCK(h1, k1); 203 | } 204 | 205 | #else /*UNALIGNED_SAFE*/ 206 | /* This CPU does not handle unaligned word access */ 207 | 208 | /* Consume enough so that the next data byte is word aligned */ 209 | int i = -(long)ptr & 3; 210 | if(i && i <= len) { 211 | DOBYTES(i, h1, c, n, ptr, len); 212 | } 213 | 214 | /* We're now aligned. Process in aligned blocks. Specialise for each possible carry count */ 215 | end = ptr + len/4*4; 216 | switch(n) { /* how many bytes in c */ 217 | case 0: /* c=[----] w=[3210] b=[3210]=w c'=[----] */ 218 | for( ; ptr < end ; ptr+=4) { 219 | uint32_t k1 = READ_UINT32(ptr); 220 | DOBLOCK(h1, k1); 221 | } 222 | break; 223 | case 1: /* c=[0---] w=[4321] b=[3210]=c>>24|w<<8 c'=[4---] */ 224 | for( ; ptr < end ; ptr+=4) { 225 | uint32_t k1 = c>>24; 226 | c = READ_UINT32(ptr); 227 | k1 |= c<<8; 228 | DOBLOCK(h1, k1); 229 | } 230 | break; 231 | case 2: /* c=[10--] w=[5432] b=[3210]=c>>16|w<<16 c'=[54--] */ 232 | for( ; ptr < end ; ptr+=4) { 233 | uint32_t k1 = c>>16; 234 | c = READ_UINT32(ptr); 235 | k1 |= c<<16; 236 | DOBLOCK(h1, k1); 237 | } 238 | break; 239 | case 3: /* c=[210-] w=[6543] b=[3210]=c>>8|w<<24 c'=[654-] */ 240 | for( ; ptr < end ; ptr+=4) { 241 | uint32_t k1 = c>>8; 242 | c = READ_UINT32(ptr); 243 | k1 |= c<<24; 244 | DOBLOCK(h1, k1); 245 | } 246 | } 247 | #endif /*UNALIGNED_SAFE*/ 248 | 249 | /* Advance over whole 32-bit chunks, possibly leaving 1..3 bytes */ 250 | len -= len/4*4; 251 | 252 | /* Append any remaining bytes into carry */ 253 | DOBYTES(len, h1, c, n, ptr, len); 254 | 255 | /* Copy out new running hash and carry */ 256 | *ph1 = h1; 257 | *pcarry = (c & ~0xff) | n; 258 | } 259 | 260 | /*---------------------------------------------------------------------------*/ 261 | 262 | /* Finalize a hash. To match the original Murmur3A the total_length must be provided */ 263 | uint32_t PMurHash32_Result(uint32_t h, uint32_t carry, uint32_t total_length) 264 | { 265 | uint32_t k1; 266 | int n = carry & 3; 267 | if(n) { 268 | k1 = carry >> (4-n)*8; 269 | k1 *= C1; k1 = ROTL32(k1,15); k1 *= C2; h ^= k1; 270 | } 271 | h ^= total_length; 272 | 273 | /* fmix */ 274 | h ^= h >> 16; 275 | h *= 0x85ebca6b; 276 | h ^= h >> 13; 277 | h *= 0xc2b2ae35; 278 | h ^= h >> 16; 279 | 280 | return h; 281 | } 282 | 283 | /*---------------------------------------------------------------------------*/ 284 | 285 | /* Murmur3A compatable all-at-once */ 286 | uint32_t PMurHash32(uint32_t seed, const void *key, int len) 287 | { 288 | uint32_t h1=seed, carry=0; 289 | PMurHash32_Process(&h1, &carry, key, len); 290 | return PMurHash32_Result(h1, carry, len); 291 | } 292 | 293 | /*---------------------------------------------------------------------------*/ 294 | 295 | /* Provide an API suitable for smhasher */ 296 | void PMurHash32_test(const void *key, int len, uint32_t seed, void *out) 297 | { 298 | uint32_t h1=seed, carry=0; 299 | const uint8_t *ptr = (uint8_t*)key; 300 | const uint8_t *end = ptr + len; 301 | 302 | #if 0 /* Exercise the progressive processing */ 303 | while(ptr < end) { 304 | //const uint8_t *mid = ptr + rand()%(end-ptr)+1; 305 | const uint8_t *mid = ptr + (rand()&0xF); 306 | mid = mid= 199901L ) 25 | #include 26 | #define MH_UINT32 uint32_t 27 | #endif 28 | 29 | /* Otherwise try testing against max value macros from limit.h */ 30 | #if !defined(MH_UINT32) 31 | #include 32 | #if (USHRT_MAX == 0xffffffffUL) 33 | #define MH_UINT32 unsigned short 34 | #elif (UINT_MAX == 0xffffffffUL) 35 | #define MH_UINT32 unsigned int 36 | #elif (ULONG_MAX == 0xffffffffUL) 37 | #define MH_UINT32 unsigned long 38 | #endif 39 | #endif 40 | 41 | #if !defined(MH_UINT32) 42 | #error Unable to determine type name for unsigned 32-bit int 43 | #endif 44 | 45 | /* I'm yet to work on a platform where 'unsigned char' is not 8 bits */ 46 | #define MH_UINT8 unsigned char 47 | 48 | 49 | /* ------------------------------------------------------------------------- */ 50 | /* Prototypes */ 51 | 52 | #ifdef __cplusplus 53 | extern "C" { 54 | #endif 55 | 56 | void PMurHash32_Process(MH_UINT32 *ph1, MH_UINT32 *pcarry, const void *key, int len); 57 | MH_UINT32 PMurHash32_Result(MH_UINT32 h1, MH_UINT32 carry, MH_UINT32 total_length); 58 | MH_UINT32 PMurHash32(MH_UINT32 seed, const void *key, int len); 59 | 60 | void PMurHash32_test(const void *key, int len, MH_UINT32 seed, void *out); 61 | 62 | #ifdef __cplusplus 63 | } 64 | #endif 65 | -------------------------------------------------------------------------------- /src/Platform.cpp: -------------------------------------------------------------------------------- 1 | #include "Platform.h" 2 | 3 | #include 4 | 5 | void testRDTSC ( void ) 6 | { 7 | int64_t temp = rdtsc(); 8 | 9 | printf("%d",(int)temp); 10 | } 11 | 12 | #if defined(_MSC_VER) 13 | 14 | #include 15 | 16 | void SetAffinity ( int cpu ) 17 | { 18 | SetProcessAffinityMask(GetCurrentProcess(),cpu); 19 | SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_HIGHEST); 20 | } 21 | 22 | #else 23 | 24 | #include 25 | 26 | void SetAffinity ( int /*cpu*/ ) 27 | { 28 | #if !defined(__CYGWIN__) && !defined(__APPLE__) 29 | cpu_set_t mask; 30 | 31 | CPU_ZERO(&mask); 32 | 33 | CPU_SET(2,&mask); 34 | 35 | if( sched_setaffinity(0,sizeof(mask),&mask) == -1) 36 | { 37 | printf("WARNING: Could not set CPU affinity\n"); 38 | } 39 | #endif 40 | } 41 | 42 | #endif 43 | -------------------------------------------------------------------------------- /src/Platform.h: -------------------------------------------------------------------------------- 1 | //----------------------------------------------------------------------------- 2 | // Platform-specific functions and macros 3 | 4 | #pragma once 5 | 6 | void SetAffinity ( int cpu ); 7 | 8 | //----------------------------------------------------------------------------- 9 | // Microsoft Visual Studio 10 | 11 | #if defined(_MSC_VER) 12 | 13 | #define FORCE_INLINE __forceinline 14 | #define NEVER_INLINE __declspec(noinline) 15 | 16 | #include 17 | #include // Has to be included before intrin.h or VC complains about 'ceil' 18 | #include // for __rdtsc 19 | #include "pstdint.h" 20 | 21 | #define ROTL32(x,y) _rotl(x,y) 22 | #define ROTL64(x,y) _rotl64(x,y) 23 | #define ROTR32(x,y) _rotr(x,y) 24 | #define ROTR64(x,y) _rotr64(x,y) 25 | 26 | #pragma warning(disable : 4127) // "conditional expression is constant" in the if()s for avalanchetest 27 | #pragma warning(disable : 4100) 28 | #pragma warning(disable : 4702) 29 | 30 | #define BIG_CONSTANT(x) (x) 31 | 32 | // RDTSC == Read Time Stamp Counter 33 | 34 | #define rdtsc() __rdtsc() 35 | 36 | //----------------------------------------------------------------------------- 37 | // Other compilers 38 | 39 | #else // defined(_MSC_VER) 40 | 41 | #include 42 | 43 | #define FORCE_INLINE inline __attribute__((always_inline)) 44 | #define NEVER_INLINE __attribute__((noinline)) 45 | 46 | inline uint32_t rotl32 ( uint32_t x, int8_t r ) 47 | { 48 | return (x << r) | (x >> (32 - r)); 49 | } 50 | 51 | inline uint64_t rotl64 ( uint64_t x, int8_t r ) 52 | { 53 | return (x << r) | (x >> (64 - r)); 54 | } 55 | 56 | inline uint32_t rotr32 ( uint32_t x, int8_t r ) 57 | { 58 | return (x >> r) | (x << (32 - r)); 59 | } 60 | 61 | inline uint64_t rotr64 ( uint64_t x, int8_t r ) 62 | { 63 | return (x >> r) | (x << (64 - r)); 64 | } 65 | 66 | #define ROTL32(x,y) rotl32(x,y) 67 | #define ROTL64(x,y) rotl64(x,y) 68 | #define ROTR32(x,y) rotr32(x,y) 69 | #define ROTR64(x,y) rotr64(x,y) 70 | 71 | #define BIG_CONSTANT(x) (x##LLU) 72 | 73 | __inline__ unsigned long long int rdtsc() 74 | { 75 | #ifdef __x86_64__ 76 | unsigned int a, d; 77 | __asm__ volatile ("rdtsc" : "=a" (a), "=d" (d)); 78 | return (unsigned long)a | ((unsigned long)d << 32); 79 | #elif defined(__i386__) 80 | unsigned long long int x; 81 | __asm__ volatile ("rdtsc" : "=A" (x)); 82 | return x; 83 | #else 84 | #define NO_CYCLE_COUNTER 85 | return 0; 86 | #endif 87 | } 88 | 89 | #include 90 | #define _stricmp strcasecmp 91 | 92 | #endif // !defined(_MSC_VER) 93 | 94 | //----------------------------------------------------------------------------- 95 | -------------------------------------------------------------------------------- /src/Random.cpp: -------------------------------------------------------------------------------- 1 | #include "Random.h" 2 | 3 | Rand g_rand1(1); 4 | Rand g_rand2(2); 5 | Rand g_rand3(3); 6 | Rand g_rand4(4); 7 | 8 | //----------------------------------------------------------------------------- 9 | -------------------------------------------------------------------------------- /src/Random.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include "Types.h" 4 | 5 | //----------------------------------------------------------------------------- 6 | // Xorshift RNG based on code by George Marsaglia 7 | // http://en.wikipedia.org/wiki/Xorshift 8 | 9 | struct Rand 10 | { 11 | uint32_t x; 12 | uint32_t y; 13 | uint32_t z; 14 | uint32_t w; 15 | 16 | Rand() 17 | { 18 | reseed(uint32_t(0)); 19 | } 20 | 21 | Rand( uint32_t seed ) 22 | { 23 | reseed(seed); 24 | } 25 | 26 | void reseed ( uint32_t seed ) 27 | { 28 | x = 0x498b3bc5 ^ seed; 29 | y = 0; 30 | z = 0; 31 | w = 0; 32 | 33 | for(int i = 0; i < 10; i++) mix(); 34 | } 35 | 36 | void reseed ( uint64_t seed ) 37 | { 38 | x = 0x498b3bc5 ^ (uint32_t)(seed >> 0); 39 | y = 0x5a05089a ^ (uint32_t)(seed >> 32); 40 | z = 0; 41 | w = 0; 42 | 43 | for(int i = 0; i < 10; i++) mix(); 44 | } 45 | 46 | //----------------------------------------------------------------------------- 47 | 48 | void mix ( void ) 49 | { 50 | uint32_t t = x ^ (x << 11); 51 | x = y; y = z; z = w; 52 | w = w ^ (w >> 19) ^ t ^ (t >> 8); 53 | } 54 | 55 | uint32_t rand_u32 ( void ) 56 | { 57 | mix(); 58 | 59 | return x; 60 | } 61 | 62 | uint64_t rand_u64 ( void ) 63 | { 64 | mix(); 65 | 66 | uint64_t a = x; 67 | uint64_t b = y; 68 | 69 | return (a << 32) | b; 70 | } 71 | 72 | void rand_p ( void * blob, int bytes ) 73 | { 74 | uint32_t * blocks = reinterpret_cast(blob); 75 | 76 | while(bytes >= 4) 77 | { 78 | blocks[0] = rand_u32(); 79 | blocks++; 80 | bytes -= 4; 81 | } 82 | 83 | uint8_t * tail = reinterpret_cast(blocks); 84 | 85 | for(int i = 0; i < bytes; i++) 86 | { 87 | tail[i] = (uint8_t)rand_u32(); 88 | } 89 | } 90 | }; 91 | 92 | //----------------------------------------------------------------------------- 93 | 94 | extern Rand g_rand1; 95 | 96 | inline uint32_t rand_u32 ( void ) { return g_rand1.rand_u32(); } 97 | inline uint64_t rand_u64 ( void ) { return g_rand1.rand_u64(); } 98 | 99 | inline void rand_p ( void * blob, int bytes ) 100 | { 101 | uint32_t * blocks = (uint32_t*)blob; 102 | 103 | while(bytes >= 4) 104 | { 105 | *blocks++ = rand_u32(); 106 | bytes -= 4; 107 | } 108 | 109 | uint8_t * tail = (uint8_t*)blocks; 110 | 111 | for(int i = 0; i < bytes; i++) 112 | { 113 | tail[i] = (uint8_t)rand_u32(); 114 | } 115 | } 116 | 117 | //----------------------------------------------------------------------------- 118 | -------------------------------------------------------------------------------- /src/RiskyHash.h: -------------------------------------------------------------------------------- 1 | /* Originally from https://github.com/boazsegev/facil.io */ 2 | /* 3 | Copyright: Boaz Segev, 2018-2019 4 | License: MIT 5 | 6 | Feel free to copy, use and enjoy according to the license provided. 7 | */ 8 | 9 | /* the 64 bit should be equivalent to fio.h version 10 | * the 128 bit version is a slight modification of the 64 bit */ 11 | 12 | #include 13 | #include 14 | 15 | /** Converts an unaligned network ordered byte stream to a 64 bit number. */ 16 | #define fio_str2u64(c) \ 17 | ((uint64_t)((((uint64_t)((uint8_t *)(c))[0]) << 56) | \ 18 | (((uint64_t)((uint8_t *)(c))[1]) << 48) | \ 19 | (((uint64_t)((uint8_t *)(c))[2]) << 40) | \ 20 | (((uint64_t)((uint8_t *)(c))[3]) << 32) | \ 21 | (((uint64_t)((uint8_t *)(c))[4]) << 24) | \ 22 | (((uint64_t)((uint8_t *)(c))[5]) << 16) | \ 23 | (((uint64_t)((uint8_t *)(c))[6]) << 8) | (((uint8_t *)(c))[7]))) 24 | 25 | /** 64Bit left rotation, inlined. */ 26 | #define fio_lrot64(i, bits) \ 27 | (((uint64_t)(i) << ((bits)&63UL)) | ((uint64_t)(i) >> ((-(bits)) & 63UL))) 28 | 29 | /* The primes used by Risky Hash */ 30 | static const uint64_t RISKY_PRIME_0 = 0xFBBA3FA15B22113B, // 1111101110111010001111111010000101011011001000100001000100111011 31 | RISKY_PRIME_1 = 0xAB137439982B86C9; // 1010101100010011011101000011100110011000001010111000011011001001 32 | 33 | /* Risky Hash consumption round */ 34 | #define fio_risky_consume(v, w) \ 35 | (v) += (w); \ 36 | (v) = fio_lrot64((v), 33); \ 37 | (v) += (w); \ 38 | (v) *= RISKY_PRIME_0; 39 | 40 | #define fio_risky_compress(data, len, v0, v1, v2, v3) \ 41 | for (size_t i = len >> 5; i; --i) { \ 42 | fio_risky_consume(v0, fio_str2u64(data));\ 43 | fio_risky_consume(v1, fio_str2u64(data + 8)); \ 44 | fio_risky_consume(v2, fio_str2u64(data + 16)); \ 45 | fio_risky_consume(v3, fio_str2u64(data + 24)); \ 46 | data += 32; \ 47 | } \ 48 | switch (len & 24) { \ 49 | case 24: \ 50 | fio_risky_consume(v2, fio_str2u64(data + 16)); \ 51 | case 16: \ 52 | fio_risky_consume(v1, fio_str2u64(data + 8)); \ 53 | case 8: \ 54 | fio_risky_consume(v0, fio_str2u64(data)); \ 55 | data += len & 24; \ 56 | } \ 57 | uint64_t tmp = 0; \ 58 | switch ((len & 7)) { \ 59 | case 7: \ 60 | tmp |= ((uint64_t)data[6]) << 8; \ 61 | case 6: \ 62 | tmp |= ((uint64_t)data[5]) << 16; \ 63 | case 5: \ 64 | tmp |= ((uint64_t)data[4]) << 24; \ 65 | case 4: \ 66 | tmp |= ((uint64_t)data[3]) << 32; \ 67 | case 3: \ 68 | tmp |= ((uint64_t)data[2]) << 40; \ 69 | case 2: \ 70 | tmp |= ((uint64_t)data[1]) << 48; \ 71 | case 1: \ 72 | tmp |= ((uint64_t)data[0]) << 56; \ 73 | /* ((len >> 3) & 3) is a 0...3 value indicating consumption vector */ \ 74 | switch ((len >> 3) & 3) { \ 75 | case 3: \ 76 | fio_risky_consume(v3, tmp); \ 77 | break; \ 78 | case 2: \ 79 | fio_risky_consume(v2, tmp); \ 80 | break; \ 81 | case 1: \ 82 | fio_risky_consume(v1, tmp); \ 83 | break; \ 84 | case 0: \ 85 | fio_risky_consume(v0, tmp); \ 86 | break; \ 87 | } \ 88 | } 89 | 90 | uint64_t fio_risky_hash64(const void *data_, size_t len, uint64_t seed) { 91 | /* The consumption vectors initialized state */ 92 | register uint64_t v0 = seed ^ RISKY_PRIME_1; 93 | register uint64_t v1 = ~seed + RISKY_PRIME_1; 94 | register uint64_t v2 = 95 | fio_lrot64(seed, 17) ^ ((~RISKY_PRIME_1) + RISKY_PRIME_0); 96 | register uint64_t v3 = fio_lrot64(seed, 33) + (~RISKY_PRIME_1); 97 | 98 | /* reading position */ 99 | const uint8_t *data = (uint8_t *)data_; 100 | fio_risky_compress(data, len, v0, v1, v2, v3); 101 | 102 | /* merge and mix */ 103 | uint64_t result = fio_lrot64(v0, 17) + fio_lrot64(v1, 13) + 104 | fio_lrot64(v2, 47) + fio_lrot64(v3, 57); 105 | 106 | len ^= (len << 33); 107 | result += len; 108 | 109 | result += v0 * RISKY_PRIME_1; 110 | result ^= fio_lrot64(result, 13); 111 | result += v1 * RISKY_PRIME_1; 112 | result ^= fio_lrot64(result, 29); 113 | result += v2 * RISKY_PRIME_1; 114 | result ^= fio_lrot64(result, 33); 115 | result += v3 * RISKY_PRIME_1; 116 | result ^= fio_lrot64(result, 51); 117 | 118 | /* irreversible avalanche... I think */ 119 | result ^= (result >> 29) * RISKY_PRIME_0; 120 | return result; 121 | } 122 | 123 | 124 | void fio_risky_hash128(const void *data_, size_t len, uint64_t *h1, uint64_t *h2) { 125 | /* The consumption vectors initialized state */ 126 | register uint64_t v0 = h1[0] ^ RISKY_PRIME_1; 127 | register uint64_t v1 = ~h2[0] + RISKY_PRIME_1; 128 | register uint64_t v2 = 129 | fio_lrot64(h1[0], 17) ^ ((~RISKY_PRIME_1) + RISKY_PRIME_0); 130 | register uint64_t v3 = fio_lrot64(h2[0], 33) + (~RISKY_PRIME_1); 131 | 132 | const uint8_t *data = (const uint8_t *) data_; 133 | fio_risky_compress(data, len, v0, v1, v2, v3); 134 | 135 | /* merge and mix */ 136 | h1[0] = fio_lrot64(v0, 17) + fio_lrot64(v1, 13) + 137 | fio_lrot64(v2, 47) + fio_lrot64(v3, 57); 138 | h2[0] = fio_lrot64(v2, 17) + fio_lrot64(v3, 13) + 139 | fio_lrot64(v0, 47) + fio_lrot64(v1, 57); 140 | len ^= (len << 33); 141 | h1[0] += len; 142 | h1[0] += v0 * RISKY_PRIME_1; 143 | h1[0] ^= fio_lrot64(h1[0], 13); 144 | h1[0] += v1 * RISKY_PRIME_1; 145 | h1[0] ^= fio_lrot64(h1[0], 29); 146 | h1[0] += v2 * RISKY_PRIME_1; 147 | h1[0] ^= fio_lrot64(h1[0], 33); 148 | h1[0] += v3 * RISKY_PRIME_1; 149 | h1[0] ^= fio_lrot64(h1[0], 51); 150 | h1[0] ^= (h1[0] >> 29) * RISKY_PRIME_0; 151 | h2[0] ^= (h1[0] >> 29) * RISKY_PRIME_0; 152 | } 153 | 154 | #undef fio_risky_compress 155 | #undef fio_risky_consume 156 | -------------------------------------------------------------------------------- /src/RiskyTest.cpp: -------------------------------------------------------------------------------- 1 | #include "RiskyHash.h" 2 | 3 | void RiskyHash128_test(const void *key, int len, uint32_t seed, void *out) { 4 | uint64_t h1 = seed, h2 = seed; 5 | fio_risky_hash128(key, len, &h1, &h2); 6 | ((uint64_t*)out)[0] = h1; 7 | ((uint64_t*)out)[1] = h2; 8 | } 9 | 10 | void RiskyHash64_test(const void *key, int len, uint32_t seed, void *out) { 11 | ((uint64_t*)out)[0] = fio_risky_hash64(key, len, seed); 12 | 13 | /* use the first half of 128b: 14 | uint64_t h1 = seed, h2 = seed; 15 | fio_risky_hash128(key, len, &h1, &h2); 16 | ((uint64_t*)out)[0] = h1; */ 17 | 18 | /* use the second half of 128b: 19 | uint64_t h1 = seed, h2 = seed; 20 | fio_risky_hash128(key, len, &h1, &h2); 21 | ((uint64_t*)out)[0] = h2; */ 22 | } 23 | -------------------------------------------------------------------------------- /src/SpeedTest.cpp: -------------------------------------------------------------------------------- 1 | #include "SpeedTest.h" 2 | 3 | #include "Random.h" 4 | 5 | #include // for printf 6 | #include // for memset 7 | #include // for sqrt 8 | #include // for sort 9 | 10 | //----------------------------------------------------------------------------- 11 | // We view our timing values as a series of random variables V that has been 12 | // contaminated with occasional outliers due to cache misses, thread 13 | // preemption, etcetera. To filter out the outliers, we search for the largest 14 | // subset of V such that all its values are within three standard deviations 15 | // of the mean. 16 | 17 | double CalcMean ( std::vector & v ) 18 | { 19 | double mean = 0; 20 | 21 | for(int i = 0; i < (int)v.size(); i++) 22 | { 23 | mean += v[i]; 24 | } 25 | 26 | mean /= double(v.size()); 27 | 28 | return mean; 29 | } 30 | 31 | double CalcMean ( std::vector & v, int a, int b ) 32 | { 33 | double mean = 0; 34 | 35 | for(int i = a; i <= b; i++) 36 | { 37 | mean += v[i]; 38 | } 39 | 40 | mean /= (b-a+1); 41 | 42 | return mean; 43 | } 44 | 45 | double CalcStdv ( std::vector & v, int a, int b ) 46 | { 47 | double mean = CalcMean(v,a,b); 48 | 49 | double stdv = 0; 50 | 51 | for(int i = a; i <= b; i++) 52 | { 53 | double x = v[i] - mean; 54 | 55 | stdv += x*x; 56 | } 57 | 58 | stdv = sqrt(stdv / (b-a+1)); 59 | 60 | return stdv; 61 | } 62 | 63 | // Return true if the largest value in v[0,len) is more than three 64 | // standard deviations from the mean 65 | 66 | bool ContainsOutlier ( std::vector & v, size_t len ) 67 | { 68 | double mean = 0; 69 | 70 | for(size_t i = 0; i < len; i++) 71 | { 72 | mean += v[i]; 73 | } 74 | 75 | mean /= double(len); 76 | 77 | double stdv = 0; 78 | 79 | for(size_t i = 0; i < len; i++) 80 | { 81 | double x = v[i] - mean; 82 | stdv += x*x; 83 | } 84 | 85 | stdv = sqrt(stdv / double(len)); 86 | 87 | double cutoff = mean + stdv*3; 88 | 89 | return v[len-1] > cutoff; 90 | } 91 | 92 | // Do a binary search to find the largest subset of v that does not contain 93 | // outliers. 94 | 95 | void FilterOutliers ( std::vector & v ) 96 | { 97 | std::sort(v.begin(),v.end()); 98 | 99 | size_t len = 0; 100 | 101 | for(size_t x = 0x40000000; x; x = x >> 1 ) 102 | { 103 | if((len | x) >= v.size()) continue; 104 | 105 | if(!ContainsOutlier(v,len | x)) 106 | { 107 | len |= x; 108 | } 109 | } 110 | 111 | v.resize(len); 112 | } 113 | 114 | // Iteratively tighten the set to find a subset that does not contain 115 | // outliers. I'm not positive this works correctly in all cases. 116 | 117 | void FilterOutliers2 ( std::vector & v ) 118 | { 119 | std::sort(v.begin(),v.end()); 120 | 121 | int a = 0; 122 | int b = (int)(v.size() - 1); 123 | 124 | for(int i = 0; i < 10; i++) 125 | { 126 | //printf("%d %d\n",a,b); 127 | 128 | double mean = CalcMean(v,a,b); 129 | double stdv = CalcStdv(v,a,b); 130 | 131 | double cutA = mean - stdv*3; 132 | double cutB = mean + stdv*3; 133 | 134 | while((a < b) && (v[a] < cutA)) a++; 135 | while((b > a) && (v[b] > cutB)) b--; 136 | } 137 | 138 | std::vector v2; 139 | 140 | v2.insert(v2.begin(),v.begin()+a,v.begin()+b+1); 141 | 142 | v.swap(v2); 143 | } 144 | 145 | //----------------------------------------------------------------------------- 146 | // We really want the rdtsc() calls to bracket the function call as tightly 147 | // as possible, but that's hard to do portably. We'll try and get as close as 148 | // possible by marking the function as NEVER_INLINE (to keep the optimizer from 149 | // moving it) and marking the timing variables as "volatile register". 150 | int64_t total; 151 | NEVER_INLINE int64_t timehash ( pfHash hash, const void * key, int len, int seed, bool speedseed ) 152 | { 153 | volatile register int64_t begin,end; 154 | 155 | uint64_t temp[128 / 8]; 156 | if (len < 1024) 157 | { 158 | uint64_t align_len = (len + 7) / 8; 159 | uint64_t *buf = new uint64_t[align_len]; /* from rurban */ 160 | memset(buf, 0, align_len * 8); 161 | memcpy(buf, key, len); 162 | if ( speedseed ) { 163 | begin = rdtsc(); 164 | for (int i = 0; i < 32; i++) { 165 | hash(buf,len,seed,temp); 166 | seed += temp[0]; 167 | /*buf[0] ^= temp[0];*/ 168 | } 169 | end = rdtsc(); 170 | } 171 | else { 172 | begin = rdtsc(); 173 | for (int i = 0; i < 32; i++) { 174 | hash(buf,len,seed,temp); 175 | total += temp[0]; 176 | } 177 | end = rdtsc(); 178 | } 179 | delete [] buf; 180 | return (end-begin) >> 5; 181 | } 182 | begin = rdtsc(); 183 | hash(key,len,seed,temp); 184 | end = rdtsc(); 185 | return end-begin; 186 | } 187 | 188 | //----------------------------------------------------------------------------- 189 | 190 | double SpeedTest ( pfHash hash, uint32_t seed, const int trials, const int blocksize, const int align, bool speedseed ) 191 | { 192 | Rand r(seed); 193 | 194 | uint8_t * buf = new uint8_t[blocksize + 512]; 195 | 196 | uint64_t t1 = reinterpret_cast(buf); 197 | 198 | t1 = (t1 + 255) & BIG_CONSTANT(0xFFFFFFFFFFFFFF00); 199 | t1 += align; 200 | 201 | uint8_t * block = reinterpret_cast(t1); 202 | 203 | r.rand_p(block,blocksize); 204 | 205 | //---------- 206 | 207 | std::vector times; 208 | times.reserve(trials); 209 | 210 | for(int itrial = 0; itrial < trials; itrial++) 211 | { 212 | r.rand_p(block,blocksize); 213 | 214 | double t = (double)timehash(hash,block,blocksize,itrial,speedseed); 215 | 216 | if(t > 0) times.push_back(t); 217 | } 218 | 219 | //---------- 220 | 221 | std::sort(times.begin(),times.end()); 222 | 223 | FilterOutliers(times); 224 | 225 | delete [] buf; 226 | 227 | return CalcMean(times); 228 | } 229 | 230 | //----------------------------------------------------------------------------- 231 | // 256k blocks seem to give the best results. 232 | 233 | void BulkSpeedTest ( pfHash hash, uint32_t seed, bool plot ) 234 | { 235 | const int trials = 2999; 236 | const int blocksize = 256 * 1024; 237 | 238 | if ( ! plot ) 239 | printf("Bulk speed test - %d-byte keys\n",blocksize); 240 | 241 | for(int align = 0; align < 8; align++) 242 | { 243 | double cycles = SpeedTest(hash,seed,trials,blocksize,align,false); 244 | 245 | double bestbpc = double(blocksize)/cycles; 246 | 247 | double bestbps = (bestbpc * 3000000000.0 / 1048576.0); 248 | if ( plot ) 249 | printf("%2d %6.3f\n",align,bestbpc); 250 | else 251 | printf("Alignment %2d - %6.3f bytes/cycle - %7.2f MiB/sec @ 3 ghz\n",align,bestbpc,bestbps); 252 | } 253 | } 254 | 255 | //----------------------------------------------------------------------------- 256 | 257 | void TinySpeedTest ( pfHash hash, int hashsize, int keysize, uint32_t seed, double & /*outCycles*/, bool plot, bool speedseed ) 258 | { 259 | const int trials = 999999; 260 | 261 | if ( ! plot ) 262 | printf("Small key speed test - %4d-byte keys - ",keysize); 263 | else 264 | printf("%d ",keysize); 265 | 266 | double cycles = SpeedTest(hash,seed,trials,keysize,0,speedseed); 267 | 268 | if ( ! plot ) 269 | printf("%8.2f cycles/hash\n",cycles); 270 | else 271 | printf("%8.2f\n",cycles); 272 | } 273 | 274 | //----------------------------------------------------------------------------- 275 | -------------------------------------------------------------------------------- /src/SpeedTest.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include "Types.h" 4 | 5 | void BulkSpeedTest ( pfHash hash, uint32_t seed, bool plot ); 6 | void TinySpeedTest ( pfHash hash, int hashsize, int keysize, uint32_t seed, double & outCycles, bool plot, bool speedseed ); 7 | 8 | //----------------------------------------------------------------------------- 9 | -------------------------------------------------------------------------------- /src/Spooky.cpp: -------------------------------------------------------------------------------- 1 | // Spooky Hash 2 | // A 128-bit noncryptographic hash, for checksums and table lookup 3 | // By Bob Jenkins. Public domain. 4 | // Oct 31 2010: published framework, disclaimer ShortHash isn't right 5 | // Nov 7 2010: disabled ShortHash 6 | // Oct 31 2011: replace End, ShortMix, ShortEnd, enable ShortHash again 7 | 8 | #include 9 | #include "Spooky.h" 10 | 11 | #define ALLOW_UNALIGNED_READS 1 12 | 13 | // 14 | // short hash ... it could be used on any message, 15 | // but it's used by Spooky just for short messages. 16 | // 17 | void SpookyHash::Short( 18 | const void *message, 19 | size_t length, 20 | uint64 *hash1, 21 | uint64 *hash2) 22 | { 23 | uint64 buf[sc_numVars]; 24 | union 25 | { 26 | const uint8 *p8; 27 | uint32 *p32; 28 | uint64 *p64; 29 | size_t i; 30 | } u; 31 | 32 | u.p8 = (const uint8 *)message; 33 | 34 | if (!ALLOW_UNALIGNED_READS && (u.i & 0x7)) 35 | { 36 | memcpy(buf, message, length); 37 | u.p64 = buf; 38 | } 39 | 40 | size_t remainder = length%32; 41 | uint64 a=*hash1; 42 | uint64 b=*hash2; 43 | uint64 c=sc_const; 44 | uint64 d=sc_const; 45 | 46 | if (length > 15) 47 | { 48 | const uint64 *end = u.p64 + (length/32)*4; 49 | 50 | // handle all complete sets of 32 bytes 51 | for (; u.p64 < end; u.p64 += 4) 52 | { 53 | c += u.p64[0]; 54 | d += u.p64[1]; 55 | ShortMix(a,b,c,d); 56 | a += u.p64[2]; 57 | b += u.p64[3]; 58 | } 59 | 60 | //Handle the case of 16+ remaining bytes. 61 | if (remainder >= 16) 62 | { 63 | c += u.p64[0]; 64 | d += u.p64[1]; 65 | ShortMix(a,b,c,d); 66 | u.p64 += 2; 67 | remainder -= 16; 68 | } 69 | } 70 | 71 | // Handle the last 0..15 bytes, and its length 72 | d = ((uint64)length) << 56; 73 | switch (remainder) 74 | { 75 | case 15: 76 | d += ((uint64)u.p8[14]) << 48; 77 | case 14: 78 | d += ((uint64)u.p8[13]) << 40; 79 | case 13: 80 | d += ((uint64)u.p8[12]) << 32; 81 | case 12: 82 | d += u.p32[2]; 83 | c += u.p64[0]; 84 | break; 85 | case 11: 86 | d += ((uint64)u.p8[10]) << 16; 87 | case 10: 88 | d += ((uint64)u.p8[9]) << 8; 89 | case 9: 90 | d += (uint64)u.p8[8]; 91 | case 8: 92 | c += u.p64[0]; 93 | break; 94 | case 7: 95 | c += ((uint64)u.p8[6]) << 48; 96 | case 6: 97 | c += ((uint64)u.p8[5]) << 40; 98 | case 5: 99 | c += ((uint64)u.p8[4]) << 32; 100 | case 4: 101 | c += u.p32[0]; 102 | break; 103 | case 3: 104 | c += ((uint64)u.p8[2]) << 16; 105 | case 2: 106 | c += ((uint64)u.p8[1]) << 8; 107 | case 1: 108 | c += (uint64)u.p8[0]; 109 | break; 110 | case 0: 111 | c += sc_const; 112 | d += sc_const; 113 | } 114 | ShortEnd(a,b,c,d); 115 | *hash1 = a; 116 | *hash2 = b; 117 | } 118 | 119 | 120 | 121 | 122 | // do the whole hash in one call 123 | void SpookyHash::Hash128( 124 | const void *message, 125 | size_t length, 126 | uint64 *hash1, 127 | uint64 *hash2) 128 | { 129 | if (length < sc_bufSize) 130 | { 131 | Short(message, length, hash1, hash2); 132 | return; 133 | } 134 | 135 | uint64 h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11; 136 | uint64 buf[sc_numVars]; 137 | uint64 *end; 138 | union 139 | { 140 | const uint8 *p8; 141 | uint64 *p64; 142 | size_t i; 143 | } u; 144 | size_t remainder; 145 | 146 | h0=h3=h6=h9 = *hash1; 147 | h1=h4=h7=h10 = *hash2; 148 | h2=h5=h8=h11 = sc_const; 149 | 150 | u.p8 = (const uint8 *)message; 151 | end = u.p64 + (length/sc_blockSize)*sc_numVars; 152 | 153 | // handle all whole sc_blockSize blocks of bytes 154 | if (ALLOW_UNALIGNED_READS || ((u.i & 0x7) == 0)) 155 | { 156 | while (u.p64 < end) 157 | { 158 | Mix(u.p64, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 159 | u.p64 += sc_numVars; 160 | } 161 | } 162 | else 163 | { 164 | while (u.p64 < end) 165 | { 166 | memcpy(buf, u.p64, sc_blockSize); 167 | Mix(buf, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 168 | u.p64 += sc_numVars; 169 | } 170 | } 171 | 172 | // handle the last partial block of sc_blockSize bytes 173 | remainder = (length - ((const uint8 *)end-(const uint8 *)message)); 174 | memcpy(buf, end, remainder); 175 | memset(((uint8 *)buf)+remainder, 0, sc_blockSize-remainder); 176 | ((uint8 *)buf)[sc_blockSize-1] = remainder; 177 | Mix(buf, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 178 | 179 | // do some final mixing 180 | End(h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 181 | *hash1 = h0; 182 | *hash2 = h1; 183 | } 184 | 185 | 186 | 187 | // init spooky state 188 | void SpookyHash::Init(uint64 seed1, uint64 seed2) 189 | { 190 | m_length = 0; 191 | m_remainder = 0; 192 | m_state[0] = seed1; 193 | m_state[1] = seed2; 194 | } 195 | 196 | 197 | // add a message fragment to the state 198 | void SpookyHash::Update(const void *message, size_t length) 199 | { 200 | uint64 h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11; 201 | size_t newLength = length + m_remainder; 202 | uint8 remainder; 203 | union 204 | { 205 | const uint8 *p8; 206 | uint64 *p64; 207 | size_t i; 208 | } u; 209 | const uint64 *end; 210 | 211 | // Is this message fragment too short? If it is, stuff it away. 212 | if (newLength < sc_bufSize) 213 | { 214 | memcpy(&((uint8 *)m_data)[m_remainder], message, length); 215 | m_length = length + m_length; 216 | m_remainder = (uint8)newLength; 217 | return; 218 | } 219 | 220 | // init the variables 221 | if (m_length < sc_bufSize) 222 | { 223 | h0=h3=h6=h9 = m_state[0]; 224 | h1=h4=h7=h10 = m_state[1]; 225 | h2=h5=h8=h11 = sc_const; 226 | } 227 | else 228 | { 229 | h0 = m_state[0]; 230 | h1 = m_state[1]; 231 | h2 = m_state[2]; 232 | h3 = m_state[3]; 233 | h4 = m_state[4]; 234 | h5 = m_state[5]; 235 | h6 = m_state[6]; 236 | h7 = m_state[7]; 237 | h8 = m_state[8]; 238 | h9 = m_state[9]; 239 | h10 = m_state[10]; 240 | h11 = m_state[11]; 241 | } 242 | m_length = length + m_length; 243 | 244 | // if we've got anything stuffed away, use it now 245 | if (m_remainder) 246 | { 247 | uint8 prefix = sc_bufSize-m_remainder; 248 | memcpy(&(((uint8 *)m_data)[m_remainder]), message, prefix); 249 | u.p64 = m_data; 250 | Mix(u.p64, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 251 | Mix(&u.p64[sc_numVars], h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 252 | u.p8 = ((const uint8 *)message) + prefix; 253 | length -= prefix; 254 | } 255 | else 256 | { 257 | u.p8 = (const uint8 *)message; 258 | } 259 | 260 | // handle all whole blocks of sc_blockSize bytes 261 | end = u.p64 + (length/sc_blockSize)*sc_numVars; 262 | remainder = (uint8)(length-((const uint8 *)end-u.p8)); 263 | if (ALLOW_UNALIGNED_READS || (u.i & 0x7) == 0) 264 | { 265 | while (u.p64 < end) 266 | { 267 | Mix(u.p64, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 268 | u.p64 += sc_numVars; 269 | } 270 | } 271 | else 272 | { 273 | while (u.p64 < end) 274 | { 275 | memcpy(m_data, u.p8, sc_blockSize); 276 | Mix(m_data, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 277 | u.p64 += sc_numVars; 278 | } 279 | } 280 | 281 | // stuff away the last few bytes 282 | m_remainder = remainder; 283 | memcpy(m_data, end, remainder); 284 | 285 | // stuff away the variables 286 | m_state[0] = h0; 287 | m_state[1] = h1; 288 | m_state[2] = h2; 289 | m_state[3] = h3; 290 | m_state[4] = h4; 291 | m_state[5] = h5; 292 | m_state[6] = h6; 293 | m_state[7] = h7; 294 | m_state[8] = h8; 295 | m_state[9] = h9; 296 | m_state[10] = h10; 297 | m_state[11] = h11; 298 | } 299 | 300 | 301 | // report the hash for the concatenation of all message fragments so far 302 | void SpookyHash::Final(uint64 *hash1, uint64 *hash2) 303 | { 304 | // init the variables 305 | if (m_length < sc_bufSize) 306 | { 307 | Short( m_data, m_length, hash1, hash2); 308 | return; 309 | } 310 | 311 | const uint64 *data = (const uint64 *)m_data; 312 | uint8 remainder = m_remainder; 313 | 314 | uint64 h0 = m_state[0]; 315 | uint64 h1 = m_state[1]; 316 | uint64 h2 = m_state[2]; 317 | uint64 h3 = m_state[3]; 318 | uint64 h4 = m_state[4]; 319 | uint64 h5 = m_state[5]; 320 | uint64 h6 = m_state[6]; 321 | uint64 h7 = m_state[7]; 322 | uint64 h8 = m_state[8]; 323 | uint64 h9 = m_state[9]; 324 | uint64 h10 = m_state[10]; 325 | uint64 h11 = m_state[11]; 326 | 327 | if (remainder >= sc_blockSize) 328 | { 329 | // m_data can contain two blocks; handle any whole first block 330 | Mix(data, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 331 | data += sc_numVars; 332 | remainder -= sc_blockSize; 333 | } 334 | 335 | // mix in the last partial block, and the length mod sc_blockSize 336 | memset(&((uint8 *)data)[remainder], 0, (sc_blockSize-remainder)); 337 | 338 | ((uint8 *)data)[sc_blockSize-1] = remainder; 339 | Mix(data, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 340 | 341 | // do some final mixing 342 | End(h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 343 | 344 | *hash1 = h0; 345 | *hash2 = h1; 346 | } 347 | 348 | -------------------------------------------------------------------------------- /src/SpookyTest.cpp: -------------------------------------------------------------------------------- 1 | #include "Spooky.h" 2 | #include "SpookyV2.h" 3 | 4 | void SpookyHash32_test(const void *key, int len, uint32_t seed, void *out) { 5 | *(uint32_t*)out = SpookyHash::Hash32(key, len, seed); 6 | } 7 | 8 | void SpookyHash64_test(const void *key, int len, uint32_t seed, void *out) { 9 | *(uint64_t*)out = SpookyHash::Hash64(key, len, seed); 10 | } 11 | 12 | void SpookyHash128_test(const void *key, int len, uint32_t seed, void *out) { 13 | uint64_t h1 = seed, h2 = seed; 14 | SpookyHash::Hash128(key, len, &h1, &h2); 15 | ((uint64_t*)out)[0] = h1; 16 | ((uint64_t*)out)[1] = h2; 17 | } 18 | 19 | void SpookyHash128V2_test(const void *key, int len, uint32_t seed, void *out) { 20 | uint64_t h1 = seed, h2 = seed; 21 | SpookyHashV2::Hash128(key, len, &h1, &h2); 22 | ((uint64_t*)out)[0] = h1; 23 | ((uint64_t*)out)[1] = h2; 24 | } 25 | 26 | void SpookyHash64V2_test(const void *key, int len, uint32_t seed, void *out) { 27 | *(uint64_t*)out = SpookyHashV2::Hash64(key, len, seed); 28 | } 29 | -------------------------------------------------------------------------------- /src/SpookyV2.cpp: -------------------------------------------------------------------------------- 1 | // Spooky Hash 2 | // A 128-bit noncryptographic hash, for checksums and table lookup 3 | // By Bob Jenkins. Public domain. 4 | // Oct 31 2010: published framework, disclaimer ShortHash isn't right 5 | // Nov 7 2010: disabled ShortHash 6 | // Oct 31 2011: replace End, ShortMix, ShortEnd, enable ShortHash again 7 | // April 10 2012: buffer overflow on platforms without unaligned reads 8 | // July 12 2012: was passing out variables in final to in/out in short 9 | // July 30 2012: I reintroduced the buffer overflow 10 | // August 5 2012: SpookyV2: d = should be d += in short hash, and remove extra mix from long hash 11 | 12 | #include 13 | #include "SpookyV2.h" 14 | 15 | #define ALLOW_UNALIGNED_READS 1 16 | 17 | // 18 | // short hash ... it could be used on any message, 19 | // but it's used by Spooky just for short messages. 20 | // 21 | void SpookyHashV2::Short( 22 | const void *message, 23 | size_t length, 24 | uint64 *hash1, 25 | uint64 *hash2) 26 | { 27 | uint64 buf[2*sc_numVars]; 28 | union 29 | { 30 | const uint8 *p8; 31 | uint32 *p32; 32 | uint64 *p64; 33 | size_t i; 34 | } u; 35 | 36 | u.p8 = (const uint8 *)message; 37 | 38 | if (!ALLOW_UNALIGNED_READS && (u.i & 0x7)) 39 | { 40 | memcpy(buf, message, length); 41 | u.p64 = buf; 42 | } 43 | 44 | size_t remainder = length%32; 45 | uint64 a=*hash1; 46 | uint64 b=*hash2; 47 | uint64 c=sc_const; 48 | uint64 d=sc_const; 49 | 50 | if (length > 15) 51 | { 52 | const uint64 *end = u.p64 + (length/32)*4; 53 | 54 | // handle all complete sets of 32 bytes 55 | for (; u.p64 < end; u.p64 += 4) 56 | { 57 | c += u.p64[0]; 58 | d += u.p64[1]; 59 | ShortMix(a,b,c,d); 60 | a += u.p64[2]; 61 | b += u.p64[3]; 62 | } 63 | 64 | //Handle the case of 16+ remaining bytes. 65 | if (remainder >= 16) 66 | { 67 | c += u.p64[0]; 68 | d += u.p64[1]; 69 | ShortMix(a,b,c,d); 70 | u.p64 += 2; 71 | remainder -= 16; 72 | } 73 | } 74 | 75 | // Handle the last 0..15 bytes, and its length 76 | d += ((uint64)length) << 56; 77 | switch (remainder) 78 | { 79 | case 15: 80 | d += ((uint64)u.p8[14]) << 48; 81 | case 14: 82 | d += ((uint64)u.p8[13]) << 40; 83 | case 13: 84 | d += ((uint64)u.p8[12]) << 32; 85 | case 12: 86 | d += u.p32[2]; 87 | c += u.p64[0]; 88 | break; 89 | case 11: 90 | d += ((uint64)u.p8[10]) << 16; 91 | case 10: 92 | d += ((uint64)u.p8[9]) << 8; 93 | case 9: 94 | d += (uint64)u.p8[8]; 95 | case 8: 96 | c += u.p64[0]; 97 | break; 98 | case 7: 99 | c += ((uint64)u.p8[6]) << 48; 100 | case 6: 101 | c += ((uint64)u.p8[5]) << 40; 102 | case 5: 103 | c += ((uint64)u.p8[4]) << 32; 104 | case 4: 105 | c += u.p32[0]; 106 | break; 107 | case 3: 108 | c += ((uint64)u.p8[2]) << 16; 109 | case 2: 110 | c += ((uint64)u.p8[1]) << 8; 111 | case 1: 112 | c += (uint64)u.p8[0]; 113 | break; 114 | case 0: 115 | c += sc_const; 116 | d += sc_const; 117 | } 118 | ShortEnd(a,b,c,d); 119 | *hash1 = a; 120 | *hash2 = b; 121 | } 122 | 123 | 124 | 125 | 126 | // do the whole hash in one call 127 | void SpookyHashV2::Hash128( 128 | const void *message, 129 | size_t length, 130 | uint64 *hash1, 131 | uint64 *hash2) 132 | { 133 | if (length < sc_bufSize) 134 | { 135 | Short(message, length, hash1, hash2); 136 | return; 137 | } 138 | 139 | uint64 h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11; 140 | uint64 buf[sc_numVars]; 141 | uint64 *end; 142 | union 143 | { 144 | const uint8 *p8; 145 | uint64 *p64; 146 | size_t i; 147 | } u; 148 | size_t remainder; 149 | 150 | h0=h3=h6=h9 = *hash1; 151 | h1=h4=h7=h10 = *hash2; 152 | h2=h5=h8=h11 = sc_const; 153 | 154 | u.p8 = (const uint8 *)message; 155 | end = u.p64 + (length/sc_blockSize)*sc_numVars; 156 | 157 | // handle all whole sc_blockSize blocks of bytes 158 | if (ALLOW_UNALIGNED_READS || ((u.i & 0x7) == 0)) 159 | { 160 | while (u.p64 < end) 161 | { 162 | Mix(u.p64, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 163 | u.p64 += sc_numVars; 164 | } 165 | } 166 | else 167 | { 168 | while (u.p64 < end) 169 | { 170 | memcpy(buf, u.p64, sc_blockSize); 171 | Mix(buf, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 172 | u.p64 += sc_numVars; 173 | } 174 | } 175 | 176 | // handle the last partial block of sc_blockSize bytes 177 | remainder = (length - ((const uint8 *)end-(const uint8 *)message)); 178 | memcpy(buf, end, remainder); 179 | memset(((uint8 *)buf)+remainder, 0, sc_blockSize-remainder); 180 | ((uint8 *)buf)[sc_blockSize-1] = remainder; 181 | 182 | // do some final mixing 183 | End(buf, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 184 | *hash1 = h0; 185 | *hash2 = h1; 186 | } 187 | 188 | 189 | 190 | // init spooky state 191 | void SpookyHashV2::Init(uint64 seed1, uint64 seed2) 192 | { 193 | m_length = 0; 194 | m_remainder = 0; 195 | m_state[0] = seed1; 196 | m_state[1] = seed2; 197 | } 198 | 199 | 200 | // add a message fragment to the state 201 | void SpookyHashV2::Update(const void *message, size_t length) 202 | { 203 | uint64 h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11; 204 | size_t newLength = length + m_remainder; 205 | uint8 remainder; 206 | union 207 | { 208 | const uint8 *p8; 209 | uint64 *p64; 210 | size_t i; 211 | } u; 212 | const uint64 *end; 213 | 214 | // Is this message fragment too short? If it is, stuff it away. 215 | if (newLength < sc_bufSize) 216 | { 217 | memcpy(&((uint8 *)m_data)[m_remainder], message, length); 218 | m_length = length + m_length; 219 | m_remainder = (uint8)newLength; 220 | return; 221 | } 222 | 223 | // init the variables 224 | if (m_length < sc_bufSize) 225 | { 226 | h0=h3=h6=h9 = m_state[0]; 227 | h1=h4=h7=h10 = m_state[1]; 228 | h2=h5=h8=h11 = sc_const; 229 | } 230 | else 231 | { 232 | h0 = m_state[0]; 233 | h1 = m_state[1]; 234 | h2 = m_state[2]; 235 | h3 = m_state[3]; 236 | h4 = m_state[4]; 237 | h5 = m_state[5]; 238 | h6 = m_state[6]; 239 | h7 = m_state[7]; 240 | h8 = m_state[8]; 241 | h9 = m_state[9]; 242 | h10 = m_state[10]; 243 | h11 = m_state[11]; 244 | } 245 | m_length = length + m_length; 246 | 247 | // if we've got anything stuffed away, use it now 248 | if (m_remainder) 249 | { 250 | uint8 prefix = sc_bufSize-m_remainder; 251 | memcpy(&(((uint8 *)m_data)[m_remainder]), message, prefix); 252 | u.p64 = m_data; 253 | Mix(u.p64, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 254 | Mix(&u.p64[sc_numVars], h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 255 | u.p8 = ((const uint8 *)message) + prefix; 256 | length -= prefix; 257 | } 258 | else 259 | { 260 | u.p8 = (const uint8 *)message; 261 | } 262 | 263 | // handle all whole blocks of sc_blockSize bytes 264 | end = u.p64 + (length/sc_blockSize)*sc_numVars; 265 | remainder = (uint8)(length-((const uint8 *)end-u.p8)); 266 | if (ALLOW_UNALIGNED_READS || (u.i & 0x7) == 0) 267 | { 268 | while (u.p64 < end) 269 | { 270 | Mix(u.p64, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 271 | u.p64 += sc_numVars; 272 | } 273 | } 274 | else 275 | { 276 | while (u.p64 < end) 277 | { 278 | memcpy(m_data, u.p8, sc_blockSize); 279 | Mix(m_data, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 280 | u.p64 += sc_numVars; 281 | } 282 | } 283 | 284 | // stuff away the last few bytes 285 | m_remainder = remainder; 286 | memcpy(m_data, end, remainder); 287 | 288 | // stuff away the variables 289 | m_state[0] = h0; 290 | m_state[1] = h1; 291 | m_state[2] = h2; 292 | m_state[3] = h3; 293 | m_state[4] = h4; 294 | m_state[5] = h5; 295 | m_state[6] = h6; 296 | m_state[7] = h7; 297 | m_state[8] = h8; 298 | m_state[9] = h9; 299 | m_state[10] = h10; 300 | m_state[11] = h11; 301 | } 302 | 303 | 304 | // report the hash for the concatenation of all message fragments so far 305 | void SpookyHashV2::Final(uint64 *hash1, uint64 *hash2) 306 | { 307 | // init the variables 308 | if (m_length < sc_bufSize) 309 | { 310 | *hash1 = m_state[0]; 311 | *hash2 = m_state[1]; 312 | Short( m_data, m_length, hash1, hash2); 313 | return; 314 | } 315 | 316 | const uint64 *data = (const uint64 *)m_data; 317 | uint8 remainder = m_remainder; 318 | 319 | uint64 h0 = m_state[0]; 320 | uint64 h1 = m_state[1]; 321 | uint64 h2 = m_state[2]; 322 | uint64 h3 = m_state[3]; 323 | uint64 h4 = m_state[4]; 324 | uint64 h5 = m_state[5]; 325 | uint64 h6 = m_state[6]; 326 | uint64 h7 = m_state[7]; 327 | uint64 h8 = m_state[8]; 328 | uint64 h9 = m_state[9]; 329 | uint64 h10 = m_state[10]; 330 | uint64 h11 = m_state[11]; 331 | 332 | if (remainder >= sc_blockSize) 333 | { 334 | // m_data can contain two blocks; handle any whole first block 335 | Mix(data, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 336 | data += sc_numVars; 337 | remainder -= sc_blockSize; 338 | } 339 | 340 | // mix in the last partial block, and the length mod sc_blockSize 341 | memset(&((uint8 *)data)[remainder], 0, (sc_blockSize-remainder)); 342 | 343 | ((uint8 *)data)[sc_blockSize-1] = remainder; 344 | 345 | // do some final mixing 346 | End(data, h0,h1,h2,h3,h4,h5,h6,h7,h8,h9,h10,h11); 347 | 348 | *hash1 = h0; 349 | *hash2 = h1; 350 | } 351 | 352 | -------------------------------------------------------------------------------- /src/Stats.cpp: -------------------------------------------------------------------------------- 1 | #include "Stats.h" 2 | 3 | //----------------------------------------------------------------------------- 4 | 5 | double chooseK ( int n, int k ) 6 | { 7 | if(k > (n - k)) k = n - k; 8 | 9 | double c = 1; 10 | 11 | for(int i = 0; i < k; i++) 12 | { 13 | double t = double(n-i) / double(i+1); 14 | 15 | c *= t; 16 | } 17 | 18 | return c; 19 | } 20 | 21 | double chooseUpToK ( int n, int k ) 22 | { 23 | double c = 0; 24 | 25 | for(int i = 1; i <= k; i++) 26 | { 27 | c += chooseK(n,i); 28 | } 29 | 30 | return c; 31 | } 32 | 33 | //----------------------------------------------------------------------------- 34 | // Distribution "score" 35 | // TODO - big writeup of what this score means 36 | 37 | // Basically, we're computing a constant that says "The test distribution is as 38 | // uniform, RMS-wise, as a random distribution restricted to (1-X)*100 percent of 39 | // the bins. This makes for a nice uniform way to rate a distribution that isn't 40 | // dependent on the number of bins or the number of keys 41 | 42 | // (as long as # keys > # bins * 3 or so, otherwise random fluctuations show up 43 | // as distribution weaknesses) 44 | 45 | double calcScore ( const int * bins, const int bincount, const int keycount ) 46 | { 47 | double n = bincount; 48 | double k = keycount; 49 | 50 | // compute rms value 51 | 52 | double r = 0; 53 | 54 | for(int i = 0; i < bincount; i++) 55 | { 56 | double b = bins[i]; 57 | 58 | r += b*b; 59 | } 60 | 61 | r = sqrt(r / n); 62 | 63 | // compute fill factor 64 | 65 | double f = (k*k - 1) / (n*r*r - k); 66 | 67 | // rescale to (0,1) with 0 = good, 1 = bad 68 | 69 | return 1 - (f / n); 70 | } 71 | 72 | 73 | //---------------------------------------------------------------------------- 74 | 75 | void plot ( double n ) 76 | { 77 | double n2 = n * 1; 78 | 79 | if(n2 < 0) n2 = 0; 80 | 81 | n2 *= 100; 82 | 83 | if(n2 > 64) n2 = 64; 84 | 85 | int n3 = (int)n2; 86 | 87 | if(n3 == 0) 88 | printf("."); 89 | else 90 | { 91 | char x = '0' + char(n3); 92 | 93 | if(x > '9') x = 'X'; 94 | 95 | printf("%c",x); 96 | } 97 | } 98 | 99 | //----------------------------------------------------------------------------- 100 | -------------------------------------------------------------------------------- /src/Stats.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include "Types.h" 4 | 5 | #include 6 | #include 7 | #include 8 | #include // for std::sort 9 | #include // for memset 10 | #include // for printf 11 | 12 | double calcScore ( const int * bins, const int bincount, const int ballcount ); 13 | 14 | void plot ( double n ); 15 | 16 | inline double ExpectedCollisions ( double balls, double bins ) 17 | { 18 | return balls - bins + bins * pow(1 - 1/bins,balls); 19 | } 20 | 21 | double chooseK ( int b, int k ); 22 | double chooseUpToK ( int n, int k ); 23 | 24 | //----------------------------------------------------------------------------- 25 | 26 | inline uint32_t f3mix ( uint32_t k ) 27 | { 28 | k ^= k >> 16; 29 | k *= 0x85ebca6b; 30 | k ^= k >> 13; 31 | k *= 0xc2b2ae35; 32 | k ^= k >> 16; 33 | 34 | return k; 35 | } 36 | 37 | //----------------------------------------------------------------------------- 38 | // Sort the hash list, count the total number of collisions and return 39 | // the first N collisions for further processing 40 | 41 | template< typename hashtype > 42 | int FindCollisions ( std::vector & hashes, 43 | HashSet & collisions, 44 | int maxCollisions ) 45 | { 46 | int collcount = 0; 47 | 48 | std::sort(hashes.begin(),hashes.end()); 49 | 50 | for(size_t i = 1; i < hashes.size(); i++) 51 | { 52 | if(hashes[i] == hashes[i-1]) 53 | { 54 | collcount++; 55 | 56 | if((int)collisions.size() < maxCollisions) 57 | { 58 | collisions.insert(hashes[i]); 59 | } 60 | } 61 | } 62 | 63 | return collcount; 64 | } 65 | 66 | //----------------------------------------------------------------------------- 67 | 68 | template < class keytype, typename hashtype > 69 | int PrintCollisions ( hashfunc hash, std::vector & keys ) 70 | { 71 | int collcount = 0; 72 | 73 | typedef std::map htab; 74 | htab tab; 75 | 76 | for(size_t i = 1; i < keys.size(); i++) 77 | { 78 | keytype & k1 = keys[i]; 79 | 80 | hashtype h = hash(&k1,sizeof(keytype),0); 81 | 82 | typename htab::iterator it = tab.find(h); 83 | 84 | if(it != tab.end()) 85 | { 86 | keytype & k2 = (*it).second; 87 | 88 | printf("A: "); 89 | printbits(&k1,sizeof(keytype)); 90 | printf("B: "); 91 | printbits(&k2,sizeof(keytype)); 92 | } 93 | else 94 | { 95 | tab.insert( std::make_pair(h,k1) ); 96 | } 97 | } 98 | 99 | return collcount; 100 | } 101 | 102 | //---------------------------------------------------------------------------- 103 | // Measure the distribution "score" for each possible N-bit span up to 20 bits 104 | 105 | template< typename hashtype > 106 | double TestDistribution ( std::vector & hashes, bool drawDiagram ) 107 | { 108 | printf("Testing distribution - "); 109 | 110 | if(drawDiagram) printf("\n"); 111 | 112 | const int hashbits = sizeof(hashtype) * 8; 113 | 114 | int maxwidth = 20; 115 | 116 | // We need at least 5 keys per bin to reliably test distribution biases 117 | // down to 1%, so don't bother to test sparser distributions than that 118 | 119 | while(double(hashes.size()) / double(1 << maxwidth) < 5.0) 120 | { 121 | maxwidth--; 122 | } 123 | 124 | std::vector bins; 125 | bins.resize(1 << maxwidth); 126 | 127 | double worst = 0; 128 | int worstStart = -1; 129 | int worstWidth = -1; 130 | 131 | for(int start = 0; start < hashbits; start++) 132 | { 133 | int width = maxwidth; 134 | int bincount = (1 << width); 135 | 136 | memset(&bins[0],0,sizeof(int)*bincount); 137 | 138 | for(size_t j = 0; j < hashes.size(); j++) 139 | { 140 | hashtype & hash = hashes[j]; 141 | 142 | uint32_t index = window(&hash,sizeof(hash),start,width); 143 | 144 | bins[index]++; 145 | } 146 | 147 | // Test the distribution, then fold the bins in half, 148 | // repeat until we're down to 256 bins 149 | 150 | if(drawDiagram) printf("["); 151 | 152 | while(bincount >= 256) 153 | { 154 | double n = calcScore(&bins[0],bincount,(int)hashes.size()); 155 | 156 | if(drawDiagram) plot(n); 157 | 158 | if(n > worst) 159 | { 160 | worst = n; 161 | worstStart = start; 162 | worstWidth = width; 163 | } 164 | 165 | width--; 166 | bincount /= 2; 167 | 168 | if(width < 8) break; 169 | 170 | for(int i = 0; i < bincount; i++) 171 | { 172 | bins[i] += bins[i+bincount]; 173 | } 174 | } 175 | 176 | if(drawDiagram) printf("]\n"); 177 | } 178 | 179 | double pct = worst * 100.0; 180 | 181 | printf("Worst bias is the %3d-bit window at bit %3d - %5.3f%%",worstWidth,worstStart,pct); 182 | if(pct >= 1.0) printf(" !!!!! "); 183 | printf("\n"); 184 | 185 | return worst; 186 | } 187 | 188 | //---------------------------------------------------------------------------- 189 | 190 | template < typename hashtype > 191 | bool TestHashList ( std::vector & hashes, std::vector & collisions, bool testDist, bool drawDiagram ) 192 | { 193 | bool result = true; 194 | 195 | { 196 | size_t count = hashes.size(); 197 | 198 | double expected = (double(count) * double(count-1)) / pow(2.0,double(sizeof(hashtype) * 8 + 1)); 199 | 200 | printf("Testing collisions - Expected %8.2f, ",expected); 201 | 202 | double collcount = 0; 203 | 204 | HashSet collisions; 205 | 206 | collcount = FindCollisions(hashes,collisions,1000); 207 | 208 | printf("actual %8.2f (%5.2fx)",collcount, collcount / expected); 209 | 210 | if(sizeof(hashtype) == sizeof(uint32_t)) 211 | { 212 | // 2x expected collisions = fail 213 | 214 | // #TODO - collision failure cutoff needs to be expressed as a standard deviation instead 215 | // of a scale factor, otherwise we fail erroneously if there are a small expected number 216 | // of collisions 217 | 218 | if(double(collcount) / double(expected) > 2.0) 219 | { 220 | printf(" !!!!! "); 221 | result = false; 222 | } 223 | } 224 | else 225 | { 226 | // For all hashes larger than 32 bits, _any_ collisions are a failure. 227 | 228 | if(collcount > 0) 229 | { 230 | printf(" !!!!! "); 231 | result = false; 232 | } 233 | } 234 | 235 | printf("\n"); 236 | } 237 | 238 | //---------- 239 | 240 | if(testDist) 241 | { 242 | TestDistribution(hashes,drawDiagram); 243 | } 244 | 245 | return result; 246 | } 247 | 248 | //---------- 249 | 250 | template < typename hashtype > 251 | bool TestHashList ( std::vector & hashes, bool /*testColl*/, bool testDist, bool drawDiagram ) 252 | { 253 | std::vector collisions; 254 | 255 | return TestHashList(hashes,collisions,testDist,drawDiagram); 256 | } 257 | 258 | //----------------------------------------------------------------------------- 259 | 260 | template < class keytype, typename hashtype > 261 | bool TestKeyList ( hashfunc hash, std::vector & keys, bool testColl, bool testDist, bool drawDiagram ) 262 | { 263 | int keycount = (int)keys.size(); 264 | 265 | std::vector hashes; 266 | 267 | hashes.resize(keycount); 268 | 269 | printf("Hashing"); 270 | 271 | for(int i = 0; i < keycount; i++) 272 | { 273 | if(i % (keycount / 10) == 0) printf("."); 274 | 275 | keytype & k = keys[i]; 276 | 277 | hash(&k,sizeof(k),0,&hashes[i]); 278 | } 279 | 280 | printf("\n"); 281 | 282 | bool result = TestHashList(hashes,testColl,testDist,drawDiagram); 283 | 284 | printf("\n"); 285 | 286 | return result; 287 | } 288 | 289 | //----------------------------------------------------------------------------- 290 | // Bytepair test - generate 16-bit indices from all possible non-overlapping 291 | // 8-bit sections of the hash value, check distribution on all of them. 292 | 293 | // This is a very good test for catching weak intercorrelations between bits - 294 | // much harder to pass than the normal distribution test. However, it doesn't 295 | // really model the normal usage of hash functions in hash table lookup, so 296 | // I'm not sure it's that useful (and hash functions that fail this test but 297 | // pass the normal distribution test still work well in practice) 298 | 299 | template < typename hashtype > 300 | double TestDistributionBytepairs ( std::vector & hashes, bool drawDiagram ) 301 | { 302 | const int nbytes = sizeof(hashtype); 303 | const int hashbits = nbytes * 8; 304 | 305 | const int nbins = 65536; 306 | 307 | std::vector bins(nbins,0); 308 | 309 | double worst = 0; 310 | 311 | for(int a = 0; a < hashbits; a++) 312 | { 313 | if(drawDiagram) if((a % 8 == 0) && (a > 0)) printf("\n"); 314 | 315 | if(drawDiagram) printf("["); 316 | 317 | for(int b = 0; b < hashbits; b++) 318 | { 319 | if(drawDiagram) if((b % 8 == 0) && (b > 0)) printf(" "); 320 | 321 | bins.clear(); 322 | bins.resize(nbins,0); 323 | 324 | for(size_t i = 0; i < hashes.size(); i++) 325 | { 326 | hashtype & hash = hashes[i]; 327 | 328 | uint32_t pa = window(&hash,sizeof(hash),a,8); 329 | uint32_t pb = window(&hash,sizeof(hash),b,8); 330 | 331 | bins[pa | (pb << 8)]++; 332 | } 333 | 334 | double s = calcScore(bins,bins.size(),hashes.size()); 335 | 336 | if(drawDiagram) plot(s); 337 | 338 | if(s > worst) 339 | { 340 | worst = s; 341 | } 342 | } 343 | 344 | if(drawDiagram) printf("]\n"); 345 | } 346 | 347 | return worst; 348 | } 349 | 350 | //----------------------------------------------------------------------------- 351 | // Simplified test - only check 64k distributions, and only on byte boundaries 352 | 353 | template < typename hashtype > 354 | void TestDistributionFast ( std::vector & hashes, double & dworst, double & davg ) 355 | { 356 | const int hashbits = sizeof(hashtype) * 8; 357 | const int nbins = 65536; 358 | 359 | std::vector bins(nbins,0); 360 | 361 | dworst = -1.0e90; 362 | davg = 0; 363 | 364 | for(int start = 0; start < hashbits; start += 8) 365 | { 366 | bins.clear(); 367 | bins.resize(nbins,0); 368 | 369 | for(size_t j = 0; j < hashes.size(); j++) 370 | { 371 | hashtype & hash = hashes[j]; 372 | 373 | uint32_t index = window(&hash,sizeof(hash),start,16); 374 | 375 | bins[index]++; 376 | } 377 | 378 | double n = calcScore(&bins.front(),(int)bins.size(),(int)hashes.size()); 379 | 380 | davg += n; 381 | 382 | if(n > dworst) dworst = n; 383 | } 384 | 385 | davg /= double(hashbits/8); 386 | } 387 | 388 | //----------------------------------------------------------------------------- 389 | -------------------------------------------------------------------------------- /src/SuperFastHash.cpp: -------------------------------------------------------------------------------- 1 | #include "Platform.h" 2 | #include // for NULL 3 | 4 | /* By Paul Hsieh (C) 2004, 2005. Covered under the Paul Hsieh derivative 5 | license. See: 6 | http://www.azillionmonkeys.com/qed/weblicense.html for license details. 7 | 8 | http://www.azillionmonkeys.com/qed/hash.html */ 9 | 10 | /* 11 | #undef get16bits 12 | #if (defined(__GNUC__) && defined(__i386__)) || defined(__WATCOMC__) \ 13 | || defined(_MSC_VER) || defined (__BORLANDC__) || defined (__TURBOC__) 14 | #define get16bits(d) (*((const uint16_t *) (d))) 15 | #endif 16 | 17 | #if !defined (get16bits) 18 | #define get16bits(d) ((((uint32_t)(((const uint8_t *)(d))[1])) << 8)\ 19 | +(uint32_t)(((const uint8_t *)(d))[0]) ) 20 | #endif 21 | */ 22 | 23 | FORCE_INLINE uint16_t get16bits ( const void * p ) 24 | { 25 | return *(const uint16_t*)p; 26 | } 27 | 28 | uint32_t SuperFastHash (const signed char * data, int len) { 29 | uint32_t hash = 0, tmp; 30 | int rem; 31 | 32 | if (len <= 0 || data == NULL) return 0; 33 | 34 | rem = len & 3; 35 | len >>= 2; 36 | 37 | /* Main loop */ 38 | for (;len > 0; len--) { 39 | hash += get16bits (data); 40 | tmp = (get16bits (data+2) << 11) ^ hash; 41 | hash = (hash << 16) ^ tmp; 42 | data += 2*sizeof (uint16_t); 43 | hash += hash >> 11; 44 | } 45 | 46 | /* Handle end cases */ 47 | switch (rem) { 48 | case 3: hash += get16bits (data); 49 | hash ^= hash << 16; 50 | hash ^= data[sizeof (uint16_t)] << 18; 51 | hash += hash >> 11; 52 | break; 53 | case 2: hash += get16bits (data); 54 | hash ^= hash << 11; 55 | hash += hash >> 17; 56 | break; 57 | case 1: hash += *data; 58 | hash ^= hash << 10; 59 | hash += hash >> 1; 60 | } 61 | 62 | /* Force "avalanching" of final 127 bits */ 63 | hash ^= hash << 3; 64 | hash += hash >> 5; 65 | hash ^= hash << 4; 66 | hash += hash >> 17; 67 | hash ^= hash << 25; 68 | hash += hash >> 6; 69 | 70 | return hash; 71 | } 72 | 73 | void SuperFastHash ( const void * key, int len, uint32_t /*seed*/, void * out ) 74 | { 75 | *(uint32_t*)out = SuperFastHash((const signed char*)key,len); 76 | } 77 | -------------------------------------------------------------------------------- /src/Types.cpp: -------------------------------------------------------------------------------- 1 | #include "Types.h" 2 | 3 | #include "Random.h" 4 | 5 | #include 6 | 7 | uint32_t MurmurOAAT ( const void * blob, int len, uint32_t seed ); 8 | 9 | //----------------------------------------------------------------------------- 10 | 11 | #if defined(_MSC_VER) 12 | #pragma optimize( "", off ) 13 | #endif 14 | 15 | void blackhole ( uint32_t ) 16 | { 17 | } 18 | 19 | uint32_t whitehole ( void ) 20 | { 21 | return 0; 22 | } 23 | 24 | #if defined(_MSC_VER) 25 | #pragma optimize( "", on ) 26 | #endif 27 | 28 | uint32_t g_verify = 1; 29 | 30 | void MixVCode ( const void * blob, int len ) 31 | { 32 | g_verify = MurmurOAAT(blob,len,g_verify); 33 | } 34 | 35 | //----------------------------------------------------------------------------- 36 | 37 | bool isprime ( uint32_t x ) 38 | { 39 | uint32_t p[] = 40 | { 41 | 2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,71,73,79,83,89,97,101, 42 | 103,107,109,113,127,131,137,139,149,151,157,163,167,173,179,181,191,193,197, 43 | 199,211,223,227,229,233,239,241,251 44 | }; 45 | 46 | for(size_t i=0; i < sizeof(p)/sizeof(uint32_t); i++) 47 | { 48 | if((x % p[i]) == 0) 49 | { 50 | return false; 51 | } 52 | } 53 | 54 | for(int i = 257; i < 65536; i += 2) 55 | { 56 | if((x % i) == 0) 57 | { 58 | return false; 59 | } 60 | } 61 | 62 | return true; 63 | } 64 | 65 | void GenerateMixingConstants ( void ) 66 | { 67 | Rand r(8350147); 68 | 69 | int count = 0; 70 | 71 | int trials = 0; 72 | int bitfail = 0; 73 | int popfail = 0; 74 | int matchfail = 0; 75 | int primefail = 0; 76 | 77 | //for(uint32_t x = 1; x; x++) 78 | while(count < 100) 79 | { 80 | //if(x % 100000000 == 0) printf("."); 81 | 82 | trials++; 83 | uint32_t b = r.rand_u32(); 84 | //uint32_t b = x; 85 | 86 | //---------- 87 | // must have between 14 and 18 set bits 88 | 89 | if(popcount(b) < 16) { b = 0; popfail++; } 90 | if(popcount(b) > 16) { b = 0; popfail++; } 91 | 92 | if(b == 0) continue; 93 | 94 | //---------- 95 | // must have 3-5 bits set per 8-bit window 96 | 97 | for(int i = 0; i < 32; i++) 98 | { 99 | uint32_t c = ROTL32(b,i) & 0xFF; 100 | 101 | if(popcount(c) < 3) { b = 0; bitfail++; break; } 102 | if(popcount(c) > 5) { b = 0; bitfail++; break; } 103 | } 104 | 105 | if(b == 0) continue; 106 | 107 | //---------- 108 | // all 8-bit windows must be different 109 | 110 | uint8_t match[256]; 111 | 112 | memset(match,0,256); 113 | 114 | for(int i = 0; i < 32; i++) 115 | { 116 | uint32_t c = ROTL32(b,i) & 0xFF; 117 | 118 | if(match[c]) { b = 0; matchfail++; break; } 119 | 120 | match[c] = 1; 121 | } 122 | 123 | if(b == 0) continue; 124 | 125 | //---------- 126 | // must be prime 127 | 128 | if(!isprime(b)) 129 | { 130 | b = 0; 131 | primefail++; 132 | } 133 | 134 | if(b == 0) continue; 135 | 136 | //---------- 137 | 138 | if(b) 139 | { 140 | printf("0x%08x : 0x%08x\n",b,~b); 141 | count++; 142 | } 143 | } 144 | 145 | printf("%d %d %d %d %d %d\n",trials,popfail,bitfail,matchfail,primefail,count); 146 | } 147 | 148 | //----------------------------------------------------------------------------- 149 | -------------------------------------------------------------------------------- /src/Types.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #include "Platform.h" 4 | #include "Bitvec.h" 5 | 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | //----------------------------------------------------------------------------- 12 | // If the optimizer detects that a value in a speed test is constant or unused, 13 | // the optimizer may remove references to it or otherwise create code that 14 | // would not occur in a real-world application. To prevent the optimizer from 15 | // doing this we declare two trivial functions that either sink or source data, 16 | // and bar the compiler from optimizing them. 17 | 18 | void blackhole ( uint32_t x ); 19 | uint32_t whitehole ( void ); 20 | 21 | //----------------------------------------------------------------------------- 22 | // We want to verify that every test produces the same result on every platform 23 | // To do this, we hash the results of every test to produce an overall 24 | // verification value for the whole test suite. If two runs produce the same 25 | // verification value, then every test in both run produced the same results 26 | 27 | extern uint32_t g_verify; 28 | 29 | // Mix the given blob of data into the verification code 30 | 31 | void MixVCode ( const void * blob, int len ); 32 | 33 | 34 | //----------------------------------------------------------------------------- 35 | 36 | typedef void (*pfHash) ( const void * blob, const int len, const uint32_t seed, void * out ); 37 | 38 | struct ByteVec : public std::vector 39 | { 40 | ByteVec ( const void * key, int len ) 41 | { 42 | resize(len); 43 | memcpy(&front(),key,len); 44 | } 45 | }; 46 | 47 | template< typename hashtype, typename keytype > 48 | struct CollisionMap : public std::map< hashtype, std::vector > 49 | { 50 | }; 51 | 52 | template< typename hashtype > 53 | struct HashSet : public std::set 54 | { 55 | }; 56 | 57 | //----------------------------------------------------------------------------- 58 | 59 | template < class T > 60 | class hashfunc 61 | { 62 | public: 63 | 64 | hashfunc ( pfHash h ) : m_hash(h) 65 | { 66 | } 67 | 68 | inline void operator () ( const void * key, const int len, const uint32_t seed, uint32_t * out ) 69 | { 70 | m_hash(key,len,seed,out); 71 | } 72 | 73 | inline operator pfHash ( void ) const 74 | { 75 | return m_hash; 76 | } 77 | 78 | inline T operator () ( const void * key, const int len, const uint32_t seed ) 79 | { 80 | T result; 81 | 82 | m_hash(key,len,seed,(uint32_t*)&result); 83 | 84 | return result; 85 | } 86 | 87 | pfHash m_hash; 88 | }; 89 | 90 | //----------------------------------------------------------------------------- 91 | // Key-processing callback objects. Simplifies keyset testing a bit. 92 | 93 | struct KeyCallback 94 | { 95 | KeyCallback() : m_count(0) 96 | { 97 | } 98 | 99 | virtual ~KeyCallback() 100 | { 101 | } 102 | 103 | virtual void operator() ( const void * key, int len ) 104 | { 105 | m_count++; 106 | } 107 | 108 | virtual void reserve ( int keycount ) 109 | { 110 | }; 111 | 112 | int m_count; 113 | }; 114 | 115 | //---------- 116 | 117 | template 118 | struct HashCallback : public KeyCallback 119 | { 120 | typedef std::vector hashvec; 121 | 122 | HashCallback ( pfHash hash, hashvec & hashes ) : m_hashes(hashes), m_pfHash(hash) 123 | { 124 | m_hashes.clear(); 125 | } 126 | 127 | virtual void operator () ( const void * key, int len ) 128 | { 129 | size_t newsize = m_hashes.size() + 1; 130 | 131 | m_hashes.resize(newsize); 132 | 133 | m_pfHash(key,len,0,&m_hashes.back()); 134 | } 135 | 136 | virtual void reserve ( int keycount ) 137 | { 138 | m_hashes.reserve(keycount); 139 | } 140 | 141 | hashvec & m_hashes; 142 | pfHash m_pfHash; 143 | 144 | //---------- 145 | 146 | private: 147 | 148 | HashCallback & operator = ( const HashCallback & ); 149 | }; 150 | 151 | //---------- 152 | 153 | template 154 | struct CollisionCallback : public KeyCallback 155 | { 156 | typedef HashSet hashset; 157 | typedef CollisionMap collmap; 158 | 159 | CollisionCallback ( pfHash hash, hashset & collisions, collmap & cmap ) 160 | : m_pfHash(hash), 161 | m_collisions(collisions), 162 | m_collmap(cmap) 163 | { 164 | } 165 | 166 | virtual void operator () ( const void * key, int len ) 167 | { 168 | hashtype h; 169 | 170 | m_pfHash(key,len,0,&h); 171 | 172 | if(m_collisions.count(h)) 173 | { 174 | m_collmap[h].push_back( ByteVec(key,len) ); 175 | } 176 | } 177 | 178 | //---------- 179 | 180 | pfHash m_pfHash; 181 | hashset & m_collisions; 182 | collmap & m_collmap; 183 | 184 | private: 185 | 186 | CollisionCallback & operator = ( const CollisionCallback & c ); 187 | }; 188 | 189 | //----------------------------------------------------------------------------- 190 | 191 | template < int _bits > 192 | class Blob 193 | { 194 | public: 195 | 196 | Blob() 197 | { 198 | for(size_t i = 0; i < sizeof(bytes); i++) 199 | { 200 | bytes[i] = 0; 201 | } 202 | } 203 | 204 | Blob ( int x ) 205 | { 206 | for(size_t i = 0; i < sizeof(bytes); i++) 207 | { 208 | bytes[i] = 0; 209 | } 210 | 211 | *(int*)bytes = x; 212 | } 213 | 214 | Blob ( const Blob & k ) 215 | { 216 | for(size_t i = 0; i < sizeof(bytes); i++) 217 | { 218 | bytes[i] = k.bytes[i]; 219 | } 220 | } 221 | 222 | Blob & operator = ( const Blob & k ) 223 | { 224 | for(size_t i = 0; i < sizeof(bytes); i++) 225 | { 226 | bytes[i] = k.bytes[i]; 227 | } 228 | 229 | return *this; 230 | } 231 | 232 | Blob ( uint64_t a, uint64_t b ) 233 | { 234 | uint64_t t[2] = {a,b}; 235 | set(&t,16); 236 | } 237 | 238 | void set ( const void * blob, size_t len ) 239 | { 240 | const uint8_t * k = (const uint8_t*)blob; 241 | 242 | len = len > sizeof(bytes) ? sizeof(bytes) : len; 243 | 244 | for(size_t i = 0; i < len; i++) 245 | { 246 | bytes[i] = k[i]; 247 | } 248 | 249 | for(size_t i = len; i < sizeof(bytes); i++) 250 | { 251 | bytes[i] = 0; 252 | } 253 | } 254 | 255 | uint8_t & operator [] ( int i ) 256 | { 257 | return bytes[i]; 258 | } 259 | 260 | const uint8_t & operator [] ( int i ) const 261 | { 262 | return bytes[i]; 263 | } 264 | 265 | //---------- 266 | // boolean operations 267 | 268 | bool operator < ( const Blob & k ) const 269 | { 270 | for(size_t i = 0; i < sizeof(bytes); i++) 271 | { 272 | if(bytes[i] < k.bytes[i]) return true; 273 | if(bytes[i] > k.bytes[i]) return false; 274 | } 275 | 276 | return false; 277 | } 278 | 279 | bool operator == ( const Blob & k ) const 280 | { 281 | for(size_t i = 0; i < sizeof(bytes); i++) 282 | { 283 | if(bytes[i] != k.bytes[i]) return false; 284 | } 285 | 286 | return true; 287 | } 288 | 289 | bool operator != ( const Blob & k ) const 290 | { 291 | return !(*this == k); 292 | } 293 | 294 | //---------- 295 | // bitwise operations 296 | 297 | Blob operator ^ ( const Blob & k ) const 298 | { 299 | Blob t; 300 | 301 | for(size_t i = 0; i < sizeof(bytes); i++) 302 | { 303 | t.bytes[i] = bytes[i] ^ k.bytes[i]; 304 | } 305 | 306 | return t; 307 | } 308 | 309 | Blob & operator ^= ( const Blob & k ) 310 | { 311 | for(size_t i = 0; i < sizeof(bytes); i++) 312 | { 313 | bytes[i] ^= k.bytes[i]; 314 | } 315 | 316 | return *this; 317 | } 318 | 319 | int operator & ( int x ) 320 | { 321 | return (*(int*)bytes) & x; 322 | } 323 | 324 | Blob & operator &= ( const Blob & k ) 325 | { 326 | for(size_t i = 0; i < sizeof(bytes); i++) 327 | { 328 | bytes[i] &= k.bytes[i]; 329 | } 330 | } 331 | 332 | Blob operator << ( int c ) 333 | { 334 | Blob t = *this; 335 | 336 | lshift(&t.bytes[0],sizeof(bytes),c); 337 | 338 | return t; 339 | } 340 | 341 | Blob operator >> ( int c ) 342 | { 343 | Blob t = *this; 344 | 345 | rshift(&t.bytes[0],sizeof(bytes),c); 346 | 347 | return t; 348 | } 349 | 350 | Blob & operator <<= ( int c ) 351 | { 352 | lshift(&bytes[0],sizeof(bytes),c); 353 | 354 | return *this; 355 | } 356 | 357 | Blob & operator >>= ( int c ) 358 | { 359 | rshift(&bytes[0],sizeof(bytes),c); 360 | 361 | return *this; 362 | } 363 | 364 | //---------- 365 | 366 | private: 367 | 368 | uint8_t bytes[(_bits+7)/8]; 369 | }; 370 | 371 | typedef Blob<128> uint128_t; 372 | typedef Blob<256> uint256_t; 373 | 374 | //----------------------------------------------------------------------------- 375 | -------------------------------------------------------------------------------- /src/Xxh3Test.cpp: -------------------------------------------------------------------------------- 1 | #include "xxh3.h" 2 | 3 | void XXH3128_test(const void *key, int len, uint32_t seed, void *out) { 4 | XXH128_hash_t res = XXH3_128bits_withSeed(key, len, seed); 5 | ((uint64_t*)out)[0] = res.low64; 6 | ((uint64_t*)out)[1] = res.high64; 7 | } 8 | 9 | void XXH364_test(const void *key, int len, uint32_t seed, void *out) { 10 | ((uint64_t*)out)[0] = XXH3_64bits_withSeed(key, len, seed); 11 | } 12 | -------------------------------------------------------------------------------- /src/crc.cpp: -------------------------------------------------------------------------------- 1 | #include "Platform.h" 2 | 3 | /* 4 | * This file is derived from crc32.c from the zlib-1.1.3 distribution 5 | * by Jean-loup Gailly and Mark Adler. 6 | */ 7 | 8 | /* crc32.c -- compute the CRC-32 of a data stream 9 | * Copyright (C) 1995-1998 Mark Adler 10 | * For conditions of distribution and use, see copyright notice in zlib.h 11 | */ 12 | 13 | 14 | /* ======================================================================== 15 | * Table of CRC-32's of all single-byte values (made by make_crc_table) 16 | */ 17 | static const uint32_t crc_table[256] = { 18 | 0x00000000L, 0x77073096L, 0xee0e612cL, 0x990951baL, 0x076dc419L, 19 | 0x706af48fL, 0xe963a535L, 0x9e6495a3L, 0x0edb8832L, 0x79dcb8a4L, 20 | 0xe0d5e91eL, 0x97d2d988L, 0x09b64c2bL, 0x7eb17cbdL, 0xe7b82d07L, 21 | 0x90bf1d91L, 0x1db71064L, 0x6ab020f2L, 0xf3b97148L, 0x84be41deL, 22 | 0x1adad47dL, 0x6ddde4ebL, 0xf4d4b551L, 0x83d385c7L, 0x136c9856L, 23 | 0x646ba8c0L, 0xfd62f97aL, 0x8a65c9ecL, 0x14015c4fL, 0x63066cd9L, 24 | 0xfa0f3d63L, 0x8d080df5L, 0x3b6e20c8L, 0x4c69105eL, 0xd56041e4L, 25 | 0xa2677172L, 0x3c03e4d1L, 0x4b04d447L, 0xd20d85fdL, 0xa50ab56bL, 26 | 0x35b5a8faL, 0x42b2986cL, 0xdbbbc9d6L, 0xacbcf940L, 0x32d86ce3L, 27 | 0x45df5c75L, 0xdcd60dcfL, 0xabd13d59L, 0x26d930acL, 0x51de003aL, 28 | 0xc8d75180L, 0xbfd06116L, 0x21b4f4b5L, 0x56b3c423L, 0xcfba9599L, 29 | 0xb8bda50fL, 0x2802b89eL, 0x5f058808L, 0xc60cd9b2L, 0xb10be924L, 30 | 0x2f6f7c87L, 0x58684c11L, 0xc1611dabL, 0xb6662d3dL, 0x76dc4190L, 31 | 0x01db7106L, 0x98d220bcL, 0xefd5102aL, 0x71b18589L, 0x06b6b51fL, 32 | 0x9fbfe4a5L, 0xe8b8d433L, 0x7807c9a2L, 0x0f00f934L, 0x9609a88eL, 33 | 0xe10e9818L, 0x7f6a0dbbL, 0x086d3d2dL, 0x91646c97L, 0xe6635c01L, 34 | 0x6b6b51f4L, 0x1c6c6162L, 0x856530d8L, 0xf262004eL, 0x6c0695edL, 35 | 0x1b01a57bL, 0x8208f4c1L, 0xf50fc457L, 0x65b0d9c6L, 0x12b7e950L, 36 | 0x8bbeb8eaL, 0xfcb9887cL, 0x62dd1ddfL, 0x15da2d49L, 0x8cd37cf3L, 37 | 0xfbd44c65L, 0x4db26158L, 0x3ab551ceL, 0xa3bc0074L, 0xd4bb30e2L, 38 | 0x4adfa541L, 0x3dd895d7L, 0xa4d1c46dL, 0xd3d6f4fbL, 0x4369e96aL, 39 | 0x346ed9fcL, 0xad678846L, 0xda60b8d0L, 0x44042d73L, 0x33031de5L, 40 | 0xaa0a4c5fL, 0xdd0d7cc9L, 0x5005713cL, 0x270241aaL, 0xbe0b1010L, 41 | 0xc90c2086L, 0x5768b525L, 0x206f85b3L, 0xb966d409L, 0xce61e49fL, 42 | 0x5edef90eL, 0x29d9c998L, 0xb0d09822L, 0xc7d7a8b4L, 0x59b33d17L, 43 | 0x2eb40d81L, 0xb7bd5c3bL, 0xc0ba6cadL, 0xedb88320L, 0x9abfb3b6L, 44 | 0x03b6e20cL, 0x74b1d29aL, 0xead54739L, 0x9dd277afL, 0x04db2615L, 45 | 0x73dc1683L, 0xe3630b12L, 0x94643b84L, 0x0d6d6a3eL, 0x7a6a5aa8L, 46 | 0xe40ecf0bL, 0x9309ff9dL, 0x0a00ae27L, 0x7d079eb1L, 0xf00f9344L, 47 | 0x8708a3d2L, 0x1e01f268L, 0x6906c2feL, 0xf762575dL, 0x806567cbL, 48 | 0x196c3671L, 0x6e6b06e7L, 0xfed41b76L, 0x89d32be0L, 0x10da7a5aL, 49 | 0x67dd4accL, 0xf9b9df6fL, 0x8ebeeff9L, 0x17b7be43L, 0x60b08ed5L, 50 | 0xd6d6a3e8L, 0xa1d1937eL, 0x38d8c2c4L, 0x4fdff252L, 0xd1bb67f1L, 51 | 0xa6bc5767L, 0x3fb506ddL, 0x48b2364bL, 0xd80d2bdaL, 0xaf0a1b4cL, 52 | 0x36034af6L, 0x41047a60L, 0xdf60efc3L, 0xa867df55L, 0x316e8eefL, 53 | 0x4669be79L, 0xcb61b38cL, 0xbc66831aL, 0x256fd2a0L, 0x5268e236L, 54 | 0xcc0c7795L, 0xbb0b4703L, 0x220216b9L, 0x5505262fL, 0xc5ba3bbeL, 55 | 0xb2bd0b28L, 0x2bb45a92L, 0x5cb36a04L, 0xc2d7ffa7L, 0xb5d0cf31L, 56 | 0x2cd99e8bL, 0x5bdeae1dL, 0x9b64c2b0L, 0xec63f226L, 0x756aa39cL, 57 | 0x026d930aL, 0x9c0906a9L, 0xeb0e363fL, 0x72076785L, 0x05005713L, 58 | 0x95bf4a82L, 0xe2b87a14L, 0x7bb12baeL, 0x0cb61b38L, 0x92d28e9bL, 59 | 0xe5d5be0dL, 0x7cdcefb7L, 0x0bdbdf21L, 0x86d3d2d4L, 0xf1d4e242L, 60 | 0x68ddb3f8L, 0x1fda836eL, 0x81be16cdL, 0xf6b9265bL, 0x6fb077e1L, 61 | 0x18b74777L, 0x88085ae6L, 0xff0f6a70L, 0x66063bcaL, 0x11010b5cL, 62 | 0x8f659effL, 0xf862ae69L, 0x616bffd3L, 0x166ccf45L, 0xa00ae278L, 63 | 0xd70dd2eeL, 0x4e048354L, 0x3903b3c2L, 0xa7672661L, 0xd06016f7L, 64 | 0x4969474dL, 0x3e6e77dbL, 0xaed16a4aL, 0xd9d65adcL, 0x40df0b66L, 65 | 0x37d83bf0L, 0xa9bcae53L, 0xdebb9ec5L, 0x47b2cf7fL, 0x30b5ffe9L, 66 | 0xbdbdf21cL, 0xcabac28aL, 0x53b39330L, 0x24b4a3a6L, 0xbad03605L, 67 | 0xcdd70693L, 0x54de5729L, 0x23d967bfL, 0xb3667a2eL, 0xc4614ab8L, 68 | 0x5d681b02L, 0x2a6f2b94L, 0xb40bbe37L, 0xc30c8ea1L, 0x5a05df1bL, 69 | 0x2d02ef8dL 70 | }; 71 | 72 | /* ========================================================================= */ 73 | 74 | #define DO1(buf) crc = crc_table[((int)crc ^ (*buf++)) & 0xff] ^ (crc >> 8); 75 | #define DO2(buf) DO1(buf); DO1(buf); 76 | #define DO4(buf) DO2(buf); DO2(buf); 77 | #define DO8(buf) DO4(buf); DO4(buf); 78 | 79 | /* ========================================================================= */ 80 | 81 | void crc32 ( const void * key, int len, uint32_t seed, void * out ) 82 | { 83 | uint8_t * buf = (uint8_t*)key; 84 | uint32_t crc = seed ^ 0xffffffffL; 85 | 86 | while (len >= 8) 87 | { 88 | DO8(buf); 89 | len -= 8; 90 | } 91 | 92 | while(len--) 93 | { 94 | DO1(buf); 95 | } 96 | 97 | crc ^= 0xffffffffL; 98 | 99 | *(uint32_t*)out = crc; 100 | } 101 | -------------------------------------------------------------------------------- /src/lookup3.cpp: -------------------------------------------------------------------------------- 1 | // lookup3 by Bob Jekins, code is public domain. 2 | 3 | #include "Platform.h" 4 | 5 | #define rot(x,k) (((x)<<(k)) | ((x)>>(32-(k)))) 6 | 7 | #define mix(a,b,c) \ 8 | { \ 9 | a -= c; a ^= rot(c, 4); c += b; \ 10 | b -= a; b ^= rot(a, 6); a += c; \ 11 | c -= b; c ^= rot(b, 8); b += a; \ 12 | a -= c; a ^= rot(c,16); c += b; \ 13 | b -= a; b ^= rot(a,19); a += c; \ 14 | c -= b; c ^= rot(b, 4); b += a; \ 15 | } 16 | 17 | #define final(a,b,c) \ 18 | { \ 19 | c ^= b; c -= rot(b,14); \ 20 | a ^= c; a -= rot(c,11); \ 21 | b ^= a; b -= rot(a,25); \ 22 | c ^= b; c -= rot(b,16); \ 23 | a ^= c; a -= rot(c,4); \ 24 | b ^= a; b -= rot(a,14); \ 25 | c ^= b; c -= rot(b,24); \ 26 | } 27 | 28 | uint32_t lookup3 ( const void * key, int length, uint32_t initval ) 29 | { 30 | uint32_t a,b,c; /* internal state */ 31 | 32 | a = b = c = 0xdeadbeef + ((uint32_t)length) + initval; 33 | 34 | const uint32_t *k = (const uint32_t *)key; /* read 32-bit chunks */ 35 | 36 | /*------ all but last block: aligned reads and affect 32 bits of (a,b,c) */ 37 | while (length > 12) 38 | { 39 | a += k[0]; 40 | b += k[1]; 41 | c += k[2]; 42 | mix(a,b,c); 43 | length -= 12; 44 | k += 3; 45 | } 46 | 47 | switch(length) 48 | { 49 | case 12: c+=k[2]; b+=k[1]; a+=k[0]; break; 50 | case 11: c+=k[2]&0xffffff; b+=k[1]; a+=k[0]; break; 51 | case 10: c+=k[2]&0xffff; b+=k[1]; a+=k[0]; break; 52 | case 9 : c+=k[2]&0xff; b+=k[1]; a+=k[0]; break; 53 | case 8 : b+=k[1]; a+=k[0]; break; 54 | case 7 : b+=k[1]&0xffffff; a+=k[0]; break; 55 | case 6 : b+=k[1]&0xffff; a+=k[0]; break; 56 | case 5 : b+=k[1]&0xff; a+=k[0]; break; 57 | case 4 : a+=k[0]; break; 58 | case 3 : a+=k[0]&0xffffff; break; 59 | case 2 : a+=k[0]&0xffff; break; 60 | case 1 : a+=k[0]&0xff; break; 61 | case 0 : { return c; } /* zero length strings require no mixing */ 62 | } 63 | 64 | final(a,b,c); 65 | 66 | return c; 67 | } 68 | 69 | void lookup3_test ( const void * key, int len, uint32_t seed, void * out ) 70 | { 71 | *(uint32_t*)out = lookup3(key,len,seed); 72 | } 73 | -------------------------------------------------------------------------------- /src/meow_hash.h: -------------------------------------------------------------------------------- 1 | /* ======================================================================== 2 | 3 | Meow - A Fast Non-cryptographic Hash 4 | (C) Copyright 2018 by Molly Rocket, Inc. (https://mollyrocket.com) 5 | 6 | See https://mollyrocket.com/meowhash for details. 7 | */ 8 | 9 | /* based on meow_hash v0.4 (https://github.com/cmuratori/meow_hash) 10 | * Same algo, except that it uses 128 bit seed and mixes the seed into the 11 | * initial state and uses 2 AESDEC rounds instead of 1. 12 | */ 13 | static inline __m128i 14 | Meow128_AESDEC_Memx2( __m128i R, const uint8_t *S ) 15 | { 16 | R = _mm_aesdec_si128((R), _mm_loadu_si128((__m128i *)(S))); 17 | R = _mm_aesdec_si128((R), _mm_loadu_si128((__m128i *)(S))); 18 | return R; 19 | } 20 | 21 | static inline __m128i 22 | Meow128_AESDECx2( __m128i R, __m128i S ) 23 | { 24 | R = _mm_aesdec_si128((R), (S)); 25 | R = _mm_aesdec_si128((R), (S)); 26 | return R; 27 | } 28 | 29 | static void 30 | hash_meow128( const void *p, size_t sz, uint64_t *x1, uint64_t *x2 ) 31 | { 32 | static const uint8_t MeowShiftAdjust[] = {0,1,2,3, 4,5,6,7, 33 | 8,9,10,11, 12,13,14,15, 34 | 128,128,128,128, 128,128,128,128, 35 | 128,128,128,128, 128,128,128,0}; 36 | static const uint8_t MeowMaskLen[] = {255,255,255,255, 255,255,255,255, 37 | 255,255,255,255, 255,255,255,255, 38 | 0,0,0,0, 0,0,0,0, 39 | 0,0,0,0, 0,0,0,0}; 40 | static const uint8_t MeowS0Init[] = { 0, 1, 2, 3, 4, 5, 6, 7, 41 | 8, 9,10,11, 12,13,14,15}; 42 | static const uint8_t MeowS1Init[] = {16,17,18,19, 20,21,22,23, 43 | 24,25,26,27, 28,29,30,31}; 44 | static const uint8_t MeowS2Init[] = {32,33,34,35, 36,37,38,39, 45 | 40,41,42,43, 44,45,46,47}; 46 | static const uint8_t MeowS3Init[] = {48,49,50,51, 52,53,54,55, 47 | 56,57,58,59, 60,61,62,63}; 48 | __m128i S0 = *(__m128i *) MeowS0Init; 49 | __m128i S1 = *(__m128i *) MeowS1Init; 50 | __m128i S2 = *(__m128i *) MeowS2Init; 51 | __m128i S3 = *(__m128i *) MeowS3Init; 52 | 53 | const uint8_t * Source = (const uint8_t *) p; 54 | size_t Len = sz; 55 | const uint32_t Len8 = (uint32_t) Len & 15; 56 | const uint32_t Len128 = (uint32_t) Len & 48; 57 | 58 | __m128i Mixer = _mm_set_epi64x(*x2 + sz + 1, *x1 - sz); 59 | S0 ^= Mixer; 60 | S1 ^= Mixer; 61 | S2 ^= Mixer; 62 | S3 ^= Mixer; 63 | 64 | while ( Len >= 64 ) { 65 | S0 = Meow128_AESDEC_Memx2(S0, Source); 66 | S1 = Meow128_AESDEC_Memx2(S1, Source + 16); 67 | S2 = Meow128_AESDEC_Memx2(S2, Source + 32); 68 | S3 = Meow128_AESDEC_Memx2(S3, Source + 48); 69 | 70 | Len -= 64; 71 | Source += 64; 72 | } 73 | 74 | if ( Len8 ) { 75 | __m128i Partial; 76 | const uint8_t * Overhang = Source + Len128; 77 | uint32_t Align = ((size_t)(intptr_t) Overhang) & 15; 78 | 79 | if ( Align != 0 ) { /* presumes page size is >= 4k */ 80 | uint32_t End = ((size_t)(intptr_t) Overhang) & (size_t) (4096 - 1); 81 | 82 | if ( End <= (4096 - 16) || (End + Len8) > 4096 ) 83 | Align = 0; 84 | 85 | Partial = _mm_shuffle_epi8( 86 | _mm_loadu_si128( 87 | (__m128i *)( Overhang - Align ) ), 88 | _mm_loadu_si128( (__m128i *) &MeowShiftAdjust[ Align ] ) ); 89 | 90 | Partial = 91 | _mm_and_si128( Partial, 92 | _mm_loadu_si128( (__m128i *) &MeowMaskLen[ 16 - Len8 ] ) ); 93 | } 94 | else { 95 | Partial = 96 | _mm_and_si128( *(__m128i *) Overhang, 97 | _mm_loadu_si128( (__m128i *) &MeowMaskLen[ 16 - Len8 ] ) ); 98 | } 99 | S3 = Meow128_AESDECx2(S3, Partial); 100 | } 101 | switch ( Len128 ) { 102 | case 48: S2 = Meow128_AESDEC_Memx2(S2, Source + 32); /* FALLTHRU */ 103 | case 32: S1 = Meow128_AESDEC_Memx2(S1, Source + 16); /* FALLTHRU */ 104 | case 16: S0 = Meow128_AESDEC_Memx2(S0, Source); break; 105 | } 106 | 107 | S3 = _mm_aesdec_si128(S3, Mixer); 108 | S2 = _mm_aesdec_si128(S2, Mixer); 109 | S1 = _mm_aesdec_si128(S1, Mixer); 110 | S0 = _mm_aesdec_si128(S0, Mixer); 111 | 112 | S2 = _mm_aesdec_si128(S2, S3); 113 | S0 = _mm_aesdec_si128(S0, S1); 114 | 115 | S2 = _mm_aesdec_si128(S2, Mixer); 116 | 117 | S0 = _mm_aesdec_si128(S0, S2); 118 | S0 = _mm_aesdec_si128(S0, Mixer); 119 | 120 | *x1 = _mm_extract_epi64(S0, 0); 121 | *x2 = _mm_extract_epi64(S0, 1); 122 | } 123 | 124 | -------------------------------------------------------------------------------- /src/sha1.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | SHA-1 in C 3 | By Steve Reid 4 | 100% Public Domain 5 | 6 | ----------------- 7 | Modified 7/98 8 | By James H. Brown 9 | Still 100% Public Domain 10 | 11 | Corrected a problem which generated improper hash values on 16 bit machines 12 | Routine SHA1Update changed from 13 | void SHA1Update(SHA1_CTX* context, unsigned char* data, unsigned int 14 | len) 15 | to 16 | void SHA1Update(SHA1_CTX* context, unsigned char* data, unsigned 17 | long len) 18 | 19 | The 'len' parameter was declared an int which works fine on 32 bit machines. 20 | However, on 16 bit machines an int is too small for the shifts being done 21 | against 22 | it. This caused the hash function to generate incorrect values if len was 23 | greater than 8191 (8K - 1) due to the 'len << 3' on line 3 of SHA1Update(). 24 | 25 | Since the file IO in main() reads 16K at a time, any file 8K or larger would 26 | be guaranteed to generate the wrong hash (e.g. Test Vector #3, a million 27 | "a"s). 28 | 29 | I also changed the declaration of variables i & j in SHA1Update to 30 | unsigned long from unsigned int for the same reason. 31 | 32 | These changes should make no difference to any 32 bit implementations since 33 | an 34 | int and a long are the same size in those environments. 35 | 36 | -- 37 | I also corrected a few compiler warnings generated by Borland C. 38 | 1. Added #include for exit() prototype 39 | 2. Removed unused variable 'j' in SHA1Final 40 | 3. Changed exit(0) to return(0) at end of main. 41 | 42 | ALL changes I made can be located by searching for comments containing 'JHB' 43 | ----------------- 44 | Modified 8/98 45 | By Steve Reid 46 | Still 100% public domain 47 | 48 | 1- Removed #include and used return() instead of exit() 49 | 2- Fixed overwriting of finalcount in SHA1Final() (discovered by Chris Hall) 50 | 3- Changed email address from steve@edmweb.com to sreid@sea-to-sky.net 51 | 52 | ----------------- 53 | Modified 4/01 54 | By Saul Kravitz 55 | Still 100% PD 56 | Modified to run on Compaq Alpha hardware. 57 | 58 | ----------------- 59 | Modified 07/2002 60 | By Ralph Giles 61 | Still 100% public domain 62 | modified for use with stdint types, autoconf 63 | code cleanup, removed attribution comments 64 | switched SHA1Final() argument order for consistency 65 | use SHA1_ prefix for public api 66 | move public api to sha1.h 67 | */ 68 | 69 | /* 70 | Test Vectors (from FIPS PUB 180-1) 71 | "abc" 72 | A9993E36 4706816A BA3E2571 7850C26C 9CD0D89D 73 | "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq" 74 | 84983E44 1C3BD26E BAAE4AA1 F95129E5 E54670F1 75 | A million repetitions of "a" 76 | 34AA973C D4C4DAA4 F61EEB2B DBAD2731 6534016F 77 | */ 78 | 79 | #include 80 | #include 81 | #include 82 | 83 | #include "sha1.h" 84 | 85 | #if defined(_MSC_VER) 86 | #pragma warning(disable : 4267) 87 | #pragma warning(disable : 4996) 88 | #pragma warning(disable : 4100) 89 | #endif 90 | 91 | void SHA1_Transform(uint32_t state[5], const uint8_t buffer[64]); 92 | 93 | #define rol ROTL32 94 | 95 | /* blk0() and blk() perform the initial expand. */ 96 | /* I got the idea of expanding during the round function from SSLeay */ 97 | /* FIXME: can we do this in an endian-proof way? */ 98 | 99 | #ifdef WORDS_BIGENDIAN 100 | #define blk0(i) block->l[i] 101 | #else 102 | #define blk0(i) (block->l[i] = (rol(block->l[i],24)&0xFF00FF00) | (rol(block->l[i],8)&0x00FF00FF)) 103 | #endif 104 | #define blk(i) (block->l[i&15] = rol(block->l[(i+13)&15]^block->l[(i+8)&15] ^ block->l[(i+2)&15]^block->l[i&15],1)) 105 | 106 | /* (R0+R1), R2, R3, R4 are the different operations used in SHA1 */ 107 | #define R0(v,w,x,y,z,i) z+=((w&(x^y))^y)+blk0(i)+0x5A827999+rol(v,5);w=rol(w,30); 108 | #define R1(v,w,x,y,z,i) z+=((w&(x^y))^y)+blk(i)+0x5A827999+rol(v,5);w=rol(w,30); 109 | #define R2(v,w,x,y,z,i) z+=(w^x^y)+blk(i)+0x6ED9EBA1+rol(v,5);w=rol(w,30); 110 | #define R3(v,w,x,y,z,i) z+=(((w|x)&y)|(w&x))+blk(i)+0x8F1BBCDC+rol(v,5);w=rol(w,30); 111 | #define R4(v,w,x,y,z,i) z+=(w^x^y)+blk(i)+0xCA62C1D6+rol(v,5);w=rol(w,30); 112 | 113 | 114 | /* Hash a single 512-bit block. This is the core of the algorithm. */ 115 | void SHA1_Transform(uint32_t state[5], const uint8_t buffer[64]) 116 | { 117 | uint32_t a, b, c, d, e; 118 | typedef union { 119 | uint8_t c[64]; 120 | uint32_t l[16]; 121 | } CHAR64LONG16; 122 | CHAR64LONG16* block; 123 | 124 | block = (CHAR64LONG16*)buffer; 125 | 126 | /* Copy context->state[] to working vars */ 127 | a = state[0]; 128 | b = state[1]; 129 | c = state[2]; 130 | d = state[3]; 131 | e = state[4]; 132 | 133 | /* 4 rounds of 20 operations each. Loop unrolled. */ 134 | R0(a,b,c,d,e, 0); R0(e,a,b,c,d, 1); R0(d,e,a,b,c, 2); R0(c,d,e,a,b, 3); 135 | R0(b,c,d,e,a, 4); R0(a,b,c,d,e, 5); R0(e,a,b,c,d, 6); R0(d,e,a,b,c, 7); 136 | R0(c,d,e,a,b, 8); R0(b,c,d,e,a, 9); R0(a,b,c,d,e,10); R0(e,a,b,c,d,11); 137 | R0(d,e,a,b,c,12); R0(c,d,e,a,b,13); R0(b,c,d,e,a,14); R0(a,b,c,d,e,15); 138 | R1(e,a,b,c,d,16); R1(d,e,a,b,c,17); R1(c,d,e,a,b,18); R1(b,c,d,e,a,19); 139 | R2(a,b,c,d,e,20); R2(e,a,b,c,d,21); R2(d,e,a,b,c,22); R2(c,d,e,a,b,23); 140 | R2(b,c,d,e,a,24); R2(a,b,c,d,e,25); R2(e,a,b,c,d,26); R2(d,e,a,b,c,27); 141 | R2(c,d,e,a,b,28); R2(b,c,d,e,a,29); R2(a,b,c,d,e,30); R2(e,a,b,c,d,31); 142 | R2(d,e,a,b,c,32); R2(c,d,e,a,b,33); R2(b,c,d,e,a,34); R2(a,b,c,d,e,35); 143 | R2(e,a,b,c,d,36); R2(d,e,a,b,c,37); R2(c,d,e,a,b,38); R2(b,c,d,e,a,39); 144 | R3(a,b,c,d,e,40); R3(e,a,b,c,d,41); R3(d,e,a,b,c,42); R3(c,d,e,a,b,43); 145 | R3(b,c,d,e,a,44); R3(a,b,c,d,e,45); R3(e,a,b,c,d,46); R3(d,e,a,b,c,47); 146 | R3(c,d,e,a,b,48); R3(b,c,d,e,a,49); R3(a,b,c,d,e,50); R3(e,a,b,c,d,51); 147 | R3(d,e,a,b,c,52); R3(c,d,e,a,b,53); R3(b,c,d,e,a,54); R3(a,b,c,d,e,55); 148 | R3(e,a,b,c,d,56); R3(d,e,a,b,c,57); R3(c,d,e,a,b,58); R3(b,c,d,e,a,59); 149 | R4(a,b,c,d,e,60); R4(e,a,b,c,d,61); R4(d,e,a,b,c,62); R4(c,d,e,a,b,63); 150 | R4(b,c,d,e,a,64); R4(a,b,c,d,e,65); R4(e,a,b,c,d,66); R4(d,e,a,b,c,67); 151 | R4(c,d,e,a,b,68); R4(b,c,d,e,a,69); R4(a,b,c,d,e,70); R4(e,a,b,c,d,71); 152 | R4(d,e,a,b,c,72); R4(c,d,e,a,b,73); R4(b,c,d,e,a,74); R4(a,b,c,d,e,75); 153 | R4(e,a,b,c,d,76); R4(d,e,a,b,c,77); R4(c,d,e,a,b,78); R4(b,c,d,e,a,79); 154 | 155 | /* Add the working vars back into context.state[] */ 156 | state[0] += a; 157 | state[1] += b; 158 | state[2] += c; 159 | state[3] += d; 160 | state[4] += e; 161 | 162 | /* Wipe variables */ 163 | a = b = c = d = e = 0; 164 | } 165 | 166 | 167 | /* SHA1Init - Initialize new context */ 168 | void SHA1_Init(SHA1_CTX* context) 169 | { 170 | /* SHA1 initialization constants */ 171 | context->state[0] = 0x67452301; 172 | context->state[1] = 0xEFCDAB89; 173 | context->state[2] = 0x98BADCFE; 174 | context->state[3] = 0x10325476; 175 | context->state[4] = 0xC3D2E1F0; 176 | context->count[0] = 0; 177 | context->count[1] = 0; 178 | } 179 | 180 | 181 | /* Run your data through this. */ 182 | void SHA1_Update(SHA1_CTX* context, const uint8_t* data, const size_t len) 183 | { 184 | size_t i, j; 185 | 186 | j = (context->count[0] >> 3) & 63; 187 | if ((context->count[0] += len << 3) < (len << 3)) context->count[1]++; 188 | 189 | context->count[1] += (len >> 29); 190 | 191 | if ((j + len) > 63) 192 | { 193 | memcpy(&context->buffer[j], data, (i = 64-j)); 194 | SHA1_Transform(context->state, context->buffer); 195 | 196 | for ( ; i + 63 < len; i += 64) 197 | { 198 | SHA1_Transform(context->state, data + i); 199 | } 200 | 201 | j = 0; 202 | } 203 | else i = 0; 204 | memcpy(&context->buffer[j], &data[i], len - i); 205 | } 206 | 207 | 208 | /* Add padding and return the message digest. */ 209 | void SHA1_Final(SHA1_CTX* context, uint8_t digest[SHA1_DIGEST_SIZE]) 210 | { 211 | uint32_t i; 212 | uint8_t finalcount[8]; 213 | 214 | for (i = 0; i < 8; i++) { 215 | finalcount[i] = (unsigned char)((context->count[(i >= 4 ? 0 : 1)] 216 | >> ((3-(i & 3)) * 8) ) & 255); /* Endian independent */ 217 | } 218 | SHA1_Update(context, (uint8_t *)"\200", 1); 219 | while ((context->count[0] & 504) != 448) { 220 | SHA1_Update(context, (uint8_t *)"\0", 1); 221 | } 222 | SHA1_Update(context, finalcount, 8); /* Should cause a SHA1_Transform() */ 223 | for (i = 0; i < SHA1_DIGEST_SIZE; i++) { 224 | digest[i] = (uint8_t) 225 | ((context->state[i>>2] >> ((3-(i & 3)) * 8) ) & 255); 226 | } 227 | 228 | /* Wipe variables */ 229 | i = 0; 230 | memset(context->buffer, 0, 64); 231 | memset(context->state, 0, 20); 232 | memset(context->count, 0, 8); 233 | memset(finalcount, 0, 8); /* SWR */ 234 | } 235 | 236 | //----------------------------------------------------------------------------- 237 | 238 | void sha1_32a ( const void * key, int len, uint32_t seed, void * out ) 239 | { 240 | SHA1_CTX context; 241 | 242 | uint8_t digest[20]; 243 | 244 | SHA1_Init(&context); 245 | SHA1_Update(&context, (uint8_t*)key, len); 246 | SHA1_Final(&context, digest); 247 | 248 | memcpy(out,&digest[0],4); 249 | } 250 | 251 | //----------------------------------------------------------------------------- 252 | // self test 253 | 254 | //#define TEST 255 | 256 | #ifdef TEST 257 | 258 | static char *test_data[] = { 259 | "abc", 260 | "abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq", 261 | "A million repetitions of 'a'"}; 262 | static char *test_results[] = { 263 | "A9993E36 4706816A BA3E2571 7850C26C 9CD0D89D", 264 | "84983E44 1C3BD26E BAAE4AA1 F95129E5 E54670F1", 265 | "34AA973C D4C4DAA4 F61EEB2B DBAD2731 6534016F"}; 266 | 267 | 268 | void digest_to_hex(const uint8_t digest[SHA1_DIGEST_SIZE], char *output) 269 | { 270 | int i,j; 271 | char *c = output; 272 | 273 | for (i = 0; i < SHA1_DIGEST_SIZE/4; i++) { 274 | for (j = 0; j < 4; j++) { 275 | sprintf(c,"%02X", digest[i*4+j]); 276 | c += 2; 277 | } 278 | sprintf(c, " "); 279 | c += 1; 280 | } 281 | *(c - 1) = '\0'; 282 | } 283 | 284 | int main(int argc, char** argv) 285 | { 286 | int k; 287 | SHA1_CTX context; 288 | uint8_t digest[20]; 289 | char output[80]; 290 | 291 | fprintf(stdout, "verifying SHA-1 implementation... "); 292 | 293 | for (k = 0; k < 2; k++){ 294 | SHA1_Init(&context); 295 | SHA1_Update(&context, (uint8_t*)test_data[k], strlen(test_data[k])); 296 | SHA1_Final(&context, digest); 297 | digest_to_hex(digest, output); 298 | 299 | if (strcmp(output, test_results[k])) { 300 | fprintf(stdout, "FAIL\n"); 301 | fprintf(stderr,"* hash of \"%s\" incorrect:\n", test_data[k]); 302 | fprintf(stderr,"\t%s returned\n", output); 303 | fprintf(stderr,"\t%s is correct\n", test_results[k]); 304 | return (1); 305 | } 306 | } 307 | /* million 'a' vector we feed separately */ 308 | SHA1_Init(&context); 309 | for (k = 0; k < 1000000; k++) 310 | SHA1_Update(&context, (uint8_t*)"a", 1); 311 | SHA1_Final(&context, digest); 312 | digest_to_hex(digest, output); 313 | if (strcmp(output, test_results[2])) { 314 | fprintf(stdout, "FAIL\n"); 315 | fprintf(stderr,"* hash of \"%s\" incorrect:\n", test_data[2]); 316 | fprintf(stderr,"\t%s returned\n", output); 317 | fprintf(stderr,"\t%s is correct\n", test_results[2]); 318 | return (1); 319 | } 320 | 321 | /* success */ 322 | fprintf(stdout, "ok\n"); 323 | return(0); 324 | } 325 | #endif /* TEST */ 326 | -------------------------------------------------------------------------------- /src/sha1.h: -------------------------------------------------------------------------------- 1 | /* public api for steve reid's public domain SHA-1 implementation */ 2 | /* this file is in the public domain */ 3 | 4 | #pragma once 5 | 6 | #include "Platform.h" 7 | 8 | struct SHA1_CTX 9 | { 10 | uint32_t state[5]; 11 | uint32_t count[2]; 12 | uint8_t buffer[64]; 13 | }; 14 | 15 | #define SHA1_DIGEST_SIZE 20 16 | 17 | void SHA1_Init(SHA1_CTX* context); 18 | void SHA1_Update(SHA1_CTX* context, const uint8_t* data, const size_t len); 19 | void SHA1_Final(SHA1_CTX* context, uint8_t digest[SHA1_DIGEST_SIZE]); 20 | 21 | void sha1_32a ( const void * key, int len, uint32_t seed, void * out ); -------------------------------------------------------------------------------- /src/xxh3.h: -------------------------------------------------------------------------------- 1 | /* 2 | * xxHash - Extremely Fast Hash algorithm 3 | * Development source file for `xxh3` 4 | * Copyright (C) 2019-2020 Yann Collet 5 | * 6 | * BSD 2-Clause License (https://www.opensource.org/licenses/bsd-license.php) 7 | * 8 | * Redistribution and use in source and binary forms, with or without 9 | * modification, are permitted provided that the following conditions are 10 | * met: 11 | * 12 | * * Redistributions of source code must retain the above copyright 13 | * notice, this list of conditions and the following disclaimer. 14 | * * Redistributions in binary form must reproduce the above 15 | * copyright notice, this list of conditions and the following disclaimer 16 | * in the documentation and/or other materials provided with the 17 | * distribution. 18 | * 19 | * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20 | * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21 | * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22 | * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23 | * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24 | * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25 | * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26 | * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27 | * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | * 31 | * You can contact the author at: 32 | * - xxHash homepage: https://www.xxhash.com 33 | * - xxHash source repository: https://github.com/Cyan4973/xxHash 34 | */ 35 | 36 | /* 37 | * Note: This file used to host the source code of XXH3_* variants. 38 | * during the development period. 39 | * The source code is now properly integrated within xxhash.h. 40 | * 41 | * xxh3.h is no longer useful, 42 | * but it is still provided for compatibility with source code 43 | * which used to include it directly. 44 | * 45 | * Programs are now highly discourage to include xxh3.h. 46 | * Include `xxhash.h` instead, which is the officially supported interface. 47 | * 48 | * In the future, xxh3.h will start to generate warnings, then errors, 49 | * then it will be removed from source package and from include directory. 50 | */ 51 | 52 | /* Simulate the same impact as including the old xxh3.h source file */ 53 | 54 | #define XXH_INLINE_ALL 55 | #include "xxhash.h" 56 | -------------------------------------------------------------------------------- /src/xxhash.c: -------------------------------------------------------------------------------- 1 | /* 2 | * xxHash - Extremely Fast Hash algorithm 3 | * Copyright (C) 2012-2020 Yann Collet 4 | * 5 | * BSD 2-Clause License (https://www.opensource.org/licenses/bsd-license.php) 6 | * 7 | * Redistribution and use in source and binary forms, with or without 8 | * modification, are permitted provided that the following conditions are 9 | * met: 10 | * 11 | * * Redistributions of source code must retain the above copyright 12 | * notice, this list of conditions and the following disclaimer. 13 | * * Redistributions in binary form must reproduce the above 14 | * copyright notice, this list of conditions and the following disclaimer 15 | * in the documentation and/or other materials provided with the 16 | * distribution. 17 | * 18 | * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 19 | * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 20 | * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 21 | * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 22 | * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 24 | * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 25 | * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 26 | * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 27 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 28 | * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 29 | * 30 | * You can contact the author at: 31 | * - xxHash homepage: https://www.xxhash.com 32 | * - xxHash source repository: https://github.com/Cyan4973/xxHash 33 | */ 34 | 35 | 36 | /* 37 | * xxhash.c instantiates functions defined in xxhash.h 38 | */ 39 | 40 | #define XXH_STATIC_LINKING_ONLY /* access advanced declarations */ 41 | #define XXH_IMPLEMENTATION /* access definitions */ 42 | 43 | #include "xxhash.h" 44 | --------------------------------------------------------------------------------