10 |

11 |

12 |

Bloom filters

13 | 14 |

15 | Bloom filter 16 |

17 | 18 | 19 | Image released to the public domain by David Eppstein. 20 | 21 |

22 | 23 |

24 |

Me

25 |

26 | Shane Tomlinson 27 |

28 |

29 | https://shanetomlinson.com 30 |

31 |

32 | @shane_tomlinson 33 |

34 |

35 | https://github.com/shane-tomlinson 36 |

37 |

38 | 39 |

40 |

Code

41 | https://github.com/LondonAlgorithms/Bloomfilters 42 |

43 | 44 |

45 |

Start with a problem

46 |

47 | 48 |

49 |

50 | top 25 passwords 51 |

52 | 53 | Top 25 Most Used Internet Passwords - An infographic by the team at Tom Fanelli - Infographic Marketing. 54 | 55 |

56 | 57 |

58 |

Goal

59 |

60 | Ban users from using one of the top 50k most common passwords. 61 |

62 | 63 |

Math(s) speak: determine if an item is a member of a set.

64 | 65 |

66 | 67 |

68 |

Constraints

69 |

No data can be sent to the server.
Download must be "reasonably" sized.

73 |

74 | 75 |

76 |

What data structures could we use?

77 |

78 | 79 |

80 |

Hash tables

81 | 82 |

83 | Create a bit array, hash entries to a position in array. 84 |

85 | 86 |

87 | Download hash table, run password through hash function, check if bit is set. 88 |

89 | 90 | 91 |

92 |

Collisions

93 | 94 |

Inevitable. See birthday problem.
Causes false positives.

98 |

99 |

100 | 101 |

102 |

Reducing collisions

103 | 104 |

Expand the size of the hash table.
Use more than one hash function.

108 |

109 | 110 |

111 |

What if the occasional false positive is OK?

112 |

113 | 114 |

115 |

Bloom filter

116 | 117 |

118 | A Bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom in 1970, that is used to test whether an element is a member of a set. 119 |

120 | 121 |

122 | 123 | https://en.wikipedia.org/wiki/Bloom_filter 124 | 125 |

126 | 127 |

128 | 129 |

130 |

It looks like an onion

131 | 132 |

133 |

134 | 135 |

136 |

Properties

137 | 138 |

Has two primary functions, add and test.
If an item is in the set, test will always return true.
If an item is not in the set, test might return true.
Rate of false positives is configurable.

144 |

145 | 146 |

147 |

Secondary properties

148 | 149 |

Hash table can be downloaded and tested against.
False positive rate of <1% uses ~9.6 bits per member.
Supports union and intersection of two or more filters.

154 |

155 | 156 |

157 |

Nomenclature

158 | 159 |

k: number of hash functions.
m: size of hash table, in bits.
n: size of set.

164 |

165 | 166 |

167 |

Bloom filter algorithm

168 |

169 | 170 |

171 | 172 |

Create

173 |

Create a bit array of size m.
Fill bit array with all 0s.

177 | 178 |

Add

179 |

Run item through k independent hash functions.
Set the corresponding entries in the bit array to 1.

183 | 184 |

Test

185 |

Run item through k independent hash functions.
188 | Check the corresponding entries in the bit array. 189 |
1. If all entries are 1, return true, otherwise return false.
192 |

194 |

195 | 196 |

197 |

K independent ... WTF?

198 |

199 | 200 |

201 |

K independent hash functions

202 | 203 |

204 | Can be simulated with only two hash functions. 205 |

206 | 207 |

208 | To find an arbitrary i < k: 209 |

210 |


211 |           hash(i) = (a + b * i ) % m;
212 |

213 | 214 |

215 | Where a is the result of Hash 1, b is the result of Hash 2, and m is the size 216 | of the bit array. 217 |

218 |

219 | 220 |

221 |

To adjust false positive rate

222 | 223 |

Fiddle with k and m.

224 | 225 |

226 |

227 | Increasing m gives us more space (bigger download). 228 |

229 |

230 | Increasing k gives us a better chance to find "0"s, but fills the hash table more quickly. 231 |

232 |

233 | 234 |

235 | Look at this table. 236 |

237 |

238 | 239 | 240 |

241 |