├── .travis.yml ├── LICENSE.txt ├── Makefile ├── README.md ├── adhole ├── main.go ├── sigwait_unix.go └── sigwait_windows.go └── genlist ├── main.go └── sources.go /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | go: 3 | - 1.3 4 | script: 5 | - cd adhole && go build -v 6 | - cd ../genlist && go build -v 7 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2014 - 2019, Piotr S. Staszewski 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | 2. Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 15 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 16 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 17 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 18 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 19 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 20 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 21 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 22 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 23 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 24 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: adhole genlist 2 | 3 | adhole/adhole: adhole/main.go adhole/sigwait_unix.go adhole/sigwait_windows.go 4 | cd adhole; \ 5 | gofmt -w *.go; \ 6 | go build . 7 | 8 | genlist/genlist: genlist/main.go genlist/sources.go 9 | cd genlist; \ 10 | gofmt -w *.go; \ 11 | go build . 12 | 13 | adhole: adhole/adhole 14 | genlist: genlist/genlist 15 | .PHONY: adhole 16 | .PHONY: genlist 17 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # adhole [![Build Status](https://travis-ci.org/drbig/adhole.svg?branch=master)](https://travis-ci.org/drbig/adhole) 2 | 3 | AdHole is a simple transparent advertisement and tracking blocker intended for 4 | personal use. 5 | 6 | It works by providing two elements: first a DNS server that, based on a list of 7 | domains, either relies the query to a real server, or returns its own IP 8 | address; and a micro HTTP server that serves an empty GIF image for any 9 | request. As such its functionality is akin to 10 | [AdBlock](https://chrome.google.com/webstore/detail/adblock/gighmmpiobklfepjocnamgkkbiglidom?hl=en) 11 | , but less pretty (you'll probably see empty space where ads 12 | used to be) and completely transparent (i.e. will work for any browser, on any 13 | OS on any machine, locally or over any network). Note that you can just as well 14 | block *any* domains. 15 | 16 | I use AdHole in 'production' on my 17 | [RaspberryPi](http://www.raspberrypi.org/)-powered WiFi router, where it 18 | provides ad and tracking blocking to all my wireless toys. 19 | 20 | My programming goal was to make it as simple and minimal as possible. Therefore 21 | instead of a full DNS stack (and especially parsing) I work directly on 22 | `[]byte` slices and try to reuse the data already on hand. It was also a nice 23 | little adventure in dealing with binary protocols the low-level way. 24 | 25 | ## Changelog 26 | 27 | - **2016-11-11** Added locking around the queries map. Can't really test it as the RPi is obviously single-core. 28 | 29 | ## Building 30 | 31 | On a Unix-like system you should be able to get everything built using just: 32 | 33 | $ make 34 | cd adhole; \ 35 | gofmt -w *.go; \ 36 | go build . 37 | cd genlist; \ 38 | gofmt -w *.go; \ 39 | go build . 40 | 41 | Otherwise just run `go build .` in either or both of `adhole/` and `genlist/`. 42 | 43 | ## Usage 44 | 45 | $ ./adhole 46 | Usage: ./adhole [options] key upstream proxy list.txt 47 | 48 | key - password used for /debug actions protection 49 | upstream - real upstream DNS address, e.g. 8.8.8.8 50 | proxy - servers' bind address, e.g. 127.0.0.1 51 | list.txt - text file with domains to block 52 | 53 | -dport=53: DNS server port 54 | -hport=80: HTTP server port 55 | -t=5s: upstream query timeout 56 | -v=false: be verbose 57 | 58 | Note that you will need root privileges to run it on the default ports. 59 | 60 | List format is simply: one domain name per line. All subdomains of a given 61 | domain will be blocked, so there is no need to use `*`. Domains should also not 62 | end with a dot. The parser should also be indifferent to line endings. Example 63 | list file: 64 | 65 | 101com.com 66 | 101order.com 67 | 103bees.com 68 | 123found.com 69 | 123pagerank.com 70 | 71 | To get a decent list of domains to block I recommend going 72 | [here](http://pgl.yoyo.org/adservers/) and generating a 'plain non-HTML list -- 73 | as a plain list of hostnames (no HTML)' with 'no links back to this page' and 74 | 'view list as plain text:' ticked (that was so verbose...). Or alternatively 75 | [this](http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext) 76 | should work as a direct link to the current list... 77 | 78 | Or you can use the experimental `genlist` utility. Currently I've implemented 79 | fetching for the above list and for 80 | [EasyList](https://easylist.adblockplus.org/en/). 81 | If you have a better / your favourite source of domain blacklist 82 | please look at `sources.go` and implement a fetcher. As a side note, sources 83 | that are meant to be used by AdBlock (or equivalent) are not the best 84 | candidates here, due to the fact that AdBlock is much more subtle, i.e. with 85 | current extraction method a list generated with EasyList input will block 86 | google.com, bbc.co.uk and imdb.com (and probably many more sites that you 87 | care about). Also the whole thing is brittle by nature, hence the permanent 88 | 'experimental' status. 89 | 90 | $ ./genlist 91 | Usage: ./genlist [options] list|all|source source source... 92 | 93 | list - show available sources 94 | all - combine all sources 95 | source - use only specified source(s) 96 | 97 | $ ./genlist list 98 | There are 2 sources available: 99 | 01 - pgl - http://pgl.yoyo.org/adservers 100 | 02 - easylist - https://easylist.adblockplus.org 101 | 102 | $ ./genlist all > /tmp/blacklist.txt 103 | Processing list for pgl 104 | Processing list for easylist 105 | Got 9829 domains total 106 | 107 | Example [systemd](http://www.freedesktop.org/wiki/Software/systemd/) service 108 | file: 109 | 110 | [Unit] 111 | Description=DNS blackhole 112 | After=network.target 113 | 114 | [Service] 115 | ExecStart=/home/user/adhole SecretKey 192.168.0.3 192.168.0.21 /home/user/blacklist.txt 116 | 117 | [Install] 118 | WantedBy=multi-user.target 119 | 120 | Thanks to the great [expvar](http://golang.org/pkg/expvar/) package you can 121 | monitor some statistics by visiting `http://proxy.addr/debug/vars`. The 122 | following items are relevant: 123 | 124 | * `stateIsRunning` - if false all queries are relied to upstream 125 | * `statsQuestions` - number of received queries 126 | * `statsRelayed` - number of queries relayed to the real server 127 | * `statsBlocked` - number of queries blocked 128 | * `statsTimedout` - number of relayed queries that timed out 129 | * `statsServed` - number of HTTP requests served 130 | * `statsErrors` - number of errors encountered 131 | * `statsRules` - number of items read from the blacklist 132 | 133 | You can also do the following actions via HTTP: 134 | 135 | * `/debug/reload` - will reload the list.txt file 136 | * `/debug/toggle` - toggle blocking on and off 137 | 138 | You'll need to append `&key=YOURKEY` to the above. Unauthorized hits will 139 | be logged. Note that you may set the key to `""` (i.e. an empty key) and 140 | therefore disable the authentication. 141 | 142 | **Tested on:** 143 | 144 | * Linux - amd64, armv6l 145 | * Windows XP - i686 146 | 147 | ## Contributing 148 | 149 | Feel free to either use GitHub's pull request or send me patches directly. 150 | 151 | I also welcome any feedback regarding stability and what OS/platform you run 152 | AdHole on (just out of curiosity). 153 | 154 | ## Bugs / Todo 155 | 156 | * Edge cases (multiple questions per query, anybody?) 157 | * Even less data shuffling 158 | * IPv4 only 159 | 160 | ## Copyright 161 | 162 | Copyright (c) 2014 - 2019 Piotr S. Staszewski 163 | 164 | Absolutely no warranty. See LICENSE.txt for details. 165 | -------------------------------------------------------------------------------- /adhole/main.go: -------------------------------------------------------------------------------- 1 | // See LICENSE.txt for licensing information. 2 | 3 | package main 4 | 5 | import ( 6 | "bufio" 7 | "bytes" 8 | "expvar" 9 | "flag" 10 | "fmt" 11 | "io" 12 | "log" 13 | "net" 14 | "net/http" 15 | "os" 16 | "strings" 17 | "sync" 18 | "time" 19 | ) 20 | 21 | // query wraps Host name and clients UDPAddr. 22 | type query struct { 23 | Host string 24 | From *net.UDPAddr 25 | } 26 | 27 | // queryMap is a synced map for keeping track of relied queries. 28 | type queryMap struct { 29 | mu sync.RWMutex 30 | qs map[int]*query 31 | } 32 | 33 | // String prints human-readable representation of a query. 34 | func (q *query) String() string { 35 | return fmt.Sprintf("from %s about %s", q.From, q.Host) 36 | } 37 | 38 | // toggle is a synced bool wrapper for expvar. 39 | type toggle struct { 40 | mu sync.RWMutex 41 | b bool 42 | } 43 | 44 | // String converts a toggle to string. 45 | func (t *toggle) String() string { 46 | t.mu.RLock() 47 | defer t.mu.RUnlock() 48 | if t.b { 49 | return "true" 50 | } 51 | return "false" 52 | } 53 | 54 | // Value returns a toggle value. 55 | func (t *toggle) Value() bool { 56 | t.mu.RLock() 57 | defer t.mu.RUnlock() 58 | return t.b 59 | } 60 | 61 | // Toggle toggles a toggle. 62 | func (t *toggle) Toggle() bool { 63 | t.mu.Lock() 64 | defer t.mu.Unlock() 65 | t.b = !t.b 66 | return t.b 67 | } 68 | 69 | // Flags. 70 | var ( 71 | flagVerbose = flag.Bool("v", false, "be verbose") 72 | flagHTTPPort = flag.Int("hport", 80, "HTTP server port") 73 | flagDNSPort = flag.Int("dport", 53, "DNS server port") 74 | flagTimeout = flag.Duration("t", 5*time.Second, "upstream query timeout") 75 | ) 76 | 77 | // Expvar exported statistics counters. 78 | var ( 79 | cntMsgs = expvar.NewInt("statsQuestions") 80 | cntRelayed = expvar.NewInt("statsRelayed") 81 | cntBlocked = expvar.NewInt("statsBlocked") 82 | cntTimedout = expvar.NewInt("statsTimedout") 83 | cntServed = expvar.NewInt("statsServed") 84 | cntErrors = expvar.NewInt("statsErrors") 85 | cntRules = expvar.NewInt("statsRules") 86 | ) 87 | 88 | // 'Static' variables. 89 | // answer will become static once the proxy address bytes have been added. 90 | var ( 91 | // answer is the header of a DNS query response without the domain name and 92 | // the resource data. 93 | // 94 | // Good sources of information on the DNS protocol can be found at: 95 | // http://www.firewall.cx/networking-topics/protocols/domain-name-system-dns 96 | // http://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml 97 | // 98 | // Bytes described: 99 | // 2 - Type = 0x0001 - A 100 | // 2 - Class = 0x0001 - IN 101 | // 4 - TTL = 0xffffffff - if anyone respects it, this should reduce hits 102 | // 2 - Data Length = 0x0004 - number of resource bytes, which for IN A IPv4 address is exactly 4 103 | answer = []byte("\x00\x01\x00\x01\xff\xff\xff\xff\x00\x04") 104 | 105 | // pixel is a hex representation of an 'empty' 1x1 GIF image. 106 | pixel = "\x47\x49\x46\x38\x39\x61\x01\x00\x01\x00\x80\x00\x00\xff\xff" + 107 | "\xff\x00\x00\x00\x21\xf9\x04\x01\x00\x00\x00\x00\x2c\x00\x00" + 108 | "\x00\x00\x01\x00\x01\x00\x00\x02\x02\x44\x01\x00\x3b" 109 | ) 110 | 111 | var ( 112 | proxy *net.UDPConn 113 | upstream *net.UDPConn 114 | queries queryMap 115 | blocked map[string]bool 116 | blocking = &toggle{b: true} 117 | key string 118 | list string 119 | ) 120 | 121 | func init() { 122 | expvar.Publish("stateIsRunning", blocking) 123 | } 124 | 125 | func main() { 126 | flag.Usage = func() { 127 | fmt.Fprintf(os.Stderr, "Usage: %s [options] key upstream proxy list.txt\n\n"+ 128 | "key - password used for /debug actions protection\n"+ 129 | "upstream - real upstream DNS address, e.g. 8.8.8.8\n"+ 130 | "proxy - servers' bind address, e.g. 127.0.0.1\n"+ 131 | "list.txt - text file with domains to block\n\n", 132 | os.Args[0], 133 | ) 134 | flag.PrintDefaults() 135 | return 136 | } 137 | flag.Parse() 138 | 139 | if len(flag.Args()) < 4 { 140 | flag.Usage() 141 | os.Exit(1) 142 | } 143 | 144 | key = flag.Arg(0) 145 | upIP := parseIPv4(flag.Arg(1), "upstream") 146 | proxyIP := parseIPv4(flag.Arg(2), "proxy") 147 | answer = append(answer, proxyIP...) 148 | list = flag.Arg(3) 149 | parseList(list) 150 | 151 | var err error 152 | upAddr := &net.UDPAddr{IP: upIP, Port: 53} 153 | upstream, err = net.DialUDP("udp4", nil, upAddr) 154 | if err != nil { 155 | fmt.Fprintf(os.Stderr, "ERROR: %s\n", err) 156 | os.Exit(2) 157 | } 158 | defer upstream.Close() 159 | 160 | proxyAddr := &net.UDPAddr{IP: proxyIP, Port: *flagDNSPort} 161 | proxy, err = net.ListenUDP("udp4", proxyAddr) 162 | if err != nil { 163 | fmt.Fprintf(os.Stderr, "ERROR: %s\n", err) 164 | os.Exit(2) 165 | } 166 | defer proxy.Close() 167 | 168 | queries.qs = make(map[int]*query, 4096) 169 | 170 | go runServerHTTP(proxyIP.String()) 171 | go runServerUpstreamDNS() 172 | go runServerLocalDNS() 173 | 174 | sigwait() 175 | } 176 | 177 | // parseIPv4 parses a string to an IPv4 address or dies. 178 | func parseIPv4(arg string, msg string) (ip net.IP) { 179 | ip = net.ParseIP(arg) 180 | if ip == nil { 181 | fmt.Fprintf(os.Stderr, "ERROR: Can't parse %s IP '%s'\n", msg, arg) 182 | os.Exit(2) 183 | } 184 | ip = ip.To4() 185 | if ip == nil { 186 | fmt.Fprintln(os.Stderr, "ERROR: IPv6 is not supported, sorry") 187 | os.Exit(3) 188 | } 189 | return 190 | } 191 | 192 | // parseList loads a block list file into blocked and updates rules counter. 193 | func parseList(path string) { 194 | file, err := os.Open(path) 195 | if err != nil { 196 | fmt.Fprintln(os.Stderr, "ERROR:", err) 197 | os.Exit(2) 198 | } 199 | defer file.Close() 200 | 201 | blocked = make(map[string]bool, 4096) 202 | counter := 0 203 | scn := bufio.NewScanner(file) 204 | for scn.Scan() { 205 | counter++ 206 | blocked[scn.Text()+"."] = true 207 | } 208 | log.Printf("DNS: Parsed %d entries from list\n", counter) 209 | cntRules.Set(int64(counter)) 210 | return 211 | } 212 | 213 | // runServerLocalDNS listens for incoming DNS queries and dispatches them for processing. 214 | func runServerLocalDNS() { 215 | log.Println("DNS: Started local server at", proxy.LocalAddr()) 216 | 217 | buf := make([]byte, 512) 218 | for { 219 | n, addr, err := proxy.ReadFromUDP(buf) 220 | if err != nil { 221 | log.Println("DNS ERROR (1):", err) 222 | cntErrors.Add(1) 223 | continue 224 | } 225 | 226 | msg := make([]byte, n) 227 | copy(msg, buf[:n]) 228 | cntMsgs.Add(1) 229 | go handleDNS(msg, addr) 230 | } 231 | } 232 | 233 | // runServerUpstreamDNS listens for upstream answers and relies them to original clients. 234 | func runServerUpstreamDNS() { 235 | log.Println("DNS: Started upstream server") 236 | 237 | buf := make([]byte, 512) 238 | for { 239 | n, _, err := upstream.ReadFromUDP(buf) 240 | if err != nil { 241 | log.Println("DNS ERROR (2):", err) 242 | cntErrors.Add(1) 243 | continue 244 | } 245 | 246 | id := int(uint16(buf[0])<<8 + uint16(buf[1])) 247 | queries.mu.RLock() 248 | if query, ok := queries.qs[id]; ok { 249 | queries.mu.RUnlock() 250 | queries.mu.Lock() 251 | delete(queries.qs, id) 252 | queries.mu.Unlock() 253 | _, err := proxy.WriteTo(buf[:n], query.From) 254 | if err != nil { 255 | log.Printf("DNS ERROR: Query id %d %s %s", id, query, err) 256 | cntErrors.Add(1) 257 | continue 258 | } 259 | if *flagVerbose { 260 | log.Println("DNS: Relayed answer to query", id) 261 | } 262 | cntRelayed.Add(1) 263 | } else { 264 | queries.mu.RUnlock() 265 | } 266 | } 267 | } 268 | 269 | // handleDNS peeks the query and either relies it to the upstream DNS server or returns 270 | // a static answer with the 'fake' IP. 271 | func handleDNS(msg []byte, from *net.UDPAddr) { 272 | var domain bytes.Buffer 273 | var block bool 274 | 275 | id := int(uint16(msg[0])<<8 + uint16(msg[1])) 276 | if *flagVerbose { 277 | log.Printf("DNS: Query id %d from %s\n", id, from) 278 | } 279 | 280 | count := uint8(msg[5]) // question counter 281 | offset := 12 // point to first domain name 282 | 283 | if count != 1 { 284 | log.Printf("DNS WARN: Query id %d from %s has %d questions\n", id, from, count) 285 | return 286 | } 287 | 288 | for { 289 | length := int8(msg[offset]) 290 | if length == 0 { 291 | break 292 | } 293 | offset++ 294 | domain.WriteString(string(msg[offset : offset+int(length)])) 295 | domain.WriteString(".") 296 | offset += int(length) 297 | } 298 | host := domain.String() 299 | testHost := host 300 | parts := strings.Split(testHost, ".") 301 | try := 1 302 | for { 303 | if _, ok := blocked[testHost]; ok { 304 | block = true 305 | break 306 | } 307 | parts = parts[1:] 308 | if len(parts) < 3 { 309 | break 310 | } 311 | testHost = strings.Join(parts, ".") 312 | try++ 313 | } 314 | 315 | if blocking.Value() && block { 316 | if *flagVerbose { 317 | log.Printf("DNS: Blocking (%d) %s\n", try, host) 318 | } 319 | cntBlocked.Add(1) 320 | 321 | msg[2] = uint8(129) // flags upper byte 322 | msg[3] = uint8(128) // flags lower byte 323 | msg[7] = uint8(1) // answer counter 324 | 325 | msg = append(msg, msg[12:12+1+len(host)]...) // domain 326 | msg = append(msg, answer...) // payload 327 | _, err := proxy.WriteTo(msg, from) 328 | if err != nil { 329 | log.Println("DNS ERROR (3):", err) 330 | cntErrors.Add(1) 331 | return 332 | } 333 | if *flagVerbose { 334 | log.Println("DNS: Sent fake answer") 335 | } 336 | } else { 337 | if *flagVerbose { 338 | log.Println("DNS: Asking upstream") 339 | } 340 | queries.mu.Lock() 341 | queries.qs[id] = &query{From: from, Host: host} 342 | queries.mu.Unlock() 343 | _, err := upstream.Write(msg) 344 | if err != nil { 345 | log.Println("DNS ERROR (4):", err) 346 | cntErrors.Add(1) 347 | queries.mu.Lock() 348 | delete(queries.qs, id) 349 | queries.mu.Unlock() 350 | return 351 | } 352 | go func(queryID int) { 353 | time.Sleep(*flagTimeout) 354 | queries.mu.RLock() 355 | if query, ok := queries.qs[queryID]; ok { 356 | fmt.Printf("DNS WARN: Query id %d %s timed out\n", queryID, query) 357 | cntTimedout.Add(1) 358 | queries.mu.RUnlock() 359 | queries.mu.Lock() 360 | delete(queries.qs, queryID) 361 | queries.mu.Unlock() 362 | } else { 363 | queries.mu.RUnlock() 364 | } 365 | return 366 | }(id) 367 | } 368 | return 369 | } 370 | 371 | // authHTTP checks if user supplied proper key. 372 | func authHTTP(req *http.Request) bool { 373 | if val := req.FormValue("key"); val == key { 374 | return true 375 | } 376 | log.Printf("HTTP: Unauthorized access to %s from %s\n", req.RequestURI, req.RemoteAddr) 377 | return false 378 | } 379 | 380 | // handleHTTP returns an 'empty' 1x1 GIF image for any URL. 381 | func handleHTTP(w http.ResponseWriter, req *http.Request) { 382 | if *flagVerbose { 383 | log.Printf("HTTP: Request %s %s %s\n", req.Method, req.Host, req.RequestURI) 384 | } 385 | cntServed.Add(1) 386 | w.Header()["Content-type"] = []string{"image/gif"} 387 | io.WriteString(w, pixel) 388 | return 389 | } 390 | 391 | // handleReload reloads the rules and redirects to the debug page. 392 | func handleReload(w http.ResponseWriter, req *http.Request) { 393 | if authHTTP(req) { 394 | parseList(list) 395 | log.Println("Rules reloaded:", cntRules) 396 | } 397 | http.Redirect(w, req, "/debug/vars", http.StatusSeeOther) 398 | return 399 | } 400 | 401 | // handleToggle toggles blocking and redirects to the debug page. 402 | func handleToggle(w http.ResponseWriter, req *http.Request) { 403 | if authHTTP(req) { 404 | log.Println("Blocking toggled to:", blocking.Toggle()) 405 | } 406 | http.Redirect(w, req, "/debug/vars", http.StatusSeeOther) 407 | return 408 | } 409 | 410 | // runServerHTTP starts the HTTP server. 411 | func runServerHTTP(host string) { 412 | addr := fmt.Sprintf("%s:%d", host, *flagHTTPPort) 413 | http.HandleFunc("/", handleHTTP) 414 | http.HandleFunc("/debug/reload", handleReload) 415 | http.HandleFunc("/debug/toggle", handleToggle) 416 | log.Println("HTTP: Started at", addr) 417 | log.Fatalln(http.ListenAndServe(addr, nil)) 418 | panic("not reachable") 419 | } 420 | 421 | // vim: ts=4 sw=4 sts=4 422 | -------------------------------------------------------------------------------- /adhole/sigwait_unix.go: -------------------------------------------------------------------------------- 1 | // See LICENSE.txt for licensing information. 2 | // +build !windows 3 | 4 | package main 5 | 6 | import ( 7 | "log" 8 | "os" 9 | "os/signal" 10 | "syscall" 11 | ) 12 | 13 | // sigwait processes signals such as a CTRL-C hit. 14 | func sigwait() { 15 | sig := make(chan os.Signal) 16 | signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM, syscall.SIGKILL) 17 | 18 | <-sig 19 | log.Println("Signal received, stopping") 20 | 21 | return 22 | } 23 | -------------------------------------------------------------------------------- /adhole/sigwait_windows.go: -------------------------------------------------------------------------------- 1 | // See LICENSE.txt for licensing information. 2 | // +build windows 3 | 4 | package main 5 | 6 | import ( 7 | "log" 8 | "os" 9 | "os/signal" 10 | ) 11 | 12 | // sigwait processes signals such as a CTRL-C hit. 13 | func sigwait() { 14 | sig := make(chan os.Signal) 15 | signal.Notify(sig, os.Interrupt, os.Kill) 16 | 17 | <-sig 18 | log.Println("Signal received, stopping") 19 | 20 | return 21 | } 22 | -------------------------------------------------------------------------------- /genlist/main.go: -------------------------------------------------------------------------------- 1 | // See LICENSE.txt for licensing information. 2 | 3 | package main 4 | 5 | import ( 6 | "bufio" 7 | "flag" 8 | "fmt" 9 | "net/http" 10 | "os" 11 | "sync" 12 | ) 13 | 14 | // Source represents a remote data source and holds a function 15 | // that will extract a domain name from a line of data. 16 | type Source struct { 17 | Name string 18 | Desc string 19 | URL string 20 | Extractor func(string) *string 21 | } 22 | 23 | var ( 24 | domains map[string]bool 25 | wg sync.WaitGroup 26 | ) 27 | 28 | // Process will fetch the source data, extract domain names 29 | // and put them into the global domains map. 30 | func (s *Source) Process() { 31 | fmt.Fprintf(os.Stderr, "Processing list for %s\n", s.Name) 32 | defer wg.Done() 33 | resp, err := http.Get(s.URL) 34 | if err != nil { 35 | fmt.Fprintf(os.Stderr, "Error: (%s) %s\n", s.Name, err) 36 | return 37 | } 38 | defer resp.Body.Close() 39 | scanner := bufio.NewScanner(resp.Body) 40 | for scanner.Scan() { 41 | domain := s.Extractor(scanner.Text()) 42 | if domain != nil { 43 | domains[*domain] = true 44 | } 45 | } 46 | return 47 | } 48 | 49 | func main() { 50 | flag.Usage = func() { 51 | fmt.Fprintf(os.Stderr, "Usage: %s [options] list|all|source source source...\n\n"+ 52 | "list - show available sources\n"+ 53 | "all - combine all sources\n"+ 54 | "source - use only specified source(s)\n\n", 55 | os.Args[0]) 56 | flag.PrintDefaults() 57 | return 58 | } 59 | flag.Parse() 60 | if len(flag.Args()) < 1 { 61 | flag.Usage() 62 | os.Exit(1) 63 | } 64 | 65 | domains = make(map[string]bool, 4096) 66 | available := make(map[string]*Source, len(sources)) 67 | for _, src := range sources { 68 | if _, exists := available[src.Name]; exists { 69 | panic(fmt.Sprintf("Duplicate source %s!", src.Name)) 70 | } 71 | available[src.Name] = src 72 | } 73 | 74 | switch flag.Arg(0) { 75 | case "all": 76 | for _, src := range sources { 77 | wg.Add(1) 78 | go src.Process() 79 | } 80 | case "list": 81 | fmt.Fprintf(os.Stderr, "There are %d sources available:\n", len(sources)) 82 | for i, src := range sources { 83 | fmt.Fprintf(os.Stderr, "%02d - %-10s - %s\n", i+1, src.Name, src.Desc) 84 | } 85 | return 86 | default: 87 | for _, name := range flag.Args() { 88 | src, exists := available[name] 89 | if !exists { 90 | fmt.Fprintf(os.Stderr, "Source '%s' not found, skipping...\n", name) 91 | continue 92 | } 93 | wg.Add(1) 94 | go src.Process() 95 | } 96 | } 97 | 98 | wg.Wait() 99 | fmt.Fprintf(os.Stderr, "Got %d domains total\n", len(domains)) 100 | for domain, _ := range domains { 101 | fmt.Println(domain) 102 | } 103 | } 104 | -------------------------------------------------------------------------------- /genlist/sources.go: -------------------------------------------------------------------------------- 1 | // See LICENSE.txt for licensing information. 2 | 3 | package main 4 | 5 | import ( 6 | "regexp" 7 | ) 8 | 9 | var ( 10 | easylistRegexp *regexp.Regexp 11 | ) 12 | 13 | // put any extractor global stuff here. 14 | func init() { 15 | easylistRegexp = regexp.MustCompile("^\\|\\|((\\w+\\.)+[[:alpha:]]+)[^[:alpha:]]") 16 | } 17 | 18 | // sources defines an array of available blacklist sources. 19 | var sources = [...]*Source{ 20 | &Source{ 21 | Name: "pgl", 22 | Desc: "http://pgl.yoyo.org/adservers", 23 | URL: "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=%0A0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaint%0Aext", 24 | Extractor: func(line string) *string { 25 | return &line 26 | }, 27 | }, 28 | &Source{ 29 | Name: "easylist", 30 | Desc: "https://easylist.adblockplus.org", 31 | URL: "https://easylist-downloads.adblockplus.org/easylist.txt", 32 | Extractor: func(line string) *string { 33 | match := easylistRegexp.FindStringSubmatch(line) 34 | if match == nil { 35 | return nil 36 | } 37 | return &match[1] 38 | }, 39 | }, 40 | } 41 | --------------------------------------------------------------------------------