├── .gitignore ├── .travis.yml ├── README.md ├── build_rules.py └── test_build_rules.py /.gitignore: -------------------------------------------------------------------------------- 1 | /build_rules.pyc 2 | /test_build_rules.pyc 3 | /blacklist.conf 4 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: '2.7' 3 | install: python build_rules.py 4 | script: python -m unittest test_build_rules 5 | 6 | deploy: 7 | provider: releases 8 | api_key: 9 | secure: BUNMxIJpjaWu9Yx2kwbY+cg3taY50oWiJEFaESHdpdgDoNU2Uzkkq+6wL3Gp6YUwXNmo/jSexdiDpwwGI0ouiHDFejLlnwAonY/k6Ae3UyutRxzSD+ehsbdDA60WaDhOowevg+dACuDrkPRK8/V+NjL1iaN6PVg5C7b3VXK1/fGb4erP6vtbevYDW0gkZgnDfLK6s2JySuv0c46eiw1nzhsXdC/0yGR7B0JGyOppmoQGonM3RTb/7uB4M0HIS5Xo+rOnbhG3qfC/0uw/1os69e+92nPcJoZbNR0dW7z4yoVNcfe0V9Kp5RH9bQ5TXpZrlCOlrzCLS3kHUAUyl40N8eHjoofmCdo41ud9+1hi/dTDRy/n84MN84tD9VkIulIsDSHQudsg3rZA7J8lEB7iMmky1CDnt5NoHmZNokO6M1YZLUrai15G7vFO5P+wNzBLJy2chDwK6h7LhX6qINGCw8QDhj5vZM4i3of9Dd8gUrz0UXORnPXMTz+h05KrxE33WTPsXqqS5wrn+V8yTPNK3rESawA11c6v1apr9Mmg/mfsdpsFSRQYyXpTlsxfCYP4va2uoYaq69ffmZ/qLOQmpiXpQFwqdZrJuV6yUHPBI5oSsYeegSIOWb+Iasew9C9X/+DhInIAHpj1wgcjVrX3SzXTAmyB4BZURp9OESk6hHc= 10 | file: blacklist.conf 11 | skip_cleanup: true 12 | on: 13 | repo: ndfred/unifi-pi-hole 14 | 15 | before_deploy: 16 | - git config --local user.name "Frédéric Sagnes" 17 | - git config --local user.email "frederic.sagnes@gmail.com" 18 | - git tag "v0.2.$TRAVIS_BUILD_NUMBER" 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Unifi Pi-hole 2 | 3 | Get the [Pi-hole](https://github.com/pi-hole/pi-hole) project working on my [Unifi Security Gateway](https://www.ui.com/unifi-routing/usg/). Inspired by the [blacklist project](https://github.com/britannic/blacklist). 4 | 5 | ## Installing 6 | 7 | SSH into your USG and run these commands: 8 | 9 | configure 10 | delete service dns forwarding blacklist 11 | set service dns forwarding blacklist dns-redirect-ip 0.0.0.0 12 | 13 | edit service dns forwarding blacklist domains source unifi-pi-hole 14 | set url https://github.com/ndfred/unifi-pi-hole/releases/download/v0.1.0/domains.txt 15 | set description "Consolidated domains list from https://github.com/ndfred/unifi-pi-hole" 16 | set prefix 0.0.0.0 17 | top 18 | 19 | edit service dns forwarding blacklist hosts source unifi-pi-hole 20 | set url https://github.com/ndfred/unifi-pi-hole/releases/download/v0.1.0/hosts.txt 21 | set description "Consolidated hosts list from https://github.com/ndfred/unifi-pi-hole" 22 | set prefix 0.0.0.0 23 | top 24 | 25 | commit; save; exit 26 | 27 | I have yet to work on refreshing the list automatically, but this should get you started. 28 | 29 | ## Building the hosts list 30 | 31 | Clone the repo and run the `build_rules.py` script to download and parse the rules files, and generate the `hosts.txt` and `domains.txt` files: 32 | 33 | $ python build_rules.py 34 | Parsing https://hosts-file.net/grm.txt 35 | Parsing https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts 36 | Parsing https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts 37 | [...] 38 | Parsing https://zerodot1.gitlab.io/CoinBlockerLists/hosts 39 | Wrote 735586 host names in hosts.txt and domains.txt 40 | 41 | You can then publish the files on a web server or copy them directly to your [Unifi Security Gateway](https://www.ui.com/unifi-routing/usg/). 42 | 43 | The lists come from the [Firebog website](https://firebog.net), which backs the [Pi-hole setup script](https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh), and aggregates all the safe lists. I might support more advances lists with whitelisting in the future. 44 | 45 | ## Testing [![Build Status](https://travis-ci.com/ndfred/unifi-pi-hole.svg?branch=master)](https://travis-ci.com/ndfred/unifi-pi-hole/) 46 | 47 | Just run the test script: 48 | 49 | $ python test_build_rules.py 50 | ........... 51 | ---------------------------------------------------------------------- 52 | Ran 11 tests in 0.186s 53 | 54 | OK 55 | 56 | ## Configuration reference 57 | 58 | I SSH-ed into my USG, put myself in configuration mode, and queried completion suggestions to get to the documentation: 59 | 60 | # set service dns forwarding blacklist 61 | Possible completions: 62 | disabled Option to disable blacklisting 63 | dns-redirect-ip 64 | Global redirect IP address for hosts and domains (zones) 65 | domains Configure DNS forwarding blacklist DOMAINS 66 | exclude domains to GLOBALLY EXCLUDE from DNS forwarding domains and hosts blacklist 67 | hosts Configure DNS forwarding blacklist hosts (must be fully qualified domain names) 68 | 69 | # set service dns forwarding blacklist domains 70 | Possible completions: 71 | dns-redirect-ip 72 | Blackhole IP address for domains 73 | exclude Domains to EXCLUDE from DNS forwarding blacklist 74 | include Domains to INCLUDE in the DNS forwarding blacklist 75 | source Blacklisted domains source name 76 | 77 | # set service dns forwarding blacklist hosts 78 | Possible completions: 79 | dns-redirect-ip 80 | Blackhole IP address for hosts - overrides global blackhole IP 81 | exclude Hosts to EXCLUDE from DNS forwarding blacklist 82 | include Hosts to INCLUDE in the DNS forwarding blacklist 83 | source Blacklisted hosts source name 84 | 85 | # set service dns forwarding blacklist domains source unifi-pi-hole 86 | Possible completions: 87 | description Blacklist domain source description 88 | dns-redirect-ip 89 | Blackhole IP address for a domain source - overrides global blackhole IP 90 | file A path and filename that provides a list of domains to blacklist, e.g. /config/user-data/hacked_domains.txt 91 | prefix Prefix string must include all text before the domain name 92 | url A blacklist source url that provides a list of domain names to block 93 | 94 | # set service dns forwarding blacklist exclude 95 | Possible completions: 96 | domains to GLOBALLY EXCLUDE from DNS forwarding domains and hosts blacklist 97 | -------------------------------------------------------------------------------- /build_rules.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import sys 5 | import urllib2 6 | import re 7 | 8 | # See appendToListsFile in https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh 9 | AD_LISTS = [ 10 | ('StevenBlack\'s Unified Hosts List', 'https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'), 11 | # ('MalwareDomains', 'https://mirror1.malwaredomains.com/files/justdomains'), 12 | ('Cameleon', 'http://sysctl.org/cameleon/hosts'), 13 | # ('ZeusTracker', 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist'), 14 | ('Disconnect.me Tracking', 'https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt'), 15 | ('Disconnect.me Ads', 'https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt'), 16 | # ('Hosts-file.net Ads', 'https://hosts-file.net/ad_servers.txt'), 17 | ] 18 | # See https://v.firebog.net/hosts/lists.php 19 | FIREBOG_CONSERVATIVE_URLS_LIST = 'https://v.firebog.net/hosts/lists.php?type=tick' 20 | DOMAIN_EXPR = re.compile(r'^[a-zA-Z0-9\.\-_]+$') 21 | ZERO_IP_PREFIXES = ('0.0.0.0 ', '127.0.0.1 ', '0 ', ':: ') 22 | INVALID_DOMAINS = frozenset(('localhost', '0.0.0.0')) 23 | OUTPUT_BLACKLIST_PATH = 'blacklist.conf' 24 | DOMAIN_EXTENSIONS_URL = 'https://publicsuffix.org/list/public_suffix_list.dat' 25 | DOMAIN_EXTENSIONS = None 26 | 27 | def get_domain_extensions(): 28 | global DOMAIN_EXTENSIONS 29 | 30 | if DOMAIN_EXTENSIONS == None: 31 | DOMAIN_EXTENSIONS = frozenset(parse_host_file(DOMAIN_EXTENSIONS_URL)) 32 | 33 | return DOMAIN_EXTENSIONS 34 | 35 | def download_file(url): 36 | request = urllib2.Request(url) 37 | # Needed to bypass Cloudflare's bot detection 38 | request.add_header('User-Agent', 'curl/7.54.0') 39 | 40 | return urllib2.urlopen(request).read() 41 | 42 | def download_ads_list_urls(url): 43 | return [(list_url, list_url) for list_url in download_file(url).split('\n') if list_url] 44 | 45 | def cleanup_domain_line(line): 46 | if '#' in line: 47 | line = line[:line.index('#')] 48 | 49 | line = line.strip().replace('\t ', ' ').replace('\t', ' ') 50 | 51 | return line 52 | 53 | def is_domain(domain): 54 | if not '.' in domain: 55 | return True 56 | 57 | extension = domain[domain.index('.') + 1:] 58 | 59 | if extension in get_domain_extensions(): 60 | return True 61 | 62 | return False 63 | 64 | def parse_domain_line(line): 65 | original_line = line 66 | line = cleanup_domain_line(line) 67 | domain = None 68 | 69 | if DOMAIN_EXPR.match(line): 70 | domain = line 71 | else: 72 | for prefix in ZERO_IP_PREFIXES: 73 | if line.startswith(prefix): 74 | domain = line[len(prefix):].strip() 75 | 76 | if domain in INVALID_DOMAINS: 77 | domain = None 78 | 79 | # Uncomment to visually debug unmatched lines and make sure we parse all hosts 80 | # if original_line and domain is None and not original_line.startswith('#'): 81 | # print original_line.decode('utf8') 82 | 83 | return domain 84 | 85 | def parse_host_file(url): 86 | found_domains = False 87 | 88 | for line in download_file(url).split('\n'): 89 | domain = parse_domain_line(line) 90 | 91 | if domain: 92 | found_domains = True 93 | yield domain 94 | 95 | if not found_domains: 96 | raise Exception('Couldn\'t find any domains in URL %s' % url) 97 | 98 | def remove_duplicate_domains(domains): 99 | top_level_domains = set() 100 | top_level_domains_suffixes = set() 101 | hosts = set() 102 | filtered_hosts = set() 103 | 104 | for domain in domains: 105 | if is_domain(domain): 106 | top_level_domains.add(domain) 107 | top_level_domains_suffixes.add('.%s' % domain) 108 | else: 109 | hosts.add(domain) 110 | 111 | top_level_domains_suffixes = tuple(top_level_domains_suffixes) 112 | 113 | for host in hosts: 114 | if not host.endswith(top_level_domains_suffixes): 115 | filtered_hosts.add(host) 116 | 117 | return sorted(list(top_level_domains.union(filtered_hosts))) 118 | 119 | def output_hosts(ads_lists_ulrs=FIREBOG_CONSERVATIVE_URLS_LIST, output_blacklist_path=OUTPUT_BLACKLIST_PATH): 120 | # ads_lists = download_ads_list_urls(ads_lists_ulrs) 121 | ads_lists = AD_LISTS 122 | domains = [] 123 | 124 | for name, url in ads_lists: 125 | print 'Parsing %s' % name 126 | domains += parse_host_file(url) 127 | 128 | domains = remove_duplicate_domains(domains) 129 | 130 | with open(output_blacklist_path, 'w') as blacklist_file: 131 | for domain in domains: 132 | blacklist_file.write('server=/%s/\n' % domain) 133 | 134 | print 'Wrote %d host names in %s' % (len(domains), output_blacklist_path) 135 | 136 | return 0 137 | 138 | def main(): 139 | output_hosts() 140 | 141 | return 0 142 | 143 | if __name__ == '__main__': 144 | sys.exit(main()) 145 | -------------------------------------------------------------------------------- /test_build_rules.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | from build_rules import * 5 | import unittest 6 | 7 | class TestDomainParsing(unittest.TestCase): 8 | def test_domain_expr(self): 9 | self.assertFalse(DOMAIN_EXPR.match('Hello world')) 10 | self.assertFalse(DOMAIN_EXPR.match('https://www.google.com')) 11 | self.assertTrue(DOMAIN_EXPR.match('google.com')) 12 | self.assertTrue(DOMAIN_EXPR.match('my-google.com')) 13 | self.assertTrue(DOMAIN_EXPR.match('sub.my-google.com')) 14 | self.assertTrue(DOMAIN_EXPR.match('sub.google.com')) 15 | self.assertTrue(DOMAIN_EXPR.match('927697--storno-sicher-konto_identity.sicherheitsvorbeugung-schutz.cf')) 16 | 17 | def test_cleanup_domain_line(self): 18 | self.assertEqual(cleanup_domain_line(' '), '') 19 | self.assertEqual(cleanup_domain_line('# Comment'), '') 20 | self.assertEqual(cleanup_domain_line('domain.com # Comment'), 'domain.com') 21 | self.assertEqual(cleanup_domain_line(' domain.com # Comment'), 'domain.com') 22 | self.assertEqual(cleanup_domain_line('0.0.0.0 domain.com # Comment'), '0.0.0.0 domain.com') 23 | self.assertEqual(cleanup_domain_line('0.0.0.0\t domain.com # Comment'), '0.0.0.0 domain.com') 24 | 25 | def test_StevenBlack(self): 26 | self.assertIsNone(parse_domain_line('# This hosts file is a merged collection of hosts from reputable sources,')) 27 | self.assertIsNone(parse_domain_line('# ===============================================================')) 28 | self.assertIsNone(parse_domain_line('127.0.0.1 localhost')) 29 | self.assertIsNone(parse_domain_line('::1 localhost')) 30 | self.assertIsNone(parse_domain_line('0.0.0.0 0.0.0.0')) 31 | self.assertEqual(parse_domain_line('0.0.0.0 1493361689.rsc.cdn77.org'), '1493361689.rsc.cdn77.org') 32 | self.assertEqual(parse_domain_line('0.0.0.0 123greetings.com # contains one link to distributor of adware or spyware'), '123greetings.com') 33 | 34 | def test_MalwareDomains(self): 35 | self.assertIsNone(parse_domain_line('')) 36 | self.assertEqual(parse_domain_line('amazon.co.uk.security-check.ga'), 'amazon.co.uk.security-check.ga') 37 | 38 | def test_Cameleon(self): 39 | self.assertIsNone(parse_domain_line('# Last updated : 2018-03-17')) 40 | self.assertIsNone(parse_domain_line('127.0.0.1 localhost')) 41 | self.assertEqual(parse_domain_line('127.0.0.1 0.r.msn.com'), '0.r.msn.com') 42 | self.assertEqual(parse_domain_line('127.0.0.1 0.r.msn.com'), '0.r.msn.com') 43 | 44 | def test_ZeusTracker(self): 45 | self.assertIsNone(parse_domain_line('##############################################################################')) 46 | self.assertEqual(parse_domain_line('039b1ee.netsolhost.com'), '039b1ee.netsolhost.com') 47 | 48 | def test_Disconnect(self): 49 | self.assertIsNone(parse_domain_line('# Basic tracking list by Disconnect')) 50 | self.assertEqual(parse_domain_line('bango.combango.org'), 'bango.combango.org') 51 | 52 | def test_Hostsfile(self): 53 | self.assertIsNone(parse_domain_line('# hpHosts - Ad and Tracking servers only')) 54 | self.assertIsNone(parse_domain_line('# Hosts: 45737')) 55 | self.assertEqual(parse_domain_line('127.0.0.1 005.free-counter.co.uk'), '005.free-counter.co.uk') 56 | self.assertEqual(parse_domain_line('127.0.0.1 118d654612df63bc8395-aecfeaabe29a34ea9a877711ec6d8aed.r37.cf2.rackcdn.com'), '118d654612df63bc8395-aecfeaabe29a34ea9a877711ec6d8aed.r37.cf2.rackcdn.com') 57 | 58 | def test_get_domain_extensions(self): 59 | domain_extensions = get_domain_extensions() 60 | self.assertTrue('com' in domain_extensions) 61 | self.assertTrue('co.uk' in domain_extensions) 62 | 63 | def test_is_domain(self): 64 | self.assertTrue(is_domain('google.com')) 65 | self.assertFalse(is_domain('www.google.com')) 66 | self.assertTrue(is_domain('bbc.co.uk')) 67 | 68 | def test_remove_duplicate_domains(self): 69 | self.assertEqual(remove_duplicate_domains(['domain.com', 'sub.domain.com']), ['domain.com']) 70 | self.assertEqual(remove_duplicate_domains(['domain.com', 'otherdomain.com']), ['domain.com', 'otherdomain.com']) 71 | 72 | if __name__ == '__main__': 73 | sys.exit(unittest.main()) 74 | --------------------------------------------------------------------------------