├── README.md └── PyPhDB.py /README.md: -------------------------------------------------------------------------------- 1 | # PyPhDB 2 | This script has been created to give you the ability to quickly edit your **Adlists**, **Whitelist (Exact)**, **Whitelist (Regex)**, **Blacklist**, **Blacklist (Regex)** from flat-files (pre v5.0 behaviour). 3 | 4 | ### Requirements ### 5 | * Python **3.6+** is required to run this script. 6 | * Validators Python module (for adlist url validation) 7 | 8 | #### Validators installation (required) #### 9 | 1. `sudo apt-get install python3-pip` 10 | 2. `sudo pip3 install validators` 11 | 12 | ### Docker ### 13 | This script should work with Pi-hole being run within a docker container. In order to interact with your docker container, you must use the `--docker` switch **and** specify your Pi-hole directory volume using `--directory` 14 | 15 | #### Example: #### 16 | 17 | `curl -sSl https://raw.githubusercontent.com/mmotti/PyPhDB/master/PyPhDB.py | sudo python3 - --dump --docker --directory '/your/pihole/directory'` 18 | 19 | ### How to use ### 20 | 21 | There are three main steps to this script: 22 | 23 | #### dump #### 24 | Dump the items from your Pi-hole DB to **/etc/pihole/PyPhDB** 25 | 26 | `curl -sSl https://raw.githubusercontent.com/mmotti/PyPhDB/master/PyPhDB.py | sudo python3 - --dump` 27 | 28 | #### upload #### 29 | Upload the changes to your Pi-hole DB 30 | 31 | `curl -sSl https://raw.githubusercontent.com/mmotti/PyPhDB/master/PyPhDB.py | sudo python3 - --upload` 32 | 33 | #### clean #### 34 | Optional: Remove the **/etc/pihole/PyPhDB** directory. 35 | 36 | `curl -sSl https://raw.githubusercontent.com/mmotti/PyPhDB/master/PyPhDB.py | sudo python3 - --clean` 37 | 38 | ### Further Information ### 39 | 40 | This script will export the following files to **/etc/pihole/PyPhDB**: **adlists.list** (adlists), **whitelist.list** (exact whitelist), **blacklist.list** (exact blacklist), **whitelist_regex.list** (regex whitelist), **regex.list** (regex blacklist) and **gravity.list** (gravity domains) 41 | 42 | Adlist urls, exact blacklist / whitelist domains and the regexps are validated before being allowed to enter the DB. 43 | 44 | **gravity.list** is excluded from the upload process due to the way the entries are stored. This is dumped only for diagnostic purposes. 45 | 46 | During the upload process, the script uses **INSERT OR IGNORE** to avoid issues caused by IDs changing etc. The script also determines entires that exist locally, but not in the database and will carefully remove them accordingly. 47 | -------------------------------------------------------------------------------- /PyPhDB.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import re 4 | import shutil 5 | import subprocess 6 | import sys 7 | import sqlite3 8 | import validators 9 | 10 | 11 | class PyPhDB: 12 | 13 | def __init__(self, ph_dir='/etc/pihole/'): 14 | 15 | self.path_pihole_dir = '/etc/pihole/' if not ph_dir else os.path.expanduser(ph_dir) 16 | self.path_pihole_db = os.path.join(self.path_pihole_dir, 'gravity.db') 17 | self.path_output_dir = os.path.join(self.path_pihole_dir, 'PyPhDB') 18 | self.comment = 'Imported by PyPhDB' 19 | self.connection = None 20 | self.cursor = None 21 | 22 | self.set_adlists = set() 23 | self.set_blacklist = set() 24 | self.set_whitelist = set() 25 | self.set_bl_regexps = set() 26 | self.set_wl_regexps = set() 27 | self.set_gravity = set() 28 | 29 | def access_check(self): 30 | 31 | if os.path.exists(self.path_pihole_dir): 32 | print('[i] Pi-hole directory located') 33 | if os.access(self.path_pihole_dir, os.X_OK | os.W_OK): 34 | print('[i] Write access is available to Pi-hole directory') 35 | # Does the DB exist 36 | # and is the file size greater than 0 bytes 37 | if os.path.isfile(self.path_pihole_db) and os.path.getsize(self.path_pihole_db) > 0: 38 | print('[i] Pi-hole DB located') 39 | return True 40 | else: 41 | print('[e] Write access is available but the Pi-hole DB does not exist') 42 | return False 43 | else: 44 | print('[e] Write access is not available to the Pi-hole directory.') 45 | return False 46 | else: 47 | print(f'[e] {self.path_pihole_dir} does not exist.') 48 | return False 49 | 50 | def make_connection(self): 51 | 52 | try: 53 | self.connection = sqlite3.connect(self.path_pihole_db) 54 | except sqlite3.Error as e: 55 | print('[e] Failed to connected to Pi-hole DB') 56 | return False 57 | 58 | print('[i] Connection established to Pi-hole DB') 59 | 60 | self.cursor = self.connection.cursor() 61 | 62 | return True 63 | 64 | def close_connection(self): 65 | 66 | print('[i] Closing connection to the Pi-hole DB') 67 | self.connection.close() 68 | 69 | def fetch_data(self): 70 | 71 | # adlists.list 72 | print('[i] Fetching adlists') 73 | self.cursor.execute('SELECT address FROM adlist') 74 | self.set_adlists.update(x[0] for x in self.cursor.fetchall()) 75 | 76 | # whitelist.list 77 | print('[i] Fetching whitelist') 78 | self.cursor.execute('SELECT domain FROM domainlist WHERE type = 0') 79 | self.set_whitelist.update(x[0] for x in self.cursor.fetchall()) 80 | 81 | # blacklist.list 82 | print('[i] Fetching blacklist') 83 | self.cursor.execute('SELECT domain FROM domainlist WHERE type = 1') 84 | self.set_blacklist.update(x[0] for x in self.cursor.fetchall()) 85 | 86 | # whitelist_regex.list 87 | print('[i] Fetching whitelist regexps') 88 | self.cursor.execute('SELECT domain FROM domainlist WHERE type = 2') 89 | self.set_wl_regexps.update(x[0] for x in self.cursor.fetchall()) 90 | 91 | # regex.list 92 | print('[i] Fetching blacklist regexps') 93 | self.cursor.execute('SELECT domain FROM domainlist WHERE type = 3') 94 | self.set_bl_regexps.update(x[0] for x in self.cursor.fetchall()) 95 | 96 | # gravity.list 97 | print('[i] Fetching gravity domains') 98 | self.cursor.execute('SELECT distinct(domain) FROM gravity') 99 | self.set_gravity.update(x[0] for x in self.cursor.fetchall()) 100 | 101 | def stage_output(self): 102 | 103 | # Create /etc/pihole/PyPhDB 104 | if not os.path.exists(self.path_output_dir): 105 | print('[i] Creating output directory') 106 | os.mkdir(self.path_output_dir) 107 | 108 | def dump_data(self): 109 | 110 | self.stage_output() 111 | 112 | # Create a dictionary for easier output 113 | dict_output = { 114 | 'adlists.list': self.set_adlists, 115 | 'whitelist.list': self.set_whitelist, 116 | 'blacklist.list': self.set_blacklist, 117 | 'whitelist_regex.list': self.set_wl_regexps, 118 | 'regex.list': self.set_bl_regexps, 119 | 'gravity.list': self.set_gravity 120 | } 121 | 122 | # Iterate through dictionary 123 | for k, v in dict_output.items(): 124 | # Form the file path 125 | path_file = os.path.join(self.path_output_dir, k) 126 | 127 | print(f'[i] {k}:') 128 | print(f' --> Outputting {len(v)} lines to {path_file}') 129 | 130 | with open(path_file, 'w') as fWrite: 131 | for line in sorted(v): 132 | fWrite.write(f'{line}\n') 133 | 134 | def upload_files(self): 135 | 136 | # Create a dictionary for easier output 137 | dict_upload = { 138 | 'adlists.list': self.set_adlists, 139 | 'whitelist.list': self.set_whitelist, 140 | 'blacklist.list': self.set_blacklist, 141 | 'whitelist_regex.list': self.set_wl_regexps, 142 | 'regex.list': self.set_bl_regexps 143 | } 144 | 145 | # Insert or IGNORE 146 | # Delete specifics 147 | # Delete all of type (if file cleared) with exception to adlists 148 | 149 | dict_sql = { 150 | 'adlists.list': 151 | 'INSERT OR IGNORE INTO adlist (address, comment) VALUES (?, ?)|\ 152 | DELETE FROM adlist WHERE address IN (?)', 153 | 'whitelist.list': 154 | 'INSERT OR IGNORE INTO domainlist (type, domain, enabled, comment) VALUES (0, ?, 1, ?)|\ 155 | DELETE FROM domainlist WHERE domain IN (?) AND type = 0|\ 156 | DELETE FROM domainlist WHERE type = 0', 157 | 'blacklist.list': 158 | 'INSERT OR IGNORE INTO domainlist (type, domain, enabled, comment) VALUES (1, ?, 1, ?)|\ 159 | DELETE FROM domainlist WHERE domain IN (?) AND type = 1|\ 160 | DELETE FROM domainlist WHERE type = 1', 161 | 'whitelist_regex.list': 162 | 'INSERT OR IGNORE INTO domainlist (type, domain, enabled, comment) VALUES (2, ?, 1, ?)|\ 163 | DELETE FROM domainlist WHERE domain IN (?) AND type = 2|\ 164 | DELETE FROM domainlist WHERE type = 2', 165 | 'regex.list': 166 | 'INSERT OR IGNORE INTO domainlist (type, domain, enabled, comment) VALUES (3, ?, 1, ?)|\ 167 | DELETE FROM domainlist WHERE domain IN (?) AND type = 3|\ 168 | DELETE FROM domainlist WHERE type = 3' 169 | } 170 | 171 | # Determine how each list needs to be validated 172 | validators_adlist = {'adlists.list'} 173 | validators_domain = {'whitelist.list', 'blacklist.list'} 174 | validators_regexps = {'whitelist_regex.list', 'regex.list'} 175 | 176 | # For each upload item (list) 177 | for k, v in dict_upload.items(): 178 | print(f'[i] Processing {k}') 179 | # Construct full file path 180 | path_file = os.path.join(self.path_output_dir, k) 181 | # Check if the file exists 182 | if os.path.isfile(path_file): 183 | # Create a new set to store changes 184 | set_modified = set() 185 | set_removal = set() 186 | # Read the file in the output directory to a set 187 | with open(path_file, 'r', encoding='utf-8', errors='ignore') as fOpen: 188 | # Generator for selecting only non-empty lines / non-commented lines 189 | lines = (x for x in map(str.strip, fOpen) if x and x[:1] != '#') 190 | # Use appropriate validation when reading from the files 191 | if k in validators_adlist: 192 | # For each url 193 | for line in lines: 194 | # If it's a valid URL 195 | if validators.url(line): 196 | # Add to the set 197 | set_modified.add(line) 198 | elif k in validators_domain: 199 | # For each domain 200 | for line in lines: 201 | # If it's a valid domain 202 | if validators.domain(line): 203 | # Add to the set 204 | set_modified.add(line) 205 | elif k in validators_regexps: 206 | # For each regexp 207 | for line in lines: 208 | try: 209 | # Try to compile the regexp (test if valid) 210 | re.compile(line) 211 | # If valid, add to set 212 | set_modified.add(line) 213 | except re.error: 214 | # If invalid, skip to next 215 | continue 216 | 217 | # If the set was populated 218 | if set_modified: 219 | # Check if it's identical to DB 220 | if set_modified == v: 221 | print(' --> No Changes') 222 | else: 223 | print(' --> Updating DB') 224 | # Update or Ignore 225 | self.cursor.executemany(dict_sql[k].split('|')[0], [(x, self.comment) for x in set_modified]) 226 | # Find items that are in the DB but not in the modified files (for removal from db) 227 | set_removal.update(x for x in v if x not in set_modified) 228 | # If there are items to remove from the DB 229 | if set_removal: 230 | self.cursor.executemany(dict_sql[k].split('|')[1], [(x,) for x in set_removal]) 231 | # If the file has been emptied 232 | else: 233 | # Check whether the DB is already empty or not 234 | if set_modified == v: 235 | print(' --> No Changes') 236 | continue 237 | # Check if we've got a preset query to remove all 238 | try: 239 | sql_remove_all = dict_sql[k].split('|')[2] 240 | except IndexError as e: 241 | continue 242 | # If we do, run it 243 | if sql_remove_all: 244 | print(' --> Updating DB') 245 | self.cursor.execute(dict_sql[k].split('|')[2]) 246 | else: 247 | print(' --> Local file does not exist') 248 | 249 | self.connection.commit() 250 | 251 | def clean_dump(self): 252 | 253 | if os.path.exists(self.path_output_dir): 254 | print('[i] Removing output directory') 255 | shutil.rmtree(self.path_output_dir) 256 | else: 257 | print('[i] Output directory does not exist') 258 | 259 | 260 | def restart_pihole(docker=False): 261 | 262 | # Form the command to restart Pi-hole 263 | cmd = ['pihole', 'restartdns', 'reload'] 264 | 265 | # If it's running in a docker container 266 | if docker: 267 | # Prepend list with docker commands 268 | cmd[0:0] = ['docker', 'exec'] 269 | 270 | print('[i] Restarting Pi-hole') 271 | 272 | # Try to run the reset command 273 | try: 274 | subprocess.call(cmd, stdout=subprocess.DEVNULL) 275 | except OSError as e: 276 | print(f'[e] Restart failed: {e}') 277 | exit(1) 278 | 279 | 280 | # Create a new argument parser 281 | parser = argparse.ArgumentParser() 282 | # Create mutual exclusion groups 283 | group_action = parser.add_mutually_exclusive_group() 284 | # Dump flag 285 | group_action.add_argument('-d', '--dump', help='Export elements of the Pi-hole DB', action='store_true') 286 | # Upload flag 287 | group_action.add_argument('-u', '--upload', help='Import text files to the Pi-hole DB', action='store_true') 288 | # Clean flag 289 | group_action.add_argument('-c', '--clean', help='Clean (remove) the output directory', action='store_true') 290 | # Add options group 291 | group_options = parser.add_argument_group() 292 | # Docker flag 293 | group_options.add_argument('-dc', '--docker', help='Indicate that Pi-hole is being ran within a docker container', 294 | action='store_true') 295 | # Pi-hole DIR 296 | group_options.add_argument('-dir', '--directory', help='Specify Pi-hole Directory') 297 | # Parse arguments 298 | args = parser.parse_args() 299 | 300 | # If no arguments were passed 301 | if not len(sys.argv) > 1: 302 | print('[i] No script arguments detected - Defaulted to DUMP') 303 | # Default to dump mode 304 | args.dump = True 305 | 306 | # If the docker flag is enabled and no directory is specified 307 | if args.docker and args.directory is None: 308 | parser.error('[e] If --docker is specified, you must also specify your Pi-hole volume location using --directory') 309 | 310 | # Create a new instance 311 | PyPhDB_inst = PyPhDB(ph_dir=args.directory) 312 | 313 | # Access check for DB 314 | if PyPhDB_inst.access_check(): 315 | # If the clean flag is enabled 316 | if args.clean: 317 | PyPhDB_inst.clean_dump() 318 | exit() 319 | # If we're able to access the DB 320 | if PyPhDB_inst.make_connection(): 321 | # Populate sets with data from DB 322 | PyPhDB_inst.fetch_data() 323 | # If the dump flag is enabled 324 | if args.dump: 325 | # Close the connection to the DB 326 | PyPhDB_inst.close_connection() 327 | # Dump data to disk 328 | PyPhDB_inst.dump_data() 329 | # If the upload flag is enabled 330 | elif args.upload: 331 | PyPhDB_inst.upload_files() 332 | # Close the connection to the DB 333 | PyPhDB_inst.close_connection() 334 | # Restart Pi-hole 335 | restart_pihole(docker=args.docker) 336 | else: 337 | exit(1) 338 | else: 339 | exit(1) 340 | --------------------------------------------------------------------------------