├── README.md ├── uploadr.ini └── uploadr.py /README.md: -------------------------------------------------------------------------------- 1 | flickr-uploader 2 | =============== 3 | 4 | Upload a directory of media to Flickr to use as a backup to your local storage. 5 | 6 | Interested in helping manage pull requests and issues? I need one or more collaborators since I no longer actively use this script. 7 | * trickortweak was kind enough to add me as a contributor to this repository. 8 | * I will be maintaining it mostly to answer queries and to incorporate pull requests/changes from other contributors. 9 | * I do more actively maintain this fork [flickr-uploader](https://github.com/oPromessa/flickr-uploader) also available [on pypi.org also](https://pypi.org/project/flickr-uploader/). 10 | 11 | 12 | ## Features: 13 | * Uploads images in full resolution to Flickr account (JPG, PNG...) 14 | * Reuploads modified images 15 | * Removes images from Flickr when they are removed from your local hard drive 16 | * Uploads videos (AVI, MOV, MPG, MP4, 3GP...) 17 | * Stores image information locally using a simple SQLite database 18 | * Creates "Sets" based on the folder name the media is in (getting existing sets from Flickr is managed also) 19 | * Ignores unwanted directories (like ".picasabackup" for Picasa users) 20 | * Allows specific files to be ignored (via regular expressions) 21 | * Convert RAW files (with an external tool) 22 | 23 | THIS SCRIPT IS PROVIDED WITH NO WARRANTY WHATSOEVER. PLEASE REVIEW THE SOURCE CODE TO MAKE SURE IT WILL WORK FOR YOUR NEEDS. IF YOU FIND A BUG, PLEASE REPORT IT. 24 | 25 | ## Requirements: 26 | 27 | * Python 2.7+ (for DSM use: Official package from Synology does not work for HTTPS so use package from SynoCommunity) 28 | * File write access (for the token and local database) 29 | * Flickr API key (free) 30 | 31 | ## Setup: 32 | Go to http://www.flickr.com/services/apps/create/apply and apply for an API key 33 | Edit the following variables in the uploadr.ini: 34 | 35 | * FILES_DIR = "YourDir" 36 | * FLICKR = { 37 | "title" : "", 38 | "description" : "", 39 | "tags" : "auto-upload", 40 | "is_public" : "0", 41 | "is_friend" : "0", 42 | "is_family" : "0", 43 | "api_key" : "Yourkey", 44 | "secret" : "YourSecret" 45 | } 46 | * FLICKR["api_key"] = "" 47 | * FLICKR["secret"] = "" 48 | 49 | Refer to https://www.flickr.com/services/api/upload.api.html for what each of the 50 | upload arguments above correspond to for Flickr's API. 51 | 52 | ## Usage 53 | Place the file uploadr.py in any directory and run via ssh (execution privs required): 54 | 55 | $ ./uploadr.py 56 | 57 | It will crawl through all the files from the FILES_DIR directory and begin the upload process. 58 | To check what files uploadr.py would upload and delete you can run the script withe option --dry-run: 59 | 60 | $ ./uploadr.py --dry-run 61 | 62 | In case you've changed the EXCLUDED_FOLDERS setting in your INI file and want to remove any previously 63 | uploaded files that are now ignored, run the script with the option --remove-ignored 64 | 65 | $ ./uploadr.py --remove-ignored 66 | 67 | ## Q&A 68 | * Q: Who is this script designed for? 69 | * A: Those people comfortable with the command line that want to backup their media on Flickr in full resolution. 70 | 71 | * Q: Why don't you use OAuth? 72 | * A: The older method is simpler to understand and works just as good. No need to fix what isn't broken. 73 | 74 | * Q: Are you a python ninja? 75 | * A: No, sorry. I just picked up the language to write this script because python can easily be installed on a Synology Diskstation. 76 | 77 | * Q: Is this script feature complete and fully tested? 78 | * A: Nope. It's a work in progress. I've tested it as needed for my needs, but it's possible to build additional features by contributing to the script. 79 | 80 | * Q: How to automate it with a Synology NAS ? 81 | * A: First you will need to run script at least one time in a ssh client to get the token file then with DSM 5, create an automate task, make it run once a day for example, and put this in the textbox without quotes "path_to_your_python_program path_to_your_script". For example, assuming you installed Python package from Synocommunity, command should look like "/usr/local/python/bin/python /volume1/script/flickr-uploader/uploadr.py". 82 | -------------------------------------------------------------------------------- /uploadr.ini: -------------------------------------------------------------------------------- 1 | [Config] 2 | ################################################################################ 3 | # Location to scan for new files 4 | ################################################################################ 5 | FILES_DIR = "YourDir" 6 | 7 | ################################################################################ 8 | # Flickr settings 9 | ################################################################################ 10 | # Set your own API key and secret message 11 | # Go to http://www.flickr.com/services/apps/create/apply and apply for an API key 12 | # 13 | FLICKR = { 14 | "title" : "", 15 | "description" : "", 16 | "tags" : "auto-upload", 17 | "is_public" : "0", 18 | "is_friend" : "0", 19 | "is_family" : "0", 20 | "api_key" : "YourKey", 21 | "secret" : "YourSecret" 22 | } 23 | 24 | ################################################################################ 25 | # How often to check for new files to upload (in seconds) 26 | ################################################################################ 27 | SLEEP_TIME = 1 * 60 28 | 29 | ################################################################################ 30 | # Only with --drip-feed option: 31 | # How often to wait between uploading individual files (in seconds) 32 | ################################################################################ 33 | DRIP_TIME = 1 * 60 34 | 35 | ################################################################################ 36 | # File we keep the history of uploaded files in. 37 | ################################################################################ 38 | DB_PATH = os.path.join(os.path.dirname(sys.argv[0]), "flickrdb") 39 | 40 | ################################################################################ 41 | # Location of file where we keep the lock for multiple running processes from happening 42 | ################################################################################ 43 | LOCK_PATH = os.path.join(os.path.dirname(sys.argv[0]), ".flickrlock") 44 | 45 | ################################################################################ 46 | # Location of file where we keep the tokenfile 47 | ################################################################################ 48 | TOKEN_PATH = os.path.join(os.path.dirname(sys.argv[0]), ".flickrToken") 49 | 50 | ################################################################################ 51 | # List of folder names you don't want to parse 52 | ################################################################################ 53 | EXCLUDED_FOLDERS = ["@eaDir","#recycle",".picasaoriginals","_ExcludeSync","Corel Auto-Preserve","Originals","Automatisch beibehalten von Corel"] 54 | 55 | ################################################################################ 56 | # List of filename regular expressions you wish to ignore 57 | # Regex is used to search the filename (as opposed to matching it completely) 58 | ################################################################################ 59 | IGNORED_REGEX = [] 60 | 61 | ################################################################################ 62 | # List of file extensions you agree to upload 63 | ################################################################################ 64 | ALLOWED_EXT = ["jpg","png","avi","mov","mpg","mp4","3gp"] 65 | 66 | ################################################################################ 67 | # RAW File Conversion (optional) 68 | ################################################################################ 69 | CONVERT_RAW_FILES = False 70 | RAW_EXT = ["3fr", "ari", "arw", "bay", "crw", "cr2", "cap", "dcs", "dcr", "dng", "drf", "eip", "erf", "fff", "iiq", "k25", "kdc", "mdc", "mef", "mos", "mrw", "nef", "nrw", "obm", "orf", "pef", "ptx", "pxn", "r3d", "raf", "raw", "rwl", "rw2", "rwz", "sr2", "srf", "srw", "x3f"] 71 | RAW_TOOL_PATH = "/volume1/photo/Image-ExifTool-9.69/" 72 | 73 | ################################################################################ 74 | # Files greater than this value won't be uploaded (1Mo = 1000000) 75 | ################################################################################ 76 | FILE_MAX_SIZE = 50000000 77 | 78 | ################################################################################ 79 | # Do you want to verify each time if already uploaded files have been changed? 80 | ################################################################################ 81 | MANAGE_CHANGES = True 82 | 83 | ################################################################################ 84 | # Full set name 85 | # Example: 86 | # FILES_DIR = /home/user/media 87 | # File to upload: /home/user/media/2014/05/05/photo.jpg 88 | # FULL_SET_NAME: 89 | # False: 05 90 | # True: 2014/05/05 91 | ################################################################################ 92 | FULL_SET_NAME = False 93 | 94 | ################################################################################ 95 | # Timeout for urlopen function 96 | ################################################################################ 97 | SOCKET_TIMEOUT = 60 98 | 99 | ################################################################################ 100 | # Counter for uploading, replacing attempts 101 | ################################################################################ 102 | MAX_UPLOAD_ATTEMPTS = 10 103 | -------------------------------------------------------------------------------- /uploadr.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | 5 | flickr-uploader designed for Synology Devices 6 | Upload a directory of media to Flickr to use as a backup to your local storage. 7 | 8 | Features: 9 | 10 | -Uploads both images and movies (JPG, PNG, GIF, AVI, MOV, 3GP files) 11 | -Stores image information locally using a simple SQLite database 12 | -Automatically creates "Sets" based on the folder name the media is in 13 | -Ignores ".picasabackup" directory 14 | -Automatically removes images from Flickr when they are removed from your local hard drive 15 | 16 | Requirements: 17 | 18 | -Python 2.7+ 19 | -File write access (for the token and local database) 20 | -Flickr API key (free) 21 | 22 | Setup: 23 | 24 | Go to http://www.flickr.com/services/apps/create/apply and apply for an API key Edit the following variables in the uploadr.ini 25 | 26 | FILES_DIR = "files/" 27 | FLICKR = { "api_key" : "", "secret" : "", "title" : "", "description" : "", "tags" : "auto-upload", "is_public" : "0", "is_friend" : "0", "is_family" : "1" } 28 | SLEEP_TIME = 1 * 60 29 | DRIP_TIME = 1 * 60 30 | DB_PATH = os.path.join(FILES_DIR, "fickerdb") 31 | Place the file uploadr.py in any directory and run: 32 | 33 | $ ./uploadr.py 34 | 35 | It will crawl through all the files from the FILES_DIR directory and begin the upload process. 36 | 37 | Upload files placed within a directory to your Flickr account. 38 | 39 | Inspired by: 40 | http://micampe.it/things/flickruploadr 41 | https://github.com/joelmx/flickrUploadr/blob/master/python3/uploadr.py 42 | 43 | Usage: 44 | 45 | cron entry (runs at the top of every hour ) 46 | 0 * * * * /full/path/to/uploadr.py > /dev/null 2>&1 47 | 48 | This code has been updated to use the new Auth API from flickr. 49 | 50 | You may use this code however you see fit in any form whatsoever. 51 | 52 | 53 | """ 54 | import httplib 55 | import sys 56 | import argparse 57 | import mimetools 58 | import mimetypes 59 | import os 60 | import time 61 | import urllib 62 | import urllib2 63 | import webbrowser 64 | import sqlite3 as lite 65 | import json 66 | from xml.dom.minidom import parse 67 | import hashlib 68 | try: 69 | # Use portalocker if available. Required for Windows systems 70 | import portalocker as FileLocker # noqa 71 | FILELOCK = FileLocker.lock 72 | except ImportError: 73 | # Use fcntl 74 | import fcntl as FileLocker 75 | FILELOCK = FileLocker.lockf 76 | import errno 77 | import subprocess 78 | import re 79 | import ConfigParser 80 | from multiprocessing.pool import ThreadPool 81 | 82 | if sys.version_info < (2, 7): 83 | sys.stderr.write("This script requires Python 2.7 or newer.\n") 84 | sys.stderr.write("Current version: " + sys.version + "\n") 85 | sys.stderr.flush() 86 | sys.exit(1) 87 | 88 | # 89 | # Read Config from config.ini file 90 | # 91 | 92 | config = ConfigParser.ConfigParser() 93 | config.read(os.path.join(os.path.dirname(sys.argv[0]), "uploadr.ini")) 94 | FILES_DIR = eval(config.get('Config', 'FILES_DIR')) 95 | FLICKR = eval(config.get('Config', 'FLICKR')) 96 | SLEEP_TIME = eval(config.get('Config', 'SLEEP_TIME')) 97 | DRIP_TIME = eval(config.get('Config', 'DRIP_TIME')) 98 | DB_PATH = eval(config.get('Config', 'DB_PATH')) 99 | LOCK_PATH = eval(config.get('Config', 'LOCK_PATH')) 100 | TOKEN_PATH = eval(config.get('Config', 'TOKEN_PATH')) 101 | EXCLUDED_FOLDERS = eval(config.get('Config', 'EXCLUDED_FOLDERS')) 102 | IGNORED_REGEX = [re.compile(regex) for regex in eval(config.get('Config', 'IGNORED_REGEX'))] 103 | ALLOWED_EXT = eval(config.get('Config', 'ALLOWED_EXT')) 104 | RAW_EXT = eval(config.get('Config', 'RAW_EXT')) 105 | FILE_MAX_SIZE = eval(config.get('Config', 'FILE_MAX_SIZE')) 106 | MANAGE_CHANGES = eval(config.get('Config', 'MANAGE_CHANGES')) 107 | RAW_TOOL_PATH = eval(config.get('Config', 'RAW_TOOL_PATH')) 108 | CONVERT_RAW_FILES = eval(config.get('Config', 'CONVERT_RAW_FILES')) 109 | FULL_SET_NAME = eval(config.get('Config', 'FULL_SET_NAME')) 110 | SOCKET_TIMEOUT = eval(config.get('Config', 'SOCKET_TIMEOUT')) 111 | MAX_UPLOAD_ATTEMPTS = eval(config.get('Config', 'MAX_UPLOAD_ATTEMPTS')) 112 | 113 | 114 | class APIConstants: 115 | """ APIConstants class 116 | """ 117 | 118 | base = "https://api.flickr.com/services/" 119 | rest = base + "rest/" 120 | auth = base + "auth/" 121 | upload = base + "upload/" 122 | replace = base + "replace/" 123 | 124 | def __init__(self): 125 | """ Constructor 126 | """ 127 | pass 128 | 129 | 130 | api = APIConstants() 131 | 132 | 133 | class Uploadr: 134 | """ Uploadr class 135 | """ 136 | 137 | token = None 138 | perms = "" 139 | 140 | def __init__(self): 141 | """ Constructor 142 | """ 143 | self.token = self.getCachedToken() 144 | 145 | 146 | 147 | def signCall(self, data): 148 | """ 149 | Signs args via md5 per http://www.flickr.com/services/api/auth.spec.html (Section 8) 150 | """ 151 | keys = data.keys() 152 | keys.sort() 153 | foo = "" 154 | for a in keys: 155 | foo += (a + data[a]) 156 | 157 | f = FLICKR["secret"] + "api_key" + FLICKR["api_key"] + foo 158 | # f = "api_key" + FLICKR[ "api_key" ] + foo 159 | 160 | return hashlib.md5(f).hexdigest() 161 | 162 | def urlGen(self, base, data, sig): 163 | """ urlGen 164 | """ 165 | data['api_key'] = FLICKR["api_key"] 166 | data['api_sig'] = sig 167 | encoded_url = base + "?" + urllib.urlencode(data) 168 | return encoded_url 169 | 170 | def authenticate(self): 171 | """ Authenticate user so we can upload files 172 | """ 173 | 174 | print("Getting new token") 175 | self.getFrob() 176 | self.getAuthKey() 177 | self.getToken() 178 | self.cacheToken() 179 | 180 | def getFrob(self): 181 | """ 182 | flickr.auth.getFrob 183 | 184 | Returns a frob to be used during authentication. This method call must be 185 | signed. 186 | 187 | This method does not require authentication. 188 | Arguments 189 | 190 | "api_key" (Required) 191 | Your API application key. See here for more details. 192 | """ 193 | 194 | d = { 195 | "method": "flickr.auth.getFrob", 196 | "format": "json", 197 | "nojsoncallback": "1" 198 | } 199 | sig = self.signCall(d) 200 | url = self.urlGen(api.rest, d, sig) 201 | try: 202 | response = self.getResponse(url) 203 | if (self.isGood(response)): 204 | FLICKR["frob"] = str(response["frob"]["_content"]) 205 | else: 206 | self.reportError(response) 207 | except: 208 | print("Error: cannot get frob:" + str(sys.exc_info())) 209 | 210 | def getAuthKey(self): 211 | """ 212 | Checks to see if the user has authenticated this application 213 | """ 214 | d = { 215 | "frob": FLICKR["frob"], 216 | "perms": "delete" 217 | } 218 | sig = self.signCall(d) 219 | url = self.urlGen(api.auth, d, sig) 220 | ans = "" 221 | try: 222 | webbrowser.open(url) 223 | print("Copy-paste following URL into a web browser and follow instructions:") 224 | print(url) 225 | ans = raw_input("Have you authenticated this application? (Y/N): ") 226 | except: 227 | print(str(sys.exc_info())) 228 | if (ans.lower() == "n"): 229 | print("You need to allow this program to access your Flickr site.") 230 | print("Copy-paste following URL into a web browser and follow instructions:") 231 | print(url) 232 | print("After you have allowed access restart uploadr.py") 233 | sys.exit() 234 | 235 | def getToken(self): 236 | """ 237 | http://www.flickr.com/services/api/flickr.auth.getToken.html 238 | 239 | flickr.auth.getToken 240 | 241 | Returns the auth token for the given frob, if one has been attached. This method call must be signed. 242 | Authentication 243 | 244 | This method does not require authentication. 245 | Arguments 246 | 247 | NTC: We need to store the token in a file so we can get it and then check it insted of 248 | getting a new on all the time. 249 | 250 | "api_key" (Required) 251 | Your API application key. See here for more details. 252 | frob (Required) 253 | The frob to check. 254 | """ 255 | 256 | d = { 257 | "method": "flickr.auth.getToken", 258 | "frob": str(FLICKR["frob"]), 259 | "format": "json", 260 | "nojsoncallback": "1" 261 | } 262 | sig = self.signCall(d) 263 | url = self.urlGen(api.rest, d, sig) 264 | try: 265 | res = self.getResponse(url) 266 | if (self.isGood(res)): 267 | self.token = str(res['auth']['token']['_content']) 268 | self.perms = str(res['auth']['perms']['_content']) 269 | self.cacheToken() 270 | else: 271 | self.reportError(res) 272 | except: 273 | print(str(sys.exc_info())) 274 | 275 | def getCachedToken(self): 276 | """ 277 | Attempts to get the flickr token from disk. 278 | """ 279 | if (os.path.exists(TOKEN_PATH)): 280 | return open(TOKEN_PATH).read() 281 | else: 282 | return None 283 | 284 | def cacheToken(self): 285 | """ cacheToken 286 | """ 287 | 288 | try: 289 | open(TOKEN_PATH, "w").write(str(self.token)) 290 | except: 291 | print("Issue writing token to local cache ", str(sys.exc_info())) 292 | 293 | def checkToken(self): 294 | """ 295 | flickr.auth.checkToken 296 | 297 | Returns the credentials attached to an authentication token. 298 | Authentication 299 | 300 | This method does not require authentication. 301 | Arguments 302 | 303 | "api_key" (Required) 304 | Your API application key. See here for more details. 305 | auth_token (Required) 306 | The authentication token to check. 307 | """ 308 | 309 | if (self.token == None): 310 | return False 311 | else: 312 | d = { 313 | "auth_token": str(self.token), 314 | "method": "flickr.auth.checkToken", 315 | "format": "json", 316 | "nojsoncallback": "1" 317 | } 318 | sig = self.signCall(d) 319 | 320 | url = self.urlGen(api.rest, d, sig) 321 | try: 322 | res = self.getResponse(url) 323 | if (self.isGood(res)): 324 | self.token = res['auth']['token']['_content'] 325 | self.perms = res['auth']['perms']['_content'] 326 | return True 327 | else: 328 | self.reportError(res) 329 | except: 330 | print(str(sys.exc_info())) 331 | return False 332 | 333 | def removeIgnoredMedia(self): 334 | print("*****Removing ignored files*****") 335 | 336 | if (not self.checkToken()): 337 | self.authenticate() 338 | con = lite.connect(DB_PATH) 339 | con.text_factory = str 340 | 341 | with con: 342 | cur = con.cursor() 343 | cur.execute("SELECT files_id, path FROM files") 344 | rows = cur.fetchall() 345 | 346 | for row in rows: 347 | if (self.isFileIgnored(row[1].decode('utf-8'))): 348 | success = self.deleteFile(row, cur) 349 | print("*****Completed ignored files*****") 350 | 351 | def removeDeletedMedia(self): 352 | """ Remove files deleted at the local source 353 | loop through database 354 | check if file exists 355 | if exists, continue 356 | if not exists, delete photo from fickr 357 | http://www.flickr.com/services/api/flickr.photos.delete.html 358 | """ 359 | 360 | print("*****Removing deleted files*****") 361 | 362 | if (not self.checkToken()): 363 | self.authenticate() 364 | con = lite.connect(DB_PATH) 365 | con.text_factory = str 366 | 367 | with con: 368 | cur = con.cursor() 369 | cur.execute("SELECT files_id, path FROM files") 370 | rows = cur.fetchall() 371 | 372 | for row in rows: 373 | if (not os.path.isfile(row[1].decode('utf-8'))): 374 | success = self.deleteFile(row, cur) 375 | print("*****Completed deleted files*****") 376 | 377 | def upload(self): 378 | """ upload 379 | """ 380 | 381 | print("*****Uploading files*****") 382 | 383 | allMedia = self.grabNewFiles() 384 | # If managing changes, consider all files 385 | if MANAGE_CHANGES: 386 | changedMedia = allMedia 387 | # If not, then get just the new and missing files 388 | else: 389 | con = lite.connect(DB_PATH) 390 | with con: 391 | cur = con.cursor() 392 | cur.execute("SELECT path FROM files") 393 | existingMedia = set(file[0] for file in cur.fetchall()) 394 | changedMedia = set(allMedia) - existingMedia 395 | 396 | changedMedia_count = len(changedMedia) 397 | print("Found " + str(changedMedia_count) + " files") 398 | 399 | 400 | if args.processes: 401 | pool = ThreadPool(processes=int(args.processes)) 402 | pool.map(self.uploadFile, changedMedia) 403 | else: 404 | count = 0 405 | for i, file in enumerate(changedMedia): 406 | success = self.uploadFile(file) 407 | if args.drip_feed and success and i != changedMedia_count - 1: 408 | print("Waiting " + str(DRIP_TIME) + " seconds before next upload") 409 | time.sleep(DRIP_TIME) 410 | count = count + 1; 411 | if (count % 100 == 0): 412 | print(" " + str(count) + " files processed (uploaded, md5ed or timestamp checked)") 413 | if (count % 100 > 0): 414 | print(" " + str(count) + " files processed (uploaded, md5ed or timestamp checked)") 415 | 416 | print("*****Completed uploading files*****") 417 | 418 | def convertRawFiles(self): 419 | """ convertRawFiles 420 | """ 421 | if (not CONVERT_RAW_FILES): 422 | return 423 | 424 | print "*****Converting files*****" 425 | for ext in RAW_EXT: 426 | print ("About to convert files with extension:" + ext + " files.") 427 | 428 | for dirpath, dirnames, filenames in os.walk(unicode(FILES_DIR), followlinks=True): 429 | if '.picasaoriginals' in dirnames: 430 | dirnames.remove('.picasaoriginals') 431 | if '@eaDir' in dirnames: 432 | dirnames.remove('@eaDir') 433 | for f in filenames: 434 | 435 | fileExt = f.split(".")[-1] 436 | filename = f.split(".")[0] 437 | if (fileExt.lower() == ext): 438 | 439 | if (not os.path.exists(dirpath + "/" + filename + ".JPG")): 440 | print("About to create JPG from raw " + dirpath + "/" + f) 441 | 442 | flag = "" 443 | if ext is "cr2": 444 | flag = "PreviewImage" 445 | else: 446 | flag = "JpgFromRaw" 447 | 448 | command = RAW_TOOL_PATH + "exiftool -b -" + flag + " -w .JPG -ext " + ext + " -r '" + dirpath + "/" + filename + "." + fileExt + "'" 449 | # print(command) 450 | 451 | p = subprocess.call(command, shell=True) 452 | 453 | if (not os.path.exists(dirpath + "/" + filename + ".JPG_original")): 454 | print ("About to copy tags from " + dirpath + "/" + f + " to JPG.") 455 | 456 | command = RAW_TOOL_PATH + "exiftool -tagsfromfile '" + dirpath + "/" + f + "' -r -all:all -ext JPG '" + dirpath + "/" + filename + ".JPG'" 457 | # print(command) 458 | 459 | p = subprocess.call(command, shell=True) 460 | 461 | print ("Finished copying tags.") 462 | 463 | print ("Finished converting files with extension:" + ext + ".") 464 | 465 | print "*****Completed converting files*****" 466 | 467 | def grabNewFiles(self): 468 | """ grabNewFiles 469 | """ 470 | 471 | files = [] 472 | for dirpath, dirnames, filenames in os.walk(unicode(FILES_DIR), followlinks=True): 473 | for f in filenames: 474 | filePath = os.path.join(dirpath, f) 475 | if self.isFileIgnored(filePath): 476 | continue 477 | if any(ignored.search(f) for ignored in IGNORED_REGEX): 478 | continue 479 | ext = os.path.splitext(os.path.basename(f))[1][1:].lower() 480 | if ext in ALLOWED_EXT: 481 | fileSize = os.path.getsize(dirpath + "/" + f) 482 | if (fileSize < FILE_MAX_SIZE): 483 | files.append(os.path.normpath(dirpath + "/" + f).replace("'", "\'")) 484 | else: 485 | print("Skipping file due to size restriction: " + (os.path.normpath(dirpath + "/" + f))) 486 | files.sort() 487 | return files 488 | 489 | def isFileIgnored(self, filename): 490 | for excluded_dir in EXCLUDED_FOLDERS: 491 | if excluded_dir in os.path.dirname(filename): 492 | return True 493 | 494 | return False 495 | 496 | def uploadFile(self, file): 497 | """ uploadFile 498 | """ 499 | 500 | if args.dry_run : 501 | print("Dry Run Uploading " + file + "...") 502 | return True 503 | 504 | success = False 505 | con = lite.connect(DB_PATH) 506 | con.text_factory = str 507 | with con: 508 | cur = con.cursor() 509 | cur.execute("SELECT rowid,files_id,path,set_id,md5,tagged,last_modified FROM files WHERE path = ?", (file,)) 510 | row = cur.fetchone() 511 | 512 | last_modified = os.stat(file).st_mtime; 513 | if row is None: 514 | print("Uploading " + file + "...") 515 | 516 | if FULL_SET_NAME: 517 | setName = os.path.relpath(os.path.dirname(file), FILES_DIR) 518 | else: 519 | head, setName = os.path.split(os.path.dirname(file)) 520 | try: 521 | photo = ('photo', file.encode('utf-8'), open(file, 'rb').read()) 522 | if args.title: # Replace 523 | FLICKR["title"] = args.title 524 | if args.description: # Replace 525 | FLICKR["description"] = args.description 526 | if args.tags: # Append 527 | FLICKR["tags"] += " " 528 | 529 | file_checksum = self.md5Checksum(file) 530 | d = { 531 | "auth_token": str(self.token), 532 | "perms": str(self.perms), 533 | "title": str(FLICKR["title"]), 534 | "description": str(FLICKR["description"]), 535 | # replace commas to avoid tags conflicts 536 | "tags": '{} {} checksum:{}'.format(FLICKR["tags"], setName.encode('utf-8'), file_checksum).replace(',', ''), 537 | "is_public": str(FLICKR["is_public"]), 538 | "is_friend": str(FLICKR["is_friend"]), 539 | "is_family": str(FLICKR["is_family"]) 540 | } 541 | sig = self.signCall(d) 542 | d["api_sig"] = sig 543 | d["api_key"] = FLICKR["api_key"] 544 | url = self.build_request(api.upload, d, (photo,)) 545 | 546 | res = None 547 | search_result = None 548 | for x in range(0, MAX_UPLOAD_ATTEMPTS): 549 | try: 550 | res = parse(urllib2.urlopen(url, timeout=SOCKET_TIMEOUT)) 551 | search_result = None 552 | break 553 | except (IOError, httplib.HTTPException): 554 | print(str(sys.exc_info())) 555 | print("Check is file already uploaded") 556 | time.sleep(5) 557 | 558 | search_result = self.photos_search(file_checksum) 559 | if search_result["stat"] != "ok": 560 | raise IOError(search_result) 561 | 562 | if int(search_result["photos"]["total"]) == 0: 563 | if x == MAX_UPLOAD_ATTEMPTS - 1: 564 | raise ValueError("Reached maximum number of attempts to upload, skipping") 565 | 566 | print("Not found, reuploading") 567 | continue 568 | 569 | if int(search_result["photos"]["total"]) > 1: 570 | raise IOError("More then one file with same checksum, collisions? " + search_result) 571 | 572 | if int(search_result["photos"]["total"]) == 1: 573 | break 574 | 575 | if not search_result and res.documentElement.attributes['stat'].value != "ok": 576 | print("A problem occurred while attempting to upload the file: " + file) 577 | raise IOError(str(res.toxml())) 578 | 579 | print("Successfully uploaded the file: " + file) 580 | 581 | if search_result: 582 | file_id = int(search_result["photos"]["photo"][0]["id"]) 583 | else: 584 | file_id = int(str(res.getElementsByTagName('photoid')[0].firstChild.nodeValue)) 585 | 586 | # Add to db 587 | cur.execute( 588 | 'INSERT INTO files (files_id, path, md5, last_modified, tagged) VALUES (?, ?, ?, ?, 1)', 589 | (file_id, file, file_checksum, last_modified)) 590 | success = True 591 | except: 592 | print(str(sys.exc_info())) 593 | elif (MANAGE_CHANGES): 594 | if (row[6] == None): 595 | cur.execute('UPDATE files SET last_modified = ? WHERE files_id = ?', (last_modified, row[1])) 596 | con.commit() 597 | if (row[6] != last_modified): 598 | fileMd5 = self.md5Checksum(file) 599 | if (fileMd5 != str(row[4])): 600 | self.replacePhoto(file, row[1], row[4], fileMd5, last_modified, cur, con); 601 | return success 602 | 603 | def replacePhoto(self, file, file_id, oldFileMd5, fileMd5, last_modified, cur, con): 604 | 605 | if args.dry_run : 606 | print("Dry Run Replace file " + file + "...") 607 | return True 608 | 609 | success = False 610 | print("Replacing the file: " + file + "...") 611 | try: 612 | photo = ('photo', file.encode('utf-8'), open(file, 'rb').read()) 613 | 614 | d = { 615 | "auth_token": str(self.token), 616 | "photo_id": str(file_id) 617 | } 618 | sig = self.signCall(d) 619 | d["api_sig"] = sig 620 | d["api_key"] = FLICKR["api_key"] 621 | url = self.build_request(api.replace, d, (photo,)) 622 | 623 | res = None 624 | res_add_tag = None 625 | res_get_info = None 626 | 627 | for x in range(0, MAX_UPLOAD_ATTEMPTS): 628 | try: 629 | res = parse(urllib2.urlopen(url, timeout=SOCKET_TIMEOUT)) 630 | if res.documentElement.attributes['stat'].value == "ok": 631 | res_add_tag = self.photos_add_tags(file_id, ['checksum:{}'.format(fileMd5)]) 632 | if res_add_tag['stat'] == 'ok': 633 | res_get_info = flick.photos_get_info(file_id) 634 | if res_get_info['stat'] == 'ok': 635 | tag_id = None 636 | for tag in res_get_info['photo']['tags']['tag']: 637 | if tag['raw'] == 'checksum:{}'.format(oldFileMd5): 638 | tag_id = tag['id'] 639 | break 640 | if not tag_id: 641 | print("Can't find tag {} for file {}".format(tag_id, file_id)) 642 | break 643 | else: 644 | self.photos_remove_tag(tag_id) 645 | break 646 | except (IOError, ValueError, httplib.HTTPException): 647 | print(str(sys.exc_info())) 648 | print("Replacing again") 649 | time.sleep(5) 650 | 651 | if x == MAX_UPLOAD_ATTEMPTS - 1: 652 | raise ValueError("Reached maximum number of attempts to replace, skipping") 653 | continue 654 | 655 | if res.documentElement.attributes['stat'].value != "ok" \ 656 | or res_add_tag['stat'] != 'ok' \ 657 | or res_get_info['stat'] != 'ok': 658 | print("A problem occurred while attempting to upload the file: " + file) 659 | 660 | if res.documentElement.attributes['stat'].value != "ok": 661 | raise IOError(str(res.toxml())) 662 | 663 | if res_add_tag['stat'] != 'ok': 664 | raise IOError(res_add_tag) 665 | 666 | if res_get_info['stat'] != 'ok': 667 | raise IOError(res_get_info) 668 | 669 | print("Successfully replaced the file: " + file) 670 | 671 | # Add to set 672 | cur.execute('UPDATE files SET md5 = ?,last_modified = ? WHERE files_id = ?', 673 | (fileMd5, last_modified, file_id)) 674 | con.commit() 675 | success = True 676 | except: 677 | print(str(sys.exc_info())) 678 | 679 | return success 680 | 681 | def deleteFile(self, file, cur): 682 | 683 | if args.dry_run : 684 | print("Deleting file: " + file[1].decode('utf-8')) 685 | return True 686 | 687 | success = False 688 | print("Deleting file: " + file[1].decode('utf-8')) 689 | 690 | try: 691 | d = { 692 | # FIXME: double format? 693 | "auth_token": str(self.token), 694 | "perms": str(self.perms), 695 | "format": "rest", 696 | "method": "flickr.photos.delete", 697 | "photo_id": str(file[0]), 698 | "format": "json", 699 | "nojsoncallback": "1" 700 | } 701 | sig = self.signCall(d) 702 | url = self.urlGen(api.rest, d, sig) 703 | res = self.getResponse(url) 704 | if (self.isGood(res)): 705 | 706 | # Find out if the file is the last item in a set, if so, remove the set from the local db 707 | cur.execute("SELECT set_id FROM files WHERE files_id = ?", (file[0],)) 708 | row = cur.fetchone() 709 | cur.execute("SELECT set_id FROM files WHERE set_id = ?", (row[0],)) 710 | rows = cur.fetchall() 711 | if (len(rows) == 1): 712 | print("File is the last of the set, deleting the set ID: " + str(row[0])) 713 | cur.execute("DELETE FROM sets WHERE set_id = ?", (row[0],)) 714 | 715 | # Delete file record from the local db 716 | cur.execute("DELETE FROM files WHERE files_id = ?", (file[0],)) 717 | print("Successful deletion.") 718 | success = True 719 | else: 720 | if (res['code'] == 1): 721 | # File already removed from Flicker 722 | cur.execute("DELETE FROM files WHERE files_id = ?", (file[0],)) 723 | else: 724 | self.reportError(res) 725 | except: 726 | # If you get 'attempt to write a readonly database', set 'admin' as owner of the DB file (fickerdb) and 'users' as group 727 | print(str(sys.exc_info())) 728 | return success 729 | 730 | def logSetCreation(self, setId, setName, primaryPhotoId, cur, con): 731 | print("adding set to log: " + setName.decode('utf-8')) 732 | 733 | success = False 734 | cur.execute("INSERT INTO sets (set_id, name, primary_photo_id) VALUES (?,?,?)", 735 | (setId, setName, primaryPhotoId)) 736 | cur.execute("UPDATE files SET set_id = ? WHERE files_id = ?", (setId, primaryPhotoId)) 737 | con.commit() 738 | return True 739 | 740 | def build_request(self, theurl, fields, files, txheaders=None): 741 | """ 742 | build_request/encode_multipart_formdata code is from www.voidspace.org.uk/atlantibots/pythonutils.html 743 | 744 | Given the fields to set and the files to encode it returns a fully formed urllib2.Request object. 745 | You can optionally pass in additional headers to encode into the opject. (Content-type and Content-length will be overridden if they are set). 746 | fields is a sequence of (name, value) elements for regular form fields - or a dictionary. 747 | files is a sequence of (name, filename, value) elements for data to be uploaded as files. 748 | """ 749 | 750 | content_type, body = self.encode_multipart_formdata(fields, files) 751 | if not txheaders: txheaders = {} 752 | txheaders['Content-type'] = content_type 753 | txheaders['Content-length'] = str(len(body)) 754 | 755 | return urllib2.Request(theurl, body, txheaders) 756 | 757 | def encode_multipart_formdata(self, fields, files, BOUNDARY='-----' + mimetools.choose_boundary() + '-----'): 758 | """ Encodes fields and files for uploading. 759 | fields is a sequence of (name, value) elements for regular form fields - or a dictionary. 760 | files is a sequence of (name, filename, value) elements for data to be uploaded as files. 761 | Return (content_type, body) ready for urllib2.Request instance 762 | You can optionally pass in a boundary string to use or we'll let mimetools provide one. 763 | """ 764 | 765 | CRLF = '\r\n' 766 | L = [] 767 | if isinstance(fields, dict): 768 | fields = fields.items() 769 | for (key, value) in fields: 770 | L.append('--' + BOUNDARY) 771 | L.append('Content-Disposition: form-data; name="%s"' % key) 772 | L.append('') 773 | L.append(value) 774 | for (key, filename, value) in files: 775 | filetype = mimetypes.guess_type(filename)[0] or 'application/octet-stream' 776 | L.append('--' + BOUNDARY) 777 | L.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename)) 778 | L.append('Content-Type: %s' % filetype) 779 | L.append('') 780 | L.append(value) 781 | L.append('--' + BOUNDARY + '--') 782 | L.append('') 783 | body = CRLF.join(L) 784 | content_type = 'multipart/form-data; boundary=%s' % BOUNDARY # XXX what if no files are encoded 785 | return content_type, body 786 | 787 | def isGood(self, res): 788 | """ isGood 789 | """ 790 | 791 | if (not res == "" and res['stat'] == "ok"): 792 | return True 793 | else: 794 | return False 795 | 796 | def reportError(self, res): 797 | """ reportError 798 | """ 799 | 800 | try: 801 | print("Error: " + str(res['code'] + " " + res['message'])) 802 | except: 803 | print("Error: " + str(res)) 804 | 805 | def getResponse(self, url): 806 | """ 807 | Send the url and get a response. Let errors float up 808 | """ 809 | res = None 810 | try: 811 | res = urllib2.urlopen(url, timeout=SOCKET_TIMEOUT).read() 812 | except urllib2.HTTPError, e: 813 | print(e.code) 814 | except urllib2.URLError, e: 815 | print(e.args) 816 | return json.loads(res, encoding='utf-8') 817 | 818 | def run(self): 819 | """ run 820 | """ 821 | 822 | while (True): 823 | self.upload() 824 | print("Last check: " + str(time.asctime(time.localtime()))) 825 | time.sleep(SLEEP_TIME) 826 | 827 | def createSets(self): 828 | 829 | print('*****Creating Sets*****') 830 | 831 | if args.dry_run : 832 | return True 833 | 834 | 835 | con = lite.connect(DB_PATH) 836 | con.text_factory = str 837 | with con: 838 | 839 | cur = con.cursor() 840 | cur.execute("SELECT files_id, path, set_id FROM files") 841 | 842 | files = cur.fetchall() 843 | 844 | for row in files: 845 | if FULL_SET_NAME: 846 | setName = os.path.relpath(os.path.dirname(row[1]), FILES_DIR) 847 | else: 848 | head, setName = os.path.split(os.path.dirname(row[1])) 849 | newSetCreated = False 850 | 851 | cur.execute("SELECT set_id, name FROM sets WHERE name = ?", (setName,)) 852 | 853 | set = cur.fetchone() 854 | 855 | if set == None: 856 | setId = self.createSet(setName, row[0], cur, con) 857 | print("Created the set: " + setName.decode('utf-8')) 858 | newSetCreated = True 859 | else: 860 | setId = set[0] 861 | 862 | if row[2] == None and newSetCreated == False: 863 | print("adding file to set " + row[1].decode('utf-8')) 864 | self.addFileToSet(setId, row, cur) 865 | print('*****Completed creating sets*****') 866 | 867 | def addFileToSet(self, setId, file, cur): 868 | 869 | if args.dry_run : 870 | return True 871 | 872 | try: 873 | d = { 874 | "auth_token": str(self.token), 875 | "perms": str(self.perms), 876 | "format": "json", 877 | "nojsoncallback": "1", 878 | "method": "flickr.photosets.addPhoto", 879 | "photoset_id": str(setId), 880 | "photo_id": str(file[0]) 881 | } 882 | sig = self.signCall(d) 883 | url = self.urlGen(api.rest, d, sig) 884 | 885 | res = self.getResponse(url) 886 | if (self.isGood(res)): 887 | 888 | print("Successfully added file " + str(file[1]) + " to its set.") 889 | 890 | cur.execute("UPDATE files SET set_id = ? WHERE files_id = ?", (setId, file[0])) 891 | 892 | else: 893 | if (res['code'] == 1): 894 | print("Photoset not found, creating new set...") 895 | if FULL_SET_NAME: 896 | setName = os.path.relpath(os.path.dirname(file[1]), FILES_DIR) 897 | else: 898 | head, setName = os.path.split(os.path.dirname(file[1])) 899 | con = lite.connect(DB_PATH) 900 | con.text_factory = str 901 | self.createSet(setName, file[0], cur, con) 902 | elif (res['code'] == 3): 903 | print(res['message'] + "... updating DB") 904 | cur.execute("UPDATE files SET set_id = ? WHERE files_id = ?", (setId, file[0])) 905 | else: 906 | self.reportError(res) 907 | except: 908 | print(str(sys.exc_info())) 909 | 910 | def createSet(self, setName, primaryPhotoId, cur, con): 911 | print("Creating new set: " + setName.decode('utf-8')) 912 | 913 | if args.dry_run : 914 | return True 915 | 916 | 917 | try: 918 | d = { 919 | "auth_token": str(self.token), 920 | "perms": str(self.perms), 921 | "format": "json", 922 | "nojsoncallback": "1", 923 | "method": "flickr.photosets.create", 924 | "primary_photo_id": str(primaryPhotoId), 925 | "title": setName 926 | 927 | } 928 | 929 | sig = self.signCall(d) 930 | 931 | url = self.urlGen(api.rest, d, sig) 932 | res = self.getResponse(url) 933 | if (self.isGood(res)): 934 | self.logSetCreation(res["photoset"]["id"], setName, primaryPhotoId, cur, con) 935 | return res["photoset"]["id"] 936 | else: 937 | print(d) 938 | self.reportError(res) 939 | except: 940 | print(str(sys.exc_info())) 941 | return False 942 | 943 | def setupDB(self): 944 | print("Setting up the database: " + DB_PATH) 945 | con = None 946 | try: 947 | con = lite.connect(DB_PATH) 948 | con.text_factory = str 949 | cur = con.cursor() 950 | cur.execute('CREATE TABLE IF NOT EXISTS files (files_id INT, path TEXT, set_id INT, md5 TEXT, tagged INT)') 951 | cur.execute('CREATE TABLE IF NOT EXISTS sets (set_id INT, name TEXT, primary_photo_id INTEGER)') 952 | cur.execute('CREATE UNIQUE INDEX IF NOT EXISTS fileindex ON files (path)') 953 | cur.execute('CREATE INDEX IF NOT EXISTS setsindex ON sets (name)') 954 | con.commit() 955 | cur = con.cursor() 956 | cur.execute('PRAGMA user_version') 957 | row = cur.fetchone() 958 | if (row[0] == 0): 959 | print('Adding last_modified column to database'); 960 | cur = con.cursor() 961 | cur.execute('PRAGMA user_version="1"') 962 | cur.execute('ALTER TABLE files ADD COLUMN last_modified REAL'); 963 | con.commit() 964 | con.close() 965 | except lite.Error, e: 966 | print("Error: %s" % e.args[0]) 967 | if con != None: 968 | con.close() 969 | sys.exit(1) 970 | finally: 971 | print("Completed database setup") 972 | 973 | def md5Checksum(self, filePath): 974 | with open(filePath, 'rb') as fh: 975 | m = hashlib.md5() 976 | while True: 977 | data = fh.read(8192) 978 | if not data: 979 | break 980 | m.update(data) 981 | return m.hexdigest() 982 | 983 | # Method to clean unused sets 984 | def removeUselessSetsTable(self): 985 | print('*****Removing empty Sets from DB*****') 986 | if args.dry_run : 987 | return True 988 | 989 | 990 | con = lite.connect(DB_PATH) 991 | con.text_factory = str 992 | with con: 993 | cur = con.cursor() 994 | cur.execute("SELECT set_id, name FROM sets WHERE set_id NOT IN (SELECT set_id FROM files)") 995 | unusedsets = cur.fetchall() 996 | 997 | for row in unusedsets: 998 | print("Unused set spotted about to be deleted: " + str(row[0]) + " (" + row[1].decode('utf-8') + ")") 999 | cur.execute("DELETE FROM sets WHERE set_id = ?", (row[0],)) 1000 | con.commit() 1001 | 1002 | print('*****Completed removing empty Sets from DB*****') 1003 | 1004 | # Display Sets 1005 | def displaySets(self): 1006 | con = lite.connect(DB_PATH) 1007 | con.text_factory = str 1008 | with con: 1009 | cur = con.cursor() 1010 | cur.execute("SELECT set_id, name FROM sets") 1011 | allsets = cur.fetchall() 1012 | for row in allsets: 1013 | print("Set: " + str(row[0]) + "(" + row[1] + ")") 1014 | 1015 | # Get sets from Flickr 1016 | def getFlickrSets(self): 1017 | print('*****Adding Flickr Sets to DB*****') 1018 | if args.dry_run : 1019 | return True 1020 | 1021 | con = lite.connect(DB_PATH) 1022 | con.text_factory = str 1023 | try: 1024 | d = { 1025 | "auth_token": str(self.token), 1026 | "perms": str(self.perms), 1027 | "format": "json", 1028 | "nojsoncallback": "1", 1029 | "method": "flickr.photosets.getList" 1030 | } 1031 | url = self.urlGen(api.rest, d, self.signCall(d)) 1032 | res = self.getResponse(url) 1033 | if (self.isGood(res)): 1034 | cur = con.cursor() 1035 | for row in res['photosets']['photoset']: 1036 | setId = row['id'] 1037 | setName = row['title']['_content'] 1038 | primaryPhotoId = row['primary'] 1039 | cur.execute("SELECT set_id FROM sets WHERE set_id = '" + setId + "'") 1040 | foundSets = cur.fetchone() 1041 | if foundSets == None: 1042 | print(u"Adding set #{0} ({1}) with primary photo #{2}".format(setId, setName, primaryPhotoId)) 1043 | cur.execute("INSERT INTO sets (set_id, name, primary_photo_id) VALUES (?,?,?)", 1044 | (setId, setName, primaryPhotoId)) 1045 | con.commit() 1046 | con.close() 1047 | else: 1048 | print(d) 1049 | self.reportError(res) 1050 | except: 1051 | print(str(sys.exc_info())) 1052 | print('*****Completed adding Flickr Sets to DB*****') 1053 | 1054 | def photos_search(self, checksum): 1055 | data = { 1056 | "auth_token": str(self.token), 1057 | "perms": str(self.perms), 1058 | "format": "json", 1059 | "nojsoncallback": "1", 1060 | "method": "flickr.photos.search", 1061 | "user_id": "me", 1062 | "tags": 'checksum:{}'.format(checksum), 1063 | } 1064 | 1065 | url = self.urlGen(api.rest, data, self.signCall(data)) 1066 | return self.getResponse(url) 1067 | 1068 | def people_get_photos(self): 1069 | data = { 1070 | "auth_token": str(self.token), 1071 | "perms": str(self.perms), 1072 | "format": "json", 1073 | "nojsoncallback": "1", 1074 | "user_id": "me", 1075 | "method": "flickr.people.getPhotos", 1076 | "per_page": "1" 1077 | } 1078 | 1079 | url = self.urlGen(api.rest, data, self.signCall(data)) 1080 | return self.getResponse(url) 1081 | 1082 | def photos_get_not_in_set(self): 1083 | data = { 1084 | "auth_token": str(self.token), 1085 | "perms": str(self.perms), 1086 | "format": "json", 1087 | "nojsoncallback": "1", 1088 | "method": "flickr.photos.getNotInSet", 1089 | "per_page": "1" 1090 | } 1091 | 1092 | url = self.urlGen(api.rest, data, self.signCall(data)) 1093 | return self.getResponse(url) 1094 | 1095 | def photos_add_tags(self, photo_id, tags): 1096 | tags = [tag.replace(',', '') for tag in tags] 1097 | data = { 1098 | "auth_token": str(self.token), 1099 | "perms": str(self.perms), 1100 | "format": "json", 1101 | "nojsoncallback": "1", 1102 | "method": "flickr.photos.addTags", 1103 | "photo_id": str(photo_id), 1104 | "tags": ','.join(tags) 1105 | } 1106 | 1107 | url = self.urlGen(api.rest, data, self.signCall(data)) 1108 | return self.getResponse(url) 1109 | 1110 | def photos_get_info(self, photo_id): 1111 | data = { 1112 | "auth_token": str(self.token), 1113 | "perms": str(self.perms), 1114 | "format": "json", 1115 | "nojsoncallback": "1", 1116 | "method": "flickr.photos.getInfo", 1117 | "photo_id": str(photo_id), 1118 | } 1119 | 1120 | url = self.urlGen(api.rest, data, self.signCall(data)) 1121 | return self.getResponse(url) 1122 | 1123 | def photos_remove_tag(self, tag_id): 1124 | data = { 1125 | "auth_token": str(self.token), 1126 | "perms": str(self.perms), 1127 | "format": "json", 1128 | "nojsoncallback": "1", 1129 | "method": "flickr.photos.removeTag", 1130 | "tag_id": str(tag_id), 1131 | } 1132 | 1133 | url = self.urlGen(api.rest, data, self.signCall(data)) 1134 | return self.getResponse(url) 1135 | 1136 | def print_stat(self): 1137 | con = lite.connect(DB_PATH) 1138 | con.text_factory = str 1139 | with con: 1140 | cur = con.cursor() 1141 | cur.execute("SELECT Count(*) FROM files") 1142 | 1143 | print 'Total photos on local: {}'.format(cur.fetchone()[0]) 1144 | 1145 | res = self.people_get_photos() 1146 | if res["stat"] != "ok": 1147 | raise IOError(res) 1148 | print 'Total photos on flickr: {}'.format(res["photos"]["total"]) 1149 | 1150 | res = self.photos_get_not_in_set() 1151 | if res["stat"] != "ok": 1152 | raise IOError(res) 1153 | print 'Photos not in sets on flickr: {}'.format(res["photos"]["total"]) 1154 | 1155 | 1156 | print("--------- Start time: " + time.strftime("%c") + " ---------") 1157 | if __name__ == "__main__": 1158 | # Ensure that only one instance of this script is running 1159 | try: 1160 | # FileLocker is an alias to portalocker (if available) or fcntl 1161 | FILELOCK(open(LOCK_PATH, 'w'), 1162 | FileLocker.LOCK_EX | FileLocker.LOCK_NB) 1163 | except IOError as err: 1164 | if err.errno == errno.EAGAIN: 1165 | sys.stderr.write('[%s] Script already running.\n' % time.strftime('%c')) 1166 | sys.exit(-1) 1167 | raise 1168 | finally: 1169 | pass 1170 | parser = argparse.ArgumentParser(description='Upload files to Flickr.') 1171 | parser.add_argument('-d', '--daemon', action='store_true', 1172 | help='Run forever as a daemon') 1173 | parser.add_argument('-i', '--title', action='store', 1174 | help='Title for uploaded files') 1175 | parser.add_argument('-e', '--description', action='store', 1176 | help='Description for uploaded files') 1177 | parser.add_argument('-t', '--tags', action='store', 1178 | help='Space-separated tags for uploaded files') 1179 | parser.add_argument('-r', '--drip-feed', action='store_true', 1180 | help='Wait a bit between uploading individual files') 1181 | parser.add_argument('-p', '--processes', 1182 | help='Number of photos to upload simultaneously') 1183 | parser.add_argument('-n', '--dry-run', action='store_true', 1184 | help='Dry run') 1185 | parser.add_argument('-g', '--remove-ignored', action='store_true', 1186 | help='Remove previously uploaded files, now ignored') 1187 | args = parser.parse_args() 1188 | print args.dry_run 1189 | 1190 | flick = Uploadr() 1191 | 1192 | if FILES_DIR == "": 1193 | print("Please configure the name of the folder in the script with media available to sync with Flickr.") 1194 | sys.exit() 1195 | 1196 | if FLICKR["api_key"] == "" or FLICKR["secret"] == "": 1197 | print("Please enter an API key and secret in the script file (see README).") 1198 | sys.exit() 1199 | 1200 | flick.setupDB() 1201 | 1202 | if args.daemon: 1203 | flick.run() 1204 | else: 1205 | if not flick.checkToken(): 1206 | flick.authenticate() 1207 | # flick.displaySets() 1208 | 1209 | flick.removeUselessSetsTable() 1210 | flick.getFlickrSets() 1211 | flick.convertRawFiles() 1212 | flick.upload() 1213 | flick.removeDeletedMedia() 1214 | if args.remove_ignored: 1215 | flick.removeIgnoredMedia() 1216 | flick.createSets() 1217 | flick.print_stat() 1218 | 1219 | 1220 | print("--------- End time: " + time.strftime("%c") + " ---------") 1221 | --------------------------------------------------------------------------------