├── .gitignore ├── CONTRIBUTING.md ├── LICENSE.txt ├── README.md ├── code ├── repo_sync ├── repoclean ├── reposadolib │ ├── __init__.py │ └── reposadocommon.py └── repoutil ├── docs ├── URL_rewrites.md ├── client_configuration.md ├── getting_started.md ├── reference.md ├── reposado_metadata.md ├── reposado_operation.md ├── reposado_preferences.md └── reposado_py2exe.md ├── other └── reposado.jpg └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | code/preferences.plist 3 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | Walt Disney Animation Studios welcomes contributions to reposado! 2 | 3 | ### Contributor License Agreement 4 | All contributors must complete and sign a Contributor License Agreement as discussed [here](https://www.technology.disneyanimation.com/collaboration-through-sharing). A copy can be found [here](https://docs.wixstatic.com/ugd/a2be3a_c129b5feb37444fc9c52f3ce0714b200.pdf). 5 | 6 | ### Preferred contribution methods 7 | #### Bug fixes 8 | Bug fixes should be submitted as Pull Requests. 9 | 10 | #### Enhancements/New features 11 | Before starting work on a new feature or enhancement, consider bringing your idea up for discussion on the [reposado mailing list](http://groups.google.com/group/reposado). This might save you a lot of wasted effort. Once you have code for a new feature complete, push it to your own GitHub clone. You can then ask others to test it first, or proceed directly to creating a Pull Request. 12 | 13 | Learn more about cloning a GitHub repository [here](https://help.github.com/articles/cloning-a-repository/). 14 | Learn more about GitHub pull requests [here](https://help.github.com/articles/about-pull-requests/). 15 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright 2011 Disney Enterprises, Inc. All rights reserved 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are 5 | met: 6 | 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright 11 | notice, this list of conditions and the following disclaimer in 12 | the documentation and/or other materials provided with the 13 | distribution. 14 | 15 | * The names "Disney", "Walt Disney Pictures", "Walt Disney Animation 16 | Studios" or the names of its contributors may NOT be used to 17 | endorse or promote products derived from this software without 18 | specific prior written permission from Walt Disney Pictures. 19 | 20 | Disclaimer: THIS SOFTWARE IS PROVIDED BY WALT DISNEY PICTURES AND 21 | CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, 22 | BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS 23 | FOR A PARTICULAR PURPOSE, NONINFRINGEMENT AND TITLE ARE DISCLAIMED. 24 | IN NO EVENT SHALL WALT DISNEY PICTURES, THE COPYRIGHT HOLDER OR 25 | CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 26 | EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 27 | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 28 | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND BASED ON ANY 29 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 30 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 31 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 32 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | **macOS Big Sur important information** 2 | In macOS Big Sur, Apple has removed the ability for `softwareupdate` to be pointed to a non-Apple sucatalog. This means you cannot use a Reposado server to serve Apple software updates to Big Sur (and presumably later versions of macOS) clients. 3 | 4 | **INTRODUCTION** 5 | 6 | Reposado is a set of tools written in Python that replicate the key functionality of Mac OS X Server's Software Update Service. 7 | 8 | **LICENSE** 9 | 10 | Reposado is licensed under the new BSD license. 11 | 12 | **DISCUSSION GROUP** 13 | 14 | Discussion for users and developers of Reposado is [here.](http://groups.google.com/group/reposado) 15 | 16 | **FEATURES AND CAPABILITIES** 17 | 18 | Reposado, together with Python, the "curl" binary tool and a web server such as Apache 2, enables you to host a local Apple Software Update Server on any hardware and OS of your choice. 19 | 20 | Reposado contains a tool (repo_sync) to download Software Update catalogs and (optionally) update packages from Apple's servers, enabling you to host them from a local web server. 21 | 22 | Additionally, Reposado provides a command-line tool (repoutil) that enables you to create any arbitrary number of "branches" of the Apple catalogs. These branches can contain any subset of the available updates. For example, one could create "testing" and "release" branches, and then set some clients to use the "testing" branch catalog to test newly-released updates. You would set most of your clients to use the "release" branch catalog, which would contain updates that had been through the testing process. 23 | 24 | If you configure Reposado to also download the actual updates as well as the catalogs, you can continue to offer updates that have been superseded by more recent updates. For example, if you are currently offering the 10.6.7 updates to your clients, and Apple releases a 10.6.8 update, you can continue to offer the (deprecated) 10.6.7 update until you are ready to release the newer update to your clients. You can even offer the 10.6.7 update to your "release" clients while offering the 10.6.8 update to your "testing" clients. Offering "deprecated" Apple Software Updates is a feature that is difficult with Apple's tools. 25 | 26 | **LIMITATIONS AND DEPENDENCIES** 27 | 28 | Apple's Software Update Service does a few things. Primarily, it replicates software updates from Apple's servers, downloading them to a local machine. Secondly, it functions as a web server to actually serve these updates to client machines. Reposado does not duplicate the web server portion of Apple's Software Update Service. Instead you may use any existing web server you wish. 29 | 30 | Reposado also currently relies on the command-line "curl" binary to download updates from Apple's servers. curl is available on OS X, RedHat Linux, and many other OSes, including Win32 and Win64 versions. See [http://curl.haxx.se](http://curl.haxx.se) for more information. 31 | 32 | **MORE INFO** 33 | 34 | More information and basic documentation is available here: https://github.com/wdas/reposado/tree/master/docs 35 | -------------------------------------------------------------------------------- /code/repo_sync: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # encoding: utf-8 3 | # 4 | # Copyright 2011 Disney Enterprises, Inc. All rights reserved 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are 8 | # met: 9 | 10 | # * Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | 13 | # * Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in 15 | # the documentation and/or other materials provided with the 16 | # distribution. 17 | 18 | # * The names "Disney", "Walt Disney Pictures", "Walt Disney Animation 19 | # Studios" or the names of its contributors may NOT be used to 20 | # endorse or promote products derived from this software without 21 | # specific prior written permission from Walt Disney Pictures. 22 | 23 | # Disclaimer: THIS SOFTWARE IS PROVIDED BY WALT DISNEY PICTURES AND 24 | # CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, 25 | # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS 26 | # FOR A PARTICULAR PURPOSE, NONINFRINGEMENT AND TITLE ARE DISCLAIMED. 27 | # IN NO EVENT SHALL WALT DISNEY PICTURES, THE COPYRIGHT HOLDER OR 28 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 29 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 30 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 31 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND BASED ON ANY 32 | # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 33 | # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 34 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 35 | 36 | """ 37 | repo_sync 38 | 39 | Created by Greg Neagle on 2011-03-03. 40 | """ 41 | 42 | import calendar 43 | import os 44 | import optparse 45 | import plistlib 46 | import re 47 | import shutil 48 | import subprocess 49 | #import sys 50 | import time 51 | import tempfile 52 | import urlparse 53 | from xml.dom import minidom 54 | from xml.parsers.expat import ExpatError 55 | 56 | from reposadolib import reposadocommon 57 | 58 | def _win_os_rename(src, dst): 59 | '''Non-atomic os.rename() that doesn't throw OSError under Windows 60 | 61 | Windows doesn't allow renaming a file to a filename that already exists 62 | Idea from: http://bugs.python.org/issue8828#msg106599 63 | ''' 64 | try: 65 | os.rename(src, dst) 66 | except OSError: 67 | os.unlink(dst) 68 | os.rename(src, dst) 69 | 70 | if os.name in ('nt', 'ce'): 71 | os_rename = _win_os_rename 72 | else: 73 | os_rename = os.rename 74 | 75 | def parseServerMetadata(filename): 76 | '''Parses a softwareupdate server metadata file, looking for information 77 | of interest. 78 | Returns a dictionary containing title, version, and description.''' 79 | title = '' 80 | vers = '' 81 | description = '' 82 | try: 83 | md_plist = plistlib.readPlist(filename) 84 | except (OSError, IOError, ExpatError), err: 85 | reposadocommon.print_stderr( 86 | 'Error reading %s: %s', filename, err) 87 | return {} 88 | vers = md_plist.get('CFBundleShortVersionString', '') 89 | localization = md_plist.get('localization', {}) 90 | languages = localization.keys() 91 | preferred_lang = getPreferredLocalization(languages) 92 | preferred_localization = localization.get(preferred_lang) 93 | if preferred_localization: 94 | title = preferred_localization.get('title', '') 95 | encoded_description = preferred_localization.get('description', '') 96 | if encoded_description: 97 | description = str(encoded_description) 98 | 99 | metadata = {} 100 | metadata['title'] = title 101 | metadata['version'] = vers 102 | metadata['description'] = description 103 | return metadata 104 | 105 | 106 | def parse_cdata(cdata_str): 107 | '''Parses the CDATA string from an Apple Software Update distribution file 108 | and returns a dictionary with key/value pairs. 109 | 110 | The data in the CDATA string is in the format of an OS X .strings file, 111 | which is generally: 112 | 113 | "KEY1" = "VALUE1"; 114 | "KEY2"='VALUE2'; 115 | "KEY3" = 'A value 116 | that spans 117 | multiple lines. 118 | '; 119 | 120 | Values can span multiple lines; either single or double-quotes can be used 121 | to quote the keys and values, and the alternative quote character is allowed 122 | as a literal inside the other, otherwise the quote character is escaped. 123 | 124 | //-style comments and blank lines are allowed in the string; these should 125 | be skipped by the parser unless within a value. 126 | 127 | ''' 128 | parsed_data = {} 129 | REGEX = (r"""^\s*""" 130 | r"""(?P['"]?)(?P[^'"]+)(?P=key_quote)""" 131 | r"""\s*=\s*""" 132 | r"""(?P['"])(?P.*?)(?P=value_quote);$""") 133 | regex = re.compile(REGEX, re.MULTILINE | re.DOTALL) 134 | 135 | # iterate through the string, finding all possible non-overlapping 136 | # matches 137 | for match_obj in re.finditer(regex, cdata_str): 138 | match_dict = match_obj.groupdict() 139 | if 'key' in match_dict.keys() and 'value' in match_dict.keys(): 140 | key = match_dict['key'] 141 | value = match_dict['value'] 142 | # now 'de-escape' escaped quotes 143 | quote = match_dict.get('value_quote') 144 | if quote: 145 | escaped_quote = '\\' + quote 146 | value = value.replace(escaped_quote, quote) 147 | parsed_data[key] = value 148 | 149 | return parsed_data 150 | 151 | 152 | def parseSUdist(filename, debug=False): 153 | '''Parses a softwareupdate dist file, looking for information of interest. 154 | Returns a dictionary containing su_name, title, version, and description.''' 155 | 156 | try: 157 | dom = minidom.parse(filename) 158 | except ExpatError: 159 | reposadocommon.print_stderr( 160 | 'Invalid XML in %s', filename) 161 | return None 162 | except IOError, err: 163 | reposadocommon.print_stderr( 164 | 'Error reading %s: %s', filename, err) 165 | return None 166 | 167 | su_choice_id_key = 'su' 168 | # look for > fileobj, 'compressed' # accept and handle compressed files 293 | print >> fileobj, 'silent' # no progress meter 294 | print >> fileobj, 'show-error' # print error msg to stderr 295 | print >> fileobj, 'no-buffer' # don't buffer output 296 | print >> fileobj, 'fail' # throw error if download fails 297 | print >> fileobj, 'dump-header -' # dump headers to stdout 298 | print >> fileobj, 'speed-time = 30' # give up if too slow d/l 299 | print >> fileobj, 'tlsv1' # use only TLS 1.x 300 | print >> fileobj, 'http1.1' # disable http2 301 | print >> fileobj, 'url = "%s"' % url 302 | 303 | # add additional options from our prefs 304 | if reposadocommon.pref('AdditionalCurlOptions'): 305 | for line in reposadocommon.pref('AdditionalCurlOptions'): 306 | print >> fileobj, line 307 | 308 | if os.path.exists(tempdownloadpath): 309 | if resume: 310 | # let's try to resume this download 311 | print >> fileobj, 'continue-at -' 312 | else: 313 | os.remove(tempdownloadpath) 314 | 315 | if os.path.exists(destinationpath): 316 | if etag: 317 | escaped_etag = etag.replace('"', '\\"') 318 | print >> fileobj, ('header = "If-None-Match: %s"' 319 | % escaped_etag) 320 | elif onlyifnewer: 321 | print >> fileobj, 'time-cond = "%s"' % destinationpath 322 | else: 323 | os.remove(destinationpath) 324 | 325 | fileobj.close() 326 | except Exception, err: 327 | raise CurlError(-5, 'Error writing curl directive: %s' % str(err)) 328 | 329 | cmd = [reposadocommon.pref('CurlPath'), 330 | '-q', # don't read .curlrc file 331 | '--config', # use config file 332 | curldirectivepath, 333 | '-o', tempdownloadpath] 334 | 335 | proc = subprocess.Popen(cmd, shell=False, bufsize=1, 336 | stdin=subprocess.PIPE, 337 | stdout=subprocess.PIPE, stderr=subprocess.PIPE) 338 | 339 | targetsize = 0 340 | downloadedpercent = -1 341 | 342 | while True: 343 | line = proc.stdout.readline() 344 | if line: 345 | line_stripped = line.rstrip('\r\n') 346 | if line_stripped: 347 | line = line_stripped 348 | 349 | if line.startswith('HTTP/'): 350 | header['http_result_code'] = '' 351 | header['http_result_description'] = '' 352 | try: 353 | part = line.split(None, 2) 354 | header['http_result_code'] = part[1] 355 | header['http_result_description'] = part[2] 356 | except IndexError: 357 | pass 358 | elif ': ' in line: 359 | part = line.split(': ', 1) 360 | fieldname = part[0].lower() 361 | header[fieldname] = part[1] 362 | else: 363 | # "empty" line, but not end of output. likely end of headers 364 | # for a given HTTP result section 365 | try: 366 | targetsize = int(header.get('content-length')) 367 | if (targetsize and 368 | header.get('http_result_code').startswith('2')): 369 | if reposadocommon.pref('HumanReadableSizes'): 370 | printed_size = reposadocommon.humanReadable(targetsize) 371 | else: 372 | printed_size = str(targetsize) + ' bytes' 373 | reposadocommon.print_stdout( 374 | 'Downloading %s from %s...', printed_size, url) 375 | except (ValueError, TypeError): 376 | targetsize = 0 377 | if header.get('http_result_code') == '206': 378 | # partial content because we're resuming 379 | reposadocommon.print_stderr( 380 | 'Resuming partial download for %s', 381 | os.path.basename(destinationpath)) 382 | contentrange = header.get('content-range') 383 | if contentrange.startswith('bytes'): 384 | try: 385 | targetsize = int(contentrange.split('/')[1]) 386 | except (ValueError, TypeError): 387 | targetsize = 0 388 | 389 | elif proc.poll() != None: 390 | break 391 | 392 | retcode = proc.poll() 393 | if retcode: 394 | curlerr = proc.stderr.read().rstrip('\n') 395 | if curlerr: 396 | curlerr = curlerr.split(None, 2)[2] 397 | if os.path.exists(tempdownloadpath): 398 | if (not resume) or (retcode == 33): 399 | # 33 means server doesn't support range requests 400 | # and so cannot resume downloads, so 401 | os.remove(tempdownloadpath) 402 | raise CurlError(retcode, curlerr) 403 | else: 404 | temp_download_exists = os.path.isfile(tempdownloadpath) 405 | http_result = header.get('http_result_code') 406 | if (downloadedpercent != 100 and 407 | http_result.startswith('2') and 408 | temp_download_exists): 409 | downloadedsize = os.path.getsize(tempdownloadpath) 410 | if downloadedsize >= targetsize: 411 | os_rename(tempdownloadpath, destinationpath) 412 | return header 413 | else: 414 | # not enough bytes retreived 415 | if not resume and temp_download_exists: 416 | os.remove(tempdownloadpath) 417 | raise CurlError(-5, 'Expected %s bytes, got: %s' % 418 | (targetsize, downloadedsize)) 419 | elif http_result.startswith('2') and temp_download_exists: 420 | os_rename(tempdownloadpath, destinationpath) 421 | return header 422 | elif http_result == '304': 423 | return header 424 | elif (not temp_download_exists and 425 | http_result == '200' and 426 | os.path.isfile(destinationpath) and 427 | (not (targetsize and 428 | (targetsize != os.path.getsize(destinationpath))))): 429 | # The above comparison tries to check that a) no body content was 430 | # delivered, b) the HTTP result was 200, c) there is an existing 431 | # download already, and d) [if the there was a Content-Length 432 | # returned by the server] that the file sizes match. The logic is 433 | # reversed with a 'not' in step d) to return True if the sizes 434 | # match or there is no Content-Length. 435 | 436 | # This is a test for an edge case where curl does not download 437 | # body content even if the server returned a 200 response. This 438 | # happens when curl is given the 'time-cond' option (which sends 439 | # the HTTP header If-Modified-Since to the server) and the server 440 | # responds with a 200 response but curl terminates the connection 441 | # before any body content is transferred. I.e. curl goes above 442 | # and beyond sending an If-Modified-Since and actually compares 443 | # the Last-Modified header returned to it itself to make a 444 | # decision whether to download the document body. 445 | 446 | # See curl issue report here: 447 | # https://sourceforge.net/p/curl/bugs/806/ 448 | reposadocommon.print_stderr( 449 | 'WARNING: No body provided; assuming already downloaded for %s', 450 | destinationpath) 451 | return header 452 | else: 453 | # there was a download error of some sort; clean all relevant 454 | # downloads that may be in a bad state. 455 | for filename in [tempdownloadpath, destinationpath]: 456 | try: 457 | os.unlink(filename) 458 | except OSError: 459 | pass 460 | raise HTTPError(http_result, 461 | header.get('http_result_description', '')) 462 | 463 | 464 | def getURL(url, destination_path): 465 | '''Downloads a file from url to destination_path, checking existing 466 | files by mode date or etag''' 467 | if os.path.exists(destination_path): 468 | saved_etag = get_saved_etag(url) 469 | else: 470 | saved_etag = None 471 | try: 472 | header = curl(url, destination_path, 473 | onlyifnewer=True, etag=saved_etag) 474 | except CurlError, err: 475 | err = 'Error %s: %s' % tuple(err) 476 | raise CurlDownloadError(err) 477 | 478 | except HTTPError, err: 479 | err = 'HTTP result %s: %s' % tuple(err) 480 | raise CurlDownloadError(err) 481 | 482 | err = None 483 | if header['http_result_code'] == '304': 484 | # not modified; what we have is correct 485 | #print >> sys.stderr, ('%s is already downloaded.' % url) 486 | pass 487 | else: 488 | if header.get('last-modified'): 489 | # set the modtime of the downloaded file to the modtime of the 490 | # file on the server 491 | modtimestr = header['last-modified'] 492 | modtimetuple = time.strptime(modtimestr, 493 | '%a, %d %b %Y %H:%M:%S %Z') 494 | modtimeint = calendar.timegm(modtimetuple) 495 | os.utime(destination_path, (time.time(), modtimeint)) 496 | if header.get('etag'): 497 | # store etag for future use 498 | record_etag(url, header['etag']) 499 | 500 | 501 | _ETAG = {} 502 | def get_saved_etag(url): 503 | '''Retrieves a saved etag''' 504 | #global _ETAG 505 | if _ETAG == {}: 506 | reposadocommon.getDataFromPlist('ETags.plist') 507 | if url in _ETAG: 508 | return _ETAG[url] 509 | else: 510 | return None 511 | 512 | 513 | def record_etag(url, etag): 514 | '''Saves an etag in our internal dict''' 515 | #global _ETAG 516 | _ETAG[url] = etag 517 | 518 | 519 | def writeEtagDict(): 520 | '''Writes our stored etags to disk''' 521 | reposadocommon.writeDataToPlist(_ETAG, 'ETags.plist') 522 | 523 | 524 | class ReplicationError(Exception): 525 | '''A custom error when replication fails''' 526 | pass 527 | 528 | 529 | def replicateURLtoFilesystem(full_url, root_dir=None, 530 | base_url=None, copy_only_if_missing=False, 531 | appendToFilename=''): 532 | '''Downloads a URL and stores it in the same relative path on our 533 | filesystem. Returns a path to the replicated file.''' 534 | 535 | if root_dir == None: 536 | root_dir = reposadocommon.pref('UpdatesRootDir') 537 | 538 | if base_url: 539 | if not full_url.startswith(base_url): 540 | raise ReplicationError('%s is not a resource in %s' 541 | % (full_url, base_url)) 542 | relative_url = full_url[len(base_url):].lstrip('/') 543 | else: 544 | (unused_scheme, unused_netloc, 545 | path, unused_query, unused_fragment) = urlparse.urlsplit(full_url) 546 | relative_url = path.lstrip('/') 547 | relative_url = os.path.normpath(relative_url) 548 | local_file_path = os.path.join(root_dir, relative_url) + appendToFilename 549 | local_dir_path = os.path.dirname(local_file_path) 550 | if copy_only_if_missing and os.path.exists(local_file_path): 551 | return local_file_path 552 | if not os.path.exists(local_dir_path): 553 | try: 554 | os.makedirs(local_dir_path) 555 | except OSError, oserr: 556 | raise ReplicationError(oserr) 557 | try: 558 | getURL(full_url, local_file_path) 559 | except CurlDownloadError, err: 560 | raise ReplicationError(err) 561 | return local_file_path 562 | 563 | 564 | class ArchiveError(Exception): 565 | '''A custom error when archiving fails''' 566 | pass 567 | 568 | 569 | def archiveCatalog(catalogpath): 570 | '''Makes a copy of our catalog in our archive folder, 571 | marking with a date''' 572 | archivedir = os.path.join(os.path.dirname(catalogpath), 'archive') 573 | if not os.path.exists(archivedir): 574 | try: 575 | os.makedirs(archivedir) 576 | except OSError, oserr: 577 | raise ArchiveError(oserr) 578 | # get modtime of original file 579 | modtime = int(os.stat(catalogpath).st_mtime) 580 | # make a string from the mod time 581 | modtimestring = time.strftime('.%Y-%m-%d-%H%M%S', time.localtime(modtime)) 582 | catalogname = os.path.basename(catalogpath) 583 | # remove the '.apple' from the end of the catalogname 584 | if catalogname.endswith('.apple'): 585 | catalogname = catalogname[0:-6] 586 | archivepath = os.path.join(archivedir, catalogname + modtimestring) 587 | if not os.path.exists(archivepath): 588 | try: 589 | catalog = plistlib.readPlist(catalogpath) 590 | plistlib.writePlist(catalog, archivepath) 591 | # might as well set the mod time of the archive file to match 592 | os.utime(archivepath, (time.time(), modtime)) 593 | except (OSError, IOError, ExpatError), err: 594 | reposadocommon.print_stderr( 595 | 'Error archiving %s: %s', catalogpath, err) 596 | 597 | 598 | def getPreferredLocalization(list_of_localizations): 599 | '''Picks the best localization from a list of available 600 | localizations. If we're running on OS X, we use 601 | NSBundle.preferredLocalizationsFromArray_forPreferences_, 602 | else we look for PreferredLocalizations in our preferences''' 603 | try: 604 | from Foundation import NSBundle 605 | except ImportError: 606 | # Foundation NSBundle isn't available, use prefs instead 607 | languages = (reposadocommon.pref('PreferredLocalizations') 608 | or ['English', 'en']) 609 | for language in languages: 610 | if language in list_of_localizations: 611 | return language 612 | else: 613 | preferred_langs = ( 614 | NSBundle.preferredLocalizationsFromArray_forPreferences_( 615 | list_of_localizations, None)) 616 | if preferred_langs: 617 | return preferred_langs[0] 618 | 619 | if 'English' in list_of_localizations: 620 | return 'English' 621 | elif 'en' in list_of_localizations: 622 | return 'en' 623 | return None 624 | 625 | 626 | def cleanUpTmpDir(): 627 | """Cleans up our temporary directory.""" 628 | global TMPDIR 629 | if TMPDIR: 630 | try: 631 | shutil.rmtree(TMPDIR) 632 | except (OSError, IOError): 633 | pass 634 | TMPDIR = None 635 | 636 | 637 | TMPDIR = None 638 | def sync(fast_scan=False, download_packages=True): 639 | '''Syncs Apple's Software Updates with our local store. 640 | Returns a dictionary of products.''' 641 | global TMPDIR 642 | TMPDIR = tempfile.mkdtemp() 643 | if reposadocommon.LOGFILE: 644 | print 'Output logged to %s' % reposadocommon.LOGFILE 645 | reposadocommon.print_stdout('repo_sync run started') 646 | catalog_urls = reposadocommon.pref('AppleCatalogURLs') 647 | products = reposadocommon.getProductInfo() 648 | 649 | # clear cached AppleCatalog listings 650 | for item in products.keys(): 651 | products[item]['AppleCatalogs'] = [] 652 | replicated_products = [] 653 | 654 | for catalog_url in catalog_urls: 655 | localcatalogpath = ( 656 | reposadocommon.getLocalPathNameFromURL(catalog_url) + '.apple') 657 | if os.path.exists(localcatalogpath): 658 | archiveCatalog(localcatalogpath) 659 | try: 660 | localcatalogpath = replicateURLtoFilesystem( 661 | catalog_url, appendToFilename='.apple') 662 | except ReplicationError, err: 663 | reposadocommon.print_stderr( 664 | 'Could not replicate %s: %s', catalog_url, err) 665 | continue 666 | try: 667 | catalog = plistlib.readPlist(localcatalogpath) 668 | except (OSError, IOError, ExpatError), err: 669 | reposadocommon.print_stderr( 670 | 'Error reading %s: %s', localcatalogpath, err) 671 | continue 672 | if 'Products' in catalog: 673 | product_keys = list(catalog['Products'].keys()) 674 | reposadocommon.print_stdout('%s products found in %s', 675 | len(product_keys), catalog_url) 676 | product_keys.sort() 677 | for product_key in product_keys: 678 | if product_key in replicated_products: 679 | products[product_key]['AppleCatalogs'].append( 680 | catalog_url) 681 | else: 682 | if not product_key in products: 683 | products[product_key] = {} 684 | products[product_key]['AppleCatalogs'] = [catalog_url] 685 | product = catalog['Products'][product_key] 686 | products[product_key]['CatalogEntry'] = product 687 | if download_packages and 'ServerMetadataURL' in product: 688 | try: 689 | unused_path = replicateURLtoFilesystem( 690 | product['ServerMetadataURL'], 691 | copy_only_if_missing=fast_scan) 692 | except ReplicationError, err: 693 | reposadocommon.print_stderr( 694 | 'Could not replicate %s: %s', 695 | product['ServerMetadataURL'], err) 696 | continue 697 | 698 | if download_packages: 699 | for package in product.get('Packages', []): 700 | # TO-DO: Check 'Size' attribute and make sure 701 | # we have enough space on the target 702 | # filesystem before attempting to download 703 | if 'URL' in package: 704 | try: 705 | unused_path = replicateURLtoFilesystem( 706 | package['URL'], 707 | copy_only_if_missing=fast_scan) 708 | except ReplicationError, err: 709 | reposadocommon.print_stderr( 710 | 'Could not replicate %s: %s', 711 | package['URL'], err) 712 | continue 713 | if 'MetadataURL' in package: 714 | try: 715 | unused_path = replicateURLtoFilesystem( 716 | package['MetadataURL'], 717 | copy_only_if_missing=fast_scan) 718 | except ReplicationError, err: 719 | reposadocommon.print_stderr( 720 | 'Could not replicate %s: %s', 721 | package['MetadataURL'], err) 722 | continue 723 | if 'IntegrityDataURL' in package: 724 | try: 725 | unused_path = replicateURLtoFilesystem( 726 | package['IntegrityDataURL'], 727 | copy_only_if_missing=fast_scan) 728 | except ReplicationError, err: 729 | reposadocommon.print_stderr( 730 | 'Could not replicate %s: %s', 731 | package['IntegrityDataURL'], err) 732 | continue 733 | 734 | # calculate total size 735 | size = 0 736 | for package in product.get('Packages', []): 737 | size += package.get('Size', 0) 738 | 739 | distributions = product['Distributions'] 740 | preferred_lang = getPreferredLocalization( 741 | distributions.keys()) 742 | preferred_dist = None 743 | 744 | for dist_lang in distributions.keys(): 745 | dist_url = distributions[dist_lang] 746 | if (download_packages or 747 | dist_lang == preferred_lang): 748 | try: 749 | dist_path = replicateURLtoFilesystem( 750 | dist_url, 751 | copy_only_if_missing=fast_scan) 752 | if dist_lang == preferred_lang: 753 | preferred_dist = dist_path 754 | except ReplicationError, err: 755 | reposadocommon.print_stderr( 756 | 'Could not replicate %s: %s', dist_url, err) 757 | 758 | if not preferred_dist: 759 | # we didn't download the .dist for the preferred 760 | # language. Let's use English. 761 | if 'English' in distributions.keys(): 762 | dist_lang = 'English' 763 | elif 'en' in distributions.keys(): 764 | dist_lang = 'en' 765 | else: 766 | # no English or en.dist! 767 | reposadocommon.print_stderr( 768 | 'No usable .dist file found!') 769 | continue 770 | dist_url = distributions[dist_lang] 771 | preferred_dist = reposadocommon.getLocalPathNameFromURL( 772 | dist_url) 773 | 774 | dist = parseSUdist(preferred_dist) 775 | if not dist: 776 | reposadocommon.print_stderr( 777 | 'Could not get data from dist file: %s', 778 | preferred_dist) 779 | continue 780 | products[product_key]['title'] = dist['title'] 781 | products[product_key]['version'] = dist['version'] 782 | products[product_key]['size'] = str(size) 783 | products[product_key]['description'] = dist['description'] 784 | products[product_key]['PostDate'] = product['PostDate'] 785 | products[product_key]['pkg_refs'] = dist['pkg_refs'] 786 | 787 | # if we got this far, we've replicated the product data 788 | replicated_products.append(product_key) 789 | 790 | # record original catalogs in case the product is 791 | # deprecated in the future 792 | #if not 'OriginalAppleCatalogs' in products[product_key]: 793 | # products[product_key]['OriginalAppleCatalogs'] = \ 794 | # products[product_key]['AppleCatalogs'] 795 | 796 | # If AppleCatalogs list is non-empty, record to 797 | # OriginalAppleCatalogs in case the product is deprecated 798 | # in the future 799 | # 800 | # (This is a change from the original implementation to 801 | # account for products being mistakenly released for the 802 | # wrong sucatalogs and later corrected. The assumption now 803 | # is that a change in available catalogs means Apple is 804 | # fixing a mistake; disappearing from all catalogs means 805 | # an item is deprecated.) 806 | if products[product_key]['AppleCatalogs']: 807 | products[product_key]['OriginalAppleCatalogs'] = ( 808 | products[product_key]['AppleCatalogs']) 809 | 810 | # record products we've successfully downloaded 811 | reposadocommon.writeDownloadStatus(replicated_products) 812 | # write our ETags to disk for future use 813 | writeEtagDict() 814 | # record our product cache 815 | reposadocommon.writeProductInfo(products) 816 | # write our local (filtered) catalogs 817 | reposadocommon.writeLocalCatalogs(localcatalogpath) 818 | 819 | # clean up tmpdir 820 | cleanUpTmpDir() 821 | reposadocommon.print_stdout('repo_sync run ended') 822 | 823 | 824 | def main(): 825 | '''Main command processing''' 826 | parser = optparse.OptionParser() 827 | parser.set_usage('''Usage: %prog [options]''') 828 | parser.add_option('--log', dest='logfile', metavar='LOGFILE', 829 | help='Log all output to LOGFILE. No output to STDOUT.') 830 | parser.add_option('--recheck', action='store_true', 831 | help='Recheck already downloaded packages for changes.') 832 | 833 | options, unused_arguments = parser.parse_args() 834 | if reposadocommon.validPreferences(): 835 | if not os.path.exists(reposadocommon.pref('CurlPath')): 836 | reposadocommon.print_stderr('ERROR: curl tool not found at %s', 837 | reposadocommon.pref('CurlPath')) 838 | exit(-1) 839 | if not reposadocommon.pref('LocalCatalogURLBase'): 840 | download_packages = False 841 | else: 842 | download_packages = True 843 | if options.logfile: 844 | reposadocommon.LOGFILE = options.logfile 845 | elif reposadocommon.pref('RepoSyncLogFile'): 846 | reposadocommon.LOGFILE = reposadocommon.pref('RepoSyncLogFile') 847 | 848 | sync(fast_scan=(not options.recheck), 849 | download_packages=download_packages) 850 | 851 | 852 | if __name__ == '__main__': 853 | main() 854 | -------------------------------------------------------------------------------- /code/repoclean: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # encoding: utf-8 3 | # 4 | # Copyright 2019 Disney Enterprises, Inc. All rights reserved 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are 8 | # met: 9 | 10 | # * Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | 13 | # * Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in 15 | # the documentation and/or other materials provided with the 16 | # distribution. 17 | 18 | # * The names "Disney", "Walt Disney Pictures", "Walt Disney Animation 19 | # Studios" or the names of its contributors may NOT be used to 20 | # endorse or promote products derived from this software without 21 | # specific prior written permission from Walt Disney Pictures. 22 | 23 | # Disclaimer: THIS SOFTWARE IS PROVIDED BY WALT DISNEY PICTURES AND 24 | # CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, 25 | # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS 26 | # FOR A PARTICULAR PURPOSE, NONINFRINGEMENT AND TITLE ARE DISCLAIMED. 27 | # IN NO EVENT SHALL WALT DISNEY PICTURES, THE COPYRIGHT HOLDER OR 28 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 29 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 30 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 31 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND BASED ON ANY 32 | # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 33 | # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 34 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 35 | 36 | '''A tool to clean abandonded products from a reposado repo''' 37 | 38 | from __future__ import absolute_import 39 | from __future__ import print_function 40 | 41 | import os 42 | import shutil 43 | import sys 44 | 45 | from reposadolib import reposadocommon 46 | 47 | 48 | # Python 2 and 3 wrapper for raw_input/input 49 | try: 50 | # Python 2 51 | get_input = raw_input # pylint: disable=raw_input-builtin, invalid-name 52 | except NameError: 53 | # Python 3 54 | get_input = input # pylint: disable=input-builtin, invalid-name 55 | 56 | 57 | def find_all_product_dirs_on_disk(): 58 | '''Walks downloads dir and returns a set of all directories that appear to 59 | contain a product''' 60 | root_dir = reposadocommon.pref('UpdatesRootDir') 61 | downloads_dir = os.path.join(root_dir, "content/downloads") 62 | product_dirs = set() 63 | for (path, _, files) in os.walk(downloads_dir): 64 | for a_file in files: 65 | if a_file.endswith((".smd", ".pkm", ".pkg", ".mpkg", ".tar")): 66 | product_dirs.add(path) 67 | break 68 | return product_dirs 69 | 70 | 71 | def get_product_location(product): 72 | '''Returns local path to replicated product.''' 73 | if not 'CatalogEntry' in product: 74 | # something is wrong with the product entry 75 | return None 76 | catalog_entry = product['CatalogEntry'] 77 | product_url = None 78 | if 'ServerMetadataURL' in catalog_entry: 79 | product_url = catalog_entry['ServerMetadataURL'] 80 | else: 81 | try: 82 | # get the URL for the first package in the Packages array 83 | product_url = catalog_entry['Packages'][0]['URL'] 84 | except (KeyError, IndexError): 85 | return None 86 | filepath = reposadocommon.getLocalPathNameFromURL(product_url) 87 | # return the directory this pkg is in 88 | return os.path.dirname(filepath) 89 | 90 | 91 | def all_product_dirs_from_productinfo(): # pylint: disable=invalid-name 92 | '''Returns a set of all the product directories for products in our 93 | ProductInformation.plist''' 94 | product_dirs = set() 95 | products = reposadocommon.getProductInfo() 96 | for product in products: 97 | product_location = get_product_location(products[product]) 98 | if product_location: 99 | product_dirs.add(product_location) 100 | return product_dirs 101 | 102 | 103 | def main(): 104 | '''Here's the main thing we do!''' 105 | if not reposadocommon.pref('LocalCatalogURLBase'): 106 | print("We're not replicating products, so nothing to clean up!") 107 | return 108 | 109 | print("Finding all product directories stored on disk...") 110 | disk_dirs = find_all_product_dirs_on_disk() 111 | print(" Found %s products" % len(disk_dirs)) 112 | 113 | print("Finding all product directories in our internal database...") 114 | info_dirs = all_product_dirs_from_productinfo() 115 | print(" Found %s products" % len(info_dirs)) 116 | 117 | missing_disk_dirs = info_dirs - disk_dirs 118 | if missing_disk_dirs: 119 | print("\nFound %s products that might not be on-disk:" 120 | % len(missing_disk_dirs)) 121 | print("\n".join(sorted(missing_disk_dirs))) 122 | 123 | orphaned_dirs = disk_dirs - info_dirs 124 | if orphaned_dirs: 125 | print("\nFound %s abandoned/orphaned product directories:" 126 | % len(orphaned_dirs)) 127 | print("\n".join(sorted(orphaned_dirs))) 128 | print() 129 | answer = get_input( 130 | 'Remove abandoned product directories ' 131 | '(WARNING: this cannot be undone)? [y/n]: ') 132 | if answer.lower().startswith('y'): 133 | for directory in orphaned_dirs: 134 | print ("Removing %s..." % directory) 135 | try: 136 | shutil.rmtree(directory) 137 | except (OSError, IOError) as err: 138 | print('Error: %s' % err, file=sys.stderr) 139 | else: 140 | print("Can't find anything to clean up!") 141 | 142 | 143 | if __name__ == '__main__': 144 | main() 145 | -------------------------------------------------------------------------------- /code/reposadolib/__init__.py: -------------------------------------------------------------------------------- 1 | # this is needed to make Python recognize the directory as a module package. 2 | # 3 | # Warning: do NOT put any Python imports here that require ObjC. 4 | -------------------------------------------------------------------------------- /code/reposadolib/reposadocommon.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # encoding: utf-8 3 | # 4 | # Copyright 2011 Disney Enterprises, Inc. All rights reserved 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are 8 | # met: 9 | 10 | # * Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | 13 | # * Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in 15 | # the documentation and/or other materials provided with the 16 | # distribution. 17 | 18 | # * The names "Disney", "Walt Disney Pictures", "Walt Disney Animation 19 | # Studios" or the names of its contributors may NOT be used to 20 | # endorse or promote products derived from this software without 21 | # specific prior written permission from Walt Disney Pictures. 22 | 23 | # Disclaimer: THIS SOFTWARE IS PROVIDED BY WALT DISNEY PICTURES AND 24 | # CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, 25 | # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS 26 | # FOR A PARTICULAR PURPOSE, NONINFRINGEMENT AND TITLE ARE DISCLAIMED. 27 | # IN NO EVENT SHALL WALT DISNEY PICTURES, THE COPYRIGHT HOLDER OR 28 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 29 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 30 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 31 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND BASED ON ANY 32 | # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 33 | # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 34 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 35 | 36 | """ 37 | reposadocommon.py 38 | 39 | Created by Greg Neagle on 2011-03-03. 40 | """ 41 | 42 | from __future__ import absolute_import 43 | from __future__ import print_function 44 | import sys 45 | import os 46 | import imp 47 | import plistlib 48 | import time 49 | import urlparse 50 | import warnings 51 | from xml.parsers.expat import ExpatError 52 | from xml.dom import minidom 53 | 54 | def get_main_dir(): 55 | '''Returns the directory name of the script or the directory name of the exe 56 | if py2exe was used 57 | Code from http://www.py2exe.org/index.cgi/HowToDetermineIfRunningFromExe 58 | ''' 59 | if (hasattr(sys, "frozen") or hasattr(sys, "importers") 60 | or imp.is_frozen("__main__")): 61 | return os.path.dirname(sys.executable) 62 | return os.path.dirname(sys.argv[0]) 63 | 64 | def prefsFilePath(): 65 | '''Returns path to our preferences file.''' 66 | return os.path.join(get_main_dir(), 'preferences.plist') 67 | 68 | 69 | def pref(prefname): 70 | '''Returns a preference.''' 71 | default_prefs = { 72 | 'AppleCatalogURLs': [ 73 | ('http://swscan.apple.com/content/catalogs/' 74 | 'index.sucatalog'), 75 | ('http://swscan.apple.com/content/catalogs/' 76 | 'index-1.sucatalog'), 77 | ('http://swscan.apple.com/content/catalogs/others/' 78 | 'index-leopard.merged-1.sucatalog'), 79 | ('http://swscan.apple.com/content/catalogs/others/' 80 | 'index-leopard-snowleopard.merged-1.sucatalog'), 81 | ('http://swscan.apple.com/content/catalogs/others/' 82 | 'index-lion-snowleopard-leopard.merged-1.sucatalog'), 83 | ('http://swscan.apple.com/content/catalogs/others/' 84 | 'index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog'), 85 | ('https://swscan.apple.com/content/catalogs/others/' 86 | 'index-10.9-mountainlion-lion-snowleopard-leopard.merged-1' 87 | '.sucatalog'), 88 | ('https://swscan.apple.com/content/catalogs/others/' 89 | 'index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1' 90 | '.sucatalog'), 91 | ('https://swscan.apple.com/content/catalogs/others/' 92 | 'index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-' 93 | 'snowleopard-leopard.merged-1.sucatalog'), 94 | ('https://swscan.apple.com/content/catalogs/others/' 95 | 'index-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-' 96 | 'snowleopard-leopard.merged-1.sucatalog'), 97 | ('https://swscan.apple.com/content/catalogs/others/' 98 | 'index-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-' 99 | 'leopard.merged-1.sucatalog'), 100 | ('https://swscan.apple.com/content/catalogs/others/' 101 | 'index-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-' 102 | 'leopard.merged-1.sucatalog'), 103 | ('https://swscan.apple.com/content/catalogs/others/' 104 | 'index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard' 105 | '.merged-1.sucatalog'), 106 | ], 107 | 'PreferredLocalizations': ['English', 'en'], 108 | 'CurlPath': '/usr/bin/curl' 109 | } 110 | try: 111 | prefs = plistlib.readPlist(prefsFilePath()) 112 | except (IOError, ExpatError): 113 | prefs = default_prefs 114 | if prefname in prefs: 115 | return prefs[prefname] 116 | elif prefname in default_prefs: 117 | return default_prefs[prefname] 118 | else: 119 | return None 120 | 121 | 122 | def validPreferences(): 123 | '''Validates our preferences to make sure needed values are defined 124 | and paths exist. Returns boolean.''' 125 | prefs_valid = True 126 | for pref_name in ['UpdatesRootDir', 'UpdatesMetadataDir']: 127 | preference = pref(pref_name) 128 | if not preference: 129 | print_stderr('ERROR: %s is not defined in %s.' % 130 | (pref_name, prefsFilePath())) 131 | prefs_valid = False 132 | elif not os.path.exists(preference): 133 | print_stderr('WARNING: %s "%s" does not exist.' 134 | ' Will attempt to create it.' % 135 | (pref_name, preference)) 136 | return prefs_valid 137 | 138 | 139 | def configure_prefs(): 140 | """Configures prefs for use""" 141 | _prefs = {} 142 | keysAndPrompts = [ 143 | ('UpdatesRootDir', 144 | 'Filesystem path to store replicated catalogs and updates'), 145 | ('UpdatesMetadataDir', 146 | 'Filesystem path to store Reposado metadata'), 147 | ('LocalCatalogURLBase', 148 | 'Base URL for your local Software Update Service\n(Example: ' 149 | 'http://su.your.org -- leave empty if you are not replicating ' 150 | 'updates)'), 151 | ] 152 | if not os.path.exists(pref('CurlPath')): 153 | keysAndPrompts.append( 154 | ('CurlPath', 'Path to curl tool (Example: /usr/bin/curl)')) 155 | 156 | for (key, prompt) in keysAndPrompts: 157 | newvalue = raw_input('%15s [%s]: ' % (prompt, pref(key))) 158 | _prefs[key] = newvalue or pref(key) or '' 159 | 160 | prefspath = prefsFilePath() 161 | # retrieve current preferences 162 | try: 163 | prefs = plistlib.readPlist(prefspath) 164 | except (IOError, ExpatError): 165 | prefs = {} 166 | # merge edited preferences 167 | for key in _prefs.keys(): 168 | prefs[key] = _prefs[key] 169 | # write preferences to our file 170 | try: 171 | plistlib.writePlist(prefs, prefspath) 172 | except (IOError, ExpatError): 173 | print_stderr('Could not save configuration to %s', prefspath) 174 | else: 175 | # check to make sure they're valid 176 | unused_value = validPreferences() 177 | 178 | 179 | def str_to_ascii(s): 180 | """Given str (unicode, latin-1, or not) return ascii. 181 | 182 | Args: 183 | s: str, likely in Unicode-16BE, UTF-8, or Latin-1 charset 184 | Returns: 185 | str, ascii form, no >7bit chars 186 | """ 187 | try: 188 | return unicode(s).encode('ascii', 'ignore') 189 | except UnicodeDecodeError: 190 | return s.decode('ascii', 'ignore') 191 | 192 | 193 | def concat_message(msg, *args): 194 | """Concatenates a string with any additional arguments; drops unicode.""" 195 | msg = str_to_ascii(msg) 196 | if args: 197 | args = [str_to_ascii(arg) for arg in args] 198 | try: 199 | msg = msg % tuple(args) 200 | except TypeError: 201 | warnings.warn( 202 | 'String format does not match concat args: %s' % ( 203 | str(sys.exc_info()))) 204 | return msg 205 | 206 | 207 | def log(msg): 208 | """Generic logging function""" 209 | # date/time format string 210 | if not LOGFILE: 211 | return 212 | formatstr = '%b %d %H:%M:%S' 213 | try: 214 | fileobj = open(LOGFILE, mode='a', buffering=1) 215 | try: 216 | print(time.strftime(formatstr), msg.encode('UTF-8'), file=fileobj) 217 | except (OSError, IOError): 218 | pass 219 | fileobj.close() 220 | except (OSError, IOError): 221 | pass 222 | 223 | 224 | def print_stdout(msg, *args): 225 | """ 226 | Prints message and args to stdout. 227 | """ 228 | output = concat_message(msg, *args) 229 | if LOGFILE: 230 | log(output) 231 | else: 232 | print(output) 233 | sys.stdout.flush() 234 | 235 | 236 | def print_stderr(msg, *args): 237 | """ 238 | Prints message and args to stderr. 239 | """ 240 | output = concat_message(msg, *args) 241 | if LOGFILE: 242 | log(output) 243 | else: 244 | print(concat_message(msg, *args), file=sys.stderr) 245 | 246 | 247 | def humanReadable(size_in_bytes): 248 | """Returns sizes in human-readable units.""" 249 | try: 250 | size_in_bytes = int(size_in_bytes) 251 | except ValueError: 252 | size_in_bytes = 0 253 | units = [(" KB", 10**6), (" MB", 10**9), (" GB", 10**12), (" TB", 10**15)] 254 | for suffix, limit in units: 255 | if size_in_bytes > limit: 256 | continue 257 | else: 258 | return str(round(size_in_bytes/float(limit/2**10), 1)) + suffix 259 | 260 | 261 | def writeDataToPlist(data, filename): 262 | '''Writes a dict or list to a plist in our metadata dir''' 263 | metadata_dir = pref('UpdatesMetadataDir') 264 | if not os.path.exists(metadata_dir): 265 | try: 266 | os.makedirs(metadata_dir) 267 | except OSError as errmsg: 268 | print_stderr( 269 | 'Could not create missing %s because %s', 270 | metadata_dir, errmsg) 271 | try: 272 | plistlib.writePlist(data, 273 | os.path.join(metadata_dir, filename)) 274 | except (IOError, OSError, TypeError) as errmsg: 275 | print_stderr( 276 | 'Could not write %s because %s', filename, errmsg) 277 | 278 | 279 | def getDataFromPlist(filename): 280 | '''Reads data from a plist in our metadata dir''' 281 | metadata_dir = pref('UpdatesMetadataDir') 282 | try: 283 | return plistlib.readPlist( 284 | os.path.join(metadata_dir, filename)) 285 | except (IOError, ExpatError): 286 | return {} 287 | 288 | 289 | def getDownloadStatus(): 290 | '''Reads download status info from disk''' 291 | return getDataFromPlist('DownloadStatus.plist') 292 | 293 | 294 | def writeDownloadStatus(download_status_list): 295 | '''Writes download status info to disk''' 296 | writeDataToPlist(download_status_list, 'DownloadStatus.plist') 297 | 298 | 299 | def getCatalogBranches(): 300 | '''Reads catalog branches info from disk''' 301 | return getDataFromPlist('CatalogBranches.plist') 302 | 303 | 304 | def writeCatalogBranches(catalog_branches): 305 | '''Writes catalog branches info to disk''' 306 | writeDataToPlist(catalog_branches, 'CatalogBranches.plist') 307 | 308 | 309 | def getProductInfo(): 310 | '''Reads Software Update product info from disk''' 311 | return getDataFromPlist('ProductInfo.plist') 312 | 313 | 314 | def writeProductInfo(product_info_dict): 315 | '''Writes Software Update product info to disk''' 316 | writeDataToPlist(product_info_dict, 'ProductInfo.plist') 317 | 318 | 319 | def getFilenameFromURL(url): 320 | '''Gets the filename from a URL''' 321 | (unused_scheme, unused_netloc, 322 | path, unused_query, unused_fragment) = urlparse.urlsplit(url) 323 | return os.path.basename(path) 324 | 325 | 326 | def getLocalPathNameFromURL(url, root_dir=None): 327 | '''Derives the appropriate local path name based on the URL''' 328 | if root_dir is None: 329 | root_dir = pref('UpdatesRootDir') 330 | (unused_scheme, unused_netloc, 331 | path, unused_query, unused_fragment) = urlparse.urlsplit(url) 332 | relative_path = path.lstrip('/') 333 | return os.path.join(root_dir, relative_path) 334 | 335 | 336 | def rewriteOneURL(full_url): 337 | '''Rewrites a single URL to point to our local replica''' 338 | our_base_url = pref('LocalCatalogURLBase') 339 | if not full_url.startswith(our_base_url): 340 | # only rewrite the URL if needed 341 | (unused_scheme, unused_netloc, 342 | path, unused_query, unused_fragment) = urlparse.urlsplit(full_url) 343 | return our_base_url + path 344 | else: 345 | return full_url 346 | 347 | 348 | def rewriteURLsForProduct(product): 349 | '''Rewrites the URLs for a product''' 350 | if 'ServerMetadataURL' in product: 351 | product['ServerMetadataURL'] = rewriteOneURL( 352 | product['ServerMetadataURL']) 353 | for package in product.get('Packages', []): 354 | if 'URL' in package: 355 | package['URL'] = rewriteOneURL(package['URL']) 356 | if 'MetadataURL' in package: 357 | package['MetadataURL'] = rewriteOneURL( 358 | package['MetadataURL']) 359 | if 'IntegrityDataURL' in package: 360 | package['IntegrityDataURL'] = rewriteOneURL( 361 | package['IntegrityDataURL']) 362 | # workaround for 10.8.2 issue where client ignores local pkg 363 | # and prefers Apple's URL. Need to revisit as we better understand this 364 | # issue 365 | if 'Digest' in package: 366 | # removing the Digest causes 10.8.2 to use the replica's URL 367 | # instead of Apple's 368 | del package['Digest'] 369 | distributions = product['Distributions'] 370 | for dist_lang in distributions.keys(): 371 | distributions[dist_lang] = rewriteOneURL( 372 | distributions[dist_lang]) 373 | 374 | 375 | def rewriteURLs(catalog): 376 | '''Rewrites all the URLs in the given catalog to point to our local 377 | replica''' 378 | if pref('LocalCatalogURLBase') is None: 379 | return 380 | if 'Products' in catalog: 381 | product_keys = list(catalog['Products'].keys()) 382 | for product_key in product_keys: 383 | product = catalog['Products'][product_key] 384 | rewriteURLsForProduct(product) 385 | 386 | 387 | def writeAllBranchCatalogs(): 388 | '''Writes out all branch catalogs. Used when we edit branches.''' 389 | for catalog_URL in pref('AppleCatalogURLs'): 390 | localcatalogpath = getLocalPathNameFromURL(catalog_URL) 391 | if os.path.exists(localcatalogpath): 392 | writeBranchCatalogs(localcatalogpath) 393 | else: 394 | print_stderr( 395 | 'WARNING: %s does not exist. Perhaps you need to run repo_sync?' 396 | % localcatalogpath) 397 | 398 | 399 | def writeBranchCatalogs(localcatalogpath): 400 | '''Writes our branch catalogs''' 401 | catalog = plistlib.readPlist(localcatalogpath) 402 | downloaded_products = catalog['Products'] 403 | product_info = getProductInfo() 404 | 405 | localcatalogname = os.path.basename(localcatalogpath) 406 | # now strip the '.sucatalog' bit from the name 407 | # so we can use it to construct our branch catalog names 408 | if localcatalogpath.endswith('.sucatalog'): 409 | localcatalogpath = localcatalogpath[0:-10] 410 | 411 | # now write filtered catalogs (branches) 412 | catalog_branches = getCatalogBranches() 413 | for branch in catalog_branches.keys(): 414 | branchcatalogpath = localcatalogpath + '_' + branch + '.sucatalog' 415 | print_stdout('Building %s...' % os.path.basename(branchcatalogpath)) 416 | # embed branch catalog name into the catalog for troubleshooting 417 | # and validation 418 | catalog['_CatalogName'] = os.path.basename(branchcatalogpath) 419 | catalog['Products'] = {} 420 | for product_key in catalog_branches[branch]: 421 | if product_key in downloaded_products.keys(): 422 | # add the product to the Products dict 423 | # for this catalog 424 | catalog['Products'][product_key] = \ 425 | downloaded_products[product_key] 426 | elif pref('LocalCatalogURLBase') and product_key in product_info: 427 | # Product might have been deprecated by Apple, 428 | # so we check cached product info 429 | # Check to see if this product was ever in this 430 | # catalog 431 | original_catalogs = product_info[product_key].get( 432 | 'OriginalAppleCatalogs', []) 433 | for original_catalog in original_catalogs: 434 | if original_catalog.endswith(localcatalogname): 435 | # this item was originally in this catalog, so 436 | # we can add it to the branch 437 | catalog_entry = \ 438 | product_info[product_key].get('CatalogEntry') 439 | title = product_info[product_key].get('title') 440 | version = product_info[product_key].get('version') 441 | if catalog_entry: 442 | print_stderr( 443 | 'WARNING: Product %s (%s-%s) in branch %s ' 444 | 'has been deprecated. Will use cached info ' 445 | 'and packages.', 446 | product_key, title, version, branch) 447 | rewriteURLsForProduct(catalog_entry) 448 | catalog['Products'][product_key] = catalog_entry 449 | continue 450 | else: 451 | # item is not listed in the main catalog and we don't have a 452 | # local cache of product info. It either was never in this 453 | # catalog or has been removed by Apple. In either case, we just 454 | # skip the item -- we can't add it to the catalog. 455 | pass 456 | 457 | plistlib.writePlist(catalog, branchcatalogpath) 458 | 459 | 460 | def writeAllLocalCatalogs(): 461 | '''Writes out all local and branch catalogs. Used when we purge products.''' 462 | for catalog_URL in pref('AppleCatalogURLs'): 463 | localcatalogpath = getLocalPathNameFromURL(catalog_URL) + '.apple' 464 | if os.path.exists(localcatalogpath): 465 | writeLocalCatalogs(localcatalogpath) 466 | 467 | 468 | def writeLocalCatalogs(applecatalogpath): 469 | '''Writes our local catalogs based on the Apple catalog''' 470 | catalog = plistlib.readPlist(applecatalogpath) 471 | # rewrite the URLs within the catalog to point to the items on our 472 | # local server instead of Apple's 473 | rewriteURLs(catalog) 474 | # remove the '.apple' from the end of the localcatalogpath 475 | if applecatalogpath.endswith('.apple'): 476 | localcatalogpath = applecatalogpath[0:-6] 477 | else: 478 | localcatalogpath = applecatalogpath 479 | 480 | print_stdout('Building %s...' % os.path.basename(localcatalogpath)) 481 | catalog['_CatalogName'] = os.path.basename(localcatalogpath) 482 | downloaded_products_list = getDownloadStatus() 483 | 484 | downloaded_products = {} 485 | product_keys = list(catalog['Products'].keys()) 486 | # filter Products, removing those that haven't been downloaded 487 | for product_key in product_keys: 488 | if product_key in downloaded_products_list: 489 | downloaded_products[product_key] = \ 490 | catalog['Products'][product_key] 491 | else: 492 | print_stderr('WARNING: did not add product %s to ' 493 | 'catalog %s because it has not been downloaded.', 494 | product_key, os.path.basename(applecatalogpath)) 495 | catalog['Products'] = downloaded_products 496 | 497 | # write raw (unstable/development) catalog 498 | # with all downloaded Apple updates enabled 499 | plistlib.writePlist(catalog, localcatalogpath) 500 | 501 | # now write filtered catalogs (branches) based on this catalog 502 | writeBranchCatalogs(localcatalogpath) 503 | 504 | 505 | def readXMLfile(filename): 506 | '''Return dom from XML file or None''' 507 | try: 508 | dom = minidom.parse(filename) 509 | except ExpatError: 510 | print_stderr( 511 | 'Invalid XML in %s', filename) 512 | return None 513 | except IOError as err: 514 | print_stderr( 515 | 'Error reading %s: %s', filename, err) 516 | return None 517 | return dom 518 | 519 | 520 | def writeXMLtoFile(node, path): 521 | '''Write XML dom node to file''' 522 | xml_string = node.toxml('utf-8') 523 | try: 524 | fileobject = open(path, mode='w') 525 | print(xml_string, file=fileobject) 526 | fileobject.close() 527 | except (OSError, IOError): 528 | print_stderr('Couldn\'t write XML to %s' % path) 529 | 530 | 531 | def remove_config_data_attribute(product_list): 532 | '''Wrapper to emulate previous behavior of remove-only only operation.''' 533 | check_or_remove_config_data_attribute(product_list, remove_attr=True) 534 | 535 | 536 | def check_or_remove_config_data_attribute( 537 | product_list, remove_attr=False, products=None, suppress_output=False): 538 | '''Loop through the type="config-data" attributes from the distribution 539 | options for a list of products. Return a list of products that have 540 | this attribute set or if `remove_attr` is specified then remove the 541 | attribute from the distribution file. 542 | 543 | This makes softwareupdate find and display updates like 544 | XProtectPlistConfigData and Gatekeeper Configuration Data, which it 545 | normally does not.''' 546 | if not products: 547 | products = getProductInfo() 548 | config_data_products = set() 549 | for key in product_list: 550 | if key in products: 551 | if products[key].get('CatalogEntry'): 552 | distributions = products[key]['CatalogEntry'].get( 553 | 'Distributions', {}) 554 | for lang in distributions.keys(): 555 | distPath = getLocalPathNameFromURL( 556 | products[key]['CatalogEntry']['Distributions'][lang]) 557 | if not os.path.exists(distPath): 558 | continue 559 | dom = readXMLfile(distPath) 560 | if dom: 561 | found_config_data = False 562 | option_elements = ( 563 | dom.getElementsByTagName('options') or []) 564 | for element in option_elements: 565 | if 'type' in element.attributes.keys(): 566 | if (element.attributes['type'].value 567 | == 'config-data'): 568 | found_config_data = True 569 | config_data_products.add(key) 570 | if remove_attr: 571 | element.removeAttribute('type') 572 | # done editing dom 573 | if found_config_data and remove_attr: 574 | try: 575 | writeXMLtoFile(dom, distPath) 576 | except (OSError, IOError): 577 | pass 578 | else: 579 | if not suppress_output: 580 | print_stdout('Updated dist: %s', distPath) 581 | elif not found_config_data: 582 | if not suppress_output: 583 | print_stdout('No config-data in %s', distPath) 584 | return list(config_data_products) 585 | 586 | LOGFILE = None 587 | def main(): 588 | '''Placeholder''' 589 | pass 590 | 591 | 592 | if __name__ == '__main__': 593 | main() 594 | -------------------------------------------------------------------------------- /code/repoutil: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # encoding: utf-8 3 | # 4 | # Copyright 2011 Disney Enterprises, Inc. All rights reserved 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are 8 | # met: 9 | 10 | # * Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions and the following disclaimer. 12 | 13 | # * Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions and the following disclaimer in 15 | # the documentation and/or other materials provided with the 16 | # distribution. 17 | 18 | # * The names "Disney", "Walt Disney Pictures", "Walt Disney Animation 19 | # Studios" or the names of its contributors may NOT be used to 20 | # endorse or promote products derived from this software without 21 | # specific prior written permission from Walt Disney Pictures. 22 | 23 | # Disclaimer: THIS SOFTWARE IS PROVIDED BY WALT DISNEY PICTURES AND 24 | # CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, 25 | # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS 26 | # FOR A PARTICULAR PURPOSE, NONINFRINGEMENT AND TITLE ARE DISCLAIMED. 27 | # IN NO EVENT SHALL WALT DISNEY PICTURES, THE COPYRIGHT HOLDER OR 28 | # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 29 | # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 30 | # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 31 | # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND BASED ON ANY 32 | # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 33 | # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 34 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 35 | 36 | '''A tool to replicate most of the functionality of 37 | Apple Software Update server''' 38 | 39 | import optparse 40 | import os 41 | import shutil 42 | 43 | from reposadolib import reposadocommon 44 | 45 | 46 | def deleteBranchCatalogs(branchname): 47 | '''Removes catalogs corresponding to a deleted branch''' 48 | for catalog_URL in reposadocommon.pref('AppleCatalogURLs'): 49 | localcatalogpath = reposadocommon.getLocalPathNameFromURL(catalog_URL) 50 | # now strip the '.sucatalog' bit from the name 51 | if localcatalogpath.endswith('.sucatalog'): 52 | localcatalogpath = localcatalogpath[0:-10] 53 | branchcatalogpath = localcatalogpath + '_' + branchname + '.sucatalog' 54 | if os.path.exists(branchcatalogpath): 55 | reposadocommon.print_stdout( 56 | 'Removing %s', os.path.basename(branchcatalogpath)) 57 | os.remove(branchcatalogpath) 58 | 59 | 60 | def getProductLocation(product, product_id): 61 | '''Returns local path to replicated product 62 | We pass in the product dictionary to avoid calling 63 | reposadocommon.getProductInfo(), which is slow.''' 64 | if not 'CatalogEntry' in product: 65 | # something is wrong with the product entry 66 | return None 67 | catalog_entry = product['CatalogEntry'] 68 | product_url = None 69 | if 'ServerMetadataURL' in catalog_entry: 70 | product_url = catalog_entry['ServerMetadataURL'] 71 | else: 72 | try: 73 | # get the URL for the first package in the Packages array 74 | product_url = catalog_entry['Packages'][0]['URL'] 75 | except (KeyError, IndexError): 76 | return None 77 | filepath = reposadocommon.getLocalPathNameFromURL(product_url) 78 | # return the directory this pkg is in 79 | return os.path.dirname(filepath) 80 | 81 | 82 | def getRestartNeeded(product): 83 | '''Returns "Yes" if all pkg_refs require a restart or shutdown, 84 | "No" if none do, and "Sometimes" if some do and some don't. 85 | Returns "UNKNOWN" if there is no pkg_ref data for the update.''' 86 | 87 | pkgs = product.get('pkg_refs', {}).keys() 88 | pkg_count = len(pkgs) 89 | if pkg_count == 0: 90 | return "UNKNOWN" 91 | restart_count = 0 92 | for pkg in pkgs: 93 | if 'RestartAction' in product['pkg_refs'][pkg]: 94 | restart_count += 1 95 | if restart_count == 0: 96 | # no pkgs require a restart/shutdown/logout 97 | return "No" 98 | elif restart_count == pkg_count: 99 | # all pkgs require a restart/shutdown/logout 100 | return "Yes" 101 | else: 102 | # some pkgs require a restart/shutdown/logout 103 | return "Sometimes" 104 | 105 | 106 | def print_info(key): 107 | '''Prints detail for a specific product''' 108 | products = reposadocommon.getProductInfo() 109 | if key in products: 110 | product = products[key] 111 | downloaded_products_list = reposadocommon.getDownloadStatus() 112 | if key in downloaded_products_list: 113 | status = "Downloaded" 114 | else: 115 | status = "Not downloaded" 116 | catalog_branches = reposadocommon.getCatalogBranches() 117 | branchlist = [branch for branch in catalog_branches.keys() 118 | if key in catalog_branches[branch]] 119 | 120 | reposadocommon.print_stdout('Product: %s', key) 121 | reposadocommon.print_stdout('Title: %s', product.get('title')) 122 | reposadocommon.print_stdout('Version: %s', product.get('version')) 123 | reposadocommon.print_stdout('Size: %s', 124 | reposadocommon.humanReadable(product.get('size', 0))) 125 | reposadocommon.print_stdout( 126 | 'Post Date: %s', product.get('PostDate')) 127 | reposadocommon.print_stdout( 128 | 'RestartNeeded: %s', getRestartNeeded(product)) 129 | if reposadocommon.pref('LocalCatalogURLBase'): 130 | # we're replicating products locally 131 | reposadocommon.print_stdout('Status: %s', status) 132 | if status == 'Downloaded': 133 | reposadocommon.print_stdout( 134 | 'Location: %s' % getProductLocation(product, key)) 135 | if products[key].get('AppleCatalogs'): 136 | reposadocommon.print_stdout('AppleCatalogs:') 137 | for catalog in product['AppleCatalogs']: 138 | reposadocommon.print_stdout(' %s', catalog) 139 | else: 140 | reposadocommon.print_stdout(' Product is deprecated.') 141 | if product.get('OriginalAppleCatalogs'): 142 | reposadocommon.print_stdout('OriginalAppleCatalogs:') 143 | for catalog in product['OriginalAppleCatalogs']: 144 | reposadocommon.print_stdout(' %s', catalog) 145 | reposadocommon.print_stdout('Branches:') 146 | if branchlist: 147 | for branch in branchlist: 148 | reposadocommon.print_stdout(' %s', branch) 149 | else: 150 | reposadocommon.print_stdout(' ') 151 | reposadocommon.print_stdout('HTML Description:') 152 | reposadocommon.print_stdout(product.get('description')) 153 | else: 154 | reposadocommon.print_stdout('No product id %s found.', key) 155 | 156 | 157 | def print_dist(key): 158 | '''Print the .dist file for a specific product for every language in 159 | PreferredLocalizations''' 160 | products = reposadocommon.getProductInfo() 161 | languages = reposadocommon.pref('PreferredLocalizations') 162 | if key in products: 163 | if products[key].get('CatalogEntry'): 164 | if products[key]['CatalogEntry'].get('Distributions'): 165 | for lang in languages: 166 | if products[key]['CatalogEntry']['Distributions'].get(lang): 167 | distPath = reposadocommon.getLocalPathNameFromURL( 168 | products[key]['CatalogEntry'][ 169 | 'Distributions'][lang]) 170 | try: 171 | distFd = open(distPath, 'r') 172 | distContents = distFd.read() 173 | distFd.close() 174 | reposadocommon.print_stdout(distContents) 175 | except (IOError, OSError), errorMsg: 176 | reposadocommon.print_stderr( 177 | 'Error getting %s dist file for product %s:\n%s' 178 | % (lang, key, errorMsg)) 179 | else: 180 | reposadocommon.print_stdout('No product id %s found.', key) 181 | 182 | 183 | def list_branches(): 184 | '''Prints catalog branch names''' 185 | catalog_branches = reposadocommon.getCatalogBranches() 186 | for key in catalog_branches.keys(): 187 | reposadocommon.print_stdout(key) 188 | 189 | 190 | def print_product_line(key, products, catalog_branches=None): 191 | '''Prints a line of product info''' 192 | if key in products: 193 | if not catalog_branches: 194 | branchlist = '' 195 | else: 196 | branchlist = [branch for branch in catalog_branches.keys() 197 | if key in catalog_branches[branch]] 198 | branchlist.sort() 199 | deprecation_state = '' 200 | if not products[key].get('AppleCatalogs'): 201 | # not in any Apple catalogs 202 | deprecation_state = '(Deprecated)' 203 | try: 204 | post_date = products[key].get('PostDate').strftime('%Y-%m-%d') 205 | except BaseException: 206 | post_date = 'None' 207 | reposadocommon.print_stdout( 208 | '%-15s %-50s %-10s %-10s %s %s', 209 | key, 210 | products[key].get('title'), 211 | products[key].get('version'), 212 | post_date, 213 | branchlist, 214 | deprecation_state) 215 | else: 216 | reposadocommon.print_stdout('%-15s ', key) 217 | 218 | 219 | def list_branch(branchname, sort_order='date', reverse_sort=False): 220 | '''List products in a given catalog branch''' 221 | catalog_branches = reposadocommon.getCatalogBranches() 222 | if branchname in catalog_branches: 223 | list_products(sort_order, reverse_sort, catalog_branches[branchname]) 224 | else: 225 | reposadocommon.print_stderr( 226 | 'ERROR: %s is not a valid branch name.' % branchname) 227 | 228 | 229 | def diff_branches(branch_list): 230 | '''Displays differences between two branches''' 231 | catalog_branches = reposadocommon.getCatalogBranches() 232 | for branch in branch_list: 233 | if not branch in catalog_branches: 234 | reposadocommon.print_stderr( 235 | 'ERROR: %s is not a valid branch name.' % branch) 236 | return 237 | branch1 = set(catalog_branches[branch_list[0]]) 238 | branch2 = set(catalog_branches[branch_list[1]]) 239 | unique_to_first = branch1 - branch2 240 | unique_to_second = branch2 - branch1 241 | if len(unique_to_first) == 0 and len(unique_to_second) == 0: 242 | reposadocommon.print_stdout( 243 | 'No differences between %s and %s.' % branch_list) 244 | else: 245 | reposadocommon.print_stdout('Unique to \'%s\':', branch_list[0]) 246 | if len(unique_to_first): 247 | list_products(list_of_productids=unique_to_first) 248 | else: 249 | reposadocommon.print_stdout('') 250 | reposadocommon.print_stdout('\nUnique to \'%s\':', branch_list[1]) 251 | if len(unique_to_second): 252 | list_products(list_of_productids=unique_to_second) 253 | else: 254 | reposadocommon.print_stdout('') 255 | 256 | 257 | def list_deprecated(sort_order='date', reverse_sort=False): 258 | '''Find products that are no longer referenced in Apple\'s catalogs''' 259 | products = reposadocommon.getProductInfo() 260 | list_of_productids = [key for key in products.keys() 261 | if not products[key].get('AppleCatalogs')] 262 | list_products(sort_order, reverse_sort, list_of_productids) 263 | 264 | 265 | def list_non_deprecated(sort_order='date', reverse_sort=False): 266 | '''Find products that are referenced in Apple\'s catalogs''' 267 | products = reposadocommon.getProductInfo() 268 | list_of_productids = [key for key in products.keys() 269 | if products[key].get('AppleCatalogs')] 270 | list_products(sort_order, reverse_sort, list_of_productids) 271 | 272 | 273 | def list_config_data(sort_order='date', reverse_sort=False): 274 | '''Find updates with \'type="config-data"\' attribute''' 275 | product_info = reposadocommon.getProductInfo() 276 | product_list = product_info.keys() 277 | matching_products = reposadocommon.check_or_remove_config_data_attribute( 278 | product_list, remove_attr=False, products=product_info, 279 | suppress_output=True) 280 | list_products(sort_order, reverse_sort, matching_products) 281 | 282 | 283 | def list_products(sort_order='date', reverse_sort=False, 284 | list_of_productids=None): 285 | '''Prints a list of Software Update products''' 286 | 287 | def sort_by_key(a, b): 288 | """Internal comparison function for use with sorting""" 289 | return cmp(a['sort_key'], b['sort_key']) 290 | 291 | sort_keys = {'date': 'PostDate', 292 | 'title': 'title', 293 | 'id': 'id'} 294 | 295 | sort_key = sort_keys.get(sort_order, 'PostDate') 296 | errormessages = [] 297 | products = reposadocommon.getProductInfo() 298 | catalog_branches = reposadocommon.getCatalogBranches() 299 | product_list = [] 300 | if list_of_productids == None: 301 | list_of_productids = products.keys() 302 | for productid in list_of_productids: 303 | if not productid in products: 304 | errormessages.append( 305 | 'Skipped product %s because it does not exist ' 306 | 'in the ProductInfo database.' % productid) 307 | continue 308 | product_dict = {} 309 | product_dict['key'] = productid 310 | if sort_key == 'id': 311 | product_dict['sort_key'] = productid 312 | else: 313 | try: 314 | product_dict['sort_key'] = products[productid][sort_key] 315 | except KeyError: 316 | errormessages.append( 317 | 'Product %s is missing the sort key %s -- ' 318 | 'Product info database may be incomplete' 319 | % (productid, sort_key)) 320 | continue 321 | product_list.append(product_dict) 322 | product_list.sort(sort_by_key) 323 | if reverse_sort: 324 | product_list.reverse() 325 | for product in product_list: 326 | print_product_line(product['key'], products, catalog_branches) 327 | for error in errormessages: 328 | reposadocommon.print_stderr('WARNING: %s' % error) 329 | 330 | 331 | def add_product_to_branch(parameters): 332 | '''Adds one or more products to a branch. Takes a list of strings. 333 | The last string must be the name of a branch catalog. All other 334 | strings must be product_ids.''' 335 | # sanity checking 336 | for item in parameters: 337 | if item.startswith('-'): 338 | reposadocommon.print_stderr( 339 | 'Ambiguous parameters: can\'t tell if ' 340 | '%s is a parameter or an option!', item) 341 | return 342 | branch_name = parameters[-1] 343 | product_id_list = parameters[0:-1] 344 | 345 | # remove all duplicate product ids 346 | product_id_list = list(set(product_id_list)) 347 | 348 | catalog_branches = reposadocommon.getCatalogBranches() 349 | if not branch_name in catalog_branches: 350 | reposadocommon.print_stderr('Catalog branch %s doesn\'t exist!', 351 | branch_name) 352 | return 353 | 354 | products = reposadocommon.getProductInfo() 355 | if 'all' in product_id_list: 356 | product_id_list = products.keys() 357 | elif 'non-deprecated' in product_id_list: 358 | product_id_list = [key for key in products.keys() 359 | if products[key].get('AppleCatalogs')] 360 | 361 | for product_id in product_id_list: 362 | if not product_id in products: 363 | reposadocommon.print_stderr( 364 | 'Product %s doesn\'t exist!', product_id) 365 | else: 366 | try: 367 | title = products[product_id]['title'] 368 | vers = products[product_id]['version'] 369 | except KeyError: 370 | reposadocommon.print_stderr( 371 | 'Product %s is missing a title or version!\n' 372 | 'Product info database may be incomplete.\n' 373 | 'Info for product:\n%s', 374 | product_id, products[product_id]) 375 | # skip this one and move on 376 | continue 377 | if product_id in catalog_branches[branch_name]: 378 | reposadocommon.print_stderr( 379 | '%s (%s-%s) is already in branch %s!', 380 | product_id, title, vers, branch_name) 381 | else: 382 | reposadocommon.print_stdout( 383 | 'Adding %s (%s-%s) to branch %s...', 384 | product_id, title, vers, branch_name) 385 | catalog_branches[branch_name].append(product_id) 386 | 387 | reposadocommon.writeCatalogBranches(catalog_branches) 388 | reposadocommon.writeAllBranchCatalogs() 389 | 390 | 391 | def remove_product_from_branch(parameters): 392 | '''Removes one or more products from a branch. Takes a list of strings. 393 | The last string must be the name of a branch catalog. All other 394 | strings must be product_ids.''' 395 | 396 | # sanity checking 397 | for item in parameters: 398 | if item.startswith('-'): 399 | reposadocommon.print_stderr( 400 | 'Ambiguous parameters: can\'t tell if ' 401 | '%s is a parameter or an option!', item) 402 | return 403 | 404 | branch_name = parameters[-1] 405 | product_id_list = parameters[0:-1] 406 | 407 | catalog_branches = reposadocommon.getCatalogBranches() 408 | if not branch_name in catalog_branches: 409 | reposadocommon.print_stderr( 410 | 'Catalog branch %s doesn\'t exist!', branch_name) 411 | return 412 | 413 | products = reposadocommon.getProductInfo() 414 | if 'deprecated' in product_id_list: 415 | product_id_list = [key for key in catalog_branches[branch_name] 416 | if not products[key].get('AppleCatalogs')] 417 | else: 418 | # remove all duplicate product ids 419 | product_id_list = list(set(product_id_list)) 420 | 421 | for product_id in product_id_list: 422 | if product_id in products: 423 | title = products[product_id].get('title') 424 | vers = products[product_id].get('version') 425 | else: 426 | reposadocommon.print_stderr( 427 | 'Product %s doesn\'t exist!', product_id) 428 | title = 'UNKNOWN' 429 | vers = 'UNKNOWN' 430 | if not product_id in catalog_branches[branch_name]: 431 | reposadocommon.print_stderr('%s (%s-%s) is not in branch %s!', 432 | product_id, title, vers, branch_name) 433 | continue 434 | 435 | reposadocommon.print_stdout('Removing %s (%s-%s) from branch %s...', 436 | product_id, title, vers, branch_name) 437 | catalog_branches[branch_name].remove(product_id) 438 | reposadocommon.writeCatalogBranches(catalog_branches) 439 | reposadocommon.writeAllBranchCatalogs() 440 | 441 | 442 | def purge_product(product_ids, force=False): 443 | '''Removes products from the ProductInfo.plist and purges their local 444 | replicas (if they exist). Warns and skips if a product is not deprecated 445 | or is in any branch, unless force == True. If force == True, product is 446 | also removed from all branches. This action is destructive and cannot be 447 | undone. 448 | product_ids is a list of productids.''' 449 | 450 | # sanity checking 451 | for item in product_ids: 452 | if item.startswith('-'): 453 | reposadocommon.print_stderr('Ambiguous parameters: can\'t tell if ' 454 | '%s is a parameter or an option!', item) 455 | return 456 | 457 | products = reposadocommon.getProductInfo() 458 | catalog_branches = reposadocommon.getCatalogBranches() 459 | downloaded_product_list = reposadocommon.getDownloadStatus() 460 | 461 | if 'all-deprecated' in product_ids: 462 | product_ids.remove('all-deprecated') 463 | deprecated_productids = [key for key in products.keys() 464 | if not products[key].get('AppleCatalogs')] 465 | product_ids.extend(deprecated_productids) 466 | 467 | # remove all duplicate product ids 468 | product_ids = list(set(product_ids)) 469 | 470 | for product_id in product_ids: 471 | if not product_id in products: 472 | reposadocommon.print_stderr( 473 | 'Product %s does not exist in the ProductInfo database. ' 474 | 'Skipping.', product_id) 475 | continue 476 | product = products[product_id] 477 | product_short_info = ( 478 | '%s (%s-%s)' 479 | % (product_id, product.get('title'), product.get('version'))) 480 | if product.get('AppleCatalogs') and not force: 481 | reposadocommon.print_stderr( 482 | 'WARNING: Product %s is in Apple catalogs:\n %s', 483 | product_short_info, '\n '.join(product['AppleCatalogs'])) 484 | reposadocommon.print_stderr('Skipping product %s', product_id) 485 | continue 486 | branches_with_product = [branch for branch in catalog_branches.keys() 487 | if product_id in catalog_branches[branch]] 488 | if branches_with_product: 489 | if not force: 490 | reposadocommon.print_stderr( 491 | 'WARNING: Product %s is in catalog branches:\n %s', 492 | product_short_info, '\n '.join(branches_with_product)) 493 | reposadocommon.print_stderr('Skipping product %s', product_id) 494 | continue 495 | else: 496 | # remove product from all branches 497 | for branch_name in branches_with_product: 498 | reposadocommon.print_stdout( 499 | 'Removing %s from branch %s...', 500 | product_short_info, branch_name) 501 | catalog_branches[branch_name].remove(product_id) 502 | 503 | local_copy = getProductLocation(product, product_id) 504 | if local_copy: 505 | # remove local replica 506 | reposadocommon.print_stdout( 507 | 'Removing replicated %s from %s...', 508 | product_short_info, local_copy) 509 | try: 510 | shutil.rmtree(local_copy) 511 | except (OSError, IOError), err: 512 | reposadocommon.print_stderr( 513 | 'Error: %s', err) 514 | # but not fatal, so keep going... 515 | # delete product from ProductInfo database 516 | del products[product_id] 517 | # delete product from downloaded product list 518 | if product_id in downloaded_product_list: 519 | downloaded_product_list.remove(product_id) 520 | 521 | # write out changed catalog branches, productInfo, 522 | # and rebuild our local and branch catalogs 523 | reposadocommon.writeDownloadStatus(downloaded_product_list) 524 | reposadocommon.writeCatalogBranches(catalog_branches) 525 | reposadocommon.writeProductInfo(products) 526 | reposadocommon.writeAllLocalCatalogs() 527 | 528 | 529 | def copy_branches(source_branch, dest_branch, force=False): 530 | '''Copies source_branch to dest_branch, replacing dest_branch''' 531 | # sanity checking 532 | for branch in [source_branch, dest_branch]: 533 | if branch.startswith('-'): 534 | reposadocommon.print_stderr( 535 | 'Ambiguous parameters: can\'t tell if %s is a branch name or' 536 | ' option!', branch) 537 | return 538 | catalog_branches = reposadocommon.getCatalogBranches() 539 | if not source_branch in catalog_branches: 540 | reposadocommon.print_stderr('Branch %s does not exist!', source_branch) 541 | return 542 | if dest_branch in catalog_branches and not force: 543 | answer = raw_input( 544 | 'Really replace contents of branch %s with branch %s? [y/n] ' 545 | % (dest_branch, source_branch)) 546 | if not answer.lower().startswith('y'): 547 | return 548 | catalog_branches[dest_branch] = catalog_branches[source_branch] 549 | reposadocommon.print_stdout('Copied contents of branch %s to branch %s.', 550 | source_branch, dest_branch) 551 | reposadocommon.writeCatalogBranches(catalog_branches) 552 | reposadocommon.writeAllBranchCatalogs() 553 | 554 | 555 | def delete_branch(branchname, force=False): 556 | '''Deletes a branch''' 557 | catalog_branches = reposadocommon.getCatalogBranches() 558 | if not branchname in catalog_branches: 559 | reposadocommon.print_stderr('Branch %s does not exist!', branchname) 560 | return 561 | if not force: 562 | answer = raw_input('Really remove branch %s? [y/n] ' % branchname) 563 | if not answer.lower().startswith('y'): 564 | return 565 | del catalog_branches[branchname] 566 | deleteBranchCatalogs(branchname) 567 | reposadocommon.writeCatalogBranches(catalog_branches) 568 | 569 | 570 | def new_branch(branchname): 571 | '''Creates a new empty branch''' 572 | catalog_branches = reposadocommon.getCatalogBranches() 573 | if branchname in catalog_branches: 574 | reposadocommon.print_stderr('Branch %s already exists!', branchname) 575 | return 576 | catalog_branches[branchname] = [] 577 | reposadocommon.writeCatalogBranches(catalog_branches) 578 | 579 | 580 | def configure(): 581 | '''Configures reposado preferences.''' 582 | reposadocommon.configure_prefs() 583 | 584 | 585 | def remove_config_data(product_ids): 586 | '''Remove the config-data attribute from product dist files''' 587 | if len(product_ids) == 1 and product_ids[0] == 'all': 588 | '''Removes the config-data attribute from all products''' 589 | reposadocommon.print_stdout( 590 | 'Checking all products for config-data attributes...') 591 | product_info = reposadocommon.getProductInfo() 592 | product_list = product_info.keys() 593 | updated_products = reposadocommon.check_or_remove_config_data_attribute( 594 | product_list, remove_attr=True, products=product_info, 595 | suppress_output=True) 596 | if updated_products: 597 | reposadocommon.print_stdout( 598 | 'config-data attribute removed from:') 599 | for key in updated_products: 600 | reposadocommon.print_stdout( 601 | ' %s: %s-%s', 602 | key, product_info[key]['title'], product_info[key]['version']) 603 | else: 604 | reposadocommon.print_stdout( 605 | 'No products with config-data attributes found.') 606 | else: 607 | reposadocommon.remove_config_data_attribute(product_ids) 608 | 609 | 610 | def main(): 611 | '''Main command processing''' 612 | 613 | p = optparse.OptionParser() 614 | p.set_usage('''Usage: %prog [options]''') 615 | #p.add_option('--sync', action='store_true', 616 | # help="""Synchronize Apple updates""") 617 | p.add_option('--configure', action='store_true', 618 | help='Configure Reposado preferences.') 619 | p.add_option('--products', '--updates', action='store_true', 620 | dest='products', 621 | help='List available updates.') 622 | p.add_option('--deprecated', action='store_true', 623 | help='List deprecated updates.') 624 | p.add_option('--non-deprecated', action='store_true', 625 | help='List non-deprecated updates.') 626 | p.add_option('--config-data', action='store_true', 627 | help="""List updates with 'type="config-data"' attribute""") 628 | p.add_option('--sort', metavar='SORT_ORDER', default='date', 629 | help='Sort list.\n' 630 | 'Available sort orders are: date, title, id.') 631 | p.add_option('--reverse', action='store_true', 632 | help='Reverse sort order.') 633 | p.add_option('--branches', '--catalogs', 634 | dest='list_branches', action='store_true', 635 | help='List available branch catalogs.') 636 | p.add_option('--new-branch', 637 | metavar='BRANCH_NAME', 638 | help='Create new empty branch BRANCH_NAME.') 639 | p.add_option('--delete-branch', 640 | metavar='BRANCH_NAME [--force]', 641 | help='Delete branch BRANCH_NAME.') 642 | p.add_option('--copy-branch', nargs=2, 643 | metavar='SOURCE_BRANCH DEST_BRANCH [--force]', 644 | help='Copy all items from SOURCE_BRANCH to ' 645 | 'DEST_BRANCH. If DEST_BRANCH does not exist, ' 646 | 'it will be created.') 647 | p.add_option('--list-branch', '--list-catalog', 648 | dest='branch', 649 | metavar='BRANCH_NAME', 650 | help='List updates in branch BRANCH_NAME.') 651 | p.add_option('--diff', '--diff-branch', '--diff-branches', 652 | dest='diff_branch', nargs=2, 653 | metavar='BRANCH1_NAME BRANCH2_NAME', 654 | help='Display differences between two branches.') 655 | p.add_option('--product-info', '--info', metavar='PRODUCT_ID', 656 | dest='info', 657 | help='Print info on a specific update.') 658 | p.add_option('--product-dist', '--dist', metavar='PRODUCT_ID', 659 | dest='dist', 660 | help='Print the contents of the .dist file for a specific ' 661 | 'update.') 662 | p.add_option('--add-product', '--add-products', 663 | '--add-update', '--add-updates', '--add', 664 | dest='add_product', nargs=2, 665 | metavar='PRODUCT_ID [PRODUCT_ID ...] BRANCH_NAME', 666 | help='Add one or more PRODUCT_IDs to catalog branch ' 667 | 'BRANCH_NAME. --add-product all BRANCH_NAME will add ' 668 | 'all cached products, including deprecated products, to ' 669 | 'catalog BRANCH_NAME. --add-product non-deprecated ' 670 | 'BRANCH_NAME will add all non-deprecated products to ' 671 | 'catalog BRANCH_NAME.') 672 | p.add_option('--remove-product', '--remove-products', nargs=2, 673 | metavar='PRODUCT_ID [PRODUCT_ID ...] BRANCH_NAME', 674 | help='Remove one or more PRODUCT_IDs from catalog branch ' 675 | 'BRANCH_NAME. --remove-product deprecated will remove ' 676 | 'all deprecated products from BRANCH_NAME.') 677 | p.add_option('--remove-config-data', 678 | metavar='PRODUCT_ID [PRODUCT_ID ...]', 679 | help='Remove the \'type="config-data"\' attribute from one or ' 680 | 'more PRODUCT_IDs.') 681 | p.add_option('--purge-product', '--purge-products', 682 | metavar='PRODUCT_ID [PRODUCT_ID ...] [--force]', 683 | help='Purge one or more PRODUCT_IDs from product ' 684 | 'database and remove any locally replicated version.') 685 | p.add_option('--force', action='store_true', 686 | help='Force purge of product, force copy or force delete a ' 687 | 'branch. Must be used with --purge-product, --copy-branch ' 688 | 'or --delete-branch options.') 689 | 690 | options, arguments = p.parse_args() 691 | 692 | if options.configure: 693 | configure() 694 | if options.products: 695 | list_products(sort_order=options.sort, reverse_sort=options.reverse) 696 | if options.deprecated: 697 | list_deprecated(sort_order=options.sort, reverse_sort=options.reverse) 698 | if options.non_deprecated: 699 | list_non_deprecated(sort_order=options.sort, reverse_sort=options.reverse) 700 | if options.config_data: 701 | list_config_data(sort_order=options.sort, reverse_sort=options.reverse) 702 | if options.branch: 703 | list_branch(options.branch, sort_order=options.sort, 704 | reverse_sort=options.reverse) 705 | if options.list_branches: 706 | list_branches() 707 | if options.info: 708 | print_info(options.info) 709 | if options.dist: 710 | print_dist(options.dist) 711 | if options.new_branch: 712 | new_branch(options.new_branch) 713 | if options.copy_branch: 714 | copy_branches( 715 | options.copy_branch[0], options.copy_branch[1], force=options.force) 716 | if options.delete_branch: 717 | delete_branch(options.delete_branch, force=options.force) 718 | if options.diff_branch: 719 | diff_branches(options.diff_branch) 720 | if options.add_product: 721 | params = list(options.add_product) 722 | params.extend(arguments) 723 | add_product_to_branch(params) 724 | if options.remove_product: 725 | params = list(options.remove_product) 726 | params.extend(arguments) 727 | remove_product_from_branch(params) 728 | if options.purge_product: 729 | product_ids = [options.purge_product] 730 | product_ids.extend(arguments) 731 | purge_product(product_ids, force=options.force) 732 | if options.remove_config_data: 733 | params = [options.remove_config_data] 734 | params.extend(arguments) 735 | remove_config_data(params) 736 | 737 | 738 | if __name__ == '__main__': 739 | main() 740 | -------------------------------------------------------------------------------- /docs/URL_rewrites.md: -------------------------------------------------------------------------------- 1 | # URL rewrites 2 | 3 | ## Introduction 4 | 5 | Apple's Software Update service has the ability to offer the "correct" catalog to a client that requests simply "index.sucatalog". This ability was first added with OS X Server 10.6.6, and involved the use of Apache's mod_rewrite. 6 | 7 | This feature is handy, as it greatly simplifies client configuration. You no longer need to take client OS version into consideration when setting the CatalogURL for the client. 8 | 9 | Lion Server's Software Update service has a similar capability, but this is done via a CGI instead. 10 | 11 | Since Reposado doesn't handle the web-serving part of offering Apple software updates, Reposado itself cannot provide a similar feature: instead you must configure your web server to do URL rewrites, or write your own CGI to provide this functionality. 12 | 13 | 14 | ## Apache2 mod_rewrite example 15 | 16 | If you are using Apache2 as your webserver, you may be able to configure mod_rewrite to return the "correct" OS-specific sucatalog: 17 | 18 | Here is an example .htaccess file you could place at the root of your Reposado repo: 19 | 20 | RewriteEngine On 21 | Options FollowSymLinks 22 | RewriteBase / 23 | RewriteCond %{HTTP_USER_AGENT} Darwin/8 24 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/index$1.sucatalog [L] 25 | RewriteCond %{HTTP_USER_AGENT} Darwin/9 26 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-leopard.merged-1$1.sucatalog [L] 27 | RewriteCond %{HTTP_USER_AGENT} Darwin/10 28 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-leopard-snowleopard.merged-1$1.sucatalog [L] 29 | RewriteCond %{HTTP_USER_AGENT} Darwin/11 30 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 31 | RewriteCond %{HTTP_USER_AGENT} Darwin/12 32 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 33 | RewriteCond %{HTTP_USER_AGENT} Darwin/13 34 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 35 | RewriteCond %{HTTP_USER_AGENT} Darwin/14 36 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 37 | RewriteCond %{HTTP_USER_AGENT} Darwin/15 38 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 39 | RewriteCond %{HTTP_USER_AGENT} Darwin/16 40 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 41 | RewriteCond %{HTTP_USER_AGENT} Darwin/17 42 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 43 | RewriteCond %{HTTP_USER_AGENT} Darwin/18 44 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 45 | RewriteCond %{HTTP_USER_AGENT} Darwin/19 46 | RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L] 47 | 48 | 49 | This requires Apache2 to be configured to actually pay attention to mod_rewrite rules in .htaccess files. See your Apache and mod_rewrite documentation for details. 50 | 51 | Note that the format for these rules is specific for use in an .htaccess file. The rules would need to be changed if in the main Apache config file. 52 | 53 | 54 | ## Other web servers 55 | 56 | Other web servers support URL rewriting; the specifics are slightly different, but the general concepts are similar. 57 | 58 | ### nginx 59 | 60 | Heig Gregorian has contributed this example of an Nginx configuration. This is an excerpt from the 'server' section of his Nginx config: 61 | 62 | 63 | 64 | if ( $http_user_agent ~ "Darwin/8" ){ 65 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/index$1.sucatalog last; 66 | } 67 | if ( $http_user_agent ~ "Darwin/9" ){ 68 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-leopard.merged-1$1.sucatalog last; 69 | } 70 | if ( $http_user_agent ~ "Darwin/10" ){ 71 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-leopard-snowleopard.merged-1$1.sucatalog last; 72 | } 73 | if ( $http_user_agent ~ "Darwin/11" ){ 74 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-lion-snowleopard-leopard.merged-1$1.sucatalog last; 75 | } 76 | if ( $http_user_agent ~ "Darwin/12" ){ 77 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 78 | } 79 | if ( $http_user_agent ~ "Darwin/13" ){ 80 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 81 | } 82 | if ( $http_user_agent ~ "Darwin/14" ){ 83 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 84 | } 85 | if ( $http_user_agent ~ "Darwin/15" ){ 86 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 87 | } 88 | if ( $http_user_agent ~ "Darwin/16" ){ 89 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 90 | } 91 | if ( $http_user_agent ~ "Darwin/17" ){ 92 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 93 | } 94 | if ( $http_user_agent ~ "Darwin/18" ){ 95 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 96 | } 97 | if ( $http_user_agent ~ "Darwin/19" ){ 98 | rewrite ^/index(.*)\.sucatalog$ /content/catalogs/others/index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog last; 99 | } 100 | 101 | Again, consult Nginx documentation for further information about URL rewriting. 102 | 103 | ### IIS 104 | 105 | This was tested successfully on Microsoft IIS 7. IIS 7 does not have URL Rewrite support installed by default, but it can be added by installing a module for the specific version of IIS. 106 | 107 | IIS does not support serving certain Apple specific extensions, so the following MIME Types will need to be added. This is example is taken from the web.config in the root of the reposado directory: 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | Here is an example web.config file located in the html directory of a reposado repository: 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | ## Testing 157 | 158 | Reposado can help you test your URL rewrites. When catalogs are written out to disk, a key named "_CatalogName" is added to the catalog with the base filename of the catalog. This allows you to verify that the catalog being returned is the one you expect. 159 | 160 | For example, I want to test that I'm getting the Lion catalog: 161 | 162 | % curl --user-agent "Darwin/11.4.0" http://su.example.com/index_testing.sucatalog > /tmp/testing 163 | % Total % Received % Xferd Average Speed Time Time Time Current 164 | Dload Upload Total Spent Left Speed 165 | 100 1024k 100 1024k 0 0 25.1M 0 --:--:-- --:--:-- --:--:-- 32.2M 166 | 167 | % tail -5 /tmp/testing 168 | 169 | _CatalogName 170 | index-lion-snowleopard-leopard.merged-1_testing.sucatalog 171 | 172 | 173 | 174 | The _CatalogName is "index-lion-snowleopard-leopard.merged-1_testing.sucatalog", which is what I expect. 175 | 176 | I can repeat the test for Snow Leopard, this time against the "release" branch: 177 | 178 | % curl --user-agent "Darwin/10.8.0" http://su.example.com/index_release.sucatalog > /tmp/release 179 | % Total % Received % Xferd Average Speed Time Time Time Current 180 | Dload Upload Total Spent Left Speed 181 | 100 912k 100 912k 0 0 29.7M 0 --:--:-- --:--:-- --:--:-- 31.8M 182 | 183 | % tail -5 /tmp/release 184 | 185 | _CatalogName 186 | index-leopard-snowleopard.merged-1_release.sucatalog 187 | 188 | 189 | 190 | 191 | ## Conclusion 192 | 193 | Adding URL rewriting to your Reposado web server configuration makes client configuration much simpler, as you no longer have to worry about which OS the client is running. 194 | -------------------------------------------------------------------------------- /docs/client_configuration.md: -------------------------------------------------------------------------------- 1 | # Configuring clients to use your Reposado server 2 | 3 | If you've never used the Software Update Service on Mac OS X Server, you may be unfamiliar with configuring Mac OS X clients to use a Software Update Server other than Apple's "main" server. 4 | 5 | This setting may be controlled by setting the value of CatalogURL in /Library/Preferences/com.apple.SoftwareUpdate.plist. This is commonly done using the command-line 'defaults' tool: 6 | 7 | sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL 8 | 9 | where \ is the URL to the catalog file. 10 | 11 | This preference can also be managed via configuration profile, which in turn can be deployed via MDM. 12 | 13 | In recent versions of macOS you may also use `sudo softwareupdate --set-catalog ` and for the versions that support it, that might be a "better" method than using `defaults write`. 14 | 15 | You can use URL rewriting on your web server to simplify client configuration. See [URL_rewrites.md](./URL_rewrites.md) for more on this. 16 | 17 | 18 | ## Tiger Clients 19 | 20 | Tiger clients should use a CatalogURL of the form: 21 | 22 | http://su.yourorg.com/content/catalogs/index.sucatalog 23 | 24 | This will offer the same updates as if the client was pointed directly at Apple's servers. If you are using branch catalogs to filter available updates, or to offer deprecated updates, the CatalogURL will take the form of: 25 | 26 | http://su.yourorg.com/content/catalogs/index_.sucatalog 27 | 28 | where \ is the name of one of the branch catalogs you've created. 29 | 30 | 31 | ## Leopard Clients 32 | 33 | Leopard clients should use a CatalogURL of the form: 34 | 35 | http://su.yourorg.com/content/catalogs/others/index-leopard.merged-1.sucatalog 36 | 37 | Branch CatalogURLs take the form of: 38 | 39 | http://su.yourorg.com/content/catalogs/others/index-leopard.merged-1_.sucatalog 40 | 41 | 42 | ## Snow Leopard Clients 43 | 44 | Snow Leopard clients should use a CatalogURL of the form: 45 | 46 | http://su.yourorg.com/content/catalogs/others/index-leopard-snowleopard.merged-1.sucatalog 47 | 48 | Branch CatalogURLs take the form of: 49 | 50 | http://su.yourorg.com/content/catalogs/others/index-leopard-snowleopard.merged-1_.sucatalog 51 | 52 | 53 | ## Lion Clients 54 | 55 | Lion clients should use a CatalogURL of the form: 56 | 57 | http://su.yourorg.com/content/catalogs/others/index-lion-snowleopard-leopard.merged-1.sucatalog 58 | 59 | Branch CatalogURLs take the form of: 60 | 61 | http://su.yourorg.com/content/catalogs/others/index-lion-snowleopard-leopard.merged-1_.sucatalog 62 | 63 | 64 | ## Mountain Lion Clients 65 | 66 | Mountain Lion clients should use a CatalogURL of the form: 67 | 68 | http://su.yourorg.com/content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 69 | 70 | Branch CatalogURLs take the form of: 71 | 72 | http://su.yourorg.com/content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1_.sucatalog 73 | 74 | 75 | ## Mavericks Clients 76 | 77 | Mavericks clients should use a CatalogURL of the form: 78 | 79 | http://su.yourorg.com/content/catalogs/others/index-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 80 | 81 | Branch CatalogURLs take the form of: 82 | 83 | http://su.yourorg.com/content/catalogs/others/index-10.9-mountainlion-lion-snowleopard-leopard.merged-1_.sucatalog 84 | 85 | 86 | ## Yosemite Clients 87 | 88 | Yosemite clients should use a CatalogURL of the form: 89 | 90 | http://su.yourorg.com/content/catalogs/others/index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 91 | 92 | Branch CatalogURLs take the form of: 93 | 94 | http://su.yourorg.com/content/catalogs/others/index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1_.sucatalog 95 | 96 | 97 | ## El Capitan Clients 98 | 99 | El Capitan clients should use a CatalogURL of the form: 100 | 101 | http://su.yourorg.com/content/catalogs/others/index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 102 | 103 | Branch CatalogURLs take the form of: 104 | 105 | http://su.yourorg.com/content/catalogs/others/index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1_.sucatalog 106 | 107 | 108 | ## Sierra Clients 109 | 110 | Sierra clients should use a CatalogURL of the form: 111 | 112 | http://su.yourorg.com/content/catalogs/others/index-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 113 | 114 | Branch CatalogURLs take the form of: 115 | 116 | http://su.yourorg.com/content/catalogs/others/index-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1_.sucatalog 117 | 118 | 119 | ## High Sierra Clients 120 | 121 | High Sierra clients should use a CatalogURL of the form: 122 | 123 | http://su.yourorg.com/content/catalogs/others/index-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 124 | 125 | Branch CatalogURLs take the form of: 126 | 127 | http://su.yourorg.com/content/catalogs/others/index-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1_.sucatalog 128 | 129 | 130 | ## Mojave Clients 131 | 132 | Testing with the beta releases indicates that Mojave's `softwareupdate` requires the use of https. It also does Extended Validation of TLS certs by default. To disable this, you can set a preference in the com.apple.SoftwareUpdate preferences domain: 133 | 134 | `sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate SUDisableEVCheck -bool YES` 135 | (Or use a configuration profile to manage this preference.) 136 | 137 | Mojave clients should use a CatalogURL of the form: 138 | 139 | https://su.yourorg.com/content/catalogs/others/index-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 140 | 141 | Branch CatalogURLs take the form of: 142 | 143 | https://su.yourorg.com/content/catalogs/others/index-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1_.sucatalog 144 | 145 | (More importantly, this means the softwareupdate catalog and products must be served via https -- if you are replicating products locally, this means the LocalCatalogURLBase must also be an https URL.) 146 | -------------------------------------------------------------------------------- /docs/getting_started.md: -------------------------------------------------------------------------------- 1 | # Getting Started with Reposado 2 | 3 | What you need: 4 | 5 | - The Reposado tools 6 | - Python 2.5-2.7 with `plistlib`. (Reposado has been tested with Python 2.6, but should work with 2.5-2.7 as long as `plistlib` is available. `plistlib` was included as a Mac-specific library with Python 2.5, and for all platforms with Python 2.6.) 7 | - `curl` binary 8 | - A web server 9 | - Storage space for the catalogs and update packages. If you are replicating the update packages, you'll need approximately 300GB of space as of March 2016. The need for space grows as additional updates are released by Apple. If you are only replicating catalogs and not the updates themselves, you'll probably need less than 100MB of space, though the exact amount of space needed depends on the number of branch catalogs you create. 10 | 11 | 1. Download the Reposado tools, typically by doing a git clone of the repo, or downloading the source from GitHub. The tools are in the code directory. The tools do not need to be any place in particular to operate; you may put them wherever you like. 12 | 13 | 2. Create a directory in which to store replicated catalogs and updates, and another to store Reposado metadata. These may share a parent directory, like so: 14 | 15 | /Volumes/data/reposado/html 16 | /Volumes/data/reposado/metadata 17 | 18 | Make sure you have enough space for the replicated catalogs and updates. Make sure these directories are writable by the user `repo_sync` will run as, and readable by the user your webserver runs as. 19 | 20 | 3. Configure your web server to serve the contents of the updates root directory you created ("/Volumes/data/reposado/html" in the example above). If you are planning to replicate and serve the actual update packages as well as the catalogs, make note of the URL needed to access the updates root directory via HTTP. This will be the LocalCatalogURLBase when configuring Reposado in the next step. 21 | 22 | 4. Configure Reposado by running: 23 | 24 | ``` 25 | repoutil --configure 26 | ``` 27 | 28 | You'll be asked three questions: 29 | 30 | ``` 31 | Path to store replicated catalogs and updates: 32 | Path to store Reposado metadata: 33 | Base URL for your local Software Update Service 34 | (Example: http://su.your.org -- leave empty if you are not replicating updates): 35 | ``` 36 | 37 | Answer appropriately using your decisions in steps 2 and 3 above. 38 | See [reposado_preferences.md](./reposado_preferences.md) for more details on Reposado preferences. 39 | 40 | NOTE: if you enter the paths by dragging folders into a Terminal window from the Finder, be aware that the Finder adds an extra space to the end of pathnames. Delete this space or your results may not be what you expect. You can always run `repoutil --configure` again to check and modify the configuration if needed. 41 | 42 | 5. Run `repo_sync` to replicate update catalogs and (optionally) update packages to the UpdatesRootDir directory. The first time you do this it may take several hours to complete if you are replicating packages as well as catalogs. 43 | 44 | 6. Test things so far by visiting a catalog URL in your browser. If http://su.myorg.com is the URL for the updates root directory you created earlier, then http://su.myorg.com/content/catalogs/others/index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 45 | is the CatalogURL for the El Capitan updates catalog. You should see a plist version of the updates catalog displayed in your browser. 46 | 47 | 7. Next test: run `softwareupdate` on a client, again pointing it to your Catalog URL: 48 | 49 | ``` 50 | softwareupdate --set-catalog "http://su.myorg.com/content/catalogs/others/index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog" 51 | softwareupdate -l 52 | ``` 53 | 54 | NOTE: commands are different in versions of OS X prior to 10.9 Mavericks: 55 | 56 | ``` 57 | softwareupdate -l --CatalogURL "http://su.myorg.com/content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog" 58 | ``` 59 | 60 | If there are no errors, you've successfully configured Reposado and successfully replicated Apple Software Updates. 61 | -------------------------------------------------------------------------------- /docs/reference.md: -------------------------------------------------------------------------------- 1 | # repo_sync 2 | 3 | *repo_sync* downloads software update catalogs from Apple's servers and (if configured) also downloads software update packages to be served from your web server. 4 | 5 | 6 | ## Options 7 | 8 | --recheck 9 | 10 | Normally, once downloaded, software update packages are not checked for changes against Apple's servers. If you'd like to reverify all downloaded packages (at the cost of increased execution time and additional Internet bandwidth usage), include the --recheck option when calling *repo_sync*. 11 | 12 | --log PATH_TO_LOGFILE 13 | 14 | Specify a log file to redirect all output to. 15 | 16 | 17 | # repoutil 18 | 19 | *repoutil* is a tool for getting info about available updates, managing branch catalogs, and configuring Reposado. 20 | 21 | 22 | ## Options 23 | 24 | 25 | ### CONFIGURING REPOSADO 26 | 27 | repoutil --configure 28 | 29 | Configure key preferences for Reposado. Note this command does not allow you to edit all available Reposado preferences. See [reposado_preferences](./reposado_preferences.md) for more information. 30 | 31 | 32 | ### LISTING AVAILABLE PRODUCTS (UPDATES) 33 | 34 | repoutil --products [--sort ] [--reverse] 35 | repoutil --updates [--sort ] [--reverse] 36 | 37 | List available updates. You may optionally sort by date, title, or id, and optionally reverse the sort order. The default sort order is by post date, so that newest updates are listed last. 38 | Example: 39 | 40 | repoutil --products 41 | 061-1688 Final Cut Express 2.0.3 2005-04-12 [] 42 | 061-1704 iMovie HD Update 5.0.2 2005-04-14 [] 43 | 061-1702 iDVD Update 5.0.1 2005-04-14 [] 44 | [...] 45 | zzz041-0453 Security Update 2011-002 1.0 2011-04-14 [] 46 | zzz041-0654 Security Update 2011-002 1.0 2011-04-14 [] 47 | zzz041-0656 Security Update 2011-002 1.0 2011-04-14 [] 48 | 041-0560 Safari 5.0.5 2011-04-14 ['testing'] 49 | zzzz041-0565 Safari 5.0.5 2011-04-14 ['testing'] 50 | zzzz041-0531 iTunes 10.2.2 2011-04-18 ['testing'] 51 | zzzz041-0532 iTunes 10.2.2 2011-04-18 ['testing'] 52 | 53 | 54 | ### LISTING DEPRECATED UPDATES 55 | 56 | repoutil --deprecated 57 | repoutil --non-deprecated 58 | 59 | List deprecated or non-deprecated updates. Deprecated products are updates that are no longer available from Apple. 60 | They may have been withdrawn, or have been superseded by newer versions. Example: 61 | 62 | repoutil --deprecated 63 | 031-41757 Digital Camera RAW Compatibility Update 6.17 2015-10-19 ['testing'] (Deprecated) 64 | 031-34805 Safari 9.0.1 2015-10-21 ['testing'] (Deprecated) 65 | zzzz031-36002 iTunes 12.3.1 2015-10-21 ['testing'] (Deprecated) 66 | 67 | ### LISTING CRITICAL UPDATES 68 | 69 | repoutil --config-data 70 | 71 | List critical updates. These are updates that are marked as "config-data" by Apple. 72 | Depending on the setting on the client, those updates can be automatically installed without 73 | any user interaction. 74 | 75 | repoutil --config-data 76 | 031-12170 Gatekeeper Configuration Data 1.0 2015-02-09 ['testing'] 77 | 031-38723 XProtectPlistConfigData 1.0 2015-10-19 [] (Deprecated) 78 | 031-52942 XProtectPlistConfigData 1.0 2016-03-05 ['testing'] 79 | 80 | 81 | ### LISTING AVAILABLE BRANCH CATALOGS 82 | 83 | repoutil --branches 84 | 85 | List available branch catalogs. Example: 86 | 87 | repoutil --branches 88 | release 89 | testing 90 | 91 | 92 | ### CREATING A NEW BRANCH CATALOG 93 | 94 | repoutil --new-branch BRANCH_NAME 95 | 96 | Creates a new empty branch named BRANCH_NAME. Example: 97 | 98 | repoutil --new-branch testing 99 | 100 | 101 | ### DELETING A BRANCH CATALOG 102 | 103 | repoutil --delete-branch BRANCH_NAME 104 | 105 | Deletes the branch named BRANCH_NAME. 106 | 107 | 108 | ### COPYING ALL PRODUCTS FROM ONE BRANCH TO ANOTHER 109 | 110 | repoutil --copy-branch SOURCE_BRANCH DEST_BRANCH 111 | 112 | Copies all items from SOURCE_BRANCH to DEST_BRANCH, completely replacing the contents of DEST_BRANCH with the contents of SOURCE_BRANCH. If DEST_BRANCH does not exist, it will be created. 113 | 114 | 115 | ### LISTING PRODUCTS IN A BRANCH CATALOG 116 | 117 | repoutil --list-branch BRANCH_NAME [--sort ] [--reverse] 118 | repoutil --list-catalog BRANCH_NAME [--sort ] [--reverse] 119 | 120 | List updates in branch BRANCH_NAME. You may optionally sort these. 121 | 122 | repoutil--list-branch testing 123 | zzzz041-0565 Safari 5.0.5 2011-04-14 ['testing'] 124 | 041-0560 Safari 5.0.5 2011-04-14 ['testing'] 125 | zzzz041-0532 iTunes 10.2.2 2011-04-18 ['testing'] 126 | zzzz041-0531 iTunes 10.2.2 2011-04-18 ['testing'] 127 | 128 | 129 | ### GETTING DETAILED INFO ON A PRODUCT 130 | 131 | repoutil --product-info PRODUCT_ID 132 | repoutil --info PRODUCT_ID 133 | 134 | Prints detailed info on a specific update. Example: 135 | 136 | repoutil --info 041-0560 137 | Product: 041-0560 138 | Title: Safari 139 | Version: 5.0.5 140 | Size: 47.1 MB 141 | Post Date: 2011-04-14 17:13:16 142 | AppleCatalogs: 143 | http://swscan.apple.com/content/catalogs/index.sucatalog 144 | http://swscan.apple.com/content/catalogs/others/index-leopard.merged-1.sucatalog 145 | http://swscan.apple.com/content/catalogs/others/index-leopard-snowleopard.merged-1.sucatalog 146 | Branches: 147 | testing 148 | HTML Description: 149 | 150 | 151 | 152 | 153 | 154 | 155 | 159 | 160 | 161 |

This update is recommended for all Safari users and includes the latest security updates.

162 | 163 |

For information on the security content of this update, please visit this website: http://support.apple.com/kb/HT1222.

164 |

165 | 166 | 167 | 168 | 169 | ### ADDING PRODUCTS TO A BRANCH CATALOG 170 | 171 | repoutil --add-product PRODUCT_ID [PRODUCT_ID ...] BRANCH_NAME 172 | repoutil --add-update=PRODUCT_ID [PRODUCT_ID ...] BRANCH_NAME 173 | repoutil --add=PRODUCT_ID [PRODUCT_ID ...] BRANCH_NAME 174 | 175 | Add one or more PRODUCT_IDs to catalog branch BRANCH_NAME. 176 | You may add all cached products, optionally including deprecated products to a branch catalog by specifying 'non-deprecated' or 'all': 177 | 178 | repoutil --add-product non-deprecated BRANCH_NAME 179 | repoutil --add-product all BRANCH_NAME 180 | 181 | 182 | ### REMOVING PRODUCTS FROM A BRANCH CATALOG 183 | 184 | repoutil --remove-product=PRODUCT_ID [PRODUCT_ID ...] BRANCH_NAME 185 | 186 | Remove one or more PRODUCT_IDs from catalog branch BRANCH_NAME. You may remove all deprecated products from a branch by specifying 'deprecated': 187 | 188 | repoutil --remove-product deprecated BRANCH_NAME 189 | 190 | 191 | ### PURGING PRODUCTS ENTIRELY 192 | 193 | repoutil --purge-product [ --force 204 | 205 | Note: if you purge a deprecated product, it is deleted forever and cannot be redownloaded from Apple. 206 | -------------------------------------------------------------------------------- /docs/reposado_metadata.md: -------------------------------------------------------------------------------- 1 | # Reposado metadata 2 | 3 | This document describes the metadata files generated by Reposado. 4 | 5 | Reposado's metadata files are stored in the directory you specify in Reposado's preferences as the UpdatesMetadataDir. 6 | 7 | Currently, the following files are generated and stored in this directory: 8 | 9 | - CatalogBranches.plist 10 | 11 | This plist has an entry for each catalog branch you've defined. The name of each branch is a key, and each branch is a list of strings. Each string is a ProductID. Example: 12 | 13 | release 14 | 15 | zzzz061-9636 16 | zzzz041-0306 17 | 18 | testing 19 | 20 | zzzz061-9636 21 | zzzz041-0306 22 | 23 | 24 | 25 | - DownloadStatus.plist 26 | 27 | As each product is successfully downloaded, its ProductID is added to this list. Items not in this list are excluded from the local catalogs. 28 | 29 | 30 | - ETags.plist 31 | 32 | ETags for each item downloaded from Apple's servers are stored here. Reposado uses these to only download changed items from Apple's servers. 33 | 34 | 35 | - ProductInfo.plist 36 | 37 | This is a cache of information for every product seen in any catalog downloaded from Apple's servers. This info is used when listing available products and when offering deprecated products that have been removed from Apple catalogs. 38 | -------------------------------------------------------------------------------- /docs/reposado_operation.md: -------------------------------------------------------------------------------- 1 | # Reposado operation 2 | 3 | A basic guide to Reposado operation. 4 | 5 | See [getting_started.md](./getting_started.md) for initial setup, configuration, and testing. 6 | 7 | Once you have successfully set up and configured Reposado, you have a local mirror of Apple's Software Update servers. Your clients can use the locally mirrored catalogs (and optionally, update packages) for Apple updates. 8 | 9 | You'll almost certainly want to run the *repo_sync* tool periodically to download catalog and package updates. The exact mechanism by which you might do this varies from platform to platform. On a Linux or other flavor of Unix machine, you'd typically add a cron job to do this. You may also do this on OS X or OS X Server, or you can implement a periodic job, or a launchd task. You may run this as frequently or infrequently as you wish, but Apple's tools sync with Apple once a day, so you might consider that as a guide. 10 | 11 | If all you want or need is a local replica of Apple's Software Updates in order to conserve your organization's Internet bandwidth usage, you need not do anything other than the tasks of the initial configuration and setting up periodic execution of the sync script. If you'd like to be able to manage which updates are made available to your client machines, the *repoutil* tool provides the means to do so. 12 | 13 | 14 | ## CatalogURLs 15 | 16 | By default, Reposado replicates these update catalogs from Apple's servers: 17 | 18 | http://swscan.apple.com/content/catalogs/index.sucatalog 19 | http://swscan.apple.com/content/catalogs/index-1.sucatalog 20 | http://swscan.apple.com/content/catalogs/others/index-leopard.merged-1.sucatalog 21 | http://swscan.apple.com/content/catalogs/others/index-leopard-snowleopard.merged-1.sucatalog 22 | http://swscan.apple.com/content/catalogs/others/index-lion-snowleopard-leopard.merged-1.sucatalog 23 | http://swscan.apple.com/content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 24 | 25 | These are the update catalogs for Tiger, Leopard, and Snow Leopard, Lion and Mountain Lion clients, respectively. You may point any or all of your client machines to the replicated versions of these catalogs. The clients will get the latest updates from Apple as soon as they've been replicated to your Reposado server. 26 | 27 | If you'd like to control the availability of updates, you must create "branch" catalogs, which contain subsets of the available updates. You then point some or all of your clients to these branch catalogs instead of the "raw" catalogs that come from Apple. 28 | 29 | 30 | ## Creating branch catalogs 31 | 32 | Start by creating an empty branch: 33 | 34 | repoutil --new-branch testing 35 | 36 | This creates a new, empty branch named "testing". This name is appended to the "raw" apple catalog names, so that the CatalogURLs become something like: 37 | 38 | http://su.myorg.com/content/catalogs/index_testing.sucatalog 39 | http://su.myorg.com/content/catalogs/others/index-leopard.merged-1_testing.sucatalog 40 | http://su.myorg.com/content/catalogs/others/index-leopard-snowleopard.merged-1_testing.sucatalog 41 | 42 | ..but not until you've added something to the testing branch. 43 | 44 | 45 | ## Adding products to a branch catalog 46 | 47 | Get a list of products: 48 | 49 | repoutil --products 50 | ...long list omitted for brevity... 51 | 041-0560 Safari 5.0.5 2011-04-14 [] 52 | zzzz041-0565 Safari 5.0.5 2011-04-14 [] 53 | zzzz041-0531 iTunes 10.2.2 2011-04-18 [] 54 | zzzz041-0532 iTunes 10.2.2 2011-04-18 [] 55 | 56 | Add both Safaris and iTunes to testing: 57 | 58 | repoutil --add-product 041-0560 zzzz041-0565 zzzz041-0531 zzzz041-0532 testing 59 | Adding 041-0560 (Safari-5.0.5) to branch testing... 60 | Adding zzzz041-0565 (Safari-5.0.5) to branch testing... 61 | Adding zzzz041-0531 (iTunes-10.2.2) to branch testing... 62 | Adding zzzz041-0532 (iTunes-10.2.2) to branch testing... 63 | 64 | And now the testing catalogs are available at URLs similar to those listed above, and the testing catalogs offer only Safari and iTunes. 65 | 66 | 67 | ## Removing products from a branch catalog 68 | 69 | You can remove products from branch catalogs: 70 | 71 | repoutil --remove-product zzzz041-0531 zzzz041-0532 testing 72 | Removing zzzz041-0531 (iTunes-10.2.2) from branch testing... 73 | Removing zzzz041-0532 (iTunes-10.2.2) from branch testing... 74 | 75 | would remove both iTunes 10.2.2. 76 | 77 | You can list the contents of branch catalogs: 78 | 79 | repoutil --list-branch testing 80 | 041-0560 Safari 5.0.5 2011-04-14 ['testing'] 81 | zzzz041-0565 Safari 5.0.5 2011-04-14 ['testing'] 82 | 83 | and copy one branch to another: 84 | 85 | repoutil --copy-branch testing release 86 | Really replace contents of branch release with branch testing? [y/n] y 87 | Copied contents of branch testing to branch release. 88 | 89 | 90 | ## One possible branch catalog workflow 91 | 92 | A very small number of your machines are configured to use the "raw" catalogs from Apple. As new updates are released, after a short delay (a day or so?) you add them to the "testing" branch. Your "testing" group of machines are configured to use the "testing" CatalogURLs for their updates. After a time of testing to make sure there are no issues, you add new updates to the "release" branch. Most machines in your organization are configured to use the "release" CatalogURLs. 93 | 94 | In this way, new updates are tested before being released to the majority of your machines. 95 | 96 | Another use for branch catalogs: if you had a set of machines that must remain on a specific OS release for compatibility with a specific application, you could create one or more special branch catalogs that contained no Mac OS X updates, but only updates to Safari, iTunes, the iLife and iWork apps. In this way you could update the other applications while leaving the OS itself at a specific version. 97 | 98 | 99 | ## Deprecated products 100 | 101 | Items from Apple's Software Update service can become "deprecated". This commonly occurs when a newer version of an update is made available. For example, when Apple releases a new update to iTunes, all older iTunes updates are deprecated and no longer available from Apple's Software Update servers. Similarly, new updates for Mac OS X cause older updates to be deprecated. 102 | 103 | This behavior can sometimes present problems for system administrators. Let's say you had made the 10.6.6 update available to all the machines you manage, and some had updated and some had not yet. Apple then released the Mac OS X 10.6.7 update, which causes the 10.6.6 update to disappear from Apple's update servers. If you are not ready to move to Mac OS X 10.6.7 (because you need testing time), but some of your machines are still running 10.6.5 or earlier, if you are using Apple's Software Update Service there is no way to update those machines to 10.6.6 using Apple's tools. 104 | 105 | However, Reposado caches all updates it downloads from Apple and does not automatically remove deprecated updates. This enables you to continue to offer deprecated updates in a branch catalog until you are ready to replace the deprecated update with the new version. 106 | 107 | This feature is also a responsibility -- it is the admin's responsibility to remove deprecated products from branch catalogs when adding their updated versions to the same branch. Deprecated products are tagged as such in product listings: 108 | 109 | repoutil --list-branch testing 110 | zzzz061-9636 iTunes 10.2 2011-03-02 ['release', 'testing'] (Deprecated) 111 | zzzz041-0306 iTunes 10.2 2011-03-02 ['release', 'testing'] (Deprecated) 112 | 113 | When adding a newer version of iTunes to the testing or release branches, you'd want to be certain to remove these older, deprecated versions. 114 | 115 | 116 | ## Purging deprecated products 117 | 118 | If you no longer want to keep deprecated products in your repository you can remove these products one by one as described under "Removing products from a branch catalog". If you have not removed deprecated products in a while your repository may include many deprecated products, however. Removing them individually could be very tedious. Instead, you can remove all deprecated products from your repository at the same time by substituting 'all-deprecated' for the product. 119 | 120 | 121 | -------------------------------------------------------------------------------- /docs/reposado_preferences.md: -------------------------------------------------------------------------------- 1 | # Reposado preferences 2 | 3 | Reposado's configuration is defined in a plist file named **preferences.plist** located in the same directory as the *repoutil* script. 4 | 5 | Two key/values are required: 6 | 7 | - UpdatesRootDir 8 | 9 | A string providing a path where the catalogs and update packages should be stored. Example: 10 | 11 | /Volumes/data/reposado/html 12 | 13 | - UpdatesMetadataDir 14 | 15 | A string providing a path where metadata used by reposado should be should be stored. Example: 16 | 17 | /Volumes/data/reposado/metadata 18 | 19 | If you are replicating the updates as well as the catalogs, you must also include: 20 | 21 | - LocalCatalogURLBase 22 | 23 | This is the "root" URL for your local Software Update repo. Reposado will re-write all product URLs in the update catalogs to use this root URL. For example, a LocalCatalogURLBase of "http://su.myorg.com" will result in a Snow Leopard update catalog URLs like: 24 | 25 | http://su.myorg.com/content/catalogs/others/index-leopard-snowleopard.merged-1.sucatalog 26 | 27 | If LocalCatalogURLBase is undefined, only Apple catalogs will be replicated and the URLs will not be re-written. The actual Software Update packages will not be downloaded. This allows you to have custom catalogs for Apple Software Update, but clients will still download the actual update packages from Apple's servers. If Reposado is configured this way, you will not be able to offer deprecated updates to clients. 28 | 29 | Note: if you are serving your softwareupdate repo on a non-standard port (standard ports are 80 for http and 443 for https) the alternate port is part of the LocalCatalogURLBase. An example is "http://su.myorg.com:8088" 30 | 31 | *repoutil --config* will allow you to quickly and easily edit the above three values (UpdatesRootDir, UpdatesMetadataDir, LocalCatalogURLBase). 32 | 33 | 34 | ## Optional keys 35 | 36 | 37 | The following keys are optional and may be defined in preferences.plist for special configurations: 38 | 39 | - AdditionalCurlOptions 40 | 41 | This is an array of strings that will be used as part of a configuration file passed to curl. A common use for this is to configure HTTP proxy information if needed for your site. Example: 42 | 43 | AdditionalCurlOptions 44 | 45 | proxy = "web-proxy.yourcompany.com:8080" 46 | 47 | 48 | See the curl documentation for available options and formatting. 49 | 50 | - AppleCatalogURLs 51 | 52 | This is an array of strings that specify the Apple SUS catalog URLs to replicate. If this is undefined, it defaults to: 53 | 54 | AppleCatalogURLs 55 | 56 | http://swscan.apple.com/content/catalogs/index.sucatalog 57 | http://swscan.apple.com/content/catalogs/index-1.sucatalog 58 | http://swscan.apple.com/content/catalogs/others/index-leopard.merged-1.sucatalog 59 | http://swscan.apple.com/content/catalogs/others/index-leopard-snowleopard.merged-1.sucatalog 60 | http://swscan.apple.com/content/catalogs/others/index-lion-snowleopard-leopard.merged-1.sucatalog 61 | http://swscan.apple.com/content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 62 | https://swscan.apple.com/content/catalogs/others/index-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 63 | https://swscan.apple.com/content/catalogs/others/index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 64 | https://swscan.apple.com/content/catalogs/others/index-10.15-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 65 | https://swscan.apple.com/content/catalogs/others/index-10.14-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 66 | https://swscan.apple.com/content/catalogs/others/index-10.13-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 67 | https://swscan.apple.com/content/catalogs/others/index-10.12-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 68 | https://swscan.apple.com/content/catalogs/others/index-10.11-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog 69 | 70 | 71 | As of the last update to this document, this is the current set of available Apple Software Update catalogs. 72 | 73 | - PreferredLocalizations 74 | 75 | A list of preferred language localizations for software update descriptions. Defaults to: 76 | 77 | PreferredLocalizations 78 | 79 | English 80 | en 81 | 82 | 83 | - CurlPath 84 | Path to the curl binary tool. Defaults to: 85 | 86 | CurlPath 87 | /usr/bin/curl 88 | 89 | 90 | - RepoSyncLogFile 91 | 92 | Path to a log file for *repo_sync* output. 93 | 94 | Example: 95 | 96 | RepoSyncLogFile 97 | /var/log/reposado_sync.log 98 | 99 | Defaults to no log file. 100 | 101 | - HumanReadableSizes 102 | 103 | Enable human-readable file sizes in download messages e.g. KB, MB. 104 | 105 | Example: 106 | 107 | HumanReadableSizes 108 | 109 | 110 | Defaults to displaying size in bytes. 111 | 112 | 113 | ## Example preferences.plist 114 | 115 | 116 | 117 | 118 | 119 | LocalCatalogURLBase 120 | http://su.myorg.com 121 | UpdatesRootDir 122 | /Volumes/data/reposado/html 123 | UpdatesMetadataDir 124 | /Volumes/data/reposado/metadata 125 | 126 | 127 | -------------------------------------------------------------------------------- /docs/reposado_py2exe.md: -------------------------------------------------------------------------------- 1 | # Using reposade with py2exe 2 | 3 | To successfully use reposado with [py2exe](http://www.py2exe.org/) (a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation) some minor modification were made to the original code. 4 | 5 | You just need to install py2exe and run 6 | 7 | python setup.py py2exe 8 | 9 | in the main directory were reposados setup.py resides. 10 | Of course you have to supply a working curl binary for windows, but that is just one file that you can put anywhere (just be sure to specify the path) 11 | 12 | You will end up with a folder dist in which you find all files that you need to run it without a Python installation. Especially the two main scripts *repo_sync* and *repoutil* are converted to Windows executables. You can run them just from the command line or wherever you want. 13 | 14 | reposados configuration file **preferences.plist** will always be created and searched in the main directory were these two exes reside, regardless from where you ran the commands. -------------------------------------------------------------------------------- /other/reposado.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wdas/reposado/1361303c2bb35284338a4a89acb25e16c36420f3/other/reposado.jpg -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | from distutils.core import setup 4 | 5 | if sys.platform == 'win32': 6 | import py2exe 7 | 8 | # Utility function to read the README file. 9 | # Used for the long_description. It's nice, because now 1) we have a top level 10 | # README file and 2) it's easier to type in the README file than to put a raw 11 | # string in below ... 12 | def read(fname): 13 | return open(os.path.join(os.path.dirname(__file__), fname)).read() 14 | 15 | if sys.platform == 'win32': 16 | setup( 17 | name = "reposado", 18 | version = "git", 19 | author = "Greg Neagle", 20 | author_email = "reposado@googlegroups.com", 21 | maintainer = "Brent B", 22 | maintainer_email = "brent.bb+py@gmail.com", 23 | description = ("Host Apple Software Updates on the hardware and OS of your choice."), 24 | license = "BSD", 25 | keywords = "apple software update repository", 26 | url = "https://github.com/wdas/reposado", 27 | packages=['reposadolib'], 28 | package_dir={'reposadolib': 'code/reposadolib'}, 29 | scripts=["code/repo_sync","code/repoutil"], 30 | long_description=read('README.md'), 31 | classifiers=[ 32 | "Intended Audience :: System Administrators", 33 | "Development Status :: 1 - Alpha", 34 | "Topic :: Utilities", 35 | "License :: OSI Approved :: BSD License", 36 | "Topic :: System :: Archiving :: Mirroring", 37 | "Topic :: System :: Installation/Setup", 38 | ], 39 | console=["code/repo_sync","code/repoutil"], 40 | ) 41 | else: 42 | setup( 43 | name = "reposado", 44 | version = "git", 45 | author = "Greg Neagle", 46 | author_email = "reposado@googlegroups.com", 47 | maintainer = "Brent B", 48 | maintainer_email = "brent.bb+py@gmail.com", 49 | description = ("Host Apple Software Updates on the hardware and OS of your choice."), 50 | license = "BSD", 51 | keywords = "apple software update repository", 52 | url = "https://github.com/wdas/reposado", 53 | packages=['reposadolib'], 54 | package_dir={'reposadolib': 'code/reposadolib'}, 55 | scripts=["code/repo_sync","code/repoutil"], 56 | long_description=read('README.md'), 57 | classifiers=[ 58 | "Intended Audience :: System Administrators", 59 | "Development Status :: 1 - Alpha", 60 | "Topic :: Utilities", 61 | "License :: OSI Approved :: BSD License", 62 | "Topic :: System :: Archiving :: Mirroring", 63 | "Topic :: System :: Installation/Setup", 64 | ], 65 | ) 66 | --------------------------------------------------------------------------------