├── .gitignore ├── README.md ├── catalogs ├── 10.10.0.catalog ├── 10.10.1.catalog ├── 10.10.2.catalog ├── 10.10.3.catalog ├── 10.10.4.catalog ├── 10.9.5.catalog └── example.catalog ├── stew └── uptodate /.gitignore: -------------------------------------------------------------------------------- 1 | build* 2 | cache* 3 | log* 4 | output* 5 | tmp* 6 | .stew_config 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | stew 2 | ======= 3 | 4 | This script processes catalog(s) of packages, verifies the sha1 hash of each package, downloads any missing or updated packages, 5 | and compiles a new image consisting of a given base OS X installer and the list of packages. 6 | 7 | Requirements 8 | ------------ 9 | 10 | + 10.9.x and above (you must build your images on a matching system) 11 | + python 2.7 12 | + InstallESD.dmg must be renamed in the format OSVERS_OSBUILD_InstallESD.dmg 13 | ``` 14 | InstallESD.dmg -> 10.9.2_13C64_InstallESD.dmg 15 | ``` 16 | 17 | Usage 18 | ----- 19 | 20 | Usage: sudo ./stew [options] 21 | 22 | Options: 23 | -h, --help show this help message and exit 24 | -b CATALOG, --build=CATALOG 25 | Specify catalog to process 26 | -c, --configure Set up or recreate configuration file 27 | -u PACKAGE, --upload=PACKAGE 28 | Upload package to webserver 29 | -C FILENAME, --checksum=FILENAME 30 | Return checksum of a cached package 31 | 32 | 33 | Configuration 34 | ------------- 35 | 36 | When you run ```stew``` for the first time, or with the ```-c``` option, you will be asked to populate a config file. This file is stored at ```~/.stew_config``` and will be recreated any time you use the ```-c``` option. This facilitates the uploading of packages using ```scp```, so setting up a key-based login for your webserver is highly recommended. 37 | 38 | You will be asked to provide the following information: 39 | 40 | + The FQDN of your webserver (example: mypackagerepo.mycorp.com) 41 | + The full path of your remote repo folder (example: /var/www/html/packages) 42 | + the web repo is just one single folder, which will contain all OS Installers, packages, and disk images 43 | + The login user for your webserver (example: stewuser) 44 | 45 | Uploading packages 46 | ------------------ 47 | 48 | ```stew``` supports both the pkg and dmg format for processing packages. You may simply upload packages with the ```stew -u``` option, or you may wish to copy the companion ```uptodate``` script to your $PATH and use that. ```uptodate``` also offers the ability to list catalogs and automates the editing and updating of catalog files: 49 | 50 | usage: uptodate [-h] [-p PACKAGE] [-c CATALOG] [-l LIST_CATALOG] 51 | 52 | optional arguments: 53 | -h, --help show this help message and exit 54 | -p PACKAGE, --package PACKAGE 55 | /path/to/package 56 | -c CATALOG, --catalog CATALOG 57 | /path/to/catalog 58 | -l LIST_CATALOG, --list_catalog LIST_CATALOG 59 | /path/to/catalog 60 | 61 | For example, to upload a package to your webserver, and update a catalog: 62 | 63 | uptodate -p /path/to/package -c /path/to/catalog 64 | 65 | To update a catalog's metadata, such as volume name and output name: 66 | 67 | uptodate -c /path/to/catalog 68 | 69 | Storing packages locally 70 | ------------------------ 71 | 72 | You do not have to use a web repo to store packages; you can store everything locally like an animal if you choose. You may simply answer the config questions with bogus info, and instead of uploading packages to the server option, place your packages in the cache folder and use the ```-C``` option to return the sha1 hash with which you can populate your catalog(s). 73 | 74 | ./stew -C ./cache/ 75 | 76 | Catalogs 77 | -------- 78 | 79 | Catalogs are json files describing information about your image and passing packages to ```stew``` for processing. Each build requires two catalogs: one providing information about the output name, volume name, and third party packages, and another describing the base OS build number and any available Apple updates. You will reference the os-catalog you want to use inside your custom catalog. For example, here is a custom catalog: 80 | 81 | { 82 | "os-catalog": "10.9.2.catalog", 83 | "volume-name": "OSX10.9.2_13C64", 84 | "output-name": "OSX10.9.2_13C64.dmg", 85 | "packages": [ 86 | [ 87 | "create_luser-1.0.pkg", 88 | "ccb2794e33cc7a3399c5c955da1b6c917ecb1bbe" 89 | ], 90 | [ 91 | "apple_setup_done.pkg", 92 | "3622b983e4d5d718aaf6b2f51fae060e737e1e3f" 93 | ], 94 | [ 95 | "command_line_tools_os_x_mavericks_for_xcode__late_october_2013.dmg", 96 | "cd325ee0b1720064292f2c86687449d0992f245b" 97 | ] 98 | ] 99 | } 100 | 101 | This catalog calls out '10.9.2.catalog' as its os-catalog. An os-catalog might look like this: 102 | 103 | { 104 | "os-installer": "10.9.0_13A603_InstallESD.dmg", 105 | "packages": [ 106 | [ 107 | "iTunes11.1.5.dmg", 108 | "d731cabbe1f9213491f3169921a45986ac944e58" 109 | ], 110 | [ 111 | "JavaForOSX2013-05.dmg", 112 | "ce78f9a916b91ec408c933bd0bde5973ca8a2dc4" 113 | ] 114 | ] 115 | } 116 | 117 | 118 | ```stew``` gathers this information, and processes all packages in the os-catalog before moving on to your defined catalog. 119 | 120 | If you do not include a ```volume-name``` definition, the default will be "Macintosh HD". If you do not include an ```output-name``` definition, the default will be the $OSVERS of your installer filename (see above). 121 | 122 | Runtime 123 | ------- 124 | 125 | Pass your catalog with the ```-b``` option, and ```stew``` will process the catalog and create an image. You can give it the relative path to the catalog file or simply name it. 126 | 127 | sudo ./stew -b example.catalog 128 | sudo ./stew -b example 129 | sudo ./stew -b catalogs/example.catalog 130 | 131 | ```stew``` caches the base sparseimage of an OS and saves it to the cache directory, which will speed up subsequent builds. 132 | 133 | Credits 134 | ------- 135 | 136 | This script was inspired by the awesome work by the [Google Mac Ops team](https://code.google.com/p/google-macops/source/browse/#svn%2Ftrunk%2Fcan_haz_image) and the wonderful [InstaDMG project](https://code.google.com/p/instadmg/). 137 | 138 | Also, much credit goes to @MaverValp and his outstanding [AutoDMG application](https://github.com/MagerValp/AutoDMG) (which you should definitely use instead of my half-baked scripts), and his patience with my infinite complaining. 139 | 140 | License 141 | ------- 142 | 143 | Copyright 2015 Joseph Chilcote 144 | 145 | Licensed under the Apache License, Version 2.0 (the "License"); 146 | you may not use this file except in compliance with the License. 147 | You may obtain a copy of the License at 148 | 149 | http://www.apache.org/licenses/LICENSE-2.0 150 | 151 | Unless required by applicable law or agreed to in writing, software 152 | distributed under the License is distributed on an "AS IS" BASIS, 153 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 154 | See the License for the specific language governing permissions and 155 | limitations under the License. 156 | -------------------------------------------------------------------------------- /catalogs/10.10.0.catalog: -------------------------------------------------------------------------------- 1 | { 2 | "os-installer": "10.10.0_14A389_InstallESD.dmg", 3 | "packages": [ 4 | [ 5 | "iTunes12.0.1.dmg", 6 | "0b0b6b00dbca55e272fab2fca3f59536cf119a8a" 7 | ], 8 | [ 9 | "Safari8.0.2Yosemite.pkg", 10 | "9f8ee68744a8ba405850f1d288812e8d8cbcae1e" 11 | ], 12 | [ 13 | "NTPUpdateYosemite.pkg", 14 | "9f240e99264165dcb278ddfc5aa38f0a0c24566f" 15 | ] 16 | ] 17 | } -------------------------------------------------------------------------------- /catalogs/10.10.1.catalog: -------------------------------------------------------------------------------- 1 | { 2 | "os-installer": "10.10.1_14B25_InstallESD.dmg", 3 | "packages": [ 4 | [ 5 | "iTunes12.0.1.dmg", 6 | "0b0b6b00dbca55e272fab2fca3f59536cf119a8a" 7 | ], 8 | [ 9 | "Safari8.0.2Yosemite.pkg", 10 | "9f8ee68744a8ba405850f1d288812e8d8cbcae1e" 11 | ], 12 | [ 13 | "NTPUpdateYosemite.pkg", 14 | "9f240e99264165dcb278ddfc5aa38f0a0c24566f" 15 | ] 16 | ] 17 | } -------------------------------------------------------------------------------- /catalogs/10.10.2.catalog: -------------------------------------------------------------------------------- 1 | { 2 | "os-installer": "10.10.2_14C109_InstallESD.dmg", 3 | "packages": [ 4 | [ 5 | "RemoteDesktopClient.pkg", 6 | "ecd973d95cb380a0746f0bcefd0e4e7b9e35ffc1" 7 | ], 8 | [ 9 | "iTunes12.1.dmg", 10 | "8da2bb4dcf428b68a34d3cd97f13f3ecadec6cca" 11 | ], 12 | [ 13 | "SecUpd2015-003Yosemite.dmg", 14 | "b3ab2951a420509debb437e4d47167a3f855100e" 15 | ] 16 | ] 17 | } -------------------------------------------------------------------------------- /catalogs/10.10.3.catalog: -------------------------------------------------------------------------------- 1 | { 2 | "os-installer": "10.10.3_14D2134_InstallESD.dmg", 3 | "packages": [ 4 | [ 5 | "itunes12.2.dmg", 6 | "e7fbbd56bfa6d1423d631d7de1e8957b5ff0d9a8" 7 | ], 8 | [ 9 | "Safari8.0.6Yosemite.pkg", 10 | "7f38285421b5e22edbcf512cd6286a14033bbd2c" 11 | ], 12 | [ 13 | "RAWCameraUpdate6.pkg", 14 | "e376a22824a633caab5669560214bd916a37a380" 15 | ] 16 | ] 17 | } -------------------------------------------------------------------------------- /catalogs/10.10.4.catalog: -------------------------------------------------------------------------------- 1 | { 2 | "os-installer": "10.10.4_14E46_InstallESD.dmg", 3 | "packages": [ 4 | [ 5 | "RemoteDesktopClient.pkg", 6 | "45f0b439a8841cc9604e8c0dbaa4f3c171258340" 7 | ], 8 | [ 9 | "itunes12.2.dmg", 10 | "e7fbbd56bfa6d1423d631d7de1e8957b5ff0d9a8" 11 | ] 12 | ] 13 | } -------------------------------------------------------------------------------- /catalogs/10.9.5.catalog: -------------------------------------------------------------------------------- 1 | { 2 | "os-installer": "10.9.5_13F34_InstallESD.dmg", 3 | "packages": [ 4 | [ 5 | "iTunes12.0.1.dmg", 6 | "0b0b6b00dbca55e272fab2fca3f59536cf119a8a" 7 | ], 8 | [ 9 | "Safari7.1.2Mavericks.pkg", 10 | "d83073b1277dd6c1ab94d7dceff090f16990281d" 11 | ], 12 | [ 13 | "iBooksDelta.pkg", 14 | "9b075180c00fe27f4a7471d7b16f80c6ca370e8f" 15 | ], 16 | [ 17 | "JavaForOSX2014-001.dmg", 18 | "0dc67455ff90ae0d20001a018465d3929f0a5334" 19 | ], 20 | [ 21 | "SecUpd2014-005Mavericks.dmg", 22 | "30fb47f161f9c7d0f74694fad660bb70c3c3b810" 23 | ], 24 | [ 25 | "NTPUpdateMavericks.pkg", 26 | "a1dd0e1e1d47ef4ef7d6f48dcc85f5ad6f2e9892" 27 | ] 28 | ] 29 | } -------------------------------------------------------------------------------- /catalogs/example.catalog: -------------------------------------------------------------------------------- 1 | { 2 | "volume-name": "OS10.10.1_14B25.dmg", 3 | "os-catalog": "10.10.1.catalog", 4 | "packages": [ 5 | [ 6 | "create_luser-1.0.pkg", 7 | "ccb2794e33cc7a3399c5c955da1b6c917ecb1bbe" 8 | ], 9 | [ 10 | "apple_setup_done.pkg", 11 | "3622b983e4d5d718aaf6b2f51fae060e737e1e3f" 12 | ], 13 | [ 14 | "commandlinetoolsosx10.10forxcode6.1.1", 15 | "5a6f9d1eb02bce4f522e6f93e22713a94daa600d" 16 | ] 17 | ], 18 | "output-name": "OS10.10.1_14B25" 19 | } 20 | -------------------------------------------------------------------------------- /stew: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python2.7 2 | """ 3 | Base image creation script. 4 | 5 | This script processes catalog(s) of packages, verifies the sha1 6 | hash of each package, downloads any missing or updated packages, 7 | and compiles a new image consisiting of a given base OS X installer 8 | and the list of packages. 9 | 10 | (Using a webserver to store packages and installers is encouraged, 11 | but not required.) 12 | 13 | Usage: sudo ./stew [options] 14 | 15 | Options: 16 | -h, --help show this help message and exit 17 | -b CATALOG, --build=CATALOG 18 | Specify catalog to process 19 | -c, --configure Set up or recreate configuration file 20 | -u PACKAGE, --upload=PACKAGE 21 | Upload package to webserver 22 | -C FILENAME, --checksum=FILENAME 23 | Return checksum of a cached package 24 | """ 25 | 26 | ############################################################################## 27 | # Copyright 2015 Joseph Chilcote 28 | # 29 | # Licensed under the Apache License, Version 2.0 (the "License"); you may not 30 | # use this file except in compliance with the License. You may obtain a copy 31 | # of the License at 32 | # 33 | # http://www.apache.org/licenses/LICENSE-2.0 34 | # 35 | # Unless required by applicable law or agreed to in writing, software 36 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 37 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 38 | # License for the specific language governing permissions and limitations 39 | # under the License. 40 | ############################################################################## 41 | 42 | __author__ = 'Joseph Chilcote (chilcote@gmail.com)' 43 | __version__ = '1.0.0' 44 | 45 | import os 46 | import re 47 | import sys 48 | import json 49 | import shutil 50 | import hashlib 51 | import urllib2 52 | import logging 53 | import datetime 54 | import platform 55 | import subprocess 56 | from optparse import OptionParser 57 | from SystemConfiguration import SCDynamicStoreCopyConsoleUser 58 | 59 | current_os = platform.mac_ver()[0] 60 | CONFIG = os.path.join(os.getenv('HOME'), '.stew_config') 61 | BUILD = os.path.join(os.getcwd(), 'build') 62 | CACHE = os.path.join(os.getcwd(), 'cache') 63 | LOG = os.path.join(os.getcwd(), 'log') 64 | OUTPUT = os.path.join(os.getcwd(), 'output') 65 | project_dirs = [BUILD, CACHE, LOG, OUTPUT] 66 | timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M') 67 | log_file = os.path.join(LOG, '%s.log' % timestamp) 68 | env = os.environ.copy() 69 | env['CM_BUILD'] = 'CM_BUILD' 70 | env['COMMAND_LINE_INSTALL'] = '1' 71 | 72 | class Stew(object): 73 | """Object for building the image.""" 74 | 75 | def __init__(self, volume_name, output_name, config, 76 | os_installer, pkgs, BUILD): 77 | self.cwd = os.getcwd() 78 | self.build = BUILD 79 | self.volume_name = volume_name 80 | self.output_name = output_name 81 | self.config = config 82 | self.os_installer = os_installer 83 | self.pkgs = pkgs 84 | #self.timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M') 85 | self.os_version = run_cmd(['sw_vers'])[0].split('\t')[2][:4] 86 | self.installer_choices = '%s_InstallerChoices.xml' % self.os_version 87 | self.osbuild = self.os_installer.split("_")[1] 88 | self.sb_cache = os.path.join(CACHE, self.osbuild) + '.sparsebundle' 89 | self.sb_cache_exists = False 90 | 91 | def setup_build_folder(self): 92 | """Creates the build directory if missing.""" 93 | try: 94 | os.mkdir(self.build) 95 | except OSError: 96 | try: 97 | os.mkdir(os.path.dirname(self.build)) 98 | os.mkdir(self.build) 99 | except OSError: 100 | logging.error('Unable to create build folder %s' % self.build) 101 | 102 | def create_sparsebundle(self, sb_cache=None): 103 | """Creates the sparsebundle on which to install the base system.""" 104 | print "Preparing build environment..." 105 | sparsebundle_path = os.path.join(self.build, 'stew.sparsebundle') 106 | if os.path.exists(sparsebundle_path): 107 | shutil.rmtree(sparsebundle_path) 108 | if not sb_cache: 109 | cmd = ['hdiutil', 'create', sparsebundle_path, '-size', '30G', 110 | '-volname', self.volume_name, '-layout', 'GPTSPUD', 111 | '-fs', 'JHFS+', '-mode','775', '-uid', '0', 112 | '-gid', '80'] 113 | (unused_stdout, stderr, unused_rc) = run_cmd(cmd) 114 | if stderr: 115 | logging.warning('Failed to create %s: %s' % (sparsebundle_path, 116 | stderr)) 117 | raise SystemExit 118 | else: 119 | return sparsebundle_path 120 | else: 121 | logging.debug('Found sparsebundle cache: %s' % sb_cache) 122 | logging.debug('Copying %s to %s' % (sb_cache, sparsebundle_path)) 123 | shutil.copytree(os.path.join(CACHE, sb_cache), 124 | sparsebundle_path) 125 | return sparsebundle_path 126 | 127 | def mount_sparsebundle(self, sparsebundle): 128 | """Mounts the sparsebundle to prepare for install.""" 129 | cmd = ['hdiutil', 'attach', '-owners', 'on', '-nobrowse', 130 | '-noverify', '-noautoopen', sparsebundle] 131 | logging.debug('Mounting %s' % sparsebundle) 132 | (stdout, stderr, unused_rc) = run_cmd(cmd) 133 | if stderr: 134 | logging.error('Unable to mount %s: %s' % (sparsebundle, stderr)) 135 | else: 136 | if self.sb_cache_exists: 137 | return stdout.split('\n')[-3].split('\t')[-1] 138 | else: 139 | return stdout.split('\n')[-2].split('\t')[-1] 140 | 141 | def mount_installer(self): 142 | """Mounts the InstallDMG volume""" 143 | baseimage_path = os.path.join(CACHE, self.os_installer) 144 | cmd = ['hdiutil', 'attach', '-nobrowse','-noverify', 145 | '-noautoopen', baseimage_path] 146 | logging.debug('Mounting %s' % baseimage_path) 147 | (stdout, stderr, unused_rc) = run_cmd(cmd) 148 | if stderr: 149 | logging.error('Unable to mount %s: %s' % (baseimage_path, stderr)) 150 | raise SystemExit 151 | else: 152 | return stdout.split('\n')[-2].split('\t')[-1] 153 | 154 | def install_base(self, mountpoint, installer_mount): 155 | """Installs the base system into sparsebundle""" 156 | cmd = ['installer', '-pkg', '%s/Packages/OSInstall.mpkg' % installer_mount, 157 | '-target', mountpoint] 158 | logging.info('Installing OS X into %s...' % mountpoint) 159 | (unused_stdout, stderr, unused_rc) = run_cmd(cmd) 160 | cmd = ['hdiutil', 'detach', '-force', installer_mount] 161 | logging.debug('Detaching %s' % installer_mount) 162 | (unused_stdout, stderr, unused_rc) = run_cmd(cmd) 163 | if stderr: 164 | logging.error('Failed to unmount: %s' % stderr) 165 | 166 | def detach_mountpoint(self, mountpoint, sparsebundle): 167 | """Detaches sparsebundle""" 168 | logging.debug('Detaching %s' % mountpoint) 169 | cmd = ['hdiutil', 'detach', '-force', mountpoint] 170 | (unused_stdout, stderr, unused_rc) = run_cmd(cmd) 171 | if stderr: 172 | logging.error('Unable to detach %s: %s' % (mountpoint, sparsebundle)) 173 | 174 | def cache_base(self, sparsebundle): 175 | logging.debug('Saving cache to %s' % self.sb_cache) 176 | shutil.copytree(sparsebundle, self.sb_cache) 177 | 178 | def mount_dmg(self, dmg): 179 | dmg_path = os.path.join(CACHE, dmg) 180 | cmd = ['hdiutil', 'attach', '-nobrowse','-noverify', 181 | '-noautoopen', dmg_path] 182 | logging.debug('Mounting %s' % dmg_path) 183 | (stdout, stderr, unused_rc) = run_cmd(cmd) 184 | if stderr: 185 | logging.error('Unable to mount %s: %s' % (dmg_path, stderr)) 186 | else: 187 | return stdout.split('\n')[-2].split('\t')[-1] 188 | 189 | def install_packages(self, pkgs, mountpoint): 190 | """Installs all pkgs into sparsebundle""" 191 | print 'Installing packages...' 192 | for pkg in pkgs: 193 | if pkg[0].endswith('.dmg'): 194 | dmg_mount = self.mount_dmg(pkg[0]) 195 | for f in os.listdir(dmg_mount): 196 | if f.endswith('.pkg') or f.endswith('.mpkg'): 197 | pkg_to_install = os.path.join(dmg_mount, f) 198 | elif pkg[0].endswith('.pkg'): 199 | dmg_mount = False 200 | pkg_to_install = os.path.join(CACHE, pkg[0]) 201 | logging.debug('Installing %s' % pkg_to_install) 202 | cmd = ['installer', '-pkg', pkg_to_install, '-target', 203 | mountpoint, '-verboseR'] 204 | (stdout, stderr, unused_rc) = run_cmd(cmd) 205 | if stderr: 206 | logging.error('Failure installing %s: %s' % (pkg_to_install, stderr)) 207 | else: 208 | logging.info('Successfully installed %s' % pkg_to_install) 209 | if dmg_mount: 210 | self.detach_mountpoint(dmg_mount, pkg[0]) 211 | 212 | def convert_sparsebundle(self, mountpoint, sparsebundle): 213 | """Detaches and converts sparsebundle""" 214 | print 'Converting and scanning disk image for restore...' 215 | if os.path.basename(mountpoint) != self.volume_name: 216 | logging.debug('Renaming %s to %s' % (mountpoint, self.volume_name)) 217 | cmd = ['diskutil', 'renameVolume', mountpoint, self.volume_name] 218 | (unused_stdout, stderr, unused_rc) = run_cmd(cmd) 219 | if stderr: 220 | logging.error('Unable to rename %s to %s' % (mountpoint, self.volume_name)) 221 | else: 222 | mountpoint = "/Volumes/%s" % self.volume_name 223 | logging.debug('Detaching %s' % mountpoint) 224 | cmd = ['hdiutil', 'detach', '-force', mountpoint] 225 | (unused_stdout, stderr, unused_rc) = run_cmd(cmd) 226 | if stderr: 227 | logging.error('Unable to detach %s: %s' % (mountpoint, sparsebundle)) 228 | else: 229 | if self.output_name.endswith(".dmg"): 230 | self.output_name = self.output_name[:-4] 231 | image_name = '%s_%s.hfs.dmg' % (self.output_name, timestamp) 232 | image_file = os.path.join(OUTPUT, image_name) 233 | logging.debug('Converting %s to %s' % (sparsebundle, image_file)) 234 | cmd = ['hdiutil', 'convert', sparsebundle, '-format', 235 | 'UDZO', '-o', image_file] 236 | (unused_stdout, stderr, unused_rc) = run_cmd(cmd) 237 | if stderr: 238 | logging.error('Image conversion failed: %s' % stderr) 239 | else: 240 | logging.debug('ASR imagescanning %s' % image_file) 241 | cmd = ['asr', 'imagescan', '-source', image_file] 242 | run_cmd(cmd) 243 | return image_file 244 | 245 | def cleanup(self, sparsebundle): 246 | """Removes temporary files""" 247 | try: 248 | shutil.rmtree(sparsebundle) 249 | except OSError, e: 250 | logging.error('Could not remove sparsebundle %s: %s' % (sparsebundle, e)) 251 | 252 | def build_image(self): 253 | """Builds the image""" 254 | if os.path.exists(self.sb_cache): 255 | self.sb_cache_exists = True 256 | if self.sb_cache_exists: 257 | sparsebundle = self.create_sparsebundle(sb_cache=self.sb_cache) 258 | else: 259 | sparsebundle = self.create_sparsebundle() 260 | mountpoint = self.mount_sparsebundle(sparsebundle) 261 | if not self.sb_cache_exists: 262 | installer_mount = self.mount_installer() 263 | self.install_base(mountpoint, installer_mount) 264 | self.detach_mountpoint(mountpoint, sparsebundle) 265 | self.cache_base(sparsebundle) 266 | self.sb_cache_exists = True 267 | mountpoint = self.mount_sparsebundle(sparsebundle) 268 | self.install_packages(self.pkgs, mountpoint) 269 | image_output = self.convert_sparsebundle(mountpoint, sparsebundle) 270 | set_perms(os.path.abspath(image_output)) 271 | print 'Image saved to: %s' % image_output 272 | self.cleanup(sparsebundle) 273 | 274 | def run_cmd(cmd, stream_out=False): 275 | """Runs a command and returns a tuple of stdout, stderr, returncode.""" 276 | if stream_out: 277 | task = subprocess.Popen(cmd) 278 | else: 279 | task = subprocess.Popen(cmd, stdout=subprocess.PIPE, 280 | stderr=subprocess.PIPE, env=env) 281 | (stdout, stderr) = task.communicate() 282 | return stdout, stderr, task.returncode 283 | 284 | def create_dir(dirpath): 285 | """Creates the given directory if missing.""" 286 | dirpath = os.path.join(os.getcwd(), dirpath) 287 | try: 288 | os.mkdir(dirpath) 289 | except OSError: 290 | try: 291 | os.mkdir(os.path.dirname(dirpath)) 292 | os.mkdir(dirpath) 293 | except OSError: 294 | print 'Unable to create folder %s' % dirpath 295 | 296 | def create_config(config): 297 | """Creates the config file with user-provided info.""" 298 | d = {} 299 | 300 | l = [('webserver', 301 | 'Enter the FQDN of your webserver ' 302 | '(i.e. mypackageserver.example.com)...\n' 303 | 'Server FQDN: ' 304 | ), 305 | ('path', 306 | 'Enter the storage path on your webserver ' 307 | '(i.e. /var/www/html/packages)...\n' 308 | 'Server path: ' 309 | ), 310 | ('login', 311 | 'Enter the ssh user for your webserver...\n' 312 | 'Login user: ' 313 | ) 314 | ] 315 | 316 | for i in l: 317 | unanswered = True 318 | while unanswered: 319 | try: 320 | answer = raw_input('%s' % i[1]) 321 | except KeyboardInterrupt: 322 | logging.error('Aborting configuration.') 323 | sys.exit() 324 | if answer != '': 325 | unanswered = False 326 | d[i[0]] = answer 327 | with open(config, 'w') as f: 328 | json.dump(d, f, indent=4, separators=(',', ': ')) 329 | 330 | print '\nCongiration file setup complete...\n' \ 331 | 'Use SSH keys for best results...\n' 332 | 333 | def import_config(config): 334 | """Returns data from config file.""" 335 | try: 336 | with open(config) as f: 337 | d = json.load(f) 338 | except NameError as err: 339 | print '%s; bailing script.' % err 340 | sys.exit(1) 341 | except IOError as err: 342 | print '%s: %s' % (err.strerror, config) 343 | sys.exit(1) 344 | return d['webserver'], d['path'], d['login'] 345 | 346 | def import_catalog(catalog): 347 | '''Imports user-defined catalog''' 348 | try: 349 | with open(catalog) as f: 350 | d = json.load(f) 351 | except NameError as err: 352 | print '%s; bailing script.' % err 353 | sys.exit(1) 354 | except IOError as err: 355 | print '%s: %s' % (err.strerror, catalog) 356 | sys.exit(1) 357 | return d 358 | 359 | def process_os_installer(base, webserver, webpath): 360 | """Checks for base installer and dowloads if needed.""" 361 | pkg_url = "http://%s/%s/%s" % (webserver, webpath, base) 362 | if os.path.exists(os.path.join(os.getcwd(), 'cache', base)): 363 | logging.debug('%s found in cache' % base) 364 | else: 365 | logging.debug('%s needs to be downloaded' % base) 366 | logging.debug('Downloading %s' % pkg_url) 367 | download_target = '%s/%s' % (CACHE, base) 368 | download_pkg(pkg_url, download_target) 369 | logging.info('%s downloaded to cache' % base) 370 | 371 | def process_pkgs(pkgs, webserver, webpath): 372 | """Checks for package(s) and dowloads if needed.""" 373 | for pkg in pkgs: 374 | downloaded = False 375 | verified = False 376 | pkg_url = 'http://%s/%s/%s' % (webserver, webpath, pkg[0]) 377 | if not os.path.exists(os.path.join(os.getcwd(), 'cache', pkg[0])): 378 | logging.debug('%s missing and needs to be downloaded' % pkg[0]) 379 | logging.debug('Downloading %s' % pkg_url) 380 | download_target = "%s/%s" % (CACHE, pkg[0]) 381 | download_pkg(pkg_url, download_target) 382 | downloaded = True 383 | l_sha1 = get_checksum(os.path.join(os.getcwd(), 'cache', pkg[0])) 384 | if l_sha1 != pkg[1]: 385 | logging.debug('%s sha1 mismatch and needs to be downloaded' % pkg[0]) 386 | logging.debug('downloading %s' % pkg_url) 387 | os.remove(os.path.join(CACHE, pkg[0])) 388 | download_target = "%s/%s" % (CACHE, pkg[0]) 389 | download_pkg(pkg_url, download_target) 390 | downloaded = True 391 | l_sha1 = get_checksum(os.path.join(os.getcwd(), 'cache', pkg[0])) 392 | if l_sha1 != pkg[1]: 393 | logging.warning('WARNING: %s does not match catalog.' % pkg[0]) 394 | sys.exit() 395 | if downloaded: 396 | logging.info('%s downloaded and verified' % pkg[0]) 397 | else: 398 | logging.info('%s found in cache and verified' % pkg[0]) 399 | 400 | def download_pkg(pkgurl, target): 401 | """Downloads package.""" 402 | try: 403 | pkg_dl = urllib2.urlopen(pkgurl) 404 | tmpfile = open(target, 'wb') 405 | shutil.copyfileobj(pkg_dl, tmpfile) 406 | set_perms(os.path.abspath(target)) 407 | except urllib2.URLError, e: 408 | logging.warning('Download of %s failed with error %s' % (pkgurl, e)) 409 | sys.exit() 410 | except IOError, e: 411 | logging.error('Could not write %s to disk; check disk permissions!' % target) 412 | logging.error('Error: %s' % e) 413 | sys.exit() 414 | 415 | def upload_pkg(pkg, login, webserver, serverpath): 416 | """Uploads package.""" 417 | cmd = ['scp', pkg, '%s@%s:%s' % (login, webserver, serverpath)] 418 | run_cmd(cmd) 419 | 420 | def set_perms(path): 421 | '''Sets the permissions of the given file''' 422 | uid = SCDynamicStoreCopyConsoleUser(None, None, None)[1] 423 | if uid != 0: 424 | os.chown(path, int(uid), 20) 425 | 426 | def get_checksum(pkg): 427 | """Returns sha1 checksum of package.""" 428 | statinfo = os.stat(pkg) 429 | if statinfo.st_size/1048576 < 200: 430 | f_content = open(pkg, 'r').read() 431 | f_hash = hashlib.sha1(f_content).hexdigest() 432 | return f_hash 433 | else: 434 | cmd = ['shasum', pkg] 435 | (stdout, unused_sterr, unused_rc) = run_cmd(cmd) 436 | return stdout.split()[0] 437 | 438 | def main(): 439 | parser = OptionParser(usage='Usage: sudo ./%prog [options]') 440 | parser.add_option('-b', '--build', dest='catalog', 441 | help='Specify catalog to process') 442 | parser.add_option('-c', '--configure', dest='configure', 443 | action='store_true', 444 | help='Set up or recreate configuration file') 445 | parser.add_option('-u', '--upload', dest='package', 446 | help='Upload package to webserver') 447 | parser.add_option('-C', '--checksum', dest='filename', 448 | help='Return checksum of a cached package') 449 | 450 | (options, unused_args) = parser.parse_args() 451 | 452 | if options.configure or not os.path.exists(CONFIG): 453 | for i in project_dirs: 454 | if not os.path.exists(i): 455 | create_dir(i) 456 | if not os.path.exists(CONFIG): 457 | print 'Config file does not exist! Creating now...\n' 458 | else: 459 | print 'Recreating config file...\n' 460 | os.remove(CONFIG) 461 | create_config(CONFIG) 462 | if not options.configure: 463 | parser.print_help() 464 | sys.exit(0) 465 | 466 | webserver, serverpath, login = import_config(CONFIG) 467 | webpath = os.path.basename(serverpath) 468 | 469 | if options.package: 470 | package = options.package 471 | print 'Uploading %s to %s:%s' % (package, webserver, serverpath) 472 | upload_pkg(package, login, webserver, serverpath) 473 | print os.path.basename(package), get_checksum(package) 474 | sys.exit(0) 475 | 476 | if options.filename: 477 | package = options.filename 478 | if "cache" not in options.filename: 479 | package = "cache/%s" % package 480 | print os.path.basename(package), get_checksum(package) 481 | sys.exit(0) 482 | 483 | if options.catalog: 484 | if os.getuid() != 0: 485 | print 'Using the -b option requires root!' 486 | parser.print_help() 487 | sys.exit(1) 488 | 489 | logging.basicConfig(format='%(asctime)s - %(levelname)s: %(message)s', 490 | datefmt='%m/%d/%Y %I:%M:%S %p', 491 | level=logging.DEBUG, 492 | filename=log_file) 493 | console = logging.StreamHandler() 494 | console.setLevel(logging.INFO) 495 | formatter = logging.Formatter('%(message)s') 496 | console.setFormatter(formatter) 497 | logging.getLogger('').addHandler(console) 498 | 499 | catalog = options.catalog 500 | if os.path.splitext(catalog)[1] != '.catalog': 501 | catalog = '%s.catalog' % catalog 502 | if not "catalogs" in catalog: 503 | catalog = "catalogs/%s" % catalog 504 | if not os.path.exists(catalog): 505 | print "File %s does not exist." % catalog 506 | sys.exit(1) 507 | catalog_data = import_catalog(catalog) 508 | volume_name = catalog_data['volume-name'] 509 | output_name = catalog_data['output-name'] 510 | os_catalog_data = import_catalog('catalogs/' + catalog_data['os-catalog']) 511 | os_installer = os_catalog_data['os-installer'] 512 | pkgs = os_catalog_data['packages'] + catalog_data['packages'] 513 | if not volume_name: 514 | volume_name = "Macintosh HD" 515 | if not output_name: 516 | output_name = '%s.dmg' % os_installer.split("_")[0] 517 | logging.info('Build started at: %s' % datetime.datetime.now()) 518 | logging.info('Current OS: %s' % current_os) 519 | logging.info('Base catalog: %s' % catalog_data['os-catalog']) 520 | logging.info('Base installer: %s' % os_installer) 521 | logging.info('Volume name: %s' % volume_name) 522 | logging.info('Output name: %s' % output_name) 523 | print 'Processing catalog(s)...' 524 | process_os_installer(os_installer, webserver, webpath) 525 | process_pkgs(pkgs, webserver, webpath) 526 | stew = Stew(volume_name, output_name, CONFIG, os_installer, 527 | pkgs, BUILD) 528 | stew.build_image() 529 | logging.info('Build finished at %s' % datetime.datetime.now()) 530 | else: 531 | parser.print_help() 532 | 533 | if __name__ == '__main__': 534 | main() -------------------------------------------------------------------------------- /uptodate: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python2.7 2 | """ 3 | Companion script to stew. 4 | 5 | This script uploads a given package, generates a sha1 6 | hash of the package, and optionally updates a given catalog 7 | with the package information. 8 | 9 | usage: uptodate [-h] [-p PACKAGE] [-c CATALOG] [-l LIST_CATALOG] 10 | 11 | optional arguments: 12 | -h, --help show this help message and exit 13 | -p PACKAGE, --package PACKAGE 14 | /path/to/package 15 | -c CATALOG, --catalog CATALOG 16 | /path/to/catalog 17 | -l LIST_CATALOG, --list_catalog LIST_CATALOG 18 | /path/to/catalog 19 | """ 20 | 21 | ############################################################################## 22 | # Copyright 2015 Joseph Chilcote 23 | # 24 | # Licensed under the Apache License, Version 2.0 (the "License"); you may not 25 | # use this file except in compliance with the License. You may obtain a copy 26 | # of the License at 27 | # 28 | # http://www.apache.org/licenses/LICENSE-2.0 29 | # 30 | # Unless required by applicable law or agreed to in writing, software 31 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 32 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 33 | # License for the specific language governing permissions and limitations 34 | # under the License. 35 | ############################################################################## 36 | 37 | __author__ = 'Joseph Chilcote (chilcote@gmail.com)' 38 | __version__ = '1.0.0' 39 | 40 | import os 41 | import sys 42 | import re 43 | import subprocess 44 | import hashlib 45 | import json 46 | import argparse 47 | 48 | CONFIG = os.path.join(os.getenv('HOME'), '.stew_config') 49 | 50 | def colored(text, color=None): 51 | if not os.getenv('ANSI_COLORS_DISABLED'): 52 | fmt_str = '\033[%dm' 53 | reset = '\033[0m' 54 | colors = { 55 | 'grey': 30, 56 | 'gray': 30, 57 | 'red': 31, 58 | 'green': 32, 59 | 'yellow': 33, 60 | 'blue': 34, 61 | 'magenta': 35, 62 | 'cyan': 36, 63 | 'white': 37, 64 | } 65 | if color is not None: 66 | text = fmt_str % (colors[color]) + text + reset 67 | return text 68 | 69 | def run_cmd(cmd, stream_out=False): 70 | '''Runs a command and returns a tuple of stdout, stderr, returncode.''' 71 | if stream_out: 72 | task = subprocess.Popen(cmd) 73 | else: 74 | task = subprocess.Popen(cmd, stdout=subprocess.PIPE, 75 | stderr=subprocess.PIPE) 76 | (stdout, stderr) = task.communicate() 77 | return stdout, stderr, task.returncode 78 | 79 | def import_config(config): 80 | '''Returns data from config file.''' 81 | try: 82 | with open(config) as f: 83 | d = json.load(f) 84 | except NameError as err: 85 | print colored('%s; bailing script.' % err, 'red') 86 | sys.exit(1) 87 | except IOError as err: 88 | print colored('%s: %s' % (err.strerror, config), 'red') 89 | sys.exit(1) 90 | return d['webserver'], d['path'], d['login'] 91 | 92 | def import_catalog(catalog): 93 | '''Imports user-defined catalog''' 94 | try: 95 | with open(catalog) as f: 96 | d = json.load(f) 97 | except NameError as err: 98 | print colored('%s; bailing script.' % err, 'red') 99 | sys.exit(1) 100 | except IOError as err: 101 | print colored('%s: %s' % (err.strerror, catalog), 'red') 102 | sys.exit(1) 103 | return d 104 | 105 | def get_checksum(pkg): 106 | '''Returns sha1 checksum of package.''' 107 | statinfo = os.stat(pkg) 108 | if statinfo.st_size/1048576 < 200: 109 | f_content = open(pkg, 'r').read() 110 | f_hash = hashlib.sha1(f_content).hexdigest() 111 | return f_hash 112 | else: 113 | cmd = ['shasum', pkg] 114 | (stdout, unused_sterr, unused_rc) = run_cmd(cmd) 115 | return stdout.split()[0] 116 | 117 | def upload_pkg(pkg, login, webserver, serverpath): 118 | '''Uploads package.''' 119 | cmd = ['scp', pkg, '%s@%s:%s' % (login, webserver, serverpath)] 120 | run_cmd(cmd) 121 | 122 | def update_catalog(catalog, d): 123 | '''Updates catalog''' 124 | with open(catalog, 'w') as f: 125 | json.dump(d, f, indent=4, separators=(',', ': ')) 126 | 127 | def print_catalog(catalog, entry=None): 128 | '''Prints catalog''' 129 | if 'os-catalog' in catalog.keys(): 130 | print colored('OS Catalog:\t%s' % catalog['os-catalog'], 'cyan') 131 | print colored('Output Name:\t%s' % catalog['output-name'], 'cyan') 132 | print colored('Volume Name:\t%s' % catalog['volume-name'], 'cyan') 133 | else: 134 | print colored('OS Installer:\t%s' % catalog['os-installer'], 'cyan') 135 | print colored('Packages:', 'cyan') 136 | for i in catalog['packages']: 137 | if catalog['packages'].index(i) == entry: 138 | print colored('\t\t%s (%s)' % (i[0], i[1]), 'yellow') 139 | else: 140 | print colored('\t\t%s' % i[0], 'cyan') 141 | 142 | def main(): 143 | '''Main method''' 144 | parser = argparse.ArgumentParser() 145 | parser.add_argument('-p', '--package', help='/path/to/package') 146 | parser.add_argument('-c', '--catalog', help='/path/to/catalog') 147 | parser.add_argument('-l', '--list_catalog', help='/path/to/catalog') 148 | args = parser.parse_args() 149 | 150 | if not os.path.exists(CONFIG): 151 | print colored('Your stew environment has not been configured.' \ 152 | '\nPlease run stew -c to create a config file.', 'red') 153 | sys.exit(1) 154 | 155 | webserver, serverpath, login = import_config(CONFIG) 156 | webpath = os.path.basename(serverpath) 157 | 158 | if args.package: 159 | package = args.package 160 | checksum = get_checksum(package) 161 | print colored('Uploading package: %s...' % package, 'yellow') 162 | upload_pkg(package, login, webserver, serverpath) 163 | 164 | if args.catalog: 165 | catalog = import_catalog(args.catalog) 166 | replace = raw_input(colored('Are you replacing a package? (y/n): ', 'green')) 167 | if replace == 'y': 168 | for i in enumerate(catalog['packages']): 169 | print colored('%s: %s (%s)' % (i[0], i[1][0], i[1][1]), 'yellow') 170 | answer = raw_input(colored('Choose a package to replace: ', 'green')) 171 | print colored('Updating catalog: %s...' % args.catalog, 'yellow') 172 | print colored('Removing: %s...' % catalog['packages'][int(answer)][0], 'yellow') 173 | print colored('Adding: %s...' % os.path.basename(package), 'yellow') 174 | catalog['packages'][int(answer)] = (os.path.basename(package), checksum) 175 | entry = int(answer) 176 | elif replace == 'n': 177 | l = catalog['packages'] 178 | print colored('Adding: %s...' % os.path.basename(package), 'yellow') 179 | l.append([os.path.basename(package), checksum]) 180 | catalog['packages'] = l 181 | entry = len(catalog['packages']) - 1 182 | else: 183 | print colored('%s is not a valid option, bailing script.' % replace, 'red') 184 | sys.exit(1) 185 | update_catalog(args.catalog, catalog) 186 | print_catalog(catalog, entry) 187 | else: 188 | print colored('%s %s' % (os.path.basename(package), checksum), 'cyan') 189 | 190 | elif args.catalog and not args.package: 191 | catalog = import_catalog(args.catalog) 192 | if 'os-installer' in catalog.keys(): 193 | os_installer = raw_input(colored('New os-installer: ', 'green')) 194 | if os_installer: 195 | catalog['os-installer'] = os_installer 196 | else: 197 | os_catalog = raw_input(colored('New os-catalog: ', 'green')) 198 | output_name = raw_input(colored('New output-name: ', 'green')) 199 | volume_name = raw_input(colored('New volume-name: ', 'green')) 200 | if os_catalog: 201 | catalog['os-catalog'] = os_catalog 202 | if output_name: 203 | catalog['output-name'] = output_name 204 | if volume_name: 205 | catalog['volume-name'] = volume_name 206 | update_catalog(args.catalog, catalog) 207 | print_catalog(catalog) 208 | 209 | elif args.list_catalog: 210 | catalog = import_catalog(args.list_catalog) 211 | print_catalog(catalog) 212 | 213 | else: 214 | parser.print_help() 215 | 216 | if __name__ == '__main__': 217 | main() --------------------------------------------------------------------------------