├── .gitignore ├── NOTICE ├── test ├── README.test ├── testvalues.py ├── botograbber.py ├── urllibgrabber.py └── signing.py ├── package ├── s3.conf ├── spec ├── README.md ├── s3.py └── LICENSE /.gitignore: -------------------------------------------------------------------------------- 1 | BUILD 2 | BUILDROOT 3 | RPMS 4 | *~ 5 | \#* 6 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Yum S3 Plugin 2 | Copyright 2011, Robert Mela 3 | Copyright 2011, Jens Braeuer 4 | 5 | May be used under the terms of the Apache License 2.0, provided in the LICENSE 6 | file in this directory and available at time of this writing at 7 | http://www.apache.org/licenses/LICENSE-2.0.txt 8 | 9 | Licenses for Python and Yum may also apply. 10 | -------------------------------------------------------------------------------- /test/README.test: -------------------------------------------------------------------------------- 1 | Tests currently focus only on urlgrabber and request signing 2 | 3 | In order to run you either need to add YUM packages to your PYTHON_PATH 4 | or you need to comment out code from createGrabber() on down in s3.py 5 | 6 | You also need to edit testvalues.py to include you aws_key_id and aws_secret_key, 7 | and edit the urls and filenames to point to an existing file on an existing s3 bucket 8 | that you have access to. 9 | 10 | -------------------------------------------------------------------------------- /package: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | 3 | usage () { 4 | echo "$0: a small wrapper to generate a rpm." 5 | echo "Usage: $0 -d" 6 | } 7 | 8 | if [ "$1" != "-d" ]; then 9 | usage 10 | exit 1 11 | fi 12 | 13 | set -e 14 | buildroot=$(dirname $(readlink -f "$0")) 15 | mkdir -p ${buildroot}/{RPMS,BUILD,BUILDROOT} 16 | rpmbuild --verbose \ 17 | --define "_sourcedir ${buildroot}" \ 18 | --define "_topdir ${buildroot}" \ 19 | -bb "${buildroot}/spec" 20 | 21 | -------------------------------------------------------------------------------- /s3.conf: -------------------------------------------------------------------------------- 1 | ; 2 | ; s3 yum plugin - /etc/yum/pluginconf.d/s3.conf 3 | ; 4 | ; Many thanks to Steven Maggs for pointers on using and configuring Yum 5 | ; 6 | ; Copyright 2011, Robert Mela 7 | ; 8 | ; Licensed under the Apache License, Version 2.0 (the "License"); 9 | ; you may not use this file except in compliance with the License. 10 | ; You may obtain a copy of the License at 11 | ; 12 | ; http://www.apache.org/licenses/LICENSE-2.0 13 | ; 14 | ; Unless required by applicable law or agreed to in writing, software 15 | ; distributed under the License is distributed on an "AS IS" BASIS, 16 | ; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | ; See the License for the specific language governing permissions and 18 | ; limitations under the License. 19 | ; 20 | ; 21 | [main] 22 | enabled=1 23 | -------------------------------------------------------------------------------- /spec: -------------------------------------------------------------------------------- 1 | Name: yum-s3 2 | Version: 0.2.4 3 | Release: 1 4 | Summary: Amazon S3 plugin for yum. 5 | 6 | Group: unknown 7 | License: Apache License 2.0 8 | URL: git@github.com:NumberFour/yum-s3-plugin.git 9 | Source0: s3.py 10 | Source1: s3.conf 11 | BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) 12 | BuildArch: noarch 13 | 14 | Requires: python-boto 15 | 16 | %description 17 | 18 | %prep 19 | cp -p %SOURCE0 %SOURCE1 . 20 | find . 21 | 22 | %install 23 | rm -rf "${RPM_BUILD_ROOT}" 24 | 25 | mkdir -p ${RPM_BUILD_ROOT}/etc/yum/pluginconf.d/ 26 | cp s3.conf ${RPM_BUILD_ROOT}/etc/yum/pluginconf.d/ 27 | 28 | mkdir -p ${RPM_BUILD_ROOT}/usr/lib/yum-plugins/ 29 | cp s3.py ${RPM_BUILD_ROOT}/usr/lib/yum-plugins/ 30 | 31 | %clean 32 | rm -rf "${RPM_BUILD_ROOT}" 33 | 34 | %files 35 | %defattr(-,root,root,-) 36 | /etc/yum/pluginconf.d/s3.conf 37 | /usr/lib/yum-plugins 38 | 39 | %changelog 40 | -------------------------------------------------------------------------------- /test/testvalues.py: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright 2011, Robert Mela 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | # 16 | 17 | """ 18 | Configuration constants for use in tests 19 | 20 | """ 21 | 22 | 23 | SECRET_KEY = 'my_amazon_secret_key' 24 | KEY_ID = 'my_amazon_key_id' 25 | TEST_URL_BASE='http://my-bucket-name.s3.amazonaws.com/' 26 | TEST_FILE='myfile.ext' 27 | TEST_URL= TEST_URL_BASE + TEST_FILE 28 | -------------------------------------------------------------------------------- /test/botograbber.py: -------------------------------------------------------------------------------- 1 | """ 2 | unittest for BotoGrabber 3 | 4 | """ 5 | # 6 | # Copyright 2011, Robert Mela 7 | # 8 | # Licensed under the Apache License, Version 2.0 (the "License"); 9 | # you may not use this file except in compliance with the License. 10 | # You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | # See the License for the specific language governing permissions and 18 | # limitations under the License. 19 | # 20 | 21 | import sys, unittest, os, os.path 22 | from urlparse import urlparse 23 | sys.path.append('..') 24 | sys.path.append('.') 25 | import s3 26 | 27 | import testvalues 28 | 29 | 30 | SECRET_KEY = testvalues.SECRET_KEY 31 | KEY_ID = testvalues.KEY_ID 32 | TEST_FILE = testvalues.TEST_FILE 33 | TEST_URL_BASE = testvalues.TEST_URL_BASE 34 | 35 | class TestBotoGrabber(unittest.TestCase): 36 | 37 | 38 | def setUp(self): 39 | self.grabber = s3.createBotoGrabber()(KEY_ID, SECRET_KEY, baseurl=[TEST_URL_BASE]) 40 | 41 | @classmethod 42 | def removeFile(cls): 43 | if not TEST_FILE: return 44 | if os.path.exists(TEST_FILE): os.unlink(TEST_FILE) 45 | 46 | def test_urlopen(self): 47 | result = self.grabber.urlopen(TEST_FILE) 48 | data = result.read() 49 | self.assertTrue(len(data)>0) 50 | 51 | def test_urlread(self): 52 | result = self.grabber.urlread(TEST_FILE) 53 | self.assertTrue( str == type(result) ) 54 | 55 | def test_urlgrab(self): 56 | result = self.grabber.urlgrab(TEST_FILE, TEST_FILE) 57 | self.assertTrue(os.path.exists(TEST_FILE)) 58 | 59 | if __name__ == '__main__': 60 | unittest.main() 61 | try: TestBotoGrabber.removeFile() 62 | except: pass 63 | -------------------------------------------------------------------------------- /test/urllibgrabber.py: -------------------------------------------------------------------------------- 1 | """ 2 | unittest for urllib-based grabber for yum s3 plugin 3 | """ 4 | # 5 | # Copyright 2011, Robert Mela 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | # 19 | 20 | import sys, unittest, os, os.path 21 | from urlparse import urlparse 22 | sys.path.append('..') 23 | sys.path.append('.') 24 | import s3 25 | 26 | import testvalues 27 | 28 | SECRET_KEY = testvalues.SECRET_KEY 29 | KEY_ID = testvalues.KEY_ID 30 | TEST_FILE = testvalues.TEST_FILE 31 | TEST_URL = testvalues.TEST_URL 32 | TEST_URL_BASE = testvalues.TEST_URL_BASE 33 | 34 | 35 | class TestUrlLibGrabber(unittest.TestCase): 36 | 37 | 38 | def setUp(self): 39 | self.grabber = s3.createUrllibGrabber()(KEY_ID,SECRET_KEY, baseurl=[TEST_URL_BASE]) 40 | 41 | @classmethod 42 | def removeFile(cls): 43 | if not TEST_FILE: return 44 | if os.path.exists(TEST_FILE): os.unlink(TEST_FILE) 45 | 46 | def test_urlopen(self): 47 | result = self.grabber.urlopen(TEST_FILE) 48 | data = result.read() 49 | self.assertTrue(len(data)>0) 50 | 51 | def test_urlread(self): 52 | result = self.grabber.urlread(TEST_FILE) 53 | self.assertTrue( str == type(result) ) 54 | 55 | def test_urlgrab(self): 56 | url = urlparse(TEST_FILE) 57 | result = self.grabber.urlgrab(TEST_FILE) 58 | self.assertTrue(os.path.exists(TEST_FILE)) 59 | 60 | if __name__ == '__main__': 61 | unittest.main() 62 | try: TestUrlLibGrabber.removeFile() 63 | except: pass 64 | -------------------------------------------------------------------------------- /test/signing.py: -------------------------------------------------------------------------------- 1 | """ 2 | unittest for signing of S3 Rest requests 3 | 4 | """ 5 | 6 | # Copyright 2011, Robert Mela 7 | # 8 | # Licensed under the Apache License, Version 2.0 (the "License"); 9 | # you may not use this file except in compliance with the License. 10 | # You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | # See the License for the specific language governing permissions and 18 | # limitations under the License. 19 | 20 | import sys, urllib2, time 21 | sys.path.append('..') 22 | sys.path.append('.') 23 | import s3 24 | import unittest 25 | import testvalues as tv 26 | 27 | class TestSigning(unittest.TestCase): 28 | """You need to configure SECRET_KEY, KEY_ID, and TEST_URL with real values for your s3 store""" 29 | 30 | 31 | def setUp(self): 32 | print "Setup" 33 | self.grabber = s3.createUrllibGrabber() 34 | 35 | def test_01_constructor(self): 36 | grabber = s3.createUrllibGrabber() 37 | 38 | def test_03_invalidkey(self): 39 | self.failIf(tv.SECRET_KEY=='my_amazon_secret_key') 40 | self.failIf(len(tv.KEY_ID) != len('0PN5J17HBGZHT7JJ3X82')) 41 | req=urllib2.Request(tv.TEST_URL) 42 | self.grabber.s3sign(req,'bogus_key', tv.KEY_ID) 43 | self.assertRaises(urllib2.HTTPError, urllib2.urlopen, req ) 44 | 45 | def test_02_urlopen(self): 46 | self.failIf(tv.SECRET_KEY=='my_amazon_secret_key') 47 | self.failIf(len(tv.KEY_ID) != len('0PN5J17HBGZHT7JJ3X82')) 48 | req=urllib2.Request(tv.TEST_URL) 49 | self.grabber.s3sign(req,tv.SECRET_KEY, tv.KEY_ID) 50 | resp=urllib2.urlopen(req) 51 | self.assertEquals(1,1) 52 | 53 | def test_02_example_key(self): 54 | 55 | """Validating against example data from 56 | http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RESTAuthentication.html""" 57 | 58 | EXAMPLE_URL='http://johnsmith.s3.amazonaws.com/photos/puppy.jpg' 59 | EXAMPLE_KEY_ID='0PN5J17HBGZHT7JJ3X82' 60 | EXAMPLE_SECRET_KEY='uV3F3YluFJax1cknvbcGwgjvx4QpvB+leU8dUj2o' 61 | EXAMPLE_DATE= time.strptime("2007-03-27 19:36:42", "%Y-%m-%d %H:%M:%S") 62 | 63 | req = urllib2.Request(EXAMPLE_URL) 64 | self.grabber.s3sign(req, EXAMPLE_SECRET_KEY, EXAMPLE_KEY_ID, EXAMPLE_DATE) 65 | desired="AWS 0PN5J17HBGZHT7JJ3X82:xXjDGYUmKxnwqr5KXNPGldn5LbA=" 66 | actual=req.headers.get('Authorization') 67 | print "Test1 Desired: %s" % desired 68 | print "Test1 Actual : %s" % actual 69 | self.assertEqual(actual, desired) 70 | 71 | if __name__ == '__main__': 72 | unittest.main() 73 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | A S3 plugin for yum. 2 | 3 | # Motivation 4 | 5 | It is very convenient to run a yum-repository on S3. S3 frees you from running your own webserver. 6 | You also dont have to worry about scalability when tons of servers update. 7 | 8 | `createrepo` and the awesome [`s3cmd`](https://github.com/s3tools/s3cmd), is 9 | all you need. However, this only works for public repositories ... 10 | 11 | # Public vs. protected repositories 12 | 13 | ## Public repositories 14 | 15 | S3 bucket can be make public. Public buckets can be accessed with plain http. 16 | 17 | - enable "website feature" of your s3 bucket 18 | - you dont need the s3-plugin. Everything works out of the box. 19 | 20 | ## Protected repositories 21 | 22 | This is where yum-s3 kicks in. yum-s3 uses the Boto library to fetch 23 | objects from S3, which allows using credentials. 24 | 25 | # Install 26 | 27 | - run ./package to build a RPM 28 | - install 29 | 30 | # How to configure a S3 based repo 31 | 32 | [repoprivate-pool-noarch] 33 | name=repoprivate-pool-noarch 34 | baseurl=http://.s3-website-eu-west-1.amazonaws.com/ 35 | gpgcheck=0 36 | priority=1 37 | s3_enabled=1 38 | key_id= 39 | secret_key= 40 | 41 | # YUM edge cases (baseurl) 42 | 43 | The is a flaw/issue/bug in yum, that stops yum-s3 from working, when 44 | you use a createrepo with a baseurl. 45 | 46 | ## Why should I want to do this? 47 | 48 | Suppose you have different environments (CI, testing, production). You 49 | want to upload the RPM only once to save outgoing bandwidth. Normally 50 | you could create symlinks on a webserver, but this is not possible 51 | with S3. But Yum offers a dedicated feature (baseurl), what can be 52 | used to make this possible. 53 | 54 | ## Repository layout with "pool" 55 | 56 | repo/ 57 | pool/ 58 | i386/ 59 | myprogram-1.0.rpm 60 | myprogram-1.1.rpm 61 | myprogram-1.2.rpm 62 | env/ 63 | CI/ 64 | myprogram-1.2.rpm -> ../../pool/i386/myprogram-1.2.rpm 65 | testing/ 66 | myprogram-1.1.rpm -> ../../pool/i386/myprogram-1.1.rpm 67 | production/ 68 | myprogram-1.0.rpm -> ../../pool/i386/myprogram-1.0.rpm 69 | 70 | So when you add a RPM to a repository, you do the following steps 71 | 72 | - add the RPM to the /pool directory 73 | - create a symlink in the folder (like CI, testing, production) 74 | - run createrepo with the --baseurl option 75 | 76 | This will create the yum xml-files based on the symlinks that are 77 | present. So you can decide with the symlinks at createrepo-time, which 78 | packages are "visible". 79 | 80 | Giving the --baseurl option to yum will make yum go the /pool 81 | directory to fetch the actual RPM. 82 | 83 | ## How do I make yum honour the baseurl? 84 | 85 | There is a patch to yum (version 3.2.29 that ships with SL6). See 86 | https://github.com/jbraeuer/yum-s3 87 | 88 | ## Authors 89 | 90 | - @rmela 91 | 92 | - @CBarraford 93 | - @airpringle 94 | - @jbraeuer 95 | - @jcomeaux 96 | - @louisrli 97 | - @tanob 98 | 99 | Have fun! 100 | -------------------------------------------------------------------------------- /s3.py: -------------------------------------------------------------------------------- 1 | """ 2 | Yum plugin for Amazon S3 access. 3 | 4 | This plugin provides access to a protected Amazon S3 bucket using either boto 5 | or Amazon's REST authentication scheme. 6 | 7 | On CentOS this file goes into /usr/lib/yum-plugins/s3.py 8 | 9 | You will also need two configuration files. See s3.conf and s3test.repo for 10 | examples on how to deploy those. 11 | 12 | 13 | """ 14 | 15 | # Copyright 2011, Robert Mela 16 | # Copyright 2011, Jens Braeuer 17 | # 18 | # Licensed under the Apache License, Version 2.0 (the "License"); 19 | # you may not use this file except in compliance with the License. 20 | # You may obtain a copy of the License at 21 | # 22 | # http://www.apache.org/licenses/LICENSE-2.0 23 | # 24 | # Unless required by applicable law or agreed to in writing, software 25 | # distributed under the License is distributed on an "AS IS" BASIS, 26 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 27 | # See the License for the specific language governing permissions and 28 | # limitations under the License. 29 | 30 | import logging 31 | import os 32 | import sys 33 | import urllib 34 | 35 | from yum.plugins import TYPE_CORE 36 | from yum.yumRepo import YumRepository 37 | from yum import config 38 | from yum import logginglevels 39 | 40 | import yum.Errors 41 | 42 | def interactive_notify(msg): 43 | if sys.stdout.isatty(): 44 | print msg 45 | 46 | def createUrllibGrabber(): 47 | """ 48 | Fetch files from AWS without boto. This code has not been tested on RHEL 6 as EPEL ships with boto 2.x. 49 | """ 50 | import os 51 | import sys 52 | import urllib2 53 | import time, sha, hmac, base64 54 | 55 | class UrllibGrabber: 56 | @classmethod 57 | 58 | def s3sign(cls,request, secret_key, key_id, date=None): 59 | date=time.strftime("%a, %d %b %Y %H:%M:%S +0000", date or time.gmtime() ) 60 | host = request.get_host() 61 | bucket = host.split('.')[0] 62 | request.add_header('Date', date) 63 | resource = "/%s%s" % ( bucket, request.get_selector() ) 64 | sigstring = """%(method)s\n\n\n%(date)s\n%(canon_amzn_resource)s""" % { 65 | 'method':request.get_method(), 66 | #'content_md5':'', 67 | #'content_type':'', # only for PUT 68 | 'date':request.headers.get('Date'), 69 | #'canon_amzn_headers':'', 70 | 'canon_amzn_resource':resource } 71 | digest = hmac.new(secret_key, sigstring, sha ).digest() 72 | digest = base64.b64encode(digest) 73 | request.add_header('Authorization', "AWS %s:%s" % ( key_id, digest )) 74 | 75 | def __init__(self, awsAccessKey, awsSecretKey, baseurl ): 76 | try: baseurl = baseurl[0] 77 | except: pass 78 | self.baseurl = baseurl 79 | self.awsAccessKey = awsAccessKey 80 | self.awsSecretKey = awsSecretKey 81 | 82 | def _request(self,url): 83 | req = urllib2.Request("%s%s" % (self.baseurl, url)) 84 | UrllibGrabber.s3sign(req, self.awsSecretKey, self.awsAccessKey ) 85 | return req 86 | 87 | def urlgrab(self, url, filename=None, **kwargs): 88 | """urlgrab(url) copy the file to the local filesystem""" 89 | self.verbose_logger.log(logginglevels.DEBUG_4, "UrlLibGrabber urlgrab url=%s filename=%s" % ( url, filename )) 90 | req = self._request(url) 91 | if not filename: 92 | filename = req.get_selector() 93 | if filename[0] == '/': filename = filename[1:] 94 | out = open(filename, 'w+') 95 | resp = urllib2.urlopen(req) 96 | buff = resp.read(8192) 97 | while buff: 98 | out.write(buff) 99 | buff = resp.read(8192) 100 | return filename 101 | # zzz - does this return a value or something? 102 | 103 | def urlopen(self, url, **kwargs): 104 | """urlopen(url) open the remote file and return a file object""" 105 | return urllib2.urlopen( self._request(url) ) 106 | 107 | def urlread(self, url, limit=None, **kwargs): 108 | """urlread(url) return the contents of the file as a string""" 109 | return urllib2.urlopen( self._request(url) ).read() 110 | 111 | return UrllibGrabber 112 | 113 | def createBotoGrabber(): 114 | import boto 115 | from urlparse import urlparse 116 | import sys 117 | import re 118 | from urlgrabber.grabber import URLGrabber 119 | 120 | class BotoGrabber(URLGrabber): 121 | logger = logging.getLogger("yum.verbose.main") 122 | 123 | def __init__(self, awsAccessKey, awsSecretKey, baseurl): 124 | self.logger.debug("BotoGrabber init BASE_URL=%s" % baseurl) 125 | 126 | URLGrabber.__init__(self) 127 | self._handle_baseurl(baseurl) 128 | self._handle_s3(awsAccessKey, awsSecretKey) 129 | self._dump_attributes() 130 | interactive_notify("%s - %s" % (self.bucket_name, self.key_prefix)) 131 | 132 | def _handle_baseurl(self, baseurl): 133 | if type(baseurl) == list: 134 | baseurl = baseurl[0] 135 | 136 | # self.baseurl[1] is self.baseurl.netloc; self.baseurl[2] is self.baseurl.path 137 | # See http://docs.python.org/library/urlparse.html 138 | self.baseurl = urlparse(baseurl) 139 | 140 | m = re.match('(.*)\.(s3.*\.amazonaws\.com)', self.baseurl[1]) 141 | self.bucket_name = m.group(1) 142 | self.host_name = m.group(2) 143 | self.key_prefix = self.baseurl[2][1:].rstrip('/') 144 | 145 | def _handle_s3(self, awsAccessKey, awsSecretKey): 146 | self.s3 = boto.connect_s3(awsAccessKey, awsSecretKey, host=self.host_name) 147 | 148 | def _dump_attributes(self): 149 | self.logger.debug("baseurl: %s" % str(self.baseurl)) 150 | self.logger.debug("host_name: %s" % self.host_name) 151 | self.logger.debug("bucket: %s" % self.bucket_name) 152 | self.logger.debug("key_prefix: %s" % self.key_prefix) 153 | 154 | def _key_name(self,url): 155 | self.logger.debug("_key_name url=%s, key_prefix=%s" % (url, self.key_prefix)) 156 | 157 | if not url.startswith("http://"): 158 | key = "%s/%s" % (self.key_prefix, url) 159 | else: 160 | key = urlparse(url)[2] 161 | 162 | self.logger.debug("_key_name(%s) -> %s" % (url, key)) 163 | return key 164 | 165 | def _key(self, key_name): 166 | self.logger.debug("_key(%s)" % key_name) 167 | bucket = self.s3.get_bucket(self.bucket_name, validate=False) 168 | 169 | return bucket.get_key(key_name) 170 | 171 | def urlgrab(self, url, filename=None, **kwargs): 172 | """urlgrab(url) copy the file to the local filesystem""" 173 | 174 | self.logger.debug("urlgrab(url='%s',filename='%s')" % (url, filename)) 175 | 176 | key_name = self._key_name(url) 177 | key = self._key(key_name) 178 | 179 | if not key: 180 | raise Exception("Can not get key for key=%s" % key_name ) 181 | if not filename: 182 | filename = key.key 183 | 184 | key.get_contents_to_filename(filename) 185 | return filename 186 | 187 | def urlopen(self, url, **kwargs): 188 | """urlopen(url) open the remote file and return a file object""" 189 | 190 | self.logger.debug("urlopen(%s)" % url) 191 | return self._key(url) 192 | 193 | def urlread(self, url, limit=None, **kwargs): 194 | """urlread(url) return the contents of the file as a string""" 195 | 196 | self.logger.debug("urlread(%s)" % url) 197 | return self._key(url).read() 198 | 199 | return BotoGrabber 200 | 201 | def createGrabber(): 202 | logger = logging.getLogger("yum.verbose.main") 203 | try: 204 | try: 205 | grabber = createBotoGrabber() 206 | logger.debug("Using BotoGrabber") 207 | except: 208 | grabber = createUrllibGrabber() 209 | logger.debug("Using UrllibGrabber") 210 | finally: 211 | return grabber 212 | 213 | AmazonS3Grabber = createGrabber() 214 | 215 | class AmazonS3Repo(YumRepository): 216 | """ 217 | Repository object for Amazon S3. 218 | """ 219 | 220 | def __init__(self, repoid): 221 | YumRepository.__init__(self, repoid) 222 | self.enable() 223 | self.grabber = None 224 | 225 | def setupGrab(self): 226 | YumRepository.setupGrab(self) 227 | self.grabber = AmazonS3Grabber(self.key_id, self.secret_key ) 228 | 229 | def _getgrabfunc(self): raise Exception("get grabfunc!") 230 | def _getgrab(self): 231 | if not self.grabber: 232 | self.grabber = AmazonS3Grabber(self.key_id, self.secret_key, baseurl=self.baseurl ) 233 | return self.grabber 234 | 235 | grabfunc = property(lambda self: self._getgrabfunc()) 236 | grab = property(lambda self: self._getgrab()) 237 | 238 | 239 | __revision__ = "1.0.0" 240 | requires_api_version = '2.5' 241 | plugin_type = TYPE_CORE 242 | 243 | def config_hook(conduit): 244 | logger = logging.getLogger("yum.verbose.main") 245 | config.RepoConf.s3_enabled = config.BoolOption(False) 246 | config.RepoConf.key_id = config.Option() or conduit.confString('main', 'aws_access_key_id') 247 | config.RepoConf.secret_key = config.Option() or conduit.confString('main', 'aws_secret_access_key') 248 | 249 | def prereposetup_hook(conduit): 250 | """ 251 | Plugin initialization hook. Setup the S3 repositories. 252 | """ 253 | 254 | repos = conduit.getRepos() 255 | conf = conduit.getConf() 256 | cachedir = conduit.getConf().cachedir 257 | 258 | for key,repo in repos.repos.iteritems(): 259 | if isinstance(repo, YumRepository) and repo.s3_enabled and repo.isEnabled(): 260 | new_repo = AmazonS3Repo(key) 261 | new_repo.name = repo.name 262 | new_repo.baseurl = repo.baseurl 263 | new_repo.mirrorlist = repo.mirrorlist 264 | new_repo.basecachedir = repo.basecachedir 265 | new_repo.gpgcheck = repo.gpgcheck 266 | new_repo.gpgkey = repo.gpgkey 267 | new_repo.proxy = repo.proxy 268 | new_repo.enablegroups = repo.enablegroups 269 | new_repo.key_id = repo.key_id 270 | new_repo.secret_key = repo.secret_key 271 | new_repo.metadata_expire = repo.metadata_expire 272 | if hasattr(repo, 'priority'): 273 | new_repo.priority = repo.priority 274 | if hasattr(repo, 'base_persistdir'): 275 | new_repo.base_persistdir = repo.base_persistdir 276 | 277 | repos.delete(repo.id) 278 | repos.add(new_repo) 279 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | --------------------------------------------------------------------------------