├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── app.py ├── pip-requirements.txt ├── s3_uploader.cfg.sample ├── s3_uploader.py ├── s3_uploader_icon.icns ├── s3_uploader_icon.ico ├── s3_uploader_icon.idraw ├── s3_uploader_icon.png ├── s3_uploader_icon_favicon.ico └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | .virtualenv 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Charles Daniel 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | 2 | dist/S3Uploader.app: dependencies data s3_uploader.py app.py 3 | python setup.py py2app 4 | 5 | dependencies: 6 | pip install -r pip-requirements.txt 7 | 8 | data: dependencies 9 | botocore_dir=$$(python -c 'import botocore; print botocore.__path__[0]'); \ 10 | echo "BOTOCORE $${botocore_dir}"; \ 11 | cp -R $${botocore_dir}/data data; \ 12 | cp -R $${botocore_dir}/vendored/requests/cacert.pem data/ 13 | 14 | clean: 15 | -rm -f *.pyc 16 | -rm -rf data 17 | -rm -rf build 18 | -rm -rf dist 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # S3 Uploader 2 | ##### by Charles Daniel 3 | 4 | [![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/ahXHOTSKi-4/0.jpg)](https://www.youtube.com/watch?v=ahXHOTSKi-4) 5 | 6 | ### Introduction 7 | 8 | This is a very simple S3 File Uploader app written in Python+Tkinter/ttk and uses Boto3 for the actual S3 interaction. 9 | Additionally it uses py2app to create a standalone OSX app that can be launched by clicking an icon. 10 | A configuration file can be used to specify the credentials as well as a list of buckets one should be uploading files into. It's mainly for giving it to non-technical people (along with a custom configuration file) so they can easily upload files to whatever bucket you like. 11 | 12 | Each file uploading is done in a separate thread so the UI won't be blocked and one can upload any number of files concurrently. It's been tested with binary files of up to 62GB, haven't had a chance to try anything larger yet. 13 | 14 | ### Building the App 15 | 16 | **Prerequisites** 17 | 18 | - Python2.7 (hasn't been tested in anything else) 19 | - Tkinter/ttk (should already be installed in a standard Python) 20 | - Boto3 21 | - Py2App 22 | 23 | **Building a standalone app on OSX using py2app** 24 | 25 | Run `make` 26 | 27 | This builds the app into the `dist/` directory. 28 | You'll want to make sure you have a configuration file located at `~/s3_uploader.cfg`. 29 | Double click the new app in the `dist/` directory to launch it. 30 | 31 | ### Running directly using Python 32 | 33 | You can run the s3_uploader directly using Python on the command line: 34 | 35 | `python s3_uploader.py` 36 | 37 | ### Configuration 38 | 39 | There's a sample configuration file `s3_uploader.cfg.sample` that can be copied and used as the starting point for the configurations. Make sure you update the `aws_access_key_id` and `aws_secret_access_key` values. Add a list of s3 buckets for the pulldown by entering a comma separated list of bucket names as `s3_buckets`. 40 | 41 | Alternatively you should be able to make a similar section called `[s3_uploader]` in any other file (including the `~/.aws/credentials`) 42 | 43 | By default the app looks for the `~/s3_uploader.cfg` file for the configurations, however you should be able to open a different configuration file by going to File > Open Config from the menubar. 44 | 45 | 46 | ### TODO 47 | 48 | - Get things to work with py2exe or pyinstaller to build a windows app 49 | -------------------------------------------------------------------------------- /app.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | os.environ["AWS_DATA_PATH"] = 'data/' 4 | os.environ["AWS_CACERT"] = 'data/cacert.pem' 5 | 6 | import s3_uploader 7 | 8 | s3_uploader.main('~/s3_uploader.cfg') 9 | -------------------------------------------------------------------------------- /pip-requirements.txt: -------------------------------------------------------------------------------- 1 | boto3==1.3.0 2 | py2app==0.10 3 | -------------------------------------------------------------------------------- /s3_uploader.cfg.sample: -------------------------------------------------------------------------------- 1 | # A config file must contain at least the 's3_uploader' section 2 | [s3_uploader] 3 | aws_access_key_id= 4 | aws_secret_access_key= 5 | # Enter a list of buckets separated by commas 6 | s3_buckets=bucket1,bucket2,bucket3... 7 | -------------------------------------------------------------------------------- /s3_uploader.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from Tkinter import * 4 | from ttk import * 5 | import Tkconstants, tkFileDialog, tkSimpleDialog 6 | import boto3 7 | import botocore 8 | from os.path import expanduser 9 | import os 10 | import thread 11 | import ConfigParser 12 | import webbrowser 13 | import datetime 14 | import ntpath 15 | 16 | class Uploader(Frame): 17 | def __init__(self, root, filename, bucket_name, s3_filename): 18 | Frame.__init__(self, root) 19 | 20 | self._filename = filename 21 | self._bucket_name = bucket_name 22 | self._s3_filename = s3_filename 23 | 24 | # Used in percent & rate metric calculations 25 | self._size = float(os.path.getsize(filename)) 26 | self._seen_so_far = 0 27 | self._last_timestamp = None 28 | self._last_updated = None 29 | self._rate_samples = [ 0 for _i in xrange(0, 10) ] 30 | 31 | self._thread_should_exit = False 32 | 33 | s3_filelabel = Label(self, text="{}/{}".format(bucket_name, s3_filename)) 34 | s3_filelabel.bind("", lambda e: webbrowser.open_new("https://s3.amazonaws.com/{}/{}".format(bucket_name, s3_filename))) 35 | s3_filelabel.grid(row=0, column=0, sticky=W, padx=10, pady=2) 36 | 37 | progress_frame = Frame(self) 38 | progress_frame.grid(row=0, column=1, padx=10, pady=2) 39 | 40 | self.progress = Progressbar(progress_frame, orient='horizontal', mode='determinate') 41 | self.progress.grid(row=0, column=0, columnspan=3, pady=0) 42 | 43 | self.percent_label = Label(progress_frame, text="?", font="Sans 8") 44 | self.percent_label.grid(row=1, column=2, padx=4, pady=0) 45 | 46 | self.rate_label = Label(progress_frame, text="?", font="Sans 8") 47 | self.rate_label.grid(row=1, column=0, padx=2, pady=0) 48 | 49 | self.rate_unit_label = Label(progress_frame, text="?", font="Sans 8") 50 | self.rate_unit_label.grid(row=1, column=1, padx=2, pady=0) 51 | 52 | self.cancel_button = Button(self, text="Cancel", command=self.cancel_transfer) 53 | self.cancel_button.grid(row=0, column=3, padx=10, pady=2) 54 | 55 | def update_progress(self, bytes_amount): 56 | # This method can be called many times a second. So we need to sample every so often (fractional seconds). 57 | # The rates are sampled and stored in a list then the average is used for the display. 58 | 59 | _now = datetime.datetime.now() 60 | self._seen_so_far += bytes_amount 61 | percentage = (self._seen_so_far / self._size) * 100 62 | percent = int(percentage) 63 | 64 | if (not self._last_updated) or ((_now - self._last_updated).total_seconds() > 0.25): 65 | self._last_updated = _now 66 | self.progress["value"] = percent 67 | self.percent_label["text"] = "{}%".format(percent) 68 | 69 | diff_secs = (_now - self._last_timestamp).total_seconds() 70 | self._last_timestamp = _now 71 | raw_rate = bytes_amount / diff_secs 72 | self._rate_samples.append(raw_rate) 73 | self._rate_samples.pop(0) 74 | 75 | #rate = max(self._rate_samples) 76 | rate = sum(self._rate_samples) / len(self._rate_samples) 77 | 78 | # Make the rate more human-readable 79 | sizes = ['KB','MB','GB','TB'] 80 | rate_unit = 'B' 81 | for s in sizes: 82 | if (rate // 1024) > 0: 83 | rate /= 1024 84 | rate_unit = s 85 | else: 86 | break 87 | 88 | self.rate_label["text"] = "{}".format(round(rate, 2)) 89 | self.rate_unit_label["text"] = "{}/s".format(rate_unit) 90 | 91 | # If something set the _thread_should_exit flag then we should force the thread to exit 92 | # which will also kill the upload. 93 | if self._thread_should_exit: 94 | self.cancel_button["text"] = "Clear" 95 | thread.exit() 96 | 97 | # If we're done the upload at 100% then just convert the cancel button to Clear 98 | if percent >= 100: 99 | self.cancel_button["text"] = "Clear" 100 | 101 | def _do_transfer(self, aws_access_key_id, aws_secret_access_key): 102 | try: 103 | aws_session = boto3.session.Session(aws_access_key_id, aws_secret_access_key) 104 | cacert = os.environ.get('AWS_CACERT', None) 105 | s3 = aws_session.client('s3', verify=cacert) 106 | print "{} START TIME: {}".format(self._s3_filename, datetime.datetime.now()) 107 | self._last_timestamp = datetime.datetime.now() 108 | s3.upload_file(self._filename, self._bucket_name, self._s3_filename, Callback=self.update_progress) 109 | print "{} END TIME: {}".format(self._s3_filename, datetime.datetime.now()) 110 | self.percent_label["text"] = "100%" 111 | self.progress["value"] = 100 112 | self.cancel_button["text"] = "Clear" 113 | except botocore.exceptions.ClientError as e: 114 | print "ERROR ", e.message 115 | self.show_error([e.message]) 116 | 117 | def show_error(self, errors): 118 | self.errors = Frame(self) 119 | for error in errors: 120 | Label(self.errors, text=error, style='Error.TLabel', font="Sans 8").pack(fill=BOTH) 121 | 122 | self.errors.grid(row=1, columnspan=4, padx=10) 123 | self.cancel_button["text"] = "Clear" 124 | 125 | def do_transfer(self, *args): 126 | thread.start_new_thread(self._do_transfer, args) 127 | 128 | def cancel_transfer(self): 129 | if self.cancel_button["text"] == "Cancel": 130 | # If the cancel button says "Cancel" then we want to set the exit flag to True 131 | self._thread_should_exit = True 132 | elif self.cancel_button["text"] == "Clear": 133 | # Otherwise if the cancel button says "Clear" we have to remove the upload frame 134 | self.grid_forget() 135 | 136 | 137 | class SecureEntry(Frame): 138 | def __init__(self, root, variable): 139 | Frame.__init__(self, root) 140 | 141 | self.entry = Entry(self, textvariable=variable) 142 | self.entry.pack(side=LEFT) 143 | 144 | self.toggle_button = Button(self, width=3, command=self.toggle_entry_security) 145 | self.toggle_button.pack(side=LEFT, padx=0, pady=0) 146 | self.toggle_entry_security(show=False) 147 | 148 | def toggle_entry_security(self, show=None): 149 | if show == None: 150 | show = (self.entry['show'] == '*') # toggle to opposite by default 151 | 152 | if show == True: 153 | self.entry['show'] = '' 154 | self.toggle_button['text'] = "\xe0\xb2\xa0_\xe0\xb2\xa0" 155 | else: 156 | self.entry['show'] = '*' 157 | self.toggle_button['text'] = ">_<" 158 | 159 | 160 | class S3FileUploader(Frame): 161 | def __init__(self, root, config_file): 162 | Frame.__init__(self, root) 163 | 164 | r = self.r = 0 165 | self.error_row = r 166 | padx = 10 167 | pady = 2 168 | 169 | r+=1 170 | Label(self, text="S3 Uploader", font="Sans 16 bold").grid(row=r, columnspan=2, pady=pady) 171 | 172 | r+=1 173 | Label(self, text="AWS AccessKeyID:").grid(row=r, sticky=W, padx=padx, pady=pady) 174 | self.aws_access_key_id = StringVar(self) 175 | SecureEntry(self, self.aws_access_key_id).grid(row=r, column=1, sticky=W, padx=padx) 176 | 177 | r+=1 178 | Label(self, text="AWS AccessSecretKey:").grid(row=r, sticky=W, padx=padx, pady=pady) 179 | self.aws_secret_access_key = StringVar(self) 180 | SecureEntry(self, self.aws_secret_access_key).grid(row=r, column=1, sticky=W, padx=padx, pady=pady) 181 | 182 | r+=1 183 | Label(self, text="File:").grid(row=r, sticky=W, padx=padx, pady=pady) 184 | self.file_button = Button(self, text='Choose Local File...', command=self.askopenfilename) 185 | self.file_button.grid(row=r, column=1, sticky=W, padx=padx, pady=pady) 186 | 187 | r+=1 188 | Label(self, text="S3 Bucket:").grid(row=r, sticky=W, padx=padx, pady=pady) 189 | self.s3_bucket = StringVar(self) 190 | self.s3_bucket_list = OptionMenu(self, self.s3_bucket) 191 | self.s3_bucket_list.grid(row=r, column=1, sticky=W, padx=padx, pady=pady) 192 | 193 | r+=1 194 | Label(self, text="S3 Filename:").grid(row=r, sticky=W, padx=padx, pady=pady) 195 | self.s3_name = StringVar(self) 196 | Entry(self, textvariable=self.s3_name).grid(row=r, column=1, sticky=W, padx=padx, pady=pady) 197 | 198 | r+=1 199 | Button(self, text='START UPLOAD', command=self.upload_to_s3).grid(row=r, columnspan=2, padx=padx, pady=pady) 200 | 201 | r+=1 202 | Separator(self, orient="horizontal").grid(row=r, columnspan=2, padx=padx, pady=pady) 203 | 204 | self.r = r 205 | 206 | self.errors = None 207 | self.config_file = config_file 208 | self.load_config(config_file) 209 | self.filename = None 210 | 211 | def prompt_load_config(self): 212 | initialdir = self.config_file if self.config_file else "~/" 213 | config_filename = tkFileDialog.askopenfilename(parent=self, initialdir=expanduser(initialdir), title="Choose config file") #, filetypes=[('cfg files', '.cfg')]) 214 | if config_filename: 215 | self.load_config(config_filename) 216 | 217 | def load_config(self, config_file): 218 | self.clear_errors() 219 | config = ConfigParser.ConfigParser() 220 | config_file_handle = expanduser(config_file) 221 | config.read(config_file_handle) 222 | try: 223 | aws_access_key_id = config.get("s3_uploader", "aws_access_key_id") 224 | self.aws_access_key_id.set(aws_access_key_id) 225 | 226 | aws_secret_access_key = config.get("s3_uploader", "aws_secret_access_key") 227 | self.aws_secret_access_key.set(aws_secret_access_key) 228 | 229 | s3_buckets = config.get("s3_uploader", "s3_buckets") 230 | s3_buckets = s3_buckets.split(',') 231 | self.s3_bucket_list.set_menu(s3_buckets[0], *(s3_buckets)) 232 | 233 | self.config_file = config_file 234 | except ConfigParser.NoSectionError: 235 | print "Failed to read Config File, missing section 's3_uploader'" 236 | self.show_errors([u"\u26A0 Failed to read Config File, missing section 's3_uploader'"]) 237 | 238 | def askopenfilename(self): 239 | # define options for opening or saving a file 240 | self.file_opt = options = {} 241 | #options['defaultextension'] = '' #'.csv' 242 | #options['filetypes'] = [('all files', '.*'), ('text files', '.csv')] 243 | options['initialdir'] = expanduser(self.filename) if self.filename else expanduser("~") 244 | #options['initialfile'] = '' 245 | options['parent'] = self 246 | options['title'] = 'Choose File to upload' 247 | 248 | # get filename 249 | _filename = tkFileDialog.askopenfilename(**self.file_opt) 250 | 251 | # open file on your own 252 | if _filename: 253 | self.filename = _filename 254 | self.file_button["text"] = self.filename if (len(self.filename) <= 33) else "...{}".format(self.filename[-30:]) 255 | self.s3_name.set(ntpath.basename(self.filename)) 256 | 257 | def show_errors(self, errors): 258 | self.errors = Frame(self) 259 | for error in errors: 260 | Label(self.errors, text=error, style='Error.TLabel').pack(fill=BOTH) 261 | 262 | self.errors.grid(row=self.error_row, columnspan=2, pady=20) 263 | 264 | def clear_errors(self): 265 | if self.errors: 266 | self.errors.destroy() 267 | self.errors = None 268 | 269 | def upload_to_s3(self): 270 | self.clear_errors() 271 | 272 | aws_access_key_id = self.aws_access_key_id.get().strip() 273 | aws_secret_access_key = self.aws_secret_access_key.get().strip() 274 | filename = self.filename 275 | bucket = self.s3_bucket.get() 276 | name = self.s3_name.get().strip() 277 | 278 | errors = [] 279 | if not aws_access_key_id: 280 | errors.append(u"\u26A0 AWS Access Key ID is required") 281 | if not aws_secret_access_key: 282 | errors.append(u"\u26A0 AWS Access Secret Key is required") 283 | if not filename: 284 | errors.append(u"\u26A0 A File must be chosen for upload") 285 | if not bucket: 286 | errors.append(u"\u26A0 Choose an S3 bucket to upload the file to") 287 | if not name: 288 | errors.append(u"\u26A0 Enter a S3 filename to upload the file as") 289 | 290 | if errors: 291 | self.show_errors(errors) 292 | return 293 | 294 | self.r += 1 295 | up = Uploader(self, filename, bucket, name) 296 | up.grid(row=self.r, columnspan=2, sticky=NSEW) 297 | 298 | up.do_transfer(aws_access_key_id, aws_secret_access_key) 299 | 300 | 301 | class AboutWin(tkSimpleDialog.Dialog): 302 | def __init__(self, root): 303 | self._parent = root 304 | 305 | def show(self): 306 | tkSimpleDialog.Dialog.__init__(self, self._parent, "About S3 Uploader") 307 | 308 | def body(self, master): 309 | fr = Frame(master) 310 | Label(fr, text="S3 Uploader", font="Sans 14 bold").pack(padx=50) 311 | Label(fr, text="by Charles Daniel", font="Sans 10").pack(padx=50) 312 | Button(fr, text="OK", command=self.destroy).pack(padx=50, pady=10) 313 | fr.pack() 314 | 315 | def destroy(self): 316 | tkSimpleDialog.Dialog.destroy(self) 317 | 318 | def buttonbox(self): 319 | pass 320 | 321 | 322 | def main(config_file): 323 | root = Tk() 324 | s = Style(root) 325 | s.configure('Error.TLabel', foreground='red', background='yellow') 326 | s.configure('.', padx=10, pady=10) 327 | 328 | root.title("S3 File Uploader") 329 | root['background'] = '#ececec' 330 | 331 | about_win = AboutWin(root) 332 | s3uploader_win = S3FileUploader(root, config_file) 333 | s3uploader_win.pack() 334 | 335 | menubar = Menu(root) 336 | filemenu = Menu(menubar, tearoff=0) 337 | filemenu.add_command(label="About", command=about_win.show) 338 | filemenu.add_separator() 339 | filemenu.add_command(label="Open Config...", command=s3uploader_win.prompt_load_config) 340 | filemenu.add_separator() 341 | filemenu.add_command(label="Exit", command=root.quit) 342 | menubar.add_cascade(label="S3Uploader", menu=filemenu) 343 | root.config(menu=menubar) 344 | 345 | root.mainloop() 346 | 347 | if __name__=='__main__': 348 | main("~/s3_uploader.cfg") 349 | -------------------------------------------------------------------------------- /s3_uploader_icon.icns: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/charlesdaniel/s3_uploader/93114b0296041fc0905937927a6ef70672607972/s3_uploader_icon.icns -------------------------------------------------------------------------------- /s3_uploader_icon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/charlesdaniel/s3_uploader/93114b0296041fc0905937927a6ef70672607972/s3_uploader_icon.ico -------------------------------------------------------------------------------- /s3_uploader_icon.idraw: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/charlesdaniel/s3_uploader/93114b0296041fc0905937927a6ef70672607972/s3_uploader_icon.idraw -------------------------------------------------------------------------------- /s3_uploader_icon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/charlesdaniel/s3_uploader/93114b0296041fc0905937927a6ef70672607972/s3_uploader_icon.png -------------------------------------------------------------------------------- /s3_uploader_icon_favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/charlesdaniel/s3_uploader/93114b0296041fc0905937927a6ef70672607972/s3_uploader_icon_favicon.ico -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | This is a setup.py script generated by py2applet 3 | 4 | Usage: 5 | python setup.py py2app 6 | """ 7 | 8 | from setuptools import setup 9 | 10 | APP = ['app.py'] 11 | DATA_FILES = ['data'] 12 | OPTIONS = { 13 | 'iconfile': 's3_uploader_icon.icns', 14 | 'argv_emulation': True, 15 | 'includes': 'HTMLParser,ConfigParser,boto3,botocore,boto3.s3.inject' 16 | } 17 | 18 | setup( 19 | name="S3Uploader", 20 | version="1.0", 21 | app=APP, 22 | data_files=DATA_FILES, 23 | options={'py2app': OPTIONS}, 24 | setup_requires=['py2app'], 25 | ) 26 | --------------------------------------------------------------------------------