├── .gitignore ├── CONTRIBUTING.md ├── LICENSE ├── Procfile ├── README.md ├── app.json ├── deep21 ├── __init__.py ├── aws.py ├── management │ ├── __init__.py │ └── commands │ │ ├── __init__.py │ │ └── publish.py ├── manifest.yaml ├── models.py ├── settings.py ├── urls.py ├── views.py └── wsgi.py ├── env.template ├── manage.py ├── requirements.txt └── runtime.txt /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | *.py[cod] 3 | venv/ 4 | *.sqlite3 5 | .DS_Store 6 | .env 7 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Pull requests 2 | 3 | ## Code style 4 | 5 | > Programs must be written for people to read, and only incidentally for 6 | > machines to execute. 7 | > 8 | > -- Harold Abelson, Structure and Interpretation of Computer Programs 9 | 10 | At a minimum, changes to the code should be compliant with [PEP 11 | 8](https://www.python.org/dev/peps/pep-0008/). For general design principles, 12 | consult [The Zen of Python](https://www.python.org/dev/peps/pep-0020/). 13 | 14 | ## Commit message format 15 | 16 | Commit your changes using the following commit message format (with the same 17 | capitalization, spacing, and punctuation): 18 | 19 | ``` 20 | logging: Move function to module level 21 | ``` 22 | 23 | The first part of the message is the scope and the second part is a description 24 | of the change, written in the imperative tense. 25 | 26 | ## Commit history 27 | 28 | Commits should be organized into logical units. Keep your git history clean by 29 | [rewriting](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History) it as 30 | necessary using `git rebase -i` (interactive rebase) and `git push -f` (force 31 | push). Commits addressing review comments and test failures should be squashed 32 | if necessary. 33 | 34 | # Opening issues 35 | 36 | ## Bug reports 37 | 38 | Bug reports should include clear instructions to reproduce the bug. Include a 39 | stack trace if applicable. 40 | 41 | ## Security issues 42 | 43 | Critical security bugs should be sent via email to security@21.co and should not 44 | be publicly disclosed. Eligible security disclosures are eligible, at our sole 45 | discretion, for a monetary bounty of up to $1000 based on severity. 46 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015-2017, 21 Inc. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 14 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 17 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 18 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 19 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 20 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 21 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 22 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | 24 | The views and conclusions contained in the software and documentation are those 25 | of the authors and should not be interpreted as representing official policies, 26 | either expressed or implied, of 21 Inc. 27 | -------------------------------------------------------------------------------- /Procfile: -------------------------------------------------------------------------------- 1 | web: gunicorn deep21.wsgi --log-level=info --log-file - 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # A tutorial for using 21 and AWS for deep learning 2 | 3 | See the tutorial at [https://21.co/learn/deep-learning-aws](https://21.co/learn/deep-learning-aws/) 4 | 5 | ## Quick deploy 6 | 7 | [![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/21dotco/two1-deep-learning/tree/master) 8 | 9 | **Important:** You will not be able to quick-deploy without setting up an AWS 10 | account and the needed security profiles, etc. See the tutorial at 11 | [https://21.co/learn/deep-learning-aws](https://21.co/learn/deep-learning-aws/) 12 | for an in-depth description. Here is a short list of the parameters you will 13 | need 14 | 15 | - `AWS_ACCESS_KEY_ID` 16 | - `AWS_SECRET_ACCESS_KEY` 17 | - `AWS_DEFAULT_REGION` 18 | - `EC2_SSH_KEYPAIR_ID` 19 | - `EC2_IAM_INSTANCE_PROFILE_ARN` 20 | - `EC2_SECURITY_GROUP_NAME` 21 | - `S3_BUCKET_NAME` 22 | - `IMGUR_CLIENT_ID` 23 | - `IMGUR_CLIENT_SECRET` 24 | - `TWO1_USERNAME` 25 | - `TWO1_WALLET_MNEMONIC` 26 | 27 | You can get imgur API keys [here](https://api.imgur.com/oauth2/addclient). 28 | 29 | Finally, if you are going to use GPU instances, you should request a instance 30 | limit increase from AWS. Currently they only allow for 2 simultaneously running 31 | GPU instances. We have the environment variable `EC2_MAX_NUM_INSTANCES` to 32 | gracefully fail when the limit is reached, and it's set to 1 by default. 33 | 34 | If you clone this repository directly (and don't use one of the tagged 35 | releases) note that `ALLOWED_HOSTS` is set to a wildcard to enable the 36 | quick-deploy button above. For security purposes you should change this to your 37 | specific domain, and note that it's only enforced when DEBUG mode is off. 38 | -------------------------------------------------------------------------------- /app.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "21 Deep Learning", 3 | "description": "An example 21 endpoint that uses AWS for deep learning.", 4 | "keywords": [ 5 | "django", 6 | "heroku", 7 | "aws", 8 | "21", 9 | "bitcoin", 10 | "deep learning", 11 | "style transfer", 12 | "neural art" 13 | ], 14 | "website": "https://21.co", 15 | "repository": "https://github.com/21dotco/two1-deep-learning", 16 | "logo": "https://assets.21.co/shared/img/21logo.png", 17 | "scripts": { 18 | "postdeploy": "python manage.py migrate" 19 | }, 20 | "env": { 21 | "DISABLE_COLLECTSTATIC": { 22 | "description": "Disable static asset collection", 23 | "value": "1" 24 | }, 25 | "HASHIDS_SALT": { 26 | "description": "A salt for hashids", 27 | "generator": "secret" 28 | }, 29 | "TWO1_USERNAME": { 30 | "description": "21.co username" 31 | }, 32 | "TWO1_WALLET_MNEMONIC": { 33 | "description": "21 wallet mnemonic" 34 | }, 35 | "AWS_DEFAULT_REGION": { 36 | "description": "AWS region", 37 | "value": "us-east-1" 38 | }, 39 | "AWS_ACCESS_KEY_ID": { 40 | "description": "AWS access key" 41 | }, 42 | "AWS_SECRET_ACCESS_KEY": { 43 | "description": "AWS secret access key" 44 | }, 45 | "EC2_SSH_KEYPAIR_ID": { 46 | "description": "The ID of the EC2 SSH keypair to ssh into launched instances" 47 | }, 48 | "EC2_IAM_INSTANCE_PROFILE_ARN": { 49 | "description": "The IAM instance profile ARN to use for launched instances" 50 | }, 51 | "EC2_SECURITY_GROUP_NAME": { 52 | "description": "The EC2 security group to use for launched instances" 53 | }, 54 | "S3_BUCKET_NAME": { 55 | "description": "The S3 bucket to store inputs and outupts in" 56 | }, 57 | "IMGUR_CLIENT_ID": { 58 | "description": "The imgur API client ID" 59 | }, 60 | "IMGUR_CLIENT_SECRET": { 61 | "description": "The imgur API secret." 62 | }, 63 | "BUY_PRICE": { 64 | "description": "The price to buy this endpoint, in satoshis.", 65 | "value": "125000" 66 | }, 67 | "DEBUG": { 68 | "description": "Whether to run the app in DEBUG mode", 69 | "value": "False" 70 | } 71 | }, 72 | "formation": { 73 | "web": { 74 | "quantity": 1 75 | } 76 | }, 77 | "addons": [ 78 | "heroku-postgresql:hobby-dev" 79 | ], 80 | "buildpacks": [ 81 | { 82 | "url": "heroku/python" 83 | } 84 | ] 85 | } 86 | -------------------------------------------------------------------------------- /deep21/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/21dotco/two1-deep-learning/efe40771d58cb6d80a1bc0bc67e09c7e0e86ceca/deep21/__init__.py -------------------------------------------------------------------------------- /deep21/aws.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import logging 4 | import boto3 5 | 6 | from deep21 import settings 7 | 8 | logger = logging.getLogger(__name__) 9 | logging.basicConfig(stream=sys.stdout, level=logging.INFO) 10 | 11 | USERDATA_TEMPLATE = """#cloud-config 12 | 13 | runcmd: 14 | - export LD_LIBRARY_PATH=/home/ubuntu/torch-distro/install/lib:/usr/local/cuda/lib64:/home/ubuntu/cudnn/:$LD_LIBRARY_PATH 15 | - export PATH=/home/ubuntu/torch-distro/install/bin:/home/ubuntu/anaconda/bin:/usr/local/cuda/bin:$PATH 16 | - export DYLD_LIBRARY_PATH=/home/ubuntu/torch-distro/install/lib:$DYLD_LIBRARY_PATH 17 | - export PYTHONPATH=/home/ubuntu/caffe/python:$PYTHONPATH 18 | - export TH_RELEX_ROOT=/home/ubuntu/th-relation-extraction 19 | - export HOME=/home/ubuntu 20 | - cd /style-transfer-torch 21 | - aws --region {region} s3 cp s3://{bucket}/{content} ./{content} 22 | - aws --region {region} s3 cp s3://{bucket}/{style} ./{style} 23 | - th neural_style.lua -style_image {style} -content_image {content} -output_image {output} -gpu 0 -backend cudnn -cudnn_autotune -print_iter 50 -image_size 500 -num_iterations 500 -init image 24 | - aws --region {region} s3 cp ./{output} s3://{bucket}/{output} 25 | - shutdown -h now 26 | """ 27 | 28 | 29 | def num_running_instances(): 30 | ec2 = boto3.client('ec2') 31 | 32 | instances = ec2.describe_instances( 33 | Filters=[ 34 | {'Name': 'instance-state-name', 'Values': ['running', 'pending']}, 35 | {'Name': 'instance-type', 'Values': [settings.EC2_INSTANCE_TYPE]}, 36 | {'Name': 'image-id', 'Values': [settings.EC2_AMI_ID]}, 37 | ] 38 | ) 39 | 40 | return len(instances['Reservations']) 41 | 42 | 43 | def upload_to_s3(local_filename, s3_filename): 44 | logger.info('Uploading file {} to S3://{}'.format(local_filename, s3_filename)) 45 | s3 = boto3.client('s3') 46 | s3.upload_file(local_filename, settings.S3_BUCKET_NAME, s3_filename) 47 | 48 | 49 | def download_from_s3(local_filename, s3_filename): 50 | logger.info('Downloading file to {} from S3://{}'.format(local_filename, s3_filename)) 51 | s3 = boto3.client('s3') 52 | s3.download_file(settings.S3_BUCKET_NAME, s3_filename, local_filename) 53 | 54 | 55 | def spin_up(data): 56 | ec2 = boto3.resource('ec2') 57 | 58 | instances = ec2.create_instances( 59 | ImageId=settings.EC2_AMI_ID, 60 | InstanceType=settings.EC2_INSTANCE_TYPE, 61 | KeyName=settings.EC2_SSH_KEYPAIR_ID, 62 | MinCount=1, 63 | MaxCount=1, 64 | IamInstanceProfile={ 65 | 'Arn': settings.EC2_IAM_INSTANCE_PROFILE_ARN 66 | }, 67 | InstanceInitiatedShutdownBehavior='terminate', 68 | SecurityGroupIds=[settings.EC2_SECURITY_GROUP_NAME], 69 | UserData=USERDATA_TEMPLATE.format(**data) 70 | ) 71 | 72 | instance = instances[0] 73 | logger.info('Spinning up instance with id {} at {}'.format(instance.id, instance.launch_time)) 74 | 75 | return instance.id 76 | 77 | 78 | def launch(local_filepaths, data): 79 | ''' 80 | Copy the local files to S3, and then launch the aws instance with 81 | the appropriate parameters for the USER_DATA script. 82 | ''' 83 | 84 | for key, filepath in local_filepaths.items(): 85 | filename = os.path.split(filepath)[1] 86 | data[key] = filename 87 | 88 | if key in [settings.CONTENT_SUFFIX, settings.STYLE_SUFFIX]: 89 | upload_to_s3(filepath, filename) 90 | 91 | data['bucket'] = settings.S3_BUCKET_NAME 92 | data['region'] = settings.AWS_DEFAULT_REGION 93 | return spin_up(data) 94 | -------------------------------------------------------------------------------- /deep21/management/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/21dotco/two1-deep-learning/efe40771d58cb6d80a1bc0bc67e09c7e0e86ceca/deep21/management/__init__.py -------------------------------------------------------------------------------- /deep21/management/commands/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/21dotco/two1-deep-learning/efe40771d58cb6d80a1bc0bc67e09c7e0e86ceca/deep21/management/commands/__init__.py -------------------------------------------------------------------------------- /deep21/management/commands/publish.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from datetime import datetime 3 | 4 | from django.core.management.base import BaseCommand 5 | 6 | from deep21 import settings 7 | from two1.commands import publish 8 | from two1.server import rest_client 9 | 10 | 11 | class Command(BaseCommand): 12 | help = 'Publish your app to the marketplace' 13 | 14 | def __init__(self): 15 | super().__init__() 16 | self._logger = logging.getLogger('deep21.publish') 17 | self._username = settings.TWO1_USERNAME 18 | self._client = rest_client.TwentyOneRestClient(username=self._username, wallet=settings.WALLET) 19 | 20 | def handle(self, *args, **options): 21 | manifest_path = 'deep21/manifest.yaml' 22 | app_name = 'Deep21' 23 | try: 24 | publish._publish(self._client, manifest_path, '21market', True, {}) 25 | self._logger.info( 26 | '%s publishing %s - published: True, Timestamp: %s' % 27 | (self._username, app_name, datetime.now()) 28 | ) 29 | except Exception as e: 30 | self._logger.error( 31 | '%s publishing %s - published: False, error: %s, Timestamp: %s' % 32 | (self._username, app_name, e, datetime.now()) 33 | ) 34 | -------------------------------------------------------------------------------- /deep21/manifest.yaml: -------------------------------------------------------------------------------- 1 | basePath: / 2 | host: 3 | schemes: [https] 4 | swagger: '2.0' 5 | x-21-manifest-path: /manifest 6 | info: 7 | contact: 8 | email: support@21.co 9 | name: 21dotco 10 | url: 'https://21.co' 11 | description: Transfer the style of one image to another image using deep learning. 12 | license: 13 | name: MIT LICENSE 14 | url: 'https://opensource.org/licenses/MIT' 15 | termsOfService: https://opensource.org/licenses/MIT 16 | title: Deep Learning AWS Tutorial 17 | version: '1.0.0' 18 | x-21-app-image: https://cdn.filepicker.io/api/file/mFDb3ur8R6OhsP3uvNVU 19 | x-21-category: entertainment 20 | x-21-keywords: 21 | neural 22 | network 23 | x-21-quick-buy: "21 buy url \"http:///buy\" --data '{\"style\":\"http://i.imgur.com/3is4qSi.jpg\", \"content\":\"http://i.imgur.com/8FwCskQ.jpg\"}" 24 | x-21-total-price: 25 | min: 200000 26 | max: 200000 27 | x-21-usage: '' 28 | paths: 29 | /buy: 30 | post: 31 | summary: Buy a style transfer 32 | consumes: 33 | - application/json 34 | produces: 35 | - application/json 36 | parameters: 37 | - in: body 38 | name: data 39 | description: parameters for the style transfer 40 | required: true 41 | schema: 42 | $ref: "#/definitions/Data" 43 | responses: 44 | "200": 45 | description: Successful initialization of style transfer algorithm 46 | /redeem: 47 | get: 48 | summary: Redeems a token 49 | description: Checks to see if the computation is done, and if so returns an imgur link to the finished image. 50 | produces: 51 | - application/json 52 | parameters: 53 | - in: query 54 | name: token 55 | description: The token to redeem 56 | required: true 57 | type: string 58 | responses: 59 | "200": 60 | description: successful operation 61 | schema: 62 | type: object 63 | properties: 64 | message: 65 | type: string 66 | status: 67 | type: string 68 | url: 69 | type: string 70 | "400": 71 | description: Invalid or redeemed token 72 | schema: 73 | type: object 74 | properties: 75 | message: 76 | type: string 77 | status: 78 | type: string 79 | definitions: 80 | Data: 81 | type: object 82 | properties: 83 | style: 84 | type: string 85 | content: 86 | type: string 87 | -------------------------------------------------------------------------------- /deep21/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | from django.utils import timezone 3 | 4 | 5 | class Request(models.Model): 6 | ''' 7 | A model representing a single request from a user. 8 | ''' 9 | 10 | created = models.DateTimeField(default=timezone.now) 11 | 12 | ''' 13 | A token given to the user, a hashid of the database id, 14 | which is also used to name files on S3, etc. 15 | ''' 16 | token = models.CharField(max_length=100, null=True, default=None) 17 | 18 | ''' 19 | The server filepath for output image to store temporarily between 20 | fetching from s3 and uploading to imgur. 21 | ''' 22 | output_filepath = models.CharField(max_length=150, null=True) 23 | 24 | ''' 25 | The name of the output file on S3. 26 | ''' 27 | output_s3_filename = models.CharField(max_length=150, null=True) 28 | 29 | ''' 30 | True when the token has successfully been redeemed, False otherwise. 31 | ''' 32 | redeemed = models.BooleanField(default=False) 33 | -------------------------------------------------------------------------------- /deep21/settings.py: -------------------------------------------------------------------------------- 1 | """ 2 | Django settings for deep21 project. 3 | """ 4 | 5 | import os 6 | import dotenv 7 | import dj_database_url 8 | from two1.wallet import Two1Wallet 9 | 10 | 11 | BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 12 | PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__)) 13 | 14 | dotenv_file = os.path.join(BASE_DIR, ".env") 15 | if os.path.isfile(dotenv_file): 16 | dotenv.load_dotenv(dotenv_file) 17 | 18 | ''' 19 | Project-specific env vars 20 | ''' 21 | 22 | # AWS access keys 23 | AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY") 24 | AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY") 25 | 26 | # imgur access keys 27 | IMGUR_CLIENT_ID = os.environ.get("IMGUR_CLIENT_ID") 28 | IMGUR_CLIENT_SECRET = os.environ.get("IMGUR_CLIENT_SECRET") 29 | 30 | # EC2 instance configuration keys 31 | S3_BUCKET_NAME = os.environ.get("S3_BUCKET_NAME") 32 | EC2_SSH_KEYPAIR_ID = os.environ.get("EC2_SSH_KEYPAIR_ID") 33 | EC2_SECURITY_GROUP_NAME = os.environ.get("EC2_SECURITY_GROUP_NAME") 34 | EC2_IAM_INSTANCE_PROFILE_ARN = os.environ.get("EC2_IAM_INSTANCE_PROFILE_ARN") 35 | EC2_MAX_NUM_INSTANCES = int(os.environ.get("EC2_MAX_NUM_INSTANCES", '1')) 36 | 37 | # if you change the default region or the custom AMI, you may 38 | # need to change/set these config variables. 39 | EC2_AMI_ID = os.environ.get("EC2_AMI_ID", 'ami-1ab24377') 40 | EC2_INSTANCE_TYPE = os.environ.get("EC2_INSTANCE_TYPE", 'g2.2xlarge') 41 | AWS_DEFAULT_REGION = os.environ.get("AWS_DEFAULT_REGION", 'us-east-1') 42 | 43 | CONTENT_SUFFIX = 'content' 44 | STYLE_SUFFIX = 'style' 45 | OUTPUT_SUFFIX = 'output' 46 | 47 | BUY_PRICE = int(os.environ.get('BUY_PRICE', '125000')) 48 | HASHIDS_SALT = os.environ.get("HASHIDS_SALT") 49 | TMP_DIR = os.environ.get("TMP_DIR", '/tmp/') 50 | 51 | DEBUG = os.environ.get("DEBUG", "False").lower() in ['true', 't', '1'] 52 | 53 | ''' 54 | 21 settings 55 | ''' 56 | 57 | TWO1_WALLET_MNEMONIC = os.environ.get("TWO1_WALLET_MNEMONIC") 58 | TWO1_USERNAME = os.environ.get("TWO1_USERNAME") 59 | WALLET = Two1Wallet.import_from_mnemonic(mnemonic=TWO1_WALLET_MNEMONIC) 60 | 61 | ''' 62 | Django settings 63 | ''' 64 | 65 | APPEND_SLASH = False 66 | SECRET_KEY = 'not-used' 67 | 68 | ALLOWED_HOSTS = ['*'] 69 | 70 | # Application definition 71 | 72 | INSTALLED_APPS = [ 73 | 'django.contrib.admin', 74 | 'django.contrib.auth', 75 | 'django.contrib.contenttypes', 76 | 'django.contrib.sessions', 77 | 'django.contrib.messages', 78 | 'django.contrib.staticfiles', 79 | 'django_extensions', 80 | 'two1.bitserv.django', 81 | 'deep21', 82 | ] 83 | 84 | MIDDLEWARE_CLASSES = [ 85 | 'django.middleware.security.SecurityMiddleware', 86 | 'django.contrib.sessions.middleware.SessionMiddleware', 87 | 'django.middleware.common.CommonMiddleware', 88 | 'django.middleware.csrf.CsrfViewMiddleware', 89 | 'django.contrib.auth.middleware.AuthenticationMiddleware', 90 | 'django.contrib.auth.middleware.SessionAuthenticationMiddleware', 91 | 'django.contrib.messages.middleware.MessageMiddleware', 92 | 'django.middleware.clickjacking.XFrameOptionsMiddleware', 93 | ] 94 | 95 | ROOT_URLCONF = 'deep21.urls' 96 | 97 | TEMPLATES = [ 98 | { 99 | 'BACKEND': 'django.template.backends.django.DjangoTemplates', 100 | 'DIRS': [], 101 | 'APP_DIRS': True, 102 | 'OPTIONS': { 103 | 'context_processors': [ 104 | 'django.template.context_processors.debug', 105 | 'django.template.context_processors.request', 106 | 'django.contrib.auth.context_processors.auth', 107 | 'django.contrib.messages.context_processors.messages', 108 | ], 109 | }, 110 | }, 111 | ] 112 | 113 | WSGI_APPLICATION = 'deep21.wsgi.application' 114 | 115 | # Database 116 | DATABASES = {} 117 | DATABASES['default'] = dj_database_url.config(conn_max_age=600) 118 | 119 | LANGUAGE_CODE = 'en-us' 120 | TIME_ZONE = 'UTC' 121 | USE_I18N = True 122 | USE_L10N = True 123 | USE_TZ = True 124 | 125 | STATIC_URL = '/static/' 126 | -------------------------------------------------------------------------------- /deep21/urls.py: -------------------------------------------------------------------------------- 1 | """ 2 | deep21 URL Configuration 3 | """ 4 | from django.conf.urls import url, include 5 | import deep21.views 6 | 7 | urlpatterns = [ 8 | url(r'^buy$', deep21.views.buy), 9 | url(r'^redeem$', deep21.views.redeem), 10 | url(r'^manifest$', deep21.views.manifest), 11 | url(r'^payments/', include('two1.bitserv.django.urls')), 12 | ] 13 | -------------------------------------------------------------------------------- /deep21/views.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import logging 4 | import requests 5 | import shutil 6 | import pyimgur 7 | import botocore 8 | import hashids 9 | import yaml 10 | 11 | from django.core.exceptions import ValidationError, ObjectDoesNotExist 12 | from django.http import JsonResponse 13 | from rest_framework.decorators import api_view 14 | from two1.bitserv.django import payment 15 | 16 | from deep21 import settings 17 | from deep21 import aws 18 | from deep21.models import Request 19 | 20 | hasher = hashids.Hashids(salt=settings.HASHIDS_SALT, min_length=5) 21 | 22 | logger = logging.getLogger(__name__) 23 | logging.basicConfig(stream=sys.stdout, level=logging.INFO) 24 | 25 | 26 | def filepaths(token): 27 | names = {} 28 | 29 | for fname in [settings.CONTENT_SUFFIX, settings.STYLE_SUFFIX, settings.OUTPUT_SUFFIX]: 30 | names[fname] = '{}{}-{}.jpg'.format(settings.TMP_DIR, token, fname) 31 | 32 | return names 33 | 34 | 35 | def fetch_files(data, filepath_dict): 36 | ''' 37 | Fetch the files given by urls in data['style'] and data['content'] 38 | and save them to the corresponding file paths given in filepath_dict. 39 | ''' 40 | logger.info('Fetching remote files') 41 | for key, filepath in filepath_dict.items(): 42 | if key != settings.OUTPUT_SUFFIX: 43 | file_url = data[key] 44 | logger.info('Fetching remote {} file: {}'.format(key, file_url)) 45 | response = requests.get(file_url, stream=True) 46 | 47 | if response.status_code == 200: 48 | with open(filepath, 'wb') as outfile: 49 | response.raw.decode_content = True 50 | shutil.copyfileobj(response.raw, outfile) 51 | else: 52 | raise FileNotFoundError('Received 404 when fetching {}'.format(file_url)) 53 | 54 | 55 | def validate_buy_params(data): 56 | validated_data = {} 57 | 58 | try: 59 | content = data['content'] 60 | if not content.endswith('.jpg'): 61 | raise ValidationError('Content image must be a url to a .jpg file') 62 | validated_data['content'] = content 63 | 64 | style = data['style'] 65 | if not style.endswith('.jpg'): 66 | raise ValidationError('Content image must be a url to a .jpg file') 67 | validated_data['style'] = style 68 | except KeyError as e: 69 | raise ValidationError("'{}' must be specified as a POST parameter".format(e.args[0])) 70 | 71 | if aws.num_running_instances() >= settings.EC2_MAX_NUM_INSTANCES: 72 | raise ValidationError('All EC2 instances are busy. Try again in 20 minutes.') 73 | 74 | return validated_data 75 | 76 | 77 | def _execute_buy(data): 78 | request = Request.objects.create() 79 | 80 | request.token = hasher.encode(request.id) 81 | filepath_dict = filepaths(request.token) 82 | 83 | request.output_filepath = filepath_dict[settings.OUTPUT_SUFFIX] 84 | request.output_s3_filename = os.path.split(request.output_filepath)[1] 85 | request.save() 86 | 87 | try: 88 | fetch_files(data, filepath_dict) 89 | except FileNotFoundError as e: 90 | return JsonResponse({"error": e.message}, status=404) 91 | 92 | try: 93 | aws.launch(filepath_dict, data) 94 | except Exception as e: 95 | return JsonResponse({"error": "Error with AWS: {}".format(str(e))}, status=500) 96 | 97 | return JsonResponse({"token": request.token}, status=200) 98 | 99 | 100 | @api_view(['POST']) 101 | @payment.required(settings.BUY_PRICE) 102 | def buy(request): 103 | try: 104 | data = validate_buy_params(request.data) 105 | except ValidationError as error: 106 | return JsonResponse({"error": error.message}, status=400) 107 | 108 | return _execute_buy(data) 109 | 110 | 111 | def try_download_output(request): 112 | s3_filename = request.output_s3_filename 113 | output_filepath = request.output_filepath 114 | aws.download_from_s3(output_filepath, s3_filename) 115 | 116 | 117 | def validate_redeem_params(request): 118 | try: 119 | token = request.GET['token'] 120 | except KeyError: 121 | raise ValidationError({ 122 | "error_message": 123 | "'token' must be specified as a GET parameter" 124 | }) 125 | 126 | return token 127 | 128 | 129 | def _redeem(token): 130 | try: 131 | request = Request.objects.get(token=token) 132 | if request.redeemed: 133 | raise ValueError() 134 | 135 | try_download_output(request) 136 | except botocore.exceptions.ClientError as e: 137 | logger.error('Download from S3 failed with error: {}'.format(str(e))) 138 | return JsonResponse({'status': 'working', 'message': 'Not yet finished.'}, status=202) 139 | except ObjectDoesNotExist: 140 | logger.error('User requested token {} that does not exist'.format(token)) 141 | return JsonResponse({'error': 'Invalid or redeemed token.'}, status=400) 142 | except ValueError: 143 | logger.error('User requested token {} that was already redeemed'.format(token)) 144 | return JsonResponse({'error': 'Invalid or redeemed token.'}, status=400) 145 | 146 | imgur = pyimgur.Imgur(settings.IMGUR_CLIENT_ID) 147 | uploaded_image = imgur.upload_image(request.output_filepath, title='Style transfer output {}'.format(token)) 148 | url = uploaded_image.link 149 | 150 | request.redeemed = True 151 | request.save() 152 | 153 | return JsonResponse({"status": "finished", "url": url, "message": "Thanks!"}, status=200) 154 | 155 | 156 | @api_view(['GET']) 157 | def redeem(request): 158 | try: 159 | token = validate_redeem_params(request) 160 | except ValidationError as error: 161 | return JsonResponse({"error": error.message}, status=400) 162 | 163 | return _redeem(token) 164 | 165 | 166 | @api_view(['GET']) 167 | def manifest(request): 168 | with open(settings.BASE_DIR + "/deep21/manifest.yaml", 'r') as infile: 169 | return JsonResponse(yaml.load(infile), status=200) 170 | -------------------------------------------------------------------------------- /deep21/wsgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | WSGI config for deep21 project. 3 | 4 | It exposes the WSGI callable as a module-level variable named ``application``. 5 | 6 | For more information on this file, see 7 | https://docs.djangoproject.com/en/1.8/howto/deployment/wsgi/ 8 | """ 9 | 10 | import os 11 | 12 | from django.core.wsgi import get_wsgi_application 13 | 14 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "deep21.settings") 15 | 16 | application = get_wsgi_application() 17 | -------------------------------------------------------------------------------- /env.template: -------------------------------------------------------------------------------- 1 | DATABASE_URL=sqlite:///db.sqlite3 2 | 3 | AWS_ACCESS_KEY_ID= 4 | AWS_SECRET_ACCESS_KEY= 5 | AWS_DEFAULT_REGION=us-east-1 # needed to use the custom AMI from the tutorial, ami-1ab24377 6 | 7 | IMGUR_CLIENT_ID= 8 | IMGUR_CLIENT_SECRET= 9 | 10 | S3_BUCKET_NAME= 11 | EC2_SSH_KEYPAIR_ID= 12 | EC2_IAM_INSTANCE_PROFILE_ARN= 13 | EC2_SECURITY_GROUP_NAME= 14 | EC2_MAX_NUM_INSTANCES=1 15 | 16 | HASHIDS_SALT= 17 | 18 | TWO1_WALLET_MNEMONIC= 19 | TWO1_USERNAME= 20 | 21 | DEBUG=True # set this to False when you deploy to production 22 | -------------------------------------------------------------------------------- /manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import sys 4 | 5 | if __name__ == "__main__": 6 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "deep21.settings") 7 | 8 | from django.core.management import execute_from_command_line 9 | 10 | execute_from_command_line(sys.argv) 11 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | aiohttp==0.17.3 2 | boto3==1.3.1 3 | botocore==1.4.26 4 | Django~=1.8.0 5 | django-extensions==1.5.7 6 | django-storages==1.1.8 7 | djangorestframework==3.2.3 8 | dj-database-url==0.4.1 9 | gunicorn==19.3.0 10 | psycopg2==2.6.1 11 | flake8==2.4.1 12 | pep8==1.5.7 13 | pyflakes==0.8.1 14 | python-dotenv==0.1.3 15 | requests==2.7.0 16 | hashids==1.1.0 17 | pyimgur==0.5.2 18 | two1==3.4.1 19 | pyyaml==3.11 20 | -------------------------------------------------------------------------------- /runtime.txt: -------------------------------------------------------------------------------- 1 | python-3.5.1 2 | --------------------------------------------------------------------------------