├── .gitignore ├── .travis.yml ├── README.rst ├── dj_elastictranscoder ├── __init__.py ├── admin.py ├── migrations │ ├── 0001_initial.py │ └── __init__.py ├── models.py ├── signals.py ├── south_migrations │ ├── 0001_initial.py │ └── __init__.py ├── transcoder.py ├── urls.py ├── utils.py └── views.py ├── docs └── images │ └── workflow.jpg ├── setup.cfg ├── setup.py ├── testsapp ├── __init__.py ├── fixtures │ ├── oncomplete.json │ ├── onerror.json │ ├── onprogress.json │ └── submit.json ├── models.py ├── requirements.txt ├── test_job.py └── tests_settings.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg-info 2 | *.pyc 3 | *.swp 4 | .cache 5 | .coverage 6 | .DS_Store 7 | .eggs 8 | .tox 9 | /dist 10 | /wheelhouse 11 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - "2.7" 5 | 6 | env: 7 | matrix: 8 | - TOX_ENV=py27-dj14 9 | - TOX_ENV=py27-dj15 10 | - TOX_ENV=py27-dj16 11 | - TOX_ENV=py27-dj17 12 | - TOX_ENV=py27-dj18 13 | - TOX_ENV=py33-dj15 14 | - TOX_ENV=py33-dj16 15 | - TOX_ENV=py33-dj17 16 | - TOX_ENV=py33-dj18 17 | - TOX_ENV=py34-dj15 18 | - TOX_ENV=py34-dj16 19 | - TOX_ENV=py34-dj17 20 | - TOX_ENV=py34-dj18 21 | - TOX_ENV=pypy-dj14 22 | - TOX_ENV=pypy-dj15 23 | - TOX_ENV=pypy-dj16 24 | - TOX_ENV=pypy-dj17 25 | - TOX_ENV=pypy-dj18 26 | - TOX_ENV=py27-cov 27 | 28 | install: 29 | - pip install tox 30 | 31 | script: 32 | - tox -e $TOX_ENV 33 | after_success: 34 | - coveralls 35 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Django Elastic Transcoder 2 | ========================= 3 | 4 | |Build Status| |Coverage Status| 5 | 6 | ``django-elastic-transcoder`` is an `Django` app, let you integrate AWS Elastic Transcoder in Django easily. 7 | 8 | What is provided in this package? 9 | 10 | - ``Transcoder`` class 11 | - URL endpoint for receive SNS notification 12 | - Signals for PROGRESS, ERROR, COMPLETE 13 | - ``EncodeJob`` model 14 | 15 | Workflow 16 | ----------- 17 | 18 | .. image:: https://github.com/StreetVoice/django-elastic-transcoder/blob/master/docs/images/workflow.jpg 19 | 20 | 21 | Install 22 | ------- 23 | 24 | First, install ``dj_elastictranscode`` with ``pip`` 25 | 26 | .. code:: sh 27 | 28 | $ pip install django-elastic-transcoder 29 | 30 | Then, add ``dj_elastictranscoder`` to ``INSTALLED_APPS`` 31 | 32 | .. code:: python 33 | 34 | INSTALLED_APPS = ( 35 | ... 36 | 'dj_elastictranscoder', 37 | ... 38 | ) 39 | 40 | Bind ``urls.py`` 41 | 42 | .. code:: python 43 | 44 | urlpatterns = patterns('', 45 | ... 46 | url(r'^dj_elastictranscoder/', include('dj_elastictranscoder.urls')), 47 | ... 48 | ) 49 | 50 | Migrate 51 | 52 | .. code:: sh 53 | 54 | $ ./manage.py migrate 55 | 56 | Setting up AWS Elastic Transcoder 57 | --------------------------------- 58 | 59 | 1. Create a new ``Pipeline`` in AWS Elastic Transcoder. 60 | 2. Hookup every Notification. 61 | 3. Subscribe SNS Notification through HTTP 62 | 4. You are ready to encode! 63 | 64 | 65 | Required Django settings 66 | ------------------------- 67 | 68 | Please settings up variables below to make this app works. 69 | 70 | .. code:: python 71 | 72 | AWS_ACCESS_KEY_ID = 73 | AWS_SECRET_ACCESS_KEY = 74 | AWS_REGION = 75 | 76 | Usage 77 | ----- 78 | 79 | For instance, encode an mp3 80 | 81 | .. code:: python 82 | 83 | from dj_elastictranscoder.transcoder import Transcoder 84 | 85 | input = { 86 | 'Key': 'path/to/input.mp3', 87 | } 88 | 89 | outputs = [{ 90 | 'Key': 'path/to/output.mp3', 91 | 'PresetId': '1351620000001-300040' # for example: 128k mp3 audio preset 92 | }] 93 | 94 | pipeline_id = '' 95 | 96 | transcoder = Transcoder(pipeline_id) 97 | transcoder.encode(input, outputs) 98 | 99 | # your can also create a EncodeJob for object automatically 100 | transcoder.create_job_for_object(obj) 101 | 102 | 103 | # Transcoder can also work standalone without Django 104 | # just pass region and required aws key/secret to Transcoder, when initiate 105 | 106 | transcoder = Transcoder(pipeline_id, AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) 107 | 108 | 109 | Setting Up AWS SNS endpoint 110 | --------------------------------- 111 | 112 | AWS Elastic Transcoder can send various SNS notification to notify your application, like ``PROGRESS``, ``ERROR``, ``WARNING`` and ``COMPLETE`` 113 | 114 | So this package provide a endpoint to receieve these notifications, for you to update transcode progress. without checking by your self. 115 | 116 | Go to SNS section in AWS WebConsole to choose topic and subscribe with the url below. 117 | 118 | ``http:///dj_elastictranscoder/endpoint/`` 119 | 120 | Before notification get started to work, you have to activate SNS subscription, you will receive email with activation link. 121 | 122 | After subscribe is done, you will receive SNS notification. 123 | 124 | 125 | Signals 126 | ----------- 127 | 128 | This package provide various signals for you to get notification, and do more things in your application. you can check the signals usage in tests.py for more usage example. 129 | 130 | * transcode_onprogress 131 | * transcode_onerror 132 | * transcode_oncomplete 133 | 134 | 135 | .. |Build Status| image:: https://travis-ci.org/StreetVoice/django-elastic-transcoder.png?branch=master 136 | :target: https://travis-ci.org/StreetVoice/django-elastic-transcoder 137 | .. |Coverage Status| image:: https://coveralls.io/repos/StreetVoice/django-elastic-transcoder/badge.png?branch=master 138 | :target: https://coveralls.io/r/StreetVoice/django-elastic-transcoder?branch=master 139 | -------------------------------------------------------------------------------- /dj_elastictranscoder/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = '1.0.4' 2 | -------------------------------------------------------------------------------- /dj_elastictranscoder/admin.py: -------------------------------------------------------------------------------- 1 | from django.contrib import admin 2 | 3 | from .models import EncodeJob 4 | 5 | 6 | class EncodeJobAdmin(admin.ModelAdmin): 7 | list_display = ('id', 'state', 'message') 8 | list_filters = ('state',) 9 | 10 | admin.site.register(EncodeJob, EncodeJobAdmin) 11 | -------------------------------------------------------------------------------- /dj_elastictranscoder/migrations/0001_initial.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import unicode_literals 3 | 4 | from django.db import models, migrations 5 | 6 | 7 | class Migration(migrations.Migration): 8 | 9 | dependencies = [ 10 | ('contenttypes', '0001_initial'), 11 | ] 12 | 13 | operations = [ 14 | migrations.CreateModel( 15 | name='EncodeJob', 16 | fields=[ 17 | ('id', models.CharField(max_length=100, serialize=False, primary_key=True)), 18 | ('object_id', models.PositiveIntegerField()), 19 | ('state', models.PositiveIntegerField(default=0, db_index=True, choices=[(0, b'Submitted'), (1, b'Progressing'), (2, b'Error'), (3, b'Warning'), (4, b'Complete')])), 20 | ('message', models.TextField()), 21 | ('created_at', models.DateTimeField(auto_now_add=True)), 22 | ('last_modified', models.DateTimeField(auto_now=True)), 23 | ('content_type', models.ForeignKey(to='contenttypes.ContentType')), 24 | ], 25 | ), 26 | ] 27 | -------------------------------------------------------------------------------- /dj_elastictranscoder/migrations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StreetVoice/django-elastic-transcoder/b26a6c9e6a482777ebaddc03d8cae96e7012bd07/dj_elastictranscoder/migrations/__init__.py -------------------------------------------------------------------------------- /dj_elastictranscoder/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | from django.contrib.contenttypes.models import ContentType 3 | 4 | import django 5 | if django.VERSION >= (1, 8): 6 | from django.contrib.contenttypes.fields import GenericForeignKey 7 | else: 8 | from django.contrib.contenttypes.generic import GenericForeignKey 9 | 10 | 11 | class EncodeJob(models.Model): 12 | STATE_CHOICES = ( 13 | (0, 'Submitted'), 14 | (1, 'Progressing'), 15 | (2, 'Error'), 16 | (3, 'Warning'), 17 | (4, 'Complete'), 18 | ) 19 | id = models.CharField(max_length=100, primary_key=True) 20 | content_type = models.ForeignKey(ContentType) 21 | object_id = models.PositiveIntegerField() 22 | state = models.PositiveIntegerField(choices=STATE_CHOICES, default=0, db_index=True) 23 | content_object = GenericForeignKey() 24 | message = models.TextField() 25 | created_at = models.DateTimeField(auto_now_add=True) 26 | last_modified = models.DateTimeField(auto_now=True) 27 | -------------------------------------------------------------------------------- /dj_elastictranscoder/signals.py: -------------------------------------------------------------------------------- 1 | from django.dispatch import Signal 2 | 3 | 4 | transcode_onprogress = Signal(providing_args=['job', 'job_response']) 5 | transcode_oncomplete = Signal(providing_args=['job', 'job_response']) 6 | transcode_onerror = Signal(providing_args=['job', 'job_response']) 7 | -------------------------------------------------------------------------------- /dj_elastictranscoder/south_migrations/0001_initial.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from south.utils import datetime_utils as datetime 3 | from south.db import db 4 | from south.v2 import SchemaMigration 5 | from django.db import models 6 | 7 | 8 | class Migration(SchemaMigration): 9 | 10 | def forwards(self, orm): 11 | # Adding model 'EncodeJob' 12 | db.create_table(u'dj_elastictranscoder_encodejob', ( 13 | ('id', self.gf('django.db.models.fields.CharField')(max_length=100, primary_key=True)), 14 | ('content_type', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['contenttypes.ContentType'])), 15 | ('object_id', self.gf('django.db.models.fields.PositiveIntegerField')()), 16 | ('state', self.gf('django.db.models.fields.PositiveIntegerField')(default=0, db_index=True)), 17 | ('message', self.gf('django.db.models.fields.TextField')()), 18 | ('created_at', self.gf('django.db.models.fields.DateTimeField')(auto_now_add=True, blank=True)), 19 | ('last_modified', self.gf('django.db.models.fields.DateTimeField')(auto_now=True, blank=True)), 20 | )) 21 | db.send_create_signal(u'dj_elastictranscoder', ['EncodeJob']) 22 | 23 | 24 | def backwards(self, orm): 25 | # Deleting model 'EncodeJob' 26 | db.delete_table(u'dj_elastictranscoder_encodejob') 27 | 28 | 29 | models = { 30 | u'contenttypes.contenttype': { 31 | 'Meta': {'ordering': "('name',)", 'unique_together': "(('app_label', 'model'),)", 'object_name': 'ContentType', 'db_table': "'django_content_type'"}, 32 | 'app_label': ('django.db.models.fields.CharField', [], {'max_length': '100'}), 33 | u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}), 34 | 'model': ('django.db.models.fields.CharField', [], {'max_length': '100'}), 35 | 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'}) 36 | }, 37 | u'dj_elastictranscoder.encodejob': { 38 | 'Meta': {'object_name': 'EncodeJob'}, 39 | 'content_type': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['contenttypes.ContentType']"}), 40 | 'created_at': ('django.db.models.fields.DateTimeField', [], {'auto_now_add': 'True', 'blank': 'True'}), 41 | 'id': ('django.db.models.fields.CharField', [], {'max_length': '100', 'primary_key': 'True'}), 42 | 'last_modified': ('django.db.models.fields.DateTimeField', [], {'auto_now': 'True', 'blank': 'True'}), 43 | 'message': ('django.db.models.fields.TextField', [], {}), 44 | 'object_id': ('django.db.models.fields.PositiveIntegerField', [], {}), 45 | 'state': ('django.db.models.fields.PositiveIntegerField', [], {'default': '0', 'db_index': 'True'}) 46 | } 47 | } 48 | 49 | complete_apps = ['dj_elastictranscoder'] -------------------------------------------------------------------------------- /dj_elastictranscoder/south_migrations/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StreetVoice/django-elastic-transcoder/b26a6c9e6a482777ebaddc03d8cae96e7012bd07/dj_elastictranscoder/south_migrations/__init__.py -------------------------------------------------------------------------------- /dj_elastictranscoder/transcoder.py: -------------------------------------------------------------------------------- 1 | from django.contrib.contenttypes.models import ContentType 2 | 3 | from .models import EncodeJob 4 | from .utils import get_setting_or_raise 5 | 6 | 7 | class Transcoder(object): 8 | 9 | def start_job(self, obj, transcode_kwargs, message=''): 10 | raise NotImplementedError() 11 | 12 | 13 | class AWSTranscoder(Transcoder): 14 | 15 | def __init__(self, access_key_id=None, secret_access_key=None, pipeline_id=None, region=None): 16 | if not access_key_id: 17 | access_key_id = get_setting_or_raise('AWS_ACCESS_KEY_ID') 18 | self.access_key_id = access_key_id 19 | 20 | if not secret_access_key: 21 | secret_access_key = get_setting_or_raise('AWS_SECRET_ACCESS_KEY') 22 | self.secret_access_key = secret_access_key 23 | 24 | if not pipeline_id: 25 | pipeline_id = get_setting_or_raise('AWS_TRANSCODER_PIPELINE_ID') 26 | self.pipeline_id = pipeline_id 27 | 28 | if not region: 29 | region = get_setting_or_raise('AWS_REGION') 30 | self.region = region 31 | 32 | from boto3.session import Session 33 | 34 | boto_session = Session( 35 | aws_access_key_id=self.access_key_id, 36 | aws_secret_access_key=self.secret_access_key, 37 | region_name=self.region, 38 | ) 39 | self.client = boto_session.client('elastictranscoder') 40 | 41 | def start_job(self, obj, transcode_kwargs, message=''): 42 | """ 43 | https://boto3.readthedocs.io/en/latest/reference/services/elastictranscoder.html#ElasticTranscoder.Client.create_job 44 | """ 45 | 46 | if 'PipelineId' not in transcode_kwargs: 47 | transcode_kwargs['PipelineId'] = self.pipeline_id 48 | 49 | ret = self.client.create_job(**transcode_kwargs) 50 | 51 | content_type = ContentType.objects.get_for_model(obj) 52 | job = EncodeJob() 53 | job.id = ret['Job']['Id'] 54 | job.content_type = content_type 55 | job.object_id = obj.pk 56 | job.message = message 57 | job.save() 58 | 59 | 60 | class QiniuTranscoder(Transcoder): 61 | 62 | def __init__( 63 | self, 64 | access_key=None, 65 | secret_key=None, 66 | pipeline_id=None, 67 | bucket_name=None, 68 | notify_url=None, 69 | ): 70 | if not access_key: 71 | access_key = get_setting_or_raise('QINIU_ACCESS_KEY') 72 | self.access_key = access_key 73 | 74 | if not secret_key: 75 | secret_key = get_setting_or_raise('QINIU_SECRET_KEY') 76 | self.secret_key = secret_key 77 | 78 | if not pipeline_id: 79 | pipeline_id = get_setting_or_raise('QINIU_TRANSCODE_PIPELINE_ID') 80 | self.pipeline_id = pipeline_id 81 | 82 | if not bucket_name: 83 | bucket_name = get_setting_or_raise('QINIU_TRANSCODE_BUCKET_NAME') 84 | self.bucket_name = bucket_name 85 | 86 | if not notify_url: 87 | notify_url = get_setting_or_raise('QINIU_TRANSCODE_NOTIFY_URL') 88 | self.notify_url = notify_url 89 | 90 | from qiniu import Auth 91 | 92 | self.client = Auth(self.access_key, self.secret_key) 93 | 94 | def start_job(self, obj, transcode_kwargs, message=''): 95 | """ 96 | https://developer.qiniu.com/dora/manual/1248/audio-and-video-transcoding-avthumb 97 | """ 98 | 99 | from qiniu import PersistentFop 100 | 101 | if 'force' not in transcode_kwargs: 102 | transcode_kwargs['force'] = 1 103 | 104 | pfop = PersistentFop(self.client, self.bucket_name, self.pipeline_id, self.notify_url) 105 | ret, info = pfop.execute(**transcode_kwargs) 106 | 107 | content_type = ContentType.objects.get_for_model(obj) 108 | job = EncodeJob() 109 | job.id = ret['persistentId'] 110 | job.content_type = content_type 111 | job.object_id = obj.pk 112 | job.message = message 113 | job.save() 114 | 115 | 116 | class AliyunTranscoder(Transcoder): 117 | 118 | def __init__( 119 | self, 120 | access_key_id=None, 121 | access_key_secret=None, 122 | pipeline_id=None, 123 | region=None, 124 | notify_url=None 125 | ): 126 | if not access_key_id: 127 | access_key_id = get_setting_or_raise('ALIYUN_TRANSCODE_ACCESS_KEY_ID') 128 | self.access_key_id = access_key_id 129 | 130 | if not access_key_secret: 131 | access_key_secret = get_setting_or_raise('ALIYUN_TRANSCODE_ACCESS_KEY_SECRET') 132 | self.access_key_secret = access_key_secret 133 | 134 | if not pipeline_id: 135 | pipeline_id = get_setting_or_raise('ALIYUN_TRANSCODE_PIPELINE_ID') 136 | self.pipeline_id = pipeline_id 137 | 138 | if not region: 139 | region = get_setting_or_raise('ALIYUN_TRANSCODE_REGION') 140 | self.region = region 141 | 142 | if not notify_url: 143 | notify_url = get_setting_or_raise('ALIYUN_TRANSCODE_NOTIFY_URL') 144 | self.notify_url = notify_url 145 | 146 | from aliyunsdkcore import client 147 | 148 | self.client = client.AcsClient(self.access_key_id, self.access_key_secret, self.region) 149 | 150 | def start_job(self, obj, transcode_kwargs, message=''): 151 | """ 152 | https://help.aliyun.com/document_detail/57347.html?spm=5176.doc56767.6.724.AJ8z3E 153 | """ 154 | 155 | import json 156 | from aliyunsdkmts.request.v20140618 import SubmitJobsRequest 157 | 158 | request = SubmitJobsRequest.SubmitJobsRequest() 159 | request.set_accept_format('json') 160 | request.set_Input(json.dumps(transcode_kwargs.get('input_file'))) 161 | request.set_OutputBucket(transcode_kwargs.get('bucket')) 162 | request.set_OutputLocation(transcode_kwargs.get('oss_location')) 163 | request.set_Outputs(json.dumps(transcode_kwargs.get('outputs'))) 164 | request.set_PipelineId(self.pipeline_id) 165 | response = json.loads(self.client.do_action_with_exception(request).decode('utf-8')) 166 | 167 | content_type = ContentType.objects.get_for_model(obj) 168 | job = EncodeJob() 169 | job.id = response['JobResultList']['JobResult'][0]['Job']['JobId'] 170 | job.content_type = content_type 171 | job.object_id = obj.pk 172 | job.message = message 173 | job.save() 174 | -------------------------------------------------------------------------------- /dj_elastictranscoder/urls.py: -------------------------------------------------------------------------------- 1 | import django 2 | 3 | 4 | if django.VERSION >= (1, 9): 5 | from django.conf.urls import url 6 | from dj_elastictranscoder import views 7 | 8 | urlpatterns = [ 9 | url(r'^endpoint/$', views.aws_endpoint), 10 | url(r'^aws_endpoint/$', views.aws_endpoint, name='aws_endpoint'), 11 | url(r'^qiniu_endpoint/$', views.qiniu_endpoint, name='qiniu_endpoint'), 12 | url(r'^aliyun_endpoint', views.aliyun_endpoint, name='aliyun_endpoint'), 13 | ] 14 | 15 | else: 16 | try: 17 | from django.conf.urls import url, patterns 18 | except ImportError: 19 | from django.conf.urls.defaults import url, patterns # Support for Django < 1.4 20 | 21 | urlpatterns = patterns( 22 | 'dj_elastictranscoder.views', 23 | url(r'^endpoint/$', 'aws_endpoint'), 24 | url(r'^aws_endpoint/$', 'aws_endpoint', name='aws_endpoint'), 25 | url(r'^qiniu_endpoint/$', 'qiniu_endpoint', name='qiniu_endpoint'), 26 | url(r'^aliyun_endpoint', 'aliyun_endpoint', name='aliyun_endpoint'), 27 | ) 28 | -------------------------------------------------------------------------------- /dj_elastictranscoder/utils.py: -------------------------------------------------------------------------------- 1 | from django.conf import settings 2 | from django.core.exceptions import ImproperlyConfigured 3 | 4 | 5 | def get_setting_or_raise(setting_name): 6 | try: 7 | value = getattr(settings, setting_name) 8 | except AttributeError: 9 | raise ImproperlyConfigured('Please provide {0}'.format(setting_name)) 10 | return value 11 | -------------------------------------------------------------------------------- /dj_elastictranscoder/views.py: -------------------------------------------------------------------------------- 1 | import json 2 | 3 | from django.core.mail import mail_admins 4 | from django.http import Http404, HttpResponse, HttpResponseBadRequest 5 | from django.views.decorators.csrf import csrf_exempt 6 | from django.views.decorators.http import require_http_methods 7 | 8 | from .models import EncodeJob 9 | from .signals import ( 10 | transcode_onprogress, 11 | transcode_onerror, 12 | transcode_oncomplete 13 | ) 14 | 15 | 16 | @csrf_exempt 17 | def aws_endpoint(request): 18 | """ 19 | Receive SNS notification 20 | """ 21 | 22 | try: 23 | webhook = request.read().decode('utf-8') 24 | data = json.loads(webhook) 25 | except ValueError: 26 | return HttpResponseBadRequest('Invalid JSON') 27 | 28 | # handle SNS subscription 29 | if data['Type'] == 'SubscriptionConfirmation': 30 | subscribe_url = data['SubscribeURL'] 31 | subscribe_body = """ 32 | Please visit this URL below to confirm your subscription with SNS 33 | 34 | %s """ % subscribe_url 35 | 36 | mail_admins('Please confirm SNS subscription', subscribe_body) 37 | return HttpResponse('OK') 38 | 39 | # handle job response 40 | message = json.loads(data['Message']) 41 | state = message['state'] 42 | 43 | job = EncodeJob.objects.get(pk=message['jobId']) 44 | 45 | # https://docs.aws.amazon.com/elastictranscoder/latest/developerguide/notifications.html 46 | if state == 'PROGRESSING': 47 | job.message = webhook 48 | job.state = 1 49 | job.save() 50 | transcode_onprogress.send(sender=None, job=job, job_response=data) 51 | elif state == 'COMPLETED': 52 | job.message = webhook 53 | job.state = 4 54 | job.save() 55 | transcode_oncomplete.send(sender=None, job=job, job_response=data) 56 | elif state == 'ERROR': 57 | job.message = webhook 58 | job.state = 2 59 | job.save() 60 | transcode_onerror.send(sender=None, job=job, job_response=data) 61 | else: 62 | raise RuntimeError('Invalid state') 63 | 64 | return HttpResponse('Done') 65 | 66 | 67 | @csrf_exempt 68 | @require_http_methods(['POST', ]) 69 | def qiniu_endpoint(request): 70 | """ 71 | Receive Qiniu notification 72 | """ 73 | 74 | try: 75 | webhook = request.read().decode('utf-8') 76 | data = json.loads(webhook) 77 | except ValueError: 78 | return HttpResponseBadRequest('Invalid JSON') 79 | 80 | code = data['code'] 81 | job_id = data['id'] 82 | 83 | job = EncodeJob.objects.get(pk=job_id) 84 | 85 | # https://developer.qiniu.com/dora/manual/1294/persistent-processing-status-query-prefop 86 | if code in (1, 2): # Progressing 87 | job.message = webhook 88 | job.state = 1 89 | job.save() 90 | transcode_onprogress.send(sender=None, job=job, job_response=data) 91 | elif code == 0: # Complete 92 | job.message = webhook 93 | job.state = 4 94 | job.save() 95 | transcode_oncomplete.send(sender=None, job=job, job_response=data) 96 | elif code == 3 or code == 4: # Error 97 | job.message = webhook 98 | job.state = 2 99 | job.save() 100 | transcode_onerror.send(sender=None, job=job, job_response=data) 101 | else: 102 | raise RuntimeError('Invalid code') 103 | 104 | return HttpResponse('Done') 105 | 106 | 107 | @csrf_exempt 108 | @require_http_methods(['POST', ]) 109 | def aliyun_endpoint(request): 110 | """ 111 | Receive Aliyun notification 112 | """ 113 | 114 | try: 115 | webhook = request.read().decode('utf-8') 116 | data = json.loads(webhook) 117 | except ValueError: 118 | return HttpResponseBadRequest('Invalid JSON') 119 | 120 | message = json.loads(data['Message']) 121 | if message['Type'] == 'Transcode': 122 | state = message['state'] 123 | job_id = message['jobId'] 124 | 125 | try: 126 | job = EncodeJob.objects.get(pk=job_id) 127 | except EncodeJob.DoesNotExist: 128 | raise Http404 129 | 130 | # https://help.aliyun.com/document_detail/57347.html?spm=5176.doc29208.6.724.4zQQQ4 131 | if state == 'Success': # Complate 132 | job.message = webhook 133 | job.state = 4 134 | job.save() 135 | transcode_oncomplete.send(sender=None, job=job, job_response=job_id) 136 | elif state == 'Fail': # Error 137 | job.message = webhook 138 | job.state = 2 139 | job.save() 140 | transcode_onerror.send(sender=None, job=job, job_response=data) 141 | else: 142 | raise RuntimeError('Invalid code') 143 | return HttpResponse('Done') 144 | -------------------------------------------------------------------------------- /docs/images/workflow.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StreetVoice/django-elastic-transcoder/b26a6c9e6a482777ebaddc03d8cae96e7012bd07/docs/images/workflow.jpg -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | from setuptools import setup, find_packages 5 | 6 | 7 | def get_version(): 8 | code = None 9 | path = os.path.join( 10 | os.path.dirname(os.path.abspath(__file__)), 11 | 'dj_elastictranscoder', 12 | '__init__.py', 13 | ) 14 | with open(path) as f: 15 | for line in f: 16 | if line.startswith('__version__'): 17 | code = line[len('__version__ = '):] 18 | break 19 | return eval(code) 20 | 21 | 22 | if sys.argv[-1] == 'wheel': 23 | os.system('pip wheel --wheel-dir=wheelhouse .') 24 | sys.exit() 25 | 26 | setup( 27 | name='django-elastic-transcoder', 28 | version=get_version(), 29 | description="Django with AWS elastic transcoder", 30 | long_description=open('README.rst').read(), 31 | author='tzangms', 32 | author_email='tzangms@streetvoice.com', 33 | url='http://github.com/StreetVoice/django-elastic-transcoder', 34 | license='MIT', 35 | packages=find_packages(exclude=('testsapp', )), 36 | include_package_data=True, 37 | zip_safe=False, 38 | install_requires=[ 39 | "boto3 >= 1.1", 40 | "django >= 1.3, < 2.0", 41 | "qiniu >= 7.0.8", 42 | ], 43 | classifiers=[ 44 | "Intended Audience :: Developers", 45 | "Operating System :: OS Independent", 46 | "Programming Language :: Python", 47 | "Programming Language :: Python :: 2", 48 | 'Programming Language :: Python :: 2.7', 49 | 'Programming Language :: Python :: 3', 50 | 'Programming Language :: Python :: 3.4', 51 | "Topic :: Software Development :: Libraries :: Python Modules", 52 | "Environment :: Web Environment", 53 | "Framework :: Django", 54 | ], 55 | keywords='django,aws,elastic,transcoder,qiniu,audio,aliyun', 56 | ) 57 | -------------------------------------------------------------------------------- /testsapp/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StreetVoice/django-elastic-transcoder/b26a6c9e6a482777ebaddc03d8cae96e7012bd07/testsapp/__init__.py -------------------------------------------------------------------------------- /testsapp/fixtures/oncomplete.json: -------------------------------------------------------------------------------- 1 | { 2 | "Type" : "Notification", 3 | "MessageId" : "", 4 | "TopicArn" : "", 5 | "Subject" : "Amazon Elastic Transcoder has finished transcoding job 1396802241671-jkmme8.", 6 | "Message" : "{\n \"state\" : \"COMPLETED\",\n \"version\" : \"2012-09-25\",\n \"jobId\" : \"1396802241671-jkmme8\",\n \"pipelineId\" : \"pipeline1\",\n \"input\" : {\n \"key\" : \"input.mp3\"\n },\n \"outputs\" : [ {\n \"id\" : \"1\",\n \"presetId\" : \"1351620000001-300040\",\n \"key\" : \"output.mp3\",\n \"status\" : \"Complete\",\n \"duration\" : 110\n } ]\n}", 7 | "Timestamp" : "2014-04-06T16:37:37.265Z", 8 | "SignatureVersion" : "1", 9 | "Signature" : "", 10 | "SigningCertURL" : "", 11 | "UnsubscribeURL" : "" 12 | } 13 | -------------------------------------------------------------------------------- /testsapp/fixtures/onerror.json: -------------------------------------------------------------------------------- 1 | { 2 | "Type" : "Notification", 3 | "MessageId" : "c030f5f7-da71-5280-a24b-be4c5e925dd2", 4 | "TopicArn" : "", 5 | "Subject" : "The Amazon Elastic Transcoder job 1396802206674-45b89f has failed.", 6 | "Message" : "{\n \"state\" : \"ERROR\",\n \"errorCode\" : 3002,\n \"messageDetails\" : \"3002 25319782-210b-45b2-a8a2-fb929b87d46b: The specified object could not be saved in the specified bucket because an object by that name already exists: bucket=bucket_name, key=output.mp3.\",\n \"version\" : \"2012-09-25\",\n \"jobId\" : \"1396802241671-jkmme8\",\n \"pipelineId\" : \"piepeline1\",\n \"input\" : {\n \"key\" : \"input.mp3\"\n },\n \"outputs\" : [ {\n \"id\" : \"1\",\n \"presetId\" : \"1351620000001-300040\",\n \"key\" : \"output.mp3\",\n \"status\" : \"Error\",\n \"statusDetail\" : \"3002 25319782-210b-45b2-a8a2-fb929b87d46b: The specified object could not be saved in the specified bucket because an object by that name already exists: bucket=bucket_name, key=hello.mp3.\",\n \"errorCode\" : 3002\n } ]\n}", 7 | "Timestamp" : "2014-04-06T16:36:51.568Z", 8 | "SignatureVersion" : "1", 9 | "Signature" : "", 10 | "SigningCertURL" : "", 11 | "UnsubscribeURL" : "" 12 | } 13 | -------------------------------------------------------------------------------- /testsapp/fixtures/onprogress.json: -------------------------------------------------------------------------------- 1 | { 2 | "Type" : "Notification", 3 | "MessageId" : "4912a9bc-c7a6-570f-8d07-e20742a2da5c", 4 | "TopicArn" : "", 5 | "Subject" : "Amazon Elastic Transcoder has scheduled job 1396802241671-jkmme8 for transcoding.", 6 | "Message" : "{\n \"state\" : \"PROGRESSING\",\n \"version\" : \"2012-09-25\",\n \"jobId\" : \"1396802241671-jkmme8\",\n \"pipelineId\" : \"pipeline1\",\n \"input\" : {\n \"key\" : \"input.mp3\"\n },\n \"outputs\" : [ {\n \"id\" : \"1\",\n \"presetId\" : \"1351620000001-300040\",\n \"key\" : \"output.mp3\",\n \"status\" : \"Progressing\"\n } ]\n}", 7 | "Timestamp" : "2014-04-06T16:37:24.159Z", 8 | "SignatureVersion" : "1", 9 | "Signature" : "", 10 | "SigningCertURL" : "", 11 | "UnsubscribeURL" : "" 12 | } 13 | -------------------------------------------------------------------------------- /testsapp/fixtures/submit.json: -------------------------------------------------------------------------------- 1 | {"Job": {"Status": "Submitted", "Playlists": [], "Outputs": [{"Status": "Submitted", "Rotate": null, "StatusDetail": null, "PresetId": "1351620000001-300040", "Composition": null, "Width": null, "Watermarks": [], "AlbumArt": null, "Key": "hello.mp3", "Duration": null, "SegmentDuration": null, "ThumbnailPattern": null, "Height": null, "Id": "1"}], "PipelineId": "1396795968443-34a7hg", "OutputKeyPrefix": null, "Output": {"Status": "Submitted", "Rotate": null, "StatusDetail": null, "PresetId": "1351620000001-300040", "Composition": null, "Width": null, "Watermarks": [], "AlbumArt": null, "Key": "output.mp3", "Duration": null, "SegmentDuration": null, "ThumbnailPattern": null, "Height": null, "Id": "1"}, "Input": {"Container": null, "FrameRate": null, "Key": "input.mp3", "AspectRatio": null, "Resolution": null, "Interlaced": null}, "Id": "1396802241671-jkmme8", "Arn": ""}} 2 | -------------------------------------------------------------------------------- /testsapp/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | class Item(models.Model): 4 | name = models.CharField(max_length=100) 5 | -------------------------------------------------------------------------------- /testsapp/requirements.txt: -------------------------------------------------------------------------------- 1 | pytest 2 | pytest-django 3 | pytest-cov 4 | boto 5 | south -------------------------------------------------------------------------------- /testsapp/test_job.py: -------------------------------------------------------------------------------- 1 | import os.path 2 | import json 3 | 4 | from django.test import TestCase 5 | from django.dispatch import receiver 6 | from django.db import models 7 | from django.contrib.contenttypes.models import ContentType 8 | 9 | from dj_elastictranscoder.models import EncodeJob 10 | from dj_elastictranscoder.signals import ( 11 | transcode_onprogress, 12 | transcode_onerror, 13 | transcode_oncomplete 14 | ) 15 | 16 | from .models import Item 17 | 18 | 19 | 20 | PROJECT_ROOT = os.path.dirname(os.path.realpath(__file__)) 21 | FIXTURE_DIRS = os.path.join(PROJECT_ROOT, 'fixtures') 22 | 23 | 24 | # ====================== 25 | # define signal receiver 26 | # ====================== 27 | 28 | @receiver(transcode_onprogress) 29 | def encode_onprogress(sender, message, **kwargs): 30 | job = EncodeJob.objects.get(pk=message['jobId']) 31 | job.message = 'Progress' 32 | job.state = 1 33 | job.save() 34 | 35 | 36 | @receiver(transcode_onerror) 37 | def encode_onerror(sender, message, **kwargs): 38 | job = EncodeJob.objects.get(pk=message['jobId']) 39 | job.message = message['messageDetails'] 40 | job.state = 2 41 | job.save() 42 | 43 | 44 | @receiver(transcode_oncomplete) 45 | def job_record(sender, message, **kwargs): 46 | job = EncodeJob.objects.get(pk=message['jobId']) 47 | job.message = 'Success' 48 | job.state = 4 49 | job.save() 50 | 51 | 52 | class SNSNotificationTest(TestCase): 53 | urls = 'dj_elastictranscoder.urls' 54 | 55 | def setUp(self): 56 | item = Item.objects.create(name='Hello') 57 | content_type = ContentType.objects.get_for_model(Item) 58 | self.job_id = '1396802241671-jkmme8' 59 | 60 | self.job = EncodeJob.objects.create(id=self.job_id, content_type=content_type, object_id=item.id) 61 | 62 | def test_initial(self): 63 | job = EncodeJob.objects.get(id=self.job_id) 64 | self.assertEqual(job.state, 0) 65 | 66 | 67 | def test_onprogress(self): 68 | with open(os.path.join(FIXTURE_DIRS, 'onprogress.json')) as f: 69 | content = f.read() 70 | 71 | resp = self.client.post('/endpoint/', content, content_type="application/json") 72 | self.assertEqual(resp.status_code, 200) 73 | self.assertContains(resp, 'Done') 74 | 75 | job = EncodeJob.objects.get(id=self.job_id) 76 | self.assertEqual(job.state, 1) 77 | 78 | def test_onerror(self): 79 | with open(os.path.join(FIXTURE_DIRS, 'onerror.json')) as f: 80 | content = f.read() 81 | 82 | resp = self.client.post('/endpoint/', content, content_type="application/json") 83 | self.assertEqual(resp.status_code, 200) 84 | self.assertContains(resp, 'Done') 85 | 86 | job = EncodeJob.objects.get(id=self.job_id) 87 | self.assertEqual(job.state, 2) 88 | 89 | 90 | def test_oncomplete(self): 91 | with open(os.path.join(FIXTURE_DIRS, 'oncomplete.json')) as f: 92 | content = f.read() 93 | 94 | resp = self.client.post('/endpoint/', content, content_type="application/json") 95 | self.assertEqual(resp.status_code, 200) 96 | self.assertContains(resp, 'Done') 97 | 98 | job = EncodeJob.objects.get(id=self.job_id) 99 | self.assertEqual(job.state, 4) 100 | 101 | 102 | class SignalTest(TestCase): 103 | 104 | def test_transcode_onprogress(self): 105 | """ 106 | test for transcode_onprogress signal 107 | """ 108 | 109 | # assume an encode job was submitted 110 | item = Item.objects.create(name='Hello') 111 | 112 | ctype = ContentType.objects.get_for_model(item) 113 | 114 | job = EncodeJob() 115 | job.id = '1396802241671-jkmme8' 116 | job.content_type = ctype 117 | job.object_id = item.id 118 | job.save() 119 | 120 | # 121 | with open(os.path.join(FIXTURE_DIRS, 'onprogress.json')) as f: 122 | resp = json.loads(f.read()) 123 | message = json.loads(resp['Message']) 124 | 125 | # send signal 126 | transcode_onprogress.send(sender=None, message=message) 127 | 128 | # 129 | job = EncodeJob.objects.get(pk=message['jobId']) 130 | 131 | # 132 | self.assertEqual(1, EncodeJob.objects.count()) 133 | self.assertEqual('1396802241671-jkmme8', job.id) 134 | self.assertEqual('Progress', job.message) 135 | self.assertEqual(1, job.state) 136 | 137 | 138 | def test_transcode_onerror(self): 139 | """ 140 | test for transcode_onerror signal 141 | """ 142 | 143 | # assume an encode job was submitted 144 | item = Item.objects.create(name='Hello') 145 | 146 | ctype = ContentType.objects.get_for_model(item) 147 | 148 | job = EncodeJob() 149 | job.id = '1396802241671-jkmme8' 150 | job.content_type = ctype 151 | job.object_id = item.id 152 | job.save() 153 | 154 | # 155 | with open(os.path.join(FIXTURE_DIRS, 'onerror.json')) as f: 156 | resp = json.loads(f.read()) 157 | message = json.loads(resp['Message']) 158 | 159 | # send signal 160 | transcode_onerror.send(sender=None, message=message) 161 | 162 | # 163 | job = EncodeJob.objects.get(pk=message['jobId']) 164 | error_message = "3002 25319782-210b-45b2-a8a2-fb929b87d46b: The specified object could not be saved in the specified bucket because an object by that name already exists: bucket=bucket_name, key=output.mp3." 165 | 166 | # 167 | self.assertEqual(1, EncodeJob.objects.count()) 168 | self.assertEqual('1396802241671-jkmme8', job.id) 169 | self.assertEqual(error_message, job.message) 170 | self.assertEqual(2, job.state) 171 | 172 | def test_transcode_oncomplete(self): 173 | """ 174 | test for transcode_oncomplete signal 175 | """ 176 | 177 | # assume an encode job was submitted 178 | item = Item.objects.create(name='Hello') 179 | 180 | ctype = ContentType.objects.get_for_model(item) 181 | 182 | job = EncodeJob() 183 | job.id = '1396802241671-jkmme8' 184 | job.content_type = ctype 185 | job.object_id = item.id 186 | job.save() 187 | 188 | # 189 | with open(os.path.join(FIXTURE_DIRS, 'oncomplete.json')) as f: 190 | resp = json.loads(f.read()) 191 | message = json.loads(resp['Message']) 192 | 193 | # send signal 194 | transcode_oncomplete.send(sender=None, message=message) 195 | 196 | # 197 | job = EncodeJob.objects.get(pk=message['jobId']) 198 | 199 | # 200 | self.assertEqual(1, EncodeJob.objects.count()) 201 | self.assertEqual('1396802241671-jkmme8', job.id) 202 | self.assertEqual('Success', job.message) 203 | self.assertEqual(4, job.state) 204 | 205 | 206 | """ 207 | class TranscoderTest(TestCase): 208 | def test_transcoder(self): 209 | from .transcoder import Transcoder 210 | 211 | input = { 212 | 'Key': 'music/00/09/00094930/6c55503185ac4a42b68d01d8277cd84e.mp3', 213 | } 214 | 215 | outputs = [{ 216 | 'Key': 'hello.mp3', 217 | 'PresetId': '1351620000001-300040' # for example: 128k mp3 audio preset 218 | }] 219 | 220 | pipeline_id = '' 221 | 222 | transcoder = Transcoder(pipeline_id, 'ap-southeast-1') 223 | transcoder.encode(input, outputs) 224 | """ 225 | -------------------------------------------------------------------------------- /testsapp/tests_settings.py: -------------------------------------------------------------------------------- 1 | DATABASES = { 2 | 'default': { 3 | 'ENGINE': 'django.db.backends.sqlite3', 4 | 'NAME': ':memory:' 5 | } 6 | } 7 | INSTALLED_APPS = [ 8 | 'django.contrib.admin', 9 | 'django.contrib.auth', 10 | 'django.contrib.contenttypes', 11 | 'django.contrib.sessions', 12 | 'django.contrib.sites', 13 | 'dj_elastictranscoder', 14 | 'testsapp', 15 | ] 16 | SITE_ID = 1 17 | DEBUG = False 18 | ROOT_URLCONF = '' 19 | SECRET_KEY='test' -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | # for py 3.x we are using only django 1.6.x as 1.5.x had only "experimental py3 support" 3 | envlist = 4 | py{27,py}-dj{14,15,16,17,18}, 5 | py{33,34}-dj{15,16,17,18}, 6 | py27-cov 7 | skipsdist = True 8 | usedevelop = True 9 | 10 | [testenv] 11 | commands = {posargs:py.test --create-db -vv} 12 | basepython = 13 | py27: python2.7 14 | py33: python3.3 15 | py34: python3.4 16 | pypy: pypy 17 | deps = 18 | -rtestsapp/requirements.txt 19 | dj14: django>=1.4,<1.4.999 20 | dj15: django>=1.5,<1.5.999 21 | dj16: django>=1.6,<1.6.999 22 | dj17: django>=1.7,<1.7.999 23 | dj18: django>=1.8,<1.8.999 24 | dj19: https://github.com/django/django/archive/master.tar.gz#egg=django 25 | setenv = 26 | DJANGO_SETTINGS_MODULE = testsapp.tests_settings 27 | PYTHONPATH = {toxinidir}/testsapp:{toxinidir} 28 | 29 | [testenv:py27-cov] 30 | commands = py.test --cov=dj_elastictranscoder 31 | deps = 32 | -rtestsapp/requirements.txt 33 | django>=1.8,<1.8.999 34 | --------------------------------------------------------------------------------