├── lib └── wse-plugin-s3upload.jar ├── README.md ├── version.txt ├── LICENSE.txt └── src └── com └── wowza └── wms └── plugin └── s3upload └── ModuleS3Upload.java /lib/wse-plugin-s3upload.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/WowzaMediaSystems/wse-plugin-s3upload/HEAD/lib/wse-plugin-s3upload.jar -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # S3Upload 2 | The **ModuleS3Upload** module for [Wowza Streaming Engine™ media server software](https://www.wowza.com/products/streaming-engine) automatically uploads finished recordings to an Amazon S3 bucket. It uses the Amazon Web Services (AWS) SDK for Java to upload the recorded files. 3 | 4 | This repo includes a [compiled version](/lib/wse-plugin-s3upload.jar). 5 | 6 | ## Prerequisites 7 | Wowza Streaming Engine 4.7.2.02 or later is recommended. For earlier versions see note below regarding AWS SDK version. 8 | 9 | [AWS SDK for Java](https://aws.amazon.com/sdk-for-java/) 10 | 11 | **Note:** For earlier versions of Wowza Streaming Engine™, AWS SDK version 1.10.77 or earlier is required. As a minimum, the following packages are required. 12 | 13 | -[AWS Java SDK For Amazon S3](http://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3/1.10.77) 14 | 15 | -[AWS SDK For Java Core](http://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-core/1.10.77) 16 | 17 | -[AWS Java SDK For AWS KMS](http://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-kms/1.10.77) (it's not clear if this package is actually required. It's only referenced from AmazonS3EncryptionClient which isn't used in the S3 uploader) 18 | 19 | The version of [Apache httpclient](http://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient) that ships with Wowza Streaming Engine prior to 4.7.2.02 isn't compatible with the later versions of the AWS SDK 20 | 21 | ## Usage 22 | When a recording is finished, a temporary file named **[recording-name].upload** is created to track the recording and sort any data that may be needed to resume the file upload later if it's interrupted. AWS TransferManager uploads the recorded file, splitting it into a multipart upload if required. After the recorded file is uploaded, the temporary **[recording-name].upload** file is deleted. 23 | 24 | When the Wowza Streaming Engine application starts or restarts, the module checks to see if any interrupted uploads must be completed. Interrupted single part uploads are restarted from the beginning while interrupted multipart uploads are resumed from the last complete part. If the module is set to not resume uploads after interruptions (**s3UploadResumeUploads** = **false**), incomplete multipart uploads are deleted from the S3 bucket. 25 | 26 | ## More resources 27 | To use the compiled version of this module, see [How to upload recorded media to an Amazon S3 bucket (S3Upload)](https://www.wowza.com/docs/how-to-upload-recorded-media-to-an-amazon-s3-bucket-modules3upload). 28 | 29 | [Wowza Streaming Engine Server-Side API Reference](https://www.wowza.com/resources/serverapi/) 30 | 31 | [How to extend Wowza Streaming Engine using the Wowza IDE](https://www.wowza.com/docs/how-to-extend-wowza-streaming-engine-using-the-wowza-ide) 32 | 33 | Wowza Media Systems™ provides developers with a platform to create streaming applications and solutions. See [Wowza Developer Tools](https://www.wowza.com/developer) to learn more about our APIs and SDK. 34 | 35 | ## Contact 36 | [Wowza Media Systems, LLC](https://www.wowza.com/contact) 37 | 38 | ## License 39 | This code is distributed under the [Wowza Public License](/LICENSE.txt). 40 | -------------------------------------------------------------------------------- /version.txt: -------------------------------------------------------------------------------- 1 | Build 55 2 | * Added support for path variables when defining s3UploadFilePrefix and streamRecorderOutputPath properties 3 | 4 | Build 54 5 | * Added `s3UploadStripRecorderVersioning` property to remove recorder versioning from the file name before versioning on S3 so that the files aren't versioned twice 6 | * Improved version checks to make sure current uploads and unfinished multi part uploads are included. 7 | 8 | Build 53 9 | * Touch appInstance more often to keep it from shutting down while preparing to start an upload 10 | 11 | Build 52 12 | * Fixed problem with trying to extract the region name from the endpoint property 13 | * Updated logging for region name matching so that it doesn't trigger the wrong error message 14 | 15 | Build 51 16 | * Fixed null region name when `useDefaultRegion` was set 17 | * Add stack trace logging to the IllegalStateException that we log when the AWS SDK version is too old 18 | 19 | Build 50 20 | * Added support for all AWS S3 authentication methods. If credentials aren't set in properties then the `Default Credentials Provider Chain` will be used. See https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default for details. 21 | * Added `s3UploadAwsProfile` property to define a aws profile to use. default: null 22 | * Added `s3UploadAwsProfilePath` property to define a different location for the aws profile settings default: null 23 | * Added `s3UploadRegion` property to define the region that the s3 bucket is located in. default: null 24 | * Added `s3UploadUseDefaultRegion` property to tell the uploader to use the same region as the EC2 instance or the default region set by the AWS SDK if `s3UploadRegion` isn't set. default: true 25 | * Added `s3UploadAllowBucketRegionOverride` property to turn on global bucket access so that a region mismatch won't cause uploads to fail. default: true 26 | * Added support for S3 Canned ACLs. `s3UploadCannedAcl` property is used to specify the the canned ACL from the list here, https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. default: null 27 | * Modified workflow so that s3 failures wouldn't prevent recordings from being marked for upload, allowing them to upload automatically once service is resumed. 28 | 29 | Build 49 30 | * Changed storageDir field to reference `streamRecorderOutputPath` property, if set 31 | 32 | Build 48 33 | * Fixed problem with deleting local files if a file prefix was set 34 | 35 | Build 47 36 | * Added `s3UploadDelay` property to delay the start of an upload to allow for cases where a stream is re-published. Default value is 0 37 | * Added `s3UploadFilePrefix` property to specify a path to upload the files to. Default not set 38 | * Added `s3UploadVersionFile` property to control file versioning. When enabled, we will check to see if the file exists in the s3 bucket and add an incrementing version number to the new uploaded file as required. Default is false (don't version) 39 | * Added `s3UploadDebugLog` property to enable extra debug logging 40 | 41 | Build 46 42 | * Added method to restart failed uploads after a timeout without having to wait for an appInstance restart 43 | * Added Boolean property `s3UploadRestartFailedUploads`. Default true 44 | * Added Integer Property `s3UploadRestartFailedUploadTimeout`. Default 60000 45 | * Streamlined upload initialisation so all uploads use a single method 46 | * Fixed bug in file search so that subfolders are indexed correctly 47 | 48 | Build 45 49 | * Fixed problem with extracting mediaName where leading separator chars were not being removed on Windows platforms 50 | 51 | Build 44 52 | * Fixed module initialisation error caused by missing property values 53 | * Changed method used to shutdown the transfer manager so that the S3Client isn't closed immediately which was throwing a socket exception if there are active transfers 54 | * Added code to touch the appInstance while an upload is in progress so it doesn't automatically time out until the uploads are complete 55 | 56 | Build 43 57 | * Fixed bug that was introduced in previous commit which prevented uploads from resuming after an application restart 58 | 59 | Build 42 60 | * S3 ACL File permissions (#3) 61 | 62 | Build 37 63 | * Added a property (s3UploadEndpoint) for setting the S3 endpoint that the bucket is in. Required for locations that only support V4 authentication and optional for other locations. Not required for the default location 64 | * Changed bucket detection so we can make sure the user has permission to write to it 65 | * Added better logging in onAppStart to detect problems with connecting to S3 66 | * Updated README.md required packages list 67 | 68 | Build 34 69 | * Add try/catch blocks to capture and log exceptions that are caused when an incorrect AWS package is used 70 | * Updated README.md to add aws-java-sdk-kms to the list of AWS packages that are required 71 | 72 | Build 33 73 | * Fixed typo in property name, 's3UploadDeleteOriginalFiles' 74 | 75 | Build 32 76 | * Merge pull request #1 from gtd/patch-1 77 | 78 | Build 28 79 | * Updated README to note specific version of AWS SDK files that are required 80 | 81 | Build 1 82 | * Initial commit 83 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Wowza Public License 2 | Version 1.0 (February 1, 2016) 3 | 4 | Please note: this License ("License") contains unique terms that differ from many common "open source" and similar license terms 5 | published by the Free Software Foundation and the Mozilla Organization.THE ACCOMPANYING CODE IS PROVIDED UNDER THE TERMS OF THIS 6 | WOWZA PUBLIC LICENSE AGREEMENT. ANY ACCESS TO OR USE, REPRODUCTION, MODIFICATION, DISTRIBUTION OR OTHER EXPLOITATION OF THE 7 | ACCOMPANYING CODE CONSTITUTES YOUR ACCEPTANCE OF THIS LICENSE AGREEMENT. NOTHING OTHER THAN THIS LICENSE GRANTS YOU PERMISSION TO 8 | ACCESS, USE REPRODUCE, MODIFY, DISTRIBUTE OR OTHERWISE EXPLOIT THE SOURCE CODE OR ITS DERIVATIVE WORKS. IF YOU DO NOT ACCEPT ALL 9 | OF THESE TERMS AND CONDITIONS, DO NOT ACCESS, USE, REPRODUCE, MODIFY, DISTRIBUTE OR OTHERWISE EXPLOIT THE ACCOMPANYING CODE. 10 | 11 | 1. Wowza License Grant to You. 12 | 13 | a. Wowza grants You a revocable (only by Wowza and only as provided herein) worldwide, royalty-free, non-exclusive license: 14 | 15 | i. under intellectual property rights (other than patent or trademark) Licensable by Wowza, to reproduce, modify, run, and 16 | distribute the Covered Code; and 17 | 18 | ii. under Patent Claims infringed by the making, using, or importing the Original Code, to make, have made, use, import, 19 | and practice the Original Code (or portions thereof). 20 | 21 | b. For the avoidance of doubt and subject to Section 1.a.ii, above, Wowza grants You no license under Patent Claims infringed 22 | by the making, using, practicing, or importing of Modifications. 23 | 24 | c. This License applies to code to which Wowza has attached the notice in Exhibit A, and to related Covered Code. 25 | 26 | d. Notwithstanding Section 1.a.ii above, no patent license is granted: 1) for code that You delete from the Original Code; 27 | 2) for Code that you separate from the Original Code; or 3) for any Modification of the Original Code. 28 | 29 | e. For the avoidance of doubt, the rights granted in this Section 1 grant you permission to make, run, market, and receive 30 | compensation for Covered Code, whether standing alone, or as a part of a software or service offering by You, provided you comply 31 | with this License and maintain this License's full effect. 32 | 33 | 2. Contributor Grants. 34 | 35 | a. To the extent You are a Contributor, You hereby grant all Recipients a world-wide, royalty-free, non-exclusive license: 36 | 37 | i. under intellectual property rights (other than patent or trademark) Licensable by Contributor, to reproduce, modify, 38 | run, and distribute the Contributor Version; and 39 | 40 | ii. under Patent Claims infringed by the making, using, or importing of the Contributor Version, to make, have made, use, 41 | and import: 1) Modifications made by that Contributor (or portions thereof); and 2) the combination of Modifications made by that 42 | Contributor with its Contributor Version (or portions of such combination). 43 | 44 | 3. Title; Reservation of Rights. As between You and Wowza, Wowza retains all title, ownership rights, and intellectual property 45 | rights in the Original Code. Wowza and the other Contributors, if applicable, reserve all rights not expressly granted to You 46 | hereunder, and no other licenses are granted or implied. Wowza may license the Original Code, including Modifications 47 | incorporated therein, on different terms from those contained in this License. 48 | 49 | 4. Distribution Obligations. 50 | 51 | a. You will ensure that any Modifications that You create or to which You contribute are governed by the terms of this 52 | License. If You Convey such Modifications or resulting Covered Code (except as required in Section 5), You will do so solely 53 | under the terms of this License. 54 | 55 | b. If You Convey the Source Code version of Covered Code you will do so solely under the terms of this License or any future 56 | version of this License (as described below), and You must include a copy of this License with each and every copy of such Source 57 | Code you Convey. 58 | 59 | c. You will not offer or impose any terms on any Source Code version that alters or restricts the applicable version of this 60 | License or the recipients' rights hereunder. For the avoidance of doubt, the obligation to provide licenses to Wowza (as provided 61 | in Section 5, above) follows the Original Code and the Covered Code, and applies to each Contributor and recipient of Covered 62 | Code. 63 | 64 | 5. Obligations to Provide Code to Wowza; Additional License Grants to Wowza. 65 | 66 | a. You will provide Wowza with a complete copy of any Covered Code and related documentation for Modifications created or 67 | contributed by You, and will do so promptly following either a request by Wowza or the first time You Convey such Modifications. 68 | 69 | b. If You are a Contributor and do not Convey the Modifications in Source Code form, You hereby grant Wowza an unrestricted, 70 | nonexclusive, worldwide, perpetual, irrevocable, royalty-free right, to use, reproduce, modify, display, perform, sublicense and 71 | distribute and otherwise exploit all Covered Code and and related documentation for Modifications, and to grant third parties the 72 | right to do so, including without limitation as a part of or with the Covered Code under all intellectual property rights 73 | (including any patent rights); and Wowza has the right to license or to otherwise transfer to third parties its rights to such 74 | Modifications without notice or any obligation to You, including without limitation the obligation to account for any profits 75 | obtained by Wowza. 76 | 77 | 6. Notices; Descriptions of Modifications Required. 78 | 79 | a. You must cause all Covered Code to which You contribute to contain a file documenting the changes You made to create that 80 | Covered Code and the date of any change. You must include a prominent statement that the Modification is derived, directly or 81 | indirectly, from Original Code provided by Wowza Media Systems, LLC, and including the name of Wowza in: (a) the Source Code; and 82 | (b) in any notice in an Executable version or related documentation in which You describe the origin or ownership of the Covered 83 | Code. 84 | 85 | b. You must duplicate the notice provided in Exhibit A in each file of the Source Code. If it is not possible to put such 86 | notice in a particular Source Code file due to its structure, then You must include such notice in a location where a user would 87 | be likely to look for such a notice. If You created one or more Modification(s) You may add your name as a Contributor to the 88 | notice described in Exhibit A. You must also duplicate this License in any documentation for the Source Code where You describe 89 | recipients' rights or ownership rights relating to Covered Code. 90 | 91 | c. If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Code 92 | due to applicable statute, judicial order, or regulation, then this License does not terminate but: (a) You do not have any 93 | rights from Wowza pursuant to Section 1.a; (b) You do not have any rights from other Contributors pursuant to Section 2.a; and 94 | (c) You must not access, run, use, have made, reproduce, modify, distribute, or exploit the Covered Code. 95 | 96 | 7. License Versions. 97 | 98 | a. Wowza may publish a revised or new version of the License from time to time in its sole discretion without prior notice. 99 | Each version will be given a distinguishing version number. 100 | 101 | b. Subject to Wowza's right to terminate or revoke this License, once Covered Code has been published under a particular 102 | version of the License, You may always continue to use it under the terms of that version. 103 | 104 | c. No one other than Wowza or its successors or assigns has the right to modify the terms applicable to Covered Code created 105 | under this License. 106 | 107 | 8. Disclaimer of Warranty; Limitation of Liability. 108 | 109 | a. THE COVERED CODE IS PROVIDED UNDER THIS LICENSE ON AN "AS IS" BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR 110 | IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, 111 | LACK OF VIRUSES, ACCURACY OR COMPLETENESS OF RESPONSES, WARRANTIES IMPLIED FROM A COURSE OF DEALING OR COURSE OF PERFORMANCE, OR 112 | WARRANTIES THAT THE OPERATION OF THE COVERED CODE WILL BE UNINTERRUPTED, ERROR FREE, OR PRODUCE PARTICULAR RESULTS. THE ENTIRE 113 | RISK AS TO THE QUALITY AND PERFORMANCE OF THE COVERED CODE IS WITH YOU. SHOULD ANY COVERED CODE PROVE DEFECTIVE IN ANY RESPECT, 114 | YOU (NOT WOWZA OR ANY OTHER CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION. THIS DISCLAIMER OF 115 | WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE. NO USE OF ANY COVERED CODE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS 116 | DISCLAIMER. 117 | 118 | b. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL WOWZA, ANY OTHER CONTRIBUTOR, OR ANY PERSON WHO COVEYS 119 | COVERED CODE, OR ANY SUPPLIER OF ANY OF SUCH PARTIES, BE LIABLE FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES 120 | ARISING OUT OF THE USE OR INABILITY TO USE THE COVERED CODE, INCLUDING BUT NOT LIMITED TO LOST PROFITS OR BUSINESS OPPORTUNITIES, 121 | LOSS OF USE, BUSINESS INTERRUPTION, OR LOSS OF DATA, UNDER ANY THEORY OF LIABILITY, WHETHER BASED IN CONTRACT, TORT, NEGLIGENCE, 122 | PRODUCT LIABILITY, OR OTHERWISE EVEN IF SUCH PARTY HAS BEEN INFORMED OF THE POSSIBILITY OF SUCH DAMAGES. 123 | 124 | c. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, WOWZA'S TOTAL AGGREGATE LIABILITY TO ALL LICENSEES AND CONTRIBUTORS 125 | HEREUNDER WILL NOT EXCEED THE GREATER OF (a) LICENSE FEES RECEIVED BY WOWZA FOR COVERED CODE; OR (b) FIVE HUNDRED U.S. DOLLARS 126 | (US $500.00). 127 | 128 | d. YOU UNDERSTAND AND AGREE THAT THE FOREGOING DISCLAIMER OF WARRANTIES AND LIMITATIONS OF LIABILITY CONSTITUTES AN ESSENTIAL 129 | PART OF THIS AGREEMENT AND THAT WOWZA, ANY OTHER CONTRIBUTOR, OR ANY PERSON WHO COVEYS COVERED CODE WOULD NOT COVEY THE COVERED 130 | CODE ABSENT THESE PROTECTIONS. 131 | 132 | 9. Termination. 133 | 134 | a. This License and the rights granted to You hereunder will terminate automatically if You fail to comply with the terms 135 | herein. Notwithstanding the foregoing, if you promptly cease and cure all violations of this License and notify Wowza that you 136 | have done so, then this License and the rights granted to You hereunder are reinstated. Sections 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 137 | and 12 survive termination of this License. 138 | 139 | b. If You initiate litigation and assert a patent or other intellectual property infringement claim against Wowza or another 140 | Recipient or Contributor related to the Covered Code or any Executable, this License and the rights granted to You hereunder will 141 | terminate automatically. 142 | 143 | 10. Your Responsibility for Claims. Notwithstanding any limitations of liability, to the maximum extent permitted by law, You 144 | agree to indemnify, defend, and hold harmless Wowza, and pay the amount of any final judgment or settlement to which we consent, 145 | against any claim or demand alleging that: (a) your Modification(s) or Covered Code resulting from Your Modification(s), or 146 | Wowza's accessing, use, reproduction, modification, distribution, or other exploitation of the same, infringes the intellectual 147 | property rights of any third party; or (2) a third party has been harmed or injured by Your actions or omissions related to, or 148 | breach of, this License. 149 | 150 | 11. Miscellaneous. This License represents the complete agreement concerning the subject matter hereof. If any provision of this 151 | License is held to be unenforceable, such provision will be reformed only to the extent necessary to make it enforceable. This 152 | License will be governed by New York law provisions (except to the extent applicable law, if any, provides otherwise), excluding 153 | its conflict-of-law provisions. With respect to disputes in which at least one party is a citizen of, or an entity chartered or 154 | registered to do business in the United States of America, any litigation relating to this License will be subject to the 155 | jurisdiction of the Federal Courts of Colorado, with venue lying in Denver County, Colorado. The application of the United 156 | Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any law or regulation which provides 157 | that the language of a contract will be construed against the drafter will not apply to this License. You agree that, if you 158 | export or re-export the Covered Code or any modifications to it, You are responsible for compliance with the United States Export 159 | Administration Regulations and hereby agree to comply with the same, and to indemnify Wowza and all other Contributors for any 160 | liability incurred as a result. 161 | 162 | 12. Definitions. 163 | 164 | a. "Contributor" means each entity that creates or contributes to the creation of Modifications. 165 | 166 | b. "Convey" means, with respect to a work, to copy, distribute (with or without modification), make available to the public, 167 | or otherwise exploit it (other than executing it on a computer that, without permission, would make you directly or indirectly 168 | liable for infringement under applicable copyright law. 169 | 170 | c. "Contributor Version" means the combination of the Original Code, prior Modifications used by a Contributor, and the 171 | Modifications made by that particular Contributor. 172 | 173 | d. "Covered Code" means the Original Code or Modifications or the combination of the Original Code and Modifications, in each 174 | case including portions thereof. 175 | 176 | e. "Executable" means Covered Code in any form other than Source Code. 177 | 178 | f. "License" means this document. 179 | 180 | g. "Licensable" means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or 181 | subsequently acquired, any and all of the rights conveyed herein. 182 | 183 | h. "Modifications" means any addition to or deletion from the substance or structure of either the Original Code or any 184 | previous Modifications. When Covered Code is released as a series of files, a Modification is: (1) any addition to or deletion 185 | from the contents of a file containing Original Code or previous Modifications; and (2) any new file that contains any part of 186 | the Original Code or previous Modifications. 187 | 188 | i. "Original Code" means Source Code of computer software code which is described in the Source Code notice required by 189 | Exhibit A as Original Code. 190 | 191 | j. "Patent Claims" means any patent claim(s), now owned or hereafter acquired, including without limitation, method, process, 192 | and apparatus claims, in any patent Licensable by grantor. 193 | 194 | k. "Recipient" means any individual or entity who directly or indirectly receives the Covered Code under this Agreement, 195 | including all Contributors. 196 | 197 | l. "Source Code" means the preferred form of the Covered Code for making modifications to it, including all modules it 198 | contains, plus any associated interface definition files, scripts used to control compilation and installation of an Executable, 199 | or source code differential comparisons against the Original Code. The Source Code can be in a compressed or archival form, 200 | provided the appropriate decompression or de-archiving software is widely available for no charge. 201 | 202 | m. "Wowza" means Wowza Media Systems, LLC, 523 Park Point Drive, Suite 300, Golden, Colorado USA, ATTN: Legal Department. 203 | 204 | n. "You" (or "Your") means an individual or a legal entity exercising rights under, and complying with all of the terms of, 205 | this License or a future version of this License. For legal entities, "You" includes any entity which controls, is controlled by, 206 | or is under common control with You. 207 | 208 | Exhibit A 209 | License Notice 210 | 211 | Copyright (c) 2007 - 2016, Wowza Media Systems, LLC. All rights reserved. 212 | 213 | The accompanying code is exclusively provided under the terms of the Wowza Public License, which is available at 214 | www.wowza.com/legal. 215 | 216 | No Wowza trademark or name may be used to imply endorsement by Wowza of products or services which make use of or are derived 217 | from this software without the prior written permission of Wowza Media Systems. THE ACCOMPANYING CODE IS PROVIDED BY 218 | WOWZA MEDIA SYSTEMS, LLC AND CONTRIBUTORS ON AN "AS IS" BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, 219 | INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, LACK OF 220 | VIRUSES, ACCURACY OR COMPLETENESS OF RESPONSES, WARRANTIES IMPLIED FROM A COURSE OF DEALING, OR COURSE OF PERFORMANCE, OR 221 | WARRANTIES THAT THE OPERATION OF THE ACCOMPANYING CODE WILL BE UNINTERRUPTED, ERROR FREE, OR PRODUCE PARTICULAR RESULTS. THE 222 | ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE ACCOMPANYING CODE IS WITH YOU. SHOULD ANY OF THE ACCOMPANYING CODE PROVE 223 | DEFECTIVE IN ANY RESPECT, YOU (NOT WOWZA OR ANY OTHER CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR 224 | CORRECTION. THIS DISCLAIMER OF WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE. NO USE OF ANY OF THE ACCOMPANYING CODE IS 225 | AUTHORIZED HEREUNDER EXCEPT UNDER THIS DISCLAIMER. 226 | 227 | TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL WOWZA MEDIA SYSTEMS, LLC OR ANY CONTRIBUTOR BE LIABLE FOR ANY 228 | INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE ACCOMPANYING CODE, 229 | INCLUDING BUT NOT LIMITED TO LOST PROFITS OR BUSINESS OPPORTUNITIES, LOSS OF USE, BUSINESS INTERRUPTION, OR LOSS OF DATA, UNDER 230 | ANY THEORY OF LIABILITY, WHETHER BASED IN CONTRACT, TORT, NEGLIGENCE, PRODUCT LIABILITY, OR OTHERWISE EVEN IF WOWZA OR A 231 | CONTRIBUTOR HAS BEEN INFORMED OF THE POSSIBILITY OF SUCH DAMAGES. 232 | 233 | TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, WOWZA MEDIA SYSTEM, LLC'S TOTAL AGGREGATE LIABILITY TO ANY LICENSEES AND 234 | CONTRIBUTORS HEREUNDER WILL NOT EXCEED TWENTY-FIVE U.S. DOLLARS (US $25.00). -------------------------------------------------------------------------------- /src/com/wowza/wms/plugin/s3upload/ModuleS3Upload.java: -------------------------------------------------------------------------------- 1 | /* 2 | * This code and all components (c) Copyright 2006 - 2018, Wowza Media Systems, LLC. All rights reserved. 3 | * This code is licensed pursuant to the Wowza Public License version 1.0, available at www.wowza.com/legal. 4 | */ 5 | package com.wowza.wms.plugin.s3upload; 6 | 7 | import java.io.File; 8 | import java.io.FileFilter; 9 | import java.io.FileInputStream; 10 | import java.io.FileOutputStream; 11 | import java.io.IOException; 12 | import java.util.ArrayList; 13 | import java.util.Collections; 14 | import java.util.Comparator; 15 | import java.util.Date; 16 | import java.util.HashMap; 17 | import java.util.Iterator; 18 | import java.util.List; 19 | import java.util.Map; 20 | import java.util.Timer; 21 | import java.util.TimerTask; 22 | import java.util.regex.Matcher; 23 | import java.util.regex.Pattern; 24 | 25 | import com.amazonaws.AmazonServiceException; 26 | import com.amazonaws.auth.AWSCredentialsProvider; 27 | import com.amazonaws.auth.AWSStaticCredentialsProvider; 28 | import com.amazonaws.auth.BasicAWSCredentials; 29 | import com.amazonaws.auth.profile.ProfileCredentialsProvider; 30 | import com.amazonaws.event.ProgressEvent; 31 | import com.amazonaws.event.ProgressEventType; 32 | import com.amazonaws.regions.Regions; 33 | import com.amazonaws.services.s3.AmazonS3; 34 | import com.amazonaws.services.s3.AmazonS3ClientBuilder; 35 | import com.amazonaws.services.s3.model.AccessControlList; 36 | import com.amazonaws.services.s3.model.CannedAccessControlList; 37 | import com.amazonaws.services.s3.model.GroupGrantee; 38 | import com.amazonaws.services.s3.model.HeadBucketRequest; 39 | import com.amazonaws.services.s3.model.HeadBucketResult; 40 | import com.amazonaws.services.s3.model.ListMultipartUploadsRequest; 41 | import com.amazonaws.services.s3.model.MultipartUploadListing; 42 | import com.amazonaws.services.s3.model.Permission; 43 | import com.amazonaws.services.s3.model.PutObjectRequest; 44 | import com.amazonaws.services.s3.transfer.PersistableTransfer; 45 | import com.amazonaws.services.s3.transfer.PersistableUpload; 46 | import com.amazonaws.services.s3.transfer.TransferManager; 47 | import com.amazonaws.services.s3.transfer.TransferManagerBuilder; 48 | import com.amazonaws.services.s3.transfer.Upload; 49 | import com.amazonaws.services.s3.transfer.internal.S3SyncProgressListener; 50 | import com.wowza.util.JSON; 51 | import com.wowza.util.StringUtils; 52 | import com.wowza.wms.application.IApplicationInstance; 53 | import com.wowza.wms.application.WMSProperties; 54 | import com.wowza.wms.logging.WMSLogger; 55 | import com.wowza.wms.logging.WMSLoggerFactory; 56 | import com.wowza.wms.logging.WMSLoggerIDs; 57 | import com.wowza.wms.module.ModuleBase; 58 | import com.wowza.wms.stream.IMediaStream; 59 | import com.wowza.wms.stream.IMediaWriterActionNotify; 60 | 61 | public class ModuleS3Upload extends ModuleBase 62 | { 63 | private class UploadTask extends TimerTask 64 | { 65 | private final String mediaName; 66 | private final long delay; 67 | private long lastAge = 0; 68 | 69 | UploadTask(String mediaName, long delay, long age) 70 | { 71 | this.mediaName = mediaName; 72 | this.delay = delay; 73 | lastAge = age; 74 | } 75 | 76 | @Override 77 | public void run() 78 | { 79 | boolean doUpload = false; 80 | boolean finished = false; 81 | 82 | synchronized(lock) 83 | { 84 | while (true) 85 | { 86 | if (shuttingDown) 87 | { 88 | if (debugLog) 89 | logger.info(MODULE_NAME + ".UploadTask.run() shutting down [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 90 | finished = true; 91 | break; 92 | } 93 | 94 | File uploadFile = new File(storageDir, mediaName + ".upload"); 95 | if (!uploadFile.exists()) 96 | { 97 | logger.warn(MODULE_NAME + ".UploadTask.run() .uploadfile missing [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 98 | finished = true; 99 | break; 100 | } 101 | 102 | long age = getFileAge(mediaName); 103 | if (age < lastAge) 104 | { 105 | logger.warn(MODULE_NAME + ".UploadTask.run() media file has been modified [" + appInstance.getContextStr() + "/" + mediaName + "] age: " + age + " < lastAge: " + lastAge, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 106 | finished = true; 107 | break; 108 | } 109 | lastAge = age; 110 | 111 | if (age >= delay) 112 | { 113 | if (debugLog) 114 | logger.info(MODULE_NAME + ".UploadTask.run() age >= delay [" + appInstance.getContextStr() + "/" + mediaName + "] age: " + age + ", delay: " + delay, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 115 | finished = true; 116 | doUpload = true; 117 | break; 118 | } 119 | if (debugLog) 120 | logger.info(MODULE_NAME + ".UploadTask.run() age < delay [" + appInstance.getContextStr() + "/" + mediaName + "] age: " + age + ", delay: " + delay, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 121 | break; 122 | } 123 | if (finished) 124 | { 125 | if (debugLog) 126 | logger.info(MODULE_NAME + ".UploadTask.run() removing timer [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 127 | uploadTimers.remove(mediaName); 128 | cancel(); 129 | } 130 | else 131 | { 132 | touchAppInstance(); 133 | } 134 | } 135 | 136 | if (doUpload) 137 | { 138 | if (debugLog) 139 | logger.info(MODULE_NAME + ".UploadTask.run() starting upload [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 140 | startUpload(mediaName); 141 | } 142 | } 143 | } 144 | 145 | private class MyFilter implements FileFilter 146 | { 147 | 148 | private String suffix; 149 | 150 | MyFilter(String suffix) 151 | { 152 | this.suffix = suffix; 153 | } 154 | 155 | @Override 156 | public boolean accept(File pathname) 157 | { 158 | return pathname.isDirectory() || pathname.getName().toLowerCase().endsWith(suffix.toLowerCase()); 159 | } 160 | 161 | } 162 | 163 | private class WriteListener implements IMediaWriterActionNotify 164 | { 165 | 166 | @Override 167 | public void onWriteComplete(IMediaStream stream, File file) 168 | { 169 | String mediaName = getMediaName(file.getPath()); 170 | if (debugLog) 171 | logger.info(MODULE_NAME + ".onWriteComplete [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 172 | 173 | if (transferManager == null) 174 | { 175 | logger.warn(MODULE_NAME + ".WriteListener.onWriteComplete Cannot upload file because S3 Transfer Manager isn't loaded: [" + appInstance.getContextStr() + "/" + mediaName + "]"); 176 | } 177 | 178 | File uploadFile = null; 179 | synchronized(lock) 180 | { 181 | try 182 | { 183 | uploadFile = new File(file.getPath() + ".upload"); 184 | if (uploadFile.exists()) 185 | { 186 | if (debugLog) 187 | logger.info(MODULE_NAME + ".onWriteComplete .upload file exists (deleting) [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 188 | uploadFile.delete(); 189 | } 190 | uploadFile.createNewFile(); 191 | if (!shuttingDown) 192 | startUpload(mediaName, uploadDelay); 193 | } 194 | catch (IOException e) 195 | { 196 | logger.error(MODULE_NAME + ".WriteListener.onWriteComplete Cannot create .upload file: [" + appInstance.getContextStr() + "/" + mediaName + "]", e); 197 | } 198 | } 199 | } 200 | 201 | @Override 202 | public void onFLVAddMetadata(IMediaStream stream, Map extraMetadata) 203 | { 204 | // no-op 205 | } 206 | } 207 | 208 | private class ProgressListener extends S3SyncProgressListener 209 | { 210 | final String mediaName; 211 | final String uploadName; 212 | 213 | ProgressListener(String mediaName, String uploadName) 214 | { 215 | this.mediaName = mediaName; 216 | this.uploadName = uploadName; 217 | } 218 | 219 | @Override 220 | public void progressChanged(ProgressEvent progressEvent) 221 | { 222 | if (progressEvent.getEventType().isTransferEvent()) 223 | { 224 | ProgressEventType type = progressEvent.getEventType(); 225 | switch (type) 226 | { 227 | case TRANSFER_COMPLETED_EVENT: 228 | if (debugLog) 229 | logger.info(MODULE_NAME + ".ProgressListener.progressChanged [" + appInstance.getContextStr() + "/" + mediaName + "] event: " + type.toString(), WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 230 | synchronized(lock) 231 | { 232 | if (uploadName != null) 233 | currentUploads.remove(uploadName); 234 | File uploadFile = new File(storageDir, mediaName + ".upload"); 235 | uploadFile.delete(); 236 | } 237 | if (deleteOriginalFiles) 238 | { 239 | File mediaFile = new File(storageDir, mediaName); 240 | mediaFile.delete(); 241 | } 242 | break; 243 | 244 | case TRANSFER_FAILED_EVENT: 245 | if (debugLog) 246 | logger.warn(MODULE_NAME + ".ProgressListener.progressChanged [" + appInstance.getContextStr() + "/" + mediaName + "] event: " + type.toString(), WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 247 | synchronized(lock) 248 | { 249 | if (uploadName != null) 250 | currentUploads.remove(uploadName); 251 | if (debugLog) 252 | logger.info(MODULE_NAME + ".ProgressListener.progressChanged [" + appInstance.getContextStr() + "/" + mediaName + "] event: " + type.toString() + ", shutting down", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 253 | if (shuttingDown) 254 | break; 255 | } 256 | 257 | if (restartFailedUploads) 258 | { 259 | if (debugLog) 260 | logger.info(MODULE_NAME + ".ProgressListener.progressChanged [" + appInstance.getContextStr() + "/" + mediaName + "] event: " + type.toString() + ", restarting upload", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 261 | long age = getFileAge(mediaName); 262 | startUpload(mediaName, restartFailedUploadsTimeout + age); 263 | } 264 | break; 265 | 266 | default: 267 | if (debugLog) 268 | logger.info(MODULE_NAME + ".ProgressListener.progressChanged [" + appInstance.getContextStr() + "/" + mediaName + "] event: " + type.toString(), WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 269 | break; 270 | } 271 | } 272 | touchAppInstance(); 273 | } 274 | 275 | @Override 276 | public void onPersistableTransfer(final PersistableTransfer transfer) 277 | { 278 | if (debugLog) 279 | logger.info(MODULE_NAME + ".ProgressListener.onPersistableTransfer() [" + appInstance.getContextStr() + "/" + mediaName + "] data: " + transfer.serialize(), WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 280 | appInstance.getVHost().getThreadPool().execute(new Runnable() 281 | { 282 | 283 | @Override 284 | public void run() 285 | { 286 | synchronized(lock) 287 | { 288 | FileOutputStream fos = null; 289 | File tmp = new File(storageDir, mediaName + ".upload"); 290 | try 291 | { 292 | if (!tmp.exists()) 293 | tmp.createNewFile(); 294 | fos = new FileOutputStream(tmp); 295 | transfer.serialize(fos); 296 | } 297 | catch (Exception e) 298 | { 299 | 300 | } 301 | finally 302 | { 303 | if (fos != null) 304 | { 305 | try 306 | { 307 | fos.close(); 308 | } 309 | catch (Exception e) 310 | { 311 | } 312 | } 313 | } 314 | } 315 | } 316 | }); 317 | } 318 | } 319 | 320 | public static final String MODULE_NAME = "ModuleS3Upload"; 321 | public static final String PROP_NAME_PREFIX = "s3Upload"; 322 | 323 | private WMSLogger logger = null; 324 | private IApplicationInstance appInstance = null; 325 | 326 | private TransferManager transferManager = null; 327 | private AccessControlList acl = null; 328 | private CannedAccessControlList cannedAcl = null; 329 | 330 | private String accessKey = null; 331 | private String secretKey = null; 332 | private String awsProfile = null; 333 | private String awsProfilePath = null; 334 | private String bucketName = null; 335 | private String filePrefix = null; 336 | private String endpoint = null; 337 | private String regionName = null; 338 | private File storageDir = null; 339 | private Map uploadTimers = new HashMap(); 340 | private List currentUploads = new ArrayList(); 341 | 342 | private boolean checkBucket = true; 343 | private boolean useDefaultRegion = true; 344 | private boolean allowBucketRegionOverride = true; 345 | private boolean debugLog = false; 346 | private boolean shuttingDown = false; 347 | private boolean resumeUploads = true; 348 | private boolean versionFile = false; 349 | private boolean stripRecorderVersioning = true; 350 | private boolean deleteOriginalFiles = false; 351 | private boolean restartFailedUploads = true; 352 | 353 | private long restartFailedUploadsTimeout = 60000l; 354 | private long uploadDelay = 0l; 355 | private long lastTouch = -1; 356 | private long touchTimeout = 2500; 357 | 358 | private Object lock = new Object(); 359 | 360 | public void onAppStart(IApplicationInstance appInstance) 361 | { 362 | this.appInstance = appInstance; 363 | logger = WMSLoggerFactory.getLoggerObj(appInstance); 364 | logger.info(MODULE_NAME + ".onAppStart [" + appInstance.getContextStr() + " : build #55]"); 365 | touchTimeout = appInstance.getApplicationInstanceTouchTimeout() / 2; 366 | 367 | try 368 | { 369 | WMSProperties props = appInstance.getProperties(); 370 | String storageDirStr = appInstance.decodeStorageDir(appInstance.getStreamRecorderProperties().getPropertyStr("streamRecorderOutputPath", appInstance.getStreamStorageDir())); 371 | storageDir = new File(storageDirStr); 372 | accessKey = props.getPropertyStr("s3UploadAccessKey", accessKey); 373 | secretKey = props.getPropertyStr("s3UploadSecretKey", secretKey); 374 | awsProfile = props.getPropertyStr("s3UploadAwsProfile", awsProfile); 375 | awsProfilePath = props.getPropertyStr("s3UploadAwsProfilePath", awsProfilePath); 376 | bucketName = props.getPropertyStr("s3UploadBucketName", bucketName); 377 | filePrefix = appInstance.decodeStorageDir(props.getPropertyStr("s3UploadFilePrefix", filePrefix)); 378 | 379 | // prefer to set region rather than endpoint which will be deprecated at some point. 380 | regionName = props.getPropertyStr("s3UploadRegion", regionName); 381 | if (StringUtils.isEmpty(regionName)) 382 | { 383 | endpoint = props.getPropertyStr("s3UploadEndpoint", endpoint); 384 | regionName = getRegion(); 385 | } 386 | // if region or endpoint isn't set then use the default region. 387 | // disable if region can be determined via the DefaultAwsRegionProviderChain. 388 | useDefaultRegion = props.getPropertyBoolean("s3UploadUseDefaultRegion", useDefaultRegion); 389 | // turn on global bucket access so that uploads won't fail if the region is incorrect. 390 | allowBucketRegionOverride = props.getPropertyBoolean("s3UploadAllowBucketRegionOverride", allowBucketRegionOverride); 391 | checkBucket = props.getPropertyBoolean("s3UploadCheckBucket", checkBucket); 392 | debugLog = props.getPropertyBoolean("s3UploadDebugLog", debugLog); 393 | resumeUploads = props.getPropertyBoolean("s3UploadResumeUploads", resumeUploads); 394 | restartFailedUploads = props.getPropertyBoolean("s3UploadRestartFailedUploads", restartFailedUploads); 395 | restartFailedUploadsTimeout = props.getPropertyLong("s3UploadRestartFailedUploadTimeout", restartFailedUploadsTimeout); 396 | versionFile = props.getPropertyBoolean("s3UploadVersionFile", versionFile); 397 | stripRecorderVersioning = props.getPropertyBoolean("s3UploadStripRecorderVersioning", stripRecorderVersioning); 398 | deleteOriginalFiles = props.getPropertyBoolean("s3UploadDeletOriginalFiles", deleteOriginalFiles); 399 | // fix typo in property name 400 | deleteOriginalFiles = props.getPropertyBoolean("s3UploadDeleteOriginalFiles", deleteOriginalFiles); 401 | uploadDelay = props.getPropertyLong("s3UploadDelay", uploadDelay); 402 | 403 | // This value should be the URI representation of the "Group Grantee" found here http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html under "Amazon S3 Predefined Groups" 404 | String aclGroupGranteeUri = props.getPropertyStr("s3UploadACLGroupGranteeUri"); 405 | // This should be a string that represents the level of permissions we want to grant to the "Group Grantee" access to the file to be uploaded 406 | String aclPermissionRule = props.getPropertyStr("s3UploadACLPermissionRule"); 407 | 408 | GroupGrantee grantee = null; 409 | Permission permission = null; 410 | 411 | // With the passed property, check if it maps to a specified GroupGrantee 412 | if (!StringUtils.isEmpty(aclGroupGranteeUri)) 413 | grantee = GroupGrantee.parseGroupGrantee(aclGroupGranteeUri); 414 | // In order for the parsing to work correctly, we will go ahead and force uppercase on the string passed' 415 | if (!StringUtils.isEmpty(aclPermissionRule)) 416 | permission = Permission.parsePermission(aclPermissionRule.toUpperCase()); 417 | 418 | // If we have properties for specifying permissions on the file upload, create the AccessControlList object and set the Grantee and Permissions 419 | if (grantee != null && permission != null) 420 | { 421 | acl = new AccessControlList(); 422 | acl.grantPermission(grantee, permission); 423 | } 424 | 425 | String cannedAclStr = props.getPropertyStr("s3UploadCannedAcl"); 426 | if (!StringUtils.isEmpty(cannedAclStr)) 427 | { 428 | for (CannedAccessControlList c : CannedAccessControlList.values()) 429 | { 430 | if (c.toString().equals(cannedAclStr)) 431 | { 432 | cannedAcl = c; 433 | break; 434 | } 435 | } 436 | } 437 | 438 | AmazonS3 s3Client = null; 439 | AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard(); 440 | Regions region = null; 441 | try 442 | { 443 | region = Regions.fromName(regionName); 444 | } 445 | catch (IllegalArgumentException e) 446 | { 447 | if (useDefaultRegion) 448 | { 449 | region = Regions.getCurrentRegion() != null ? Regions.fromName(Regions.getCurrentRegion().getName()) : Regions.DEFAULT_REGION; 450 | // set the regionName to the default region. Used in the bucket check later. 451 | if (region != null) 452 | regionName = region.getName(); 453 | } 454 | } 455 | finally 456 | { 457 | if (region != null) 458 | { 459 | builder.withRegion(region); 460 | if (allowBucketRegionOverride) 461 | { 462 | builder.withForceGlobalBucketAccessEnabled(true); 463 | } 464 | } 465 | } 466 | AWSCredentialsProvider credentialsProvider = null; 467 | 468 | // backwards compatibility 469 | if (!StringUtils.isEmpty(accessKey) && !StringUtils.isEmpty(secretKey)) 470 | { 471 | logger.info(MODULE_NAME + ".onAppStart: [" + appInstance.getContextStr() + "] using supplied aws credentials", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 472 | credentialsProvider = new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)); 473 | } 474 | else if (!StringUtils.isEmpty(awsProfile)) 475 | { 476 | logger.info(MODULE_NAME + ".onAppStart: [" + appInstance.getContextStr() + "] using aws profile: " + awsProfile, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 477 | if (StringUtils.isEmpty(awsProfilePath)) 478 | { 479 | credentialsProvider = new ProfileCredentialsProvider(awsProfile); 480 | } 481 | else 482 | { 483 | credentialsProvider = new ProfileCredentialsProvider(awsProfilePath, awsProfile); 484 | } 485 | } 486 | else 487 | { 488 | logger.info(MODULE_NAME + ".onAppStart: [" + appInstance.getContextStr() + "] using default aws credentials provider chain", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 489 | 490 | } 491 | 492 | if (credentialsProvider != null) 493 | builder.withCredentials(credentialsProvider); 494 | 495 | s3Client = builder.build(); 496 | 497 | if (checkBucket) 498 | { 499 | // check that the bucket exists and the s3Client can access it. 500 | // fails with a 404 response if the bucket doesn't exist and a 403 response if the s3Client doesn't have permission to access it. 501 | // fails with a 301 response if the bucket is in a different region and allowBucketRegionOverride isn't set (otherwise log a warning). 502 | HeadBucketResult headBucketResult = s3Client.headBucket(new HeadBucketRequest(bucketName)); 503 | String bucketRegion = headBucketResult.getBucketRegion(); 504 | if (!bucketRegion.equalsIgnoreCase(regionName)) 505 | logger.warn(MODULE_NAME + ".onAppStart: [" + appInstance.getContextStr() + "] bucket region doesn't match configured region. (b:c)[" + bucketRegion + ":" + regionName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 506 | } 507 | transferManager = TransferManagerBuilder.standard().withS3Client(s3Client).build(); 508 | logger.info(MODULE_NAME + ".onAppStart [" + appInstance.getContextStr() + "] Local Storage Dir: " + storageDirStr + ", S3 Bucket Name: " + bucketName + ", File Prefix: " + filePrefix + ", Resume Uploads: " + resumeUploads + ", Delete Original Files: " + deleteOriginalFiles 509 | + ", Version Files: " + versionFile + ", Upload Delay: " + uploadDelay, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 510 | 511 | appInstance.getVHost().getThreadPool().execute(new Runnable() 512 | { 513 | 514 | @Override 515 | public void run() 516 | { 517 | resumeUploads(); 518 | } 519 | }); 520 | } 521 | catch (IllegalStateException ise) 522 | { 523 | logger.error(MODULE_NAME + ".onAppStart [" + appInstance.getContextStr() + "] Illegal State Exception thrown. The installed version of AWS SDK may not be compatible with this version of Wowza Streaming Engine. Please check and upgrade your version of AWS SDK.", ise); 524 | } 525 | catch (AmazonServiceException ase) 526 | { 527 | int status = ase.getStatusCode(); 528 | String message = ase.getErrorMessage(); 529 | 530 | logger.warn(MODULE_NAME + ".onAppStart: [" + appInstance.getContextStr() + "] missing S3 bucket: " + bucketName + ", S3 returned status: " + status + ", message: " + message, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 531 | } 532 | catch (Exception e) 533 | { 534 | logger.error(MODULE_NAME + ".onAppStart [" + appInstance.getContextStr() + "] exception: " + e.getMessage(), e); 535 | } 536 | catch (Throwable t) 537 | { 538 | logger.error(MODULE_NAME + ".onAppStart [" + appInstance.getContextStr() + "] throwable exception: " + t.getMessage(), t); 539 | } 540 | 541 | appInstance.addMediaWriterListener(new WriteListener()); 542 | } 543 | 544 | public void onAppStop(IApplicationInstance appInstance) 545 | { 546 | logger.info(MODULE_NAME + ".onAppStop [" + appInstance.getContextStr() + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 547 | synchronized(lock) 548 | { 549 | shuttingDown = true; 550 | Iterator iter = uploadTimers.keySet().iterator(); 551 | while (iter.hasNext()) 552 | { 553 | String mediaName = iter.next(); 554 | Timer t = uploadTimers.get(mediaName); 555 | if (t != null) 556 | t.cancel(); 557 | if (debugLog) 558 | logger.info(MODULE_NAME + ".onAppStop stopping pending upload [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 559 | iter.remove(); 560 | } 561 | } 562 | 563 | try 564 | { 565 | if (transferManager != null) 566 | { 567 | transferManager.shutdownNow(false); 568 | } 569 | } 570 | catch (Exception e) 571 | { 572 | logger.error(MODULE_NAME + ".onAppStop [" + appInstance.getContextStr() + "] exception: " + e.getMessage(), e); 573 | } 574 | } 575 | 576 | private void resumeUploads() 577 | { 578 | if (debugLog) 579 | logger.info(MODULE_NAME + ".resumeUploads " + (resumeUploads ? "resuming" : "aborting") + " unfinished Uploads [" + appInstance.getContextStr() + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 580 | 581 | if (transferManager != null && !resumeUploads) 582 | { 583 | transferManager.abortMultipartUploads(bucketName, new Date()); 584 | } 585 | 586 | List uploadFiles = getMatchingFiles(storageDir, ".upload"); 587 | 588 | for (File uploadFile : uploadFiles) 589 | { 590 | if (!resumeUploads) 591 | { 592 | uploadFile.delete(); 593 | } 594 | else 595 | { 596 | String mediaName = getMediaName(uploadFile.getPath()); 597 | startUpload(mediaName, uploadDelay); 598 | } 599 | } 600 | } 601 | 602 | private void startUpload(String mediaName, long delay) 603 | { 604 | synchronized(lock) 605 | { 606 | Timer t = uploadTimers.remove(mediaName); 607 | if (t != null) 608 | t.cancel(); 609 | long age = getFileAge(mediaName); 610 | if (delay > 0 && age != -1 && age < delay) 611 | { 612 | t = new Timer("UploadTimer: [" + appInstance.getContextStr() + "/" + mediaName + "]"); 613 | long timerDelay = Math.min(delay - age, touchTimeout); 614 | t.schedule(new UploadTask(mediaName, delay, age), timerDelay, timerDelay / 2); 615 | uploadTimers.put(mediaName, t); 616 | if (debugLog) 617 | logger.info(MODULE_NAME + ".startUpload (delayed) for [" + appInstance.getContextStr() + "/" + mediaName + "] age: " + age + ", delay: " + delay + ", timerDelay: " + timerDelay, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 618 | } 619 | else 620 | { 621 | if (debugLog) 622 | logger.info(MODULE_NAME + ".startUpload (now) for [" + appInstance.getContextStr() + "/" + mediaName + "] age: " + age + ", delay: " + delay, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 623 | startUpload(mediaName); 624 | } 625 | } 626 | } 627 | 628 | private void startUpload(String mediaName) 629 | { 630 | touchAppInstance(); 631 | 632 | File uploadFile = new File(storageDir, mediaName + ".upload"); 633 | if (uploadFile == null || !uploadFile.exists()) 634 | return; 635 | 636 | if (transferManager != null) 637 | { 638 | Upload upload = null; 639 | FileInputStream fis = null; 640 | String uploadName = null; 641 | try 642 | { 643 | if (uploadFile.length() == 0) 644 | { 645 | if (debugLog) 646 | logger.info(MODULE_NAME + ".startUpload new or single part upload for [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 647 | 648 | File mediaFile = new File(storageDir, mediaName); 649 | 650 | if (mediaFile.exists()) 651 | { 652 | uploadName = mediaName; 653 | if (!StringUtils.isEmpty(filePrefix)) 654 | { 655 | uploadName = filePrefix + (filePrefix.endsWith("/") ? "" : "/") + uploadName; 656 | } 657 | if (versionFile) 658 | { 659 | uploadName = getMediaNameVersion(uploadName); 660 | } 661 | // In order to support setting ACL permissions for the file upload, we will wrap the upload properties in a PutObjectRequest 662 | PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, uploadName, mediaFile); 663 | 664 | // if the user has specified ACL properties, setup the putObjectRequest with the acl permissions generated 665 | if (acl != null) 666 | { 667 | putObjectRequest.withAccessControlList(acl); 668 | } 669 | // else add cannedACL if one is set 670 | else if (cannedAcl != null) 671 | { 672 | putObjectRequest.withCannedAcl(cannedAcl); 673 | } 674 | 675 | upload = transferManager.upload(putObjectRequest); 676 | } 677 | else 678 | { 679 | logger.warn(MODULE_NAME + ".startUpload mediaFile doesn't exist [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 680 | uploadFile.delete(); 681 | } 682 | } 683 | else 684 | { 685 | if (debugLog) 686 | logger.info(MODULE_NAME + ".startUpload resuming multipart upload for [" + appInstance.getContextStr() + "/" + mediaName + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 687 | fis = new FileInputStream(uploadFile); 688 | // Deserialize PersistableUpload information from disk. 689 | PersistableUpload persistableUpload = PersistableTransfer.deserializeFrom(fis); 690 | upload = transferManager.resumeUpload(persistableUpload); 691 | JSON json = new JSON(persistableUpload.serialize()); 692 | uploadName = json.getString("key"); 693 | } 694 | if (upload != null) 695 | { 696 | currentUploads.add(uploadName); 697 | upload.addProgressListener(new ProgressListener(mediaName, uploadName)); 698 | } 699 | } 700 | catch (Exception e) 701 | { 702 | logger.error(MODULE_NAME + ".startUpload error starting or resuming upload: [" + appInstance.getContextStr() + "/" + uploadFile.getName() + "]", e); 703 | } 704 | finally 705 | { 706 | if (fis != null) 707 | { 708 | try 709 | { 710 | fis.close(); 711 | } 712 | catch (IOException e) 713 | { 714 | } 715 | } 716 | } 717 | } 718 | else 719 | { 720 | logger.warn(MODULE_NAME + ".startUpload problem starting or resuming upload: [" + appInstance.getContextStr() + "/" + uploadFile.getName() + "] Amazon S3 TransferManager not running."); 721 | } 722 | } 723 | 724 | private List getMatchingFiles(File dir, String suffix) 725 | { 726 | List ret = new ArrayList(); 727 | 728 | File[] files = dir.listFiles(new MyFilter(suffix)); 729 | 730 | if (files != null && files.length > 0) 731 | { 732 | for (File file : files) 733 | { 734 | if (file.isDirectory()) 735 | ret.addAll(getMatchingFiles(file, suffix)); 736 | else 737 | { 738 | if (debugLog) 739 | logger.info(MODULE_NAME + ".getMatchingFile add file: " + file.getName(), WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 740 | ret.add(file); 741 | } 742 | } 743 | } 744 | 745 | Collections.sort(ret, new Comparator() 746 | { 747 | public int compare(File f1, File f2) 748 | { 749 | return Long.valueOf(f1.lastModified()).compareTo(f2.lastModified()); 750 | } 751 | }); 752 | 753 | return ret; 754 | } 755 | 756 | private long getFileAge(String mediaFile) 757 | { 758 | long age = -1; 759 | 760 | File file = new File(storageDir, mediaFile); 761 | if (file.exists()) 762 | age = System.currentTimeMillis() - file.lastModified(); 763 | 764 | return age; 765 | } 766 | 767 | private String getMediaName(String path) 768 | { 769 | String mediaName = path.replace(storageDir.getPath(), ""); 770 | if (mediaName.startsWith(File.separator)) 771 | mediaName = mediaName.substring(File.separator.length()); 772 | if (mediaName.endsWith(".upload")) 773 | mediaName = mediaName.substring(0, mediaName.indexOf(".upload")); 774 | 775 | return mediaName; 776 | } 777 | 778 | private String getMediaNameVersion(String mediaName) 779 | { 780 | if (stripRecorderVersioning) 781 | { 782 | Pattern pattern = Pattern.compile("(.*)(_\\d+)(\\.\\w+)"); 783 | Matcher matcher = pattern.matcher(mediaName); 784 | if (debugLog) 785 | logger.info(MODULE_NAME + ".getMediaNameVersion stripRecorderVersioning: " + mediaName + ": " + matcher.toString(), WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 786 | if (matcher.matches()) 787 | { 788 | mediaName = matcher.group(1) + matcher.group(3); 789 | if (debugLog) 790 | logger.info(MODULE_NAME + ".getMediaNameVersion stripRecorderVersioning new mediaName: " + mediaName, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 791 | } 792 | } 793 | 794 | boolean exists = doesObjectExistOnS3(mediaName); 795 | 796 | if (!exists) 797 | return mediaName; 798 | 799 | String newName = mediaName; 800 | String oldName = mediaName; 801 | String oldExt = ""; 802 | int oldExtIndex = oldName.lastIndexOf("."); 803 | if (oldExtIndex >= 0) 804 | { 805 | oldExt = oldName.substring(oldExtIndex); 806 | oldName = oldName.substring(0, oldExtIndex); 807 | } 808 | 809 | int version = 0; 810 | while (true) 811 | { 812 | newName = oldName + "_" + Integer.toString(version) + oldExt; 813 | exists = doesObjectExistOnS3(newName); 814 | if (debugLog) 815 | logger.info(MODULE_NAME + ".getMediaNameVersion doesObjectExist: " + newName + ": " + Boolean.toString(exists), WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 816 | if (!exists) 817 | break; 818 | version++; 819 | touchAppInstance(); 820 | } 821 | if (debugLog) 822 | logger.info(MODULE_NAME + ".getMediaNameVersion using: " + newName, WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 823 | return newName; 824 | } 825 | 826 | private boolean doesObjectExistOnS3(String mediaName) 827 | { 828 | AmazonS3 s3 = transferManager.getAmazonS3Client(); 829 | 830 | boolean exists = currentUploads.contains(mediaName); 831 | 832 | if (!exists) 833 | exists = s3.doesObjectExist(bucketName, mediaName); 834 | 835 | if (!exists) 836 | { 837 | ListMultipartUploadsRequest request = new ListMultipartUploadsRequest(bucketName).withPrefix(mediaName); 838 | MultipartUploadListing multipartUploads = s3.listMultipartUploads(request); 839 | if (!multipartUploads.getMultipartUploads().isEmpty()) 840 | exists = true; 841 | } 842 | 843 | return exists; 844 | } 845 | 846 | private String getRegion() 847 | { 848 | if (!StringUtils.isEmpty(regionName)) 849 | return regionName; 850 | 851 | try 852 | { 853 | Pattern pattern = Pattern.compile("(s3\\.dualstack.|s3\\.|s3-)(.+)\\.amazonaws.com"); 854 | if (!StringUtils.isEmpty(endpoint)) 855 | { 856 | Matcher matcher = pattern.matcher(endpoint); 857 | if (matcher.matches()) 858 | regionName = matcher.group(2); 859 | 860 | if (StringUtils.isEmpty(regionName)) 861 | logger.warn(MODULE_NAME + ".getRegion [" + appInstance.getContextStr() + "] Unable to extract region name from endpoint. [" + endpoint + "]"); 862 | } 863 | } 864 | catch (Exception e) 865 | { 866 | logger.warn(MODULE_NAME + ".getRegion [" + appInstance.getContextStr() + "] Exception throw while trying to extract region name from endpoint. [" + endpoint + "]", e); 867 | 868 | } 869 | return regionName; 870 | } 871 | 872 | private void touchAppInstance() 873 | { 874 | // touch the appInstance so it doesn't timeout while we are still uploading. 875 | long now = System.currentTimeMillis(); 876 | if (now - touchTimeout >= lastTouch) 877 | { 878 | if (debugLog) 879 | logger.info(MODULE_NAME + " touching appInstance [" + appInstance.getContextStr() + "]", WMSLoggerIDs.CAT_application, WMSLoggerIDs.EVT_comment); 880 | appInstance.touch(); 881 | lastTouch = now; 882 | } 883 | } 884 | } 885 | --------------------------------------------------------------------------------