├── .gitignore ├── .mergify.yml ├── LICENSE.txt └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | 3 | build 4 | .idea -------------------------------------------------------------------------------- /.mergify.yml: -------------------------------------------------------------------------------- 1 | pull_request_rules: 2 | - name: Automatic merge for snapshot branch when CI passes and author is elasticmachine or apmmachine 3 | conditions: 4 | - or: 5 | - author=elasticmachine 6 | - author=apmmachine 7 | - and: 8 | - check-success=package-storage/pr-merge 9 | - base=snapshot 10 | actions: 11 | merge: 12 | method: squash 13 | 14 | - name: Delete upstream branch after merging changes automatically or it's closed 15 | conditions: 16 | - or: 17 | - merged 18 | - closed 19 | - or: 20 | - head~=^update-snapshot.* 21 | - head~=^update-apm.* 22 | actions: 23 | delete_head_branch: 24 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | ELASTIC LICENSE AGREEMENT 2 | 3 | PLEASE READ CAREFULLY THIS ELASTIC LICENSE AGREEMENT (THIS "AGREEMENT"), WHICH 4 | CONSTITUTES A LEGALLY BINDING AGREEMENT AND GOVERNS ALL OF YOUR USE OF ALL OF 5 | THE ELASTIC SOFTWARE WITH WHICH THIS AGREEMENT IS INCLUDED ("ELASTIC SOFTWARE") 6 | THAT IS PROVIDED IN OBJECT CODE FORMAT, AND, IN ACCORDANCE WITH SECTION 2 BELOW, 7 | CERTAIN OF THE ELASTIC SOFTWARE THAT IS PROVIDED IN SOURCE CODE FORMAT. BY 8 | INSTALLING OR USING ANY OF THE ELASTIC SOFTWARE GOVERNED BY THIS AGREEMENT, YOU 9 | ARE ASSENTING TO THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE 10 | WITH SUCH TERMS AND CONDITIONS, YOU MAY NOT INSTALL OR USE THE ELASTIC SOFTWARE 11 | GOVERNED BY THIS AGREEMENT. IF YOU ARE INSTALLING OR USING THE SOFTWARE ON 12 | BEHALF OF A LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE THE ACTUAL 13 | AUTHORITY TO AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT ON BEHALF OF 14 | SUCH ENTITY. 15 | 16 | Posted Date: April 20, 2018 17 | 18 | This Agreement is entered into by and between Elasticsearch BV ("Elastic") and 19 | You, or the legal entity on behalf of whom You are acting (as applicable, 20 | "You"). 21 | 22 | 1. OBJECT CODE END USER LICENSES, RESTRICTIONS AND THIRD PARTY OPEN SOURCE 23 | SOFTWARE 24 | 25 | 1.1 Object Code End User License. Subject to the terms and conditions of 26 | Section 1.2 of this Agreement, Elastic hereby grants to You, AT NO CHARGE and 27 | for so long as you are not in breach of any provision of this Agreement, a 28 | License to the Basic Features and Functions of the Elastic Software. 29 | 30 | 1.2 Reservation of Rights; Restrictions. As between Elastic and You, Elastic 31 | and its licensors own all right, title and interest in and to the Elastic 32 | Software, and except as expressly set forth in Sections 1.1, and 2.1 of this 33 | Agreement, no other license to the Elastic Software is granted to You under 34 | this Agreement, by implication, estoppel or otherwise. You agree not to: (i) 35 | reverse engineer or decompile, decrypt, disassemble or otherwise reduce any 36 | Elastic Software provided to You in Object Code, or any portion thereof, to 37 | Source Code, except and only to the extent any such restriction is prohibited 38 | by applicable law, (ii) except as expressly permitted in this Agreement, 39 | prepare derivative works from, modify, copy or use the Elastic Software Object 40 | Code or the Commercial Software Source Code in any manner; (iii) except as 41 | expressly permitted in Section 1.1 above, transfer, sell, rent, lease, 42 | distribute, sublicense, loan or otherwise transfer, Elastic Software Object 43 | Code, in whole or in part, to any third party; (iv) use Elastic Software 44 | Object Code for providing time-sharing services, any software-as-a-service, 45 | service bureau services or as part of an application services provider or 46 | other service offering (collectively, "SaaS Offering") where obtaining access 47 | to the Elastic Software or the features and functions of the Elastic Software 48 | is a primary reason or substantial motivation for users of the SaaS Offering 49 | to access and/or use the SaaS Offering ("Prohibited SaaS Offering"); (v) 50 | circumvent the limitations on use of Elastic Software provided to You in 51 | Object Code format that are imposed or preserved by any License Key, or (vi) 52 | alter or remove any Marks and Notices in the Elastic Software. If You have any 53 | question as to whether a specific SaaS Offering constitutes a Prohibited SaaS 54 | Offering, or are interested in obtaining Elastic's permission to engage in 55 | commercial or non-commercial distribution of the Elastic Software, please 56 | contact elastic_license@elastic.co. 57 | 58 | 1.3 Third Party Open Source Software. The Commercial Software may contain or 59 | be provided with third party open source libraries, components, utilities and 60 | other open source software (collectively, "Open Source Software"), which Open 61 | Source Software may have applicable license terms as identified on a website 62 | designated by Elastic. Notwithstanding anything to the contrary herein, use of 63 | the Open Source Software shall be subject to the license terms and conditions 64 | applicable to such Open Source Software, to the extent required by the 65 | applicable licensor (which terms shall not restrict the license rights granted 66 | to You hereunder, but may contain additional rights). To the extent any 67 | condition of this Agreement conflicts with any license to the Open Source 68 | Software, the Open Source Software license will govern with respect to such 69 | Open Source Software only. Elastic may also separately provide you with 70 | certain open source software that is licensed by Elastic. Your use of such 71 | Elastic open source software will not be governed by this Agreement, but by 72 | the applicable open source license terms. 73 | 74 | 2. COMMERCIAL SOFTWARE SOURCE CODE 75 | 76 | 2.1 Limited License. Subject to the terms and conditions of Section 2.2 of 77 | this Agreement, Elastic hereby grants to You, AT NO CHARGE and for so long as 78 | you are not in breach of any provision of this Agreement, a limited, 79 | non-exclusive, non-transferable, fully paid up royalty free right and license 80 | to the Commercial Software in Source Code format, without the right to grant 81 | or authorize sublicenses, to prepare Derivative Works of the Commercial 82 | Software, provided You (i) do not hack the licensing mechanism, or otherwise 83 | circumvent the intended limitations on the use of Elastic Software to enable 84 | features other than Basic Features and Functions or those features You are 85 | entitled to as part of a Subscription, and (ii) use the resulting object code 86 | only for reasonable testing purposes. 87 | 88 | 2.2 Restrictions. Nothing in Section 2.1 grants You the right to (i) use the 89 | Commercial Software Source Code other than in accordance with Section 2.1 90 | above, (ii) use a Derivative Work of the Commercial Software outside of a 91 | Non-production Environment, in any production capacity, on a temporary or 92 | permanent basis, or (iii) transfer, sell, rent, lease, distribute, sublicense, 93 | loan or otherwise make available the Commercial Software Source Code, in whole 94 | or in part, to any third party. Notwithstanding the foregoing, You may 95 | maintain a copy of the repository in which the Source Code of the Commercial 96 | Software resides and that copy may be publicly accessible, provided that you 97 | include this Agreement with Your copy of the repository. 98 | 99 | 3. TERMINATION 100 | 101 | 3.1 Termination. This Agreement will automatically terminate, whether or not 102 | You receive notice of such Termination from Elastic, if You breach any of its 103 | provisions. 104 | 105 | 3.2 Post Termination. Upon any termination of this Agreement, for any reason, 106 | You shall promptly cease the use of the Elastic Software in Object Code format 107 | and cease use of the Commercial Software in Source Code format. For the 108 | avoidance of doubt, termination of this Agreement will not affect Your right 109 | to use Elastic Software, in either Object Code or Source Code formats, made 110 | available under the Apache License Version 2.0. 111 | 112 | 3.3 Survival. Sections 1.2, 2.2. 3.3, 4 and 5 shall survive any termination or 113 | expiration of this Agreement. 114 | 115 | 4. DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITY 116 | 117 | 4.1 Disclaimer of Warranties. TO THE MAXIMUM EXTENT PERMITTED UNDER APPLICABLE 118 | LAW, THE ELASTIC SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, 119 | AND ELASTIC AND ITS LICENSORS MAKE NO WARRANTIES WHETHER EXPRESSED, IMPLIED OR 120 | STATUTORY REGARDING OR RELATING TO THE ELASTIC SOFTWARE. TO THE MAXIMUM EXTENT 121 | PERMITTED UNDER APPLICABLE LAW, ELASTIC AND ITS LICENSORS SPECIFICALLY 122 | DISCLAIM ALL IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR 123 | PURPOSE AND NON-INFRINGEMENT WITH RESPECT TO THE ELASTIC SOFTWARE, AND WITH 124 | RESPECT TO THE USE OF THE FOREGOING. FURTHER, ELASTIC DOES NOT WARRANT RESULTS 125 | OF USE OR THAT THE ELASTIC SOFTWARE WILL BE ERROR FREE OR THAT THE USE OF THE 126 | ELASTIC SOFTWARE WILL BE UNINTERRUPTED. 127 | 128 | 4.2 Limitation of Liability. IN NO EVENT SHALL ELASTIC OR ITS LICENSORS BE 129 | LIABLE TO YOU OR ANY THIRD PARTY FOR ANY DIRECT OR INDIRECT DAMAGES, 130 | INCLUDING, WITHOUT LIMITATION, FOR ANY LOSS OF PROFITS, LOSS OF USE, BUSINESS 131 | INTERRUPTION, LOSS OF DATA, COST OF SUBSTITUTE GOODS OR SERVICES, OR FOR ANY 132 | SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, IN CONNECTION WITH 133 | OR ARISING OUT OF THE USE OR INABILITY TO USE THE ELASTIC SOFTWARE, OR THE 134 | PERFORMANCE OF OR FAILURE TO PERFORM THIS AGREEMENT, WHETHER ALLEGED AS A 135 | BREACH OF CONTRACT OR TORTIOUS CONDUCT, INCLUDING NEGLIGENCE, EVEN IF ELASTIC 136 | HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 137 | 138 | 5. MISCELLANEOUS 139 | 140 | This Agreement completely and exclusively states the entire agreement of the 141 | parties regarding the subject matter herein, and it supersedes, and its terms 142 | govern, all prior proposals, agreements, or other communications between the 143 | parties, oral or written, regarding such subject matter. This Agreement may be 144 | modified by Elastic from time to time, and any such modifications will be 145 | effective upon the "Posted Date" set forth at the top of the modified 146 | Agreement. If any provision hereof is held unenforceable, this Agreement will 147 | continue without said provision and be interpreted to reflect the original 148 | intent of the parties. This Agreement and any non-contractual obligation 149 | arising out of or in connection with it, is governed exclusively by Dutch law. 150 | This Agreement shall not be governed by the 1980 UN Convention on Contracts 151 | for the International Sale of Goods. All disputes arising out of or in 152 | connection with this Agreement, including its existence and validity, shall be 153 | resolved by the courts with jurisdiction in Amsterdam, The Netherlands, except 154 | where mandatory law provides for the courts at another location in The 155 | Netherlands to have jurisdiction. The parties hereby irrevocably waive any and 156 | all claims and defenses either might otherwise have in any such action or 157 | proceeding in any of such courts based upon any alleged lack of personal 158 | jurisdiction, improper venue, forum non conveniens or any similar claim or 159 | defense. A breach or threatened breach, by You of Section 2 may cause 160 | irreparable harm for which damages at law may not provide adequate relief, and 161 | therefore Elastic shall be entitled to seek injunctive relief without being 162 | required to post a bond. You may not assign this Agreement (including by 163 | operation of law in connection with a merger or acquisition), in whole or in 164 | part to any third party without the prior written consent of Elastic, which 165 | may be withheld or granted by Elastic in its sole and absolute discretion. 166 | Any assignment in violation of the preceding sentence is void. Notices to 167 | Elastic may also be sent to legal@elastic.co. 168 | 169 | 6. DEFINITIONS 170 | 171 | The following terms have the meanings ascribed: 172 | 173 | 6.1 "Affiliate" means, with respect to a party, any entity that controls, is 174 | controlled by, or which is under common control with, such party, where 175 | "control" means ownership of at least fifty percent (50%) of the outstanding 176 | voting shares of the entity, or the contractual right to establish policy for, 177 | and manage the operations of, the entity. 178 | 179 | 6.2 "Basic Features and Functions" means those features and functions of the 180 | Elastic Software that are eligible for use under a Basic license, as set forth 181 | at https://www.elastic.co/subscriptions, as may be modified by Elastic from 182 | time to time. 183 | 184 | 6.3 "Commercial Software" means the Elastic Software Source Code in any file 185 | containing a header stating the contents are subject to the Elastic License or 186 | which is contained in the repository folder labeled "x-pack", unless a LICENSE 187 | file present in the directory subtree declares a different license. 188 | 189 | 6.4 "Derivative Work of the Commercial Software" means, for purposes of this 190 | Agreement, any modification(s) or enhancement(s) to the Commercial Software, 191 | which represent, as a whole, an original work of authorship. 192 | 193 | 6.5 "License" means a limited, non-exclusive, non-transferable, fully paid up, 194 | royalty free, right and license, without the right to grant or authorize 195 | sublicenses, solely for Your internal business operations to (i) install and 196 | use the applicable Features and Functions of the Elastic Software in Object 197 | Code, and (ii) permit Contractors and Your Affiliates to use the Elastic 198 | software as set forth in (i) above, provided that such use by Contractors must 199 | be solely for Your benefit and/or the benefit of Your Affiliates, and You 200 | shall be responsible for all acts and omissions of such Contractors and 201 | Affiliates in connection with their use of the Elastic software that are 202 | contrary to the terms and conditions of this Agreement. 203 | 204 | 6.6 "License Key" means a sequence of bytes, including but not limited to a 205 | JSON blob, that is used to enable certain features and functions of the 206 | Elastic Software. 207 | 208 | 6.7 "Marks and Notices" means all Elastic trademarks, trade names, logos and 209 | notices present on the Documentation as originally provided by Elastic. 210 | 211 | 6.8 "Non-production Environment" means an environment for development, testing 212 | or quality assurance, where software is not used for production purposes. 213 | 214 | 6.9 "Object Code" means any form resulting from mechanical transformation or 215 | translation of Source Code form, including but not limited to compiled object 216 | code, generated documentation, and conversions to other media types. 217 | 218 | 6.10 "Source Code" means the preferred form of computer software for making 219 | modifications, including but not limited to software source code, 220 | documentation source, and configuration files. 221 | 222 | 6.11 "Subscription" means the right to receive Support Services and a License 223 | to the Commercial Software. 224 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DEPRECATED 2 | 3 | Since migration to Package Storage V2 was completed, this repository is not intended to be used anymore. 4 | More information can be found at https://github.com/elastic/elastic-package/blob/main/docs/howto/use_package_storage_v2.md 5 | 6 | # Package Storage 7 | 8 | This is a storage repository for the packages served through the package registry service. See the [`package-registry` repository](https://github.com/elastic/package-registry) for basic registry API usage and examples. 9 | 10 | The `package-storage` repository contains 3 branches with the packages for the different environments: 11 | 12 | * snapshot 13 | * staging 14 | * production 15 | 16 | Here's how these branches relate to repositories and other aspects of packages. 17 | 18 | | | Snapshot | Staging | Production | 19 | |------------------- |------------------------- |------------------------ |----------------------- | 20 | | URL | [epr-snapshot.elastic.co](https://epr-snapshot.elastic.co/search) | [epr-staging.elastic.co](https://epr-staging.elastic.co/search) | [epr.elastic.co](https://epr.elastic.co/search) | 21 | | How to add a package | Commit to elastic/integrations* | [`elastic-package promote`](https://github.com/elastic/elastic-package#elastic-package-promote) | [`elastic-package promote`](https://github.com/elastic/elastic-package#elastic-package-promote) | 22 | | Allow version overwrite?| yes** | if needed | no | 23 | | Allow version removal? | yes | special exceptions only | only version increments | 24 | | Stack Ver vs Storage Ver| [main branch](https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/epm/registry/registry_url.ts#L24) | all `-SNAPSHOT` Kibana versions | all shipped or BC versions*** | 25 | | Registry Version | Fixed dev or latest stable release | Stable release | Stable release | 26 | | Branch | snapshot | staging | production | 27 | | Packages | snapshot+staging+prod | staging+production | production | 28 | | Release | Manual | Manual | Manual | 29 | | Docker image | snapshot | staging | production | 30 | 31 | `*` https://github.com/elastic/integrations is the development home of most packages, though not all. Promotion process to package storage repository is discussed below. 32 | 33 | ** removing packages can cause short-term dev problems when a package is in use and then is deleted, some manual maintenance is expected and this does not reflect user experience in production. 34 | 35 | *** for example, during the development of 8.0 as main, all 7.x -SNAPSHOT stack versions (including cloud deploys) will use the `staging` repo to facilitate testing prior to package release, while the formal build candidates and shipped stack versions of Kibana are coded to use the `production` registry. To update the package storage branch used in a self managed Kibana deploy you can set the below, for example, in the kibana/config/kibana.yml file: 36 | xpack.fleet.registryUrl: "htttp://epr-snapshot.elastic.co/" 37 | 38 | # Update Package Registry for a distribution 39 | 40 | Each distribution (snapshot, staging, production) uses a specific version of the registry. The registry to be used is referenced in the Dockerfile. To update the package registry version, the referenced must be updated. 41 | 42 | In the Dockerfile there is a reference looking similar to: 43 | 44 | ``` 45 | ARG PACKAGE_REGISTRY=v0.6.0 46 | FROM docker.elastic.co/package-registry/package-registry:${PACKAGE_REGISTRY} 47 | 48 | LABEL package-registry=${PACKAGE_REGISTRY} 49 | ``` 50 | 51 | The above example uses `v0.6.0` as the registry version. Each tagged version in the registry can be used or commits made to package-registry main. A valid registry version would also be `02ca71731cdc092db213cd4c9069db43b74ac01b` as this is a commit hash from main. 52 | 53 | In addition to updating the Dockerfile, also the package-registry version on the Golang side must be updated as it is used for some testing. For this, run the following commands: 54 | 55 | ``` 56 | go get github.com/elastic/package-registry@v0.7.0 57 | mage modTidy 58 | ``` 59 | 60 | This will update the go module files. 61 | 62 | With all the changes above, a pull request can be opened. As soon as the pull request is merged, a new Docker image is automatically built by CI and can be deployed. 63 | 64 | As each branch/distribution references its own version, each branch has to be updated with the above. It is encourage to keep the branches in sync related to registry versions. 65 | 66 | # Pull request titles 67 | 68 | To easily differentiate PRs against snapshot, staging and production, each PR to one of these branches should be prefixed with `[{branch-name}]`, for example `[snapshot]`. This also makes sure if the same package is promoted to all 3 branches, they don't have the exact same names. 69 | 70 | # Package promotion 71 | 72 | A package will usually go through different stages from snapshot to staging to production. It is recommended to use the [`elastic-package` tool](https://github.com/elastic/elastic-package) to achieve package promotion. It is the package developer's burden to fulfill tests on snapshot stage and to coordinate any desired tests on staging prior to promoting to production storage. Note that production is in use by all 7.10+ released versions of Elastic and testing should reflect this to avoid problems. 73 | 74 | So, briefly, to promote a package from one distribution to another, it must first be added to the new distribution and as soon as it is added, removed from the old distribution. As an example, a package `foo-1.2.3` is in `snapshot`. Now the content is copied over to `staging` and as soon as the package is merged, the package `foo-1.2.3` should be removed from `snapshot`. For example see this PR [promoting the `system-0.10.0` package from `snapshot` to `staging`](https://github.com/elastic/package-storage/pull/824) and this corresponding PR [removing the `system-0.10.0` package from `snapshot`](https://github.com/elastic/package-storage/pull/825). 75 | 76 | If the same package version exists in both branches, the first one is taken. In the above case this is `staging`. 77 | 78 | As the staging distribution consists of `production + staging` packages and `snapshot` of `production + staging + snapshot` packages, a package that was promoted to the next stage, is always also available in the previous stage. 79 | 80 | The above implies, that as soon as the packages for `staging` as an example are built and the Docker image is available, also `snapshot` Docker image must be rebuilt as it depends on the `staging` one. 81 | 82 | # Release distribution 83 | 84 | Currently each distribution is released manually but should be release automatically in the future. To release a distribution with a new package, first the building of the Docker image must be completed which is automatic. For `snapshot` this pushes a new image to `docker.elastic.co/package-registry/distribution:snapshot`. At the same time, an image is built for each commit hash, for example `docker.elastic.co/package-registry/distribution:48f3935a72b0c5aacc6fec8ef36d559b089a238b`. The distribution specific images are constantly overwritten, the commit hash images stays as is in case an environment is need that does not change. 85 | 86 | As soon as the Docker image is built and added to the Docker registry, the rollout command for the k8s cluster can be run. For `staging` this looks as following: 87 | 88 | ``` 89 | kubectl rollout restart deployment package-registry-staging-vanilla -n package-registry 90 | ``` 91 | 92 | Each environment has its own deployment. Note: The above command only works if you have access to the specific k8s clusters. 93 | 94 | As soon as the rollout is triggered, it takes up to a minute until all containers are updated and from there, on the CDN side it takes the cache time to update the assets. New packages will be available immediately as they were not served by the CDN previously. 95 | 96 | # Branches 97 | 98 | The current `package-storage` repository has a few branches. This is a quick summary of the use cases of each branch: 99 | 100 | * production: Packages for epr.elastic.co 101 | * staging: Packages for epr-staging.elastic.co 102 | * snapshot: Packages for epr-snapshot.elastic.co 103 | * production-7.9: Packages for epr-7-9.elastic.co. These packages are served for Kibana 7.9 versions. It is expected that all future changes to packages go into the production branch. 104 | * experimental: Packages for epr-experimental.elastic.co. These packages are served for Kibana 7.8 and will disappear in the future. No updates should happen to this branch. 105 | * main: Contains docs and comment scripts for the distribution branches. 106 | 107 | # Tags 108 | 109 | At the time of the Elastic Stack release, a tag off the `production` branch is created with an Elastic Stack release version. This will allow environments like Kibana tests to use a specific "known" tag for their testing. An additional benefit is that in the future if users potentially run the registry on premise, these tags can be used to have the on premise registry aligned with the Elastic Stack version instead of constantly pulling the most recent version. 110 | 111 | These tags trigger the build of the following docker images (7.9.0 as an example): 112 | 113 | ``` 114 | docker.elastic.co/package-registry/distribution:7.9.0 115 | ``` 116 | 117 | The tag does not contain a `v` prefix like most other version tags do. The reason is that this tag is used for the docker image name where the v prefix is not used. 118 | --------------------------------------------------------------------------------