├── .github ├── PULL_REQUEST_TEMPLATE.md ├── dependabot.yml └── workflows │ ├── ci.yml │ └── codeql.yml ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── NOTICE ├── README.md ├── ecr ├── base.go ├── doc.go ├── doc_test.go ├── fake_ecr_client_test.go ├── fetcher.go ├── fetcher_test.go ├── internal │ ├── testdata │ │ ├── digest.go │ │ ├── media_type.go │ │ ├── media_type_docker.go │ │ └── media_type_oci_image.go │ └── util │ │ ├── http │ │ ├── redact.go │ │ └── redact_test.go │ │ └── oci │ │ ├── redact.go │ │ └── redact_test.go ├── layer_writer.go ├── layer_writer_test.go ├── manifest_writer.go ├── manifest_writer_test.go ├── pusher.go ├── pusher_test.go ├── ref.go ├── ref_test.go ├── resolver.go ├── resolver_test.go └── stream │ ├── chunked_processor.go │ └── chunked_processor_test.go ├── example ├── ecr-copy │ └── main.go ├── ecr-pull │ ├── main.go │ └── progress.go └── ecr-push │ ├── main.go │ └── progress.go ├── go.mod └── go.sum /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. 7 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | # Automatic upgrade for Go modules. 4 | - package-ecosystem: "gomod" 5 | directory: "/" 6 | schedule: 7 | interval: "daily" 8 | 9 | # Automatic upgrade for GitHub Actions packages. 10 | - package-ecosystem: "github-actions" 11 | directory: "/" 12 | schedule: 13 | interval: "daily" 14 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | on: [push, pull_request] 2 | 3 | jobs: 4 | build: 5 | runs-on: ubuntu-latest 6 | strategy: 7 | matrix: 8 | go: [ '1.23', '1.24' ] 9 | name: Go ${{ matrix.go }} build 10 | steps: 11 | - uses: actions/checkout@v4 12 | - uses: actions/setup-go@v5 13 | with: 14 | go-version: ${{ matrix.go }} 15 | - name: build 16 | run: make build 17 | - name: test 18 | run: make cover 19 | 20 | lint: 21 | runs-on: ubuntu-latest 22 | name: lint 23 | steps: 24 | - uses: actions/checkout@v4 25 | - uses: actions/setup-go@v5 26 | with: 27 | go-version: '1.24' 28 | - name: golangci-lint 29 | uses: golangci/golangci-lint-action@v6 30 | with: 31 | version: v1.64.7 32 | -------------------------------------------------------------------------------- /.github/workflows/codeql.yml: -------------------------------------------------------------------------------- 1 | name: "CodeQL Scan" 2 | 3 | on: 4 | push: 5 | branches: [ "master" ] 6 | pull_request: 7 | branches: [ "master" ] 8 | schedule: 9 | - cron: '25 21 * * 5' 10 | 11 | jobs: 12 | analyze: 13 | name: Analyze 14 | runs-on: ubuntu-22.04 15 | permissions: 16 | actions: read 17 | contents: read 18 | security-events: write 19 | 20 | strategy: 21 | fail-fast: false 22 | matrix: 23 | language: [ 'go' ] 24 | 25 | steps: 26 | - name: Checkout repository 27 | uses: actions/checkout@v4 28 | 29 | # Initializes the CodeQL tools for scanning. 30 | - name: Initialize CodeQL 31 | uses: github/codeql-action/init@v3 32 | with: 33 | languages: ${{ matrix.language }} 34 | 35 | - name: Build 36 | run: make build 37 | 38 | - name: Perform CodeQL Analysis 39 | uses: github/codeql-action/analyze@v3 40 | with: 41 | category: "/language:${{matrix.language}}" 42 | 43 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /vendor 2 | /bin 3 | /tmp 4 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | 6 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/awslabs/amazon-ecr-containerd-resolver/issues), or [recently closed](https://github.com/awslabs/amazon-ecr-containerd-resolver/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/amazon-ecr-containerd-resolver/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/awslabs/amazon-ecr-containerd-resolver/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"). You 4 | # may not use this file except in compliance with the License. A copy of 5 | # the License is located at 6 | # 7 | # http://aws.amazon.com/apache2.0/ 8 | # 9 | # or in the "license" file accompanying this file. This file is 10 | # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 11 | # ANY KIND, either express or implied. See the License for the specific 12 | # language governing permissions and limitations under the License. 13 | 14 | ROOT := $(shell pwd) 15 | 16 | all: build 17 | 18 | SOURCEDIR=./ 19 | SOURCES := $(shell find $(SOURCEDIR) -name '*.go' | grep -v './vendor') 20 | PULLDIR=$(SOURCEDIR)/example/ecr-pull 21 | PULL_BINARY=$(ROOT)/bin/ecr-pull 22 | PUSHDIR=$(SOURCEDIR)/example/ecr-push 23 | PUSH_BINARY=$(ROOT)/bin/ecr-push 24 | COPYDIR=$(SOURCEDIR)/example/ecr-copy 25 | COPY_BINARY=$(ROOT)/bin/ecr-copy 26 | 27 | export GO111MODULE=on 28 | 29 | .PHONY: build 30 | build: $(PULL_BINARY) $(PUSH_BINARY) $(COPY_BINARY) 31 | 32 | $(PULL_BINARY): $(SOURCES) 33 | cd $(PULLDIR) && go build -o $(PULL_BINARY) . 34 | 35 | $(PUSH_BINARY): $(SOURCES) 36 | cd $(PUSHDIR) && go build -o $(PUSH_BINARY) . 37 | 38 | $(COPY_BINARY): $(SOURCES) 39 | cd $(COPYDIR) && go build -o $(COPY_BINARY) . 40 | 41 | .PHONY: test 42 | test: $(SOURCES) 43 | go test -race -v $(shell go list ./... | grep -v '/vendor/') 44 | 45 | .PHONY: cover 46 | cover: $(SOURCES) 47 | mkdir -p tmp 48 | go test -race -coverprofile=tmp/coverage.out -v $(shell go list ./... | grep -v '/vendor/') 49 | go tool cover -html=tmp/coverage.out -o tmp/coverage.html 50 | @echo Code coverage report is generated as tmp/coverage.html 51 | 52 | .PHONY: clean 53 | clean: 54 | @rm $(PULL_BINARY) ||: 55 | @rm $(PUSH_BINARY) ||: 56 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Amazon ECR containerd resolver 2 | Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Amazon ECR containerd resolver 2 | 3 | [![.github/workflows/ci.yml](https://github.com/awslabs/amazon-ecr-containerd-resolver/actions/workflows/ci.yml/badge.svg)](https://github.com/awslabs/amazon-ecr-containerd-resolver/actions/workflows/ci.yml) 4 | [![CodeQL Scan](https://github.com/awslabs/amazon-ecr-containerd-resolver/actions/workflows/codeql.yml/badge.svg)](https://github.com/awslabs/amazon-ecr-containerd-resolver/actions/workflows/codeql.yml) 5 | [![Go Report Card](https://goreportcard.com/badge/github.com/awslabs/amazon-ecr-containerd-resolver)](https://goreportcard.com/report/github.com/awslabs/amazon-ecr-containerd-resolver) 6 | 7 | The Amazon ECR containerd resolver is an implementation of a 8 | [containerd](https://github.com/containerd/containerd) 9 | `Resolver`, `Fetcher`, and `Pusher` that can pull images from Amazon ECR and 10 | push images to Amazon ECR using the Amazon ECR API instead of the Docker 11 | Registry API. 12 | 13 | > *Note:* This repository is a proof-of-concept and is not recommended for 14 | > production use. 15 | 16 | ## Usage 17 | 18 | ### Pull images 19 | ```go 20 | resolver, _ := ecr.NewResolver() 21 | img, err := client.Pull( 22 | namespaces.NamespaceFromEnv(context.TODO()), 23 | "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/myrepository:mytag", 24 | containerd.WithResolver(resolver), 25 | containerd.WithPullUnpack, 26 | containerd.WithSchema1Conversion) 27 | ``` 28 | 29 | ### Push images 30 | ```go 31 | ctx := namespaces.NamespaceFromEnv(context.TODO()) 32 | 33 | img, _ := client.ImageService().Get( 34 | ctx, 35 | "docker.io/library/busybox:latest") 36 | resolver, _ := ecr.NewResolver() 37 | err = client.Push( 38 | ctx, 39 | "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/myrepository:mytag", 40 | img.Target, 41 | containerd.WithResolver(resolver)) 42 | ``` 43 | 44 | Two small example programs are provided in the [example](example) 45 | directory demonstrating how to use the resolver with containerd. 46 | 47 | ### `ref` 48 | 49 | containerd specifies images with a `ref`. `ref`s are different from Docker 50 | image names, as `ref`s intend to encode an identifier, but not a retrieval 51 | mechanism. `ref`s start with a DNS-style namespace that can be used to select 52 | separate `Resolver`s to use. 53 | 54 | The canonical `ref` format used by the amazon-ecr-containerd-resolver is 55 | `ecr.aws/` followed by the ARN of the repository and a label and/or a digest. 56 | 57 | ### Parallel downloads 58 | 59 | This resolver supports request parallelization for individual layers. This 60 | takes advantage of HTTP [range requests](https://tools.ietf.org/html/rfc7233) to 61 | download different parts of the same file in parallel. This is an approach to 62 | achieving higher throughput when [downloading from Amazon 63 | S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/optimizing-performance-design-patterns.html#optimizing-performance-parallelization), 64 | which provides the raw blob storage for layers in Amazon ECR. 65 | 66 | Request parallelization is not enabled by default, and the default Go HTTP 67 | client is used instead. To enable request parallelization, you can use the 68 | `WithLayerDownloadParallelism` resolver option to set the amount of 69 | parallelization per layer. 70 | 71 | When enabled, the layer will be divided into equal-sized chunks (except for the 72 | last chunk) and downloaded with the set amount of parallelism. The chunks range 73 | in size from 1 MiB to 20 MiB; anything smaller than 1 MiB will not be 74 | parallelized and anything larger than 20 MiB * *parallelism* will use a larger 75 | number of chunks (though only with the specified amount of parallelism). 76 | 77 | Initial testing suggests that a parallelism setting of `4` results in 3x faster 78 | layer downloads, but increases the amount of memory consumption between 15-20x. 79 | Further testing is still needed. 80 | 81 | This support is backed by the [htcat library](https://github.com/htcat/htcat). 82 | 83 | ## Building 84 | 85 | The Amazon ECR containerd resolver manages its dependencies with [Go modules](https://github.com/golang/go/wiki/Modules) and requires Go 1.23 or greater. 86 | If you have Go 1.23 or greater installed, you can build the example programs with `make`. 87 | 88 | ## License 89 | 90 | The Amazon ECR containerd resolver is licensed under the Apache 2.0 License. 91 | -------------------------------------------------------------------------------- /ecr/base.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "errors" 21 | 22 | "github.com/aws/aws-sdk-go/aws" 23 | "github.com/aws/aws-sdk-go/aws/request" 24 | "github.com/aws/aws-sdk-go/service/ecr" 25 | "github.com/containerd/containerd/images" 26 | "github.com/containerd/containerd/log" 27 | "github.com/containerd/containerd/reference" 28 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 29 | ) 30 | 31 | var ( 32 | errImageNotFound = errors.New("ecr: image not found") 33 | errGetImageUnhandled = errors.New("ecr: unable to get images") 34 | 35 | // supportedImageMediaTypes lists supported content types for images. 36 | supportedImageMediaTypes = []string{ 37 | ocispec.MediaTypeImageIndex, 38 | ocispec.MediaTypeImageManifest, 39 | images.MediaTypeDockerSchema2Manifest, 40 | images.MediaTypeDockerSchema2ManifestList, 41 | images.MediaTypeDockerSchema1Manifest, 42 | } 43 | ) 44 | 45 | type ecrBase struct { 46 | client ecrAPI 47 | ecrSpec ECRSpec 48 | } 49 | 50 | // ecrAPI contains only the ECR APIs that are called by the resolver 51 | // See https://docs.aws.amazon.com/sdk-for-go/api/service/ecr/ecriface/ for the 52 | // full interface from the SDK. 53 | type ecrAPI interface { 54 | BatchGetImageWithContext(aws.Context, *ecr.BatchGetImageInput, ...request.Option) (*ecr.BatchGetImageOutput, error) 55 | GetDownloadUrlForLayerWithContext(aws.Context, *ecr.GetDownloadUrlForLayerInput, ...request.Option) (*ecr.GetDownloadUrlForLayerOutput, error) 56 | BatchCheckLayerAvailabilityWithContext(aws.Context, *ecr.BatchCheckLayerAvailabilityInput, ...request.Option) (*ecr.BatchCheckLayerAvailabilityOutput, error) 57 | InitiateLayerUpload(*ecr.InitiateLayerUploadInput) (*ecr.InitiateLayerUploadOutput, error) 58 | UploadLayerPart(*ecr.UploadLayerPartInput) (*ecr.UploadLayerPartOutput, error) 59 | CompleteLayerUpload(*ecr.CompleteLayerUploadInput) (*ecr.CompleteLayerUploadOutput, error) 60 | PutImageWithContext(aws.Context, *ecr.PutImageInput, ...request.Option) (*ecr.PutImageOutput, error) 61 | } 62 | 63 | // getImage fetches the reference's image from ECR. 64 | func (b *ecrBase) getImage(ctx context.Context) (*ecr.Image, error) { 65 | return b.runGetImage(ctx, ecr.BatchGetImageInput{ 66 | ImageIds: []*ecr.ImageIdentifier{b.ecrSpec.ImageID()}, 67 | AcceptedMediaTypes: aws.StringSlice(supportedImageMediaTypes), 68 | }) 69 | } 70 | 71 | // getImageByDescriptor retrieves an image from ECR for a given OCI descriptor. 72 | func (b *ecrBase) getImageByDescriptor(ctx context.Context, desc ocispec.Descriptor) (*ecr.Image, error) { 73 | // If the reference includes both a digest & tag for an image and that 74 | // digest matches the descriptor's digest then both are specified when 75 | // requesting an image from ECR. Mutation of the image that pushes an image 76 | // with a new digest to the tag, will cause the query to fail as the 77 | // combination of tag AND digest does not match this modified tag. 78 | // 79 | // This stronger matching works well for repositories using immutable tags; 80 | // in the case of immutable tags, a ref like 81 | // ecr.aws/arn:aws:ecr:us-west-2:111111111111:repository/example-name:tag-name@sha256:$digest 82 | // would necessarily refer to the same image unless tag-name is deleted and 83 | // recreated with an different image. 84 | // 85 | // Consumers wanting to use a strong reference without assuming immutable 86 | // tags should instead provide refs that specify digests, excluding its 87 | // corresponding tag. 88 | // 89 | // See the ECR docs on image tag mutability for details: 90 | // 91 | // https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html 92 | // 93 | ident := &ecr.ImageIdentifier{ImageDigest: aws.String(desc.Digest.String())} 94 | if b.ecrSpec.Spec().Digest() == desc.Digest { 95 | if tag, _ := b.ecrSpec.TagDigest(); tag != "" { 96 | ident.ImageTag = aws.String(tag) 97 | } 98 | } 99 | 100 | input := ecr.BatchGetImageInput{ 101 | ImageIds: []*ecr.ImageIdentifier{ident}, 102 | } 103 | 104 | // Request exact mediaType when known. 105 | if desc.MediaType != "" { 106 | input.AcceptedMediaTypes = []*string{aws.String(desc.MediaType)} 107 | } else { 108 | input.AcceptedMediaTypes = aws.StringSlice(supportedImageMediaTypes) 109 | } 110 | 111 | return b.runGetImage(ctx, input) 112 | } 113 | 114 | // runGetImage submits and handles the response for requests of ECR images. 115 | func (b *ecrBase) runGetImage(ctx context.Context, batchGetImageInput ecr.BatchGetImageInput) (*ecr.Image, error) { 116 | // Allow only a single image to be fetched at a time. 117 | if len(batchGetImageInput.ImageIds) != 1 { 118 | return nil, errGetImageUnhandled 119 | } 120 | 121 | batchGetImageInput.RegistryId = aws.String(b.ecrSpec.Registry()) 122 | batchGetImageInput.RepositoryName = aws.String(b.ecrSpec.Repository) 123 | 124 | log.G(ctx).WithField("batchGetImageInput", batchGetImageInput).Trace("ecr.base.image: requesting images") 125 | 126 | batchGetImageOutput, err := b.client.BatchGetImageWithContext(ctx, &batchGetImageInput) 127 | if err != nil { 128 | log.G(ctx).WithError(err).Error("ecr.base.image: failed to get image") 129 | return nil, err 130 | } 131 | log.G(ctx).WithField("batchGetImageOutput", batchGetImageOutput).Trace("ecr.base.image: api response") 132 | 133 | // Summarize image request failures for handled errors. Only the first 134 | // failure is checked as only a single ImageIdentifier is allowed to be 135 | // queried for. 136 | if len(batchGetImageOutput.Failures) > 0 { 137 | failure := batchGetImageOutput.Failures[0] 138 | switch aws.StringValue(failure.FailureCode) { 139 | // Requested image with a corresponding tag and digest does not exist. 140 | // This failure will generally occur when pushing an updated (or new) 141 | // image with a tag. 142 | case ecr.ImageFailureCodeImageTagDoesNotMatchDigest: 143 | log.G(ctx).WithField("failure", failure).Debug("ecr.base.image: no matching image with specified digest") 144 | return nil, errImageNotFound 145 | // Requested image doesn't resolve to a known image. A new image will 146 | // result in an ImageNotFound error when checked before push. 147 | case ecr.ImageFailureCodeImageNotFound: 148 | log.G(ctx).WithField("failure", failure).Debug("ecr.base.image: no image found") 149 | return nil, errImageNotFound 150 | // Requested image identifiers are invalid. 151 | case ecr.ImageFailureCodeInvalidImageDigest, ecr.ImageFailureCodeInvalidImageTag: 152 | log.G(ctx).WithField("failure", failure).Error("ecr.base.image: invalid image identifier") 153 | return nil, reference.ErrInvalid 154 | // Unhandled failure reported for image request made. 155 | default: 156 | log.G(ctx).WithField("failure", failure).Warn("ecr.base.image: unhandled image request failure") 157 | return nil, errGetImageUnhandled 158 | } 159 | } 160 | 161 | return batchGetImageOutput.Images[0], nil 162 | } 163 | -------------------------------------------------------------------------------- /ecr/doc.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | // Package ecr provides implementations of the containerd Resolver, Fetcher, and 17 | // Pusher interfaces that can use the Amazon ECR API to push and pull images. 18 | // 19 | // References 20 | // 21 | // containerd specifies images with a reference, or a "ref". References are 22 | // different from Docker image names, as references encode an identifier, but 23 | // not a retrieval mechanism. refs start with a DNS-style namespace that can be 24 | // used to select separate Resolvers to use. 25 | // 26 | // The canonical ref format used by this package is ecr.aws/ followed by the ARN 27 | // of the repository and a label and/or a digest. Valid references are of the 28 | // form "ecr.aws/arn:aws:ecr:::repository/:". 29 | // 30 | // License 31 | // 32 | // This package is licensed under the Apache 2.0 license. 33 | package ecr 34 | -------------------------------------------------------------------------------- /ecr/doc_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr_test 17 | 18 | import ( 19 | "context" 20 | "fmt" 21 | 22 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr" 23 | "github.com/containerd/containerd" 24 | "github.com/containerd/containerd/namespaces" 25 | ) 26 | 27 | func ExampleNewResolver_pull() { 28 | client, _ := containerd.New("/run/containerd/containerd.sock") 29 | resolver, _ := ecr.NewResolver() 30 | img, _ := client.Pull( 31 | namespaces.NamespaceFromEnv(context.TODO()), 32 | "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/myrepository:mytag", 33 | containerd.WithResolver(resolver), 34 | containerd.WithPullUnpack, 35 | containerd.WithSchema1Conversion) 36 | fmt.Println(img.Name()) 37 | } 38 | 39 | func ExampleNewResolver_push() { 40 | client, _ := containerd.New("/run/containerd/containerd.sock") 41 | ctx := namespaces.NamespaceFromEnv(context.TODO()) 42 | 43 | img, _ := client.ImageService().Get( 44 | ctx, 45 | "docker.io/library/busybox:latest") 46 | resolver, _ := ecr.NewResolver() 47 | _ = client.Push( 48 | ctx, 49 | "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/myrepository:mytag", 50 | img.Target, 51 | containerd.WithResolver(resolver)) 52 | } 53 | -------------------------------------------------------------------------------- /ecr/fake_ecr_client_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "github.com/aws/aws-sdk-go/aws" 20 | "github.com/aws/aws-sdk-go/aws/request" 21 | "github.com/aws/aws-sdk-go/service/ecr" 22 | ) 23 | 24 | // fakeECRClient is a fake that can be used for testing the ecrAPI interface. 25 | // Each method is backed by a function contained in the struct. Nil functions 26 | // will cause panics when invoked. 27 | type fakeECRClient struct { 28 | BatchGetImageFn func(aws.Context, *ecr.BatchGetImageInput, ...request.Option) (*ecr.BatchGetImageOutput, error) 29 | GetDownloadUrlForLayerFn func(aws.Context, *ecr.GetDownloadUrlForLayerInput, ...request.Option) (*ecr.GetDownloadUrlForLayerOutput, error) 30 | BatchCheckLayerAvailabilityFn func(aws.Context, *ecr.BatchCheckLayerAvailabilityInput, ...request.Option) (*ecr.BatchCheckLayerAvailabilityOutput, error) 31 | InitiateLayerUploadFn func(*ecr.InitiateLayerUploadInput) (*ecr.InitiateLayerUploadOutput, error) 32 | UploadLayerPartFn func(*ecr.UploadLayerPartInput) (*ecr.UploadLayerPartOutput, error) 33 | CompleteLayerUploadFn func(*ecr.CompleteLayerUploadInput) (*ecr.CompleteLayerUploadOutput, error) 34 | PutImageFn func(aws.Context, *ecr.PutImageInput, ...request.Option) (*ecr.PutImageOutput, error) 35 | } 36 | 37 | var _ ecrAPI = (*fakeECRClient)(nil) 38 | 39 | func (f *fakeECRClient) BatchGetImageWithContext(ctx aws.Context, arg *ecr.BatchGetImageInput, opts ...request.Option) (*ecr.BatchGetImageOutput, error) { 40 | return f.BatchGetImageFn(ctx, arg, opts...) 41 | } 42 | 43 | func (f *fakeECRClient) GetDownloadUrlForLayerWithContext(ctx aws.Context, arg *ecr.GetDownloadUrlForLayerInput, opts ...request.Option) (*ecr.GetDownloadUrlForLayerOutput, error) { 44 | return f.GetDownloadUrlForLayerFn(ctx, arg, opts...) 45 | } 46 | 47 | func (f *fakeECRClient) BatchCheckLayerAvailabilityWithContext(ctx aws.Context, arg *ecr.BatchCheckLayerAvailabilityInput, opts ...request.Option) (*ecr.BatchCheckLayerAvailabilityOutput, error) { 48 | return f.BatchCheckLayerAvailabilityFn(ctx, arg, opts...) 49 | } 50 | 51 | func (f *fakeECRClient) InitiateLayerUpload(arg *ecr.InitiateLayerUploadInput) (*ecr.InitiateLayerUploadOutput, error) { 52 | return f.InitiateLayerUploadFn(arg) 53 | } 54 | 55 | func (f *fakeECRClient) UploadLayerPart(arg *ecr.UploadLayerPartInput) (*ecr.UploadLayerPartOutput, error) { 56 | return f.UploadLayerPartFn(arg) 57 | } 58 | 59 | func (f *fakeECRClient) CompleteLayerUpload(arg *ecr.CompleteLayerUploadInput) (*ecr.CompleteLayerUploadOutput, error) { 60 | return f.CompleteLayerUploadFn(arg) 61 | } 62 | 63 | func (f *fakeECRClient) PutImageWithContext(ctx aws.Context, arg *ecr.PutImageInput, opts ...request.Option) (*ecr.PutImageOutput, error) { 64 | return f.PutImageFn(ctx, arg, opts...) 65 | } 66 | -------------------------------------------------------------------------------- /ecr/fetcher.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "bytes" 20 | "context" 21 | "errors" 22 | "fmt" 23 | "io" 24 | "net/http" 25 | "net/url" 26 | "strings" 27 | 28 | "github.com/aws/aws-sdk-go/aws" 29 | "github.com/aws/aws-sdk-go/service/ecr" 30 | httputil "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/util/http" 31 | ociutil "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/util/oci" 32 | "github.com/containerd/containerd/errdefs" 33 | "github.com/containerd/containerd/images" 34 | "github.com/containerd/containerd/log" 35 | "github.com/containerd/containerd/remotes" 36 | "github.com/htcat/htcat" 37 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 38 | "golang.org/x/net/context/ctxhttp" 39 | ) 40 | 41 | // ecrFetcher implements the containerd remotes.Fetcher interface and can be 42 | // used to pull images from Amazon ECR. 43 | type ecrFetcher struct { 44 | ecrBase 45 | parallelism int 46 | httpClient *http.Client 47 | } 48 | 49 | var _ remotes.Fetcher = (*ecrFetcher)(nil) 50 | 51 | func (f *ecrFetcher) Fetch(ctx context.Context, desc ocispec.Descriptor) (io.ReadCloser, error) { 52 | ctx = log.WithLogger(ctx, log.G(ctx).WithField("desc", ociutil.RedactDescriptor(desc))) 53 | log.G(ctx).Debug("ecr.fetch") 54 | 55 | // need to do different things based on the media type 56 | switch desc.MediaType { 57 | case 58 | images.MediaTypeDockerSchema1Manifest, 59 | images.MediaTypeDockerSchema2Manifest, 60 | images.MediaTypeDockerSchema2ManifestList, 61 | ocispec.MediaTypeImageIndex, 62 | ocispec.MediaTypeImageManifest: 63 | return f.fetchManifest(ctx, desc) 64 | case 65 | images.MediaTypeDockerSchema2Layer, 66 | images.MediaTypeDockerSchema2LayerGzip, 67 | images.MediaTypeDockerSchema2Config, 68 | ocispec.MediaTypeImageLayerGzip, 69 | ocispec.MediaTypeImageLayerZstd, 70 | ocispec.MediaTypeImageLayer, 71 | ocispec.MediaTypeImageConfig: 72 | return f.fetchLayer(ctx, desc) 73 | case 74 | images.MediaTypeDockerSchema2LayerForeign, 75 | images.MediaTypeDockerSchema2LayerForeignGzip: 76 | return f.fetchForeignLayer(ctx, desc) 77 | default: 78 | log.G(ctx). 79 | WithField("media type", desc.MediaType). 80 | Error("ecr.fetcher: unimplemented media type") 81 | return nil, unimplemented 82 | } 83 | } 84 | 85 | func (f *ecrFetcher) fetchManifest(ctx context.Context, desc ocispec.Descriptor) (io.ReadCloser, error) { 86 | var ( 87 | image *ecr.Image 88 | err error 89 | ) 90 | // A digest is required to fetch by digest alone. When a digest is not 91 | // provided the fetch is based on the parsed ECR resource - specifying both 92 | // a digest and tag in the request if possible. 93 | if desc.Digest == "" { 94 | log.G(ctx).Debug("ecr.fetcher.manifest: fetch image by tag") 95 | image, err = f.getImage(ctx) 96 | } else { 97 | log.G(ctx).Debug("ecr.fetcher.manifest: fetch image by digest") 98 | image, err = f.getImageByDescriptor(ctx, desc) 99 | } 100 | if err != nil { 101 | return nil, err 102 | } 103 | if image == nil { 104 | return nil, errors.New("fetchManifest: nil image") 105 | } 106 | 107 | return io.NopCloser(bytes.NewReader([]byte(aws.StringValue(image.ImageManifest)))), nil 108 | } 109 | 110 | func (f *ecrFetcher) fetchLayer(ctx context.Context, desc ocispec.Descriptor) (io.ReadCloser, error) { 111 | log.G(ctx).Debug("ecr.fetcher.layer") 112 | getDownloadUrlForLayerInput := &ecr.GetDownloadUrlForLayerInput{ 113 | RegistryId: aws.String(f.ecrSpec.Registry()), 114 | RepositoryName: aws.String(f.ecrSpec.Repository), 115 | LayerDigest: aws.String(desc.Digest.String()), 116 | } 117 | output, err := f.client.GetDownloadUrlForLayerWithContext(ctx, getDownloadUrlForLayerInput) 118 | if err != nil { 119 | return nil, err 120 | } 121 | 122 | downloadURL := aws.StringValue(output.DownloadUrl) 123 | ctx = log.WithLogger(ctx, log.G(ctx).WithField("url", httputil.RedactHTTPQueryValuesFromURL(downloadURL))) 124 | if f.parallelism > 0 { 125 | return f.fetchLayerHtcat(ctx, desc, downloadURL) 126 | } 127 | return f.fetchLayerURL(ctx, desc, downloadURL) 128 | } 129 | 130 | func (f *ecrFetcher) fetchForeignLayer(ctx context.Context, desc ocispec.Descriptor) (io.ReadCloser, error) { 131 | log.G(ctx).Debug("ecr.fetcher.layer.foreign") 132 | if len(desc.URLs) < 1 { 133 | log.G(ctx).Error("cannot pull foreign layer without URL") 134 | } 135 | var err error 136 | for _, layerURL := range desc.URLs { 137 | redactedDownloadURL := httputil.RedactHTTPQueryValuesFromURL(layerURL) 138 | log.G(ctx).WithField("url", redactedDownloadURL).Debug("ecr.fetcher.layer.foreign: fetching from URL") 139 | var rdc io.ReadCloser 140 | rdc, err = f.fetchLayerURL(ctx, desc, layerURL) 141 | if err == nil { 142 | return rdc, nil 143 | } 144 | log.G(ctx).WithField("url", redactedDownloadURL).WithError(err).Warn("ecr.fetcher.layer.foreign: unable to fetch from URL") 145 | } 146 | return nil, err 147 | } 148 | 149 | func (f *ecrFetcher) fetchLayerURL(ctx context.Context, desc ocispec.Descriptor, downloadURL string) (io.ReadCloser, error) { 150 | req, err := http.NewRequest(http.MethodGet, downloadURL, nil) 151 | if err != nil { 152 | log.G(ctx). 153 | WithError(err). 154 | Error("ecr.fetcher.layer.url: failed to create HTTP request") 155 | return nil, err 156 | } 157 | log.G(ctx).Debug("ecr.fetcher.layer.url") 158 | 159 | req.Header.Set("Accept", strings.Join([]string{desc.MediaType, `*`}, ", ")) 160 | resp, err := f.doRequest(ctx, req) 161 | if err != nil { 162 | return nil, err 163 | } 164 | if resp.StatusCode > 299 { 165 | resp.Body.Close() 166 | redactedDownloadURL := httputil.RedactHTTPQueryValuesFromURL(downloadURL) 167 | if resp.StatusCode == http.StatusNotFound { 168 | return nil, fmt.Errorf("content at %v not found: %w", redactedDownloadURL, errdefs.ErrNotFound) 169 | } 170 | return nil, fmt.Errorf("ecr.fetcher.layer.url: unexpected status code %v: %v", redactedDownloadURL, resp.Status) 171 | } 172 | log.G(ctx).Debug("ecr.fetcher.layer.url: returning body") 173 | return resp.Body, nil 174 | } 175 | 176 | func (f *ecrFetcher) doRequest(ctx context.Context, req *http.Request) (*http.Response, error) { 177 | client := f.httpClient 178 | resp, err := ctxhttp.Do(ctx, client, req) 179 | if err != nil { 180 | return nil, fmt.Errorf("failed to do request: %w", httputil.RedactHTTPQueryValuesFromURLError(err)) 181 | } 182 | return resp, nil 183 | } 184 | 185 | func (f *ecrFetcher) fetchLayerHtcat(ctx context.Context, desc ocispec.Descriptor, downloadURL string) (io.ReadCloser, error) { 186 | log.G(ctx).Debug("ecr.fetcher.layer.htcat") 187 | parsedURL, err := url.Parse(downloadURL) 188 | if err != nil { 189 | log.G(ctx). 190 | WithError(err). 191 | Error("ecr.fetcher.layer.htcat: failed to parse URL") 192 | return nil, err 193 | } 194 | hc := f.httpClient 195 | if hc == nil { 196 | hc = http.DefaultClient 197 | } 198 | htc := htcat.New(hc, parsedURL, f.parallelism) 199 | pr, pw := io.Pipe() 200 | go func() { 201 | defer pw.Close() 202 | _, err := htc.WriteTo(pw) 203 | if err != nil { 204 | log.G(ctx). 205 | WithError(httputil.RedactHTTPQueryValuesFromURLError(err)). 206 | Error("ecr.fetcher.layer.htcat: failed to download layer") 207 | } 208 | }() 209 | return pr, nil 210 | } 211 | -------------------------------------------------------------------------------- /ecr/fetcher_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "bytes" 20 | "context" 21 | "crypto/rand" 22 | "errors" 23 | "fmt" 24 | "io" 25 | "net/http" 26 | "net/http/httptest" 27 | "testing" 28 | "time" 29 | 30 | "github.com/aws/aws-sdk-go/aws" 31 | "github.com/aws/aws-sdk-go/aws/arn" 32 | "github.com/aws/aws-sdk-go/aws/request" 33 | "github.com/aws/aws-sdk-go/service/ecr" 34 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/testdata" 35 | "github.com/containerd/containerd/errdefs" 36 | "github.com/containerd/containerd/images" 37 | "github.com/opencontainers/go-digest" 38 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 39 | "github.com/stretchr/testify/assert" 40 | "github.com/stretchr/testify/require" 41 | ) 42 | 43 | func TestFetchUnimplemented(t *testing.T) { 44 | fetcher := &ecrFetcher{} 45 | desc := ocispec.Descriptor{ 46 | MediaType: "never-implemented", 47 | } 48 | _, err := fetcher.Fetch(context.Background(), desc) 49 | assert.EqualError(t, err, unimplemented.Error()) 50 | } 51 | 52 | func TestFetchForeignLayer(t *testing.T) { 53 | // setup 54 | const expectedBody = "hello, this is dog" 55 | 56 | fetcher := &ecrFetcher{} 57 | 58 | // test both media types 59 | for _, mediaType := range []string{ 60 | images.MediaTypeDockerSchema2LayerForeign, 61 | images.MediaTypeDockerSchema2LayerForeignGzip, 62 | } { 63 | t.Run(mediaType, func(t *testing.T) { 64 | requests := 0 65 | ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 66 | requests++ 67 | if r.URL.Path == "/missing" { 68 | w.WriteHeader(http.StatusNotFound) 69 | return 70 | } 71 | fmt.Fprint(w, expectedBody) 72 | })) 73 | defer ts.Close() 74 | 75 | // input 76 | desc := ocispec.Descriptor{ 77 | MediaType: mediaType, 78 | URLs: []string{ 79 | ts.URL + "/missing", 80 | ts.URL + "/ok", 81 | }, 82 | } 83 | 84 | reader, err := fetcher.Fetch(context.Background(), desc) 85 | require.NoError(t, err, "fetch should succeed from test server") 86 | defer reader.Close() 87 | 88 | output, err := io.ReadAll(reader) 89 | assert.NoError(t, err, "should have a valid byte buffer") 90 | assert.Equal(t, expectedBody, string(output)) 91 | 92 | assert.Equal(t, requests, len(desc.URLs), "should have tried all URLs until success") 93 | }) 94 | } 95 | } 96 | 97 | func TestFetchForeignLayerNotFound(t *testing.T) { 98 | ts := httptest.NewServer(http.NotFoundHandler()) 99 | defer ts.Close() 100 | 101 | fetcher := &ecrFetcher{} 102 | mediaType := images.MediaTypeDockerSchema2LayerForeignGzip 103 | 104 | desc := ocispec.Descriptor{ 105 | MediaType: mediaType, 106 | URLs: []string{ts.URL}, 107 | } 108 | 109 | _, err := fetcher.Fetch(context.Background(), desc) 110 | assert.Error(t, err) 111 | assert.True(t, errors.Is(err, errdefs.ErrNotFound)) 112 | } 113 | 114 | func TestFetchManifest(t *testing.T) { 115 | const ( 116 | registry = "registry" 117 | repository = "repository" 118 | imageManifest = "image manifest" 119 | imageDigest = "sha256:18019fb68413973fcde9ff917d333bbaa228c4aaebba9ad0ca5ffec26e4f3541" 120 | imageTag = "tag" 121 | imageTagDigest = "tag@" + imageDigest 122 | ) 123 | 124 | // Test all supported media types 125 | for _, mediaType := range supportedImageMediaTypes { 126 | // Test variants of Object (tag, digest, and combination). 127 | for _, testObject := range []struct { 128 | ImageIdentifier ecr.ImageIdentifier 129 | Object string 130 | }{ 131 | // Tag alone - used on first get image. 132 | {Object: imageTag, ImageIdentifier: ecr.ImageIdentifier{ImageTag: aws.String(imageTag)}}, 133 | // Tag and digest assertive fetch 134 | {Object: imageTagDigest, ImageIdentifier: ecr.ImageIdentifier{ImageTag: aws.String(imageTag), ImageDigest: aws.String(imageDigest)}}, 135 | // Digest fetch 136 | {Object: "@" + imageDigest, ImageIdentifier: ecr.ImageIdentifier{ImageDigest: aws.String(imageDigest)}}, 137 | } { 138 | fakeClient := &fakeECRClient{} 139 | fetcher := &ecrFetcher{ 140 | ecrBase: ecrBase{ 141 | client: fakeClient, 142 | ecrSpec: ECRSpec{ 143 | arn: arn.ARN{ 144 | AccountID: registry, 145 | }, 146 | Repository: repository, 147 | Object: testObject.Object, 148 | }, 149 | }, 150 | } 151 | desc := ocispec.Descriptor{ 152 | MediaType: mediaType, 153 | } 154 | if testObject.ImageIdentifier.ImageDigest != nil { 155 | desc.Digest = digest.Digest(aws.StringValue(testObject.ImageIdentifier.ImageDigest)) 156 | } 157 | 158 | t.Run(mediaType+"_"+testObject.Object, func(t *testing.T) { 159 | callCount := 0 160 | fakeClient.BatchGetImageFn = func(_ aws.Context, input *ecr.BatchGetImageInput, _ ...request.Option) (*ecr.BatchGetImageOutput, error) { 161 | callCount++ 162 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 163 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 164 | 165 | assert.ElementsMatch(t, []*ecr.ImageIdentifier{&testObject.ImageIdentifier}, input.ImageIds) 166 | 167 | // Fetching populated descriptors uses a narrower requested 168 | // content type. 169 | requestedTypes := aws.StringValueSlice(input.AcceptedMediaTypes) 170 | t.Logf("requestedTypes: %q", requestedTypes) 171 | if testObject.ImageIdentifier.ImageDigest != nil { 172 | expectedTypes := []string{desc.MediaType} 173 | t.Logf("expectedTypes: %q", expectedTypes) 174 | assert.Equal(t, expectedTypes, requestedTypes, 175 | "mediaType should match the descriptor") 176 | } else { 177 | expectedTypes := supportedImageMediaTypes 178 | t.Logf("expectedTypes: %q", expectedTypes) 179 | assert.Equal(t, expectedTypes, requestedTypes, 180 | "mediaType should allow any supported type") 181 | } 182 | 183 | return &ecr.BatchGetImageOutput{ 184 | Images: []*ecr.Image{{ImageManifest: aws.String(imageManifest)}}, 185 | }, nil 186 | } 187 | 188 | reader, err := fetcher.Fetch(context.Background(), desc) 189 | require.NoError(t, err, "fetch") 190 | defer reader.Close() 191 | assert.Equal(t, 1, callCount, "BatchGetImage should be called once") 192 | manifest, err := io.ReadAll(reader) 193 | require.NoError(t, err, "reading manifest") 194 | assert.Equal(t, imageManifest, string(manifest)) 195 | }) 196 | } 197 | } 198 | } 199 | 200 | func TestFetchManifestAPIError(t *testing.T) { 201 | ref := "ecr.aws/arn:aws:ecr:fake:123456789012:repository/foo/bar:latest" 202 | mediaType := ocispec.MediaTypeImageManifest 203 | 204 | fakeClient := &fakeECRClient{ 205 | BatchGetImageFn: func(aws.Context, *ecr.BatchGetImageInput, ...request.Option) (*ecr.BatchGetImageOutput, error) { 206 | return nil, errors.New("expected") 207 | }, 208 | } 209 | resolver := &ecrResolver{ 210 | clients: map[string]ecrAPI{ 211 | "fake": fakeClient, 212 | }, 213 | } 214 | fetcher, err := resolver.Fetcher(context.Background(), ref) 215 | require.NoError(t, err, "failed to create fetcher") 216 | _, err = fetcher.Fetch(context.Background(), ocispec.Descriptor{MediaType: mediaType}) 217 | assert.EqualError(t, err, "expected") 218 | } 219 | 220 | func TestFetchManifestNotFound(t *testing.T) { 221 | ref := "ecr.aws/arn:aws:ecr:fake:123456789012:repository/foo/bar:latest" 222 | mediaType := ocispec.MediaTypeImageManifest 223 | 224 | fakeClient := &fakeECRClient{ 225 | BatchGetImageFn: func(aws.Context, *ecr.BatchGetImageInput, ...request.Option) (*ecr.BatchGetImageOutput, error) { 226 | return &ecr.BatchGetImageOutput{ 227 | Failures: []*ecr.ImageFailure{ 228 | {FailureCode: aws.String(ecr.ImageFailureCodeImageNotFound)}, 229 | }, 230 | }, nil 231 | }, 232 | } 233 | resolver := &ecrResolver{ 234 | clients: map[string]ecrAPI{ 235 | "fake": fakeClient, 236 | }, 237 | } 238 | fetcher, err := resolver.Fetcher(context.Background(), ref) 239 | require.NoError(t, err, "failed to create fetcher") 240 | _, err = fetcher.Fetch(context.Background(), ocispec.Descriptor{MediaType: mediaType}) 241 | assert.Error(t, err) 242 | } 243 | 244 | func TestFetchLayer(t *testing.T) { 245 | registry := "registry" 246 | repository := "repository" 247 | layerDigest := testdata.InsignificantDigest.String() 248 | fakeClient := &fakeECRClient{} 249 | fetcher := &ecrFetcher{ 250 | ecrBase: ecrBase{ 251 | client: fakeClient, 252 | ecrSpec: ECRSpec{ 253 | arn: arn.ARN{ 254 | AccountID: registry, 255 | }, 256 | Repository: repository, 257 | }, 258 | }, 259 | } 260 | expectedBody := "hello this is dog" 261 | ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 262 | fmt.Fprint(w, expectedBody) 263 | })) 264 | defer ts.Close() 265 | 266 | // test all supported media types 267 | for _, mediaType := range []string{ 268 | images.MediaTypeDockerSchema2Layer, 269 | images.MediaTypeDockerSchema2LayerGzip, 270 | images.MediaTypeDockerSchema2Config, 271 | ocispec.MediaTypeImageLayerGzip, 272 | ocispec.MediaTypeImageLayerZstd, 273 | ocispec.MediaTypeImageLayer, 274 | ocispec.MediaTypeImageConfig, 275 | } { 276 | t.Run(mediaType, func(t *testing.T) { 277 | callCount := 0 278 | fakeClient.GetDownloadUrlForLayerFn = func(_ aws.Context, input *ecr.GetDownloadUrlForLayerInput, _ ...request.Option) (*ecr.GetDownloadUrlForLayerOutput, error) { 279 | callCount++ 280 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 281 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 282 | assert.Equal(t, layerDigest, aws.StringValue(input.LayerDigest)) 283 | return &ecr.GetDownloadUrlForLayerOutput{DownloadUrl: aws.String(ts.URL)}, nil 284 | } 285 | desc := ocispec.Descriptor{ 286 | MediaType: mediaType, 287 | Digest: digest.Digest(layerDigest), 288 | } 289 | reader, err := fetcher.Fetch(context.Background(), desc) 290 | assert.NoError(t, err, "fetch") 291 | defer reader.Close() 292 | assert.Equal(t, 1, callCount, "GetDownloadURLForLayer should be called once") 293 | body, err := io.ReadAll(reader) 294 | assert.NoError(t, err, "reading body") 295 | assert.Equal(t, expectedBody, string(body)) 296 | }) 297 | } 298 | } 299 | 300 | func TestFetchLayerAPIError(t *testing.T) { 301 | fakeClient := &fakeECRClient{ 302 | GetDownloadUrlForLayerFn: func(aws.Context, *ecr.GetDownloadUrlForLayerInput, ...request.Option) (*ecr.GetDownloadUrlForLayerOutput, error) { 303 | return nil, errors.New("expected") 304 | }, 305 | } 306 | fetcher := &ecrFetcher{ 307 | ecrBase: ecrBase{ 308 | client: fakeClient, 309 | }, 310 | } 311 | desc := ocispec.Descriptor{ 312 | MediaType: ocispec.MediaTypeImageLayerGzip, 313 | } 314 | _, err := fetcher.Fetch(context.Background(), desc) 315 | assert.Error(t, err) 316 | } 317 | 318 | func TestFetchLayerHtcat(t *testing.T) { 319 | registry := "registry" 320 | repository := "repository" 321 | layerDigest := testdata.InsignificantDigest.String() 322 | fakeClient := &fakeECRClient{} 323 | fetcher := &ecrFetcher{ 324 | ecrBase: ecrBase{ 325 | client: fakeClient, 326 | ecrSpec: ECRSpec{ 327 | arn: arn.ARN{ 328 | AccountID: registry, 329 | }, 330 | Repository: repository, 331 | }, 332 | }, 333 | parallelism: 2, 334 | } 335 | // need >1mb of content for htcat to do parallel requests 336 | const ( 337 | kB = 1024 * 1 338 | mB = 1024 * kB 339 | ) 340 | expectedBody := make([]byte, 30*mB) 341 | _, err := rand.Read(expectedBody) 342 | assert.NoError(t, err) 343 | handlerCallCount := 0 344 | ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 345 | handlerCallCount++ 346 | http.ServeContent(w, r, "", time.Now(), bytes.NewReader(expectedBody)) 347 | })) 348 | defer ts.Close() 349 | 350 | downloadURLCallCount := 0 351 | fakeClient.GetDownloadUrlForLayerFn = func(_ aws.Context, input *ecr.GetDownloadUrlForLayerInput, _ ...request.Option) (*ecr.GetDownloadUrlForLayerOutput, error) { 352 | downloadURLCallCount++ 353 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 354 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 355 | assert.Equal(t, layerDigest, aws.StringValue(input.LayerDigest)) 356 | return &ecr.GetDownloadUrlForLayerOutput{DownloadUrl: aws.String(ts.URL)}, nil 357 | } 358 | desc := ocispec.Descriptor{ 359 | MediaType: images.MediaTypeDockerSchema2Layer, 360 | Digest: digest.Digest(layerDigest), 361 | } 362 | reader, err := fetcher.Fetch(context.Background(), desc) 363 | assert.NoError(t, err, "fetch") 364 | defer reader.Close() 365 | assert.Equal(t, 1, downloadURLCallCount, "GetDownloadURLForLayer should be called once") 366 | body, err := io.ReadAll(reader) 367 | assert.NoError(t, err, "reading body") 368 | assert.Equal(t, expectedBody, body) 369 | assert.True(t, handlerCallCount > 1, "ServeContent should be called more than once: %d", handlerCallCount) 370 | } 371 | -------------------------------------------------------------------------------- /ecr/internal/testdata/digest.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | import "github.com/opencontainers/go-digest" 4 | 5 | const ( 6 | // InsignificantDigest is an arbitrary value for consistent, placeholder use 7 | // cases in tests. 8 | InsignificantDigest digest.Digest = "insignificant-digest" 9 | // LayerDigest is used for consistent, placeholder layer digests in tests. 10 | LayerDigest digest.Digest = "layer-digest" 11 | // ImageDigest is used for consistent, placeholder image digests in tests. 12 | ImageDigest digest.Digest = "image-digest" 13 | ) 14 | -------------------------------------------------------------------------------- /ecr/internal/testdata/media_type.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | import ( 4 | "encoding/json" 5 | "strings" 6 | ) 7 | 8 | type MediaTypeSample interface { 9 | MediaType() string 10 | Content() string 11 | } 12 | 13 | // MediaTypeSample provides a sample document for a given mediaType. 14 | type mediaTypeSample struct { 15 | mediaType string 16 | content string 17 | } 18 | 19 | // MediaType is the defined sample's actual mediaType. 20 | func (s *mediaTypeSample) MediaType() string { 21 | return s.mediaType 22 | } 23 | 24 | // Content provides the sample's JSON data as a string. 25 | func (s *mediaTypeSample) Content() string { 26 | return strings.TrimSpace(s.content) 27 | } 28 | 29 | // EmptySample is an edge case sample, use 30 | var EmptySample MediaTypeSample = &mediaTypeSample{ 31 | mediaType: "", 32 | content: `{}`, 33 | } 34 | 35 | func WithMediaTypeRemoved(src MediaTypeSample) MediaTypeSample { 36 | m := map[string]interface{}{} 37 | err := json.Unmarshal([]byte(src.Content()), &m) 38 | if err != nil { 39 | return src 40 | } 41 | if _, ok := m["mediaType"]; ok { 42 | return src 43 | } 44 | data, err := json.MarshalIndent(m, "", " ") 45 | if err != nil { 46 | panic(err) 47 | } 48 | return &mediaTypeSample{ 49 | mediaType: src.MediaType(), 50 | content: string(data), 51 | } 52 | } 53 | -------------------------------------------------------------------------------- /ecr/internal/testdata/media_type_docker.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | import "github.com/containerd/containerd/images" 4 | 5 | // DockerSchema2Manifest provides a Docker v2 schema 2 manifest document. 6 | // 7 | // Modified sample (mediaType has been removed); the original is hosted in 8 | // https://github.com/docker/distribution and is licensed under Apache 2.0. 9 | // 10 | // https://github.com/docker/distribution/blob/742aab907b54a367e1ac7033fb9fe73b0e7344f5/docs/spec/manifest-v2-2.md#example-image-manifest 11 | // 12 | var DockerSchema2Manifest MediaTypeSample = &mediaTypeSample{ 13 | mediaType: images.MediaTypeDockerSchema2Manifest, 14 | content: ` 15 | { 16 | "schemaVersion": 2, 17 | "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 18 | "config": { 19 | "mediaType": "application/vnd.docker.container.image.v1+json", 20 | "size": 7023, 21 | "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7" 22 | }, 23 | "layers": [ 24 | { 25 | "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 26 | "size": 32654, 27 | "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f" 28 | }, 29 | { 30 | "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 31 | "size": 16724, 32 | "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" 33 | }, 34 | { 35 | "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 36 | "size": 73109, 37 | "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736" 38 | } 39 | ] 40 | } 41 | `, 42 | } 43 | 44 | // DockerSchema2ManifestList provides a Docker v2 schema 2 manifest list 45 | // document, with elements present in its manifest list. 46 | // 47 | // Modified sample (mediaType has been removed); the original is hosted in 48 | // https://github.com/docker/distribution and is licensed under Apache 2.0. 49 | // 50 | // https://github.com/docker/distribution/blob/742aab907b54a367e1ac7033fb9fe73b0e7344f5/docs/spec/manifest-v2-2.md#example-manifest-list 51 | // 52 | var DockerSchema2ManifestList = &mediaTypeSample{ 53 | mediaType: images.MediaTypeDockerSchema2ManifestList, 54 | content: ` 55 | { 56 | "schemaVersion": 2, 57 | "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", 58 | "manifests": [ 59 | { 60 | "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 61 | "size": 7143, 62 | "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f", 63 | "platform": { 64 | "architecture": "ppc64le", 65 | "os": "linux" 66 | } 67 | }, 68 | { 69 | "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 70 | "size": 7682, 71 | "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270", 72 | "platform": { 73 | "architecture": "amd64", 74 | "os": "linux", 75 | "features": [ 76 | "sse4" 77 | ] 78 | } 79 | } 80 | ] 81 | } 82 | `, 83 | } 84 | 85 | // DockerSchema1ManifestUnsigned provides a Docker v2 schema 1 manifest 86 | // document, without signatures. 87 | // 88 | // Modified sample (mediaType and signatures have been removed); the original is 89 | // hosted in https://github.com/docker/distribution and is licensed under Apache 90 | // 2.0. 91 | // 92 | // https://github.com/docker/distribution/blob/742aab907b54a367e1ac7033fb9fe73b0e7344f5/docs/spec/manifest-v2-1.md#example-manifest 93 | // 94 | var DockerSchema1ManifestUnsigned = &mediaTypeSample{ 95 | mediaType: "application/vnd.docker.distribution.manifest.v1+json", 96 | content: ` 97 | { 98 | "name": "hello-world", 99 | "tag": "latest", 100 | "architecture": "amd64", 101 | "fsLayers": [ 102 | { 103 | "blobSum": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 104 | }, 105 | { 106 | "blobSum": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 107 | }, 108 | { 109 | "blobSum": "sha256:cc8567d70002e957612902a8e985ea129d831ebe04057d88fb644857caa45d11" 110 | }, 111 | { 112 | "blobSum": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 113 | } 114 | ], 115 | "history": [ 116 | { 117 | "v1Compatibility": "{\"id\":\"e45a5af57b00862e5ef5782a9925979a02ba2b12dff832fd0991335f4a11e5c5\",\"parent\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"created\":\"2014-12-31T22:57:59.178729048Z\",\"container\":\"27b45f8fb11795b52e9605b686159729b0d9ca92f76d40fb4f05a62e19c46b4f\",\"container_config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) CMD [/hello]\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"docker_version\":\"1.4.1\",\"config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/hello\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"architecture\":\"amd64\",\"os\":\"linux\",\"Size\":0}\n" 118 | }, 119 | { 120 | "v1Compatibility": "{\"id\":\"e45a5af57b00862e5ef5782a9925979a02ba2b12dff832fd0991335f4a11e5c5\",\"parent\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"created\":\"2014-12-31T22:57:59.178729048Z\",\"container\":\"27b45f8fb11795b52e9605b686159729b0d9ca92f76d40fb4f05a62e19c46b4f\",\"container_config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) CMD [/hello]\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"docker_version\":\"1.4.1\",\"config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/hello\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"architecture\":\"amd64\",\"os\":\"linux\",\"Size\":0}\n" 121 | } 122 | ], 123 | "schemaVersion": 1 124 | } 125 | `, 126 | } 127 | 128 | // DockerSchema1Manifest provides a Docker v2 schema 1 manifest document. 129 | // 130 | // Modified sample (mediaType has been removed); the original is hosted in 131 | // https://github.com/docker/distribution and is licensed under Apache 2.0. 132 | // 133 | // https://github.com/docker/distribution/blob/742aab907b54a367e1ac7033fb9fe73b0e7344f5/docs/spec/manifest-v2-1.md#example-manifest 134 | // 135 | var DockerSchema1Manifest MediaTypeSample = &mediaTypeSample{ 136 | mediaType: images.MediaTypeDockerSchema1Manifest, 137 | content: ` 138 | { 139 | "name": "hello-world", 140 | "tag": "latest", 141 | "architecture": "amd64", 142 | "fsLayers": [ 143 | { 144 | "blobSum": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 145 | }, 146 | { 147 | "blobSum": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 148 | }, 149 | { 150 | "blobSum": "sha256:cc8567d70002e957612902a8e985ea129d831ebe04057d88fb644857caa45d11" 151 | }, 152 | { 153 | "blobSum": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 154 | } 155 | ], 156 | "history": [ 157 | { 158 | "v1Compatibility": "{\"id\":\"e45a5af57b00862e5ef5782a9925979a02ba2b12dff832fd0991335f4a11e5c5\",\"parent\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"created\":\"2014-12-31T22:57:59.178729048Z\",\"container\":\"27b45f8fb11795b52e9605b686159729b0d9ca92f76d40fb4f05a62e19c46b4f\",\"container_config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) CMD [/hello]\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"docker_version\":\"1.4.1\",\"config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/hello\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"architecture\":\"amd64\",\"os\":\"linux\",\"Size\":0}\n" 159 | }, 160 | { 161 | "v1Compatibility": "{\"id\":\"e45a5af57b00862e5ef5782a9925979a02ba2b12dff832fd0991335f4a11e5c5\",\"parent\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"created\":\"2014-12-31T22:57:59.178729048Z\",\"container\":\"27b45f8fb11795b52e9605b686159729b0d9ca92f76d40fb4f05a62e19c46b4f\",\"container_config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) CMD [/hello]\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"docker_version\":\"1.4.1\",\"config\":{\"Hostname\":\"8ce6509d66e2\",\"Domainname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"Cpuset\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"PortSpecs\":null,\"ExposedPorts\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/hello\"],\"Image\":\"31cbccb51277105ba3ae35ce33c22b69c9e3f1002e76e4c736a2e8ebff9d7b5d\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"NetworkDisabled\":false,\"MacAddress\":\"\",\"OnBuild\":[],\"SecurityOpt\":null,\"Labels\":null},\"architecture\":\"amd64\",\"os\":\"linux\",\"Size\":0}\n" 162 | } 163 | ], 164 | "schemaVersion": 1, 165 | "signatures": [ 166 | { 167 | "header": { 168 | "jwk": { 169 | "crv": "P-256", 170 | "kid": "OD6I:6DRK:JXEJ:KBM4:255X:NSAA:MUSF:E4VM:ZI6W:CUN2:L4Z6:LSF4", 171 | "kty": "EC", 172 | "x": "3gAwX48IQ5oaYQAYSxor6rYYc_6yjuLCjtQ9LUakg4A", 173 | "y": "t72ge6kIA1XOjqjVoEOiPPAURltJFBMGDSQvEGVB010" 174 | }, 175 | "alg": "ES256" 176 | }, 177 | "signature": "XREm0L8WNn27Ga_iE_vRnTxVMhhYY0Zst_FfkKopg6gWSoTOZTuW4rK0fg_IqnKkEKlbD83tD46LKEGi5aIVFg", 178 | "protected": "eyJmb3JtYXRMZW5ndGgiOjY2MjgsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNS0wNC0wOFQxODo1Mjo1OVoifQ" 179 | } 180 | ] 181 | } 182 | `, 183 | } 184 | -------------------------------------------------------------------------------- /ecr/internal/testdata/media_type_oci_image.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | import ( 4 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 5 | ) 6 | 7 | var OCIImageManifest MediaTypeSample = &mediaTypeSample{ 8 | mediaType: ocispec.MediaTypeImageManifest, 9 | content: ` 10 | { 11 | "schemaVersion": 2, 12 | "mediaType": "application/vnd.oci.image.manifest.v1+json", 13 | "config": { 14 | "mediaType": "application/vnd.oci.image.config.v1+json", 15 | "digest": "sha256:a6ff6fb34ad5a20c2b2371013918a9f0e033a77460b2f17a4041e02bd3d252d0", 16 | "size": 302 17 | }, 18 | "layers": [ 19 | { 20 | "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", 21 | "digest": "sha256:55e3debf4607c47ff150940897a656ec79859f7aa715f26ab4357065e2e20535", 22 | "size": 62599745 23 | } 24 | ] 25 | } 26 | `, 27 | } 28 | 29 | var OCIImageIndex MediaTypeSample = &mediaTypeSample{ 30 | mediaType: ocispec.MediaTypeImageIndex, 31 | content: ` 32 | { 33 | "schemaVersion": 2, 34 | "mediaType": "application/vnd.oci.image.index.v1+json", 35 | "manifests": [ 36 | { 37 | "mediaType": "application/vnd.oci.image.manifest.v1+json", 38 | "size": 3231, 39 | "digest": "sha256:babb154b919b9ad7d38786f71f9c8a3614f6d017b0ba7cada4801ceed7b2220d", 40 | "platform": { 41 | "architecture": "amd64", 42 | "os": "linux" 43 | } 44 | }, 45 | { 46 | "mediaType": "application/vnd.oci.image.manifest.v1+json", 47 | "size": 3231, 48 | "digest": "sha256:718441d735e6a7c9b24837c779cc7112995289eff976a308695a1936bc20b67b", 49 | "platform": { 50 | "architecture": "arm64", 51 | "os": "linux" 52 | } 53 | } 54 | ] 55 | } 56 | `, 57 | } 58 | -------------------------------------------------------------------------------- /ecr/internal/util/http/redact.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package http 17 | 18 | import ( 19 | "errors" 20 | "net/url" 21 | ) 22 | 23 | // RedactHTTPQueryValuesFromURLError is a log utility to parse an error as a URL 24 | // error and redact HTTP query values to prevent leaking sensitive information 25 | // like encoded credentials or tokens. 26 | func RedactHTTPQueryValuesFromURLError(err error) error { 27 | var urlErr *url.Error 28 | 29 | if err != nil && errors.As(err, &urlErr) { 30 | urlErr.URL = RedactHTTPQueryValuesFromURL(urlErr.URL) 31 | return urlErr 32 | } 33 | 34 | return err 35 | } 36 | 37 | // RedactHTTPQueryValuesFromURL is a log utility to parse a raw URL as a URL 38 | // and redact HTTP query values to prevent leaking sensitive information 39 | // like encoded credentials or tokens. 40 | func RedactHTTPQueryValuesFromURL(rawURL string) string { 41 | url, urlParseErr := url.Parse(rawURL) 42 | if urlParseErr == nil && url != nil { 43 | if query := url.Query(); len(query) > 0 { 44 | for k := range query { 45 | query.Set(k, "redacted") 46 | } 47 | url.RawQuery = query.Encode() 48 | } 49 | return url.Redacted() 50 | } 51 | return rawURL 52 | } 53 | -------------------------------------------------------------------------------- /ecr/internal/util/http/redact_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package http 17 | 18 | import ( 19 | "errors" 20 | "net/url" 21 | "strings" 22 | "testing" 23 | ) 24 | 25 | const ( 26 | // mockURL is a fake URL modeling ecr resolver fetching content from S3. 27 | mockURL = "https://s3.us-east-1.amazonaws.com/981ebdad55863b3631dce86a228a3ea230dc87673a06a7d216b1275d4dd707c9/12d7153d7eee2fd595a25e5378384f1ae4b6a1658298a54c5bd3f951ec50b7cb" 28 | 29 | // mockQuery is a fake HTTP query with sensitive information which should be redacted. 30 | mockQuery = "?username=admin&password=admin" 31 | 32 | // redactedQuery is the expected result of redacting mockQuery. 33 | // The query values will be sorted by key as a side-effect of encoding the URL query string back into the URL. 34 | // See https://pkg.go.dev/net/url#Values.Encode 35 | redactedQuery = "?password=redacted&username=redacted" 36 | ) 37 | 38 | func TestRedactHTTPQueryValuesFromURLError(t *testing.T) { 39 | testCases := []struct { 40 | Name string 41 | Description string 42 | Err error 43 | Assert func(*testing.T, error) 44 | }{ 45 | { 46 | Name: "NilError", 47 | Description: "Utility should handle nil error gracefully", 48 | Err: nil, 49 | Assert: func(t *testing.T, actual error) { 50 | if actual != nil { 51 | t.Fatalf("Expected nil error, got '%v'", actual) 52 | } 53 | }, 54 | }, 55 | { 56 | Name: "NonURLError", 57 | Description: "Utility should not modify an error if error is not a URL error", 58 | Err: errors.New("this error is not a URL error"), 59 | Assert: func(t *testing.T, actual error) { 60 | const expected = "this error is not a URL error" 61 | if strings.Compare(expected, actual.Error()) != 0 { 62 | t.Fatalf("Expected '%s', got '%v'", expected, actual) 63 | } 64 | }, 65 | }, 66 | { 67 | Name: "ErrorWithNoHTTPQuery", 68 | Description: "Utility should not modify an error if no HTTP queries are present.", 69 | Err: &url.Error{ 70 | Op: "GET", 71 | URL: mockURL, 72 | Err: errors.New("connect: connection refused"), 73 | }, 74 | Assert: func(t *testing.T, actual error) { 75 | const expected = "GET \"" + mockURL + "\": connect: connection refused" 76 | if strings.Compare(expected, actual.Error()) != 0 { 77 | t.Fatalf("Expected '%s', got '%v'", expected, actual) 78 | } 79 | }, 80 | }, 81 | { 82 | Name: "ErrorWithHTTPQuery", 83 | Description: "Utility should redact HTTP query values in errors to prevent logging sensitive information.", 84 | Err: &url.Error{ 85 | Op: "GET", 86 | URL: mockURL + mockQuery, 87 | Err: errors.New("connect: connection refused"), 88 | }, 89 | Assert: func(t *testing.T, actual error) { 90 | const expected = "GET \"" + mockURL + redactedQuery + "\": connect: connection refused" 91 | if strings.Compare(expected, actual.Error()) != 0 { 92 | t.Fatalf("Expected '%s', got '%v'", expected, actual) 93 | } 94 | }, 95 | }, 96 | } 97 | 98 | for _, testCase := range testCases { 99 | t.Run(testCase.Name, func(t *testing.T) { 100 | actual := RedactHTTPQueryValuesFromURLError(testCase.Err) 101 | testCase.Assert(t, actual) 102 | }) 103 | } 104 | } 105 | 106 | func TestRedactHTTPQueryValuesFromURL(t *testing.T) { 107 | testCases := []struct { 108 | Name string 109 | Description string 110 | URL string 111 | Expected string 112 | }{ 113 | { 114 | Name: "EmptyURL", 115 | Description: "Utility should gracefully handle an empty URL input", 116 | URL: "", 117 | Expected: "", 118 | }, 119 | { 120 | Name: "ValidURLWithoutQuery", 121 | Description: "Utility should not modify a valid URL with no HTTP query", 122 | URL: mockURL, 123 | Expected: mockURL, 124 | }, 125 | { 126 | Name: "ValidURLWithQuery", 127 | Description: "Utility should redact HTTP query values", 128 | URL: mockURL + mockQuery, 129 | Expected: mockURL + redactedQuery, 130 | }, 131 | } 132 | 133 | for _, testCase := range testCases { 134 | t.Run(testCase.Name, func(t *testing.T) { 135 | actual := RedactHTTPQueryValuesFromURL(testCase.URL) 136 | if strings.Compare(testCase.Expected, actual) != 0 { 137 | t.Fatalf("Expected '%s', got '%s'", testCase.Expected, actual) 138 | } 139 | }) 140 | } 141 | } 142 | -------------------------------------------------------------------------------- /ecr/internal/util/oci/redact.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package oci 17 | 18 | import ( 19 | httputil "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/util/http" 20 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 21 | ) 22 | 23 | // RedactDescriptor returns a copy of the provided image descriptor 24 | // with its URLs redacted. 25 | func RedactDescriptor(desc ocispec.Descriptor) ocispec.Descriptor { 26 | for i, url := range desc.URLs { 27 | desc.URLs[i] = httputil.RedactHTTPQueryValuesFromURL(url) 28 | } 29 | return desc 30 | } 31 | -------------------------------------------------------------------------------- /ecr/internal/util/oci/redact_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package oci 17 | 18 | import ( 19 | "testing" 20 | 21 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 22 | ) 23 | 24 | func TestRedactDescriptor(t *testing.T) { 25 | testCases := []struct { 26 | Name string 27 | Description string 28 | Descriptor ocispec.Descriptor 29 | Assert func(*testing.T, ocispec.Descriptor) 30 | }{ 31 | { 32 | Name: "RedactDescriptorEmptyURLs", 33 | Description: "Utility should make a descriptor copy with no URLs", 34 | Descriptor: ocispec.Descriptor{ 35 | URLs: []string{}, 36 | }, 37 | Assert: func(t *testing.T, actual ocispec.Descriptor) { 38 | if len(actual.URLs) != 0 { 39 | t.Fatalf("Expected length of 0, got length %d", len(actual.URLs)) 40 | } 41 | }, 42 | }, 43 | { 44 | Name: "RedactDescriptorURLs", 45 | Description: "Utility should make a descriptor copy with redacted URLs", 46 | Descriptor: ocispec.Descriptor{ 47 | URLs: []string{ 48 | "s3.amazon.com/foo/bar?token=12345", 49 | "s3.amazon.com/foo/baz?username=admin&password=admin", 50 | }, 51 | }, 52 | Assert: func(t *testing.T, actual ocispec.Descriptor) { 53 | if len(actual.URLs) != 2 { 54 | t.Fatalf("Expected length of 2, got length %d", len(actual.URLs)) 55 | } 56 | const expectedURL1 = "s3.amazon.com/foo/bar?token=redacted" 57 | if actual.URLs[0] != expectedURL1 { 58 | t.Fatalf("Expected %s; got %s", expectedURL1, actual.URLs[0]) 59 | } 60 | const expectedURL2 = "s3.amazon.com/foo/baz?password=redacted&username=redacted" 61 | if actual.URLs[1] != expectedURL2 { 62 | t.Fatalf("Expected %s; got %s", expectedURL2, actual.URLs[1]) 63 | } 64 | }, 65 | }, 66 | } 67 | 68 | for _, testCase := range testCases { 69 | t.Run(testCase.Name, func(t *testing.T) { 70 | actual := RedactDescriptor(testCase.Descriptor) 71 | testCase.Assert(t, actual) 72 | }) 73 | } 74 | } 75 | -------------------------------------------------------------------------------- /ecr/layer_writer.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "errors" 21 | "io" 22 | "strings" 23 | "time" 24 | 25 | "github.com/aws/aws-sdk-go/aws" 26 | "github.com/aws/aws-sdk-go/aws/awserr" 27 | "github.com/aws/aws-sdk-go/service/ecr" 28 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/stream" 29 | "github.com/containerd/containerd/content" 30 | "github.com/containerd/containerd/log" 31 | "github.com/containerd/containerd/remotes/docker" 32 | "github.com/opencontainers/go-digest" 33 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 34 | ) 35 | 36 | type layerWriter struct { 37 | ctx context.Context 38 | base *ecrBase 39 | desc ocispec.Descriptor 40 | buf io.WriteCloser 41 | tracker docker.StatusTracker 42 | ref string 43 | uploadID string 44 | err chan error 45 | } 46 | 47 | var _ content.Writer = (*layerWriter)(nil) 48 | 49 | const ( 50 | layerQueueSize = 5 51 | ) 52 | 53 | func newLayerWriter(base *ecrBase, tracker docker.StatusTracker, ref string, desc ocispec.Descriptor) (content.Writer, error) { 54 | ctx, cancel := context.WithCancel(context.Background()) 55 | ctx = log.WithLogger(ctx, log.G(ctx).WithField("desc", desc)) 56 | reader, writer := io.Pipe() 57 | lw := &layerWriter{ 58 | ctx: ctx, 59 | base: base, 60 | desc: desc, 61 | buf: writer, 62 | tracker: tracker, 63 | ref: ref, 64 | err: make(chan error), 65 | } 66 | 67 | // call InitiateLayerUpload and get upload ID 68 | initiateLayerUploadInput := &ecr.InitiateLayerUploadInput{ 69 | RegistryId: aws.String(base.ecrSpec.Registry()), 70 | RepositoryName: aws.String(base.ecrSpec.Repository), 71 | } 72 | initiateLayerUploadOutput, err := base.client.InitiateLayerUpload(initiateLayerUploadInput) 73 | if err != nil { 74 | cancel() 75 | return nil, err 76 | } 77 | lw.uploadID = aws.StringValue(initiateLayerUploadOutput.UploadId) 78 | partSize := aws.Int64Value(initiateLayerUploadOutput.PartSize) 79 | log.G(ctx). 80 | WithField("digest", desc.Digest.String()). 81 | WithField("uploadID", lw.uploadID). 82 | WithField("partSize", partSize). 83 | Debug("ecr.blob.init") 84 | 85 | go func() { 86 | defer cancel() 87 | defer close(lw.err) 88 | _, err := stream.ChunkedProcessor(reader, partSize, layerQueueSize, 89 | func(layerChunk *stream.Chunk) error { 90 | begin := layerChunk.BytesBegin 91 | end := layerChunk.BytesEnd 92 | bytesRead := end - begin 93 | log.G(ctx). 94 | WithField("digest", desc.Digest.String()). 95 | WithField("part", layerChunk.Part). 96 | WithField("begin", begin). 97 | WithField("end", end). 98 | WithField("bytes", bytesRead). 99 | Debug("ecr.layer.callback") 100 | 101 | uploadLayerPartInput := &ecr.UploadLayerPartInput{ 102 | RegistryId: aws.String(base.ecrSpec.Registry()), 103 | RepositoryName: aws.String(base.ecrSpec.Repository), 104 | UploadId: aws.String(lw.uploadID), 105 | PartFirstByte: aws.Int64(begin), 106 | PartLastByte: aws.Int64(end), 107 | LayerPartBlob: layerChunk.Bytes, 108 | } 109 | 110 | _, err := base.client.UploadLayerPart(uploadLayerPartInput) 111 | log.G(ctx). 112 | WithField("digest", desc.Digest.String()). 113 | WithField("part", layerChunk.Part). 114 | WithField("begin", begin). 115 | WithField("end", end). 116 | WithField("bytes", bytesRead). 117 | Debug("ecr.layer.callback end") 118 | if err == nil { 119 | var status docker.Status 120 | status, err = lw.tracker.GetStatus(lw.ref) 121 | if err == nil { 122 | status.Offset += int64(bytesRead) + 1 123 | status.UpdatedAt = time.Now() 124 | lw.tracker.SetStatus(lw.ref, status) 125 | } 126 | } 127 | return err 128 | }) 129 | if err != nil { 130 | lw.err <- err 131 | } 132 | log.G(ctx).WithField("digest", desc.Digest.String()).Debug("ecr.layer upload done") 133 | }() 134 | return lw, nil 135 | } 136 | 137 | func (lw *layerWriter) Write(b []byte) (int, error) { 138 | log.G(lw.ctx).WithField("len(b)", len(b)).Debug("ecr.layer.write") 139 | select { 140 | case err := <-lw.err: 141 | return 0, err 142 | case <-lw.ctx.Done(): 143 | return 0, errors.New("lw.Write: closed") 144 | default: 145 | } 146 | return lw.buf.Write(b) 147 | } 148 | 149 | func (lw *layerWriter) Close() error { 150 | return errors.New("lw.Close: not implemented") 151 | } 152 | 153 | func (lw *layerWriter) Digest() digest.Digest { 154 | return lw.desc.Digest 155 | } 156 | 157 | func (lw *layerWriter) Commit(ctx context.Context, size int64, expected digest.Digest, opts ...content.Opt) error { 158 | log.G(lw.ctx).WithField("size", size).WithField("expected", expected).Debug("ecr.layer.commit") 159 | lw.buf.Close() 160 | select { 161 | case err := <-lw.err: 162 | if err != nil { 163 | log.G(lw.ctx). 164 | WithError(err). 165 | WithField("expected", expected). 166 | Error("ecr.layer.commit: error while uploading parts") 167 | return err 168 | } 169 | case <-lw.ctx.Done(): 170 | } 171 | 172 | completeLayerUploadInput := &ecr.CompleteLayerUploadInput{ 173 | RegistryId: aws.String(lw.base.ecrSpec.Registry()), 174 | RepositoryName: aws.String(lw.base.ecrSpec.Repository), 175 | UploadId: aws.String(lw.uploadID), 176 | LayerDigests: []*string{aws.String(expected.String())}, 177 | } 178 | 179 | completeLayerUploadOutput, err := lw.base.client.CompleteLayerUpload(completeLayerUploadInput) 180 | if err != nil { 181 | // If the layer that is being uploaded already exists then return successfully instead of failing. Unfortunately 182 | // in this case we do not get the digest back from ECR, but if the client-provided digest starts with a 183 | // "sha256:" then the ECR has validated that the digest provided matches ours. If the expected digest uses a 184 | // different algorithm we have to fail as we do not know the digest ECR calculated and the expected digest 185 | // has not been validated. 186 | awsErr, ok := err.(awserr.Error) 187 | if ok && awsErr.Code() == "LayerAlreadyExistsException" && strings.HasPrefix(expected.String(), "sha256:") { 188 | log.G(lw.ctx).Debug("ecr.layer.commit: layer already exists") 189 | return nil 190 | } else { 191 | return err 192 | } 193 | } 194 | actualDigest := aws.StringValue(completeLayerUploadOutput.LayerDigest) 195 | if actualDigest != expected.String() { 196 | return errors.New("ecr: failed to validate uploaded digest") 197 | } 198 | log.G(ctx). 199 | WithField("expected", expected). 200 | WithField("actual", actualDigest). 201 | Debug("ecr.layer.commit: complete") 202 | return nil 203 | } 204 | 205 | func (lw *layerWriter) Status() (content.Status, error) { 206 | log.G(lw.ctx).Debug("ecr.layer.status") 207 | 208 | return content.Status{ 209 | Ref: lw.desc.Digest.String(), 210 | }, nil 211 | } 212 | 213 | func (lw *layerWriter) Truncate(size int64) error { 214 | log.G(lw.ctx).WithField("size", size).Debug("ecr.layer.truncate") 215 | 216 | return errors.New("ecr.layer.truncate: not implemented") 217 | } 218 | -------------------------------------------------------------------------------- /ecr/layer_writer_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "io" 21 | "testing" 22 | 23 | "github.com/aws/aws-sdk-go/aws" 24 | "github.com/aws/aws-sdk-go/aws/arn" 25 | "github.com/aws/aws-sdk-go/aws/awserr" 26 | "github.com/aws/aws-sdk-go/service/ecr" 27 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/testdata" 28 | "github.com/containerd/containerd/remotes/docker" 29 | "github.com/opencontainers/go-digest" 30 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 31 | "github.com/stretchr/testify/assert" 32 | "github.com/stretchr/testify/require" 33 | ) 34 | 35 | func TestLayerWriter(t *testing.T) { 36 | registry := "registry" 37 | repository := "repository" 38 | layerData := "layer" 39 | layerDigest := testdata.InsignificantDigest.String() 40 | uploadID := "upload" 41 | initiateLayerUploadCount, uploadLayerPartCount, completeLayerUploadCount := 0, 0, 0 42 | client := &fakeECRClient{ 43 | InitiateLayerUploadFn: func(input *ecr.InitiateLayerUploadInput) (*ecr.InitiateLayerUploadOutput, error) { 44 | initiateLayerUploadCount++ 45 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 46 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 47 | return &ecr.InitiateLayerUploadOutput{ 48 | UploadId: aws.String(uploadID), 49 | // use single-byte upload size so we can test each byte 50 | PartSize: aws.Int64(1), 51 | }, nil 52 | }, 53 | UploadLayerPartFn: func(input *ecr.UploadLayerPartInput) (*ecr.UploadLayerPartOutput, error) { 54 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 55 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 56 | assert.Equal(t, uploadID, aws.StringValue(input.UploadId)) 57 | assert.Equal(t, int64(uploadLayerPartCount), aws.Int64Value(input.PartFirstByte), "first byte") 58 | assert.Equal(t, int64(uploadLayerPartCount), aws.Int64Value(input.PartLastByte), "last byte") 59 | assert.Len(t, input.LayerPartBlob, 1, "only one byte is expected") 60 | assert.Equal(t, layerData[uploadLayerPartCount], input.LayerPartBlob[0], "invalid layer blob data") 61 | uploadLayerPartCount++ 62 | return nil, nil 63 | }, 64 | CompleteLayerUploadFn: func(input *ecr.CompleteLayerUploadInput) (*ecr.CompleteLayerUploadOutput, error) { 65 | completeLayerUploadCount++ 66 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 67 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 68 | assert.Equal(t, uploadID, aws.StringValue(input.UploadId)) 69 | assert.Equal(t, len(layerData), uploadLayerPartCount) 70 | return &ecr.CompleteLayerUploadOutput{ 71 | LayerDigest: aws.String(layerDigest), 72 | }, nil 73 | }, 74 | } 75 | ecrBase := &ecrBase{ 76 | client: client, 77 | ecrSpec: ECRSpec{ 78 | arn: arn.ARN{ 79 | AccountID: registry, 80 | }, 81 | Repository: repository, 82 | }, 83 | } 84 | 85 | desc := ocispec.Descriptor{ 86 | Digest: digest.Digest(layerDigest), 87 | } 88 | 89 | tracker := docker.NewInMemoryTracker() 90 | refKey := "refKey" 91 | tracker.SetStatus(refKey, docker.Status{}) 92 | 93 | lw, err := newLayerWriter(ecrBase, tracker, "refKey", desc) 94 | assert.NoError(t, err) 95 | assert.Equal(t, 1, initiateLayerUploadCount) 96 | assert.Equal(t, 0, uploadLayerPartCount) 97 | assert.Equal(t, 0, completeLayerUploadCount) 98 | 99 | // Writer is required to proceed any farther. 100 | require.NotNil(t, lw) 101 | 102 | n, err := lw.Write([]byte(layerData)) 103 | assert.NoError(t, err) 104 | assert.Equal(t, len(layerData), n) 105 | 106 | err = lw.Commit(context.Background(), int64(len(layerData)), desc.Digest) 107 | assert.NoError(t, err) 108 | assert.Equal(t, 1, completeLayerUploadCount) 109 | } 110 | 111 | type layerAlreadyExistsError struct{} 112 | 113 | func (l *layerAlreadyExistsError) Code() string { return "LayerAlreadyExistsException" } 114 | func (l *layerAlreadyExistsError) Error() string { return l.Code() } 115 | func (l *layerAlreadyExistsError) Message() string { return l.Code() } 116 | func (l *layerAlreadyExistsError) OrigErr() error { return l } 117 | 118 | var _ awserr.Error = (*layerAlreadyExistsError)(nil) 119 | 120 | func TestLayerWriterCommitExists(t *testing.T) { 121 | registry := "registry" 122 | repository := "repository" 123 | layerDigest := "sha256:digest" 124 | callCount := 0 125 | client := &fakeECRClient{ 126 | CompleteLayerUploadFn: func(_ *ecr.CompleteLayerUploadInput) (*ecr.CompleteLayerUploadOutput, error) { 127 | callCount++ 128 | return nil, &layerAlreadyExistsError{} 129 | }, 130 | } 131 | 132 | _, writer := io.Pipe() 133 | ctx, cancel := context.WithCancel(context.Background()) 134 | cancel() 135 | lw := layerWriter{ 136 | base: &ecrBase{ 137 | client: client, 138 | ecrSpec: ECRSpec{ 139 | arn: arn.ARN{ 140 | AccountID: registry, 141 | }, 142 | Repository: repository, 143 | }, 144 | }, 145 | buf: writer, 146 | ctx: ctx, 147 | } 148 | 149 | err := lw.Commit(context.Background(), 0, digest.Digest(layerDigest)) 150 | assert.NoError(t, err) 151 | assert.Equal(t, 1, callCount) 152 | } 153 | -------------------------------------------------------------------------------- /ecr/manifest_writer.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "bytes" 20 | "context" 21 | "errors" 22 | "fmt" 23 | "time" 24 | 25 | "github.com/aws/aws-sdk-go/aws" 26 | "github.com/aws/aws-sdk-go/service/ecr" 27 | "github.com/containerd/containerd/content" 28 | "github.com/containerd/containerd/log" 29 | "github.com/containerd/containerd/remotes/docker" 30 | "github.com/opencontainers/go-digest" 31 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 32 | ) 33 | 34 | type manifestWriter struct { 35 | ctx context.Context 36 | base *ecrBase 37 | desc ocispec.Descriptor 38 | buf bytes.Buffer 39 | tracker docker.StatusTracker 40 | ref string 41 | } 42 | 43 | var _ content.Writer = (*manifestWriter)(nil) 44 | 45 | func (mw *manifestWriter) Write(b []byte) (int, error) { 46 | log.G(mw.ctx).WithField("len(b)", len(b)).Debug("ecr.manifest.write") 47 | return mw.buf.Write(b) 48 | } 49 | 50 | func (mw *manifestWriter) Close() error { 51 | return errors.New("ecr.manifest.close: not implemented") 52 | } 53 | 54 | func (mw *manifestWriter) Digest() digest.Digest { 55 | return mw.desc.Digest 56 | } 57 | 58 | func (mw *manifestWriter) Commit(ctx context.Context, size int64, expected digest.Digest, opts ...content.Opt) error { 59 | manifest := mw.buf.String() 60 | ecrSpec := mw.base.ecrSpec 61 | 62 | log.G(mw.ctx). 63 | WithField("manifest", manifest). 64 | WithField("size", size). 65 | WithField("expected", expected.String()). 66 | Debug("ecr.manifest.commit") 67 | 68 | putImageInput := &ecr.PutImageInput{ 69 | RegistryId: aws.String(ecrSpec.Registry()), 70 | RepositoryName: aws.String(ecrSpec.Repository), 71 | ImageManifest: aws.String(manifest), 72 | ImageManifestMediaType: aws.String(mw.desc.MediaType), 73 | ImageDigest: aws.String(expected.String()), 74 | } 75 | 76 | // Tag only if this push is the image's root descriptor, as indicated by the 77 | // parsed ECRSpec. 78 | rootDigest := ecrSpec.Spec().Digest() 79 | if mw.desc.Digest == rootDigest { 80 | if tag, _ := ecrSpec.TagDigest(); tag != "" { 81 | log.G(ctx). 82 | WithField("tag", tag). 83 | WithField("ref", rootDigest.String()). 84 | Debug("ecr.manifest.commit: tag set on push") 85 | putImageInput.ImageTag = aws.String(tag) 86 | } 87 | } 88 | 89 | output, err := mw.base.client.PutImageWithContext(ctx, putImageInput) 90 | if err != nil { 91 | return fmt.Errorf("ecr: failed to put manifest: %v: %w", ecrSpec, err) 92 | } 93 | 94 | status, err := mw.tracker.GetStatus(mw.ref) 95 | if err == nil { 96 | status.Offset = int64(len(manifest)) 97 | status.UpdatedAt = time.Now() 98 | mw.tracker.SetStatus(mw.ref, status) 99 | } else { 100 | log.G(mw.ctx).WithError(err).WithField("ref", mw.ref).Warn("Failed to update status") 101 | } 102 | if output == nil { 103 | return fmt.Errorf("ecr: failed to put manifest, nil output: %v", ecrSpec) 104 | } 105 | 106 | actual := aws.StringValue(output.Image.ImageId.ImageDigest) 107 | if actual != expected.String() { 108 | return fmt.Errorf("digest mismatch: ECR returned %s, expected %s", actual, expected) 109 | } 110 | 111 | return nil 112 | } 113 | 114 | func (mw *manifestWriter) Status() (content.Status, error) { 115 | log.G(mw.ctx).Debug("ecr.manifest.status") 116 | 117 | status, err := mw.tracker.GetStatus(mw.ref) 118 | if err != nil { 119 | return content.Status{}, err 120 | } 121 | return status.Status, nil 122 | } 123 | 124 | func (mw *manifestWriter) Truncate(size int64) error { 125 | log.G(mw.ctx).WithField("size", size).Debug("ecr.manifest.truncate") 126 | return errors.New("mw.Truncate: not implemented") 127 | } 128 | -------------------------------------------------------------------------------- /ecr/manifest_writer_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "testing" 21 | 22 | "github.com/aws/aws-sdk-go/aws" 23 | "github.com/aws/aws-sdk-go/aws/arn" 24 | "github.com/aws/aws-sdk-go/aws/request" 25 | "github.com/aws/aws-sdk-go/service/ecr" 26 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/testdata" 27 | "github.com/containerd/containerd/remotes" 28 | "github.com/containerd/containerd/remotes/docker" 29 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 30 | "github.com/stretchr/testify/assert" 31 | "github.com/stretchr/testify/require" 32 | ) 33 | 34 | func TestManifestWriterCommit(t *testing.T) { 35 | const ( 36 | manifestContent = "manifest content" 37 | registry = "registry" 38 | repository = "repository" 39 | imageTag = "tag" 40 | ) 41 | 42 | // Setup an image details for push. 43 | imageDigest := testdata.InsignificantDigest 44 | imageDesc := ocispec.Descriptor{ 45 | Digest: imageDigest, 46 | MediaType: ocispec.MediaTypeImageManifest, 47 | } 48 | // root image Object has its digest appended. 49 | imageObject := imageTag + "@" + imageDigest.String() 50 | imageECRSpec := ECRSpec{ 51 | arn: arn.ARN{ 52 | AccountID: registry, 53 | }, 54 | Repository: repository, 55 | Object: imageObject, 56 | } 57 | // root image ref is the ECRSpec's formatted ref, with the digest of the 58 | // root descriptor. For a single manifest image, that's the manifest's 59 | // digest. 60 | refKey := imageECRSpec.Canonical() 61 | 62 | t.Log("image Object: ", imageObject) 63 | t.Log("image digest: ", imageDigest) 64 | 65 | callCount := 0 66 | client := &fakeECRClient{ 67 | PutImageFn: func(_ aws.Context, input *ecr.PutImageInput, _ ...request.Option) (*ecr.PutImageOutput, error) { 68 | callCount++ 69 | 70 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 71 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 72 | assert.Equal(t, imageTag, aws.StringValue(input.ImageTag), 73 | "should use image ref's tag") 74 | assert.Equal(t, manifestContent, aws.StringValue(input.ImageManifest), 75 | "should provide manifest's body") 76 | assert.Equal(t, imageDesc.MediaType, aws.StringValue(input.ImageManifestMediaType), 77 | "should include manifest's mediaType in API input") // regardless of it being in the manifest body 78 | assert.Equal(t, imageDesc.Digest.String(), aws.StringValue(input.ImageDigest), 79 | "should include manifest's digest in API input") 80 | 81 | return &ecr.PutImageOutput{ 82 | Image: &ecr.Image{ 83 | ImageId: &ecr.ImageIdentifier{ 84 | ImageTag: input.ImageTag, 85 | ImageDigest: aws.String(imageDigest.String()), 86 | }, 87 | }, 88 | }, nil 89 | }, 90 | } 91 | mw := &manifestWriter{ 92 | desc: imageDesc, 93 | base: &ecrBase{ 94 | client: client, 95 | ecrSpec: imageECRSpec, 96 | }, 97 | tracker: docker.NewInMemoryTracker(), 98 | ref: refKey, 99 | ctx: context.Background(), 100 | } 101 | 102 | count, err := mw.Write([]byte(manifestContent[:3])) 103 | require.NoError(t, err, "failed to write to manifest writer") 104 | assert.Equal(t, 3, count, "wrong number of bytes") 105 | 106 | count, err = mw.Write([]byte(manifestContent[3:])) 107 | require.NoError(t, err, "failed to write to manifest writer") 108 | assert.Equal(t, len(manifestContent)-3, count, "wrong number of bytes") 109 | 110 | assert.Equal(t, 0, callCount, "PutImage should not be called until committed") 111 | 112 | err = mw.Commit(context.Background(), int64(len(manifestContent)), imageDigest) 113 | require.NoError(t, err, "failed to commit") 114 | assert.Equal(t, 1, callCount, "PutImage should be called once") 115 | } 116 | 117 | func TestManifestWriterNoTagCommit(t *testing.T) { 118 | const ( 119 | registry = "registry" 120 | repository = "repository" 121 | imageTag = "tag" 122 | 123 | memberManifestContent = "manifest content" 124 | ) 125 | 126 | // The root image, this is the target digest which is treated as an Image 127 | // Index in this test case. 128 | imageDigest := testdata.ImageDigest 129 | // Image pushes include the root image digest in the object: 130 | // 131 | // ie: "latest@sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff" 132 | imageObject := imageTag + "@" + imageDigest.String() 133 | 134 | // A member manifest that was listed and being "pushed" in the test case. 135 | memberDesc := ocispec.Descriptor{ 136 | Digest: "member-digest", 137 | MediaType: ocispec.MediaTypeImageManifest, 138 | } 139 | // ref, for non-root descriptors, uses the internal ref naming (eg: 140 | // index-sha256:fffff...). 141 | refKey := remotes.MakeRefKey(context.Background(), memberDesc) 142 | 143 | t.Log("image Object: ", imageObject) 144 | t.Log("image digest: " + imageDigest.String()) 145 | t.Log("member digest: " + memberDesc.Digest.String()) 146 | 147 | callCount := 0 148 | client := &fakeECRClient{ 149 | PutImageFn: func(_ aws.Context, input *ecr.PutImageInput, _ ...request.Option) (*ecr.PutImageOutput, error) { 150 | callCount++ 151 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 152 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 153 | assert.NotEqual(t, aws.StringValue(input.ImageTag), imageTag, "should not include tag when pushing non-root descriptor") 154 | assert.Equal(t, memberManifestContent, aws.StringValue(input.ImageManifest), 155 | "should provide manifest's body") 156 | assert.Equal(t, memberDesc.MediaType, aws.StringValue(input.ImageManifestMediaType), 157 | "should include manifest's mediaType in API input") // regardless of it being in the manifest body 158 | assert.Equal(t, memberDesc.Digest.String(), aws.StringValue(input.ImageDigest), 159 | "should include manifest's digest in API input") 160 | 161 | return &ecr.PutImageOutput{ 162 | Image: &ecr.Image{ 163 | ImageId: &ecr.ImageIdentifier{ 164 | // Image will have the matching digest. 165 | ImageDigest: aws.String(memberDesc.Digest.String()), 166 | }, 167 | }, 168 | }, nil 169 | }, 170 | } 171 | mw := &manifestWriter{ 172 | base: &ecrBase{ 173 | client: client, 174 | ecrSpec: ECRSpec{ 175 | arn: arn.ARN{ 176 | AccountID: registry, 177 | }, 178 | Repository: repository, 179 | Object: imageObject, 180 | }, 181 | }, 182 | desc: memberDesc, 183 | tracker: docker.NewInMemoryTracker(), 184 | ref: refKey, 185 | ctx: context.Background(), 186 | } 187 | 188 | count, err := mw.Write([]byte(memberManifestContent[:3])) 189 | require.NoError(t, err, "failed to write to manifest writer") 190 | assert.Equal(t, 3, count, "wrong number of bytes") 191 | 192 | count, err = mw.Write([]byte(memberManifestContent[3:])) 193 | require.NoError(t, err, "failed to write to manifest writer") 194 | assert.Equal(t, len(memberManifestContent)-3, count, "wrong number of bytes") 195 | 196 | assert.Equal(t, 0, callCount, "PutImage should not be called until committed") 197 | 198 | err = mw.Commit(context.Background(), int64(len(memberManifestContent)), memberDesc.Digest) 199 | require.NoError(t, err, "failed to commit") 200 | assert.Equal(t, 1, callCount, "PutImage should be called once") 201 | } 202 | -------------------------------------------------------------------------------- /ecr/pusher.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "errors" 21 | "fmt" 22 | "time" 23 | 24 | "github.com/aws/aws-sdk-go/aws" 25 | "github.com/aws/aws-sdk-go/service/ecr" 26 | "github.com/containerd/containerd/content" 27 | "github.com/containerd/containerd/errdefs" 28 | "github.com/containerd/containerd/images" 29 | "github.com/containerd/containerd/log" 30 | "github.com/containerd/containerd/reference" 31 | "github.com/containerd/containerd/remotes" 32 | "github.com/containerd/containerd/remotes/docker" 33 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 34 | ) 35 | 36 | var ( 37 | errLayerNotFound = errors.New("ecr: layer not found") 38 | ) 39 | 40 | // ecrPusher implements the containerd remotes.Pusher interface and can be used 41 | // to push images to Amazon ECR. 42 | type ecrPusher struct { 43 | ecrBase 44 | tracker docker.StatusTracker 45 | } 46 | 47 | var _ remotes.Pusher = (*ecrPusher)(nil) 48 | 49 | func (p ecrPusher) Push(ctx context.Context, desc ocispec.Descriptor) (content.Writer, error) { 50 | ctx = log.WithLogger(ctx, log.G(ctx).WithField("desc", desc)) 51 | log.G(ctx).Debug("ecr.push") 52 | 53 | switch desc.MediaType { 54 | case 55 | images.MediaTypeDockerSchema1Manifest, 56 | images.MediaTypeDockerSchema2Manifest, 57 | images.MediaTypeDockerSchema2ManifestList, 58 | ocispec.MediaTypeImageIndex, 59 | ocispec.MediaTypeImageManifest: 60 | return p.pushManifest(ctx, desc) 61 | default: 62 | return p.pushBlob(ctx, desc) 63 | } 64 | } 65 | 66 | func (p ecrPusher) pushManifest(ctx context.Context, desc ocispec.Descriptor) (content.Writer, error) { 67 | log.G(ctx).Debug("ecr.pusher.manifest") 68 | exists, err := p.checkManifestExistence(ctx, desc) 69 | if err != nil { 70 | log.G(ctx).WithError(err). 71 | Error("ecr.pusher.manifest: failed to check existence") 72 | return nil, err 73 | } 74 | if exists { 75 | log.G(ctx).Debug("ecr.pusher.manifest: content already on remote") 76 | p.markStatusExists(ctx, desc) 77 | return nil, fmt.Errorf("content %v on remote: %w", desc.Digest, errdefs.ErrAlreadyExists) 78 | } 79 | 80 | ref := p.markStatusStarted(ctx, desc) 81 | 82 | return &manifestWriter{ 83 | ctx: ctx, 84 | base: &p.ecrBase, 85 | desc: desc, 86 | tracker: p.tracker, 87 | ref: ref, 88 | }, nil 89 | } 90 | 91 | func (p ecrPusher) checkManifestExistence(ctx context.Context, desc ocispec.Descriptor) (bool, error) { 92 | image, err := p.getImageByDescriptor(ctx, desc) 93 | if err != nil { 94 | if err == errImageNotFound { 95 | return false, nil 96 | } 97 | return false, err 98 | } 99 | if image == nil { 100 | return false, errors.New("ecr.pusher.manifest: unexpected nil image") 101 | } 102 | 103 | found := desc.Digest.String() == aws.StringValue(image.ImageId.ImageDigest) 104 | return found, nil 105 | } 106 | 107 | func (p ecrPusher) pushBlob(ctx context.Context, desc ocispec.Descriptor) (content.Writer, error) { 108 | log.G(ctx).Debug("ecr.pusher.blob") 109 | exists, err := p.checkBlobExistence(ctx, desc) 110 | if err != nil { 111 | log.G(ctx).WithError(err). 112 | Error("ecr.pusher.blob: failed to check existence") 113 | return nil, err 114 | } 115 | if exists { 116 | log.G(ctx).Debug("ecr.pusher.blob: content already on remote") 117 | p.markStatusExists(ctx, desc) 118 | return nil, fmt.Errorf("content %v on remote: %w", desc.Digest, errdefs.ErrAlreadyExists) 119 | } 120 | 121 | ref := p.markStatusStarted(ctx, desc) 122 | return newLayerWriter(&p.ecrBase, p.tracker, ref, desc) 123 | } 124 | 125 | func (p ecrPusher) checkBlobExistence(ctx context.Context, desc ocispec.Descriptor) (bool, error) { 126 | batchCheckLayerAvailabilityInput := &ecr.BatchCheckLayerAvailabilityInput{ 127 | RegistryId: aws.String(p.ecrSpec.Registry()), 128 | RepositoryName: aws.String(p.ecrSpec.Repository), 129 | LayerDigests: []*string{aws.String(desc.Digest.String())}, 130 | } 131 | 132 | batchCheckLayerAvailabilityOutput, err := p.client.BatchCheckLayerAvailabilityWithContext(ctx, batchCheckLayerAvailabilityInput) 133 | if err != nil { 134 | log.G(ctx).WithError(err).Error("ecr.pusher.blob: failed to check availability") 135 | return false, err 136 | } 137 | log.G(ctx). 138 | WithField("batchCheckLayerAvailability", batchCheckLayerAvailabilityOutput). 139 | Debug("ecr.pusher.blob") 140 | 141 | if len(batchCheckLayerAvailabilityOutput.Layers) == 0 { 142 | if len(batchCheckLayerAvailabilityOutput.Failures) > 0 { 143 | return false, errLayerNotFound 144 | } 145 | return false, reference.ErrInvalid 146 | } 147 | 148 | layer := batchCheckLayerAvailabilityOutput.Layers[0] 149 | return aws.StringValue(layer.LayerAvailability) == ecr.LayerAvailabilityAvailable, nil 150 | } 151 | 152 | func (p ecrPusher) markStatusExists(ctx context.Context, desc ocispec.Descriptor) string { 153 | ref := remotes.MakeRefKey(ctx, desc) 154 | p.tracker.SetStatus(ref, docker.Status{ 155 | Status: content.Status{ 156 | Ref: ref, 157 | UpdatedAt: time.Now(), 158 | }, 159 | }) 160 | return ref 161 | } 162 | 163 | func (p ecrPusher) markStatusStarted(ctx context.Context, desc ocispec.Descriptor) string { 164 | ref := remotes.MakeRefKey(ctx, desc) 165 | p.tracker.SetStatus(ref, docker.Status{ 166 | Status: content.Status{ 167 | Ref: ref, 168 | Total: desc.Size, 169 | Expected: desc.Digest, 170 | StartedAt: time.Now(), 171 | }, 172 | }) 173 | return ref 174 | } 175 | -------------------------------------------------------------------------------- /ecr/pusher_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "errors" 21 | "testing" 22 | "time" 23 | 24 | "github.com/aws/aws-sdk-go/aws" 25 | "github.com/aws/aws-sdk-go/aws/arn" 26 | "github.com/aws/aws-sdk-go/aws/request" 27 | "github.com/aws/aws-sdk-go/service/ecr" 28 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/testdata" 29 | "github.com/containerd/containerd/errdefs" 30 | "github.com/containerd/containerd/images" 31 | "github.com/containerd/containerd/remotes" 32 | "github.com/containerd/containerd/remotes/docker" 33 | "github.com/opencontainers/go-digest" 34 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 35 | "github.com/stretchr/testify/assert" 36 | "github.com/stretchr/testify/require" 37 | ) 38 | 39 | func TestPushManifestReturnsManifestWriter(t *testing.T) { 40 | registry := "registry" 41 | repository := "repository" 42 | imageTag := "tag" 43 | imageDigest := testdata.InsignificantDigest.String() 44 | fakeClient := &fakeECRClient{} 45 | pusher := &ecrPusher{ 46 | ecrBase: ecrBase{ 47 | client: fakeClient, 48 | ecrSpec: ECRSpec{ 49 | arn: arn.ARN{ 50 | AccountID: registry, 51 | }, 52 | Repository: repository, 53 | Object: imageTag, 54 | }, 55 | }, 56 | tracker: docker.NewInMemoryTracker(), 57 | } 58 | 59 | // test all supported media types 60 | for _, mediaType := range supportedImageMediaTypes { 61 | t.Run(mediaType, func(t *testing.T) { 62 | callCount := 0 63 | 64 | // Service mock 65 | 66 | fakeClient.BatchGetImageFn = func(_ aws.Context, input *ecr.BatchGetImageInput, _ ...request.Option) (*ecr.BatchGetImageOutput, error) { 67 | callCount++ 68 | 69 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 70 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 71 | 72 | assert.ElementsMatch(t, []*ecr.ImageIdentifier{ 73 | {ImageDigest: aws.String(imageDigest)}}, 74 | input.ImageIds, 75 | "should have requested image by its digest") 76 | 77 | assert.Equal(t, []string{mediaType}, aws.StringValueSlice(input.AcceptedMediaTypes), 78 | "should have requested known mediaType") 79 | 80 | return &ecr.BatchGetImageOutput{ 81 | Failures: []*ecr.ImageFailure{ 82 | {FailureCode: aws.String(ecr.ImageFailureCodeImageNotFound)}, 83 | }, 84 | }, nil 85 | } 86 | 87 | desc := ocispec.Descriptor{ 88 | MediaType: mediaType, 89 | Digest: digest.Digest(imageDigest), 90 | } 91 | 92 | // Run mocked push 93 | 94 | start := time.Now() 95 | writer, err := pusher.Push(context.Background(), desc) 96 | assert.Equal(t, 1, callCount, "BatchGetImage should be called once") 97 | require.NoError(t, err) 98 | _, ok := writer.(*manifestWriter) 99 | assert.True(t, ok, "writer should be a manifestWriter") 100 | end := time.Now() 101 | writer.Close() 102 | 103 | refKey := remotes.MakeRefKey(context.Background(), desc) 104 | status, err := pusher.tracker.GetStatus(refKey) 105 | assert.NoError(t, err, "should retrieve status") 106 | assert.WithinDuration(t, 107 | start, 108 | status.Status.StartedAt, 109 | end.Sub(start), 110 | "should be updated between start and end") 111 | }) 112 | } 113 | } 114 | 115 | func TestPushManifestAlreadyExists(t *testing.T) { 116 | registry := "registry" 117 | repository := "repository" 118 | imageTag := "tag" 119 | imageDigest := testdata.InsignificantDigest.String() 120 | fakeClient := &fakeECRClient{ 121 | BatchGetImageFn: func(aws.Context, *ecr.BatchGetImageInput, ...request.Option) (*ecr.BatchGetImageOutput, error) { 122 | return &ecr.BatchGetImageOutput{ 123 | Images: []*ecr.Image{ 124 | {ImageId: &ecr.ImageIdentifier{ImageDigest: aws.String(imageDigest)}}, 125 | }, 126 | }, nil 127 | }, 128 | } 129 | pusher := &ecrPusher{ 130 | ecrBase: ecrBase{ 131 | client: fakeClient, 132 | ecrSpec: ECRSpec{ 133 | arn: arn.ARN{ 134 | AccountID: registry, 135 | }, 136 | Repository: repository, 137 | Object: imageTag, 138 | }, 139 | }, 140 | tracker: docker.NewInMemoryTracker(), 141 | } 142 | 143 | desc := ocispec.Descriptor{ 144 | MediaType: ocispec.MediaTypeImageManifest, 145 | Digest: digest.Digest(imageDigest), 146 | } 147 | 148 | start := time.Now() 149 | _, err := pusher.Push(context.Background(), desc) 150 | assert.Error(t, err) 151 | assert.True(t, errors.Is(err, errdefs.ErrAlreadyExists)) 152 | end := time.Now() 153 | 154 | refKey := remotes.MakeRefKey(context.Background(), desc) 155 | status, err := pusher.tracker.GetStatus(refKey) 156 | assert.NoError(t, err, "should retrieve status") 157 | assert.WithinDuration(t, 158 | start, 159 | status.Status.UpdatedAt, 160 | end.Sub(start), 161 | "should be updated between start and end") 162 | } 163 | 164 | func TestPushBlobReturnsLayerWriter(t *testing.T) { 165 | registry := "registry" 166 | repository := "repository" 167 | layerDigest := testdata.InsignificantDigest.String() 168 | fakeClient := &fakeECRClient{ 169 | InitiateLayerUploadFn: func(*ecr.InitiateLayerUploadInput) (*ecr.InitiateLayerUploadOutput, error) { 170 | // layerWriter calls this during its constructor 171 | return &ecr.InitiateLayerUploadOutput{}, nil 172 | }, 173 | } 174 | pusher := &ecrPusher{ 175 | ecrBase: ecrBase{ 176 | client: fakeClient, 177 | ecrSpec: ECRSpec{ 178 | arn: arn.ARN{ 179 | AccountID: registry, 180 | }, 181 | Repository: repository, 182 | }, 183 | }, 184 | tracker: docker.NewInMemoryTracker(), 185 | } 186 | 187 | // test all supported media types 188 | for _, mediaType := range []string{ 189 | images.MediaTypeDockerSchema2Layer, 190 | images.MediaTypeDockerSchema2LayerGzip, 191 | images.MediaTypeDockerSchema2Config, 192 | ocispec.MediaTypeImageLayerGzip, 193 | ocispec.MediaTypeImageLayer, 194 | ocispec.MediaTypeImageConfig, 195 | } { 196 | t.Run(mediaType, func(t *testing.T) { 197 | callCount := 0 198 | fakeClient.BatchCheckLayerAvailabilityFn = func(_ aws.Context, input *ecr.BatchCheckLayerAvailabilityInput, _ ...request.Option) (*ecr.BatchCheckLayerAvailabilityOutput, error) { 199 | callCount++ 200 | assert.Equal(t, registry, aws.StringValue(input.RegistryId)) 201 | assert.Equal(t, repository, aws.StringValue(input.RepositoryName)) 202 | require.Len(t, input.LayerDigests, 1) 203 | assert.Equal(t, layerDigest, aws.StringValue(input.LayerDigests[0])) 204 | return &ecr.BatchCheckLayerAvailabilityOutput{ 205 | Layers: []*ecr.Layer{{ 206 | LayerAvailability: aws.String(ecr.LayerAvailabilityUnavailable), 207 | }}, 208 | }, nil 209 | } 210 | 211 | desc := ocispec.Descriptor{ 212 | MediaType: ocispec.MediaTypeImageLayerGzip, 213 | Digest: digest.Digest(layerDigest), 214 | } 215 | 216 | start := time.Now() 217 | writer, err := pusher.Push(context.Background(), desc) 218 | assert.Equal(t, 1, callCount, "BatchCheckLayerAvailability should be called once") 219 | assert.NoError(t, err) 220 | _, ok := writer.(*layerWriter) 221 | assert.True(t, ok, "writer should be a layerWriter") 222 | end := time.Now() 223 | writer.Close() 224 | 225 | refKey := remotes.MakeRefKey(context.Background(), desc) 226 | status, err := pusher.tracker.GetStatus(refKey) 227 | assert.NoError(t, err, "should retrieve status") 228 | assert.WithinDuration(t, 229 | start, 230 | status.Status.StartedAt, 231 | end.Sub(start), 232 | "should be updated between start and end") 233 | }) 234 | } 235 | } 236 | 237 | func TestPushBlobAlreadyExists(t *testing.T) { 238 | registry := "registry" 239 | repository := "repository" 240 | layerDigest := testdata.InsignificantDigest.String() 241 | fakeClient := &fakeECRClient{ 242 | BatchCheckLayerAvailabilityFn: func(aws.Context, *ecr.BatchCheckLayerAvailabilityInput, ...request.Option) (*ecr.BatchCheckLayerAvailabilityOutput, error) { 243 | return &ecr.BatchCheckLayerAvailabilityOutput{ 244 | Layers: []*ecr.Layer{{ 245 | LayerAvailability: aws.String(ecr.LayerAvailabilityAvailable), 246 | }}, 247 | }, nil 248 | }, 249 | } 250 | pusher := &ecrPusher{ 251 | ecrBase: ecrBase{ 252 | client: fakeClient, 253 | ecrSpec: ECRSpec{ 254 | arn: arn.ARN{ 255 | AccountID: registry, 256 | }, 257 | Repository: repository, 258 | }, 259 | }, 260 | tracker: docker.NewInMemoryTracker(), 261 | } 262 | 263 | desc := ocispec.Descriptor{ 264 | MediaType: ocispec.MediaTypeImageLayerGzip, 265 | Digest: digest.Digest(layerDigest), 266 | } 267 | 268 | start := time.Now() 269 | _, err := pusher.Push(context.Background(), desc) 270 | assert.Error(t, err) 271 | assert.True(t, errors.Is(err, errdefs.ErrAlreadyExists)) 272 | end := time.Now() 273 | 274 | refKey := remotes.MakeRefKey(context.Background(), desc) 275 | status, err := pusher.tracker.GetStatus(refKey) 276 | assert.NoError(t, err, "should retrieve status") 277 | assert.WithinDuration(t, 278 | start, 279 | status.Status.UpdatedAt, 280 | end.Sub(start), 281 | "should be updated between start and end") 282 | } 283 | 284 | func TestPushBlobAPIError(t *testing.T) { 285 | registry := "registry" 286 | repository := "repository" 287 | layerDigest := testdata.InsignificantDigest.String() 288 | fakeClient := &fakeECRClient{ 289 | BatchCheckLayerAvailabilityFn: func(aws.Context, *ecr.BatchCheckLayerAvailabilityInput, ...request.Option) (*ecr.BatchCheckLayerAvailabilityOutput, error) { 290 | return &ecr.BatchCheckLayerAvailabilityOutput{ 291 | Failures: []*ecr.LayerFailure{{}}, 292 | }, nil 293 | }, 294 | } 295 | pusher := &ecrPusher{ 296 | ecrBase: ecrBase{ 297 | client: fakeClient, 298 | ecrSpec: ECRSpec{ 299 | arn: arn.ARN{ 300 | AccountID: registry, 301 | }, 302 | Repository: repository, 303 | }, 304 | }, 305 | tracker: docker.NewInMemoryTracker(), 306 | } 307 | 308 | desc := ocispec.Descriptor{ 309 | MediaType: ocispec.MediaTypeImageLayerGzip, 310 | Digest: digest.Digest(layerDigest), 311 | } 312 | 313 | _, err := pusher.Push(context.Background(), desc) 314 | assert.EqualError(t, err, errLayerNotFound.Error()) 315 | } 316 | -------------------------------------------------------------------------------- /ecr/ref.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "errors" 20 | "fmt" 21 | "regexp" 22 | "strings" 23 | 24 | "github.com/aws/aws-sdk-go/aws" 25 | "github.com/aws/aws-sdk-go/aws/arn" 26 | "github.com/aws/aws-sdk-go/aws/endpoints" 27 | "github.com/aws/aws-sdk-go/service/ecr" 28 | "github.com/containerd/containerd/reference" 29 | "github.com/opencontainers/go-digest" 30 | ) 31 | 32 | const ( 33 | refPrefix = "ecr.aws/" 34 | repositoryPrefix = "repository/" 35 | arnServiceID = "ecr" 36 | ) 37 | 38 | var ( 39 | invalidARN = errors.New("ref: invalid ARN") 40 | // Expecting to match ECR image names of the form: 41 | // Example 1: 777777777777.dkr.ecr.us-west-2.amazonaws.com/my_image:latest 42 | // Example 2: 777777777777.dkr.ecr.cn-north-1.amazonaws.com.cn/my_image:latest 43 | // TODO: Support ECR FIPS endpoints, i.e "ecr-fips" in the URL instead of "ecr" 44 | ecrRegex = regexp.MustCompile(`(^[a-zA-Z0-9][a-zA-Z0-9-_]*)\.dkr\.ecr\.([a-zA-Z0-9][a-zA-Z0-9-_]*)\.amazonaws\.com(\.cn)?/.*`) 45 | errInvalidImageURI = errors.New("ecrspec: invalid image URI") 46 | ) 47 | 48 | // ECRSpec represents a parsed reference. 49 | // 50 | // Valid references are of the form "ecr.aws/arn:aws:ecr:::repository/:". 51 | type ECRSpec struct { 52 | // Repository name for this reference. 53 | Repository string 54 | // Object is the image reference's object descriptor. This may be a label or 55 | // a digest specifier. 56 | Object string 57 | // arn holds the canonical AWS resource name for this reference. 58 | arn arn.ARN 59 | } 60 | 61 | // ParseRef parses an ECR reference into its constituent parts 62 | func ParseRef(ref string) (ECRSpec, error) { 63 | if !strings.HasPrefix(ref, refPrefix) { 64 | return ECRSpec{}, invalidARN 65 | } 66 | stripped := ref[len(refPrefix):] 67 | return parseARN(stripped) 68 | } 69 | 70 | // ParseImageURI takes an ECR image URI and then constructs and returns an ECRSpec struct 71 | func ParseImageURI(input string) (ECRSpec, error) { 72 | input = strings.TrimPrefix(input, "https://") 73 | 74 | // Matching on account, region 75 | matches := ecrRegex.FindStringSubmatch(input) 76 | if len(matches) < 3 { 77 | return ECRSpec{}, errInvalidImageURI 78 | } 79 | account := matches[1] 80 | region := matches[2] 81 | 82 | // Get the correct partition given its region 83 | partition, found := endpoints.PartitionForRegion(endpoints.DefaultPartitions(), region) 84 | if !found { 85 | return ECRSpec{}, errInvalidImageURI 86 | } 87 | 88 | // Need to include the full repository path and the imageID (e.g. /eks/image-name:tag) 89 | tokens := strings.SplitN(input, "/", 2) 90 | if len(tokens) != 2 { 91 | return ECRSpec{}, errInvalidImageURI 92 | } 93 | 94 | fullRepoPath := tokens[len(tokens)-1] 95 | // Run simple checks on the provided repository. 96 | switch { 97 | case 98 | // Must not be empty 99 | fullRepoPath == "", 100 | // Must not have a partial/unsupplied label 101 | strings.HasSuffix(fullRepoPath, ":"), 102 | // Must not have a partial/unsupplied digest specifier 103 | strings.HasSuffix(fullRepoPath, "@"): 104 | return ECRSpec{}, errors.New("incomplete reference provided") 105 | } 106 | 107 | // Parse out image reference's to validate. 108 | ref, err := reference.Parse(repositoryPrefix + fullRepoPath) 109 | if err != nil { 110 | return ECRSpec{}, err 111 | } 112 | // If the digest is provided, check that it is valid. 113 | if ref.Digest() != "" { 114 | err := ref.Digest().Validate() 115 | // Digest may not be supported by the client despite it passing against 116 | // a rudimentary check. The error is different in the passing case, so 117 | // that's considered a passing check for unavailable digesters. 118 | // 119 | // https://github.com/opencontainers/go-digest/blob/ea51bea511f75cfa3ef6098cc253c5c3609b037a/digest.go#L110-L115 120 | if err != nil && err != digest.ErrDigestUnsupported { 121 | return ECRSpec{}, fmt.Errorf("%v: %w", errInvalidImageURI.Error(), err) 122 | } 123 | } 124 | 125 | return ECRSpec{ 126 | Repository: strings.TrimPrefix(ref.Locator, repositoryPrefix), 127 | Object: ref.Object, 128 | arn: arn.ARN{ 129 | Partition: partition.ID(), 130 | Service: arnServiceID, 131 | Region: region, 132 | AccountID: account, 133 | Resource: ref.Locator, 134 | }, 135 | }, nil 136 | } 137 | 138 | // Partition returns the AWS partition 139 | func (spec ECRSpec) Partition() string { 140 | return spec.arn.Partition 141 | } 142 | 143 | // Region returns the AWS region 144 | func (spec ECRSpec) Region() string { 145 | return spec.arn.Region 146 | } 147 | 148 | // Registry returns the Amazon ECR registry 149 | func (spec ECRSpec) Registry() string { 150 | return spec.arn.AccountID 151 | } 152 | 153 | // parseARN parses an ECR ARN into its constituent parts. 154 | // 155 | // An example ARN is: arn:aws:ecr:us-west-2:123456789012:repository/foo/bar 156 | func parseARN(a string) (ECRSpec, error) { 157 | parsed, err := arn.Parse(a) 158 | if err != nil { 159 | return ECRSpec{}, err 160 | } 161 | 162 | spec, err := reference.Parse(parsed.Resource) 163 | if err != nil { 164 | return ECRSpec{}, err 165 | } 166 | parsed.Resource = spec.Locator 167 | 168 | // Extract unprefixed repo name contained in the resource part. 169 | unprefixedRepo := strings.TrimPrefix(parsed.Resource, repositoryPrefix) 170 | if unprefixedRepo == parsed.Resource { 171 | return ECRSpec{}, invalidARN 172 | } 173 | 174 | return ECRSpec{ 175 | arn: parsed, 176 | Repository: unprefixedRepo, 177 | Object: spec.Object, 178 | }, nil 179 | } 180 | 181 | // Canonical returns the canonical representation for the reference 182 | func (spec ECRSpec) Canonical() string { 183 | return spec.Spec().String() 184 | } 185 | 186 | // ARN returns the canonical representation of the ECR ARN 187 | func (spec ECRSpec) ARN() string { 188 | return spec.arn.String() 189 | } 190 | 191 | // Spec returns a reference.Spec 192 | func (spec ECRSpec) Spec() reference.Spec { 193 | return reference.Spec{ 194 | Locator: refPrefix + spec.ARN(), 195 | Object: spec.Object, 196 | } 197 | } 198 | 199 | // ImageID returns an ecr.ImageIdentifier suitable for using in calls to ECR 200 | func (spec ECRSpec) ImageID() *ecr.ImageIdentifier { 201 | imageID := ecr.ImageIdentifier{} 202 | tag, digest := spec.TagDigest() 203 | if tag != "" { 204 | imageID.ImageTag = aws.String(tag) 205 | } 206 | if digest != "" { 207 | imageID.ImageDigest = aws.String(digest.String()) 208 | } 209 | return &imageID 210 | } 211 | 212 | // TagDigest returns the tag and/or digest specified by the reference 213 | func (spec ECRSpec) TagDigest() (string, digest.Digest) { 214 | tag, digest := reference.SplitObject(spec.Object) 215 | return strings.TrimSuffix(tag, "@"), digest 216 | } 217 | -------------------------------------------------------------------------------- /ecr/ref_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "errors" 20 | "fmt" 21 | "testing" 22 | 23 | "github.com/aws/aws-sdk-go/aws" 24 | "github.com/aws/aws-sdk-go/aws/arn" 25 | "github.com/aws/aws-sdk-go/service/ecr" 26 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/testdata" 27 | "github.com/stretchr/testify/assert" 28 | "github.com/stretchr/testify/require" 29 | ) 30 | 31 | func TestRefRepresentations(t *testing.T) { 32 | cases := []struct { 33 | ref string 34 | arn string 35 | spec ECRSpec 36 | err error 37 | }{ 38 | { 39 | ref: "invalid", 40 | err: invalidARN, 41 | }, 42 | { 43 | ref: "ecr.aws/arn:nope", 44 | err: errors.New("arn: not enough sections"), 45 | }, 46 | { 47 | ref: "arn:aws:ecr:us-west-2:123456789012:repository/foo/bar", 48 | err: invalidARN, 49 | }, 50 | { 51 | ref: "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/foo/bar", 52 | arn: "arn:aws:ecr:us-west-2:123456789012:repository/foo/bar", 53 | spec: ECRSpec{ 54 | arn: arn.ARN{ 55 | Partition: "aws", 56 | Region: "us-west-2", 57 | AccountID: "123456789012", 58 | Service: "ecr", 59 | Resource: "repository/foo/bar", 60 | }, 61 | Repository: "foo/bar", 62 | }, 63 | }, 64 | { 65 | ref: "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/foo/bar:latest", 66 | arn: "arn:aws:ecr:us-west-2:123456789012:repository/foo/bar", 67 | spec: ECRSpec{ 68 | arn: arn.ARN{ 69 | Partition: "aws", 70 | Region: "us-west-2", 71 | AccountID: "123456789012", 72 | Service: "ecr", 73 | Resource: "repository/foo/bar", 74 | }, 75 | Repository: "foo/bar", 76 | Object: "latest", 77 | }, 78 | }, 79 | { 80 | ref: "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/foo/bar:latest@" + testdata.ImageDigest.String(), 81 | arn: "arn:aws:ecr:us-west-2:123456789012:repository/foo/bar", 82 | spec: ECRSpec{ 83 | arn: arn.ARN{ 84 | Partition: "aws", 85 | Region: "us-west-2", 86 | AccountID: "123456789012", 87 | Service: "ecr", 88 | Resource: "repository/foo/bar", 89 | }, 90 | Repository: "foo/bar", 91 | Object: "latest@" + testdata.ImageDigest.String(), 92 | }, 93 | }, 94 | { 95 | ref: "ecr.aws/arn:aws:ecr:us-west-2:123456789012:repository/foo/bar@" + testdata.ImageDigest.String(), 96 | arn: "arn:aws:ecr:us-west-2:123456789012:repository/foo/bar", 97 | spec: ECRSpec{ 98 | arn: arn.ARN{ 99 | Partition: "aws", 100 | Region: "us-west-2", 101 | AccountID: "123456789012", 102 | Service: "ecr", 103 | Resource: "repository/foo/bar", 104 | }, 105 | Repository: "foo/bar", 106 | Object: "@" + testdata.ImageDigest.String(), 107 | }, 108 | }, 109 | } 110 | for _, tc := range cases { 111 | t.Run(fmt.Sprintf("ParseRef-%s", tc.ref), func(t *testing.T) { 112 | spec, err := ParseRef(tc.ref) 113 | assert.Equal(t, tc.spec, spec) 114 | if tc.err == nil { 115 | assert.Nil(t, err) 116 | } else { 117 | assert.Equal(t, tc.err, err) 118 | } 119 | }) 120 | if tc.err != nil { 121 | continue 122 | } 123 | t.Run(fmt.Sprintf("Canonical-%s", tc.ref), func(t *testing.T) { 124 | assert.Equal(t, tc.ref, tc.spec.Canonical()) 125 | }) 126 | t.Run(fmt.Sprintf("ARN-%s", tc.ref), func(t *testing.T) { 127 | assert.Equal(t, tc.arn, tc.spec.ARN()) 128 | }) 129 | } 130 | } 131 | 132 | func TestImageID(t *testing.T) { 133 | cases := []struct { 134 | name string 135 | spec ECRSpec 136 | imageID *ecr.ImageIdentifier 137 | }{ 138 | { 139 | name: "blank", 140 | spec: ECRSpec{ 141 | Repository: "foo/bar", 142 | }, 143 | imageID: &ecr.ImageIdentifier{}, 144 | }, 145 | { 146 | name: "tag", 147 | spec: ECRSpec{ 148 | Repository: "foo/bar", 149 | Object: "latest", 150 | }, 151 | imageID: &ecr.ImageIdentifier{ 152 | ImageTag: aws.String("latest"), 153 | }, 154 | }, 155 | { 156 | name: "digest", 157 | spec: ECRSpec{ 158 | Repository: "foo/bar", 159 | Object: "@" + testdata.ImageDigest.String(), 160 | }, 161 | imageID: &ecr.ImageIdentifier{ 162 | ImageDigest: aws.String(testdata.ImageDigest.String()), 163 | }, 164 | }, 165 | { 166 | name: "tag+digest", 167 | spec: ECRSpec{ 168 | Repository: "foo/bar", 169 | Object: "latest@" + testdata.ImageDigest.String(), 170 | }, 171 | imageID: &ecr.ImageIdentifier{ 172 | ImageTag: aws.String("latest"), 173 | ImageDigest: aws.String(testdata.ImageDigest.String()), 174 | }, 175 | }, 176 | } 177 | 178 | for _, tc := range cases { 179 | t.Run(tc.name, func(t *testing.T) { 180 | assert.Equal(t, tc.imageID, tc.spec.ImageID()) 181 | }) 182 | } 183 | } 184 | 185 | // Test ParseEcrImageNameToRef with a valid ECR image name 186 | func TestParseImageURIValid(t *testing.T) { 187 | tests := []struct { 188 | name string 189 | imageName string 190 | expected string 191 | }{ 192 | { 193 | "Standard", 194 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/my_image:latest", 195 | "ecr.aws/arn:aws:ecr:us-west-2:777777777777:repository/my_image:latest", 196 | }, 197 | { 198 | "Standard: With additional repository path", 199 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/foo/bar/my_image:latest", 200 | "ecr.aws/arn:aws:ecr:us-west-2:777777777777:repository/foo/bar/my_image:latest", 201 | }, 202 | { 203 | "Standard: Digests", 204 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/my_image@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", 205 | "ecr.aws/arn:aws:ecr:us-west-2:777777777777:repository/my_image@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", 206 | }, 207 | { 208 | "Standard: Digests with additional repository path", 209 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/baz/my_image@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", 210 | "ecr.aws/arn:aws:ecr:us-west-2:777777777777:repository/baz/my_image@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", 211 | }, 212 | { 213 | "AWS CN partition", 214 | "777777777777.dkr.ecr.cn-north-1.amazonaws.com.cn/my_image:latest", 215 | "ecr.aws/arn:aws-cn:ecr:cn-north-1:777777777777:repository/my_image:latest", 216 | }, 217 | { 218 | "AWS Gov Cloud West", 219 | "777777777777.dkr.ecr.us-gov-west-1.amazonaws.com/my_image:latest", 220 | "ecr.aws/arn:aws-us-gov:ecr:us-gov-west-1:777777777777:repository/my_image:latest", 221 | }, 222 | { 223 | "AWS Gov Cloud East", 224 | "777777777777.dkr.ecr.us-gov-east-1.amazonaws.com/my_image:latest", 225 | "ecr.aws/arn:aws-us-gov:ecr:us-gov-east-1:777777777777:repository/my_image:latest", 226 | }, 227 | } 228 | for _, tc := range tests { 229 | t.Run(tc.name, func(t *testing.T) { 230 | t.Logf("input: %q", tc.imageName) 231 | result, err := ParseImageURI(tc.imageName) 232 | require.NoError(t, err, "failed to convert image name into ref") 233 | assert.Equal(t, tc.expected, result.Canonical()) 234 | }) 235 | } 236 | } 237 | 238 | // Test ParseEcrImageNameToRef with an invalid ECR image name 239 | func TestParseImageURIInvalid(t *testing.T) { 240 | tests := []struct { 241 | name string 242 | imageName string 243 | }{ 244 | { 245 | "empty", 246 | "", 247 | }, 248 | { 249 | "no account", 250 | "dkr.ecr.us-west-2.amazonaws.com", 251 | }, 252 | { 253 | "no region", 254 | "777777777777.dkr.ecr.amazonaws.com/", 255 | }, 256 | { 257 | "not an ecr image", 258 | "docker.io/library/hello-world", 259 | }, 260 | { 261 | "missing repository", 262 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/", 263 | }, 264 | { 265 | "missing digest value", 266 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/repo-name@", 267 | }, 268 | { 269 | "missing label value", 270 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/repo-name:", 271 | }, 272 | { 273 | "missing name and label value", 274 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/:", 275 | }, 276 | { 277 | "missing typed digest part", 278 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/repo-name@sha256:", 279 | }, 280 | { 281 | "invalid typed digest part", 282 | "777777777777.dkr.ecr.us-west-2.amazonaws.com/repo-name@sha256:invalid-digest-value", 283 | }, 284 | } 285 | 286 | for _, tc := range tests { 287 | t.Run(tc.name, func(t *testing.T) { 288 | t.Logf("input: %q", tc.imageName) 289 | _, err := ParseImageURI(tc.imageName) 290 | assert.Error(t, err) 291 | }) 292 | } 293 | } 294 | -------------------------------------------------------------------------------- /ecr/resolver.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "encoding/json" 21 | "errors" 22 | "fmt" 23 | "net/http" 24 | "sync" 25 | 26 | "github.com/aws/aws-sdk-go/aws" 27 | "github.com/aws/aws-sdk-go/aws/session" 28 | "github.com/aws/aws-sdk-go/service/ecr" 29 | ecrsdk "github.com/aws/aws-sdk-go/service/ecr" 30 | "github.com/containerd/containerd/errdefs" 31 | "github.com/containerd/containerd/images" 32 | "github.com/containerd/containerd/log" 33 | "github.com/containerd/containerd/reference" 34 | "github.com/containerd/containerd/remotes" 35 | "github.com/containerd/containerd/remotes/docker" 36 | "github.com/opencontainers/go-digest" 37 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 38 | ) 39 | 40 | var ( 41 | ErrInvalidManifest = errors.New("invalid manifest") 42 | unimplemented = errors.New("unimplemented") 43 | ) 44 | 45 | type ecrResolver struct { 46 | session *session.Session 47 | clients map[string]ecrAPI 48 | clientsLock sync.Mutex 49 | tracker docker.StatusTracker 50 | layerDownloadParallelism int 51 | httpClient *http.Client 52 | } 53 | 54 | // ResolverOption represents a functional option for configuring the ECR 55 | // Resolver 56 | type ResolverOption func(*ResolverOptions) error 57 | 58 | // ResolverOptions represents available options for configuring the ECR Resolver 59 | type ResolverOptions struct { 60 | // Session is used for configuring the ECR client. If not specified, a 61 | // generic session is used. 62 | Session *session.Session 63 | // Tracker is used to track uploads to ECR. If not specified, an in-memory 64 | // tracker is used instead. 65 | Tracker docker.StatusTracker 66 | // LayerDownloadParallelism configures whether layer parts should be 67 | // downloaded in parallel. If not specified, parallelism is currently 68 | // disabled. 69 | LayerDownloadParallelism int 70 | // HTTPClient configures the HTTP client the resolver internally use for fetching. 71 | // If not specified, http.DefaultClient is used. 72 | HTTPClient *http.Client 73 | } 74 | 75 | // WithSession is a ResolverOption to use a specific AWS session.Session 76 | func WithSession(session *session.Session) ResolverOption { 77 | return func(options *ResolverOptions) error { 78 | options.Session = session 79 | return nil 80 | } 81 | } 82 | 83 | // WithTracker is a ResolverOption to use a specific docker.Tracker 84 | func WithTracker(tracker docker.StatusTracker) ResolverOption { 85 | return func(options *ResolverOptions) error { 86 | options.Tracker = tracker 87 | return nil 88 | } 89 | } 90 | 91 | // WithLayerDownloadParallelism is a ResolverOption to configure whether layer 92 | // parts should be downloaded in parallel. Layer parallelism is backed by the 93 | // htcat library and can increase the speed at which layers are downloaded at 94 | // the cost of increased memory consumption. It is recommended to test your 95 | // workload to determine whether the tradeoff is worthwhile. 96 | func WithLayerDownloadParallelism(parallelism int) ResolverOption { 97 | return func(options *ResolverOptions) error { 98 | options.LayerDownloadParallelism = parallelism 99 | return nil 100 | } 101 | } 102 | 103 | // WithHTTPClient is a ResolverOption to use a specific http.Client. 104 | func WithHTTPClient(client *http.Client) ResolverOption { 105 | return func(options *ResolverOptions) error { 106 | options.HTTPClient = client 107 | return nil 108 | } 109 | } 110 | 111 | // NewResolver creates a new remotes.Resolver capable of interacting with Amazon 112 | // ECR. NewResolver can be called with no arguments for default configuration, 113 | // or can be customized by specifying ResolverOptions. By default, NewResolver 114 | // will allocate a new AWS session.Session and an in-memory tracker for layer 115 | // progress. 116 | func NewResolver(options ...ResolverOption) (remotes.Resolver, error) { 117 | resolverOptions := &ResolverOptions{} 118 | for _, option := range options { 119 | err := option(resolverOptions) 120 | if err != nil { 121 | return nil, err 122 | } 123 | } 124 | if resolverOptions.Session == nil { 125 | awsSession, err := session.NewSession() 126 | if err != nil { 127 | return nil, err 128 | } 129 | resolverOptions.Session = awsSession 130 | } 131 | if resolverOptions.Tracker == nil { 132 | resolverOptions.Tracker = docker.NewInMemoryTracker() 133 | } 134 | 135 | if resolverOptions.HTTPClient == nil { 136 | resolverOptions.HTTPClient = http.DefaultClient 137 | } 138 | 139 | return &ecrResolver{ 140 | session: resolverOptions.Session, 141 | clients: map[string]ecrAPI{}, 142 | tracker: resolverOptions.Tracker, 143 | layerDownloadParallelism: resolverOptions.LayerDownloadParallelism, 144 | httpClient: resolverOptions.HTTPClient, 145 | }, nil 146 | } 147 | 148 | // Resolve attempts to resolve the provided reference into a name and a 149 | // descriptor. 150 | // 151 | // Valid references are of the form "ecr.aws/arn:aws:ecr:::repository/:". 152 | func (r *ecrResolver) Resolve(ctx context.Context, ref string) (string, ocispec.Descriptor, error) { 153 | ecrSpec, err := ParseRef(ref) 154 | if err != nil { 155 | return "", ocispec.Descriptor{}, err 156 | } 157 | 158 | if ecrSpec.Object == "" { 159 | return "", ocispec.Descriptor{}, reference.ErrObjectRequired 160 | } 161 | 162 | batchGetImageInput := &ecr.BatchGetImageInput{ 163 | RegistryId: aws.String(ecrSpec.Registry()), 164 | RepositoryName: aws.String(ecrSpec.Repository), 165 | ImageIds: []*ecr.ImageIdentifier{ecrSpec.ImageID()}, 166 | AcceptedMediaTypes: aws.StringSlice(supportedImageMediaTypes), 167 | } 168 | 169 | client := r.getClient(ecrSpec.Region()) 170 | 171 | batchGetImageOutput, err := client.BatchGetImageWithContext(ctx, batchGetImageInput) 172 | if err != nil { 173 | log.G(ctx). 174 | WithField("ref", ref). 175 | WithError(err). 176 | Warn("Failed while calling BatchGetImage") 177 | return "", ocispec.Descriptor{}, err 178 | } 179 | log.G(ctx). 180 | WithField("ref", ref). 181 | WithField("batchGetImageOutput", batchGetImageOutput). 182 | Debug("ecr.resolver.resolve") 183 | 184 | if len(batchGetImageOutput.Images) == 0 { 185 | return "", ocispec.Descriptor{}, reference.ErrInvalid 186 | } 187 | ecrImage := batchGetImageOutput.Images[0] 188 | 189 | mediaType := aws.StringValue(ecrImage.ImageManifestMediaType) 190 | if mediaType == "" { 191 | manifestBody := aws.StringValue(ecrImage.ImageManifest) 192 | log.G(ctx). 193 | WithField("ref", ref). 194 | WithField("manifest", manifestBody). 195 | Trace("ecr.resolver.resolve: parsing mediaType from manifest") 196 | mediaType, err = parseImageManifestMediaType(ctx, manifestBody) 197 | if err != nil { 198 | return "", ocispec.Descriptor{}, err 199 | } 200 | } 201 | log.G(ctx). 202 | WithField("ref", ref). 203 | WithField("mediaType", mediaType). 204 | Debug("ecr.resolver.resolve") 205 | // check resolved image's mediaType, it should be one of the specified in 206 | // the request. 207 | for i, accepted := range aws.StringValueSlice(batchGetImageInput.AcceptedMediaTypes) { 208 | if mediaType == accepted { 209 | break 210 | } 211 | if i+1 == len(batchGetImageInput.AcceptedMediaTypes) { 212 | log.G(ctx). 213 | WithField("ref", ref). 214 | WithField("mediaType", mediaType). 215 | Debug("ecr.resolver.resolve: unrequested mediaType, deferring to caller") 216 | } 217 | } 218 | 219 | desc := ocispec.Descriptor{ 220 | Digest: digest.Digest(aws.StringValue(ecrImage.ImageId.ImageDigest)), 221 | MediaType: mediaType, 222 | Size: int64(len(aws.StringValue(ecrImage.ImageManifest))), 223 | } 224 | // assert matching digest if the provided ref includes one. 225 | if expectedDigest := ecrSpec.Spec().Digest().String(); expectedDigest != "" && 226 | desc.Digest.String() != expectedDigest { 227 | return "", ocispec.Descriptor{}, fmt.Errorf("resolved image digest mismatch: %w", errdefs.ErrFailedPrecondition) 228 | } 229 | 230 | return ecrSpec.Canonical(), desc, nil 231 | } 232 | 233 | func (r *ecrResolver) getClient(region string) ecrAPI { 234 | r.clientsLock.Lock() 235 | defer r.clientsLock.Unlock() 236 | if _, ok := r.clients[region]; !ok { 237 | r.clients[region] = ecrsdk.New(r.session, &aws.Config{ 238 | Region: aws.String(region), 239 | HTTPClient: r.httpClient}) 240 | } 241 | return r.clients[region] 242 | } 243 | 244 | // manifestProbe provides a structure to parse and then probe a given manifest 245 | // to determine its mediaType. 246 | type manifestProbe struct { 247 | // SchemaVersion is version identifier for the manifest schema used. 248 | SchemaVersion int64 `json:"schemaVersion"` 249 | // Explicit MediaType assignment for the manifest. 250 | MediaType string `json:"mediaType,omitempty"` 251 | // Docker Schema 1 signatures. 252 | Signatures []json.RawMessage `json:"signatures,omitempty"` 253 | // OCI or Docker Manifest Lists, the list of descriptors has mediaTypes 254 | // embedded. 255 | Manifests []json.RawMessage `json:"manifests,omitempty"` 256 | } 257 | 258 | func parseImageManifestMediaType(ctx context.Context, body string) (string, error) { 259 | // The unsigned variant of Docker v2 Schema 1 manifest mediaType. 260 | const mediaTypeDockerSchema1ManifestUnsigned = "application/vnd.docker.distribution.manifest.v1+json" 261 | 262 | var manifest manifestProbe 263 | err := json.Unmarshal([]byte(body), &manifest) 264 | if err != nil { 265 | return "", fmt.Errorf("failed to unmarshall %q as a manifest: %w", body, ErrInvalidManifest) 266 | } 267 | 268 | switch manifest.SchemaVersion { 269 | case 2: 270 | // Defer to the manifest declared type. 271 | if manifest.MediaType != "" { 272 | return manifest.MediaType, nil 273 | } 274 | // Is a manifest list. 275 | if len(manifest.Manifests) > 0 { 276 | return images.MediaTypeDockerSchema2ManifestList, nil 277 | } 278 | // Is a single image manifest. 279 | return images.MediaTypeDockerSchema2Manifest, nil 280 | 281 | case 1: 282 | // Defer to the manifest declared type. 283 | if manifest.MediaType != "" { 284 | return manifest.MediaType, nil 285 | } 286 | // Is Signed Docker Schema 1 manifest. 287 | if len(manifest.Signatures) > 0 { 288 | return images.MediaTypeDockerSchema1Manifest, nil 289 | } 290 | // Is Unsigned Docker Schema 1 manifest. 291 | return mediaTypeDockerSchema1ManifestUnsigned, nil 292 | default: 293 | return "", fmt.Errorf("unsupported schema version %d: %w", manifest.SchemaVersion, ErrInvalidManifest) 294 | } 295 | } 296 | 297 | func (r *ecrResolver) Fetcher(ctx context.Context, ref string) (remotes.Fetcher, error) { 298 | log.G(ctx).WithField("ref", ref).Debug("ecr.resolver.fetcher") 299 | ecrSpec, err := ParseRef(ref) 300 | if err != nil { 301 | return nil, err 302 | } 303 | return &ecrFetcher{ 304 | ecrBase: ecrBase{ 305 | client: r.getClient(ecrSpec.Region()), 306 | ecrSpec: ecrSpec, 307 | }, 308 | parallelism: r.layerDownloadParallelism, 309 | httpClient: r.httpClient, 310 | }, nil 311 | } 312 | 313 | func (r *ecrResolver) Pusher(ctx context.Context, ref string) (remotes.Pusher, error) { 314 | log.G(ctx).WithField("ref", ref).Debug("ecr.resolver.pusher") 315 | ecrSpec, err := ParseRef(ref) 316 | if err != nil { 317 | return nil, err 318 | } 319 | 320 | // References will include a digest when the ref is being pushed to a tag to 321 | // denote *which* digest is the root descriptor in this push. 322 | tag, digest := ecrSpec.TagDigest() 323 | if tag == "" && digest != "" { 324 | log.G(ctx).WithField("ref", ref).Debug("ecr.resolver.pusher: push by digest") 325 | } 326 | 327 | // The root descriptor's digest *must* be provided in order to properly tag 328 | // manifests. A ref string will provide this as of containerd v1.3.0 - 329 | // earlier versions do not provide it. 330 | if digest == "" { 331 | return nil, errors.New("pusher: root descriptor missing from push reference") 332 | } 333 | 334 | return &ecrPusher{ 335 | ecrBase: ecrBase{ 336 | client: r.getClient(ecrSpec.Region()), 337 | ecrSpec: ecrSpec, 338 | }, 339 | tracker: r.tracker, 340 | }, nil 341 | } 342 | -------------------------------------------------------------------------------- /ecr/resolver_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package ecr 17 | 18 | import ( 19 | "context" 20 | "errors" 21 | "testing" 22 | 23 | "github.com/aws/aws-sdk-go/aws" 24 | "github.com/aws/aws-sdk-go/aws/request" 25 | "github.com/aws/aws-sdk-go/awstesting/unit" 26 | "github.com/aws/aws-sdk-go/service/ecr" 27 | "github.com/containerd/containerd/reference" 28 | "github.com/opencontainers/go-digest" 29 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 30 | "github.com/stretchr/testify/assert" 31 | "github.com/stretchr/testify/require" 32 | 33 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr/internal/testdata" 34 | ) 35 | 36 | func TestParseImageManifestMediaType(t *testing.T) { 37 | for _, sample := range []testdata.MediaTypeSample{ 38 | // Docker Schema 1 39 | testdata.WithMediaTypeRemoved(testdata.DockerSchema1Manifest), 40 | testdata.WithMediaTypeRemoved(testdata.DockerSchema1ManifestUnsigned), 41 | // Docker Schema 2 42 | testdata.DockerSchema2Manifest, 43 | testdata.WithMediaTypeRemoved(testdata.DockerSchema2Manifest), 44 | testdata.DockerSchema2ManifestList, 45 | // OCI Image Spec 46 | testdata.OCIImageIndex, 47 | testdata.OCIImageManifest, 48 | // Edge case 49 | testdata.EmptySample, 50 | } { 51 | t.Run(sample.MediaType(), func(t *testing.T) { 52 | t.Logf("content: %s", sample.Content()) 53 | actual, err := parseImageManifestMediaType(context.Background(), sample.Content()) 54 | if sample == testdata.EmptySample { 55 | assert.Error(t, err) 56 | assert.True(t, errors.Is(err, ErrInvalidManifest)) 57 | return 58 | } 59 | require.NoError(t, err) 60 | assert.Equal(t, sample.MediaType(), actual) 61 | }) 62 | } 63 | } 64 | 65 | func TestResolve(t *testing.T) { 66 | // input 67 | expectedRef := "ecr.aws/arn:aws:ecr:fake:123456789012:repository/foo/bar:latest" 68 | 69 | // expected API arguments 70 | expectedRegistryID := "123456789012" 71 | expectedRepository := "foo/bar" 72 | expectedImageTag := "latest" 73 | 74 | // API output 75 | imageDigest := testdata.ImageDigest.String() 76 | imageManifest := `{"schemaVersion": 2, "mediaType": "application/vnd.oci.image.manifest.v1+json"}` 77 | image := &ecr.Image{ 78 | RepositoryName: aws.String(expectedRepository), 79 | ImageId: &ecr.ImageIdentifier{ 80 | ImageDigest: aws.String(imageDigest), 81 | }, 82 | ImageManifest: aws.String(imageManifest), 83 | } 84 | 85 | // expected output 86 | expectedDesc := ocispec.Descriptor{ 87 | Digest: digest.Digest(imageDigest), 88 | MediaType: ocispec.MediaTypeImageManifest, 89 | Size: int64(len(imageManifest)), 90 | } 91 | 92 | fakeClient := &fakeECRClient{ 93 | BatchGetImageFn: func(ctx aws.Context, input *ecr.BatchGetImageInput, opts ...request.Option) (*ecr.BatchGetImageOutput, error) { 94 | assert.Equal(t, expectedRegistryID, aws.StringValue(input.RegistryId)) 95 | assert.Equal(t, expectedRepository, aws.StringValue(input.RepositoryName)) 96 | assert.Equal(t, []*ecr.ImageIdentifier{{ImageTag: aws.String(expectedImageTag)}}, input.ImageIds) 97 | return &ecr.BatchGetImageOutput{Images: []*ecr.Image{image}}, nil 98 | }, 99 | } 100 | resolver := &ecrResolver{ 101 | clients: map[string]ecrAPI{ 102 | "fake": fakeClient, 103 | }, 104 | } 105 | 106 | ref, desc, err := resolver.Resolve(context.Background(), expectedRef) 107 | assert.NoError(t, err) 108 | assert.Equal(t, expectedRef, ref) 109 | assert.Equal(t, expectedDesc, desc) 110 | } 111 | 112 | func TestResolveError(t *testing.T) { 113 | // input 114 | ref := "ecr.aws/arn:aws:ecr:fake:123456789012:repository/foo/bar:latest" 115 | 116 | // expected output 117 | expectedError := errors.New("expected") 118 | 119 | fakeClient := &fakeECRClient{ 120 | BatchGetImageFn: func(aws.Context, *ecr.BatchGetImageInput, ...request.Option) (*ecr.BatchGetImageOutput, error) { 121 | return nil, expectedError 122 | }, 123 | } 124 | resolver := &ecrResolver{ 125 | clients: map[string]ecrAPI{ 126 | "fake": fakeClient, 127 | }, 128 | } 129 | _, _, err := resolver.Resolve(context.Background(), ref) 130 | assert.EqualError(t, err, expectedError.Error()) 131 | } 132 | 133 | func TestResolveNoResult(t *testing.T) { 134 | // input 135 | ref := "ecr.aws/arn:aws:ecr:fake:123456789012:repository/foo/bar:latest" 136 | 137 | fakeClient := &fakeECRClient{ 138 | BatchGetImageFn: func(aws.Context, *ecr.BatchGetImageInput, ...request.Option) (*ecr.BatchGetImageOutput, error) { 139 | return &ecr.BatchGetImageOutput{}, nil 140 | }, 141 | } 142 | resolver := &ecrResolver{ 143 | clients: map[string]ecrAPI{ 144 | "fake": fakeClient, 145 | }, 146 | } 147 | _, _, err := resolver.Resolve(context.Background(), ref) 148 | assert.Error(t, err) 149 | assert.Equal(t, reference.ErrInvalid, err) 150 | } 151 | 152 | func TestResolvePusherAllowsDigest(t *testing.T) { 153 | for _, ref := range []string{ 154 | "ecr.aws/arn:aws:ecr:fake:123456789012:repository/foo/bar@" + testdata.ImageDigest.String(), 155 | } { 156 | t.Run(ref, func(t *testing.T) { 157 | resolver := &ecrResolver{ 158 | clients: map[string]ecrAPI{ 159 | "fake": &fakeECRClient{}, 160 | }, 161 | } 162 | 163 | p, err := resolver.Pusher(context.Background(), ref) 164 | assert.NoError(t, err) 165 | assert.NotNil(t, p) 166 | }) 167 | } 168 | } 169 | 170 | func TestResolvePusherAllowTagDigest(t *testing.T) { 171 | for _, ref := range []string{ 172 | "ecr.aws/arn:aws:ecr:fake:123456789012:repository/foo/bar:with-tag-and-digest@" + testdata.ImageDigest.String(), 173 | } { 174 | t.Run(ref, func(t *testing.T) { 175 | resolver := &ecrResolver{ 176 | // Stub session 177 | session: unit.Session, 178 | clients: map[string]ecrAPI{}, 179 | } 180 | p, err := resolver.Pusher(context.Background(), ref) 181 | assert.NoError(t, err) 182 | assert.NotNil(t, p) 183 | }) 184 | } 185 | } 186 | -------------------------------------------------------------------------------- /ecr/stream/chunked_processor.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | // Package stream contains functionality for processing arbitrarily large 17 | // streaming data. 18 | package stream 19 | 20 | import ( 21 | "context" 22 | "io" 23 | "time" 24 | ) 25 | 26 | // Chunk represents a single part of a full io stream. 27 | type Chunk struct { 28 | Bytes []byte // buffered content 29 | Part int64 // current part of io, starting at 1 30 | BytesBegin int64 // beginning byte range 31 | BytesEnd int64 // ending byte range 32 | ReadTime time.Duration // time spent reading buffer 33 | } 34 | 35 | type chunkedProcessor struct { 36 | ctx context.Context 37 | cancel func() 38 | readChannel chan *Chunk 39 | errorChannel chan error 40 | reader io.Reader 41 | chunkSize int64 42 | queueSize int64 43 | } 44 | 45 | // readCallbackFunc represents a callback function for processing chunks 46 | type readCallbackFunc func(*Chunk) error 47 | 48 | // ChunkedProcessor breaks an io.Reader into smaller parts (Chunks) and invokes 49 | // callbacks on those chunks. 50 | // 51 | // ChunkedProcessor asynchronously reads a io.Reader into at most queueSize 52 | // Chunks of chunkSize at a time. The caller provides a readCallback function 53 | // to handle each read Chunk, which does not block reading. readCallback 54 | // invocations are sequential and the next readCallback will not be invoked 55 | // until the previous has completed. If the queue of Chunks is full, the 56 | // ChunkedProcessor will block waiting until the next readCallback is invoked 57 | // to read from the queued Chunks. 58 | // 59 | // Parameters 60 | // 61 | // reader - the io.Reader to read. 62 | // 63 | // chunkSize - the maximum number of bytes that should be present in each chunk. 64 | // All chunks except the last chunk should be exactly chunkSize. 65 | // 66 | // queueSize - the maximum number of unprocessed chunks to buffer. 67 | // 68 | // readCallback - the callback function to invoke for each chunk. 69 | func ChunkedProcessor(reader io.Reader, chunkSize int64, queueSize int64, readCallback readCallbackFunc) (int64, error) { 70 | ctx, cancel := context.WithCancel(context.Background()) 71 | bufferedReader := &chunkedProcessor{ 72 | ctx: ctx, 73 | cancel: cancel, 74 | readChannel: make(chan *Chunk, queueSize), 75 | errorChannel: make(chan error), 76 | reader: reader, 77 | chunkSize: chunkSize, 78 | queueSize: queueSize, 79 | } 80 | defer close(bufferedReader.errorChannel) 81 | 82 | // Drain the read channel to void leaking the readIntoChunks goroutine. 83 | // When we return with an error out, we may have Chunks that have been read but not yet processed. 84 | defer func() { 85 | i := 0 86 | for range bufferedReader.readChannel { 87 | i++ 88 | } 89 | }() 90 | 91 | go bufferedReader.readIntoChunks() 92 | 93 | return bufferedReader.processChunks(readCallback) 94 | } 95 | 96 | // readIntoChunks begins event loop for reading Chunks. 97 | // On return, either the complete buffer is read (or there is an 98 | // error reading from the buffer) and the readChannel 99 | // 100 | // Can be canceled by canceling the context. 101 | func (processor *chunkedProcessor) readIntoChunks() { 102 | var currentBytes, currentPart int64 103 | defer close(processor.readChannel) 104 | 105 | for { 106 | select { 107 | case <-processor.ctx.Done(): 108 | return 109 | default: 110 | chunk, err := processor.readChunk(currentBytes, currentPart) 111 | if err != nil && err != io.EOF { 112 | processor.errorChannel <- err 113 | return 114 | } 115 | 116 | if chunk != nil { 117 | processor.readChannel <- chunk 118 | currentBytes = chunk.BytesEnd + 1 119 | currentPart++ 120 | } 121 | 122 | if err != nil && err == io.EOF { 123 | return 124 | } 125 | } 126 | } 127 | 128 | } 129 | 130 | // processChunks selects between the read & error channels provided in the 131 | // context and invokes the readCallback with the results on success. 132 | // 133 | // If an error is received in the error channel or from the read callback, 134 | // the function returns and cancels the context. 135 | func (processor *chunkedProcessor) processChunks(readCallback readCallbackFunc) (int64, error) { 136 | defer processor.cancel() 137 | 138 | lastReadByte := int64(0) 139 | eof := false 140 | 141 | for !eof { 142 | select { 143 | case chunk := <-processor.readChannel: 144 | if chunk == nil { 145 | eof = true 146 | break 147 | } 148 | lastReadByte = chunk.BytesEnd 149 | err := readCallback(chunk) 150 | 151 | if err != nil { 152 | return 0, err 153 | } 154 | case err := <-processor.errorChannel: 155 | return 0, err 156 | } 157 | } 158 | 159 | return lastReadByte, nil 160 | } 161 | 162 | // readChunk reads and returns a new Chunk to the caller. 163 | // Given the current part and bytesBegin, populates the new Chunk with 164 | // the proper offsets. Will return nil Chunk if reader is empty. 165 | func (processor *chunkedProcessor) readChunk(bytesBegin int64, part int64) (*Chunk, error) { 166 | startTime := time.Now() 167 | buffer := make([]byte, processor.chunkSize) 168 | size, err := io.ReadFull(processor.reader, buffer) 169 | if err != nil && err != io.ErrUnexpectedEOF && err != io.EOF { 170 | return nil, err 171 | } 172 | 173 | var chunk *Chunk 174 | 175 | if size > 0 { 176 | chunk = &Chunk{ 177 | Part: part, 178 | BytesBegin: bytesBegin, 179 | BytesEnd: bytesBegin + int64(size) - 1, 180 | Bytes: buffer[0:size], 181 | ReadTime: time.Since(startTime), 182 | } 183 | } 184 | 185 | if err == io.ErrUnexpectedEOF { 186 | err = io.EOF 187 | } 188 | 189 | return chunk, err 190 | } 191 | -------------------------------------------------------------------------------- /ecr/stream/chunked_processor_test.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package stream 17 | 18 | import ( 19 | "errors" 20 | "strings" 21 | "testing" 22 | 23 | "github.com/stretchr/testify/assert" 24 | ) 25 | 26 | var testBufferString = []string{"A", "B", "C", "D", "E", "F", "G"} 27 | var testReaderString = "ABCDEFG" 28 | 29 | func TestChunkedProcessorSuccess(t *testing.T) { 30 | var index int 31 | size, err := ChunkedProcessor(strings.NewReader(testReaderString), 1, 10, func(b *Chunk) error { 32 | assert.Equal(t, testBufferString[index], string(b.Bytes)) 33 | index += 1 34 | return nil 35 | }) 36 | assert.Nil(t, err) 37 | assert.Equal(t, int64(6), size) 38 | assert.Equal(t, 7, index) 39 | } 40 | 41 | func TestChunkedProcessorFail(t *testing.T) { 42 | var index int 43 | size, err := ChunkedProcessor(strings.NewReader(testReaderString), 1, 10, func(b *Chunk) error { 44 | index += 1 45 | return errors.New("error") 46 | }) 47 | assert.Error(t, err) 48 | assert.Equal(t, int64(0), size) 49 | assert.Equal(t, 1, index) 50 | } 51 | 52 | func TestChunkedProcessorBlockingFail(t *testing.T) { 53 | var index int 54 | size, err := ChunkedProcessor(strings.NewReader(testReaderString), 1, 2, func(b *Chunk) error { 55 | index += 1 56 | return errors.New("error") 57 | }) 58 | assert.Error(t, err) 59 | assert.Equal(t, int64(0), size) 60 | assert.Equal(t, 1, index) 61 | } 62 | 63 | func TestChunkedProcessorBlockingSuccess(t *testing.T) { 64 | var index int 65 | size, err := ChunkedProcessor(strings.NewReader(testReaderString), 1, 2, func(b *Chunk) error { 66 | assert.Equal(t, testBufferString[index], string(b.Bytes)) 67 | index += 1 68 | return nil 69 | }) 70 | assert.Nil(t, err) 71 | assert.Equal(t, int64(6), size) 72 | assert.Equal(t, 7, index) 73 | } 74 | 75 | var testChunkedString = []string{"ABC", "DEF", "G"} 76 | 77 | func TestChunkedProcessorChunkingSuccess(t *testing.T) { 78 | var index int 79 | size, err := ChunkedProcessor(strings.NewReader(testReaderString), 3, 2, func(b *Chunk) error { 80 | assert.Equal(t, testChunkedString[index], string(b.Bytes)) 81 | index += 1 82 | return nil 83 | }) 84 | assert.Nil(t, err) 85 | assert.Equal(t, int64(6), size) 86 | assert.Equal(t, 3, index) 87 | } 88 | 89 | func TestChunkedProcessorEmptySuccess(t *testing.T) { 90 | var index int 91 | size, err := ChunkedProcessor(strings.NewReader(""), 1, 2, func(b *Chunk) error { 92 | index += 1 93 | return nil 94 | }) 95 | assert.Nil(t, err) 96 | assert.Equal(t, int64(0), size) 97 | assert.Equal(t, 0, index) 98 | } 99 | -------------------------------------------------------------------------------- /example/ecr-copy/main.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package main 17 | 18 | import ( 19 | "context" 20 | "os" 21 | "strconv" 22 | 23 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr" 24 | "github.com/containerd/containerd" 25 | "github.com/containerd/containerd/log" 26 | "github.com/containerd/containerd/namespaces" 27 | "github.com/sirupsen/logrus" 28 | ) 29 | 30 | const ( 31 | // Default to no debug logging. 32 | defaultEnableDebug = 0 33 | ) 34 | 35 | func main() { 36 | ctx := namespaces.NamespaceFromEnv(context.Background()) 37 | 38 | if len(os.Args) < 3 { 39 | log.G(ctx).Fatal("Must provide source and destination as arguments") 40 | } else if len(os.Args) > 3 { 41 | log.G(ctx).Fatal("Must provide only the source and destination as arguments") 42 | } 43 | 44 | sourceRef := os.Args[1] 45 | destRef := os.Args[2] 46 | 47 | enableDebug := defaultEnableDebug 48 | parseEnvInt(ctx, "ECR_COPY_DEBUG", &enableDebug) 49 | if enableDebug == 1 { 50 | log.L.Logger.SetLevel(logrus.TraceLevel) 51 | } 52 | 53 | client, err := containerd.New("/run/containerd/containerd.sock") 54 | if err != nil { 55 | log.G(ctx).WithError(err).Fatal("Failed to connect to containerd") 56 | } 57 | defer client.Close() 58 | 59 | resolver, err := ecr.NewResolver() 60 | if err != nil { 61 | log.G(ctx).WithError(err).Fatal("Failed to create resolver") 62 | } 63 | 64 | log.G(ctx).WithField("sourceRef", sourceRef).Info("Pulling from Amazon ECR") 65 | img, err := client.Fetch( 66 | ctx, 67 | sourceRef, 68 | containerd.WithResolver(resolver), 69 | ) 70 | if err != nil { 71 | log.G(ctx).WithError(err).WithField("sourceRef", sourceRef).Fatal("Failed to pull") 72 | } 73 | log.G(ctx).WithField("img", img.Name).Info("Pulled successfully!") 74 | 75 | log.G(ctx).WithField("sourceRef", sourceRef).WithField("destRef", destRef).Info("Pushing to Amazon ECR") 76 | desc := img.Target 77 | err = client.Push(ctx, destRef, desc, 78 | containerd.WithResolver(resolver), 79 | ) 80 | if err != nil { 81 | log.G(ctx).WithError(err).WithField("destRef", destRef).Fatal("Failed to push") 82 | } 83 | 84 | log.G(ctx).WithField("destRef", destRef).Info("Pushed successfully!") 85 | } 86 | 87 | func parseEnvInt(ctx context.Context, varname string, val *int) { 88 | if varval := os.Getenv(varname); varval != "" { 89 | parsed, err := strconv.Atoi(varval) 90 | if err != nil { 91 | log.G(ctx).WithError(err).Fatalf("Failed to parse %s", varname) 92 | } 93 | *val = parsed 94 | } 95 | } 96 | -------------------------------------------------------------------------------- /example/ecr-pull/main.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package main 17 | 18 | import ( 19 | "context" 20 | "os" 21 | "strconv" 22 | 23 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr" 24 | "github.com/containerd/containerd" 25 | "github.com/containerd/containerd/images" 26 | "github.com/containerd/containerd/log" 27 | "github.com/containerd/containerd/namespaces" 28 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 29 | "github.com/sirupsen/logrus" 30 | ) 31 | 32 | const ( 33 | // Default to no parallel layer downloading. 34 | defaultParallelism = 0 35 | // Default to no debug logging. 36 | defaultEnableDebug = 0 37 | ) 38 | 39 | func main() { 40 | ctx := namespaces.NamespaceFromEnv(context.Background()) 41 | 42 | if len(os.Args) < 2 { 43 | log.G(ctx).Fatal("Must provide image to pull as argument") 44 | } else if len(os.Args) > 2 { 45 | log.G(ctx).Fatal("Must provide only the image to pull") 46 | } 47 | 48 | ref := os.Args[1] 49 | 50 | parallelism := defaultParallelism 51 | parseEnvInt(ctx, "ECR_PULL_PARALLEL", ¶llelism) 52 | 53 | enableDebug := defaultEnableDebug 54 | parseEnvInt(ctx, "ECR_PULL_DEBUG", &enableDebug) 55 | if enableDebug == 1 { 56 | log.L.Logger.SetLevel(logrus.TraceLevel) 57 | } 58 | 59 | address := "/run/containerd/containerd.sock" 60 | if newAddress := os.Getenv("CONTAINERD_ADDRESS"); newAddress != "" { 61 | address = newAddress 62 | } 63 | client, err := containerd.New(address) 64 | if err != nil { 65 | log.G(ctx).WithError(err).Fatal("Failed to connect to containerd") 66 | } 67 | defer client.Close() 68 | 69 | ongoing := newJobs(ref) 70 | pctx, stopProgress := context.WithCancel(ctx) 71 | progress := make(chan struct{}) 72 | go func() { 73 | showProgress(pctx, ongoing, client.ContentStore(), os.Stdout) 74 | close(progress) 75 | }() 76 | 77 | h := images.HandlerFunc(func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) { 78 | if desc.MediaType != images.MediaTypeDockerSchema1Manifest { 79 | ongoing.add(desc) 80 | } 81 | return nil, nil 82 | }) 83 | 84 | resolver, err := ecr.NewResolver(ecr.WithLayerDownloadParallelism(parallelism)) 85 | if err != nil { 86 | log.G(ctx).WithError(err).Fatal("Failed to create resolver") 87 | } 88 | 89 | log.G(ctx).WithField("ref", ref).Info("Pulling from Amazon ECR") 90 | img, err := client.Pull(ctx, ref, 91 | containerd.WithResolver(resolver), 92 | containerd.WithImageHandler(h), 93 | containerd.WithSchema1Conversion) 94 | stopProgress() 95 | if err != nil { 96 | log.G(ctx).WithError(err).WithField("ref", ref).Fatal("Failed to pull") 97 | } 98 | <-progress 99 | log.G(ctx).WithField("img", img.Name()).Info("Pulled successfully!") 100 | if skipUnpack := os.Getenv("ECR_SKIP_UNPACK"); skipUnpack != "" { 101 | return 102 | } 103 | snapshotter := containerd.DefaultSnapshotter 104 | if newSnapshotter := os.Getenv("CONTAINERD_SNAPSHOTTER"); newSnapshotter != "" { 105 | snapshotter = newSnapshotter 106 | } 107 | log.G(ctx). 108 | WithField("img", img.Name()). 109 | WithField("snapshotter", snapshotter). 110 | Info("unpacking...") 111 | err = img.Unpack(ctx, snapshotter) 112 | if err != nil { 113 | log.G(ctx).WithError(err).WithField("img", img.Name).Fatal("Failed to unpack") 114 | } 115 | } 116 | 117 | func parseEnvInt(ctx context.Context, varname string, val *int) { 118 | if varval := os.Getenv(varname); varval != "" { 119 | parsed, err := strconv.Atoi(varval) 120 | if err != nil { 121 | log.G(ctx).WithError(err).Fatalf("Failed to parse %s", varname) 122 | } 123 | *val = parsed 124 | } 125 | } 126 | -------------------------------------------------------------------------------- /example/ecr-pull/progress.go: -------------------------------------------------------------------------------- 1 | // This file is derived from CNCF's containerd project 2 | // The original code may be found : 3 | // https://github.com/containerd/containerd/blob/v1.0.2/cmd/ctr/commands/content/fetch.go#L103-L316 4 | // 5 | // Copyright 2015 The containerd Authors. 6 | // 7 | // Licensed under the Apache License, Version 2.0 (the "License"); 8 | // you may not use this file except in compliance with the License. 9 | // You may obtain a copy of the License at 10 | // 11 | // http://www.apache.org/licenses/LICENSE-2.0 12 | // 13 | // Unless required by applicable law or agreed to in writing, software 14 | // distributed under the License is distributed on an "AS IS" BASIS, 15 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | // See the License for the specific language governing permissions and 17 | // limitations under the License. 18 | // 19 | // Modifications are, where applicable, Copyright 2018 Amazon.com, Inc. or its affiliates. 20 | // Licensed under the Apache License 2.0 21 | 22 | package main 23 | 24 | import ( 25 | "context" 26 | "fmt" 27 | "io" 28 | "sync" 29 | "text/tabwriter" 30 | "time" 31 | 32 | "github.com/containerd/containerd/content" 33 | "github.com/containerd/containerd/errdefs" 34 | "github.com/containerd/containerd/log" 35 | "github.com/containerd/containerd/pkg/progress" 36 | "github.com/containerd/containerd/remotes" 37 | "github.com/docker/go-units" 38 | digest "github.com/opencontainers/go-digest" 39 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 40 | ) 41 | 42 | func showProgress(ctx context.Context, ongoing *jobs, cs content.Store, out io.Writer) { 43 | var ( 44 | ticker = time.NewTicker(100 * time.Millisecond) 45 | fw = progress.NewWriter(out) 46 | start = time.Now() 47 | statuses = map[string]StatusInfo{} 48 | done bool 49 | ) 50 | defer ticker.Stop() 51 | 52 | outer: 53 | for { 54 | select { 55 | case <-ticker.C: 56 | fw.Flush() 57 | 58 | tw := tabwriter.NewWriter(fw, 1, 8, 1, ' ', 0) 59 | 60 | resolved := "resolved" 61 | if !ongoing.isResolved() { 62 | resolved = "resolving" 63 | } 64 | statuses[ongoing.name] = StatusInfo{ 65 | Ref: ongoing.name, 66 | Status: resolved, 67 | } 68 | keys := []string{ongoing.name} 69 | 70 | activeSeen := map[string]struct{}{} 71 | if !done { 72 | active, err := cs.ListStatuses(ctx, "") 73 | if err != nil { 74 | log.G(ctx).WithError(err).Error("active check failed") 75 | continue 76 | } 77 | // update status of active entries! 78 | for _, active := range active { 79 | statuses[active.Ref] = StatusInfo{ 80 | Ref: active.Ref, 81 | Status: "downloading", 82 | Offset: active.Offset, 83 | Total: active.Total, 84 | StartedAt: active.StartedAt, 85 | UpdatedAt: active.UpdatedAt, 86 | } 87 | activeSeen[active.Ref] = struct{}{} 88 | } 89 | } 90 | 91 | // now, update the items in jobs that are not in active 92 | for _, j := range ongoing.jobs() { 93 | key := remotes.MakeRefKey(ctx, j) 94 | keys = append(keys, key) 95 | if _, ok := activeSeen[key]; ok { 96 | continue 97 | } 98 | 99 | status, ok := statuses[key] 100 | if !done && (!ok || status.Status == "downloading") { 101 | info, err := cs.Info(ctx, j.Digest) 102 | if err != nil { 103 | if !errdefs.IsNotFound(err) { 104 | log.G(ctx).WithError(err).Errorf("failed to get content info") 105 | continue outer 106 | } else { 107 | statuses[key] = StatusInfo{ 108 | Ref: key, 109 | Status: "waiting", 110 | } 111 | } 112 | } else if info.CreatedAt.After(start) { 113 | statuses[key] = StatusInfo{ 114 | Ref: key, 115 | Status: "done", 116 | Offset: info.Size, 117 | Total: info.Size, 118 | StartedAt: start, 119 | UpdatedAt: info.CreatedAt, 120 | } 121 | } else { 122 | statuses[key] = StatusInfo{ 123 | Ref: key, 124 | Status: "exists", 125 | } 126 | } 127 | } else if done { 128 | if ok { 129 | if status.Status != "done" && status.Status != "exists" { 130 | status.Status = "done" 131 | statuses[key] = status 132 | } 133 | } else { 134 | statuses[key] = StatusInfo{ 135 | Ref: key, 136 | Status: "done", 137 | } 138 | } 139 | } 140 | } 141 | 142 | var ordered []StatusInfo 143 | for _, key := range keys { 144 | ordered = append(ordered, statuses[key]) 145 | } 146 | 147 | Display(tw, ordered, start) 148 | tw.Flush() 149 | 150 | if done { 151 | fw.Flush() 152 | return 153 | } 154 | case <-ctx.Done(): 155 | done = true // allow ui to update once more 156 | } 157 | } 158 | } 159 | 160 | // jobs provides a way of identifying the download keys for a particular task 161 | // encountering during the pull walk. 162 | // 163 | // This is very minimal and will probably be replaced with something more 164 | // featured. 165 | type jobs struct { 166 | name string 167 | added map[digest.Digest]struct{} 168 | descs []ocispec.Descriptor 169 | mu sync.Mutex 170 | resolved bool 171 | } 172 | 173 | func newJobs(name string) *jobs { 174 | return &jobs{ 175 | name: name, 176 | added: map[digest.Digest]struct{}{}, 177 | } 178 | } 179 | 180 | func (j *jobs) add(desc ocispec.Descriptor) { 181 | j.mu.Lock() 182 | defer j.mu.Unlock() 183 | j.resolved = true 184 | 185 | if _, ok := j.added[desc.Digest]; ok { 186 | return 187 | } 188 | j.descs = append(j.descs, desc) 189 | j.added[desc.Digest] = struct{}{} 190 | } 191 | 192 | func (j *jobs) jobs() []ocispec.Descriptor { 193 | j.mu.Lock() 194 | defer j.mu.Unlock() 195 | 196 | var descs []ocispec.Descriptor 197 | return append(descs, j.descs...) 198 | } 199 | 200 | func (j *jobs) isResolved() bool { 201 | j.mu.Lock() 202 | defer j.mu.Unlock() 203 | return j.resolved 204 | } 205 | 206 | // StatusInfo holds the status info for an upload or download 207 | type StatusInfo struct { 208 | Ref string 209 | Status string 210 | Offset int64 211 | Total int64 212 | StartedAt time.Time 213 | UpdatedAt time.Time 214 | } 215 | 216 | // Display pretty prints out the download or upload progress 217 | func Display(w io.Writer, statuses []StatusInfo, start time.Time) { 218 | var total int64 219 | for _, status := range statuses { 220 | total += status.Offset 221 | switch status.Status { 222 | case "downloading", "uploading": 223 | var bar progress.Bar 224 | if status.Total > 0.0 { 225 | bar = progress.Bar(float64(status.Offset) / float64(status.Total)) 226 | } 227 | fmt.Fprintf(w, "%s:\t%s\t%40r\t%8.8s/%s\t\n", 228 | status.Ref, 229 | status.Status, 230 | bar, 231 | progress.Bytes(status.Offset), progress.Bytes(status.Total)) 232 | case "resolving", "waiting": 233 | bar := progress.Bar(0.0) 234 | fmt.Fprintf(w, "%s:\t%s\t%40r\t\n", 235 | status.Ref, 236 | status.Status, 237 | bar) 238 | case "done": 239 | if status.Total > 0.0 { 240 | bar := progress.Bar(1.0) 241 | duration := status.UpdatedAt.Sub(status.StartedAt) 242 | fmt.Fprintf(w, "%s:\t%s\t%40r\t%s\t%s\t%s\t\n", 243 | status.Ref, 244 | status.Status, 245 | bar, 246 | progress.Bytes(status.Total), 247 | units.HumanDuration(duration), 248 | progress.NewBytesPerSecond(status.Total, duration)) 249 | continue 250 | } 251 | fallthrough 252 | default: 253 | bar := progress.Bar(1.0) 254 | fmt.Fprintf(w, "%s:\t%s\t%40r\t\n", 255 | status.Ref, 256 | status.Status, 257 | bar) 258 | } 259 | } 260 | 261 | fmt.Fprintf(w, "elapsed: %-4.1fs\ttotal: %7.6v\t(%v)\t\n", 262 | time.Since(start).Seconds(), 263 | // TODO(stevvooe): These calculations are actually way off. 264 | // Need to account for previously downloaded data. These 265 | // will basically be right for a download the first time 266 | // but will be skewed if restarting, as it includes the 267 | // data into the start time before. 268 | progress.Bytes(total), 269 | progress.NewBytesPerSecond(total, time.Since(start))) 270 | } 271 | -------------------------------------------------------------------------------- /example/ecr-push/main.go: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You 5 | * may not use this file except in compliance with the License. A copy of 6 | * the License is located at 7 | * 8 | * http://aws.amazon.com/apache2.0/ 9 | * 10 | * or in the "license" file accompanying this file. This file is 11 | * distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 12 | * ANY KIND, either express or implied. See the License for the specific 13 | * language governing permissions and limitations under the License. 14 | */ 15 | 16 | package main 17 | 18 | import ( 19 | "context" 20 | "fmt" 21 | "os" 22 | "strconv" 23 | "text/tabwriter" 24 | "time" 25 | 26 | "github.com/awslabs/amazon-ecr-containerd-resolver/ecr" 27 | "github.com/containerd/containerd" 28 | "github.com/containerd/containerd/images" 29 | "github.com/containerd/containerd/log" 30 | "github.com/containerd/containerd/namespaces" 31 | "github.com/containerd/containerd/pkg/progress" 32 | "github.com/containerd/containerd/remotes" 33 | "github.com/containerd/containerd/remotes/docker" 34 | ocispec "github.com/opencontainers/image-spec/specs-go/v1" 35 | "github.com/sirupsen/logrus" 36 | "golang.org/x/sync/errgroup" 37 | ) 38 | 39 | const ( 40 | // Default to no debug logging. 41 | defaultEnableDebug = 0 42 | ) 43 | 44 | func main() { 45 | ctx := namespaces.NamespaceFromEnv(context.Background()) 46 | //logrus.SetLevel(logrus.DebugLevel) 47 | 48 | if len(os.Args) < 2 { 49 | log.G(ctx).Fatal("Must provide image to push as argument") 50 | } 51 | ref := os.Args[1] 52 | local := "" 53 | if len(os.Args) > 2 { 54 | local = os.Args[2] 55 | } else { 56 | local = ref 57 | } 58 | 59 | enableDebug := defaultEnableDebug 60 | parseEnvInt(ctx, "ECR_PUSH_DEBUG", &enableDebug) 61 | if enableDebug == 1 { 62 | log.L.Logger.SetLevel(logrus.TraceLevel) 63 | } 64 | 65 | client, err := containerd.New("/run/containerd/containerd.sock") 66 | if err != nil { 67 | log.G(ctx).WithError(err).Fatal("Failed to connect to containerd") 68 | } 69 | defer client.Close() 70 | 71 | tracker := docker.NewInMemoryTracker() 72 | resolver, err := ecr.NewResolver(ecr.WithTracker(tracker)) 73 | if err != nil { 74 | log.G(ctx).WithError(err).Fatal("Failed to create resolver") 75 | } 76 | 77 | img, err := client.ImageService().Get(ctx, local) 78 | if err != nil { 79 | fmt.Println(err) 80 | os.Exit(1) 81 | } 82 | 83 | ongoing := newPushJobs(tracker) 84 | eg, ctx := errgroup.WithContext(ctx) 85 | eg.Go(func() error { 86 | log.G(ctx).WithField("local", local).WithField("ref", ref).Info("Pushing to Amazon ECR") 87 | desc := img.Target 88 | 89 | jobHandler := images.HandlerFunc(func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) { 90 | ongoing.add(remotes.MakeRefKey(ctx, desc)) 91 | return nil, nil 92 | }) 93 | 94 | return client.Push(ctx, ref, desc, 95 | containerd.WithResolver(resolver), 96 | containerd.WithImageHandler(jobHandler)) 97 | }) 98 | errs := make(chan error) 99 | go func() { 100 | defer close(errs) 101 | errs <- eg.Wait() 102 | }() 103 | 104 | err = displayUploadProgress(ctx, ongoing, errs) 105 | if err != nil { 106 | log.G(ctx).WithError(err).WithField("ref", ref).Fatal("Failed to push") 107 | } 108 | log.G(ctx).WithField("ref", ref).Info("Pushed successfully!") 109 | } 110 | 111 | func displayUploadProgress(ctx context.Context, ongoing *pushjobs, errs chan error) error { 112 | var ( 113 | ticker = time.NewTicker(100 * time.Millisecond) 114 | fw = progress.NewWriter(os.Stdout) 115 | start = time.Now() 116 | done bool 117 | ) 118 | defer ticker.Stop() 119 | 120 | for { 121 | select { 122 | case <-ticker.C: 123 | fw.Flush() 124 | 125 | tw := tabwriter.NewWriter(fw, 1, 8, 1, ' ', 0) 126 | 127 | display(tw, ongoing.status(), start) 128 | tw.Flush() 129 | 130 | if done { 131 | fw.Flush() 132 | return nil 133 | } 134 | case err := <-errs: 135 | if err != nil { 136 | return err 137 | } 138 | done = true 139 | case <-ctx.Done(): 140 | done = true // allow ui to update once more 141 | } 142 | } 143 | } 144 | 145 | func parseEnvInt(ctx context.Context, varname string, val *int) { 146 | if varval := os.Getenv(varname); varval != "" { 147 | parsed, err := strconv.Atoi(varval) 148 | if err != nil { 149 | log.G(ctx).WithError(err).Fatalf("Failed to parse %s", varname) 150 | } 151 | *val = parsed 152 | } 153 | } 154 | -------------------------------------------------------------------------------- /example/ecr-push/progress.go: -------------------------------------------------------------------------------- 1 | // This file is derived from CNCF's containerd project 2 | // The original code may be found : 3 | // https://github.com/containerd/containerd/blob/v1.0.2/cmd/ctr/commands/content/push.go#L136-L194 4 | // https://github.com/containerd/containerd/blob/v1.0.2/cmd/ctr/commands/content/fetch.go#L103-L316 5 | // 6 | // Copyright 2015 The containerd Authors. 7 | // 8 | // Licensed under the Apache License, Version 2.0 (the "License"); 9 | // you may not use this file except in compliance with the License. 10 | // You may obtain a copy of the License at 11 | // 12 | // http://www.apache.org/licenses/LICENSE-2.0 13 | // 14 | // Unless required by applicable law or agreed to in writing, software 15 | // distributed under the License is distributed on an "AS IS" BASIS, 16 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | // See the License for the specific language governing permissions and 18 | // limitations under the License. 19 | // 20 | // Modifications are, where applicable, Copyright 2018 Amazon.com, Inc. or its affiliates. 21 | // Licensed under the Apache License 2.0 22 | 23 | package main 24 | 25 | import ( 26 | "fmt" 27 | "io" 28 | "sync" 29 | "time" 30 | 31 | "github.com/containerd/containerd/pkg/progress" 32 | "github.com/containerd/containerd/remotes/docker" 33 | units "github.com/docker/go-units" 34 | ) 35 | 36 | type pushjobs struct { 37 | jobs map[string]struct{} 38 | ordered []string 39 | tracker docker.StatusTracker 40 | mu sync.Mutex 41 | } 42 | 43 | func newPushJobs(tracker docker.StatusTracker) *pushjobs { 44 | return &pushjobs{ 45 | jobs: make(map[string]struct{}), 46 | tracker: tracker, 47 | } 48 | } 49 | 50 | func (j *pushjobs) add(ref string) { 51 | j.mu.Lock() 52 | defer j.mu.Unlock() 53 | 54 | if _, ok := j.jobs[ref]; ok { 55 | return 56 | } 57 | j.ordered = append(j.ordered, ref) 58 | j.jobs[ref] = struct{}{} 59 | } 60 | 61 | func (j *pushjobs) status() []StatusInfo { 62 | j.mu.Lock() 63 | defer j.mu.Unlock() 64 | 65 | statuses := make([]StatusInfo, 0, len(j.jobs)) 66 | for _, name := range j.ordered { 67 | si := StatusInfo{ 68 | Ref: name, 69 | } 70 | 71 | status, err := j.tracker.GetStatus(name) 72 | if err != nil { 73 | si.Status = "waiting" 74 | } else { 75 | si.Offset = status.Offset 76 | si.Total = status.Total 77 | si.StartedAt = status.StartedAt 78 | si.UpdatedAt = status.UpdatedAt 79 | if status.Offset >= status.Total { 80 | if status.UploadUUID == "" { 81 | si.Status = "done" 82 | } else { 83 | si.Status = "committing" 84 | } 85 | } else { 86 | si.Status = "uploading" 87 | } 88 | } 89 | statuses = append(statuses, si) 90 | } 91 | 92 | return statuses 93 | } 94 | 95 | // StatusInfo holds the status info for an upload or download 96 | type StatusInfo struct { 97 | Ref string 98 | Status string 99 | Offset int64 100 | Total int64 101 | StartedAt time.Time 102 | UpdatedAt time.Time 103 | } 104 | 105 | // Display pretty prints out the download or upload progress 106 | func display(w io.Writer, statuses []StatusInfo, start time.Time) { 107 | var total int64 108 | for _, status := range statuses { 109 | total += status.Offset 110 | switch status.Status { 111 | case "downloading", "uploading": 112 | var bar progress.Bar 113 | if status.Total > 0.0 { 114 | bar = progress.Bar(float64(status.Offset) / float64(status.Total)) 115 | } 116 | fmt.Fprintf(w, "%s:\t%s\t%40r\t%8.8s/%s\t\n", 117 | status.Ref, 118 | status.Status, 119 | bar, 120 | progress.Bytes(status.Offset), progress.Bytes(status.Total)) 121 | case "resolving", "waiting": 122 | bar := progress.Bar(0.0) 123 | fmt.Fprintf(w, "%s:\t%s\t%40r\t\n", 124 | status.Ref, 125 | status.Status, 126 | bar) 127 | case "done": 128 | if status.Total > 0.0 { 129 | bar := progress.Bar(1.0) 130 | duration := status.UpdatedAt.Sub(status.StartedAt) 131 | fmt.Fprintf(w, "%s:\t%s\t%40r\t%s\t%s\t%s\t\n", 132 | status.Ref, 133 | status.Status, 134 | bar, 135 | progress.Bytes(status.Total), 136 | units.HumanDuration(duration), 137 | progress.NewBytesPerSecond(status.Total, duration)) 138 | continue 139 | } 140 | fallthrough 141 | default: 142 | bar := progress.Bar(1.0) 143 | fmt.Fprintf(w, "%s:\t%s\t%40r\t\n", 144 | status.Ref, 145 | status.Status, 146 | bar) 147 | } 148 | } 149 | 150 | fmt.Fprintf(w, "elapsed: %-4.1fs\ttotal: %7.6v\t(%v)\t\n", 151 | time.Since(start).Seconds(), 152 | // TODO(stevvooe): These calculations are actually way off. 153 | // Need to account for previously downloaded data. These 154 | // will basically be right for a download the first time 155 | // but will be skewed if restarting, as it includes the 156 | // data into the start time before. 157 | progress.Bytes(total), 158 | progress.NewBytesPerSecond(total, time.Since(start))) 159 | } 160 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/awslabs/amazon-ecr-containerd-resolver 2 | 3 | go 1.23.0 4 | 5 | require ( 6 | github.com/aws/aws-sdk-go v1.55.7 7 | github.com/containerd/containerd v1.6.26 8 | github.com/docker/go-units v0.5.0 9 | github.com/htcat/htcat v1.0.2 10 | github.com/opencontainers/go-digest v1.0.0 11 | github.com/opencontainers/image-spec v1.1.1 12 | github.com/sirupsen/logrus v1.9.3 13 | github.com/stretchr/testify v1.10.0 14 | golang.org/x/net v0.40.0 15 | golang.org/x/sync v0.14.0 16 | ) 17 | 18 | require ( 19 | github.com/Microsoft/go-winio v0.5.2 // indirect 20 | github.com/Microsoft/hcsshim v0.9.10 // indirect 21 | github.com/containerd/cgroups v1.0.4 // indirect 22 | github.com/containerd/console v1.0.3 // indirect 23 | github.com/containerd/continuity v0.3.0 // indirect 24 | github.com/containerd/fifo v1.0.0 // indirect 25 | github.com/containerd/log v0.1.0 // indirect 26 | github.com/containerd/ttrpc v1.1.2 // indirect 27 | github.com/containerd/typeurl v1.0.2 // indirect 28 | github.com/davecgh/go-spew v1.1.1 // indirect 29 | github.com/docker/go-events v0.0.0-20190806004212-e31b211e4f1c // indirect 30 | github.com/gogo/googleapis v1.4.1 // indirect 31 | github.com/gogo/protobuf v1.3.2 // indirect 32 | github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect 33 | github.com/golang/protobuf v1.5.3 // indirect 34 | github.com/google/uuid v1.3.0 // indirect 35 | github.com/jmespath/go-jmespath v0.4.0 // indirect 36 | github.com/klauspost/compress v1.15.9 // indirect 37 | github.com/moby/locker v1.0.1 // indirect 38 | github.com/moby/sys/mountinfo v0.6.2 // indirect 39 | github.com/moby/sys/signal v0.6.0 // indirect 40 | github.com/opencontainers/runc v1.1.14 // indirect 41 | github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417 // indirect 42 | github.com/opencontainers/selinux v1.10.1 // indirect 43 | github.com/pkg/errors v0.9.1 // indirect 44 | github.com/pmezard/go-difflib v1.0.0 // indirect 45 | go.opencensus.io v0.24.0 // indirect 46 | golang.org/x/sys v0.33.0 // indirect 47 | golang.org/x/text v0.25.0 // indirect 48 | google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98 // indirect 49 | google.golang.org/grpc v1.58.3 // indirect 50 | google.golang.org/protobuf v1.33.0 // indirect 51 | gopkg.in/yaml.v3 v3.0.1 // indirect 52 | ) 53 | --------------------------------------------------------------------------------