├── .github └── workflows │ ├── ci.yml │ └── golangci-lint.yml ├── .gitignore ├── .golangci.yml ├── LICENCE ├── README.md ├── cmd └── nix_casync │ └── main.go ├── go.mod ├── go.sum ├── pkg ├── server │ ├── compression │ │ ├── compressor.go │ │ └── decompressor.go │ ├── server.go │ └── server_test.go ├── store │ ├── blobstore │ │ ├── blobstore_test.go │ │ ├── casync_store.go │ │ ├── casync_store_nar_reader.go │ │ ├── casync_store_nar_writer.go │ │ ├── memory_store.go │ │ └── types.go │ └── metadatastore │ │ ├── file_store.go │ │ ├── memory_store.go │ │ ├── metadata_store_test.go │ │ └── types.go └── util │ ├── helpers.go │ └── helpers_test.go └── test ├── 7cwx623saf2h3z23wsn26icszvskk4iy.narinfo ├── fixtures.go ├── generator └── default.nix ├── nar ├── 0rcdxyw7kjpxshv7wb1am0nvjfjbjq67cvrc8dmbsy1slc2ycbxp.nar ├── 0xmvxmsmmc6n79sk2h3r6db3yp8drmxps61mdk7iqnvc6vcsww60.nar └── 0z2vk40phzzgsg14516mfs79l9fvl276b993mlqlb4rf0fd7hnwp.nar ├── qp5h1cjd5ykcl4hyvsjhrlv68bbx8fan.narinfo └── x236iz9shqypbnm64qgqisz0jr4wmj2b.narinfo /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | pull_request: 8 | 9 | jobs: 10 | build: 11 | runs-on: ubuntu-20.04 12 | strategy: 13 | matrix: 14 | go: ['1.16', '1.17'] 15 | name: Build (Go ${{ matrix.go }}) 16 | steps: 17 | - uses: actions/checkout@v2 18 | - uses: actions/setup-go@v2 19 | with: 20 | go-version: ${{ matrix.go }} 21 | - name: go test 22 | run: go test -v ./... 23 | - name: go build 24 | run: go build ./cmd/... 25 | -------------------------------------------------------------------------------- /.github/workflows/golangci-lint.yml: -------------------------------------------------------------------------------- 1 | name: golangci-lint 2 | on: 3 | push: 4 | tags: 5 | - v* 6 | branches: 7 | - master 8 | - main 9 | pull_request: 10 | permissions: 11 | contents: read 12 | # Optional: allow read access to pull request. Use with `only-new-issues` option. 13 | # pull-requests: read 14 | jobs: 15 | golangci: 16 | name: lint 17 | runs-on: ubuntu-latest 18 | steps: 19 | - uses: actions/checkout@v2 20 | - name: golangci-lint 21 | uses: golangci/golangci-lint-action@v3 22 | with: 23 | # Optional: version of golangci-lint to use in form of v1.2 or v1.2.3 or `latest` to use the latest version 24 | version: v1.44.2 25 | 26 | # Optional: working directory, useful for monorepos 27 | # working-directory: somedir 28 | 29 | # Optional: golangci-lint command line arguments. 30 | # args: --issues-exit-code=0 31 | 32 | # Optional: show only new issues if it's a pull request. The default value is `false`. 33 | # only-new-issues: true 34 | 35 | # Optional: if set to true then the action will use pre-installed Go. 36 | # skip-go-installation: true 37 | 38 | # Optional: if set to true then the action don't cache or restore ~/go/pkg. 39 | # skip-pkg-cache: true 40 | 41 | # Optional: if set to true then the action don't cache or restore ~/.cache/go-build. 42 | # skip-build-cache: true 43 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /nix_casync 2 | -------------------------------------------------------------------------------- /.golangci.yml: -------------------------------------------------------------------------------- 1 | run: 2 | timeout: 5m 3 | 4 | linters: 5 | enable: 6 | - errname 7 | - exhaustive 8 | - gci 9 | - gochecknoglobals 10 | - gochecknoinits 11 | - goconst 12 | - gocritic 13 | - godot 14 | - gofumpt 15 | - goheader 16 | - goimports 17 | - gosec 18 | - ifshort 19 | - importas 20 | - ireturn 21 | - lll 22 | - makezero 23 | - misspell 24 | - nakedret 25 | - nestif 26 | - nilerr 27 | - nilnil 28 | - nlreturn 29 | - noctx 30 | - nolintlint 31 | - prealloc 32 | - predeclared 33 | - revive 34 | - rowserrcheck 35 | - stylecheck 36 | - tagliatelle 37 | - tenv 38 | - testpackage 39 | - unconvert 40 | - unparam 41 | - wastedassign 42 | - whitespace 43 | - wsl 44 | -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | Copyright 2021 Florian Klink 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | > :warning: **This project is archived**: This was an experiment, and most of the findings make up their way into the Tvix Store protocol. Check [my blog](https://flokli.de/posts/) and [Tvix](https://code.tvl.fyi/log/tvix) for more updates! 2 | 3 | # nix-casync 4 | A more efficient way to store and substitute Nix store paths. 5 | 6 | Docs are a bit sparse right now, please refer to 7 | https://flokli.de/posts/2021-12-10-nix-casync-intro/ for a description 8 | on how this works. 9 | 10 | ## Build 11 | 12 | ```sh 13 | $ go build ./cmd/nix_casync/ 14 | ``` 15 | 16 | ## Run 17 | ```sh 18 | ./nix_casync serve --cache-path=path/to/local 19 | ``` 20 | 21 | ### Uploading store paths 22 | ``` 23 | nix copy \ 24 | --extra-experimental-features nix-command \ 25 | --to "http://localhost:9000?compression=none" $storePath 26 | ``` 27 | 28 | ### Binary Cache 29 | As of now, `nix-casync` can be used as a space-efficient binary cache. 30 | 31 | You probably want to put some reverse proxy doing SSL in front of it, and add 32 | some protection on the `PUT` endpoints. 33 | 34 | The following section describes some internal behaviour of `nix-casync`, and 35 | how it treats Narfiles and Narinfo files. 36 | 37 | #### Narfiles 38 | Narfiles can be uploaded with most of the compression mechanisms Nix supports. 39 | 40 | The path it's uploaded at `HTTP PUT /nar/….nar[.$suffix]` doesn't really matter. 41 | 42 | Files will be decompressed, chunked, and put in a content-addressed store. 43 | 44 | Subsequently uploaded `.narinfo` files can refer to that file via the `NarHash` 45 | attribute, and downloads can happen via `HTTP GET /nar/$narhash.nar[.$suffix]`. 46 | 47 | For downloads, only a subset of compression algorithms (fast ones) are 48 | supported, as those are assembled on the fly and should really only be 49 | considered a poor-man's Content-Encoding. 50 | 51 | ##### Another note on compression 52 | While it's possible to upload with narfile compression, as written above, this 53 | is only used to compress *uploads* into the binary cache. 54 | 55 | This will have some unintuitive implications - if you upload a to 56 | `/nar/$filehash.nar.zst`, the upload won't be available on that location (but 57 | at `/nar/$narhash.nar`). 58 | 59 | This also means `HTTP HEAD` requests to "compressed locations" will `404`, and 60 | as a result, Nix clients might end up uploading the same `.nar` files multiple 61 | times. [^1] 62 | 63 | If the binary cache is remote, it is preferable to use compression during 64 | upload to reduce bandwidth usage. In that case, using a fast compression 65 | algorithm, such as `zstd` is recommended. 66 | 67 | By default, downloads are served with ZSTD Compression. This can be tweaked via 68 | the `--nar-compression` command line parameter. 69 | 70 | #### Narinfo files 71 | Narinfo files describe information about a store path, as well as some 72 | (redundant) information about the referred .nar file. 73 | 74 | Internally, `nix-casync` splits data from `.narinfo` into `NarMeta` and 75 | `PathInfo` models. 76 | 77 | When a Narfile is uploaded, the following checks are made: 78 | 79 | - The .narinfo file refers to a Narfile (via NarHash) that already exists in 80 | `nix-casync`. 81 | - The `References` field matches with what `nix-casync`'s internal bookkeeping 82 | of References in `NarMeta` matches. 83 | Right now, that field is populated on the first `.narinfo` upload , but as 84 | it's possible to determine the references just by looking at the `.nar` file 85 | itself, a reference scanner could be added to `nix-casync` directly. 86 | - All `References` in the uploaded `.narinfo` refer to `PathInfo` (aka 87 | - `.narinfo` files) that were already uploaded to `nix-casync`. 88 | 89 | 90 | 91 | [^1]: Nix won't upload the same store path multiple times, as it checks 92 | `$outhash.narinfo` for existence first - so this only applies to multiple 93 | `.narinfo` files referring to the same `.nar` file. 94 | -------------------------------------------------------------------------------- /cmd/nix_casync/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "net/http" 5 | "os" 6 | "os/signal" 7 | "path" 8 | "time" 9 | 10 | "github.com/alecthomas/kong" 11 | "github.com/flokli/nix-casync/pkg/server" 12 | "github.com/flokli/nix-casync/pkg/store/blobstore" 13 | "github.com/flokli/nix-casync/pkg/store/metadatastore" 14 | "github.com/go-chi/chi/middleware" 15 | log "github.com/sirupsen/logrus" 16 | ) 17 | 18 | var CLI struct { //nolint:gochecknoglobals 19 | Serve struct { 20 | CachePath string `name:"cache-path" help:"Path to use for a local cache, containing castr, caibx and narinfo files." type:"path" default:"/var/cache/nix-casync"` //nolint:lll 21 | NarCompression string `name:"nar-compression" help:"The compression algorithm to advertise .nar files with (zstd,gzip,brotli,none)" enum:"zstd,gzip,brotli,none" type:"string" default:"zstd"` //nolint:lll 22 | ListenAddr string `name:"listen-addr" help:"The address this service listens on" type:"string" default:"[::]:9000"` //nolint:lll 23 | Priority int `name:"priority" help:"What priority to advertise in nix-cache-info. Defaults to 40." type:"int" default:"40"` //nolint:lll 24 | AvgChunkSize int `name:"avg-chunk-size" help:"The average chunking size to use when chunking NAR files, in bytes. Max is 4 times that, Min is a quarter of this value." type:"int" default:"65536"` //nolint:lll 25 | AccessLog bool `name:"access-log" help:"Enable access logging" type:"bool" default:"true" negatable:""` //nolint:lll 26 | } `cmd:"" serve:"Serve a local nix cache."` 27 | } 28 | 29 | func main() { 30 | retcode := 0 31 | 32 | defer func() { os.Exit(retcode) }() 33 | 34 | ctx := kong.Parse(&CLI) 35 | switch ctx.Command() { 36 | case "serve": 37 | // initialize casync store 38 | castrPath := path.Join(CLI.Serve.CachePath, "castr") 39 | caibxPath := path.Join(CLI.Serve.CachePath, "caibx") 40 | 41 | blobStore, err := blobstore.NewCasyncStore(castrPath, caibxPath, CLI.Serve.AvgChunkSize) 42 | if err != nil { 43 | log.Errorf("Error initializing blobstore: %v", err) 44 | 45 | retcode = -1 46 | 47 | return 48 | } 49 | 50 | // initialize narinfo store 51 | narinfoPath := path.Join(CLI.Serve.CachePath, "narinfo") 52 | 53 | metadataStore, err := metadatastore.NewFileStore(narinfoPath) 54 | if err != nil { 55 | log.Errorf("Error initializing metadatastore: %v", err) 56 | 57 | retcode = -1 58 | 59 | return 60 | } 61 | 62 | s := server.NewServer(blobStore, metadataStore, CLI.Serve.NarCompression, CLI.Serve.Priority) 63 | defer s.Close() 64 | 65 | c := make(chan os.Signal, 1) 66 | signal.Notify(c, os.Interrupt) 67 | 68 | go func() { 69 | for range c { 70 | log.Info("Received Signal, shutting down…") 71 | s.Close() 72 | os.Exit(1) 73 | } 74 | }() 75 | 76 | log.Printf("Starting Server at %v", CLI.Serve.ListenAddr) 77 | 78 | srv := &http.Server{ 79 | Addr: CLI.Serve.ListenAddr, 80 | Handler: s.Handler, 81 | ReadTimeout: 50 * time.Second, 82 | WriteTimeout: 100 * time.Second, 83 | IdleTimeout: 150 * time.Second, 84 | } 85 | 86 | if CLI.Serve.AccessLog { 87 | srv.Handler = middleware.Logger(s.Handler) 88 | } 89 | 90 | err = srv.ListenAndServe() 91 | if err != nil { 92 | log.Errorf("Error listening: %v", err) 93 | 94 | retcode = 1 95 | 96 | return 97 | } 98 | default: 99 | panic(ctx.Command()) 100 | } 101 | } 102 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/flokli/nix-casync 2 | 3 | go 1.16 4 | 5 | require ( 6 | github.com/alecthomas/kong v0.5.0 7 | github.com/andybalholm/brotli v1.0.4 8 | github.com/folbricht/desync v0.9.0 9 | github.com/frankban/quicktest v1.14.0 // indirect 10 | github.com/go-chi/chi v1.5.4 11 | github.com/go-chi/chi/v5 v5.0.7 12 | github.com/klauspost/compress v1.15.3 13 | github.com/nix-community/go-nix v0.0.0-20220502083308-687fc4730510 14 | github.com/pierrec/lz4 v2.6.1+incompatible 15 | github.com/sirupsen/logrus v1.8.1 16 | github.com/stretchr/testify v1.7.0 17 | github.com/ulikunitz/xz v0.5.10 18 | ) 19 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 2 | cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 3 | cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= 4 | cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= 5 | cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= 6 | cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= 7 | cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= 8 | cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To= 9 | cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4= 10 | cloud.google.com/go v0.53.0 h1:MZQCQQaRwOrAcuKjiHWHrgKykt4fZyuwF2dtiG3fGW8= 11 | cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M= 12 | cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= 13 | cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= 14 | cloud.google.com/go/bigquery v1.4.0 h1:xE3CPsOgttP4ACBePh79zTKALtXwn/Edhcr16R5hMWU= 15 | cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= 16 | cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= 17 | cloud.google.com/go/datastore v1.1.0 h1:/May9ojXjRkPBNVrq+oWLqmWCkr4OU5uRY29bu0mRyQ= 18 | cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= 19 | cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= 20 | cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= 21 | cloud.google.com/go/pubsub v1.2.0 h1:Lpy6hKgdcl7a3WGSfJIFmxmcdjSpP6OmBEfcOv1Y680= 22 | cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= 23 | cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= 24 | cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos= 25 | cloud.google.com/go/storage v1.6.0 h1:UDpwYIwla4jHGzZJaEJYx1tOejbgSoNqsAfHAUYe2r8= 26 | cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= 27 | dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= 28 | github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= 29 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 30 | github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= 31 | github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= 32 | github.com/alecthomas/kong v0.5.0 h1:u8Kdw+eeml93qtMZ04iei0CFYve/WPcA5IFh+9wSskE= 33 | github.com/alecthomas/kong v0.5.0/go.mod h1:uzxf/HUh0tj43x1AyJROl3JT7SgsZ5m+icOv1csRhc0= 34 | github.com/alecthomas/participle/v2 v2.0.0-alpha7/go.mod h1:NumScqsC42o9x+dGj8/YqsIfhrIQjFEOFovxotbBirA= 35 | github.com/alecthomas/repr v0.0.0-20181024024818-d37bc2a10ba1/go.mod h1:xTS7Pm1pD1mvyM075QCDSRqH6qRLXylzS24ZTpRiSzQ= 36 | github.com/alecthomas/repr v0.0.0-20210801044451-80ca428c5142 h1:8Uy0oSf5co/NZXje7U1z8Mpep++QJOldL2hs/sBQf48= 37 | github.com/alecthomas/repr v0.0.0-20210801044451-80ca428c5142/go.mod h1:2kn6fqh/zIyPLmm3ugklbEi5hg5wS435eygvNfaDQL8= 38 | github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= 39 | github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= 40 | github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY= 41 | github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig= 42 | github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= 43 | github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= 44 | github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= 45 | github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= 46 | github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= 47 | github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= 48 | github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= 49 | github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= 50 | github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= 51 | github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= 52 | github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= 53 | github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= 54 | github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= 55 | github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= 56 | github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= 57 | github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= 58 | github.com/datadog/zstd v1.4.5 h1:PW1WRRmJ8wBxKoaY3HVedZ3k8Pfw4eAJ22yTXutS8cg= 59 | github.com/datadog/zstd v1.4.5/go.mod h1:inRp+etsHuvVqMPNTXaFlpf/Tj7wqviBtdJoPVrPEFQ= 60 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 61 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 62 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 63 | github.com/dchest/siphash v1.2.1 h1:4cLinnzVJDKxTCl9B01807Yiy+W7ZzVHj/KIroQRvT4= 64 | github.com/dchest/siphash v1.2.1/go.mod h1:q+IRvb2gOSrUnYoPqHiyHXS0FOBBOdl6tONBlVnOnt4= 65 | github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= 66 | github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no= 67 | github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= 68 | github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= 69 | github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= 70 | github.com/folbricht/desync v0.9.0 h1:8RZldLMc8IqkVOJszOPFu1cCUL4yCxl/WNPbR211gQ0= 71 | github.com/folbricht/desync v0.9.0/go.mod h1:GxC/D68FxhG5exY3eS7CQGjEvolwD23hvM9j22qvjCU= 72 | github.com/folbricht/tempfile v0.0.1 h1:kB3DubP2Fm5e3W7TrWFNZBfzFEHBoKL7Pjn0HvqKxSQ= 73 | github.com/folbricht/tempfile v0.0.1/go.mod h1:/Flpxx/6U+clQJ61jQ3y6Z7L2l6j1/ZSiU4B9EDPgWw= 74 | github.com/frankban/quicktest v1.14.0 h1:+cqqvzZV87b4adx/5ayVOaYZ2CrvM4ejQvUdBzPPUss= 75 | github.com/frankban/quicktest v1.14.0/go.mod h1:NeW+ay9A/U67EYXNFA1nPE8e/tnQv/09mUdL/ijj8og= 76 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 77 | github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= 78 | github.com/go-chi/chi v1.5.4 h1:QHdzF2szwjqVV4wmByUnTcsbIg7UGaQ0tPF2t5GcAIs= 79 | github.com/go-chi/chi v1.5.4/go.mod h1:uaf8YgoFazUOkPBG7fxPftUylNumIev9awIWOENIuEg= 80 | github.com/go-chi/chi/v5 v5.0.7 h1:rDTPXLDHGATaeHvVlLcR4Qe0zftYethFucbjVQ1PxU8= 81 | github.com/go-chi/chi/v5 v5.0.7/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8= 82 | github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= 83 | github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= 84 | github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= 85 | github.com/go-ini/ini v1.57.0 h1:Qwzj3wZQW+Plax5Ntj+GYe07DfGj1OH+aL1nMTMaNow= 86 | github.com/go-ini/ini v1.57.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8= 87 | github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= 88 | github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= 89 | github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= 90 | github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= 91 | github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= 92 | github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= 93 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= 94 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= 95 | github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 96 | github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 97 | github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 98 | github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY= 99 | github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= 100 | github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 101 | github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 102 | github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= 103 | github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= 104 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 105 | github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 106 | github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 107 | github.com/golang/protobuf v1.3.3 h1:gyjaxf+svBWX08ZjK86iN9geUJF0H6gp2IRKX6Nf6/I= 108 | github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= 109 | github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= 110 | github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= 111 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= 112 | github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 113 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 114 | github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 115 | github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 116 | github.com/google/go-cmp v0.5.6 h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ= 117 | github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 118 | github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no= 119 | github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= 120 | github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= 121 | github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= 122 | github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= 123 | github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= 124 | github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= 125 | github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= 126 | github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM= 127 | github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= 128 | github.com/gopherjs/gopherjs v0.0.0-20180825215210-0210a2f0f73c h1:16eHWuMGvCjSfgRJKqIzapE78onvvTbdi1rMkU00lZw= 129 | github.com/gopherjs/gopherjs v0.0.0-20180825215210-0210a2f0f73c/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= 130 | github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= 131 | github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= 132 | github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk= 133 | github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= 134 | github.com/hanwen/go-fuse v1.0.0 h1:GxS9Zrn6c35/BnfiVsZVWmsG803xwE7eVRDvcf/BEVc= 135 | github.com/hanwen/go-fuse v1.0.0/go.mod h1:unqXarDXqzAk0rt98O2tVndEPIpUgLD9+rwFisZH3Ok= 136 | github.com/hanwen/go-fuse/v2 v2.0.3 h1:kpV28BKeSyVgZREItBLnaVBvOEwv2PuhNdKetwnvNHo= 137 | github.com/hanwen/go-fuse/v2 v2.0.3/go.mod h1:0EQM6aH2ctVpvZ6a+onrQ/vaykxh2GH7hy3e13vzTUY= 138 | github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= 139 | github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= 140 | github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= 141 | github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= 142 | github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= 143 | github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo= 144 | github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= 145 | github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o= 146 | github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= 147 | github.com/jtolds/gls v4.2.1+incompatible h1:fSuqC+Gmlu6l/ZYAoZzx2pyucC8Xza35fpRVWLVmUEE= 148 | github.com/jtolds/gls v4.2.1+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= 149 | github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= 150 | github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= 151 | github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= 152 | github.com/klauspost/compress v1.10.10/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= 153 | github.com/klauspost/compress v1.15.3 h1:wmfu2iqj9q22SyMINp1uQ8C2/V4M1phJdmH9fG4nba0= 154 | github.com/klauspost/compress v1.15.3/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU= 155 | github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= 156 | github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= 157 | github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8= 158 | github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg= 159 | github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= 160 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 161 | github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= 162 | github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= 163 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 164 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 165 | github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= 166 | github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= 167 | github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348 h1:MtvEpTB6LX3vkb4ax0b5D2DHbNAUsen0Gx5wZoq3lV4= 168 | github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k= 169 | github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= 170 | github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= 171 | github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= 172 | github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= 173 | github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= 174 | github.com/minio/minio-go v6.0.14+incompatible h1:fnV+GD28LeqdN6vT2XdGKW8Qe/IfjJDswNVuni6km9o= 175 | github.com/minio/minio-go v6.0.14+incompatible/go.mod h1:7guKYtitv8dktvNUGrhzmNlA5wrAABTQXCoesZdFQO8= 176 | github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= 177 | github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= 178 | github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= 179 | github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= 180 | github.com/nix-community/go-nix v0.0.0-20220502083308-687fc4730510 h1:jBjiTUjswaq7SG+2vOJAVT1rl+lLmu1YR90bmSHzP44= 181 | github.com/nix-community/go-nix v0.0.0-20220502083308-687fc4730510/go.mod h1:r5pCQAjHNSDWTsy6+UbacPN8EzMVJiCAXDWbRN2qfwU= 182 | github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U= 183 | github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= 184 | github.com/pierrec/lz4 v2.6.1+incompatible h1:9UY3+iC23yxF0UfGaYrGplQ+79Rg+h/q9FV9ix19jjM= 185 | github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= 186 | github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 187 | github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 188 | github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= 189 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 190 | github.com/pkg/sftp v1.11.0 h1:4Zv0OGbpkg4yNuUtH0s8rvoYxRCNyT29NVUo6pgPmxI= 191 | github.com/pkg/sftp v1.11.0/go.mod h1:lYOWFsE0bwd1+KfKJaKeuokY15vzFx25BLbzYYoAxZI= 192 | github.com/pkg/xattr v0.4.1 h1:dhclzL6EqOXNaPDWqoeb9tIxATfBSmjqL0b4DpSjwRw= 193 | github.com/pkg/xattr v0.4.1/go.mod h1:W2cGD0TBEus7MkUgv0tNZ9JutLtVO3cXu+IBRuHqnFs= 194 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 195 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 196 | github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= 197 | github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= 198 | github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= 199 | github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 200 | github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 201 | github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= 202 | github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= 203 | github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= 204 | github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= 205 | github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU= 206 | github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg= 207 | github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= 208 | github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k= 209 | github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc= 210 | github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= 211 | github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= 212 | github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= 213 | github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= 214 | github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE= 215 | github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= 216 | github.com/smartystreets/assertions v0.0.0-20180820201707-7c9eb446e3cf h1:6V1qxN6Usn4jy8unvggSJz/NC790tefw8Zdy6OZS5co= 217 | github.com/smartystreets/assertions v0.0.0-20180820201707-7c9eb446e3cf/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= 218 | github.com/smartystreets/goconvey v0.0.0-20180222194500-ef6db91d284a h1:JSvGDIbmil4Ui/dDdFBExb7/cmkNjyX5F97oglmvCDo= 219 | github.com/smartystreets/goconvey v0.0.0-20180222194500-ef6db91d284a/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s= 220 | github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= 221 | github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= 222 | github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= 223 | github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= 224 | github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE= 225 | github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= 226 | github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= 227 | github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= 228 | github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE= 229 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 230 | github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 231 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 232 | github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 233 | github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 234 | github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY= 235 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 236 | github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= 237 | github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= 238 | github.com/ulikunitz/xz v0.5.10 h1:t92gobL9l3HE202wg3rlk19F6X+JOxl9BBrCCMYEYd8= 239 | github.com/ulikunitz/xz v0.5.10/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= 240 | github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= 241 | github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= 242 | go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= 243 | go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= 244 | go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= 245 | go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= 246 | go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8= 247 | go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= 248 | go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= 249 | go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= 250 | go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= 251 | golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= 252 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 253 | golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 254 | golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 255 | golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 256 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 257 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= 258 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 259 | golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= 260 | golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= 261 | golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= 262 | golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= 263 | golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY= 264 | golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= 265 | golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= 266 | golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= 267 | golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= 268 | golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6 h1:QE6XYQK6naiK1EPAe1g/ILLxN5RBoH5xkJk3CqlMI/Y= 269 | golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= 270 | golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= 271 | golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= 272 | golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 273 | golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= 274 | golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 275 | golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 276 | golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 277 | golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 278 | golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 279 | golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= 280 | golang.org/x/lint v0.0.0-20200130185559-910be7a94367 h1:0IiAsCRByjO2QjX7ZPkw5oU9x+n1YqRL802rjC0c3Aw= 281 | golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= 282 | golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= 283 | golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= 284 | golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= 285 | golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= 286 | golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= 287 | golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= 288 | golang.org/x/mod v0.2.0 h1:KU7oHjnv3XNWfa5COkzUifxZmxp1TyI7ImMXqFxLwvQ= 289 | golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 290 | golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 291 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 292 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 293 | golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 294 | golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 295 | golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 296 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 297 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 298 | golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 299 | golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 300 | golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= 301 | golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= 302 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 303 | golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 304 | golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 305 | golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 306 | golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 307 | golang.org/x/net v0.0.0-20200222125558-5a598a2470a0 h1:MsuvTghUPjX762sGLnGsxC3HM0B5r83wEtYcYR8/vRs= 308 | golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 309 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= 310 | golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 311 | golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 312 | golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 313 | golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw= 314 | golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= 315 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 316 | golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 317 | golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 318 | golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 319 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 320 | golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 321 | golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208 h1:qwRHBd0NqMbJxfbotnDhm2ByMI1Shq4Y6oRJo21SGJA= 322 | golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 323 | golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 324 | golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 325 | golang.org/x/sys v0.0.0-20181021155630-eda9bb28ed51/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 326 | golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 327 | golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 328 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 329 | golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 330 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 331 | golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 332 | golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 333 | golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 334 | golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 335 | golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 336 | golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 337 | golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 338 | golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 339 | golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 340 | golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 341 | golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 342 | golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 343 | golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 344 | golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 345 | golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae h1:/WDfKMnPU+m5M4xB+6x4kaepxRw6jWvR5iDRdvjHgy8= 346 | golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 347 | golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 348 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 349 | golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 350 | golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= 351 | golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= 352 | golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 353 | golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 354 | golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 355 | golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 356 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 357 | golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 358 | golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= 359 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 360 | golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 361 | golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 362 | golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 363 | golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 364 | golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 365 | golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 366 | golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 367 | golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 368 | golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 369 | golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 370 | golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 371 | golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 372 | golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 373 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 374 | golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 375 | golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 376 | golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 377 | golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 378 | golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 379 | golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 380 | golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 381 | golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 382 | golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 383 | golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 384 | golang.org/x/tools v0.0.0-20200224181240-023911ca70b2 h1:L/G4KZvrQn7FWLN/LlulBtBzrLUhqjiGfTWWDmrh+IQ= 385 | golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 386 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 387 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 388 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= 389 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 390 | google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= 391 | google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M= 392 | google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= 393 | google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= 394 | google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= 395 | google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= 396 | google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= 397 | google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= 398 | google.golang.org/api v0.18.0 h1:TgDr+1inK2XVUKZx3BYAqQg/GwucGdBkzZjWaTg/I+A= 399 | google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= 400 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= 401 | google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 402 | google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 403 | google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0= 404 | google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM= 405 | google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= 406 | google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= 407 | google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 408 | google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 409 | google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 410 | google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= 411 | google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= 412 | google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= 413 | google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8= 414 | google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= 415 | google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= 416 | google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= 417 | google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= 418 | google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= 419 | google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= 420 | google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA= 421 | google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= 422 | google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63 h1:YzfoEYWbODU5Fbt37+h7X16BWQbad7Q4S6gclTKFXM8= 423 | google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= 424 | google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= 425 | google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= 426 | google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= 427 | google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= 428 | google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= 429 | google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 430 | google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 431 | google.golang.org/grpc v1.27.1 h1:zvIju4sqAGvwKspUQOhwnpcqSbzi7/H6QomNNjTL4sk= 432 | google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 433 | gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= 434 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 435 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= 436 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 437 | gopkg.in/cheggaaa/pb.v1 v1.0.28/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw= 438 | gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= 439 | gopkg.in/ini.v1 v1.57.0 h1:9unxIsFcTt4I55uWluz+UmL95q4kdJ0buvQ1ZIqVQww= 440 | gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= 441 | gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo= 442 | gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74= 443 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 444 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 445 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 446 | gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo= 447 | gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 448 | honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 449 | honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 450 | honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 451 | honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 452 | honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= 453 | honnef.co/go/tools v0.0.1-2020.1.3 h1:sXmLre5bzIR6ypkjXCDI3jHPssRhc8KD/Ome589sc3U= 454 | honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= 455 | rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= 456 | rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0= 457 | rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= 458 | -------------------------------------------------------------------------------- /pkg/server/compression/compressor.go: -------------------------------------------------------------------------------- 1 | package compression 2 | 3 | import ( 4 | "compress/gzip" 5 | "fmt" 6 | "io" 7 | 8 | "github.com/andybalholm/brotli" 9 | "github.com/klauspost/compress/zstd" 10 | ) 11 | 12 | // NewCompressor returns an io.WriteCloser that compresses its input. 13 | // The compression type needs to be specified upfront. 14 | // Only cheap compression is supported, as this is assembled on the fly, and acts as a poorman's content-encoding. 15 | // It's the callers responsibility to close the reader when done. 16 | func NewCompressor(w io.Writer, compressionType string) (io.WriteCloser, error) { 17 | switch compressionType { 18 | case "br": 19 | b := brotli.NewWriterLevel(w, brotli.BestSpeed) 20 | 21 | return b, nil 22 | case "gzip": 23 | return gzip.NewWriterLevel(w, gzip.BestSpeed) 24 | case "zstd": 25 | return zstd.NewWriter(w, zstd.WithEncoderLevel(zstd.SpeedDefault)) 26 | } 27 | 28 | return nil, fmt.Errorf("unsupported compression type: %v", compressionType) 29 | } 30 | 31 | // NewCompressorBySuffix returns an io.WriteCloser that compresses its input. 32 | func NewCompressorBySuffix(w io.Writer, compressionSuffix string) (io.WriteCloser, error) { 33 | // try to lookup the compression type from compressionSuffixToType 34 | if compressionType, ok := compressionSuffixToType[compressionSuffix]; ok { 35 | return NewCompressor(w, compressionType) 36 | } 37 | 38 | return nil, fmt.Errorf("unknown compression suffix: %v", compressionSuffix) 39 | } 40 | -------------------------------------------------------------------------------- /pkg/server/compression/decompressor.go: -------------------------------------------------------------------------------- 1 | package compression 2 | 3 | import ( 4 | "compress/bzip2" 5 | "compress/gzip" 6 | "fmt" 7 | "io" 8 | 9 | "github.com/andybalholm/brotli" 10 | "github.com/klauspost/compress/zstd" 11 | "github.com/pierrec/lz4" 12 | "github.com/ulikunitz/xz" 13 | ) 14 | 15 | // compressionSuffixToType maps from the compression suffix Nix uses when uploading to the compression type. 16 | var compressionSuffixToType = map[string]string{ //nolint:gochecknoglobals 17 | "": "none", 18 | ".br": "br", 19 | ".bz2": "bzip2", 20 | ".gz": "gzip", // keep in mind nix defaults to gzip if Compression: field is unset or empty string 21 | ".lz4": "lz4", 22 | ".lzip": "lzip", 23 | ".xz": "xz", 24 | ".zst": "zstd", 25 | } 26 | 27 | func TypeToSuffix(compressionType string) (string, error) { 28 | for compressionSuffix, aCompressionType := range compressionSuffixToType { 29 | if aCompressionType == compressionType { 30 | return compressionSuffix, nil 31 | } 32 | } 33 | 34 | return "", fmt.Errorf("unknown compression type: %v", compressionType) 35 | } 36 | 37 | // NewDecompressor decompresses contents from an io.Reader 38 | // The compression type needs to be specified upfront. 39 | // It's the callers responsibility to close the reader when done. 40 | func NewDecompressor(r io.Reader, compressionType string) (io.ReadCloser, error) { 41 | // Nix seems to support the following compressions: 42 | // - none 43 | // - br 44 | // - bzip2, compress, grzip, gzip, lrzip, lz4, lzip, lzma, lzop, xz, zstd (via libarchive) 45 | switch compressionType { 46 | case "none": 47 | return io.NopCloser(r), nil 48 | case "br": 49 | return io.NopCloser(brotli.NewReader(r)), nil 50 | case "bzip2": 51 | return io.NopCloser(bzip2.NewReader(r)), nil 52 | case "gzip": 53 | gzipReader, err := gzip.NewReader(r) 54 | if err != nil { 55 | return nil, err 56 | } 57 | 58 | return gzipReader, nil 59 | case "lz4": 60 | return io.NopCloser(lz4.NewReader(r)), nil 61 | case "xz": 62 | xzReader, err := xz.NewReader(r) 63 | if err != nil { 64 | return nil, err 65 | } 66 | 67 | return io.NopCloser(xzReader), nil 68 | case "zstd": 69 | zstdr, err := zstd.NewReader(r) 70 | if err != nil { 71 | return nil, err 72 | } 73 | 74 | return io.NopCloser(zstdr), nil 75 | } 76 | 77 | // compress, grzip, lzrzip, lzip, lzop, lzma 78 | return nil, fmt.Errorf("unsupported compression type: %v", compressionType) 79 | } 80 | 81 | func NewDecompressorBySuffix(r io.Reader, compressionSuffix string) (io.ReadCloser, error) { 82 | // try to lookup the compression type from compressionSuffixToType 83 | if compressionType, ok := compressionSuffixToType[compressionSuffix]; ok { 84 | return NewDecompressor(r, compressionType) 85 | } 86 | 87 | return nil, fmt.Errorf("unknown compression suffix: %v", compressionSuffix) 88 | } 89 | -------------------------------------------------------------------------------- /pkg/server/server.go: -------------------------------------------------------------------------------- 1 | package server 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "io" 7 | "net/http" 8 | "os" 9 | 10 | "github.com/flokli/nix-casync/pkg/server/compression" 11 | "github.com/flokli/nix-casync/pkg/store/blobstore" 12 | "github.com/flokli/nix-casync/pkg/store/metadatastore" 13 | "github.com/go-chi/chi/v5" 14 | "github.com/nix-community/go-nix/pkg/nar/narinfo" 15 | "github.com/nix-community/go-nix/pkg/nixbase32" 16 | log "github.com/sirupsen/logrus" 17 | ) 18 | 19 | type Server struct { 20 | Handler *chi.Mux 21 | 22 | blobStore blobstore.BlobStore 23 | metadataStore metadatastore.MetadataStore 24 | 25 | narServeCompression string // zstd,gzip,brotli,none 26 | 27 | io.Closer 28 | } 29 | 30 | func NewServer(blobStore blobstore.BlobStore, 31 | metadataStore metadatastore.MetadataStore, 32 | narServeCompression string, 33 | priority int, 34 | ) *Server { 35 | r := chi.NewRouter() 36 | r.Get("/", func(w http.ResponseWriter, r *http.Request) { 37 | _, err := w.Write([]byte("nix-casync")) 38 | if err != nil { 39 | log.Errorf("Unable to write response: %v", err) 40 | } 41 | }) 42 | 43 | r.Get("/nix-cache-info", func(w http.ResponseWriter, r *http.Request) { 44 | _, err := w.Write([]byte(fmt.Sprintf("StoreDir: /nix/store\nWantMassQuery: 1\nPriority: %d\n", priority))) 45 | if err != nil { 46 | log.Errorf("Unable to write response: %v", err) 47 | } 48 | }) 49 | 50 | s := &Server{ 51 | Handler: r, 52 | blobStore: blobStore, 53 | metadataStore: metadataStore, 54 | narServeCompression: narServeCompression, 55 | } 56 | 57 | s.RegisterNarHandlers() 58 | s.RegisterNarinfoHandlers() 59 | 60 | return s 61 | } 62 | 63 | func (s *Server) Close() error { 64 | if err := s.blobStore.Close(); err != nil { 65 | return err 66 | } 67 | 68 | return s.metadataStore.Close() 69 | } 70 | 71 | func (s *Server) RegisterNarinfoHandlers() { 72 | pattern := "/{outputhash:^[" + nixbase32.Alphabet + "]{32}}.narinfo" 73 | s.Handler.Get(pattern, s.handleNarinfo) 74 | s.Handler.Head(pattern, s.handleNarinfo) 75 | s.Handler.Put(pattern, s.handleNarinfo) 76 | } 77 | 78 | func (s *Server) handleNarinfo(w http.ResponseWriter, r *http.Request) { 79 | outputhashStr := chi.URLParam(r, "outputhash") 80 | 81 | outputhash, err := nixbase32.DecodeString(outputhashStr) 82 | if err != nil { 83 | http.Error(w, fmt.Sprintf("Unable to decode outputhash: %v", err), http.StatusBadRequest) 84 | } 85 | 86 | //nolint:nestif 87 | if r.Method == http.MethodGet || r.Method == http.MethodHead { 88 | // get PathInfo 89 | pathInfo, err := s.metadataStore.GetPathInfo(r.Context(), outputhash) 90 | if err != nil { 91 | status := http.StatusInternalServerError 92 | if errors.Is(err, os.ErrNotExist) { 93 | status = http.StatusNotFound 94 | } 95 | 96 | http.Error(w, fmt.Sprintf("Error getting PathInfo: %v", err), status) 97 | 98 | return 99 | } 100 | 101 | // get NarMeta 102 | narMeta, err := s.metadataStore.GetNarMeta(r.Context(), pathInfo.NarHash) 103 | if err != nil { 104 | // if we can't retrieve the NarMeta, that's a inconsistency. 105 | log.Errorf( 106 | "Unable to find NarMeta for NarHash %s, referenced in PathInfo %s", 107 | nixbase32.EncodeToString(pathInfo.NarHash), 108 | nixbase32.EncodeToString(pathInfo.OutputHash), 109 | ) 110 | http.Error(w, fmt.Sprintf("Error getting NarMeta: %v", err), http.StatusInternalServerError) 111 | } 112 | 113 | narinfoContent, err := metadatastore.RenderNarinfo(pathInfo, narMeta, s.narServeCompression) 114 | if err != nil { 115 | http.Error(w, fmt.Sprintf("Unable to render .narinfo: %v", err), http.StatusInternalServerError) 116 | 117 | return 118 | } 119 | 120 | w.Header().Add("Content-Type", "text/x-nix-narinfo") 121 | w.Header().Add("Content-Length", fmt.Sprintf("%d", len(narinfoContent))) 122 | 123 | _, err = w.Write([]byte(narinfoContent)) 124 | if err != nil { 125 | log.Errorf("Unable to write narinfo contents: %v", err) 126 | } 127 | 128 | return 129 | } 130 | 131 | //nolint:nestif 132 | if r.Method == http.MethodPut { 133 | ni, err := narinfo.Parse(r.Body) 134 | if err != nil { 135 | log.Errorf("Error parsing .narinfo: %v", err) 136 | http.Error(w, fmt.Sprintf("Error parsing .narinfo: %v", err), http.StatusBadRequest) 137 | 138 | return 139 | } 140 | 141 | // retrieve the NarMeta 142 | narMeta, err := s.metadataStore.GetNarMeta(r.Context(), ni.NarHash.Digest) 143 | if errors.Is(err, os.ErrNotExist) { 144 | log.Error("Rejected uploading a .narinfo pointing to a non-existent narhash") 145 | http.Error(w, "Narinfo points to non-existent NarHash", http.StatusBadRequest) 146 | 147 | return 148 | } 149 | 150 | // Parse the .narinfo into a PathInfo and NarMeta struct 151 | sentPathInfo, sentNarMeta, err := metadatastore.ParseNarinfo(ni) 152 | if err != nil { 153 | log.Errorf("Unable to parse narinfo into PathInfo and NarMeta: %v", err) 154 | http.Error(w, "Unable to parse narinfo into PathInfo and NarMeta: %v", http.StatusBadRequest) 155 | } 156 | 157 | // Compare narMeta generated out of the .narinfo with the one in the store 158 | if !narMeta.IsEqualTo(sentNarMeta, false) { 159 | log.Error("Sent .narinfo with conflicting NarMeta") 160 | http.Error(w, "NarMeta is conflicting", http.StatusBadRequest) 161 | } 162 | 163 | // HACK: until we implement our own reference scanner on NAR upload, we 164 | // populate NarMeta.References[Str] on .narinfo upload, 165 | // if it's empty right now. 166 | if len(narMeta.References) == 0 && len(sentNarMeta.References) != 0 { 167 | // we need to persist PathInfo first, so PutNarMeta won't trip on the not-yet-existing PathInfo 168 | err = s.metadataStore.PutPathInfo(r.Context(), sentPathInfo) 169 | if err != nil { 170 | log.Errorf("Error putting PathInfo: %v", err) 171 | http.Error(w, fmt.Sprintf("Error putting PathInfo: %v", err), http.StatusInternalServerError) 172 | 173 | return 174 | } 175 | 176 | narMeta.ReferencesStr = sentNarMeta.ReferencesStr 177 | narMeta.References = sentNarMeta.References 178 | 179 | err = s.metadataStore.PutNarMeta(r.Context(), narMeta) 180 | if err != nil { 181 | log.Errorf("Failed to update NarMeta with References from pathinfo %v: %v", sentPathInfo.Name, err) 182 | http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) 183 | 184 | return 185 | } 186 | } else { 187 | // Do full comparison of NarMeta, including references 188 | if !narMeta.IsEqualTo(sentNarMeta, true) { 189 | log.Error("Sent .narinfo with conflicting NarMeta (References)") 190 | http.Error(w, "NarMeta (References) is conflicting", http.StatusBadRequest) 191 | } 192 | 193 | err = s.metadataStore.PutPathInfo(r.Context(), sentPathInfo) 194 | if err != nil { 195 | http.Error(w, fmt.Sprintf("Error putting PathInfo: %v", err), http.StatusInternalServerError) 196 | 197 | return 198 | } 199 | } 200 | 201 | return 202 | } 203 | 204 | http.Error(w, http.StatusText(http.StatusMethodNotAllowed), http.StatusMethodNotAllowed) 205 | } 206 | 207 | func (s *Server) RegisterNarHandlers() { 208 | patternPlain := "/nar/{narhash:^[" + nixbase32.Alphabet + "]{52}$}.nar" 209 | patternCompressed := patternPlain + `{compressionSuffix:^(\.\w+)$}` 210 | 211 | s.Handler.Get(patternPlain, s.handleNar) 212 | s.Handler.Head(patternPlain, s.handleNar) 213 | s.Handler.Get(patternCompressed, s.handleNar) 214 | s.Handler.Head(patternCompressed, s.handleNar) 215 | 216 | // When Nix uploads compressed paths (if compression=none is not set), 217 | // we simply can't know if a file exists or not. 218 | // Nix uploads (and checks for existence of) /nar/$filehash.nar.$compressionType, 219 | // not /nar/$narhash.nar.$compressionType (which is what we use) 220 | // We content-hash the decompressed contents and discard the compressed uploaded payload, 221 | // so there's no way to know if /nar/$filehash.nar.$compressionType was uploaded 222 | // This means we will return 404 whenever Nix tries to upload a compressed NAR file 223 | // This will cause Nix to unnecessarily upload Narfiles multiple times. 224 | // It's not as bad as it sounds, as this only affects multiple Narinfo files 225 | // referencing the same Narfile (and Nix might locally cache the fact it already uploaded 226 | // that Narfile) 227 | 228 | s.Handler.Put(patternPlain, s.handleNar) 229 | s.Handler.Put(patternCompressed, s.handleNar) 230 | } 231 | 232 | func (s *Server) handleNar(w http.ResponseWriter, r *http.Request) { 233 | //nolint:nestif 234 | if r.Method == http.MethodGet || r.Method == http.MethodHead { 235 | narhashStr := chi.URLParam(r, "narhash") 236 | 237 | narhash, err := nixbase32.DecodeString(narhashStr) 238 | if err != nil { 239 | http.Error(w, fmt.Sprintf("Unable to decode narHash %v: %v", narhashStr, err), http.StatusBadRequest) 240 | } 241 | 242 | blobReader, size, err := s.blobStore.GetBlob(r.Context(), narhash) 243 | if err != nil { 244 | status := http.StatusInternalServerError 245 | if errors.Is(err, os.ErrNotExist) { 246 | status = http.StatusNotFound 247 | } 248 | 249 | http.Error(w, fmt.Sprintf("Error retrieving narfile with hash %v: %v", narhashStr, err), status) 250 | 251 | return 252 | } 253 | defer blobReader.Close() 254 | 255 | // check compression suffix, and serve a compressed file depending on that. 256 | compressionSuffix := chi.URLParam(r, "compressionSuffix") 257 | 258 | var writer io.Writer = w 259 | 260 | if compressionSuffix != "" { 261 | // We only support zstd, gzip, brotli and none, as the others are way too CPU-intensive, 262 | // and never advertised anyways. 263 | compressedWriter, err := compression.NewCompressorBySuffix(w, compressionSuffix) 264 | if err != nil { 265 | // We still serve a 404 (as Nix might send a HEAD request while trying to upload xz, for example) 266 | http.Error(w, fmt.Sprintf("Unsupported compression suffix: %v", compressionSuffix), http.StatusNotFound) 267 | } 268 | defer compressedWriter.Close() 269 | writer = compressedWriter 270 | } 271 | 272 | w.Header().Add("Content-Type", "application/x-nix-nar") 273 | 274 | // We can only advertise a content-length when we serve uncompressed Narfiles 275 | if compressionSuffix == "" { 276 | w.Header().Add("Content-Length", fmt.Sprintf("%d", size)) 277 | } 278 | 279 | _, err = io.Copy(writer, blobReader) 280 | 281 | if err != nil { 282 | log.Errorf("Error sending Narfile to client: %v", err) 283 | 284 | return 285 | } 286 | 287 | return 288 | } 289 | 290 | //nolint:nestif 291 | if r.Method == http.MethodPut { 292 | blobWriter, err := s.blobStore.PutBlob(r.Context()) 293 | if err != nil { 294 | http.Error(w, fmt.Sprintf("Error initializing blobWriter: %v", err), http.StatusInternalServerError) 295 | 296 | return 297 | } 298 | defer blobWriter.Close() 299 | 300 | // There might be suffixes indicating compression, wrap the request body via the generic decompressor 301 | reader, err := compression.NewDecompressorBySuffix(r.Body, chi.URLParam(r, "compressionSuffix")) 302 | if err != nil { 303 | http.Error(w, fmt.Sprintf("Error initializing decompressor: %v", err), http.StatusInternalServerError) 304 | } 305 | 306 | // copy the body of the request into blobWriter 307 | _, err = io.Copy(blobWriter, reader) 308 | if err != nil { 309 | http.Error(w, fmt.Sprintf("Error copying to blobWriter: %v", err), http.StatusInternalServerError) 310 | } 311 | 312 | err = blobWriter.Close() 313 | if err != nil { 314 | http.Error(w, fmt.Sprintf("Error closing blobWriter: %v", err), http.StatusInternalServerError) 315 | } 316 | 317 | // Check if that NarMeta already exists 318 | _, err = s.metadataStore.GetNarMeta(r.Context(), blobWriter.Sha256Sum()) 319 | if err != nil { 320 | if !errors.Is(err, os.ErrNotExist) { 321 | http.Error(w, fmt.Sprintf("Error checking for existing NarMeta: %v", err), http.StatusInternalServerError) 322 | 323 | return 324 | } 325 | // We don't have this NarMeta yet, store it. 326 | narMeta := &metadatastore.NarMeta{ 327 | NarHash: blobWriter.Sha256Sum(), 328 | Size: blobWriter.BytesWritten(), 329 | // TODO: Scan for references, add them here instead of filling on the first .narinfo file upload 330 | } 331 | 332 | err = s.metadataStore.PutNarMeta(r.Context(), narMeta) 333 | if err != nil { 334 | http.Error(w, fmt.Sprintf("Error putting NarMeta: %v", err), http.StatusInternalServerError) 335 | 336 | return 337 | } 338 | } 339 | 340 | // We already had that NarMeta, nothing to be done 341 | return 342 | } 343 | 344 | http.Error(w, http.StatusText(http.StatusMethodNotAllowed), http.StatusMethodNotAllowed) 345 | } 346 | -------------------------------------------------------------------------------- /pkg/server/server_test.go: -------------------------------------------------------------------------------- 1 | package server_test 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "fmt" 7 | "io" 8 | "net/http" 9 | "net/http/httptest" 10 | "testing" 11 | 12 | "github.com/flokli/nix-casync/pkg/server" 13 | "github.com/flokli/nix-casync/pkg/server/compression" 14 | "github.com/flokli/nix-casync/pkg/store/blobstore" 15 | "github.com/flokli/nix-casync/pkg/store/metadatastore" 16 | "github.com/flokli/nix-casync/pkg/util" 17 | "github.com/flokli/nix-casync/test" 18 | "github.com/klauspost/compress/zstd" 19 | "github.com/nix-community/go-nix/pkg/nar/narinfo" 20 | "github.com/nix-community/go-nix/pkg/nixbase32" 21 | "github.com/stretchr/testify/assert" 22 | ) 23 | 24 | // TestHandler tests the handler. 25 | func TestHandler(t *testing.T) { 26 | blobStore := blobstore.NewMemoryStore() 27 | defer blobStore.Close() 28 | 29 | metadataStore := metadatastore.NewMemoryStore() 30 | defer metadataStore.Close() 31 | 32 | server := server.NewServer(blobStore, metadataStore, "zstd", 40) 33 | 34 | testDataT := test.GetTestDataTable() 35 | 36 | tdA, exists := testDataT["a"] 37 | if !exists { 38 | panic("testData[a] doesn't exist") 39 | } 40 | 41 | tdAOutputHash, err := util.GetHashFromStorePath(tdA.Narinfo.StorePath) 42 | if !exists { 43 | panic(err) 44 | } 45 | 46 | tdB, exists := testDataT["b"] 47 | if !exists { 48 | panic("testData[b] doesn't exist") 49 | } 50 | 51 | tdBOutputHash, err := util.GetHashFromStorePath(tdB.Narinfo.StorePath) 52 | if !exists { 53 | panic(err) 54 | } 55 | 56 | tdC, exists := testDataT["c"] 57 | if !exists { 58 | panic("testData[c] doesn't exist") 59 | } 60 | 61 | tdCOutputHash, err := util.GetHashFromStorePath(tdC.Narinfo.StorePath) 62 | if !exists { 63 | panic(err) 64 | } 65 | 66 | t.Run("Nar tests", func(t *testing.T) { 67 | narpath := "/nar/" + nixbase32.EncodeToString(tdA.Narinfo.NarHash.Digest) + ".nar" 68 | 69 | t.Run("GET non-existent .nar", func(t *testing.T) { 70 | rr := httptest.NewRecorder() 71 | ctx, cancel := context.WithCancel(context.Background()) 72 | defer cancel() 73 | 74 | req, err := http.NewRequestWithContext(ctx, "GET", narpath, nil) 75 | if err != nil { 76 | t.Fatal(err) 77 | } 78 | server.Handler.ServeHTTP(rr, req) 79 | assert.Equal(t, http.StatusNotFound, rr.Result().StatusCode) 80 | }) 81 | 82 | t.Run("PUT .nar", func(t *testing.T) { 83 | rr := httptest.NewRecorder() 84 | req, err := http.NewRequest("PUT", narpath, bytes.NewReader(tdA.NarContents)) 85 | if err != nil { 86 | t.Fatal(err) 87 | } 88 | 89 | server.Handler.ServeHTTP(rr, req) 90 | 91 | // expect status to be ok 92 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 93 | 94 | // expect body to be empty 95 | actualContents, err := io.ReadAll(rr.Result().Body) 96 | if err != nil { 97 | t.Fatal(err) 98 | } 99 | 100 | assert.Equal(t, []byte{}, actualContents) 101 | }) 102 | 103 | t.Run("GET .nar", func(t *testing.T) { 104 | rr := httptest.NewRecorder() 105 | ctx, cancel := context.WithCancel(context.Background()) 106 | defer cancel() 107 | 108 | req, err := http.NewRequestWithContext(ctx, "GET", narpath, nil) 109 | if err != nil { 110 | t.Fatal(err) 111 | } 112 | server.Handler.ServeHTTP(rr, req) 113 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 114 | assert.Equal(t, []string{"application/x-nix-nar"}, rr.Result().Header["Content-Type"]) 115 | assert.Equal(t, []string{fmt.Sprintf("%d", tdA.Narinfo.NarSize)}, rr.Result().Header["Content-Length"]) 116 | 117 | // read in the retrieved body 118 | actualContents, err := io.ReadAll(rr.Result().Body) 119 | if err != nil { 120 | t.Fatal(err) 121 | } 122 | 123 | assert.Equal(t, tdA.NarContents, actualContents) 124 | }) 125 | 126 | // get compressed .nar, which should match the uncompressed .nar after decompressing with zstd 127 | t.Run("GET compressed .nar", func(t *testing.T) { 128 | rr := httptest.NewRecorder() 129 | ctx, cancel := context.WithCancel(context.Background()) 130 | defer cancel() 131 | 132 | req, err := http.NewRequestWithContext(ctx, "GET", narpath+".zst", nil) 133 | if err != nil { 134 | t.Fatal(err) 135 | } 136 | server.Handler.ServeHTTP(rr, req) 137 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 138 | assert.Equal(t, []string{"application/x-nix-nar"}, rr.Result().Header["Content-Type"]) 139 | // We don't send the Content-Length header here, as we compress on the fly and don't know upfront 140 | 141 | // read the body into a buffer 142 | buf, err := io.ReadAll(rr.Result().Body) 143 | if err != nil { 144 | t.Fatal(err) 145 | } 146 | 147 | // read in the retrieved body 148 | zstdReader, err := zstd.NewReader(bytes.NewReader(buf)) 149 | if err != nil { 150 | t.Fatal(err) 151 | } 152 | defer zstdReader.Close() 153 | actualContents, err := io.ReadAll(zstdReader) 154 | if err != nil { 155 | t.Fatal(err) 156 | } 157 | 158 | // decompressed, it should look like the Nar we initially wrote 159 | assert.Equal(t, tdA.NarContents, actualContents) 160 | }) 161 | 162 | // TODO: remove Nar file to ensure we don't just no-op the upload 163 | // blobStore.DropAll(context.Background()) 164 | 165 | t.Run("PUT compressed .nar", func(t *testing.T) { 166 | // What name we upload it as doesn't really matter 167 | // (we still use the narhash here, even though Nix would use the file hash) 168 | // The only thing that matters is the extension. 169 | narpathZstd := "/nar/" + nixbase32.EncodeToString(tdA.Narinfo.NarHash.Digest) + ".nar.zst" 170 | 171 | // compress the .nar file on the fly, store in nb 172 | var b bytes.Buffer 173 | wc, err := compression.NewCompressor(&b, "zstd") 174 | assert.NoError(t, err, "creating a new compressor shouldn't error") 175 | _, err = wc.Write(tdA.NarContents) 176 | assert.NoError(t, err, "writing to compressor shouldn't error") 177 | err = wc.Close() 178 | assert.NoError(t, err, "closing compressor shouldn't error") 179 | 180 | rr := httptest.NewRecorder() 181 | ctx, cancel := context.WithCancel(context.Background()) 182 | defer cancel() 183 | 184 | req, err := http.NewRequestWithContext(ctx, "PUT", narpathZstd, bytes.NewReader(b.Bytes())) 185 | assert.NoError(t, err) 186 | 187 | server.Handler.ServeHTTP(rr, req) 188 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 189 | 190 | // check it exists in the store 191 | narMeta, err := metadataStore.GetNarMeta(context.Background(), tdA.Narinfo.NarHash.Digest) 192 | assert.NoError(t, err) 193 | assert.NotEqual(t, 0, narMeta.Size) 194 | }) 195 | }) 196 | 197 | t.Run("Narinfo tests", func(t *testing.T) { 198 | path := "/" + nixbase32.EncodeToString(tdAOutputHash) + ".narinfo" 199 | 200 | // synthesize a minimal narinfo for a compressed version 201 | // This is what we get served from the handler, 202 | // as zstd compression is configured. 203 | // We also use it later in the test to upload a compressed version. 204 | smallNarinfo := tdA.Narinfo 205 | smallNarinfo.URL += ".zst" 206 | 207 | smallNarinfo.FileHash = nil 208 | smallNarinfo.FileSize = 0 209 | smallNarinfo.Compression = "zstd" 210 | b := bytes.NewBufferString(smallNarinfo.String()) 211 | smallNarinfoContents := b.Bytes() 212 | 213 | t.Run("GET non-existent .narinfo", func(t *testing.T) { 214 | rr := httptest.NewRecorder() 215 | ctx, cancel := context.WithCancel(context.Background()) 216 | defer cancel() 217 | 218 | req, err := http.NewRequestWithContext(ctx, "GET", path, nil) 219 | if err != nil { 220 | t.Fatal(err) 221 | } 222 | server.Handler.ServeHTTP(rr, req) 223 | assert.Equal(t, http.StatusNotFound, rr.Result().StatusCode) 224 | }) 225 | 226 | t.Run("PUT .narinfo", func(t *testing.T) { 227 | rr := httptest.NewRecorder() 228 | req, err := http.NewRequest("PUT", path, bytes.NewReader(tdA.NarinfoContents)) 229 | if err != nil { 230 | t.Fatal(err) 231 | } 232 | 233 | server.Handler.ServeHTTP(rr, req) 234 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 235 | 236 | // expect body to be empty 237 | actualContents, err := io.ReadAll(rr.Result().Body) 238 | if err != nil { 239 | t.Fatal(err) 240 | } 241 | assert.Equal(t, []byte{}, actualContents) 242 | }) 243 | 244 | // when we retrieve it back, it should be served compressed 245 | // (as we initialize the handler with zstd compression) 246 | t.Run("GET .narinfo", func(t *testing.T) { 247 | rr := httptest.NewRecorder() 248 | ctx, cancel := context.WithCancel(context.Background()) 249 | defer cancel() 250 | 251 | req, err := http.NewRequestWithContext(ctx, "GET", path, nil) 252 | if err != nil { 253 | t.Fatal(err) 254 | } 255 | server.Handler.ServeHTTP(rr, req) 256 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 257 | assert.Equal(t, []string{"text/x-nix-narinfo"}, rr.Result().Header["Content-Type"]) 258 | assert.Equal(t, []string{fmt.Sprintf("%d", len(smallNarinfoContents))}, rr.Result().Header["Content-Length"]) 259 | 260 | // read in the retrieved body 261 | actualContents, err := io.ReadAll(rr.Result().Body) 262 | if err != nil { 263 | t.Fatal(err) 264 | } 265 | 266 | assert.Equal(t, smallNarinfoContents, actualContents) 267 | }) 268 | 269 | t.Run("PUT .narinfo referring to compressed NAR", func(t *testing.T) { 270 | rr := httptest.NewRecorder() 271 | ctx, cancel := context.WithCancel(context.Background()) 272 | defer cancel() 273 | 274 | req, err := http.NewRequestWithContext(ctx, "PUT", path, bytes.NewReader(smallNarinfoContents)) 275 | if err != nil { 276 | t.Fatal(err) 277 | } 278 | 279 | server.Handler.ServeHTTP(rr, req) 280 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 281 | 282 | // expect body to be empty 283 | actualContents, err := io.ReadAll(rr.Result().Body) 284 | if err != nil { 285 | t.Fatal(err) 286 | } 287 | assert.Equal(t, []byte{}, actualContents) 288 | }) 289 | 290 | // when we retrieve it back, it should still look like the minimal narinfo 291 | t.Run("GET .narinfo", func(t *testing.T) { 292 | rr := httptest.NewRecorder() 293 | ctx, cancel := context.WithCancel(context.Background()) 294 | defer cancel() 295 | 296 | req, err := http.NewRequestWithContext(ctx, "GET", path, nil) 297 | if err != nil { 298 | t.Fatal(err) 299 | } 300 | server.Handler.ServeHTTP(rr, req) 301 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 302 | assert.Equal(t, []string{"text/x-nix-narinfo"}, rr.Result().Header["Content-Type"]) 303 | assert.Equal(t, []string{fmt.Sprintf("%d", len(smallNarinfoContents))}, rr.Result().Header["Content-Length"]) 304 | 305 | // read in the retrieved body 306 | actualContents, err := io.ReadAll(rr.Result().Body) 307 | if err != nil { 308 | t.Fatal(err) 309 | } 310 | 311 | assert.Equal(t, smallNarinfoContents, actualContents) 312 | }) 313 | 314 | t.Run("PUT .nar for B", func(t *testing.T) { 315 | narpath := "/nar/" + nixbase32.EncodeToString(tdB.Narinfo.NarHash.Digest) + ".nar" 316 | rr := httptest.NewRecorder() 317 | ctx, cancel := context.WithCancel(context.Background()) 318 | defer cancel() 319 | 320 | req, err := http.NewRequestWithContext(ctx, "PUT", narpath, bytes.NewReader(tdB.NarContents)) 321 | if err != nil { 322 | t.Fatal(err) 323 | } 324 | 325 | server.Handler.ServeHTTP(rr, req) 326 | 327 | // expect status to be ok 328 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 329 | 330 | // expect body to be empty 331 | actualContents, err := io.ReadAll(rr.Result().Body) 332 | if err != nil { 333 | t.Fatal(err) 334 | } 335 | assert.Equal(t, []byte{}, actualContents) 336 | }) 337 | }) 338 | 339 | bNarinfoPath := "/" + nixbase32.EncodeToString(tdBOutputHash) + ".narinfo" 340 | 341 | t.Run("PUT .narinfo for B", func(t *testing.T) { 342 | rr := httptest.NewRecorder() 343 | 344 | req, err := http.NewRequest("PUT", bNarinfoPath, bytes.NewReader(tdB.NarinfoContents)) 345 | if err != nil { 346 | t.Fatal(err) 347 | } 348 | 349 | server.Handler.ServeHTTP(rr, req) 350 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 351 | 352 | // expect body to be empty 353 | actualContents, err := io.ReadAll(rr.Result().Body) 354 | if err != nil { 355 | t.Fatal(err) 356 | } 357 | assert.Equal(t, []byte{}, actualContents) 358 | }) 359 | 360 | t.Run("GET .narinfo for B", func(t *testing.T) { 361 | rr := httptest.NewRecorder() 362 | ctx, cancel := context.WithCancel(context.Background()) 363 | defer cancel() 364 | 365 | req, err := http.NewRequestWithContext(ctx, "GET", bNarinfoPath, nil) 366 | if err != nil { 367 | t.Fatal(err) 368 | } 369 | server.Handler.ServeHTTP(rr, req) 370 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 371 | 372 | // parse the .narinfo file we get back 373 | ni, err := narinfo.Parse(rr.Result().Body) 374 | assert.NoError(t, err) 375 | 376 | // assert references are preserved 377 | assert.Equal(t, tdB.Narinfo.References, ni.References) 378 | }) 379 | 380 | t.Run("PUT .nar for C", func(t *testing.T) { 381 | narpath := "/nar/" + nixbase32.EncodeToString(tdC.Narinfo.NarHash.Digest) + ".nar" 382 | rr := httptest.NewRecorder() 383 | ctx, cancel := context.WithCancel(context.Background()) 384 | defer cancel() 385 | 386 | req, err := http.NewRequestWithContext(ctx, "PUT", narpath, bytes.NewReader(tdC.NarContents)) 387 | if err != nil { 388 | t.Fatal(err) 389 | } 390 | 391 | server.Handler.ServeHTTP(rr, req) 392 | 393 | // expect status to be ok 394 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 395 | 396 | // expect body to be empty 397 | actualContents, err := io.ReadAll(rr.Result().Body) 398 | if err != nil { 399 | t.Fatal(err) 400 | } 401 | assert.Equal(t, []byte{}, actualContents) 402 | }) 403 | 404 | cNarinfoPath := "/" + nixbase32.EncodeToString(tdCOutputHash) + ".narinfo" 405 | 406 | t.Run("PUT .narinfo for C (contains self-reference)", func(t *testing.T) { 407 | rr := httptest.NewRecorder() 408 | ctx, cancel := context.WithCancel(context.Background()) 409 | defer cancel() 410 | 411 | req, err := http.NewRequestWithContext(ctx, "PUT", cNarinfoPath, bytes.NewReader(tdC.NarinfoContents)) 412 | if err != nil { 413 | t.Fatal(err) 414 | } 415 | 416 | server.Handler.ServeHTTP(rr, req) 417 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 418 | 419 | // expect body to be empty 420 | actualContents, err := io.ReadAll(rr.Result().Body) 421 | if err != nil { 422 | t.Fatal(err) 423 | } 424 | assert.Equal(t, []byte{}, actualContents) 425 | }) 426 | 427 | t.Run("GET .narinfo for C (contains self-reference)", func(t *testing.T) { 428 | rr := httptest.NewRecorder() 429 | ctx, cancel := context.WithCancel(context.Background()) 430 | defer cancel() 431 | 432 | req, err := http.NewRequestWithContext(ctx, "GET", cNarinfoPath, nil) 433 | if err != nil { 434 | t.Fatal(err) 435 | } 436 | server.Handler.ServeHTTP(rr, req) 437 | assert.Equal(t, http.StatusOK, rr.Result().StatusCode) 438 | 439 | // parse the .narinfo file we get back 440 | ni, err := narinfo.Parse(rr.Result().Body) 441 | assert.NoError(t, err) 442 | 443 | // assert references are preserved 444 | assert.Equal(t, tdC.Narinfo.References, ni.References) 445 | }) 446 | } 447 | -------------------------------------------------------------------------------- /pkg/store/blobstore/blobstore_test.go: -------------------------------------------------------------------------------- 1 | package blobstore_test 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "io" 7 | "io/ioutil" 8 | "os" 9 | "testing" 10 | 11 | "github.com/flokli/nix-casync/pkg/store/blobstore" 12 | "github.com/flokli/nix-casync/test" 13 | "github.com/stretchr/testify/assert" 14 | ) 15 | 16 | func TestCasyncStore(t *testing.T) { 17 | // populate castr dir 18 | castrDir, err := ioutil.TempDir("", "castr") 19 | if err != nil { 20 | panic(err) 21 | } 22 | 23 | t.Cleanup(func() { 24 | os.RemoveAll(castrDir) 25 | }) 26 | 27 | // populate caidx dir 28 | caidxDir, err := ioutil.TempDir("", "caidx") 29 | if err != nil { 30 | panic(err) 31 | } 32 | 33 | t.Cleanup(func() { 34 | os.RemoveAll(caidxDir) 35 | }) 36 | 37 | // init casync store 38 | caStore, err := blobstore.NewCasyncStore(castrDir, caidxDir, 65536) 39 | if err != nil { 40 | panic(err) 41 | } 42 | 43 | t.Cleanup(func() { 44 | caStore.Close() 45 | }) 46 | 47 | testBlobStore(t, caStore) 48 | } 49 | 50 | func TestMemoryStore(t *testing.T) { 51 | memoryStore := blobstore.NewMemoryStore() 52 | 53 | t.Cleanup(func() { 54 | memoryStore.Close() 55 | }) 56 | 57 | testBlobStore(t, memoryStore) 58 | } 59 | 60 | // testBlobStore runs all nar tests, with a Narstore generated by storeGenerator. 61 | func testBlobStore(t *testing.T, blobStore blobstore.BlobStore) { 62 | testDataT := test.GetTestDataTable() 63 | 64 | tdA, exists := testDataT["a"] 65 | if !exists { 66 | panic("testData[a] doesn't exist") 67 | } 68 | 69 | tdANarHash := tdA.Narinfo.NarHash.Digest 70 | tdANarSize := tdA.Narinfo.NarSize 71 | 72 | t.Run("GetBlobNotFound", func(t *testing.T) { 73 | _, _, err := blobStore.GetBlob(context.Background(), tdANarHash) 74 | if assert.Error(t, err) { 75 | assert.ErrorIsf(t, 76 | err, 77 | os.ErrNotExist, 78 | "on a non-existent blob, there should be a os.ErrNotExist in the error chain", 79 | ) 80 | } 81 | }) 82 | 83 | t.Run("PutBlob", func(t *testing.T) { 84 | w, err := blobStore.PutBlob(context.Background()) 85 | assert.NoError(t, err) 86 | defer w.Close() 87 | 88 | n, err := io.Copy(w, bytes.NewReader(tdA.NarContents)) 89 | 90 | assert.NoError(t, err) 91 | assert.Equal(t, tdANarSize, uint64(n)) 92 | assert.NoError(t, w.Close()) 93 | 94 | assert.Equal(t, tdANarHash, w.Sha256Sum(), "narhash should be correctly calculated") 95 | }) 96 | 97 | t.Run("PutBlob again", func(t *testing.T) { 98 | w, err := blobStore.PutBlob(context.Background()) 99 | assert.NoError(t, err) 100 | defer w.Close() 101 | 102 | n, err := io.Copy(w, bytes.NewReader(tdA.NarContents)) 103 | assert.NoError(t, err) 104 | 105 | assert.Equal(t, tdANarSize, uint64(n)) 106 | assert.NoError(t, w.Close()) 107 | 108 | assert.Equal(t, tdANarHash, w.Sha256Sum(), "narhash should still be correctly calculated") 109 | }) 110 | 111 | t.Run("PutNar,then abort", func(t *testing.T) { 112 | w, err := blobStore.PutBlob(context.Background()) 113 | assert.NoError(t, err) 114 | assert.NoError(t, w.Close()) 115 | }) 116 | 117 | t.Run("GetBlob", func(t *testing.T) { 118 | r, n, err := blobStore.GetBlob(context.Background(), tdANarHash) 119 | 120 | assert.NoError(t, err) 121 | assert.Equal(t, tdANarSize, uint64(n)) 122 | 123 | actualContents, err := io.ReadAll(r) 124 | assert.NoError(t, err) 125 | assert.Equal(t, tdA.NarContents, actualContents) 126 | }) 127 | 128 | t.Run("GetNar,then abort", func(t *testing.T) { 129 | r, _, err := blobStore.GetBlob(context.Background(), tdANarHash) 130 | 131 | assert.NoError(t, err) 132 | assert.NoError(t, r.Close()) 133 | }) 134 | } 135 | -------------------------------------------------------------------------------- /pkg/store/blobstore/casync_store.go: -------------------------------------------------------------------------------- 1 | package blobstore 2 | 3 | import ( 4 | "context" 5 | "encoding/hex" 6 | "io" 7 | "os" 8 | "runtime" 9 | 10 | "github.com/folbricht/desync" 11 | ) 12 | 13 | var _ BlobStore = &CasyncStore{} 14 | 15 | type CasyncStore struct { 16 | localStore desync.WriteStore 17 | localIndexStore desync.IndexWriteStore 18 | concurrency int 19 | 20 | chunkSizeAvgDefault uint64 21 | chunkSizeMinDefault uint64 22 | chunkSizeMaxDefault uint64 23 | 24 | // TODO: remote store(s)? 25 | } 26 | 27 | func NewCasyncStore(localStoreDir, localIndexStoreDir string, avgChunkSize int) (*CasyncStore, error) { 28 | // TODO: maybe use MultiStoreWithCache? 29 | err := os.MkdirAll(localStoreDir, os.ModePerm) 30 | if err != nil { 31 | return nil, err 32 | } 33 | 34 | localStore, err := desync.NewLocalStore(localStoreDir, desync.StoreOptions{}) 35 | if err != nil { 36 | return nil, err 37 | } 38 | 39 | err = os.MkdirAll(localIndexStoreDir, os.ModePerm) 40 | if err != nil { 41 | return nil, err 42 | } 43 | 44 | localIndexStore, err := desync.NewLocalIndexStore(localIndexStoreDir) 45 | if err != nil { 46 | return nil, err 47 | } 48 | 49 | concurrency := runtime.NumCPU() 50 | if concurrency > 4 { 51 | concurrency = 4 52 | } 53 | 54 | return &CasyncStore{ 55 | localStore: localStore, 56 | localIndexStore: localIndexStore, 57 | concurrency: concurrency, 58 | 59 | // values stolen from chunker_test.go 60 | chunkSizeAvgDefault: uint64(avgChunkSize), 61 | chunkSizeMinDefault: uint64(avgChunkSize) / 4, 62 | chunkSizeMaxDefault: uint64(avgChunkSize) * 4, 63 | }, nil 64 | } 65 | 66 | func (c *CasyncStore) Close() error { 67 | if err := c.localStore.Close(); err != nil { 68 | return err 69 | } 70 | 71 | return c.localIndexStore.Close() 72 | } 73 | 74 | func (c *CasyncStore) GetBlob(ctx context.Context, sha256 []byte) (io.ReadCloser, int64, error) { 75 | // retrieve .caidx 76 | caidx, err := c.localIndexStore.GetIndex(hex.EncodeToString(sha256)) 77 | if err != nil { 78 | return nil, 0, err 79 | } 80 | 81 | csnr, err := NewCasyncStoreReader( 82 | ctx, 83 | caidx, 84 | c.localStore, 85 | []desync.Seed{}, 86 | 1, 87 | nil, 88 | ) 89 | if err != nil { 90 | return nil, 0, err 91 | } 92 | 93 | return csnr, caidx.Length(), nil 94 | } 95 | 96 | func (c *CasyncStore) PutBlob(ctx context.Context) (WriteCloseHasher, error) { //nolint:ireturn 97 | return NewCasyncStoreWriter( 98 | ctx, 99 | 100 | c.localStore, 101 | c.localIndexStore, 102 | 103 | c.concurrency, 104 | c.chunkSizeMinDefault, 105 | c.chunkSizeAvgDefault, 106 | c.chunkSizeMaxDefault, 107 | ) 108 | } 109 | -------------------------------------------------------------------------------- /pkg/store/blobstore/casync_store_nar_reader.go: -------------------------------------------------------------------------------- 1 | package blobstore 2 | 3 | import ( 4 | "context" 5 | "io" 6 | "io/ioutil" 7 | "os" 8 | 9 | "github.com/folbricht/desync" 10 | ) 11 | 12 | // CasyncStoreReader provides a io.ReadCloser 13 | // on the first read, it creates a tempfile, assembles the contents into it, 14 | // then reads into that file. 15 | type CasyncStoreReader struct { 16 | io.ReadCloser 17 | 18 | ctx context.Context 19 | caidx desync.Index 20 | desyncStore desync.Store 21 | seeds []desync.Seed 22 | concurrency int 23 | pb desync.ProgressBar 24 | 25 | f *os.File 26 | fileAssembled bool // whether AssembleFile was already run 27 | } 28 | 29 | // NewCasyncStoreReader returns a properly initialized casyncStoreReader. 30 | func NewCasyncStoreReader( 31 | ctx context.Context, 32 | caidx desync.Index, 33 | desyncStore desync.Store, 34 | seeds []desync.Seed, 35 | concurrency int, 36 | pb desync.ProgressBar, 37 | ) (*CasyncStoreReader, error) { 38 | tmpFile, err := ioutil.TempFile("", "blob") 39 | if err != nil { 40 | return nil, err 41 | } 42 | // Cleanup is handled in csnr.Close(), or whenever there's an error during init 43 | 44 | return &CasyncStoreReader{ 45 | ctx: ctx, 46 | caidx: caidx, 47 | desyncStore: desyncStore, 48 | seeds: seeds, 49 | concurrency: concurrency, 50 | pb: pb, 51 | f: tmpFile, 52 | }, nil 53 | } 54 | 55 | func (csnr *CasyncStoreReader) Read(p []byte) (n int, err error) { 56 | // if this is the first read, we need to run AssembleFile into f 57 | // if there's any error, we return it. 58 | // It's up to the caller to also run Close(), which will clean up the tmpfile 59 | if !csnr.fileAssembled { 60 | _, err = desync.AssembleFile( 61 | csnr.ctx, 62 | csnr.f.Name(), 63 | csnr.caidx, 64 | csnr.desyncStore, 65 | csnr.seeds, 66 | csnr.concurrency, 67 | csnr.pb, 68 | ) 69 | if err != nil { 70 | return 0, err 71 | } 72 | 73 | // flush and seek to the beginning 74 | err = csnr.f.Sync() 75 | if err != nil { 76 | return 0, err 77 | } 78 | 79 | _, err = csnr.f.Seek(0, 0) 80 | if err != nil { 81 | return 0, err 82 | } 83 | // we successfully went till here 84 | csnr.fileAssembled = true 85 | } 86 | 87 | return csnr.f.Read(p) 88 | } 89 | 90 | func (csnr *CasyncStoreReader) Close() error { 91 | defer os.Remove(csnr.f.Name()) 92 | 93 | return csnr.f.Close() 94 | } 95 | -------------------------------------------------------------------------------- /pkg/store/blobstore/casync_store_nar_writer.go: -------------------------------------------------------------------------------- 1 | package blobstore 2 | 3 | import ( 4 | "context" 5 | "crypto/sha256" 6 | "encoding/hex" 7 | "hash" 8 | "io" 9 | "io/ioutil" 10 | "os" 11 | 12 | "github.com/folbricht/desync" 13 | ) 14 | 15 | // CasyncStoreWriter implements WriteCloseHasher. 16 | var _ WriteCloseHasher = &CasyncStoreWriter{} 17 | 18 | // CasyncStoreWriter provides a io.WriteClose[Hashe]r interface 19 | // The whole content of the blob is written to it. 20 | // Internally, it'll write it to a temporary file. 21 | // On close, its contents will be chunked, 22 | // the index added to the index store, and the chunks added to the chunk store. 23 | type CasyncStoreWriter struct { 24 | io.WriteCloser 25 | 26 | ctx context.Context 27 | 28 | desyncStore desync.WriteStore 29 | desyncIndexStore desync.IndexWriteStore 30 | 31 | concurrency int 32 | chunkSizeMinDefault uint64 33 | chunkSizeAvgDefault uint64 34 | chunkSizeMaxDefault uint64 35 | 36 | f *os.File 37 | bytesWritten uint64 38 | hash hash.Hash 39 | } 40 | 41 | // NewCasyncStoreWriter returns a properly initialized casyncStoreWriter. 42 | func NewCasyncStoreWriter( 43 | ctx context.Context, 44 | desyncStore desync.WriteStore, 45 | desyncIndexStore desync.IndexWriteStore, 46 | concurrency int, 47 | chunkSizeMinDefault uint64, 48 | chunkSizeAvgDefault uint64, 49 | chunkSizeMaxDefault uint64, 50 | ) (*CasyncStoreWriter, error) { 51 | tmpFile, err := ioutil.TempFile("", "blob") 52 | if err != nil { 53 | return nil, err 54 | } 55 | // Cleanup is handled in Close() 56 | 57 | return &CasyncStoreWriter{ 58 | ctx: ctx, 59 | 60 | desyncStore: desyncStore, 61 | desyncIndexStore: desyncIndexStore, 62 | 63 | concurrency: concurrency, 64 | chunkSizeMinDefault: chunkSizeMinDefault, 65 | chunkSizeAvgDefault: chunkSizeAvgDefault, 66 | chunkSizeMaxDefault: chunkSizeMaxDefault, 67 | 68 | f: tmpFile, 69 | hash: sha256.New(), 70 | }, nil 71 | } 72 | 73 | func (csw *CasyncStoreWriter) Write(p []byte) (int, error) { 74 | csw.hash.Write(p) 75 | csw.bytesWritten += uint64(len(p)) 76 | 77 | return csw.f.Write(p) 78 | } 79 | 80 | func (csw *CasyncStoreWriter) Close() error { 81 | // at the end, we want to remove the tempfile 82 | defer os.Remove(csw.f.Name()) 83 | 84 | // calculate how the file will be called 85 | indexName := hex.EncodeToString(csw.Sha256Sum()) 86 | 87 | // check if that same file has already been uploaded. 88 | _, err := csw.desyncIndexStore.GetIndex(indexName) 89 | 90 | if err != nil && !os.IsNotExist(err) { 91 | return err 92 | } 93 | 94 | if err == nil { 95 | // if the file already exists in the index, we're done. 96 | return nil 97 | } 98 | 99 | // flush the tempfile and seek to the start 100 | err = csw.f.Sync() 101 | if err != nil { 102 | return err 103 | } 104 | 105 | _, err = csw.f.Seek(0, 0) 106 | if err != nil { 107 | return err 108 | } 109 | 110 | // Run the chunker on the tempfile 111 | chunker, err := desync.NewChunker( 112 | csw.f, 113 | csw.chunkSizeMinDefault, 114 | csw.chunkSizeAvgDefault, 115 | csw.chunkSizeMaxDefault, 116 | ) 117 | if err != nil { 118 | return err 119 | } 120 | 121 | // upload all chunks into the store 122 | caidx, err := desync.ChunkStream(csw.ctx, 123 | chunker, 124 | csw.desyncStore, 125 | csw.concurrency, 126 | ) 127 | if err != nil { 128 | return err 129 | } 130 | 131 | // upload index into the index store 132 | // name it after the hash 133 | err = csw.desyncIndexStore.StoreIndex(indexName, caidx) 134 | if err != nil { 135 | return err 136 | } 137 | 138 | return nil 139 | } 140 | 141 | func (csw *CasyncStoreWriter) Sha256Sum() []byte { 142 | return csw.hash.Sum([]byte{}) 143 | } 144 | 145 | func (csw *CasyncStoreWriter) BytesWritten() uint64 { 146 | return csw.bytesWritten 147 | } 148 | -------------------------------------------------------------------------------- /pkg/store/blobstore/memory_store.go: -------------------------------------------------------------------------------- 1 | package blobstore 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "crypto/sha256" 7 | "encoding/hex" 8 | "hash" 9 | "io" 10 | "os" 11 | "sync" 12 | ) 13 | 14 | // MemoryStore implements BlobStore. 15 | var _ BlobStore = &MemoryStore{} 16 | 17 | type MemoryStore struct { 18 | // Go can't use []bytes as a map key 19 | blobs map[string][]byte 20 | muBlobs sync.Mutex 21 | } 22 | 23 | func NewMemoryStore() *MemoryStore { 24 | return &MemoryStore{ 25 | blobs: make(map[string][]byte), 26 | } 27 | } 28 | 29 | func (m *MemoryStore) Close() error { 30 | return nil 31 | } 32 | 33 | func (m *MemoryStore) PutBlob(ctx context.Context) (WriteCloseHasher, error) { //nolint:ireturn 34 | return &memoryStoreWriter{ 35 | hash: sha256.New(), 36 | memoryStore: m, 37 | }, nil 38 | } 39 | 40 | func (m *MemoryStore) GetBlob(ctx context.Context, sha256 []byte) (io.ReadCloser, int64, error) { 41 | m.muBlobs.Lock() 42 | v, ok := m.blobs[hex.EncodeToString(sha256)] 43 | m.muBlobs.Unlock() 44 | 45 | if ok { 46 | return io.NopCloser(bytes.NewReader(v)), int64(len(v)), nil 47 | } 48 | 49 | return nil, 0, os.ErrNotExist 50 | } 51 | 52 | // memoryStoreWriter implements WriteCloseHasher. 53 | var _ WriteCloseHasher = &memoryStoreWriter{} 54 | 55 | type memoryStoreWriter struct { 56 | memoryStore *MemoryStore 57 | contents []byte 58 | bytesWritten uint64 59 | hash hash.Hash 60 | } 61 | 62 | func (msw *memoryStoreWriter) Write(p []byte) (n int, err error) { 63 | msw.contents = append(msw.contents, p...) 64 | msw.hash.Write(p) 65 | msw.bytesWritten += uint64(len(p)) 66 | 67 | return len(p), nil 68 | } 69 | 70 | func (msw *memoryStoreWriter) Close() error { 71 | msw.memoryStore.muBlobs.Lock() 72 | msw.memoryStore.blobs[hex.EncodeToString(msw.hash.Sum([]byte{}))] = msw.contents 73 | msw.memoryStore.muBlobs.Unlock() 74 | 75 | return nil 76 | } 77 | 78 | func (msw *memoryStoreWriter) Sha256Sum() []byte { 79 | return msw.hash.Sum([]byte{}) 80 | } 81 | 82 | func (msw *memoryStoreWriter) BytesWritten() uint64 { 83 | return msw.bytesWritten 84 | } 85 | -------------------------------------------------------------------------------- /pkg/store/blobstore/types.go: -------------------------------------------------------------------------------- 1 | // Package blobstore implements some content-addressed blob stores. 2 | // You can store whatever you want in there, but need to address things by their hash to get them out. 3 | package blobstore 4 | 5 | import ( 6 | "context" 7 | "io" 8 | ) 9 | 10 | // BlobStore describes the interface of a blob store. 11 | type BlobStore interface { 12 | PutBlob(ctx context.Context) (WriteCloseHasher, error) 13 | GetBlob(ctx context.Context, sha256 []byte) (io.ReadCloser, int64, error) 14 | io.Closer 15 | } 16 | 17 | // WriteWriteCloserHashSum is a io.WriteCloser, which you can ask for a checksum. 18 | type WriteCloseHasher interface { 19 | io.WriteCloser 20 | Sha256Sum() []byte 21 | BytesWritten() uint64 22 | } 23 | -------------------------------------------------------------------------------- /pkg/store/metadatastore/file_store.go: -------------------------------------------------------------------------------- 1 | package metadatastore 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "errors" 7 | "fmt" 8 | "io" 9 | "io/ioutil" 10 | "os" 11 | "path" 12 | 13 | "github.com/nix-community/go-nix/pkg/nixbase32" 14 | ) 15 | 16 | // FileStore implements MetadataStore. 17 | var _ MetadataStore = &FileStore{} 18 | 19 | type FileStore struct { 20 | pathInfoDirectory string 21 | narMetaDirectory string 22 | } 23 | 24 | func NewFileStore(baseDirectory string) (*FileStore, error) { 25 | pathInfoDirectory := path.Join(baseDirectory, "pathinfo") 26 | 27 | err := os.MkdirAll(pathInfoDirectory, os.ModePerm) 28 | if err != nil { 29 | return nil, err 30 | } 31 | 32 | narMetaDirectory := path.Join(baseDirectory, "narmeta") 33 | 34 | err = os.MkdirAll(narMetaDirectory, os.ModePerm) 35 | if err != nil { 36 | return nil, err 37 | } 38 | 39 | return &FileStore{ 40 | pathInfoDirectory: pathInfoDirectory, 41 | narMetaDirectory: narMetaDirectory, 42 | }, nil 43 | } 44 | 45 | func (fs *FileStore) pathInfoPath(outputHash []byte) string { 46 | encodedHash := nixbase32.EncodeToString(outputHash) 47 | 48 | return path.Join(fs.pathInfoDirectory, encodedHash[:4], encodedHash+".json") 49 | } 50 | 51 | func (fs *FileStore) narMetaPath(narHash []byte) string { 52 | encodedHash := nixbase32.EncodeToString(narHash) 53 | 54 | return path.Join(fs.narMetaDirectory, encodedHash[:4], encodedHash+".json") 55 | } 56 | 57 | func (fs *FileStore) GetPathInfo(ctx context.Context, outputHash []byte) (*PathInfo, error) { 58 | p := fs.pathInfoPath(outputHash) 59 | 60 | f, err := os.Open(p) 61 | if err != nil { 62 | return nil, err 63 | } 64 | 65 | defer f.Close() 66 | 67 | b, err := io.ReadAll(f) 68 | if err != nil { 69 | return nil, err 70 | } 71 | 72 | var pathInfo PathInfo 73 | 74 | err = json.Unmarshal(b, &pathInfo) 75 | if err != nil { 76 | return nil, err 77 | } 78 | 79 | return &pathInfo, nil 80 | } 81 | 82 | func (fs *FileStore) PutPathInfo(ctx context.Context, pathinfo *PathInfo) error { 83 | err := pathinfo.Check() 84 | if err != nil { 85 | return err 86 | } 87 | // foreign key constraint: referred NarMeta needs to exist 88 | _, err = fs.GetNarMeta(ctx, pathinfo.NarHash) 89 | if err != nil { 90 | if errors.Is(err, os.ErrNotExist) { 91 | return fmt.Errorf("referred nar doesn't exist: %w", err) 92 | } 93 | 94 | return err 95 | } 96 | 97 | p := fs.pathInfoPath(pathinfo.OutputHash) 98 | dir := path.Dir(p) 99 | 100 | err = os.MkdirAll(dir, os.ModePerm) 101 | if err != nil { 102 | return err 103 | } 104 | 105 | // create a tempfile (in the same directory), write to it, then move it to where we want it to be 106 | // this is to ensure an atomic write/replacement. 107 | tmpFile, err := ioutil.TempFile(path.Dir(p), "narinfo") 108 | if err != nil { 109 | return err 110 | } 111 | 112 | defer tmpFile.Close() 113 | defer os.Remove(tmpFile.Name()) 114 | 115 | // serialize the pathinfo to json 116 | b, err := json.Marshal(pathinfo) 117 | if err != nil { 118 | return err 119 | } 120 | 121 | _, err = tmpFile.Write(b) 122 | if err != nil { 123 | return err 124 | } 125 | 126 | err = tmpFile.Sync() 127 | if err != nil { 128 | return err 129 | } 130 | 131 | err = tmpFile.Close() 132 | if err != nil { 133 | return err 134 | } 135 | 136 | return os.Rename(tmpFile.Name(), p) 137 | } 138 | 139 | func (fs *FileStore) GetNarMeta(ctx context.Context, narHash []byte) (*NarMeta, error) { 140 | p := fs.narMetaPath(narHash) 141 | 142 | f, err := os.Open(p) 143 | if err != nil { 144 | return nil, err 145 | } 146 | 147 | defer f.Close() 148 | 149 | b, err := io.ReadAll(f) 150 | if err != nil { 151 | return nil, err 152 | } 153 | 154 | var narMeta NarMeta 155 | 156 | err = json.Unmarshal(b, &narMeta) 157 | if err != nil { 158 | return nil, err 159 | } 160 | 161 | return &narMeta, nil 162 | } 163 | 164 | func (fs *FileStore) PutNarMeta(ctx context.Context, narMeta *NarMeta) error { 165 | err := narMeta.Check() 166 | if err != nil { 167 | return err 168 | } 169 | 170 | // foreign key constraint: all references need to exist 171 | for i, reference := range narMeta.References { 172 | _, err := fs.GetPathInfo(ctx, reference) 173 | if err != nil { 174 | if errors.Is(err, os.ErrNotExist) { 175 | return fmt.Errorf("referred reference %v doesn't exist: %w", narMeta.ReferencesStr[i], err) 176 | } 177 | 178 | return err 179 | } 180 | } 181 | 182 | p := fs.narMetaPath(narMeta.NarHash) 183 | dir := path.Dir(p) 184 | 185 | err = os.MkdirAll(dir, os.ModePerm) 186 | if err != nil { 187 | return err 188 | } 189 | 190 | // create a tempfile (in the same directory), write to it, then move it to where we want it to be 191 | // this is to ensure an atomic write/replacement. 192 | tmpFile, err := ioutil.TempFile(path.Dir(p), "narmeta") 193 | if err != nil { 194 | return err 195 | } 196 | 197 | defer tmpFile.Close() 198 | defer os.Remove(tmpFile.Name()) 199 | 200 | // serialize the pathinfo to json 201 | b, err := json.Marshal(narMeta) 202 | if err != nil { 203 | return err 204 | } 205 | 206 | _, err = tmpFile.Write(b) 207 | if err != nil { 208 | return err 209 | } 210 | 211 | err = tmpFile.Sync() 212 | if err != nil { 213 | return err 214 | } 215 | 216 | err = tmpFile.Close() 217 | if err != nil { 218 | return err 219 | } 220 | 221 | return os.Rename(tmpFile.Name(), p) 222 | } 223 | 224 | func (fs *FileStore) Close() error { 225 | return nil 226 | } 227 | 228 | func (fs *FileStore) DropAll(ctx context.Context) error { 229 | err := os.RemoveAll(fs.narMetaDirectory) 230 | if err != nil { 231 | return err 232 | } 233 | 234 | err = os.RemoveAll(fs.pathInfoDirectory) 235 | if err != nil { 236 | return err 237 | } 238 | 239 | err = os.MkdirAll(fs.narMetaDirectory, os.ModePerm) 240 | if err != nil { 241 | return err 242 | } 243 | 244 | err = os.MkdirAll(fs.pathInfoDirectory, os.ModePerm) 245 | if err != nil { 246 | return err 247 | } 248 | 249 | return nil 250 | } 251 | -------------------------------------------------------------------------------- /pkg/store/metadatastore/memory_store.go: -------------------------------------------------------------------------------- 1 | package metadatastore 2 | 3 | import ( 4 | "context" 5 | "encoding/hex" 6 | "errors" 7 | "fmt" 8 | "os" 9 | "sync" 10 | ) 11 | 12 | // MemoryStore implements MetadataStore. 13 | var _ MetadataStore = &MemoryStore{} 14 | 15 | type MemoryStore struct { 16 | pathInfo map[string]PathInfo 17 | muPathInfo sync.Mutex 18 | narMeta map[string]NarMeta 19 | muNarMeta sync.Mutex 20 | } 21 | 22 | func NewMemoryStore() *MemoryStore { 23 | return &MemoryStore{ 24 | pathInfo: make(map[string]PathInfo), 25 | narMeta: make(map[string]NarMeta), 26 | } 27 | } 28 | 29 | func (ms *MemoryStore) Close() error { 30 | return nil 31 | } 32 | 33 | func (ms *MemoryStore) GetPathInfo(ctx context.Context, outputHash []byte) (*PathInfo, error) { 34 | ms.muPathInfo.Lock() 35 | v, ok := ms.pathInfo[hex.EncodeToString(outputHash)] 36 | ms.muPathInfo.Unlock() 37 | 38 | if ok { 39 | return &v, nil 40 | } 41 | 42 | return nil, os.ErrNotExist 43 | } 44 | 45 | func (ms *MemoryStore) PutPathInfo(ctx context.Context, pathinfo *PathInfo) error { 46 | err := pathinfo.Check() 47 | if err != nil { 48 | return err 49 | } 50 | 51 | // foreign key constraint: referred NarMeta needs to exist 52 | _, err = ms.GetNarMeta(ctx, pathinfo.NarHash) 53 | if err != nil { 54 | if errors.Is(err, os.ErrNotExist) { 55 | return fmt.Errorf("referred nar doesn't exist: %w", err) 56 | } 57 | 58 | return err 59 | } 60 | 61 | ms.muPathInfo.Lock() 62 | ms.pathInfo[hex.EncodeToString(pathinfo.OutputHash)] = *pathinfo 63 | ms.muPathInfo.Unlock() 64 | 65 | return nil 66 | } 67 | 68 | func (ms *MemoryStore) GetNarMeta(ctx context.Context, narHash []byte) (*NarMeta, error) { 69 | ms.muNarMeta.Lock() 70 | v, ok := ms.narMeta[hex.EncodeToString(narHash)] 71 | ms.muNarMeta.Unlock() 72 | 73 | if ok { 74 | return &v, nil 75 | } 76 | 77 | return nil, os.ErrNotExist 78 | } 79 | 80 | func (ms *MemoryStore) PutNarMeta(ctx context.Context, narMeta *NarMeta) error { 81 | err := narMeta.Check() 82 | if err != nil { 83 | return err 84 | } 85 | 86 | // foreign key constraint: all references need to exist 87 | for i, reference := range narMeta.References { 88 | _, err := ms.GetPathInfo(ctx, reference) 89 | if err != nil { 90 | if errors.Is(err, os.ErrNotExist) { 91 | return fmt.Errorf("referred reference %v doesn't exist: %w", narMeta.ReferencesStr[i], err) 92 | } 93 | 94 | return err 95 | } 96 | } 97 | 98 | ms.muNarMeta.Lock() 99 | ms.narMeta[hex.EncodeToString(narMeta.NarHash)] = *narMeta 100 | ms.muNarMeta.Unlock() 101 | 102 | return nil 103 | } 104 | 105 | func (ms *MemoryStore) DropAll(ctx context.Context) error { 106 | ms.muNarMeta.Lock() 107 | ms.muPathInfo.Lock() 108 | 109 | for k := range ms.narMeta { 110 | delete(ms.narMeta, k) 111 | } 112 | 113 | for k := range ms.pathInfo { 114 | delete(ms.pathInfo, k) 115 | } 116 | 117 | ms.muNarMeta.Unlock() 118 | ms.muPathInfo.Unlock() 119 | 120 | return nil 121 | } 122 | -------------------------------------------------------------------------------- /pkg/store/metadatastore/metadata_store_test.go: -------------------------------------------------------------------------------- 1 | package metadatastore_test 2 | 3 | import ( 4 | "context" 5 | "io/ioutil" 6 | "os" 7 | "testing" 8 | 9 | "github.com/flokli/nix-casync/pkg/store/metadatastore" 10 | "github.com/flokli/nix-casync/test" 11 | "github.com/stretchr/testify/assert" 12 | ) 13 | 14 | func TestMemoryStore(t *testing.T) { 15 | memoryStore := metadatastore.NewMemoryStore() 16 | 17 | t.Cleanup(func() { 18 | memoryStore.Close() 19 | }) 20 | 21 | testMetadataStore(t, memoryStore) 22 | } 23 | 24 | func TestFileStore(t *testing.T) { 25 | tmpDir, err := ioutil.TempDir("", "narinfo") 26 | if err != nil { 27 | panic(err) 28 | } 29 | 30 | t.Cleanup(func() { 31 | os.RemoveAll(tmpDir) 32 | }) 33 | 34 | fileStore, err := metadatastore.NewFileStore(tmpDir) 35 | if err != nil { 36 | panic(err) 37 | } 38 | 39 | t.Cleanup(func() { 40 | fileStore.Close() 41 | }) 42 | 43 | testMetadataStore(t, fileStore) 44 | } 45 | 46 | // testMetadataStore runs all metadata store tests against the passed store. 47 | func testMetadataStore(t *testing.T, metadataStore metadatastore.MetadataStore) { 48 | testDataT := test.GetTestDataTable() 49 | 50 | tdA, exists := testDataT["a"] 51 | if !exists { 52 | panic("testData[a] doesn't exist") 53 | } 54 | 55 | tdAPathInfo, tdANarMeta, err := metadatastore.ParseNarinfo(tdA.Narinfo) 56 | if err != nil { 57 | t.Fatal(err) 58 | } 59 | 60 | tdB, exists := testDataT["b"] 61 | if !exists { 62 | panic("testData[b] doesn't exist") 63 | } 64 | 65 | tdBPathInfo, tdBNarMeta, err := metadatastore.ParseNarinfo(tdB.Narinfo) 66 | if err != nil { 67 | t.Fatal(err) 68 | } 69 | 70 | t.Run("NarMeta", func(t *testing.T) { 71 | t.Run("GetNarMetaNotFound", func(t *testing.T) { 72 | _, err := metadataStore.GetNarMeta(context.Background(), tdANarMeta.NarHash) 73 | if assert.Error(t, err) { 74 | assert.ErrorIsf( 75 | t, 76 | err, 77 | os.ErrNotExist, 78 | "on a non-existent NarMeta, there should be a os.ErrNotExist in the error chain", 79 | ) 80 | } 81 | }) 82 | 83 | t.Run("PutNarMeta", func(t *testing.T) { 84 | err := metadataStore.PutNarMeta(context.Background(), tdANarMeta) 85 | assert.NoError(t, err) 86 | }) 87 | 88 | t.Run("PutNarMeta again", func(t *testing.T) { 89 | err := metadataStore.PutNarMeta(context.Background(), tdANarMeta) 90 | assert.NoError(t, err) 91 | }) 92 | 93 | t.Run("GetNarMeta", func(t *testing.T) { 94 | narMeta, err := metadataStore.GetNarMeta(context.Background(), tdANarMeta.NarHash) 95 | assert.NoError(t, err) 96 | assert.Equal(t, *tdANarMeta, *narMeta) 97 | }) 98 | }) 99 | 100 | t.Run("PathInfo", func(t *testing.T) { 101 | t.Run("GetPathInfoNotFound", func(t *testing.T) { 102 | _, err := metadataStore.GetPathInfo(context.Background(), tdAPathInfo.OutputHash) 103 | if assert.Error(t, err) { 104 | assert.ErrorIsf( 105 | t, 106 | err, 107 | os.ErrNotExist, 108 | "on a non-existent PathInfo, there should be a os.ErrNotExist in the error chain", 109 | ) 110 | } 111 | }) 112 | 113 | t.Run("PutPathInfo", func(t *testing.T) { 114 | err := metadataStore.PutPathInfo(context.Background(), tdAPathInfo) 115 | assert.NoError(t, err) 116 | }) 117 | 118 | t.Run("PutPathInfo again", func(t *testing.T) { 119 | err := metadataStore.PutPathInfo(context.Background(), tdAPathInfo) 120 | assert.NoError(t, err) 121 | }) 122 | 123 | t.Run("GetPathInfo", func(t *testing.T) { 124 | pathInfo, err := metadataStore.GetPathInfo(context.Background(), tdAPathInfo.OutputHash) 125 | if assert.NoError(t, err) { 126 | assert.Equal(t, *tdAPathInfo, *pathInfo) 127 | } 128 | }) 129 | }) 130 | 131 | t.Run("Integrity Tests", func(t *testing.T) { 132 | err := metadataStore.DropAll(context.Background()) 133 | if err != nil { 134 | panic(err) 135 | } 136 | 137 | // Test it's not possible to upload A PathInfo without uploading A NarMeta first 138 | t.Run("require NarMeta first", func(t *testing.T) { 139 | err = metadataStore.PutPathInfo(context.Background(), tdAPathInfo) 140 | assert.Error(t, err) 141 | 142 | err = metadataStore.PutNarMeta(context.Background(), tdANarMeta) 143 | assert.NoError(t, err) 144 | 145 | err = metadataStore.PutPathInfo(context.Background(), tdAPathInfo) 146 | assert.NoError(t, err) 147 | }) 148 | 149 | err = metadataStore.DropAll(context.Background()) 150 | if err != nil { 151 | panic(err) 152 | } 153 | // Try to upload B, which refers to A (which is not uploaded) should fail, 154 | // until we upload A (and it's pathinfo) 155 | t.Run("require References to be uploaded first", func(t *testing.T) { 156 | // upload NarMeta for B 157 | err = metadataStore.PutNarMeta(context.Background(), tdBNarMeta) 158 | assert.Error(t, err, "uploading NarMeta with references to non-existing PathInfo should fail") 159 | 160 | // upload PathInfo for A, which should also fail without NarMeta for A 161 | err = metadataStore.PutPathInfo(context.Background(), tdAPathInfo) 162 | assert.Error(t, err, "uploading PathInfo with references to non-existing NarMeta should fail") 163 | 164 | // now try to upload NarMeta for A, then PathInfo for A, 165 | // then NarMeta for B, then PathInfo for B, which should succeed 166 | err = metadataStore.PutNarMeta(context.Background(), tdANarMeta) 167 | assert.NoError(t, err) 168 | err = metadataStore.PutPathInfo(context.Background(), tdAPathInfo) 169 | assert.NoError(t, err) 170 | err = metadataStore.PutNarMeta(context.Background(), tdBNarMeta) 171 | assert.NoError(t, err) 172 | err = metadataStore.PutPathInfo(context.Background(), tdBPathInfo) 173 | assert.NoError(t, err) 174 | }) 175 | 176 | err = metadataStore.DropAll(context.Background()) 177 | if err != nil { 178 | panic(err) 179 | } 180 | // upload NarMeta for A, then PathInfo for A, then a broken NarMeta for B 181 | t.Run("PutNarMeta with broken inconsistent references", func(t *testing.T) { 182 | err = metadataStore.PutNarMeta(context.Background(), tdANarMeta) 183 | assert.NoError(t, err) 184 | err = metadataStore.PutPathInfo(context.Background(), tdAPathInfo) 185 | assert.NoError(t, err) 186 | 187 | brokenNarMeta := *tdBNarMeta 188 | brokenNarMeta.References = [][]byte{} 189 | err = metadataStore.PutNarMeta(context.Background(), &brokenNarMeta) 190 | assert.Error(t, err, "uploading NarMeta with inconsistent References[Str] should fail") 191 | }) 192 | }) 193 | } 194 | -------------------------------------------------------------------------------- /pkg/store/metadatastore/types.go: -------------------------------------------------------------------------------- 1 | // Package metadatastore contains some datastructures that are part of a binary cache. 2 | package metadatastore 3 | 4 | import ( 5 | "bytes" 6 | "context" 7 | "fmt" 8 | "io" 9 | "strings" 10 | 11 | "github.com/flokli/nix-casync/pkg/server/compression" 12 | "github.com/flokli/nix-casync/pkg/util" 13 | "github.com/nix-community/go-nix/pkg/hash" 14 | "github.com/nix-community/go-nix/pkg/nar/narinfo" 15 | "github.com/nix-community/go-nix/pkg/nixbase32" 16 | ) 17 | 18 | type MetadataStore interface { 19 | GetPathInfo(ctx context.Context, outputHash []byte) (*PathInfo, error) 20 | PutPathInfo(ctx context.Context, pathInfo *PathInfo) error 21 | 22 | // TODO: once we have reference scanning, it shouldn't be possible to mutate existing NarMetas 23 | GetNarMeta(ctx context.Context, narHash []byte) (*NarMeta, error) 24 | PutNarMeta(ctx context.Context, narMeta *NarMeta) error 25 | DropAll(ctx context.Context) error 26 | io.Closer 27 | } 28 | 29 | type PathInfo struct { 30 | OutputHash []byte 31 | Name string 32 | 33 | NarHash []byte 34 | 35 | Deriver string 36 | System string 37 | NarinfoSignatures []*narinfo.Signature 38 | 39 | CA string 40 | } 41 | 42 | // ParseNarinfo parses a narinfo.NarInfo struct 43 | // and returns a PathInfo and NarMeta struct, or an error. 44 | func ParseNarinfo(narinfo *narinfo.NarInfo) (*PathInfo, *NarMeta, error) { 45 | // Ensure sha256 is used for hashing. 46 | if narinfo.NarHash.HashType != hash.HashTypeSha256 { 47 | return nil, nil, fmt.Errorf("unexpected hashtype: %v", narinfo.NarHash) 48 | } 49 | 50 | // Try to parse the StorePath field. 51 | // We need it later, but there's no need to lookup in NarMeta 52 | // if the StorePath field is already invalid. 53 | outputHash, err := util.GetHashFromStorePath(narinfo.StorePath) 54 | if err != nil { 55 | return nil, nil, fmt.Errorf("invalid StorePath field: %v", narinfo.StorePath) 56 | } 57 | 58 | pathInfo := &PathInfo{ 59 | OutputHash: outputHash, 60 | Name: util.GetNameFromStorePath(narinfo.StorePath), 61 | 62 | NarHash: narinfo.NarHash.Digest, 63 | 64 | Deriver: narinfo.Deriver, 65 | System: narinfo.System, 66 | NarinfoSignatures: narinfo.Signatures, 67 | 68 | CA: narinfo.CA, 69 | } 70 | 71 | // Construct References 72 | references := make([][]byte, 0, len(narinfo.References)) 73 | 74 | for _, referenceStr := range narinfo.References { 75 | hashRef, err := nixbase32.DecodeString(referenceStr[0:32]) 76 | if err != nil { 77 | return nil, nil, fmt.Errorf("unable to decode hash %v in reference %v: %w", referenceStr, narinfo.References, err) 78 | } 79 | 80 | references = append(references, hashRef) 81 | } 82 | 83 | narMeta := &NarMeta{ 84 | NarHash: narinfo.NarHash.Digest, 85 | Size: narinfo.NarSize, 86 | ReferencesStr: narinfo.References, 87 | References: references, 88 | } 89 | 90 | return pathInfo, narMeta, nil 91 | } 92 | 93 | // RenderNarinfo renders a minimal .narinfo from a PathInfo and NarMeta. 94 | // The URL is synthesized to /nar/$narhash.nar[$compressionSuffix]. 95 | func RenderNarinfo(pathInfo *PathInfo, narMeta *NarMeta, compressionType string) (string, error) { 96 | // render the narinfo 97 | narHash := &hash.Hash{ 98 | HashType: hash.HashTypeSha256, 99 | Digest: narMeta.NarHash, 100 | } 101 | narhashStr := nixbase32.EncodeToString(pathInfo.NarHash) 102 | narInfo := &narinfo.NarInfo{ 103 | StorePath: pathInfo.StorePath(), 104 | URL: "nar/" + narhashStr + ".nar", 105 | Compression: compressionType, 106 | 107 | NarHash: narHash, 108 | NarSize: narMeta.Size, 109 | 110 | References: narMeta.ReferencesStr, 111 | 112 | Deriver: pathInfo.Deriver, 113 | 114 | System: pathInfo.System, 115 | 116 | Signatures: pathInfo.NarinfoSignatures, 117 | 118 | CA: pathInfo.CA, 119 | } 120 | 121 | suffix, err := compression.TypeToSuffix(compressionType) 122 | if err != nil { 123 | return "", err 124 | } 125 | 126 | narInfo.URL += suffix 127 | 128 | return narInfo.String(), nil 129 | } 130 | 131 | func (pi *PathInfo) StorePath() string { 132 | return util.StoreDir + "/" + nixbase32.EncodeToString(pi.OutputHash) + "-" + pi.Name 133 | } 134 | 135 | func (pi *PathInfo) Check() error { 136 | if len(pi.OutputHash) != 20 { 137 | return fmt.Errorf("invalid outputhash: %v", nixbase32.EncodeToString(pi.OutputHash)) 138 | } 139 | 140 | if len(pi.Name) == 0 { 141 | return fmt.Errorf("invalid name: %v", pi.Name) 142 | } 143 | 144 | if len(pi.NarHash) != 32 { 145 | return fmt.Errorf("invalid narhash: %v", nixbase32.EncodeToString(pi.NarHash)) 146 | } 147 | // Derivers can be empty (when importing store paths), 148 | // but when they're not, they need to be at least long enough to 149 | // hold the output hash (base32 encoded), a dash and the name 150 | if !(len(pi.Deriver) == 0 || (strings.HasSuffix(pi.Deriver, ".drv") && len(pi.Deriver) > 32+1+1)) { 151 | return fmt.Errorf("invalid deriver: %v", pi.Deriver) 152 | } 153 | 154 | return nil 155 | } 156 | 157 | type NarMeta struct { 158 | NarHash []byte 159 | Size uint64 160 | 161 | References [][]byte // this refers to multiple PathInfo.OutputHash 162 | ReferencesStr []string // we still keep the strings around, so we don't need to look up all other PathInfo objects 163 | } 164 | 165 | // Check provides some sanity checking on values in the NarMeta struct. 166 | func (n *NarMeta) Check() error { 167 | if len(n.NarHash) != 32 { // 32 bytes = 256bits 168 | return fmt.Errorf("invalid narhash length: %v, must be 32", len(n.NarHash)) 169 | } 170 | 171 | if n.Size == 0 { 172 | return fmt.Errorf("invalid Size: %v", n.Size) 173 | } 174 | 175 | if len(n.References) != len(n.ReferencesStr) { 176 | return fmt.Errorf("inconsistent number of References[Str]") 177 | } 178 | 179 | // We need to be able to decode all store paths in ReferencesStr 180 | // and they should match the hashes stored in References 181 | for i, refStr := range n.ReferencesStr { 182 | hash, err := nixbase32.DecodeString(refStr[:32]) 183 | if err != nil { 184 | return fmt.Errorf("unable to encode hash from store path: %v", refStr) 185 | } 186 | 187 | if !bytes.Equal(hash, n.References[i]) { 188 | return fmt.Errorf( 189 | "inconsistent References and ReferencesStr at position %v: %v != %v", 190 | i, 191 | hash, 192 | n.References[i], 193 | ) 194 | } 195 | } 196 | 197 | return nil 198 | } 199 | 200 | // IsEqualTo returns true if the other NarMeta is equal to it 201 | // The compareReferences parameter controls whether references should be compared. 202 | func (n *NarMeta) IsEqualTo(other *NarMeta, compareReferences bool) bool { 203 | if !(n.Size == other.Size) { 204 | return false 205 | } 206 | 207 | if !bytes.Equal(n.NarHash, other.NarHash) { 208 | return false 209 | } 210 | 211 | if compareReferences { 212 | for i, refStr := range n.ReferencesStr { 213 | if refStr != other.ReferencesStr[i] { 214 | return false 215 | } 216 | } 217 | 218 | for i, refBytes := range n.References { 219 | if !bytes.Equal(refBytes, other.References[i]) { 220 | return false 221 | } 222 | } 223 | } 224 | 225 | return true 226 | } 227 | -------------------------------------------------------------------------------- /pkg/util/helpers.go: -------------------------------------------------------------------------------- 1 | package util 2 | 3 | import ( 4 | "github.com/nix-community/go-nix/pkg/nixbase32" 5 | ) 6 | 7 | const ( 8 | StoreDir = "/nix/store" // hardcoded for now 9 | ) 10 | 11 | // GetHashFromStorePath extracts the outputhash of a Nix Store path, and returns it decoded. 12 | func GetHashFromStorePath(storePath string) ([]byte, error) { 13 | offset := len(StoreDir) + 1 14 | 15 | return nixbase32.DecodeString(storePath[offset : offset+32]) 16 | } 17 | 18 | func GetNameFromStorePath(storePath string) string { 19 | offset := len(StoreDir) + 1 + 32 + 1 20 | 21 | return storePath[offset:] 22 | } 23 | -------------------------------------------------------------------------------- /pkg/util/helpers_test.go: -------------------------------------------------------------------------------- 1 | package util_test 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/flokli/nix-casync/pkg/util" 7 | "github.com/nix-community/go-nix/pkg/nixbase32" 8 | "github.com/stretchr/testify/assert" 9 | ) 10 | 11 | func TestStorePathHelpers(t *testing.T) { 12 | storePath := "/nix/store/dr76fsw7d6ws3pymafx0w0sn4rzbw7c9-etc-os-release" 13 | 14 | t.Run("getHashFromStorePath", func(t *testing.T) { 15 | hash, err := util.GetHashFromStorePath(storePath) 16 | assert.NoError(t, err) 17 | assert.Equal(t, "dr76fsw7d6ws3pymafx0w0sn4rzbw7c9", nixbase32.EncodeToString(hash)) 18 | }) 19 | 20 | t.Run("getNameFromStorePath", func(t *testing.T) { 21 | name := util.GetNameFromStorePath(storePath) 22 | assert.Equal(t, "etc-os-release", name) 23 | }) 24 | } 25 | -------------------------------------------------------------------------------- /test/7cwx623saf2h3z23wsn26icszvskk4iy.narinfo: -------------------------------------------------------------------------------- 1 | StorePath: /nix/store/7cwx623saf2h3z23wsn26icszvskk4iy-hello 2 | URL: nar/0rcdxyw7kjpxshv7wb1am0nvjfjbjq67cvrc8dmbsy1slc2ycbxp.nar 3 | Compression: none 4 | FileHash: sha256:0rcdxyw7kjpxshv7wb1am0nvjfjbjq67cvrc8dmbsy1slc2ycbxp 5 | FileSize: 328 6 | NarHash: sha256:0rcdxyw7kjpxshv7wb1am0nvjfjbjq67cvrc8dmbsy1slc2ycbxp 7 | NarSize: 328 8 | References: x236iz9shqypbnm64qgqisz0jr4wmj2b-txt 9 | Deriver: ss3sdmaabcqywsbd8bc9dnq8d203f90m-hello.drv 10 | -------------------------------------------------------------------------------- /test/fixtures.go: -------------------------------------------------------------------------------- 1 | package test 2 | 3 | import ( 4 | "bytes" 5 | _ "embed" 6 | "fmt" 7 | 8 | "github.com/nix-community/go-nix/pkg/nar/narinfo" 9 | ) 10 | 11 | //go:embed x236iz9shqypbnm64qgqisz0jr4wmj2b.narinfo 12 | var aNarinfoContents []byte 13 | 14 | //go:embed nar/0xmvxmsmmc6n79sk2h3r6db3yp8drmxps61mdk7iqnvc6vcsww60.nar 15 | var aNarContents []byte 16 | 17 | //go:embed 7cwx623saf2h3z23wsn26icszvskk4iy.narinfo 18 | var bNarinfoContents []byte 19 | 20 | //go:embed nar/0rcdxyw7kjpxshv7wb1am0nvjfjbjq67cvrc8dmbsy1slc2ycbxp.nar 21 | var bNarContents []byte 22 | 23 | //go:embed qp5h1cjd5ykcl4hyvsjhrlv68bbx8fan.narinfo 24 | var cNarinfoContents []byte 25 | 26 | //go:embed nar/0z2vk40phzzgsg14516mfs79l9fvl276b993mlqlb4rf0fd7hnwp.nar 27 | var cNarContents []byte 28 | 29 | type Data struct { 30 | NarinfoContents []byte 31 | Narinfo *narinfo.NarInfo 32 | NarContents []byte 33 | } 34 | 35 | type DataTable map[string]Data 36 | 37 | // GetTestDataTable returns testdata from //test 38 | // it's a map with the following store paths: 39 | // a is a store path without any references. 40 | // b refers to it. 41 | // c contains a self-reference. 42 | func GetTestDataTable() DataTable { 43 | testDataT := make(DataTable, 2) 44 | 45 | for _, item := range []struct { 46 | name string 47 | narinfoContents []byte 48 | narContents []byte 49 | }{ 50 | {name: "a", narinfoContents: aNarinfoContents, narContents: aNarContents}, 51 | {name: "b", narinfoContents: bNarinfoContents, narContents: bNarContents}, 52 | {name: "c", narinfoContents: cNarinfoContents, narContents: cNarContents}, 53 | } { 54 | // parse narinfo file 55 | narinfo, err := narinfo.Parse(bytes.NewReader(item.narinfoContents)) 56 | if err != nil { 57 | panic(fmt.Errorf("error parsing narinfo contents: %w", err)) 58 | } 59 | 60 | testDataT[item.name] = Data{ 61 | NarinfoContents: item.narinfoContents, 62 | Narinfo: narinfo, 63 | NarContents: item.narContents, 64 | } 65 | } 66 | 67 | return testDataT 68 | } 69 | -------------------------------------------------------------------------------- /test/generator/default.nix: -------------------------------------------------------------------------------- 1 | # This creates two derivations which depend on each other. 2 | with import { }; 3 | 4 | let 5 | txt = writeText "txt" '' 6 | Hello, World! 7 | ''; 8 | in 9 | { 10 | b = stdenv.mkDerivation { 11 | name = "hello"; 12 | dontUnpack = true; 13 | dontBuild = true; 14 | installPhase = '' 15 | mkdir -p $out 16 | ln -sfn ${txt} $out/txt 17 | ''; 18 | }; 19 | c = writeText "self" '' 20 | ${placeholder "out"} 21 | ''; 22 | } 23 | -------------------------------------------------------------------------------- /test/nar/0rcdxyw7kjpxshv7wb1am0nvjfjbjq67cvrc8dmbsy1slc2ycbxp.nar: -------------------------------------------------------------------------------- 1 | nix-archive-1(type directoryentry(nametxtnode(typesymlinktarget//nix/store/x236iz9shqypbnm64qgqisz0jr4wmj2b-txt))) -------------------------------------------------------------------------------- /test/nar/0xmvxmsmmc6n79sk2h3r6db3yp8drmxps61mdk7iqnvc6vcsww60.nar: -------------------------------------------------------------------------------- 1 | nix-archive-1(typeregularcontentsHello, World! 2 | ) -------------------------------------------------------------------------------- /test/nar/0z2vk40phzzgsg14516mfs79l9fvl276b993mlqlb4rf0fd7hnwp.nar: -------------------------------------------------------------------------------- 1 | nix-archive-1(typeregularcontents1/nix/store/qp5h1cjd5ykcl4hyvsjhrlv68bbx8fan-self 2 | ) -------------------------------------------------------------------------------- /test/qp5h1cjd5ykcl4hyvsjhrlv68bbx8fan.narinfo: -------------------------------------------------------------------------------- 1 | StorePath: /nix/store/qp5h1cjd5ykcl4hyvsjhrlv68bbx8fan-self 2 | URL: nar/0z2vk40phzzgsg14516mfs79l9fvl276b993mlqlb4rf0fd7hnwp.nar 3 | Compression: none 4 | FileHash: sha256:0z2vk40phzzgsg14516mfs79l9fvl276b993mlqlb4rf0fd7hnwp 5 | FileSize: 168 6 | NarHash: sha256:0z2vk40phzzgsg14516mfs79l9fvl276b993mlqlb4rf0fd7hnwp 7 | NarSize: 168 8 | References: qp5h1cjd5ykcl4hyvsjhrlv68bbx8fan-self 9 | Deriver: ms5j3qkm1s8y90jnz557011abscn38v0-self.drv 10 | -------------------------------------------------------------------------------- /test/x236iz9shqypbnm64qgqisz0jr4wmj2b.narinfo: -------------------------------------------------------------------------------- 1 | StorePath: /nix/store/x236iz9shqypbnm64qgqisz0jr4wmj2b-txt 2 | URL: nar/0xmvxmsmmc6n79sk2h3r6db3yp8drmxps61mdk7iqnvc6vcsww60.nar 3 | Compression: none 4 | FileHash: sha256:0xmvxmsmmc6n79sk2h3r6db3yp8drmxps61mdk7iqnvc6vcsww60 5 | FileSize: 128 6 | NarHash: sha256:0xmvxmsmmc6n79sk2h3r6db3yp8drmxps61mdk7iqnvc6vcsww60 7 | NarSize: 128 8 | References: 9 | Deriver: bbqmya8wi5wyyjgmp5qxdqackglh5s4i-txt.drv 10 | --------------------------------------------------------------------------------