├── docs └── diagrams │ ├── with-artifact-storage.drawio.png │ └── without-artifact-storage.drawio.png ├── test ├── Caddyfile └── main.tf ├── go.mod ├── go.sum ├── .gitignore ├── LICENSE ├── .circleci └── config.yml ├── main.go └── README.md /docs/diagrams/with-artifact-storage.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasonwbarnett/terraform-registry-proxy/HEAD/docs/diagrams/with-artifact-storage.drawio.png -------------------------------------------------------------------------------- /docs/diagrams/without-artifact-storage.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jasonwbarnett/terraform-registry-proxy/HEAD/docs/diagrams/without-artifact-storage.drawio.png -------------------------------------------------------------------------------- /test/Caddyfile: -------------------------------------------------------------------------------- 1 | terraform-registry.dev.local, hashicorp-releases.dev.local { 2 | reverse_proxy /* http://localhost:8555 3 | 4 | tls /tmp/terraform-registry.crt /tmp/terraform-registry.key 5 | } 6 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/jasonwbarnett/terraform-registry-proxy 2 | 3 | go 1.17 4 | 5 | require ( 6 | github.com/felixge/httpsnoop v1.0.1 // indirect 7 | github.com/gorilla/handlers v1.5.1 // indirect 8 | ) 9 | -------------------------------------------------------------------------------- /test/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | azurerm = { 4 | source = "terraform-registry.dev.local/hashicorp/azurerm" 5 | version = "=2.97.0" 6 | } 7 | } 8 | } 9 | 10 | # Configure the Microsoft Azure Provider 11 | provider "azurerm" { 12 | features {} 13 | } 14 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/felixge/httpsnoop v1.0.1 h1:lvB5Jl89CsZtGIWuTcDM1E/vkVs49/Ml7JJe07l8SPQ= 2 | github.com/felixge/httpsnoop v1.0.1/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= 3 | github.com/gorilla/handlers v1.5.1 h1:9lRY6j8DEeeBT10CvO9hGW0gmky0BprnvDI5vfhUHH4= 4 | github.com/gorilla/handlers v1.5.1/go.mod h1:t8XrUpc4KVXb7HGyJ4/cEnwQiaxrX/hz1Zv/4g96P1Q= 5 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | terraform-registry-proxy* 3 | *.exe 4 | *.exe~ 5 | *.dll 6 | *.so 7 | *.dylib 8 | 9 | # Test binary, built with `go test -c` 10 | *.test 11 | 12 | # Output of the go coverage tool, specifically when used with LiteIDE 13 | *.out 14 | 15 | # Dependency directories (remove the comment below to include it) 16 | # vendor/ 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Jason Barnett 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | # Use the latest 2.1 version of CircleCI pipeline process engine. 2 | # See: https://circleci.com/docs/2.0/configuration-reference 3 | version: 2.1 4 | 5 | # Define a job to be invoked later in a workflow. 6 | # See: https://circleci.com/docs/2.0/configuration-reference/#jobs 7 | jobs: 8 | test: 9 | working_directory: ~/repo 10 | # Specify the execution environment. You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub. 11 | # See: https://circleci.com/docs/2.0/configuration-reference/#docker-machine-macos-windows-executor 12 | docker: 13 | - image: golang:latest 14 | resource_class: small 15 | # Add steps to the job 16 | # See: https://circleci.com/docs/2.0/configuration-reference/#steps 17 | steps: 18 | - checkout 19 | - restore_cache: 20 | keys: 21 | - go-mod-v6-{{ checksum "go.sum" }} 22 | - run: 23 | name: Install Dependencies 24 | command: go mod download 25 | - save_cache: 26 | key: go-mod-v6-{{ checksum "go.sum" }} 27 | paths: 28 | - "/go/pkg/mod" 29 | - run: 30 | name: build binary 31 | command: | 32 | go build 33 | - run: 34 | name: setup local hosts 35 | command: | 36 | echo "127.0.0.1 terraform-registry.local.dev" | tee -a /etc/hosts 37 | echo "127.0.0.1 hashicorp-releases.local.dev" | tee -a /etc/hosts 38 | - run: 39 | name: generate ssl certificate 40 | command: | 41 | openssl req -x509 -newkey rsa:2048 -sha256 -days 1 -nodes \ 42 | -keyout /tmp/terraform-registry.key -out /tmp/terraform-registry.crt -subj "/CN=terraform-registry.local.dev" \ 43 | -addext "subjectAltName=DNS:terraform-registry.local.dev,DNS:hashicorp-releases.local.dev" 44 | - run: 45 | name: install terraform 46 | command: | 47 | apt update 48 | apt install -y unzip 49 | curl -LO https://releases.hashicorp.com/terraform/1.1.7/terraform_1.1.7_linux_amd64.zip 50 | unzip terraform_1.1.7_linux_amd64.zip 51 | chmod +x terraform 52 | mv terraform /usr/local/bin 53 | terraform version 54 | - run: 55 | name: install caddy 56 | command: | 57 | apt update 58 | apt install -y debian-keyring debian-archive-keyring apt-transport-https 59 | curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | tee /etc/apt/trusted.gpg.d/caddy-stable.asc 60 | curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list 61 | apt update 62 | apt install -y caddy 63 | - run: 64 | name: configure caddy 65 | command: | 66 | \cp -f ./test/Caddyfile /etc/caddy/Caddyfile 67 | /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile & 68 | 69 | # Invoke jobs via workflows 70 | # See: https://circleci.com/docs/2.0/configuration-reference/#workflows 71 | workflows: 72 | end to end test: 73 | jobs: 74 | - test 75 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "compress/gzip" 6 | "flag" 7 | "fmt" 8 | "io" 9 | "io/ioutil" 10 | "net/http" 11 | "net/http/httputil" 12 | "net/url" 13 | "os" 14 | "strconv" 15 | "strings" 16 | 17 | "github.com/gorilla/handlers" 18 | ) 19 | 20 | // WebReverseProxyConfiguration is a coniguration for the ReverseProxy 21 | type WebReverseProxyConfiguration struct { 22 | RegistryProxyHost string 23 | ReleaseProxyHost string 24 | ReleasePathPrefix string 25 | } 26 | 27 | var ( 28 | registryHost string 29 | releaseHost string 30 | releasePathPrefix string 31 | httpAddress string 32 | ) 33 | 34 | func init() { 35 | flag.StringVar(®istryHost, "registry-proxy-host", "", "FQDN of registry proxy host [Required]") 36 | flag.StringVar(&releaseHost, "release-proxy-host", "", "FQDN of release proxy host [Required]") 37 | flag.StringVar(&releasePathPrefix, "release-proxy-path-prefix", "", "The prefix path to prepend to any release artifact paths. This might be /artifactory/hashicorp-releases") 38 | flag.StringVar(&httpAddress, "http-address", ":8555", "HTTP address to listen on, e.g. :8080 or 127.0.0.1:8080") 39 | flag.Parse() 40 | 41 | if registryHost == "" { 42 | fmt.Printf("You must provide a -registry-proxy-host value\n\n") 43 | flag.Usage() 44 | os.Exit(1) 45 | } 46 | 47 | if releaseHost == "" { 48 | fmt.Printf("You must provide a -release-proxy-host value\n\n") 49 | flag.Usage() 50 | os.Exit(1) 51 | } 52 | } 53 | 54 | func main() { 55 | config := &WebReverseProxyConfiguration{ 56 | RegistryProxyHost: registryHost, 57 | ReleaseProxyHost: releaseHost, 58 | ReleasePathPrefix: releasePathPrefix, 59 | } 60 | proxy := config.NewWebReverseProxy() 61 | http.Handle("/", handlers.LoggingHandler(os.Stdout, proxy)) 62 | 63 | // Start the server 64 | http.ListenAndServe(httpAddress, nil) 65 | } 66 | 67 | // This replaces all occurrences of http://releases.hashicorp.com with 68 | // config.ReleaseProxyHost in the response body 69 | func (config *WebReverseProxyConfiguration) rewriteBody(resp *http.Response) (err error) { 70 | // Check that the server actually sent compressed data 71 | var reader io.ReadCloser 72 | switch resp.Header.Get("Content-Encoding") { 73 | case "gzip": 74 | reader, err = gzip.NewReader(resp.Body) 75 | resp.Header.Del("Content-Encoding") 76 | resp.Header.Del("Content-Length") 77 | resp.ContentLength = -1 78 | resp.Uncompressed = true 79 | defer reader.Close() 80 | default: 81 | reader = resp.Body 82 | } 83 | 84 | b, err := ioutil.ReadAll(reader) 85 | if err != nil { 86 | return err 87 | } 88 | 89 | if err = resp.Body.Close(); err != nil { 90 | return err 91 | } 92 | 93 | replacement := fmt.Sprintf("https://%s%s", config.ReleaseProxyHost, config.ReleasePathPrefix) 94 | 95 | b = bytes.ReplaceAll(b, []byte("https://releases.hashicorp.com"), []byte(replacement)) // releases 96 | body := ioutil.NopCloser(bytes.NewReader(b)) 97 | resp.Body = body 98 | resp.ContentLength = int64(len(b)) 99 | resp.Header.Set("Content-Length", strconv.Itoa(len(b))) 100 | return nil 101 | } 102 | 103 | func (config *WebReverseProxyConfiguration) NewWebReverseProxy() *httputil.ReverseProxy { 104 | director := func(req *http.Request) { 105 | if req.Host == config.RegistryProxyHost { 106 | req.URL.Scheme = "https" 107 | req.URL.Host = "registry.terraform.io" 108 | req.Host = "registry.terraform.io" 109 | req.Header.Set("User-Agent", "Terraform/1.1.7") 110 | req.Header.Set("X-Terraform-Version", "1.1.7") 111 | } else if req.Host == config.ReleaseProxyHost { 112 | req.URL.Scheme = "https" 113 | req.URL.Host = "releases.hashicorp.com" 114 | req.Host = "releases.hashicorp.com" 115 | req.Header.Set("User-Agent", "Terraform/1.1.7") 116 | } 117 | } 118 | 119 | responseDirector := func(res *http.Response) error { 120 | if server := res.Header.Get("Server"); strings.HasPrefix(server, "terraform-registry") { 121 | if err := config.rewriteBody(res); err != nil { 122 | fmt.Println("Error rewriting body!") 123 | return err 124 | } 125 | } 126 | 127 | if location := res.Header.Get("Location"); location != "" { 128 | url, err := url.ParseRequestURI(location) 129 | if err != nil { 130 | fmt.Println("Error!") 131 | return err 132 | } 133 | 134 | // Override redirect url Host with ProxyHost 135 | url.Host = config.RegistryProxyHost 136 | 137 | res.Header.Set("Location", url.String()) 138 | res.Header.Set("X-Reverse-Proxy", "terraform-registry-proxy") 139 | } 140 | return nil 141 | } 142 | 143 | return &httputil.ReverseProxy{ 144 | Director: director, 145 | ModifyResponse: responseDirector, 146 | } 147 | } 148 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # terraform-registry-proxy 2 | 3 | This app is useful if you run Terraform in an offline / airgappped / no internet connected environment. 4 | 5 | ## What is this? 6 | 7 | Hashicorp for some reason either hasn't prioritized or outright refuses to make 8 | it easy to ingest Terraform providers and modules like you might traditionally 9 | see in an Artifactory so that you don't have to cache the plugins and unzip them. 10 | Instead just let the `terraform` cli natively fetch things on-demand in an 11 | environment where direct internet access is not possible. 12 | 13 | This application is intended to be put behind a web server, e.g. [NGINX][1], [Caddy][2], 14 | [Apache][3], etc. 15 | 16 | ## How does it work? 17 | 18 | The tiny proxy app is really quite simple and does two things: 19 | 20 | 1. Proxies requests to https://registry.terraform.io and https://releases.hashicorp.com (optionally, if not using external artifact storage) 21 | 2. Re-write response bodies to update where Artifacts should be fetched from (configurable). 22 | - For example, it will replace: 23 | - original url: `https://releases.hashicorp.com/terraform-provider-azurerm/2.97.0/terraform-provider-azurerm_2.97.0_darwin_amd64.zip` 24 | - re-written to: `https://hashicorp-releases.company.com/terraform-provider-azurerm/2.97.0/terraform-provider-azurerm_2.97.0_darwin_amd64.zip` 25 | - or with Artifactory re-written to: `https://artifactory.company.com/artifactory/hashicorp-releases/terraform-provider-azurerm/2.97.0/terraform-provider-azurerm_2.97.0_darwin_amd64.zip` 26 | 27 | ## Requirements 28 | 29 | - web server 30 | - ssl certificate(s) that is/are trusted by the client where `terraform` is 31 | being run 32 | - dns record dedicated to terraform registry proxy (e.g. `terraform-registry.company.com`) 33 | - update sources in your terraform configurations 34 | 35 | ### Optionally 36 | 37 | - artifact storage (e.g. Artifactory) 38 | - dns record dedicated to hashicorp releases proxy (e.g. `hashicorp-releases.company.com`) 39 | 40 | ## Usage 41 | 42 | Two possible usages: 43 | 44 | 1. Without external artifact storage 45 | 2. With external artifact storage 46 | 47 | Read each section below for more details. We will use [Artifactory][4] as our example 48 | artifact storage and the Caddy web server for our examples as well, but any web 49 | server or artifact storage should work. 50 | 51 | These diagrams are not intended to be recommendations for specific architectures 52 | but simply showing you examples of possible ways to set it up to make it easier 53 | to get familiar with how it works. 54 | 55 | ### Without external artifact storage 56 | 57 | In this scenario both https://registry.terraform.io and 58 | https://releases.hashicorp.com are proxied through this app. 59 | 60 | You will need to setup two DNS records pointing to the web server where 61 | `terraform-registry-proxy` is running, i.e. 62 | 63 | - `terraform-registry.company.com` 64 | - `hashicorp-releases.company.com` 65 | 66 | ![with artifact storage](/docs/diagrams/without-artifact-storage.drawio.png?raw=true) 67 | 68 | ```bash 69 | ./terraform-registry-proxy -registry-proxy-host terraform-registry.company.com \ 70 | -release-proxy-host hashicorp-releases.company.com 71 | ``` 72 | 73 | ### With external artifact storage 74 | 75 | In this scenario only https://registry.terraform.io is proxied through this app. 76 | 77 | You will need to setup one DNS record pointing to the web server where 78 | `terraform-registry-proxy` is running, i.e. 79 | 80 | - `terraform-registry.company.com` 81 | 82 | It also assumes you're already proxying / mirroring https://releases.hashicorp.com. 83 | 84 | ![with artifact storage](/docs/diagrams/with-artifact-storage.drawio.png?raw=true) 85 | 86 | ```bash 87 | ./terraform-registry-proxy -registry-proxy-host terraform-registry.company.com \ 88 | -release-proxy-host artifactory.company.com \ 89 | -release-proxy-path-prefix /artifactory/hashicorp-releases 90 | ``` 91 | 92 | This assumes you've configured a generic remote repository named 93 | `hashicorp-releases` for https://registry.terraform.io in your Artifactory 94 | instance. 95 | 96 | ### Update sources in Terraform configurations 97 | 98 | After you have your infrastructure setup you need to update your Terraform 99 | configuration so it knows to pull dependencies through the proxy. 100 | 101 | Say for example this is your original configuration: 102 | 103 | ```terraform 104 | terraform { 105 | required_providers { 106 | azurerm = { 107 | source = "hashicorp/azurerm" 108 | version = "=2.97.0" 109 | } 110 | } 111 | } 112 | 113 | # Configure the Microsoft Azure Provider 114 | provider "azurerm" { 115 | features {} 116 | } 117 | ``` 118 | 119 | You would update it to this 120 | 121 | ```terraform 122 | terraform { 123 | required_providers { 124 | azurerm = { 125 | source = "terraform-registry.company.com/hashicorp/azurerm" 126 | version = "=2.97.0" 127 | } 128 | } 129 | } 130 | 131 | # Configure the Microsoft Azure Provider 132 | provider "azurerm" { 133 | features {} 134 | } 135 | ``` 136 | 137 | [1]: https://nginx.org/en/ 138 | [2]: https://caddyserver.com/ 139 | [3]: https://httpd.apache.org/ 140 | [4]: https://jfrog.com/artifactory/ 141 | --------------------------------------------------------------------------------