├── Dockerfile ├── README.md ├── config ├── default.yaml └── templates │ └── squid.conf.mako └── scripts └── start.sh /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM sameersbn/squid 2 | MAINTAINER ryan@enigmacurry.com 3 | 4 | RUN apt-get update && apt-get install -y python3 python3-yaml python3-mako 5 | 6 | ADD config /docker_configurator 7 | ADD https://raw.githubusercontent.com/EnigmaCurry/docker-configurator/master/docker_configurator.py /docker_configurator/docker_configurator.py 8 | ADD scripts/start.sh / 9 | 10 | EXPOSE 8080/tcp 11 | VOLUME ["/var/spool/squid3"] 12 | 13 | ENTRYPOINT ["/start.sh"] 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Lazy Distro Mirrors 2 | 3 | I have a pretty slow internet connection. I don't run a computer lab. 4 | I don't have users. But, I do have multiple machines running various 5 | distros of Linux. If I update them, they tend to download a lot of 6 | redundant data, which is taxing on my connection. I could run a local 7 | mirror of the Arch and Debian repositories I use, but that seems like 8 | overkill. Really what I want is to just be able to cache files that I 9 | download once so that I don't have to re-download them a second time. 10 | This is the job for a caching proxy server, like squid. Except, HTTP 11 | proxies are a pain to setup, especially with package managers. I want 12 | something that just works. 13 | 14 | Lazy Distro Mirrors is a docker container that hosts a *reverse* squid 15 | proxy. This is effectively a local mirror of a remote website (like 16 | http://mirrors.kernel.org), with transparent caching. This is not a 17 | full mirror, but it looks and behaves like one. When you download a 18 | file for the first time, it goes to the remote mirror to fetch it. 19 | Subsequent retrievals will fetch it from the local cache. It will do 20 | this until you fill up the cache (20GB by default) and then it will 21 | start deleting the least recently used file. 22 | 23 | The benefit of a reverse proxy is that you don't have to configure an 24 | HTTP proxy, instead you edit the package manager configuration to use 25 | a local URL for the mirror to use. The hard configuration is done once, 26 | on one computer, and then all the rest of the computers on your LAN 27 | get to use it by just choosing the correct mirror URL. 28 | 29 | Thank you to [Matt Wagner and his blog post](http://ma.ttwagner.com/lazy-distro-mirrors-with-squid/). 30 | I copied most of his squid configuration for this. I also copied 31 | the name he used, as I couldn't think of anything better. Being lazy 32 | seemed appropriate. 33 | 34 | ## Usage 35 | 36 | You need docker, install it however you like. [Here's the official docs 37 | for that](https://docs.docker.com/linux/). 38 | 39 | You'll need two directories, one to place the config file, and one for 40 | the squid cache directory. For example: 41 | 42 | mkdir -p /opt/docker/lazy-distro-mirrors/config 43 | mkdir -p /opt/docker/lazy-distro-mirrors/cache 44 | 45 | Download and start the docker container: 46 | 47 | docker run \ 48 | --name lazy-distro-mirrors \ 49 | -d --restart=always \ 50 | --publish 8080:8080 \ 51 | --volume /opt/docker/lazy-distro-mirrors/config:/docker_configurator/user \ 52 | --volume /opt/docker/lazy-distro-mirrors/cache:/var/spool/squid \ 53 | enigmacurry/lazy-distro-mirrors 54 | 55 | Replace with the directory names you chose. Replace the first `8080` 56 | with the port you want the proxy to run on. 57 | 58 | By now the proxy should be running on the port you chose. 59 | 60 | On first running the container, it will have written it's config file 61 | to the config directory you specified. Open up config.yaml in that 62 | location. 63 | 64 | You'll see some settings you may want to edit: 65 | 66 | mirrors: 67 | kernel-mirror: http://mirrors.kernel.org 68 | xmission-mirror: http://mirrors.xmission.com 69 | advancedhosters-mirror: http://mirrors.advancedhosters.com 70 | 71 | These are the three default mirrors I use, but you can add your own 72 | here. Each mirror has a name and a URL. Just edit the file, save it, 73 | and restart the container: 74 | 75 | docker restart lazy-distro-mirrors 76 | 77 | Finally, Squid relies upon name-based resolution to pick the mirror 78 | you're using. For instance, to use the kernel.org mirror, you'll need 79 | to go to http://kernel-mirror:8080/ - this is the local URL for that 80 | mirror. If you have a DNS server running on your router, add the IP 81 | address of your server running docker and assign it the name `kernel-mirror`. 82 | If you don't run your own DNS server (I highly recommend [dd-wrt with 83 | dnsmasq](http://cybernetnews.com/local-internal-dns-ddwrt/)), you can 84 | also just add it to your /etc/hosts file, but you'll have to do this 85 | on each machine you want to use it with: 86 | 87 | 127.0.0.1 kernel-mirror xmission-mirror advancedhosters-mirror 88 | 89 | Do the same for all mirrors you define. 90 | 91 | Now you can switch between mirrors by going to the different local 92 | URLs: 93 | 94 | http://kernel-mirror:8080/ 95 | http://xmission-mirror:8080/ 96 | http://advancedhosters-mirror:8080/ 97 | 98 | ## Package manager configuration 99 | 100 | In order to get your distro to use this for package updates, you'll 101 | need to select the local URL as your primary mirror location. Consult 102 | your distribution's documentation for how best to do that. Here's how 103 | to do it briefly: 104 | 105 | For Arch Linux, edit /etc/pacman.d/mirrorlist, put this at the top: 106 | 107 | Server = http://kernel-mirror:8080/archlinux/$repo/os/$arch 108 | 109 | If you want to force using this mirror, delete everything else in the 110 | file. 111 | 112 | For anything using apt-get, edit /etc/apt/sources.list and replace the 113 | domain with your local one. For example: 114 | 115 | deb http://us.archive.ubuntu.com/ubuntu/ trusty main restricted 116 | 117 | Becomes: 118 | 119 | deb http://kernel-mirror:8080/ubuntu/ trusty main restricted 120 | 121 | 122 | -------------------------------------------------------------------------------- /config/default.yaml: -------------------------------------------------------------------------------- 1 | # Template map - 2 | # This is a list of all the config template files and the path to 3 | # render them to. Example: 4 | # 5 | # template_map: 6 | # my_config.mako: /etc/my_app/my_config.cfg 7 | # second_config.mako: /etc/my_other_app/name.conf 8 | # 9 | template_map: 10 | squid.conf.mako: /etc/squid/squid.conf 11 | 12 | # Max size of cache in MB 13 | cache_size: 20000 14 | 15 | # List of mirrors - local name -> URL 16 | # Squid uses name-based resolution to choose the mirror. You will need to setup 17 | # your local DNS (or edit /etc/hosts) for the names of the mirrors below and 18 | # point them to the IP address of your docker server 19 | mirrors: 20 | kernel-mirror: http://mirrors.kernel.org 21 | xmission-mirror: http://mirrors.xmission.com 22 | advancedhosters-mirror: http://mirrors.advancedhosters.com 23 | -------------------------------------------------------------------------------- /config/templates/squid.conf.mako: -------------------------------------------------------------------------------- 1 | #Kudos to http://ma.ttwagner.com/lazy-distro-mirrors-with-squid/ most 2 | #of the parameters here were taken from that. 3 | 4 | http_access allow all 5 | http_port 8080 accel defaultsite=kernel-mirror vhost 6 | 7 | # This needs to be specified before the cache_dir, otherwise it's ignored ?! 8 | maximum_object_size 4096 MB 9 | 10 | # Cache settings 11 | cache_dir ufs /var/spool/squid ${cache_size} 16 256 12 | 13 | cache_replacement_policy heap LFUDA 14 | 15 | refresh_pattern -i .rpm$ 129600 100% 129600 refresh-ims override-expire 16 | refresh_pattern -i .iso$ 129600 100% 129600 refresh-ims override-expire 17 | refresh_pattern -i .deb$ 129600 100% 129600 refresh-ims override-expire 18 | refresh_pattern -i .tar.xz$ 129600 100% 129600 refresh-ims override-expire 19 | refresh_pattern ^ftp: 1440 20% 10080 20 | refresh_pattern ^gopher: 1440 0% 1440 21 | refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 22 | refresh_pattern . 0 20% 4320 23 | 24 | % for mirror, url in mirrors.items(): 25 | <% 26 | import urllib 27 | url_parsed = urllib.parse.urlparse(url) 28 | hostname = url_parsed.hostname 29 | port = url_parsed.port 30 | if port is None: 31 | port = 80 32 | %> 33 | cache_peer ${hostname} parent ${port} 0 no-query originserver name=${mirror} 34 | cache_peer_domain ${mirror} ${mirror} 35 | % endfor 36 | -------------------------------------------------------------------------------- /scripts/start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | python3 /docker_configurator/docker_configurator.py 5 | 6 | exec /sbin/entrypoint.sh 7 | 8 | --------------------------------------------------------------------------------