├── README.md
├── cache
    └── nix-cache-info
├── get_build_hashes.sh
├── get_build_outputs.sh
├── get_builds.sh
├── run.sh
└── update.sh


/README.md:
--------------------------------------------------------------------------------
  1 | # Requirements:
  2 | 
  3 | * jq to parse berlin's API responses (in $PATH)
  4 | * curl to download things
  5 | * ipfs to do ipfs things (in $PATH)
  6 | * a running ipfs daemon that is online
  7 | 
  8 | # How to run:
  9 | 
 10 | Change to the directory containing guix-ipfs-cache and run
 11 | <code>bash run.sh</code>
 12 | 
 13 | # Rationale
 14 | 
 15 | TL;DR: ipfs allows sharing of whole directory structures under persistent names (ipns), even if the directory structure changes. Since <code>guix package --substitute-url="http://localhost:8080/some/path/to/repository/of/packages"</code> expects a certain directory structure we can try and share this directory structure via ipfs/ipns. Click this link for an example:
 16 | 
 17 | http://ipfs.io/ipns/QmPMJYhxbeaSYXzNLMRbvvJknpYcJG9DcG8h2kJJmukd9i
 18 | 
 19 | (This is running via the gateway to ipfs provided by the ipfs people). Once a user has a running ipfs daemon locally they can use this:
 20 | 
 21 | http://localhost:8080/ipns/QmPMJYhxbeaSYXzNLMRbvvJknpYcJG9DcG8h2kJJmukd9i
 22 | 
 23 | as substitute-url. Read on below for some more details if interested.
 24 | 
 25 | ## ipfs daemon as "local" guix substitute-mirror
 26 | 
 27 | The ipfs daemon provides a local http-proxy to the ipfs network that allows retrieving files by their CID (content-id), a unique identifier for the respective file that derived from its contents. The default port of this proxy is TCP/8080. An url to retrieve a file from the daemon would e.g. look like this:
 28 | 
 29 | http://localhost:8080/ipfs/QmbiJmTexTp1YBv3s7eKXDTzKGaKMTLGuns2xscmhdhadu
 30 | 
 31 | These can be retrieved by any HTTP-Client:
 32 | 
 33 | <pre>
 34 | $ curl http://localhost:8080/ipfs/QmbiJmTexTp1YBv3s7eKXDTzKGaKMTLGuns2xscmhdhadu
 35 | StorePath: /gnu/store/y857mykfnc9vd1sc2lqz5g35l111fp71-opensmtpd-test
 36 | URL: nar/gzip/y857mykfnc9vd1sc2lqz5g35l111fp71-opensmtpd-test
 37 | Compression: gzip
 38 | NarHash: sha256:136lm3fxh4kbsgz32157cnmyypgynkkmrqqb03biym1nm6cdwyv7
 39 | NarSize: 2600
 40 | References: 2sznibwwvp6x0ha2j6n8s1z1brf8ra4q-opensmtpd-test-builder
 41 | FileSize: 894
 42 | System: x86_64-linux
 43 | Deriver: 8aski1xwcn6lj112yc9ylgp2y3c7sj83-opensmtpd-test.drv
 44 | Signature: 1;berlin.guixsd.org;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjQTdCQzIwRjU0QzQ1NEYyRUE1QjQ1NTU1QzExN0VDOEYzMjA1RjlEQkM4N0ZDMzc4M0M1NjBCREM5REZCQUI3QyMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjQzhGRjZGMTMyNUFENzRBRjhDRDVDOTEzREE2NThFMjM5QzJCOEQ1QjZGNzM5REM1ODdDMTVERDVCODA3RkQjKQogICAocyAjMENCMTA3NUI5NDI5MUNEODVFMDI5NDQ4N0YzMTk1QUI5RTI5ODE5ODBFQzg2RDc1NDZDMDYyRjQ1NTNBRjEzQiMpCiAgICkKICApCiAocHVibGljLWtleSAKICAoZWNjIAogICAoY3VydmUgRWQyNTUxOSkKICAgKHEgIzhEMTU2RjI5NUQyNEIwRDlBODZGQTU3NDFBODQwRkYyRDI0RjYwRjdCNkM0MTM0ODE0QUQ1NTYyNTk3MUIzOTQjKQogICApCiAgKQogKQo=
 45 | </pre>
 46 | 
 47 | Individual files can be added to the ipfs-network with the <code>ipfs add</code>. There are other mechanisms, but we'll stick to this high level view for now.
 48 | 
 49 | In addition to providing files, ipfs supports directories (also identified by CIDs). These directories are implemented as Merkle-DAGs -- a form of cryptographic data structure that is a directed acyclic graph (DAG). 
 50 | 
 51 | <pre>
 52 | $ mkdir test
 53 | $ touch test/foo
 54 | $ ipfs add -r test/
 55 | added QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH test/foo
 56 | added QmWLU2zkdpnTSzKEu1RLN43mFjayCJzXrnE8nP1KT2htNd test
 57 |  0 B / ? [----------------------------------------------------------------------------------------------------=]   0.00%
 58 | </pre>
 59 | 
 60 | One nice feature is that the ipfs daemon proxy supports lookup of files relative to a directory by clear names:
 61 | 
 62 | <pre>
 63 | $ curl http://localhost:8080/ipfs/QmWLU2zkdpnTSzKEu1RLN43mFjayCJzXrnE8nP1KT2htNd/foo
 64 | $ 
 65 | </pre>
 66 | 
 67 | In this case the file was empty. To illustrate one important point about directories in ipfs, let's change the file's content:
 68 | 
 69 | <pre>
 70 | $ echo "Hello, world!" > test/foo 
 71 | $ ipfs add -r test/
 72 | added QmeeLUVdiSTTKQqhWqsffYDtNvvvcTfJdotkNyi1KDEJtQ test/foo
 73 | added QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1 test
 74 |  14 B / 14 B [=================================================================================================] 100.00%
 75 | </pre>
 76 | 
 77 | The attentive reader will have noticed that the resulting CID of the <code>test</code> directory has changed. This will happen whenever a file's content changed or the structure of the directory changed (e.g. by adding new or removing existing files from the directory).
 78 | 
 79 | <pre>
 80 | $ curl http://localhost:8080/ipfs/QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1/foo
 81 | Hello, world!
 82 | </pre>
 83 | 
 84 | At this point we can imagine how a local ipfs daemon can act as substitute-url for guix: We simply publish a directory structure on ipfs that resembles the structure the <code>guix package</code> expects, namely:
 85 | 
 86 | * A bunch of <code>narinfo</code> files directly in the root of the directory
 87 | * A nix-cache-control text file in the root of the directory
 88 | * A bunch of <code>nar</code> files in a <code>nar/gzip</code> subdirectory.
 89 | 
 90 | An example of this would be <code>https://ipfs.io/ipfs/QmR6y77ijvafrZanwxw63Q2QHkMBK4PbhTxSgTC9m3Un3o</code> (ipfs.io runs a gateway similar to the local ipfs daemon, so please click on this link to browse the directory).
 91 | 
 92 | A user could, in principle, now use an url like this in a call to <code>guix package</code>.
 93 | 
 94 | <pre>
 95 | guix package --substitute-url="http://localhost:8080/ipfs/QmR6y77ijvafrZanwxw63Q2QHkMBK4PbhTxSgTC9m3Un3o https://mirror.hydra.gnu.org" -i emacs
 96 | </pre>
 97 | 
 98 | The attentive reader will have noticed one weakness of this approach: Everytime the root-directory of this repository changes the resulting URL will change as well. This would imply that users would have to constantly update the URL they use in the <code>substitute-url</code> parameter to <code>guix package</code>. To remedy situations like this ipfs has implemented a name system called ipns.
 99 | 
100 | ipfs allows to publish ipns names that are mutable. The name is the hash of a public part of a public/private key pair, and per default <code>ipfs name publish</code> uses the ipfs node's key.
101 | 
102 | <pre>
103 | $ ipfs name publish QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1
104 | Published to QmaeGpMRsHmeVaQFRnwtuZYdSdVgbc3Y64aDFs1ya8Frnb: /ipfs/QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1
105 | </pre>
106 | 
107 | This basically completes the necessary components to provide a stable substitute-url for guix that can have changing contents. Checkout https://ipfs.io/ipns/QmPMJYhxbeaSYXzNLMRbvvJknpYcJG9DcG8h2kJJmukd9i which is the address that this repository's code running on a server produces. Note how the first part of the url changed from <code>ipfs</code> to <code>ipns</code> (note: like said above the ipns name is specific to the particular ipfs node publishing that name):
108 | 
109 | <code>
110 | guix package --substitute-url="http://localhost:8080/ipns/QmPMJYhxbeaSYXzNLMRbvvJknpYcJG9DcG8h2kJJmukd9i https://mirror.hydra.gnu.org" -i emacs
111 | </code>
112 | 
113 | # Fetching substitutes from berlin.guixsd.org and publishing them via ipfs
114 | 
115 | The code in this repository contains a bunch of shell scripts (with no error handling whatsoever - so beware, it's just a proof of concept) that perform these steps:
116 | 
117 | * Query http://berlin.guixsd.org for newly finished builds
118 | * Fetch narinfos into <code>cache/</code>
119 | * Fetch the respective nar into <code>cache/nar/gzip</code>
120 | * build up directory structure that <code>guix package</code> expects
121 | * publish them all to ipfs
122 | * update the ipns name to point to the latest version
123 | 
124 | with binaries retrieved from berlin.guixsd.org's API that informs about finished builds.
125 | 
126 | # An optimization for size, speed and profit
127 | 
128 | Once a directory reaches a certain size, adding it recursively with <code>ipfs add</code> becomes slow as a dog (disk IO alone being a bottleneck). Also just keeping all files on disk is using up precious disk space while the promise of ipfs is that the swarm can take over hosting "responsibility". Luckily there is a way to add entries to a directory object directly without having to rerun <code>ipfs add -r</code> over and over again.
129 | 
130 | Here's an example. Let's add a directory structure under ipfs:
131 | 
132 | <pre>
133 | ipfs add -r test/
134 | added QmeeLUVdiSTTKQqhWqsffYDtNvvvcTfJdotkNyi1KDEJtQ test/foo
135 | added QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1 test
136 |  14 B / 14 B [=================================================================================================] 100.00%
137 | </pre>
138 | 
139 | and publish it under an ipns name:
140 | 
141 | <pre>
142 | $ ipfs name publish QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1
143 | Published to QmaeGpMRsHmeVaQFRnwtuZYdSdVgbc3Y64aDFs1ya8Frnb: /ipfs/QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1
144 | </pre>
145 | 
146 | If we now want to, at some later point, add to the directory published under this name we have two options:
147 | 
148 | * Remember the resulting CID of last directory state (in this case <code>QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1</code>)
149 | * Look it up first from its published name
150 | 
151 | The second option could be done like:
152 | 
153 | <pre>
154 | $ ipfs name resolve QmaeGpMRsHmeVaQFRnwtuZYdSdVgbc3Y64aDFs1ya8Frnb
155 | /ipfs/QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1
156 | </pre>
157 | 
158 | This gives us back the CID of the previous state of the directory. Now to add onto that we can perform these steps for each newly added file:
159 | 
160 | Add the file to ipfs:
161 | 
162 | <pre>
163 | $ ipfs add bar 
164 | added QmTEzo7FYzUCd5aq7bGKoMLfsbbsebpfARRZd4Znejb25R bar
165 |  4 B / 4 B [===================================================================================================] 100.00%
166 | </pre>
167 | 
168 | ...modify the directory object from above:
169 | 
170 | <pre>
171 | $ ipfs object patch add-link /ipfs/QmT429U7M2Civz8qmkA6uZ5tvLJ9njQAvzsX6BvWodJ3i1 bar QmTEzo7FYzUCd5aq7bGKoMLfsbbsebpfARRZd4Znejb25R
172 | QmcD4o2EK1qS8H4dR3usWmiZqKCGwSugUgkRaXgtcqEEaF
173 | </pre>
174 | 
175 | resulting in a new CID for the modified directory:
176 | 
177 | <pre>
178 | $ ipfs ls QmcD4o2EK1qS8H4dR3usWmiZqKCGwSugUgkRaXgtcqEEaF
179 | QmTEzo7FYzUCd5aq7bGKoMLfsbbsebpfARRZd4Znejb25R 12 bar
180 | QmeeLUVdiSTTKQqhWqsffYDtNvvvcTfJdotkNyi1KDEJtQ 22 foo
181 | </pre>
182 | 
183 | which we can then republish:
184 | 
185 | <pre>
186 | $ ipfs name publish QmcD4o2EK1qS8H4dR3usWmiZqKCGwSugUgkRaXgtcqEEaF
187 | Published to QmaeGpMRsHmeVaQFRnwtuZYdSdVgbc3Y64aDFs1ya8Frnb: /ipfs/QmcD4o2EK1qS8H4dR3usWmiZqKCGwSugUgkRaXgtcqEEaF
188 | </pre>
189 | 
190 | And voila:
191 | 
192 | <pre>
193 | $ ipfs ls /ipns/QmaeGpMRsHmeVaQFRnwtuZYdSdVgbc3Y64aDFs1ya8Frnb
194 | QmTEzo7FYzUCd5aq7bGKoMLfsbbsebpfARRZd4Znejb25R 12 bar
195 | QmeeLUVdiSTTKQqhWqsffYDtNvvvcTfJdotkNyi1KDEJtQ 22 foo
196 | </pre>
197 | 
198 | The implementation in the provided scripts is left as excercise to the interested reader for now :)
199 | 
200 | 


--------------------------------------------------------------------------------
/cache/nix-cache-info:
--------------------------------------------------------------------------------
1 | StoreDir: /gnu/store
2 | WantMassQuery: 0
3 | Priority: 100
4 | 


--------------------------------------------------------------------------------
/get_build_hashes.sh:
--------------------------------------------------------------------------------
1 | curl https://berlin.guixsd.org/api/latestbuilds?nr=30 | jq .[].buildoutputs.out.path | cut -d / -f 4 | cut -d "-" -f 1
2 | 


--------------------------------------------------------------------------------
/get_build_outputs.sh:
--------------------------------------------------------------------------------
1 | curl https://berlin.guixsd.org/api/latestbuilds?nr=10 | jq .[].buildoutputs.out.path 
2 | 


--------------------------------------------------------------------------------
/get_builds.sh:
--------------------------------------------------------------------------------
1 | curl https://berlin.guixsd.org/api/latestbuilds?nr=10 | jq .
2 | 


--------------------------------------------------------------------------------
/run.sh:
--------------------------------------------------------------------------------
1 | while true; do
2 |     sleep 300
3 |     bash update.sh
4 | done
5 | 


--------------------------------------------------------------------------------
/update.sh:
--------------------------------------------------------------------------------
 1 | echo "#### CACHE UPDATE STARTED"
 2 | mkdir -p cache/nar/gzip
 3 | 
 4 | echo "#### FINISHED JOBS"
 5 | bash get_build_hashes.sh > current_hashes.txt; 
 6 | for n in `cat current_hashes.txt`; do 
 7 |     narinfo="$n.narinfo"
 8 |     if [ ! -f "./cache/$narinfo" ]; then 
 9 |         echo get: "$narinfo";
10 |         wget https://berlin.guixsd.org/"$n".narinfo -P ./cache/
11 |         sleep 1
12 |     fi 
13 | done
14 | 
15 | echo "#### GETTING NARINFOS"
16 | rm -f current_nars.txt
17 | for n in cache/*.narinfo; do 
18 |     cat "$n" | grep ^URL | cut -d / -f 3 >> current_nars.txt
19 | done
20 | 
21 | 
22 | echo "#### GETTING NARS"
23 | for n in `cat current_nars.txt`; do
24 |     if [ ! -f "./cache/nar/gzip/$n" ]; then
25 |         echo get: "$n"
26 |         wget https://berlin.guixsd.org/nar/gzip/"$n" -P ./cache/nar/gzip/
27 |     fi
28 | done
29 | 
30 | echo "#### UPDATING IPFS"
31 | ipfs add -r cache > ipfs_log.txt
32 | 
33 | echo "#### UPDATING IPNS"
34 | cache_cid=`tail -n 1 ipfs_log.txt | cut -d ' ' -f 2`
35 | ipfs name publish "$cache_cid"
36 | echo "#### DONE."
37 | 


--------------------------------------------------------------------------------