├── .github └── CODE_OF_CONDUCT.md ├── .travis.yml ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── all-fetchers.nix ├── fetch-git ├── README.md ├── build.nix ├── run.nix ├── src │ ├── .gitignore │ ├── fetchgit │ │ ├── __init__.py │ │ ├── archives.py │ │ ├── arguments.py │ │ ├── command.py │ │ ├── fetchgit.py │ │ ├── lock.py │ │ ├── log.py │ │ ├── nixerrors.py │ │ ├── paths.py │ │ └── repository.py │ ├── setup.cfg │ └── setup.py └── tests │ ├── annotated-tag.nix │ ├── default.nix │ ├── fetch-branch.nix │ ├── fetch-middle-commit.nix │ ├── fetch-specify-name.nix │ ├── fetch-tag.nix │ ├── filter-source.nix │ └── lib.nix ├── fetch-pypi-hash ├── README.md ├── build.nix ├── fetch-pypi-hash └── run.nix ├── make-extra-builtins.nix └── test ├── default.nix └── run /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to make participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, gender identity and expression, level of experience, 9 | nationality, personal appearance, race, religion, or sexual identity and 10 | orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at 59 | [TTS-OpenSource-Office@target.com](mailto:TTS-OpenSource-Office@target.com). All 60 | complaints will be reviewed and investigated and will result in a response that 61 | is deemed necessary and appropriate to the circumstances. The project team is 62 | obligated to maintain confidentiality with regard to the reporter of an incident. 63 | Further details of specific enforcement policies may be posted separately. 64 | 65 | Project maintainers who do not follow or enforce the Code of Conduct in good 66 | faith may face temporary or permanent repercussions as determined by other 67 | members of the project's leadership. 68 | 69 | ## Attribution 70 | 71 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 72 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 73 | 74 | [homepage]: https://www.contributor-covenant.org 75 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: nix 2 | sudo: false 3 | 4 | script: 5 | - export XDG_CACHE_HOME=$HOME/.cache; cd ./test; ./run 6 | - export XDG_CACHE_HOME=$HOME/.cache; cd ./test; nix-shell -p python2Packages.requests --run ./run 7 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to reuse 2 | 3 | ### Issues 4 | 5 | Feel free to submit bugs or feature requests as issues. 6 | 7 | ### Pull Requests 8 | 9 | These rules must be followed for any contributions to be merged into master. 10 | 11 | 1. Fork this repo 12 | 2. Make any desired changes 13 | 3. Validate that your changes meet your desired use case 14 | 4. Ensure documentation has been updated 15 | 5. Open a pull request 16 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018 Target Brands, Inc. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | nix-fetchers 2 | ============== 3 | 4 | A set of morally pure fetching builtins for [Nix]. 5 | 6 | [Nix]: https://nixos.org/nix 7 | 8 | Prerequisites 9 | -------------- 10 | 11 | While the fetchers provided here are *morally* pure (that is, they 12 | always yield the same output for a given input), they rely on 13 | operations (such as network access) that `Nix` by itself can't allow. 14 | As such, they all rely on the special `exec` builtin. In vanilla 15 | `Nix`, this builtin is only available when the 16 | `allow-unsafe-native-code-during-evaluation` configuration setting is 17 | `true`. This allows *any* `Nix` evaluation to run arbitrary code as 18 | the user, though, so the [nix-plugins] project includes the 19 | [extraBuiltins] plugin to make `exec` available only in specific 20 | user-controlled circumstances. See the [extraBuiltins] documentation 21 | for more details. 22 | 23 | [nix-plugins]: https://github.com/shlevy/nix-plugins 24 | [extraBuiltins]: https://github.com/shlevy/nix-plugins#extra-builtins 25 | 26 | Project structure 27 | ------------------ 28 | 29 | Due to their privileged nature and their expected use as being 30 | (lazily) referenced for every `Nix` evaluation (in the `extraBuiltins` 31 | model), the fetchers are organized a bit unusually for `Nix` projects. 32 | 33 | Each fetcher has a subdirectory in this repo. The subdirectory 34 | contains at least two `Nix` expressions: 35 | 36 | * `build.nix`: This is a "normal" `Nix` expression like you might find 37 | in [nixpkgs]. It is callable via the normal `callPackage` mechanism 38 | and returns a derivation containing the program that will actually 39 | be run by the fetcher. 40 | * `run.nix`: This expression takes two arguments. The first is the 41 | `exec` builtin itself, the second is a package or path that includes 42 | the program built by `build.nix`. The result of applying the 43 | function in `run.nix` is usually another function corresponding to 44 | the fetcher interface itself, documented in the fetcher's 45 | `README.md`. 46 | 47 | Note that the second argument to `run.nix` may just be the result of 48 | `callPackage build.nix {}`, or it may for example be a `buildEnv` 49 | containing at least that result. 50 | 51 | This design allows the user to opt-in only to the plugins they need 52 | and avoid having their `extra-builtins.nix` (which is included in 53 | every evaluation) require a reference to or evaluation of a full 54 | `nixpkgs` set. See [make-extra-builtins] for a recommended workflow 55 | for this. 56 | 57 | [nixpkgs]: https://nixos.org/nixpkgs 58 | [make-extra-builtins]: #make-extra-builtins 59 | 60 | make-extra-builtins 61 | --------------------- 62 | 63 | This repo contains [make-extra-builtins.nix], defining a function to 64 | generate `extra-builtins.nix` from a set of fetchers. After being 65 | imported with `callPackage`, `make-extra-builtins` takes a set of 66 | directories with the same structure as the fetchers in this project 67 | and returns a derivation yielding `extra-builtins.nix` in a directory 68 | suitable for including in a top-level `extra-builtins.nix` or serving 69 | as it directly. For example, if you only wanted fetchers 70 | `fetch-monotone` and `fetch-bitkeeper`, you could do something like: 71 | 72 | ```nix 73 | let 74 | pkgs = import {}; 75 | make-extra-builtins = 76 | pkgs.callPackage {}; 77 | in make-extra-builtins { 78 | fetchers.fetch-monotone = ; 79 | fetchers.fetch-bitkeeper = ; 80 | } 81 | ``` 82 | 83 | Which will generate an `extra-builtins.nix` like: 84 | 85 | ```nix 86 | { exec, ... }: let 87 | fetcher-programs = builtins.storePath /nix/store/g13qx40xakg4zaz1bprx0ix5j8mlidlj-fetcher-programs; 88 | in { 89 | fetch-monotone = import /nix/store/v6z4bc0mzzw1b9sqc2yhn2n11fbj6nqs-run.nix exec fetcher-programs; 90 | fetch-bitkeeper = import /nix/store/0hkg88a9z08jaipcp4rn6xm62f74i22n-run.nix exec fetcher-programs; 91 | } 92 | ``` 93 | 94 | which has no dependency on `nixpkgs`. 95 | 96 | There is a top-level `all-fetchers.nix` that can be imported and 97 | passed as `fetchers` to `make-extra-builtins` if you wish to simply 98 | include all of the fetchers defined here. 99 | 100 | [make-extra-builtins.nix]: ./make-extra-builtins.nix 101 | 102 | Testing 103 | -------- 104 | 105 | The test suite can be run with the [run] script in the [test] 106 | directory. Note that the test suite relies on network connectivity 107 | (naturally), so it cannot be run within a `Nix` build itself. 108 | 109 | [run]: ./test/run 110 | [test]: ./test 111 | -------------------------------------------------------------------------------- /all-fetchers.nix: -------------------------------------------------------------------------------- 1 | builtins.listToAttrs (map (name: { 2 | inherit name; 3 | value = ./. + "/${name}"; 4 | }) [ 5 | "fetch-pypi-hash" 6 | "fetch-git" 7 | ]) 8 | -------------------------------------------------------------------------------- /fetch-git/README.md: -------------------------------------------------------------------------------- 1 | # fetch-git 2 | 3 | Fetch a git repository by tag or by a branch and revision. 4 | 5 | The result can be composed with `builtins.filterSource`. 6 | 7 | **Note:** specifying a tag name is much faster than specifying a 8 | branch and revision, as Git is able to perform a much shallower fetch 9 | with the tag itself. 10 | 11 | ## Arguments for fetching a tag 12 | 13 | * `name`: Optional, the name of the source archive 14 | * `url`: Required, the git-compatible URL to a repository 15 | * `tag`: Required, the name of the tag to fetch 16 | 17 | ### Example: 18 | 19 | ```nix 20 | fetch-git { 21 | url = "https://github.com/nixos/nixpkgs.git"; 22 | tag = "18.03"; 23 | } 24 | ``` 25 | 26 | or: 27 | 28 | ```nix 29 | fetch-git rec { 30 | name = "nixpkgs-at-tag-${tag}"; 31 | url = "https://github.com/nixos/nixpkgs.git"; 32 | tag = "18.03"; 33 | } 34 | ``` 35 | 36 | ## Arguments for fetching a branch 37 | 38 | * `name`: Optional, the name of the source archive 39 | * `url`: Required, the git-compatible URL to a repository 40 | * `branch`: Required, the name of the branch which contains `revision` 41 | * `revision`: Required, the specific revision of `branch` to fetch 42 | 43 | ### Example 44 | 45 | ```nix 46 | fetch-git { 47 | url = "https://github.com/nixos/nixpkgs.git"; 48 | branch = "release-18.03"; 49 | revision = "8bf6df2b8e8fc9e2f012901af48214feff918d9d"; 50 | } 51 | ``` 52 | 53 | or: 54 | 55 | ```nix 56 | fetch-git rec { 57 | name = "nixpkgs-${branch}-at-${revision}"; 58 | url = "https://github.com/nixos/nixpkgs.git"; 59 | branch = "release-18.03"; 60 | revision = "8bf6df2b8e8fc9e2f012901af48214feff918d9d"; 61 | } 62 | ``` 63 | 64 | ## Return value 65 | 66 | ### Success: a derivation 67 | 68 | * `outPath`: a Path to an archive of the repository 69 | * `debug`: a list of strings, debug messages from the fetcher. 70 | 71 | **Note:** this can be used directly as a source, you don't need to 72 | explicitly reference the `outPath`: 73 | 74 | ```nix 75 | runCommand "copy-nixpkgs-readme" 76 | { 77 | buildInputs = [ git ]; 78 | src = fetch-git rec { 79 | name = "nixpkgs-${branch}-at-${revision}"; 80 | url = "https://github.com/nixos/nixpkgs.git"; 81 | branch = "release-18.03"; 82 | revision = "8bf6df2b8e8fc9e2f012901af48214feff918d9d"; 83 | }; 84 | } 85 | '' 86 | cp $src/README.md $out 87 | ''; 88 | ``` 89 | 90 | ### Failure 91 | 92 | * `error`: possibly `no-cache-directory`, `commit-not-found`, 93 | `tag-not-found`. 94 | * `message`: Error details 95 | * `debug`: An array of strings, each line being a debug message from 96 | the fetcher 97 | 98 | ## Notes 99 | 100 | Successful results are cached in [XDG_CACHE_HOME], under 101 | `nix-fetchers`. 102 | 103 | ## DIFFERENCES FROM builtins.fetchGit 104 | 105 | ### Morally pure, assuming good Git hygiene and tags are immutable 106 | 107 | Specifying a git commit and branch is effectively immutable and 108 | pure 109 | 110 | Specifying a tag name, in a team where you can trust all actors 111 | to behave responsibly, is also immutable and pure. Note: this 112 | can probably only be feasibly true in an organization with strict 113 | repository protection! 114 | 115 | This purity allows for reliable caching. The upstream fetchGit, by 116 | contrast, allows fetching the tip of any branch (or arbitrary ref) and 117 | has a convenience cache with a configurable TTL unless a specific 118 | commit is specified. If you always specify a revision, this particular 119 | advantage may not matter much to you. 120 | 121 | ### Uses separate Git repositories per upstream Git repository 122 | 123 | Sacrifices some caching possibilities for much faster initial 124 | syncs. Upstream's fetch-git uses a single Git repository for all 125 | cloned repos. This works great if all you're fetching is Nixpkgs, 126 | but is extremely slow for initial syncs of other, non-nixpkgs 127 | repositories. 128 | 129 | 130 | [XDG_CACHE_HOME]: https://specifications.freedesktop.org/basedir-spec/0.7/ar01s03.html 131 | -------------------------------------------------------------------------------- /fetch-git/build.nix: -------------------------------------------------------------------------------- 1 | { bash, git, gnutar, python3, python3Packages, makeWrapper }: 2 | 3 | python3Packages.buildPythonPackage { 4 | pname = "target-nix-fetchers-fetch-git"; 5 | version = "1.0.0"; 6 | 7 | src = ./src; 8 | 9 | postFixup = '' 10 | wrapProgram $out/bin/fetch-git \ 11 | --set PATH "${git}/bin:${gnutar}/bin:${bash}/bin" \ 12 | --unset PYTHONPATH 13 | ''; 14 | } 15 | -------------------------------------------------------------------------------- /fetch-git/run.nix: -------------------------------------------------------------------------------- 1 | exec: fetch-git: 2 | { name ? "source", url, tag ? "", branch ? "", revision ? "" } @ args: 3 | # Effectively an enum: 4 | # 5 | # FetchGit: 6 | # - Tag(tag) 7 | # - Rev(branch, revision) 8 | 9 | let 10 | usageUnless = msg: cond: if cond then true else builtins.trace '' 11 | fetch-git usage error: 12 | 13 | ${msg} 14 | 15 | Fetch a tag: 16 | 17 | fetch-git { tag = "my-tag-name"; } 18 | 19 | Fetch a revision: 20 | 21 | fetch-git { branch = "my-branch-name"; revision = "my-revision"; } 22 | 23 | Received: 24 | 25 | fetch-git ${builtins.toJSON args} 26 | 27 | Need help? Reach out at https://github.com/target/nix-fetchers. 28 | 29 | '' false; 30 | 31 | in 32 | assert ((usageUnless "If the tag is specified, the branch and revision must be empty" 33 | ((tag != "") -> (branch == "" && revision == "")))); 34 | 35 | assert (usageUnless "If the branch is specified, revision must be, and tag must not be" 36 | ((branch != "") -> (revision != "" && tag == ""))); 37 | 38 | assert (usageUnless "If the revision is specified, branch must be, and tag must not be" 39 | ((revision != "") -> (branch != "" && tag == ""))); 40 | 41 | if tag != "" then 42 | exec [ 43 | "${fetch-git}/bin/fetch-git" name url "--tag" tag 44 | ] 45 | else 46 | exec [ 47 | "${fetch-git}/bin/fetch-git" name url "--branch" branch revision 48 | ] 49 | -------------------------------------------------------------------------------- /fetch-git/src/.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | *.pyc 3 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/target/nix-fetchers/b7cc311fd5ea4e56f05e31be3d14e0712831f5d5/fetch-git/src/fetchgit/__init__.py -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/archives.py: -------------------------------------------------------------------------------- 1 | import os 2 | import subprocess 3 | from fetchgit import log 4 | from fetchgit.paths import sanitize_path_segment 5 | from fetchgit.command import cmd 6 | from fetchgit.nixerrors import NixError 7 | 8 | 9 | def setup_bare(path): 10 | if not os.path.isdir(path): 11 | log.debug("About to init a bare repository at {}".format(path)) 12 | cmd(None, ["git", "init", "--bare", path]) 13 | 14 | 15 | def archive_ref(source, ref, destination): 16 | try: 17 | os.makedirs(destination) 18 | except FileExistsError: 19 | pass 20 | 21 | cmd(source, 22 | [ 23 | "bash", 24 | # Bash will run the following safe bash, and accept the defs 25 | # of $1 and $2 via the following -s 26 | "-c", "git archive --format=tar \"$1\" | tar -x -C \"$2\"", 27 | 28 | # Passing ref and destination this way is much safer than 29 | # using interpolation 30 | "-s", ref, destination 31 | ]) 32 | 33 | 34 | class BranchArchive: 35 | cache = None 36 | branch = None 37 | revision = None 38 | name = None 39 | 40 | def __init__(self, cache, branch, revision, name): 41 | self.cache = cache 42 | self.branch = branch 43 | self.revision = revision 44 | self.name = name 45 | 46 | def make_archive(self): 47 | if not self.__exists(): 48 | log.warning("Fetching branch '{}' at rev '{}' from {}".format( 49 | self.branch, 50 | self.revision, 51 | self.cache.repository 52 | )) 53 | with self.cache.lock(): 54 | bare_path = self.__bare_path() 55 | 56 | if not os.path.isdir(bare_path): 57 | setup_bare(bare_path) 58 | 59 | self.__fetch_depth(1) 60 | 61 | if not self.__commit_exists() and not self.__is_massive(): 62 | # So 1000 is just a "good start", subsequent fetches are 63 | # pretty expensive. Thus, my hope is that we can get a 64 | # nice fetch chunk here and maximize the odds we receive 65 | # the revision we need, without doing a fancy backoff 66 | # which is very slow and re-downloads quite a lot of data. 67 | self.__deepen(1000) 68 | 69 | if not self.__commit_exists(): 70 | self.__fetch_all() 71 | 72 | if not self.__commit_exists(): 73 | raise NixError( 74 | "commit-not-found", 75 | "Commit {} not found after fetching {}.".format( 76 | self.revision, 77 | self.__ref() 78 | ) 79 | ) 80 | 81 | archive_ref(bare_path, self.revision, self.__archive_path()) 82 | 83 | return self.__archive_path() 84 | 85 | def __is_massive(self): 86 | return "nixpkgs" in self.cache.repository.lower() 87 | 88 | def __ref(self): 89 | return "refs/heads/{}".format(self.branch) 90 | 91 | def __commit_exists(self): 92 | try: 93 | cmd(self.__bare_path(), ["git", "cat-file", "-e", 94 | "{}^{{commit}}".format(self.revision)]) 95 | return True 96 | except subprocess.CalledProcessError as e: 97 | log.debug( 98 | "Did not find commit {} and instead got output: {}".format( 99 | self.revision, e.output 100 | ) 101 | ) 102 | return False 103 | 104 | def __fetch_depth(self, depth): 105 | cmd(self.__bare_path(), ["git", 106 | "fetch", 107 | "--depth", 108 | str(depth), 109 | self.cache.repository, 110 | self.__ref()]) 111 | 112 | def __deepen(self, depth): 113 | cmd(self.__bare_path(), ["git", 114 | "fetch", 115 | "--deepen", 116 | str(depth), 117 | self.cache.repository, 118 | self.__ref()]) 119 | 120 | def __fetch_all(self): 121 | try: 122 | cmd(self.__bare_path(), ["git", 123 | "fetch", 124 | "--unshallow", 125 | self.cache.repository, 126 | self.__ref()]) 127 | except subprocess.CalledProcessError: 128 | # We may already have unshallowed this repository 129 | pass 130 | 131 | def __bare_path(self): 132 | return os.path.join( 133 | self.cache.root_directory(), 134 | 'bare' 135 | ) 136 | 137 | def __archive_path(self): 138 | return os.path.join( 139 | self.cache.root_directory(), 140 | 'branch-archives', 141 | sanitize_path_segment(self.branch), 142 | sanitize_path_segment(self.revision), 143 | sanitize_path_segment(self.name) 144 | ) 145 | 146 | def __exists(self): 147 | return os.path.isdir(self.__archive_path()) 148 | 149 | 150 | class TagArchive: 151 | cache = None 152 | tag = None 153 | name = None 154 | 155 | def __init__(self, cache, tag, name): 156 | self.cache = cache 157 | self.tag = tag 158 | self.name = name 159 | 160 | def make_archive(self): 161 | if not self.__exists(): 162 | log.warning("Fetching tag '{}' from {}".format( 163 | self.tag, 164 | self.cache.repository 165 | )) 166 | with self.cache.lock(): 167 | bare_path = self.__bare_path() 168 | ref = "refs/tags/{}".format(self.tag) 169 | setup_bare(bare_path) 170 | 171 | try: 172 | cmd(bare_path, ["git", 173 | "fetch", 174 | "--depth=1", 175 | self.cache.repository, 176 | ref]) 177 | except subprocess.CalledProcessError as e: 178 | log.debug( 179 | "Did not find tag {}, instead got output: {}".format( 180 | self.tag, e.output 181 | ) 182 | ) 183 | 184 | raise NixError( 185 | "tag-not-found", 186 | "Tag {} not found".format(self.tag) 187 | ) 188 | 189 | archive_ref(bare_path, "FETCH_HEAD", self.__archive_path()) 190 | 191 | return self.__archive_path() 192 | 193 | def __bare_path(self): 194 | return os.path.join( 195 | self.cache.root_directory(), 196 | 'bare' 197 | ) 198 | 199 | def __archive_path(self): 200 | return os.path.join( 201 | self.cache.root_directory(), 202 | 'tag-archives', 203 | sanitize_path_segment(self.tag), 204 | sanitize_path_segment(self.name) 205 | ) 206 | 207 | def __exists(self): 208 | return os.path.isdir(self.__archive_path()) 209 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/arguments.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | 4 | class BranchRevisionAction(argparse.Action): 5 | def __init__(self, nargs=None, **kwargs): 6 | if nargs != 2: 7 | raise ValueError("nargs must be 2") 8 | 9 | super(BranchRevisionAction, self).__init__(nargs=nargs, **kwargs) 10 | 11 | def __call__(self, parser, namespace, values, option_string=None): 12 | branch = values[0] 13 | revision = values[1] 14 | 15 | self.validate_revision(revision) 16 | 17 | setattr(namespace, "branch", branch) 18 | setattr(namespace, "revision", revision) 19 | 20 | def validate_revision(self, rev): 21 | if len(rev) != 40: 22 | raise argparse.ArgumentError( 23 | self, 24 | "{} is not a valid Git revision: it must be 40 characters long".format(rev) # noqa: E501 25 | ) 26 | 27 | try: 28 | int(rev, 16) 29 | except ValueError as e: 30 | raise argparse.ArgumentError( 31 | self, 32 | "{} is not a valid Git revision: it isn't hexadecimal".format(rev) # noqa: E501 33 | ) 34 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/command.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | from fetchgit import log 3 | 4 | 5 | def cmd(cwd, command): 6 | log.debug("Running {} in {}".format( 7 | " ".join(command), 8 | cwd 9 | )) 10 | 11 | result = subprocess.run(command, 12 | cwd=cwd, 13 | stdout=subprocess.PIPE, 14 | universal_newlines=True, 15 | check=True) 16 | log.debug("command output: {}".format(result.stdout)) 17 | return result 18 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/fetchgit.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import json 4 | import argparse 5 | 6 | from fetchgit import log 7 | from fetchgit.archives import TagArchive, BranchArchive 8 | from fetchgit.arguments import BranchRevisionAction 9 | from fetchgit.repository import RepositoryCache 10 | from fetchgit.nixerrors import NixError 11 | 12 | 13 | def dont_break_out_of_double_single_quotes(msg): 14 | return msg.replace("''", '""') # oh my word 15 | 16 | 17 | def print_success(outPath): 18 | nixPattern = """ 19 | (let 20 | data = builtins.fromJSON ''{}''; 21 | in data // {{ 22 | outPath = /. + data.outPath; 23 | }}) 24 | """ 25 | print(nixPattern.format( 26 | json.dumps({ 27 | "outPath": outPath, 28 | "debug": [dont_break_out_of_double_single_quotes(msg) 29 | for msg in log.debug_msgs] 30 | }) 31 | )) 32 | 33 | 34 | parser = argparse.ArgumentParser( 35 | description="an improved evaluation-time Git fetcher for Nix", 36 | formatter_class=argparse.RawDescriptionHelpFormatter, 37 | epilog=""" 38 | 39 | ENVIRONMENT VARIABLES 40 | 41 | fetch-git caches repositories in: 42 | 43 | if XDG_CACHE_HOME is set: 44 | 45 | XDG_CACHE_HOME/nix-fetchers/fetch-git 46 | 47 | Failing that, if HOME is set, it will store it in: 48 | 49 | HOME/.cache/nix-fetchers/fetch-git 50 | 51 | Failing that, it will raise an error. 52 | 53 | REPORTING BUGS 54 | 55 | https://github.com/target/nix-fetchers 56 | 57 | AUTHORS 58 | 59 | Original work by Shea Levy 60 | Subsequent work by Graham Christensen 61 | 62 | """ 63 | ) 64 | 65 | parser.add_argument('name', help="Name of the archive") 66 | parser.add_argument('repo', help="git-compatible URL to the repository") 67 | group = parser.add_mutually_exclusive_group(required=True) 68 | group.add_argument("--tag", nargs=1, help="Name of the tag to archive") 69 | group.add_argument("--branch", nargs=2, metavar=("branch", "revision"), 70 | action=BranchRevisionAction, 71 | help="Name and revision to archive") 72 | 73 | def main(): 74 | args = parser.parse_args() 75 | 76 | try: 77 | if args.tag is not None: 78 | c = RepositoryCache(args.repo) 79 | archive = TagArchive( 80 | c, 81 | args.tag[0], 82 | args.name 83 | ) 84 | print_success(archive.make_archive()) 85 | elif args.branch is not None: 86 | branch = args.branch 87 | commit = args.revision 88 | c = RepositoryCache(args.repo) 89 | archive = BranchArchive( 90 | c, 91 | branch, 92 | commit, 93 | args.name 94 | ) 95 | print_success(archive.make_archive()) 96 | else: 97 | parser.print_help() 98 | exit(2) 99 | except NixError as e: 100 | error_fmt = """ 101 | (let 102 | x = (builtins.fromJSON ''{}''); 103 | in builtins.trace x x) 104 | """ 105 | print(error_fmt.format( 106 | json.dumps({ 107 | "error": e.failure_type, 108 | "message": e.message, 109 | "debug": [dont_break_out_of_double_single_quotes(msg) 110 | for msg in log.debug_msgs] 111 | }) 112 | )) 113 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/lock.py: -------------------------------------------------------------------------------- 1 | import os 2 | from fetchgit import log 3 | import fcntl 4 | import errno 5 | 6 | 7 | class FileLock: 8 | fd = None 9 | 10 | def __init__(self, root): 11 | self.root = root 12 | self.lockfile = os.path.join(self.root, "lock") 13 | 14 | def __enter__(self): 15 | log.debug("About to lock {}".format(self.lockfile)) 16 | 17 | if not self._acquire(block=False): 18 | log.warning("Retrying lock acquisition on {}.".format( 19 | self.lockfile 20 | )) 21 | self._acquire(block=True) 22 | 23 | log.debug("Received lock on {}".format(self.lockfile)) 24 | 25 | def __exit__(self, *args): 26 | log.debug("Releasing lock {}".format(self.lockfile)) 27 | self._release() 28 | 29 | def _acquire(self, block): 30 | try: 31 | self.fd = os.open(self.lockfile, os.O_CREAT) 32 | if block: 33 | fcntl.flock(self.fd, fcntl.LOCK_EX) 34 | else: 35 | fcntl.flock(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB) 36 | return True 37 | except (IOError, OSError) as e: 38 | if e.errno != errno.EAGAIN: 39 | raise 40 | return False 41 | 42 | def _release(self): 43 | if self.fd is not None: 44 | try: 45 | os.unlink(self.lockfile) 46 | except: # noqa: E722 47 | # Swallow any/all exceptions because it is immaterial 48 | # if we delete the file or not. 49 | pass 50 | 51 | fcntl.flock(self.fd, fcntl.LOCK_UN) 52 | os.close(self.fd) 53 | self.fd = None 54 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/log.py: -------------------------------------------------------------------------------- 1 | 2 | import sys 3 | 4 | debug_msgs = [] 5 | 6 | 7 | def debug(msg): 8 | debug_msgs.append(msg) 9 | 10 | 11 | def warning(msg): 12 | print(f"fetch-git: {msg}", file=sys.stderr) 13 | debug(msg) 14 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/nixerrors.py: -------------------------------------------------------------------------------- 1 | class NixError(BaseException): 2 | def __init__(self, failure_type, message): 3 | self.failure_type = failure_type 4 | self.message = message 5 | 6 | super(BaseException, self).__init__() 7 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/paths.py: -------------------------------------------------------------------------------- 1 | 2 | 3 | class ScaryPathSegmentException(Exception): 4 | pass 5 | 6 | 7 | def sanitize_path_segment(segment): 8 | # Path names in Linux can contain anything except a nul byte and / 9 | # Note this causes collisions on cases like 10 | # 11 | # git@myhost.com/foo/bar 12 | # git@myhost.com/foo_bar 13 | # 14 | # but honestly. 15 | # 16 | # An alternative would be to base64 or sha256sum the input but 17 | # that seems a bit unfriendly by default. 18 | # 19 | # Python strings can't contain a nul byte, so we don't have to 20 | # guard against it. 21 | 22 | sanitized = segment.replace("/", "_") 23 | 24 | if sanitized == "..": 25 | raise ScaryPathSegmentException("Repository cannot be named '..'") 26 | elif sanitized == ".": 27 | raise ScaryPathSegmentException("Repository cannot be named '.'") 28 | elif sanitized == "": 29 | raise ScaryPathSegmentException("Repository name cannot be an empty" 30 | " string") 31 | else: 32 | return sanitized 33 | -------------------------------------------------------------------------------- /fetch-git/src/fetchgit/repository.py: -------------------------------------------------------------------------------- 1 | import os 2 | from fetchgit.paths import sanitize_path_segment 3 | from fetchgit.lock import FileLock 4 | from fetchgit.nixerrors import NixError 5 | 6 | 7 | def cache_root(repository): 8 | xdg_cache_home = os.environ.get('XDG_CACHE_HOME') 9 | home = os.environ.get('HOME') 10 | 11 | cache_base = None 12 | if xdg_cache_home is not None: 13 | cache_base = xdg_cache_home 14 | elif home is not None: 15 | cache_base = os.path.join(home, '.cache') 16 | else: 17 | raise NixError("no-cache-directory", 18 | "Specify XDG_CACHE_HOME or HOME env variables") 19 | 20 | return os.path.join(cache_base, 21 | 'nix-fetchers/fetch-git', 22 | sanitize_path_segment(repository)) 23 | 24 | 25 | class RepositoryCache: 26 | repository = None 27 | 28 | def __init__(self, repository): 29 | self.repository = repository 30 | 31 | def root_directory(self): 32 | root = cache_root(self.repository) 33 | os.makedirs(root, exist_ok=True) 34 | return root 35 | 36 | def lock(self): 37 | return FileLock(self.root_directory()) 38 | -------------------------------------------------------------------------------- /fetch-git/src/setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = target-nix-fetchers-fetch-git 3 | version = 1.0.0 4 | author = Graham Christensen 5 | author_email = graham.christensen@target.com 6 | url = https://github.com/target/nix-fetchers 7 | description = Fetch git repositories quickly. 8 | keywords = nix, git 9 | classifiers = 10 | License :: OSI Approved :: MIT License 11 | 12 | [options] 13 | packages = fetchgit 14 | py_modules = fetchgit 15 | 16 | [options.entry_points] 17 | console_scripts = 18 | fetch-git = fetchgit.fetchgit:main -------------------------------------------------------------------------------- /fetch-git/src/setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup() 4 | -------------------------------------------------------------------------------- /fetch-git/tests/annotated-tag.nix: -------------------------------------------------------------------------------- 1 | { runCommand, testlib, all-fetchers, git }: 2 | runCommand "test-fetch-tag" { 3 | buildInputs = [ git ]; 4 | src = all-fetchers.fetch-git { 5 | url = testlib.makeGitRepo { 6 | commits = 3; 7 | finalizingCommand = '' 8 | git tag -a v1.4 -m "my version 1.4" 9 | echo 4 > contents 10 | git commit -am "commit #4" 11 | ''; 12 | }; 13 | tag = "v1.4"; 14 | }; 15 | } 16 | '' 17 | cd $src 18 | if [ "$(cat contents)" = "3" ]; then 19 | echo success | tee $out 20 | else 21 | echo "3" | diff contents - 22 | fi 23 | 24 | '' 25 | -------------------------------------------------------------------------------- /fetch-git/tests/default.nix: -------------------------------------------------------------------------------- 1 | { pkgs, all-fetchers, seed ? "stable-seed" }: 2 | let 3 | testlib = pkgs.callPackage ./lib.nix { inherit seed; }; 4 | 5 | callTest = file: let 6 | # built = 7 | in { 8 | value = let 9 | attempt = builtins.tryEval (builtins.readFile (pkgs.callPackage file { inherit testlib all-fetchers; })); 10 | in if attempt.success 11 | then attempt.value 12 | else "failure"; 13 | expected = "success\n"; 14 | }; 15 | in { 16 | test_fetch_tag = callTest ./fetch-tag.nix; 17 | test_annotated_tag = callTest ./annotated-tag.nix; 18 | test_fetch_branch = callTest ./fetch-branch.nix; 19 | test_fetch_middle_commit = callTest ./fetch-middle-commit.nix; 20 | test_filter_source = callTest ./filter-source.nix; 21 | test_fetch_specify_name = callTest ./fetch-specify-name.nix; 22 | } 23 | -------------------------------------------------------------------------------- /fetch-git/tests/fetch-branch.nix: -------------------------------------------------------------------------------- 1 | { runCommand, testlib, all-fetchers, git }: 2 | runCommand "test-fetch-branch" { 3 | buildInputs = [ git ]; 4 | src = all-fetchers.fetch-git rec { 5 | url = testlib.makeGitRepo { 6 | commits = 5; 7 | }; 8 | branch = "master"; 9 | revision = testlib.latestCommit url; 10 | }; 11 | } 12 | '' 13 | cd $src 14 | if [ "$(cat contents)" = "5" ]; then 15 | echo success | tee $out 16 | else 17 | echo "5" | diff contents - 18 | fi 19 | 20 | '' 21 | -------------------------------------------------------------------------------- /fetch-git/tests/fetch-middle-commit.nix: -------------------------------------------------------------------------------- 1 | { runCommand, testlib, all-fetchers, git }: 2 | runCommand "test-fetch-midle-commit" { 3 | buildInputs = [ git ]; 4 | src = all-fetchers.fetch-git rec { 5 | url = testlib.makeGitRepo { 6 | commits = 8; 7 | }; 8 | branch = "master"; 9 | revision = builtins.elemAt (testlib.listAllCommits url) 4; 10 | }; 11 | } 12 | '' 13 | cd $src 14 | if [ "$(cat contents)" = "4" ]; then 15 | echo success | tee $out 16 | else 17 | echo "4" | diff contents - 18 | fi 19 | 20 | '' 21 | -------------------------------------------------------------------------------- /fetch-git/tests/fetch-specify-name.nix: -------------------------------------------------------------------------------- 1 | { runCommand, testlib, lib, all-fetchers, git }: 2 | runCommand "test-specify-name" { 3 | src = all-fetchers.fetch-git rec { 4 | url = testlib.makeGitRepo { 5 | commits = 1; 6 | }; 7 | name = "my-cool-source-name"; 8 | branch = "master"; 9 | revision = testlib.latestCommit url; 10 | }; 11 | } 12 | '' 13 | if echo "$src" | grep -q "my-cool-source-name"; then 14 | echo success > $out 15 | fi 16 | '' 17 | -------------------------------------------------------------------------------- /fetch-git/tests/fetch-tag.nix: -------------------------------------------------------------------------------- 1 | { runCommand, testlib, all-fetchers, git }: 2 | runCommand "test-fetch-tag" { 3 | buildInputs = [ git ]; 4 | src = all-fetchers.fetch-git { 5 | url = testlib.makeGitRepo { 6 | commits = 2; 7 | finalizingCommand = '' 8 | git tag v0.0.2 9 | ''; 10 | }; 11 | tag = "v0.0.2"; 12 | }; 13 | } 14 | '' 15 | cd $src 16 | if [ "$(cat contents)" = "2" ]; then 17 | echo success | tee $out 18 | else 19 | echo "2" | diff contents - 20 | fi 21 | 22 | '' 23 | -------------------------------------------------------------------------------- /fetch-git/tests/filter-source.nix: -------------------------------------------------------------------------------- 1 | { runCommand, testlib, lib, all-fetchers, git }: 2 | runCommand "test-filter-source" { 3 | src = builtins.filterSource 4 | (path: type: (builtins.baseNameOf path) == "keep") 5 | (all-fetchers.fetch-git rec { 6 | url = testlib.makeGitRepo { 7 | commits = 1; 8 | finalizingCommand = '' 9 | touch keep 10 | git add keep 11 | git commit -m "keep should be in the results when we filter it" 12 | ''; 13 | }; 14 | branch = "master"; 15 | revision = testlib.latestCommit url; 16 | }); 17 | } 18 | '' 19 | cd $src 20 | if [ -f ./contents ]; then 21 | echo "Found ./contents! failure!" 22 | exit 1 23 | fi 24 | 25 | if [ -f ./keep ]; then 26 | echo success > $out 27 | else 28 | echo "the file ./keep does not exist" 29 | fi 30 | '' 31 | -------------------------------------------------------------------------------- /fetch-git/tests/lib.nix: -------------------------------------------------------------------------------- 1 | { runCommand, git, jq, lib, seed }: 2 | rec { 3 | makeGitRepo = { commits ? 2, 4 | buildInputs ? [], 5 | finalizingCommand ? "" 6 | }: (runCommand "test-git-repo" { 7 | buildInputs = [ git ] ++ buildInputs; 8 | GIT_COMMITTER_NAME = "Bogus Committer"; 9 | GIT_COMMITTER_EMAIL = "bogus-committer@example.com"; 10 | GIT_AUTHOR_NAME = "Bogus Author"; 11 | GIT_AUTHOR_EMAIL = "bogus-author@example.com"; 12 | TEST_SEED = seed; 13 | } '' 14 | set -eu 15 | 16 | mkdir repo 17 | cd repo 18 | git init 19 | 20 | touch contents 21 | git add contents 22 | ${lib.concatMapStringsSep "\n" (x: '' 23 | echo "${toString x}" > contents 24 | git add contents 25 | git commit -m "Appending ${toString x} to contents" 26 | '') (lib.range 1 commits)} 27 | 28 | ${finalizingCommand} 29 | 30 | git init --bare $out 31 | git push --mirror $out 32 | ''); 33 | 34 | loadJSONFile = f: builtins.fromJSON (builtins.readFile f); 35 | 36 | # Fetch a list of all commits in the repository from the default 37 | # current HEAD, the first (lib.head) in the list will be the most 38 | # recent. The last (lib.last) will be the oldest. 39 | listAllCommits = repo: loadJSONFile (runCommand "list-all-commits" { 40 | buildInputs = [ git jq ]; 41 | } '' 42 | cd ${repo} 43 | git log --pretty=tformat:%H \ 44 | | jq --raw-input --slurp 'split("\n") | .[0:-1]' > $out 45 | ''); 46 | 47 | latestCommit = repo: lib.head (listAllCommits repo); 48 | } 49 | -------------------------------------------------------------------------------- /fetch-pypi-hash/README.md: -------------------------------------------------------------------------------- 1 | # fetch-pypi-hash 2 | 3 | Get the hash of the source tarball for a given PyPI package. 4 | 5 | ## Arguments 6 | 7 | * `name`: The name of the package. Must be in the same form as on 8 | PyPI, despite some tooling being more flexible with cases. For 9 | example, `pillow` is an error while `Pillow` is correct. 10 | * `version`: The version of the package. Must be in canonical form, as 11 | defined in [PEP 440][PEP-440]. 12 | * `filename`: The name of the source file to hash. Defaults to 13 | `${name}-${version}.tar.gz`. 14 | 15 | [PEP-440]: https://www.python.org/dev/peps/pep-0440/ 16 | 17 | ## Return value 18 | 19 | ### Success 20 | 21 | * `sha256`: The SHA-256 hash of the source tarball. 22 | 23 | ### Failure 24 | 25 | * `failure-type`: One of `"HTTP-status"`, `"JSON-parse"`, or 26 | `"missing-filename"`. 27 | * `code`: On an `HTTP-status` error, the non-200 HTTP status code. 28 | * `message`: On a `JSON-parse` error, the error message from the 29 | parser. 30 | 31 | Due to a quirk of the JSON parser in use, some errors due to valid 32 | `JSON` without the expected schema manifest as `null` hashes. 33 | 34 | Some errors currently manifest as an evaluation error. 35 | 36 | More failure types may be added in the future. 37 | 38 | ## Notes 39 | 40 | Successful results are cached in [XDG_CACHE_HOME], under 41 | `nix-fetchers`. 42 | 43 | A `3xx` `HTTP-status` error is a good indication of a non-canonical 44 | package `name`. A `404` `HTTP-status` error is a good indication of a 45 | missing package. 46 | 47 | [XDG_CACHE_HOME]: https://specifications.freedesktop.org/basedir-spec/0.7/ar01s03.html 48 | -------------------------------------------------------------------------------- /fetch-pypi-hash/build.nix: -------------------------------------------------------------------------------- 1 | { substituteAll, curl, jq, bash, coreutils, shellcheck }: 2 | 3 | substituteAll { 4 | src = ./fetch-pypi-hash; 5 | isExecutable = true; 6 | dir = "bin"; 7 | inherit curl jq bash coreutils; 8 | nativeBuildInputs = [ shellcheck ]; 9 | postInstall = '' 10 | shellcheck $out/bin/fetch-pypi-hash 11 | ''; 12 | } 13 | -------------------------------------------------------------------------------- /fetch-pypi-hash/fetch-pypi-hash: -------------------------------------------------------------------------------- 1 | #! @bash@/bin/bash 2 | 3 | set -euo pipefail 4 | 5 | export PATH=@curl@/bin:@jq@/bin:@coreutils@/bin 6 | 7 | declare tmpdir 8 | tmpdir="$(mktemp --directory --tmpdir fetch-pypi-hash.XXXXXXXXXX)" 9 | 10 | cleanup () { 11 | rm --force --recursive "$tmpdir" 12 | } 13 | 14 | trap cleanup EXIT 15 | 16 | validate_args() { 17 | if [ "$#" -ne 3 ]; then 18 | echo "USAGE: $0 NAME VERSION FILENAME" >&2 19 | exit 1 20 | fi 21 | local -r name="$1" 22 | local -r version="$2" 23 | local -r filename="$3" 24 | 25 | # Adapted from PEP-508 26 | # https://www.python.org/dev/peps/pep-0508/#names 27 | local name_regex 28 | # Either a single letter or digit 29 | name_regex+='^([A-Za-z0-9]|' 30 | # Or a sequence of letters, numbers, dots, underscores, and dashes 31 | # that starts and ends with a letter or digit 32 | name_regex+='[A-Za-z0-9][A-Za-z0-9._-]*[A-Za-z0-9])$' 33 | if ! [[ "$name" =~ $name_regex ]]; then 34 | echo "Invalid python package name $name" >&2 35 | exit 1 36 | fi 37 | 38 | # Adapted from PEP-440. 39 | # https://www.python.org/dev/peps/pep-0440/#appendix-b-parsing-version-strings-with-regular-expressions 40 | local version_regex 41 | # The version "epoch" 42 | version_regex+='^([1-9][0-9]*!)?' 43 | # The thing we normally think of as a version number 44 | version_regex+='(0|[1-9][0-9]*)(\.(0|[1-9][0-9]*))*' 45 | # Pre-release marking 46 | version_regex+='((a|b|rc)(0|[1-9][0-9]*))?' 47 | # Post release marking 48 | version_regex+='(\.post(0|[1-9][0-9]*))?' 49 | # Development release... Possibly shouldn't support these. 50 | version_regex+='(\.dev(0|[1-9][0-9]*))?$' 51 | if ! [[ "$version" =~ $version_regex ]]; then 52 | echo "Invalid python package version $version" >&2 53 | exit 1 54 | fi 55 | 56 | case "$filename" in 57 | . | .. | */*) 58 | echo "Invalid filename $filename" >&2 59 | exit 1 60 | ;; 61 | *) ;; 62 | esac 63 | } 64 | 65 | fetch_json () { 66 | local -r name="$1" 67 | local -r version="$2" 68 | 69 | code="$(curl --silent --show-error --output "$tmpdir/download" \ 70 | --write-out "%{http_code}" \ 71 | "https://pypi.org/pypi/$name/$version/json")" 72 | if [ "$code" -ne "$code" ] 2>/dev/null; then 73 | echo "Unexpected non-numeric HTTP status code from curl:" \ 74 | "$code" >&2 75 | exit 1 76 | fi 77 | 78 | if [ "$code" -ne 200 ]; then 79 | echo "{ failure-type = ''HTTP-status''; code = $code; }" 80 | exit 81 | fi 82 | } 83 | 84 | parse_json () { 85 | local -r name="$1" 86 | local -r version="$2" 87 | local -r filename="$3" 88 | local query 89 | # Find the URL... 90 | query+='.urls | ' 91 | # that is the filename we want 92 | # disable SC2016 because we *want* a literal $ here. 93 | # shellcheck disable=SC2016 94 | query+='map(select(.filename == $filename)) | ' 95 | # Fail if the filename doesn't exist 96 | query+='if length == 0 then false else ' 97 | # Otherwise there can only be one by pypi's name uniqueness 98 | # guarantees, get its hash. 99 | query+='.[0].digests.sha256 end' 100 | local hash 101 | hash=$(jq "$query" --arg filename "$filename" \ 102 | <"$tmpdir/download" 2>"$tmpdir/err" || echo "") 103 | if [ -z "$hash" ]; then 104 | local msg 105 | msg="''$(cat "$tmpdir/err")''" 106 | echo "{ failure-type = ''JSON-parse''; message = $msg; }" 107 | exit 108 | elif [ "$hash" = "false" ]; then 109 | echo "{ failure-type = ''missing-filename''; }" 110 | exit 111 | fi 112 | 113 | echo "{ sha256 = $hash; }" \ 114 | >"$tmpdir/default.nix" 115 | } 116 | 117 | main() { 118 | validate_args "$@" 119 | local -r name="$1" 120 | local -r version="$2" 121 | local -r filename="$3" 122 | 123 | local -r cache_base="${XDG_CACHE_HOME:-$HOME/.cache}/nix-fetchers" 124 | local -r cache_dir="$cache_base/fetchpypi/$name/$version/$filename" 125 | if [ ! -f "$cache_dir/default.nix" ]; then 126 | fetch_json "$name" "$version" 127 | 128 | parse_json "$name" "$version" "$filename" 129 | 130 | mkdir -p "$cache_dir" 131 | mv "$tmpdir/default.nix" "$cache_dir" 132 | fi 133 | cat "$cache_dir/default.nix" 134 | } 135 | 136 | main "$@" 137 | -------------------------------------------------------------------------------- /fetch-pypi-hash/run.nix: -------------------------------------------------------------------------------- 1 | exec: fetch-pypi-hash: 2 | { name, version, filename ? "${name}-${version}.tar.gz" }: exec [ 3 | "${fetch-pypi-hash}/bin/fetch-pypi-hash" name version filename 4 | ] 5 | -------------------------------------------------------------------------------- /make-extra-builtins.nix: -------------------------------------------------------------------------------- 1 | { callPackage, writeTextDir, buildEnv }: 2 | { fetchers }: let 3 | add-fetcher = { programs ? [], fetcher-snippets ? [] }: name: let 4 | dir = fetchers.${name}; 5 | in { 6 | programs = programs ++ [ (callPackage (dir + "/build.nix") {}) ]; 7 | fetcher-snippets = fetcher-snippets ++ [ 8 | "${name} = import ${dir + "/run.nix"} exec fetcher-programs;" 9 | ]; 10 | }; 11 | inherit (builtins.foldl' add-fetcher {} (builtins.attrNames fetchers)) 12 | programs fetcher-snippets; 13 | fetcher-programs = buildEnv { 14 | name = "fetcher-programs"; 15 | paths = programs; 16 | }; 17 | in writeTextDir "extra-builtins.nix" '' 18 | { exec, ... }: let 19 | fetcher-programs = builtins.storePath ${fetcher-programs}; 20 | in { 21 | ${builtins.concatStringsSep "\n " fetcher-snippets} 22 | } 23 | '' 24 | -------------------------------------------------------------------------------- /test/default.nix: -------------------------------------------------------------------------------- 1 | { seed ? "stable-seed" }: 2 | # Run with: ./run 3 | let 4 | comparison-tests = { 5 | fetch-pypi-hash = { 6 | value = all-fetchers.fetch-pypi-hash { 7 | name = "Pillow"; 8 | version = "5.0.0"; 9 | }; 10 | expected = { 11 | sha256 = "12f29d6c23424f704c66b5b68c02fe0b571504459605cfe36ab8158359b0e1bb"; 12 | }; 13 | }; 14 | fetch-pypi-hash-bad-filename = { 15 | value = all-fetchers.fetch-pypi-hash { 16 | name = "Pillow"; 17 | version = "5.0.0"; 18 | filename = "this-better-not-exist.tar.gz"; 19 | }; 20 | expected = { 21 | failure-type = "missing-filename"; 22 | }; 23 | }; 24 | fetch-pypi-zip = { 25 | value = all-fetchers.fetch-pypi-hash rec { 26 | name = "recordclass"; 27 | version = "0.5"; 28 | filename = "${name}-${version}.zip"; 29 | }; 30 | expected = { 31 | sha256 = "4582ed9621846cd67c7647edaa2940ddaca75fda3ad5fd77616a347025e6fa78"; 32 | }; 33 | }; 34 | } // (import ../fetch-git/tests { inherit pkgs all-fetchers seed; }); 35 | 36 | ### 37 | nixpkgs = fetchGit { url = "git://github.com/NixOS/nixpkgs.git"; 38 | ref = "release-18.03"; 39 | rev = "03667476e330f91aefe717a3e36e56015d23f848"; 40 | }; 41 | pkgs = import nixpkgs {}; 42 | make-extra-builtins = 43 | pkgs.callPackage ../make-extra-builtins.nix {}; 44 | # Simulate build/run staging 45 | extra-builtins = make-extra-builtins { 46 | fetchers = import ../all-fetchers.nix; 47 | }; 48 | exec' = builtins.exec or 49 | (throw "Tests require the allow-unsafe-native-code-during-evaluation Nix setting to be true"); 50 | all-fetchers = import "${extra-builtins}/extra-builtins.nix" { 51 | exec = exec'; 52 | }; 53 | 54 | run-test = acc: name: let 55 | inherit (comparison-tests.${name}) value expected; 56 | in if value != expected 57 | then builtins.trace "Test ${name} failed" ( 58 | builtins.trace "Expected:" ( 59 | builtins.trace expected ( 60 | builtins.trace "Got: " ( 61 | builtins.trace value "failure")))) 62 | else acc; 63 | in builtins.foldl' run-test "success" (builtins.attrNames comparison-tests) 64 | -------------------------------------------------------------------------------- /test/run: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -euo pipefail 4 | 5 | declare tmpdir 6 | tmpdir="$(mktemp -d)" 7 | 8 | cleanup () { 9 | rm -fR "$tmpdir" 10 | } 11 | 12 | trap cleanup EXIT 13 | 14 | mkdir -p "$tmpdir/cache" 15 | # share fetchGit cache... 16 | ln -s "${XDG_CACHE_HOME:-$HOME/.cache}/nix" "$tmpdir/cache/nix" 17 | export XDG_CACHE_HOME="$tmpdir/cache" 18 | 19 | declare expr 20 | expr="$(dirname "${BASH_SOURCE[0]}")/default.nix" 21 | 22 | SEED=$RANDOM 23 | echo "Seed: $SEED" >&2 24 | echo -n "Testing with an empty cache... " >&2 25 | declare empty 26 | empty="$(nix-instantiate --eval --read-write-mode \ 27 | --allow-unsafe-native-code-during-evaluation \ 28 | --arg seed "$SEED" \ 29 | "$expr")" 30 | echo "$empty" >&2 31 | [ "$empty" = "\"success\"" ] 32 | 33 | echo -n "Testing with an warm cache..." >&2 34 | declare warm 35 | warm="$(nix-instantiate --eval --read-write-mode \ 36 | --allow-unsafe-native-code-during-evaluation \ 37 | --arg seed "$SEED" \ 38 | "$expr")" 39 | echo "$warm" >&2 40 | [ "$warm" = "\"success\"" ] 41 | --------------------------------------------------------------------------------