├── .gitignore ├── .travis.yml ├── LICENSE ├── README.rst ├── default.nix ├── nox ├── cache.py ├── enumerate_tests.nix ├── nixpkgs_repo.py ├── review.py ├── search.py ├── tests │ ├── __init__.py │ └── test_review.py └── update.py ├── requirements.txt ├── screen.png ├── setup.cfg └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg-info/ 2 | /env/ 3 | __pycache__ 4 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: nix 2 | deploy: 3 | provider: pypi 4 | user: madjar 5 | password: 6 | secure: pnWh9Nr4id3S3Kz5kotR6DAQXFE8Ix+H983KHW+oMSK0T5Idn27sWyU3opUdfbZBuDoVfTufNuLB8O9Z+hnTB5Ji1lo0wIyN1GP72l1F9Rmbo45k0eTRW36y54bIFcP04nf9QeSZ3dHPMFg+FN+p80HwiIzbiWJtLmTC26aqkEQ= 7 | on: 8 | tags: true 9 | before_deploy: pip install twine 10 | distributions: sdist bdist_wheel 11 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Georges Dubus 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Nox 2 | === 3 | 4 | Nox is a small tool that makes the use of the Nix package manager 5 | easier. 6 | 7 | Nox is written in Python 3 and requires nix 1.8 and git. It is 8 | released under MIT license. 9 | 10 | Try it 11 | ------ 12 | 13 | You can install it from nixpkgs by running ``nix-env -i nox``. 14 | 15 | To try the last version, just clone the repository, run ``nix-build``, 16 | and run the resulting binaries in ``result/bin``. To install it, run 17 | ``nix-env -if .``. 18 | 19 | Search 20 | ------ 21 | 22 | Just run ``nox QUERY`` to search for a nix package. The underlying 23 | ``nix-env`` invocation is cached to make the search faster than your 24 | usual ``nix-env -qa | grep QUERY``. In addition, package descriptions 25 | are searched as well as their names. You may specify multiple queries, 26 | in which case only packages matching all of them will be listed. Queries 27 | are considered as Python-style regular expressions. 28 | 29 | .. image:: screen.png 30 | 31 | Once you have the results, type the numbers of the packages to install. 32 | 33 | Bonus: if you enter the letter 's' at the beginning of the package 34 | numbers list, a nix-shell will be started with those packages instead. 35 | 36 | Review 37 | ------ 38 | 39 | The ``nox-review`` command helps you find what has changed in nixpkgs, and 40 | build changed packages, so you're sure they are not broken. There are 3 modes: 41 | 42 | - ``nox-review wip`` compares the nixpkgs in the current working dir 43 | against a commit, so you can check that your changes break 44 | nothing. Defaults to comparing to ``HEAD`` (the last commit), but you 45 | can change it: ``nox-review wip --against master^'``. 46 | - ``nox-review pr PR`` finds the packages touched by the given PR and build 47 | them. 48 | 49 | Experimental 50 | ------------ 51 | 52 | I'm working on a new command, ``nox-update``, that will display 53 | information about what is about to be updated, especially giving info 54 | not provided by nixos-rebuild: 55 | 56 | - Why is everything being installed? 57 | - Which are package upgrades? 58 | - Which are expression changes? 59 | - Which are only rebuilds trigerred by dependency changes? 60 | - Especially, what package triggered the rebuild? 61 | 62 | A picture is better than a thousand words, so here is what it looks like for 63 | now: 64 | 65 | .. image:: http://i.imgur.com/jdOGN94.png 66 | -------------------------------------------------------------------------------- /default.nix: -------------------------------------------------------------------------------- 1 | with import { }; 2 | 3 | nox.overrideAttrs (oldAttrs : { 4 | src = ./.; 5 | buildInputs = oldAttrs.buildInputs ++ [ git ]; 6 | propagatedBuildInputs = oldAttrs.propagatedBuildInputs ++ [ python3.pkgs.psutil ]; 7 | }) 8 | -------------------------------------------------------------------------------- /nox/cache.py: -------------------------------------------------------------------------------- 1 | from dogpile.cache import make_region 2 | import getpass 3 | 4 | region = make_region().configure( 5 | 'dogpile.cache.dbm', 6 | expiration_time=36000, 7 | arguments={'filename': '/tmp/nox.dbm.'+getpass.getuser()} 8 | ) 9 | -------------------------------------------------------------------------------- /nox/enumerate_tests.nix: -------------------------------------------------------------------------------- 1 | # small utility to gather the attrname of all nixos tests 2 | # this is cpu intensive so instead only process 1/numJobs of the list 3 | { jobIndex ? 0, numJobs ? 1, disable_blacklist ? false }: 4 | let 5 | tests = (import { 6 | supportedSystems = [ builtins.currentSystem ]; 7 | }).tests; 8 | lib = (import ); 9 | blacklist = if disable_blacklist then [] else 10 | # list of patterns of tests to never rebuild 11 | # they depend on ./. so are rebuilt on each commit 12 | [ "installer" "containers-.*" "initrd-network-ssh" "boot" "ec2-.*" ]; 13 | enumerate = prefix: name: value: 14 | # an attr in tests is either { x86_64 = derivation; } or an attrset of such values. 15 | if lib.any (x: builtins.match x name != null) blacklist then [] else 16 | if lib.hasAttr builtins.currentSystem value then 17 | [ {attr="${prefix}${name}"; drv = value.${builtins.currentSystem};} ] 18 | else 19 | lib.flatten (lib.attrValues (lib.mapAttrs (enumerate (prefix + name + ".")) value)); 20 | # list of {attr="tests.foo"; drv=...} 21 | data = enumerate "" "tests" tests; 22 | # only keep a fraction of the list 23 | filterFraction = list: (lib.foldl' 24 | ({n, result}: element: { 25 | result = if n==jobIndex then result ++ [ element ] else result; 26 | n = if n+1==numJobs then 0 else n+1; 27 | }) 28 | { n=0; result = []; } 29 | list).result; 30 | myData = filterFraction data; 31 | evaluable = lib.filter ({attr, drv}: (builtins.tryEval drv).success) myData; 32 | in 33 | map ({attr, drv}: {inherit attr; drv=drv.drvPath;}) evaluable 34 | 35 | 36 | 37 | -------------------------------------------------------------------------------- /nox/nixpkgs_repo.py: -------------------------------------------------------------------------------- 1 | import os 2 | import subprocess 3 | import json 4 | from pathlib import Path 5 | from collections import defaultdict 6 | from concurrent.futures import ThreadPoolExecutor 7 | from fnmatch import fnmatch 8 | 9 | from .cache import region 10 | 11 | import click 12 | import psutil 13 | 14 | 15 | class Repo: 16 | def __init__(self): 17 | nox_dir = Path(click.get_app_dir('nox', force_posix=True)) 18 | if not nox_dir.exists(): 19 | nox_dir.mkdir() 20 | 21 | nixpkgs = nox_dir / 'nixpkgs' 22 | self.path = str(nixpkgs) 23 | 24 | if not nixpkgs.exists(): 25 | click.echo('==> Creating nixpkgs repo in {}'.format(nixpkgs)) 26 | self.git(['init', '--quiet', self.path], cwd=False) 27 | self.git('remote add origin https://github.com/NixOS/nixpkgs.git') 28 | self.git('config user.email nox@example.com') 29 | self.git('config user.name nox') 30 | 31 | if (Path.cwd() / '.git').exists(): 32 | git_version = self.git('version', output=True).strip() 33 | if git_version >= 'git version 2': 34 | click.echo("==> We're in a git repo, trying to fetch it") 35 | 36 | self.git(['fetch', str(Path.cwd()), '--update-shallow', '--quiet']) 37 | else: 38 | click.echo("==> Old version of git detected ({}, maybe on travis)," 39 | " not trying to fetch from local, fetch 50 commits from master" 40 | " instead".format(git_version)) 41 | self.git('fetch origin master --depth 50') 42 | 43 | def git(self, command, *args, cwd=None, output=False, **kwargs): 44 | if cwd is None: 45 | cwd = self.path 46 | elif cwd is False: 47 | cwd = None 48 | if isinstance(command, str): 49 | command = command.split() 50 | # suppress gpg prompt when git command tries to create/modify commit 51 | command = ['git', '-c', 'commit.gpgSign=false'] + command 52 | f = subprocess.check_output if output else subprocess.check_call 53 | return f(command, *args, cwd=cwd, universal_newlines=output, **kwargs) 54 | 55 | def checkout(self, sha): 56 | self.git(['checkout', '-f', '--quiet', sha]) 57 | 58 | def sha(self, ref): 59 | return self.git(['rev-parse', '--verify', ref], output=True).strip() 60 | 61 | def fetch(self, ref, depth=1): 62 | return self.git(['fetch', '--depth', str(depth), '--quiet', 63 | 'origin', '+refs/{}'.format(ref)]) 64 | 65 | def merge_base(self, first, second): 66 | try: 67 | return self.git(['merge-base', first, second], output=True).strip() 68 | except subprocess.CalledProcessError: 69 | return None 70 | 71 | 72 | _repo = None 73 | 74 | 75 | def get_repo(): 76 | global _repo 77 | if not _repo: 78 | _repo = Repo() 79 | return _repo 80 | 81 | 82 | class Buildable: 83 | """ 84 | attr (str): attribute name under which the buildable can be built 85 | path (str or tuple of them): for example , a list can be used to 86 | pass other arguments like --argstr foo bar 87 | hash: anything which contains the drvPath for example 88 | __slots__ = "path", "attr", "extra_args", "hash" 89 | """ 90 | def __init__(self, attr, hash, path=""): 91 | self.attr = attr 92 | self.hash = hash 93 | self.path = path 94 | 95 | def __eq__(self, other): 96 | return hash(self) == hash(other) 97 | 98 | def __hash__(self): 99 | return hash(self.hash) 100 | 101 | @property 102 | def path_args(self): 103 | if isinstance(self.path, str): 104 | return (self.path, ) 105 | return self.path 106 | 107 | def __repr__(self): 108 | return "Buildable(attr={!r}, hash={!r}, path={!r})".format(self.attr, self.hash, self.path_args) 109 | 110 | 111 | def get_build_commands(buildables, program="nix-build", extra_args=[]): 112 | """ Get the appropriate commands to use to build the given buildables """ 113 | prefix = [program] 114 | prefix += extra_args 115 | path_to_cmd = defaultdict(lambda *x: prefix[:]) 116 | for b in buildables: 117 | command = path_to_cmd[b.path_args] 118 | command.append('-A') 119 | command.append(b.attr) 120 | return [command + list(path) for path, command in path_to_cmd.items()] 121 | 122 | 123 | def at_given_sha(f): 124 | """decorator which calls the wrappee with the path of nixpkgs at the given sha 125 | 126 | Turns a function path -> 'a into a function sha -> 'a. 127 | If the sha passed is None, passes the current directory as argument. 128 | """ 129 | def _wrapped(sha, *args, **kwargs): 130 | if sha is not None: 131 | repo = get_repo() 132 | repo.checkout(sha) 133 | path = repo.path 134 | else: 135 | path = os.getcwd() 136 | return f(path, *args, **kwargs) 137 | _wrapped.__name__ = f.__name__ 138 | return _wrapped 139 | 140 | 141 | def cache_on_not_None(f): 142 | """like region.cache_on_argument() but does not cache if the key starts None""" 143 | wf = region.cache_on_arguments()(f) 144 | def _wrapped(arg, *args): 145 | if arg is None: 146 | return f(arg, *args) 147 | return wf(arg, *args) 148 | _wrapped.__name__ = f.__name__ 149 | return _wrapped 150 | 151 | 152 | @cache_on_not_None 153 | @at_given_sha 154 | def packages_for_sha(path): 155 | """List all nix packages in the repo, as a set of buildables""" 156 | output = subprocess.check_output(['nix-env', '-f', path, '-qaP', 157 | '--out-path', '--show-trace'], universal_newlines=True) 158 | return {Buildable(attr, hash) for attr, hash in 159 | map(lambda line: line.split(" ", 1), output.splitlines())} 160 | 161 | 162 | enumerate_tests = str(Path(__file__).parent / "enumerate_tests.nix") 163 | 164 | 165 | @cache_on_not_None 166 | @at_given_sha 167 | def tests_for_sha(path, disable_blacklist=False): 168 | """List all tests wich evaluate in the repo, as a set of (attr, drvPath)""" 169 | num_jobs = 32 170 | # at this size, each job takes 1~1.7 GB mem 171 | max_workers = max(1, psutil.virtual_memory().available//(1700*1024*1024)) 172 | # a job is also cpu hungry 173 | try: 174 | max_workers = min(max_workers, os.cpu_count()) 175 | except: pass 176 | 177 | def eval(i): 178 | output = subprocess.check_output(['nix-instantiate', '--eval', 179 | '--json', '--strict', '-I', "nixpkgs="+str(path), enumerate_tests, 180 | '--arg', "jobIndex", str(i), '--arg', 'numJobs', str(num_jobs), 181 | '--arg', 'disableBlacklist', str(disable_blacklist).lower(), 182 | '--show-trace'], universal_newlines=True) 183 | return json.loads(output) 184 | 185 | with ThreadPoolExecutor(max_workers=max_workers) as executor: 186 | evals = executor.map(eval, range(num_jobs)) 187 | 188 | path = ("", "--arg", "supportedSystems", "[builtins.currentSystem]") 189 | attrs = set() 190 | for partial in evals: 191 | for test in partial: 192 | b = Buildable(test["attr"], test["drv"], path=path) 193 | attrs.add(b) 194 | 195 | return attrs 196 | 197 | -------------------------------------------------------------------------------- /nox/review.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import tempfile 4 | import subprocess 5 | import re 6 | from pathlib import Path 7 | 8 | import click 9 | import requests 10 | 11 | from .nixpkgs_repo import get_repo, at_given_sha, get_build_commands, packages_for_sha, tests_for_sha 12 | 13 | 14 | @at_given_sha 15 | def build_sha(path, buildables, extra_args=[], dry_run=False): 16 | """Build the given package attributes in the given nixpkgs path""" 17 | if not buildables: 18 | click.echo('Nothing changed') 19 | return 20 | 21 | canonical_path = str(Path(path).resolve()) 22 | result_dir = tempfile.mkdtemp(prefix='nox-review-') 23 | click.echo('Building in {}: {}'.format(click.style(result_dir, bold=True), 24 | click.style(' '.join(s.attr for s in buildables), bold=True))) 25 | 26 | for command in get_build_commands(buildables, extra_args=extra_args+["-I", "nixpkgs="+canonical_path]): 27 | click.echo('Invoking {}'.format(' '.join(command))) 28 | 29 | if not dry_run: 30 | try: 31 | subprocess.check_call(command, cwd=result_dir) 32 | except subprocess.CalledProcessError: 33 | click.secho('The invocation of "{}" failed'.format(' '.join(command)), fg='red') 34 | sys.exit(1) 35 | click.echo('Result in {}'.format(click.style(result_dir, bold=True))) 36 | subprocess.check_call(['ls', '-l', result_dir]) 37 | 38 | 39 | def build_difference(old_sha, new_sha, extra_args=[], with_tests=False, disable_test_blacklist=False, dry_run=False): 40 | click.echo("Listing old packages...") 41 | before = packages_for_sha(old_sha) 42 | if with_tests: 43 | click.echo("Listing old tests...") 44 | before |= tests_for_sha(old_sha, disable_test_blacklist) 45 | click.echo("Listing new packages...") 46 | after = packages_for_sha(new_sha) 47 | if with_tests: 48 | click.echo("Listing new tests...") 49 | after |= tests_for_sha(new_sha, disable_test_blacklist) 50 | build_sha(new_sha, after-before, extra_args, dry_run) 51 | 52 | 53 | def setup_nixpkgs_config(f): 54 | def _(*args, **kwargs): 55 | with tempfile.NamedTemporaryFile() as cfg: 56 | cfg.write(b"pkgs: {}") 57 | cfg.flush() 58 | os.environ['NIXPKGS_CONFIG'] = cfg.name 59 | f(*args, **kwargs) 60 | return _ 61 | 62 | 63 | @click.group() 64 | @click.option('--keep-going', '-k', is_flag=True, help='Keep going in case of failed builds') 65 | @click.option('--dry-run', is_flag=True, help="Don't actually build packages, just print the commands that would have been run") 66 | @click.option('--with-tests', is_flag=True, help="Also rebuild affected NixOS tests") 67 | @click.option('--all-tests', is_flag=True, help="Do not blacklist tests known to be false positives") 68 | @click.pass_context 69 | def cli(ctx, keep_going, dry_run, with_tests, all_tests): 70 | """Review a change by building the touched commits""" 71 | ctx.obj = {'extra-args': []} 72 | if keep_going: 73 | ctx.obj['extra-args'].append('--keep-going') 74 | ctx.obj['dry_run'] = dry_run 75 | ctx.obj['tests'] = with_tests 76 | ctx.obj['no-blacklist'] = all_tests 77 | 78 | 79 | @cli.command(short_help='difference between working tree and a commit') 80 | @click.option('--against', default='HEAD') 81 | @click.pass_context 82 | @setup_nixpkgs_config 83 | def wip(ctx, against): 84 | """Build in the current dir the packages that different from AGAINST (default to HEAD)""" 85 | if not Path('default.nix').exists(): 86 | click.secho('"nox-review wip" must be run in a nix repository.', fg='red') 87 | return 88 | 89 | dirty_working_tree = subprocess.call('git diff --quiet --ignore-submodules HEAD'.split()) 90 | 91 | if not dirty_working_tree: 92 | if against == 'HEAD': 93 | click.secho('No uncommit changes. Did you mean to use the "--against" option?') 94 | return 95 | 96 | sha = subprocess.check_output(['git', 'rev-parse', '--verify', against]).decode().strip() 97 | 98 | build_difference(sha, None, extra_args=ctx.obj['extra-args'], with_tests=ctx.obj["tests"], disable_test_blacklist=ctx.obj["no-blacklist"], dry_run=ctx.obj['dry_run']) 99 | 100 | 101 | @cli.command('pr', short_help='changes in a pull request') 102 | @click.option('--slug', default=None, help='The GitHub "slug" of the repository in the from of owner_name/repo_name.') 103 | @click.option('--token', help='The GitHub API token to use.') 104 | @click.option('--merge/--no-merge', default=True, help='Merge the PR against its base.') 105 | @click.argument('pr', type=click.STRING) 106 | @click.pass_context 107 | @setup_nixpkgs_config 108 | def review_pr(ctx, slug, token, merge, pr): 109 | """Build the changes induced by the given pull request""" 110 | 111 | # Allow the 'pr' parameter to be either the numerical ID or an URL to the PR on GitHub. 112 | # Also if it's an URL, parse the proper --slug argument from that. 113 | m = re.match('^(?:https?://(?:www\.)?github\.com/([^/]+/[^/]+)/pull/)?([0-9]+)$', pr, re.IGNORECASE) 114 | if not m: 115 | click.echo("Error: parameter to 'nox-review pr' must be a valid pull request number or URL.") 116 | sys.exit(1) 117 | pr = m[2] 118 | if m[1]: 119 | if slug: 120 | click.echo("Error: '--slug' option can't be used together with a pull request URL.") 121 | sys.exit(1) 122 | slug = m[1] 123 | elif not slug: 124 | slug = 'NixOS/nixpkgs' 125 | 126 | pr_url = 'https://api.github.com/repos/{}/pulls/{}'.format(slug, pr) 127 | headers = {} 128 | if token: 129 | headers['Authorization'] = 'token {}'.format(token) 130 | request = requests.get(pr_url, headers=headers) 131 | if request.status_code == 403 and request.headers['X-RateLimit-Remaining'] == '0': 132 | click.secho('You have exceeded the GitHub API rate limit. Try again in about an hour.') 133 | if not token: 134 | click.secho('Or try running this again, providing an access token:') 135 | click.secho('$ nox-review pr --token=YOUR_TOKEN_HERE {}'.format(pr)) 136 | sys.exit(1) 137 | payload = request.json() 138 | click.echo('=== Reviewing PR {} : {}'.format( 139 | click.style(pr, bold=True), 140 | click.style(payload.get('title', '(n/a)'), bold=True))) 141 | 142 | base_ref = payload['base']['ref'] 143 | 144 | repo = get_repo() 145 | 146 | click.echo('==> Fetching base ({})'.format(base_ref)) 147 | base_refspec = 'heads/{}'.format(payload['base']['ref']) 148 | repo.fetch(base_refspec) 149 | base = repo.sha('FETCH_HEAD') 150 | 151 | click.echo('==> Fetching PR') 152 | head_refspec = 'pull/{}/head'.format(pr) 153 | repo.fetch(head_refspec) 154 | head = repo.sha('FETCH_HEAD') 155 | 156 | if merge: 157 | click.echo('==> Fetching extra history for merging') 158 | depth = 10 159 | while not repo.merge_base(head, base): 160 | repo.fetch(base_refspec, depth=depth) 161 | repo.fetch(head_refspec, depth=depth) 162 | depth *= 2 163 | 164 | # It looks like this isn't enough for a merge, so we fetch more 165 | repo.fetch(base_refspec, depth=depth) 166 | 167 | click.echo('==> Merging PR into base') 168 | 169 | repo.checkout(base) 170 | repo.git(['merge', head, '--no-ff', '-qm', 'Nox automatic merge']) 171 | merged = repo.sha('HEAD') 172 | 173 | old = base 174 | new = merged 175 | 176 | else: 177 | commits = requests.get(payload['commits_url'], headers=headers).json() 178 | old = commits[-1]['parents'][0]['sha'] 179 | new = payload['head']['sha'] 180 | 181 | build_difference(old, new, extra_args=ctx.obj['extra-args'], with_tests=ctx.obj["tests"], disable_test_blacklist=ctx.obj["no-blacklist"], dry_run=ctx.obj['dry_run']) 182 | -------------------------------------------------------------------------------- /nox/search.py: -------------------------------------------------------------------------------- 1 | import os 2 | import collections 3 | import json 4 | import subprocess 5 | import re 6 | 7 | import click 8 | 9 | from .cache import region 10 | 11 | 12 | class NixEvalError(Exception): 13 | pass 14 | 15 | 16 | def nix_packages_json(): 17 | click.echo('Refreshing cache') 18 | try: 19 | output = subprocess.check_output(['nix-env', '-qa', '--json', '--show-trace'], 20 | universal_newlines=True) 21 | except subprocess.CalledProcessError as e: 22 | raise NixEvalError from e 23 | return json.loads(output) 24 | 25 | 26 | Package = collections.namedtuple('Package', 'attribute name description') 27 | 28 | 29 | def key_for_path(path): 30 | try: 31 | manifest = os.path.join(path, 'manifest.nix') 32 | with open(manifest) as f: 33 | return f.read() 34 | except (FileNotFoundError, NotADirectoryError): 35 | pass 36 | if os.path.exists(os.path.join(path, '.git')): 37 | return subprocess.check_output('git rev-parse --verify HEAD'.split(), 38 | cwd=path) 39 | click.echo('Warning: could not find a version indicator for {}'.format(path)) 40 | return None 41 | 42 | 43 | def all_packages(force_refresh=False): 44 | defexpr = os.path.expanduser('~/.nix-defexpr/') 45 | paths = os.listdir(defexpr) 46 | key = str({p: key_for_path(defexpr + p) for p in paths}) 47 | 48 | if force_refresh: 49 | region.delete(key) 50 | 51 | packages_json = region.get_or_create(key, nix_packages_json) 52 | return (Package(attr, v['name'], v.get('meta', {}).get('description', '')) 53 | for attr, v in packages_json.items()) 54 | 55 | 56 | @click.command() 57 | @click.argument('queries', nargs=-1) 58 | @click.option('--force-refresh', is_flag=True) 59 | def main(queries, force_refresh): 60 | """Search a package in nix""" 61 | patterns = [re.compile(query, re.IGNORECASE) for query in queries] 62 | 63 | try: 64 | results = [p for p in all_packages(force_refresh) 65 | if any((all((pat.search(s) for pat in patterns)) for s in p))] 66 | except NixEvalError: 67 | raise click.ClickException('An error occured while running nix (displayed above). Maybe the nixpkgs eval is broken.') 68 | results.sort() 69 | for i, p in enumerate(results, 1): 70 | line = '{} {} ({})\n {}'.format( 71 | click.style(str(i), fg='black', bg='yellow'), 72 | click.style(p.name, bold=True), 73 | click.style(p.attribute, dim=True), 74 | click.style(p.description.replace("\n", "\n "))) 75 | click.echo(line) 76 | 77 | if results: 78 | def parse_input(inp): 79 | if inp[0] == 's': 80 | action = 'shell' 81 | inp = inp[1:] 82 | else: 83 | action = 'install' 84 | packages = [results[int(i) - 1] for i in inp.split()] 85 | return action, packages 86 | 87 | action, packages = click.prompt('Packages to install', 88 | value_proc=parse_input) 89 | attributes = [p.attribute for p in packages] 90 | if action == 'install': 91 | subprocess.check_call(['nix-env', '-iA', '--show-trace'] + attributes) 92 | elif action == 'shell': 93 | attributes = [a[len('nixpkgs.'):] for a in attributes] 94 | subprocess.check_call(['nix-shell', '-p', '--show-trace'] + attributes) 95 | -------------------------------------------------------------------------------- /nox/tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/madjar/nox/f732934c24593b8d6bbe2ad0e3700bd836797d80/nox/tests/__init__.py -------------------------------------------------------------------------------- /nox/tests/test_review.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | from .. import review, nixpkgs_repo 4 | 5 | 6 | class TestReview(unittest.TestCase): 7 | def test_get_build_command(self): 8 | nox = nixpkgs_repo.Buildable("nox", hash("nox")) 9 | result = review.get_build_commands([nox]) 10 | self.assertEqual([["nix-build", "-A", "nox", ""]], result) 11 | 12 | def test_build_in_path(self): 13 | nox = nixpkgs_repo.Buildable("nox", hash("nox")) 14 | # Just do a dry run to make sure there aren't any exceptions 15 | self.assertIs(None, review.build_sha(None, [nox], extra_args=[], dry_run=True)) 16 | -------------------------------------------------------------------------------- /nox/update.py: -------------------------------------------------------------------------------- 1 | import click 2 | import re 3 | import subprocess 4 | 5 | from enum import Enum 6 | from bisect import bisect 7 | from pkg_resources import parse_version 8 | from pathlib import Path 9 | from characteristic import attributes 10 | from collections import defaultdict 11 | 12 | def query(*args): 13 | return subprocess.check_output(['nix-store', '--query'] + list(args), 14 | universal_newlines=True) 15 | 16 | @attributes(['full_name', 'path'], apply_with_init=False) 17 | class NixPath: 18 | def __init__(self, path): 19 | self.path = path 20 | self.is_drv = self.path.endswith('.drv') 21 | name_slice_end = -4 if self.is_drv else None 22 | self.full_name = self.path[44:name_slice_end] 23 | 24 | m = re.search(r'-(\d.*)', self.full_name) 25 | if m: 26 | self.name = self.full_name[:m.start()] 27 | self.version = self.full_name[m.start()+1:] 28 | m = re.search(r'(\.[a-zA-Z][a-zA-Z0-9]*)+$', self.version) 29 | if m: 30 | self.extension = self.version[m.start()+1:] 31 | self.shortversion = self.version[:m.start()] 32 | else: 33 | self.extension = None 34 | self.shortversion = self.version 35 | else: 36 | self.name = self.full_name 37 | self.version = None 38 | self.extension = None 39 | self.shortversion = None 40 | 41 | def refs(self): 42 | return {NixPath(p) for p in query('--references', self.path).strip().split('\n')} 43 | 44 | def outputs(self): 45 | return set(query('--outputs', self.path).strip().split('\n')) 46 | 47 | 48 | def current_system_drv(old_path): 49 | current_system = str(Path(old_path if old_path else '/run/current-system').resolve()) 50 | return NixPath(current_system if current_system.endswith('.drv') else query('--deriver', current_system).strip()) 51 | 52 | 53 | def new_system_drv(new_path): 54 | if new_path: 55 | new_system = str(Path(new_path).resolve()) 56 | return NixPath(new_system if new_system.endswith('.drv') else query('--deriver', new_system).strip()) 57 | with subprocess.Popen(['nixos-rebuild', 'dry-run'], 58 | stderr=subprocess.PIPE, 59 | universal_newlines=True) as process: 60 | _, output = process.communicate() 61 | 62 | m = re.search(r'.*nixos-\d{2}.*', output) 63 | return m and NixPath(m.group().strip()) 64 | 65 | 66 | def display_path(pkg, bold): 67 | is_drv = pkg.is_drv 68 | name_slice_end = -4 if is_drv else None 69 | path = pkg.path 70 | return (path[:44] + 71 | click.style(path[44:name_slice_end], bold=bold) + 72 | (path[name_slice_end:] if is_drv else '')) 73 | 74 | ChangeType = Enum('ChangeType', 'source fixed expression new version normal') 75 | 76 | class DepsTree: 77 | def __init__(self, refs_tree): 78 | self.seen = set() 79 | self.refs_tree = refs_tree 80 | 81 | def show(self, pkg, opts, level=0): 82 | ctype = self.refs_tree[pkg.path][1] 83 | if pkg.path not in self.seen and (not opts['quiet'] or ctype != ChangeType.fixed): 84 | self.seen.add(pkg.path) 85 | click.echo(' '*level + display_path(pkg, bold=True) + ' : ', nl=False) 86 | if ctype == ChangeType.source: 87 | click.secho('Source file changed', bold=True) 88 | elif ctype == ChangeType.fixed: 89 | click.secho('Fixed-output derivation changed', bold=True) 90 | elif ctype == ChangeType.expression: 91 | click.secho('Expression changed', bold=True) 92 | elif ctype == ChangeType.new: 93 | click.secho('seems to be new', bold=True) 94 | elif ctype == ChangeType.version: 95 | opkg = self.refs_tree[pkg.path][2] 96 | if opkg.extension == pkg.extension: 97 | click.secho('new version ({} -> {})'.format(opkg.shortversion, pkg.shortversion), bold=True) 98 | else: 99 | click.secho('new version ({} -> {})'.format(opkg.version, pkg.version), bold=True) 100 | elif ctype == ChangeType.normal: 101 | click.echo() 102 | 103 | if self.refs_tree[pkg.path][0]: 104 | (removed_packages, recurse_packages) = self.refs_tree[pkg.path][3:] 105 | level=level+1 106 | for rpkg in removed_packages: 107 | click.echo(' '*level + display_path(rpkg, bold=True) + ' : seems to be removed') 108 | for rpkg in recurse_packages: 109 | self.show(rpkg, opts, level) 110 | elif not opts['quiet']: 111 | click.echo(' '*level + display_path(pkg, bold=False) + ' [...]') 112 | 113 | def diff_pkgs(refs_tree, current_drv, new_drv, opts, level=0): 114 | if new_drv.path in refs_tree: 115 | return 116 | 117 | if not current_drv: 118 | refs_tree[new_drv.path] = (False, ChangeType.new, current_drv) 119 | return 120 | 121 | if not current_drv.is_drv or not new_drv.is_drv: 122 | refs_tree[new_drv.path] = (False, ChangeType.source, current_drv) 123 | return 124 | 125 | ctype = ChangeType.normal 126 | if current_drv.version != new_drv.version: 127 | if level > opts['max_level']: 128 | refs_tree[new_drv.path] = (False, ChangeType.version, current_drv) 129 | return 130 | else: 131 | ctype = ChangeType.version 132 | 133 | # Fixed-output derivation changed, but content didn't 134 | if current_drv.outputs() == new_drv.outputs(): 135 | if level > opts['max_level']: 136 | refs_tree[new_drv.path] = (False, ChangeType.fixed, current_drv) 137 | return 138 | else: 139 | ctype = ChangeType.fixed 140 | 141 | old_pkgs = current_drv.refs() 142 | new_pkgs = new_drv.refs() 143 | 144 | if old_pkgs == new_pkgs: 145 | refs_tree[new_drv.path] = (False, ChangeType.expression if ctype == ChangeType.normal else ctype, current_drv); 146 | return 147 | 148 | removed_packages = old_pkgs - new_pkgs 149 | current_fullnames = defaultdict(list) 150 | current_names = defaultdict(list) 151 | for drv in removed_packages: 152 | current_fullnames[drv.full_name].append(drv) 153 | if drv.version: 154 | current_names[(drv.name, bool(drv.extension))].append((parse_version(drv.version), drv)) 155 | for l in current_names.values(): 156 | l.sort() 157 | 158 | recurse_packages = new_pkgs - old_pkgs 159 | for pkg in recurse_packages: 160 | previous = None 161 | pkgs = current_fullnames[pkg.full_name] 162 | if pkgs: 163 | previous = pkgs[0] 164 | elif pkg.version: 165 | versions = current_names[(pkg.name, bool(pkg.extension))] 166 | v = parse_version(pkg.version) 167 | prev = bisect(versions, (v, pkg)) - 1 168 | if prev >= 0: 169 | previous = versions[prev][1] 170 | 171 | if previous: 172 | current_fullnames[previous.full_name].remove(previous) 173 | if previous.version: 174 | current_names[(previous.name, bool(previous.extension))].remove((parse_version(previous.version), previous)) 175 | removed_packages.discard(previous) 176 | 177 | diff_pkgs(refs_tree, previous, pkg, opts, level + 1) 178 | 179 | refs_tree[new_drv.path] = (True, ctype, current_drv, sorted(removed_packages), sorted(recurse_packages)) 180 | 181 | 182 | @click.command() 183 | @click.option('--max-level', default=0, type=click.INT) 184 | @click.option('--quiet', default=False, is_flag=True) 185 | @click.argument('old-path', default='', type=click.Path(exists=True)) 186 | @click.argument('new-path', default='', type=click.Path(exists=True)) 187 | def main(old_path, new_path, **opts): 188 | new_drv = new_system_drv(new_path) 189 | if not new_drv: 190 | click.echo('No system updates') 191 | return 192 | 193 | current_drv = current_system_drv(old_path) 194 | 195 | refs_tree = {} 196 | diff_pkgs(refs_tree, current_drv, new_drv, opts) 197 | tree = DepsTree(refs_tree) 198 | tree.show(new_drv, opts) 199 | 200 | # TODO : option -> display only thing to install, deps tree without 201 | # repetition, deps tree with omission of repeated paths, or full tree 202 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | click 2 | dogpile.cache 3 | characteristic 4 | requests -------------------------------------------------------------------------------- /screen.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/madjar/nox/f732934c24593b8d6bbe2ad0e3700bd836797d80/screen.png -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = nix-nox 3 | version = 0.0.7 4 | author = Georges Dubus 5 | author-email = georges.dubus@compiletoi.net 6 | summary = Tools to make nix nicer to use 7 | description-file = README.rst 8 | license = BSD 9 | classifer = 10 | Development Status :: 3 - Alpha 11 | Programming Language :: Python :: 3 12 | 13 | [files] 14 | packages = nox 15 | 16 | [entry_points] 17 | console_scripts = 18 | nox = nox.search:main 19 | nox-update = nox.update:main 20 | nox-review = nox.review:cli 21 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | setup_requires=['pbr'], 5 | pbr=True, 6 | package_data={'': ['enumerate_tests.nix']}, 7 | include_package_data=True, 8 | test_suite="nox.tests" 9 | ) 10 | --------------------------------------------------------------------------------