├── .envrc ├── .gitignore ├── README.md ├── default.nix ├── hello.py ├── nix ├── default.nix ├── python-packages-overrides.nix ├── python-packages.nix ├── sources.json └── sources.nix ├── pip2nix.ini ├── requirements.txt ├── setup.cfg ├── setup.py └── shell.nix /.envrc: -------------------------------------------------------------------------------- 1 | eval "$(lorri direnv)" -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /result 2 | __pycache__/ 3 | *.egg-info/ 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Update 2 | 3 | I see this repo is getting some minor popularity (judging from stars) so some people find the information valuable. Since I wrote this, I figured out a a better (in my opinoion) way of writing and distributing python applications with Nix. 4 | 5 | I created another tool called [Nix-CDE](https://github.com/takeda/nix-cde) that abstracts all the boilerplate and leaves configuration files similar to `configuration.nix` from NixOS. I believe that is much better and easier way to work on Python applications with Nix. The new way utilizes `poetry2nix` so it expects your Python project to use Poetry. It shouldn't be hard to switch from `setuptools` and due to PyPA pushing people to stop using `setuptools` many likely already moved away from it. 6 | 7 | Of course I'm still leaving this repo for all those for whom the new way doesn't work. 8 | 9 | # Integration of Nix with python (setuptools) 10 | 11 | This works best if you also have [direnv](https://direnv.net/) and [lorri](https://github.com/target/lorri) installed, all you need to do is just cd to the directory and you'll have all tooling available necessary for application developement (specified in `shell.nix`) (if you don't use direnv then just type `nix-shell` to enter something similar to virtualenv). 12 | 13 | In that mode the application is also installed e.g. you will have `hello` command available that will call `main()` in `hello.py` if you make changes to the file nothing needs to be built, it immediately takes effect as if the package was installed with `pip -e`. 14 | 15 | If you use direnv + lorri you just need to enter directory (if not just type `nix-shell`) and suddenly you have everything you need and the application is installed (try executing "hello" which will execute the python code, if you modify hello.py it immediately takes effect as if you were using `pip -e`) 16 | 17 | If you call `nix build` you'll get a result directory with `result/bin/hello` that just works as if it was a binary program (nix will make sure that all depended packages are available) 18 | 19 | # Using setup.cfg 20 | 21 | The configuration can be specified in a declarative way using `setup.cfg` you can get list of available options [here](https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files). In projects where you have a directory (with `__init__.py` file aka a package) you should replace `py_modules = hello` with `packages = find:` the `find:` takes care automatically including all directories that contain `__init__.py`, you can also specify them manually. 22 | 23 | ```ini 24 | [options.entry_points] 25 | console_scripts= 26 | hello = hello:main 27 | ``` 28 | Creates `hello` command that will invoke `main()` function in `hello` module. 29 | 30 | `install_requires` are *immediate* dependencies used by your application, you should only specify what application is using `pip-compile` will figure out their dependencies. 31 | 32 | Providing versions is optional, but a good practice you can get more information about it [here](https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-dependencies). You might not be familiar with `~=` for example: `psycopg2 ~= 2.8.5` this basically translates to `psycopg2 >= 2.8.5,< 2.9.0` basically it fixes package to specific version but allow updating the last number (in semantic versioning that's patch level so it would only allow bugfixes). NOTE: If you ever work with time and depend on `pytz`, you generally don't want to fix version on that package. `pytz` contains timze zones information which change all the time, so having the latest version of `pytz` ensures your calculations are correct. 33 | 34 | # Updating python packages 35 | 36 | ```shell-script 37 | $ pip-compile -Uv # generates requirements.txt from setup.cfg 38 | $ pip2nix generate # generates nix/python-packages.nix from requirements.txt 39 | ``` 40 | 41 | # Updating nix packages 42 | 43 | ```shell-script 44 | $ niv update # updates nix/sources.json 45 | ``` 46 | 47 | # Using packages with C library dependencies 48 | 49 | Since setuptools doesn't store any information about C library dependencies, if you use libraries that depend on them e.g. psycopg2, you need to provide them manually. You can see an example of [adding postgres 12 dependency to psycopg2](https://github.com/takeda/example_python_project/blob/master/nix/python-packages-overrides.nix). 50 | The great thing about it is that you achive 100% reproducible build that encompasses not only the python package dependencies, but also ensures that exact same python binary is used with exactly the same C libraries. 51 | -------------------------------------------------------------------------------- /default.nix: -------------------------------------------------------------------------------- 1 | { pkgs ? import ./nix {} 2 | , pythonPackages ? "python38Packages" 3 | }: 4 | 5 | let 6 | inherit (pkgs.lib) fix extends; 7 | 8 | basePythonPackages = with builtins; if isAttrs pythonPackages 9 | then pythonPackages 10 | else getAttr pythonPackages pkgs; 11 | 12 | pythonPackagesGenerated = pkgs.callPackage ./nix/python-packages.nix {}; 13 | pythonPackagesOverrides = pkgs.callPackage ./nix/python-packages-overrides.nix { inherit basePythonPackages; }; 14 | 15 | # extract package name, version & dependencies from setup.cfg 16 | setupcfg = import (pkgs.runCommand "setup.cfg" {} ''${pkgs.setupcfg2nix}/bin/setupcfg2nix ${./setup.cfg} > $out''); 17 | 18 | pythonPackagesLocalOverrides = self: super: { 19 | hello = super.buildSetupcfg { 20 | info = setupcfg; 21 | src = pkgs.gitignoreSource ./.; 22 | application = true; 23 | }; 24 | }; 25 | 26 | myPythonPackages = 27 | (fix 28 | (extends pythonPackagesLocalOverrides 29 | (extends pythonPackagesOverrides 30 | (extends pythonPackagesGenerated 31 | basePythonPackages.__unfix__)))); 32 | 33 | in myPythonPackages.hello 34 | -------------------------------------------------------------------------------- /hello.py: -------------------------------------------------------------------------------- 1 | from tqdm import tqdm 2 | import psycopg2 3 | 4 | def main(): 5 | print("Hello world!") 6 | for i in tqdm(range(int(9e6))): 7 | pass 8 | -------------------------------------------------------------------------------- /nix/default.nix: -------------------------------------------------------------------------------- 1 | { sources ? import ./sources.nix }: 2 | import sources.nixpkgs { 3 | overlays = [ 4 | (self: super: { 5 | inherit sources; 6 | inherit (import sources.gitignore { inherit (super) lib; }) gitignoreSource; 7 | inherit (import sources.niv { pkgs = super; }) niv; 8 | pip2nix = import sources.pip2nix { pkgs = super; pythonPackages = "python38Packages"; }; 9 | }) 10 | ]; 11 | } 12 | -------------------------------------------------------------------------------- /nix/python-packages-overrides.nix: -------------------------------------------------------------------------------- 1 | # Generated by pip2nix 0.8.0.dev1 2 | # Adjust to your needs, e.g. to provide C libraries. 3 | 4 | { pkgs, basePythonPackages }: 5 | 6 | self: super: { 7 | 8 | # Example adjustment for lxml: It needs a few C libraries 9 | # 10 | # lxml = super.lxml.override (attrs: { 11 | # buildInputs = with self; [ 12 | # pkgs.libxml2 13 | # pkgs.libxslt 14 | # ]; 15 | # }); 16 | 17 | psycopg2 = super.psycopg2.override (attrs: { 18 | nativeBuildInputs = with self; [ 19 | pkgs.postgresql_12 20 | ]; 21 | }); 22 | 23 | # Common needs 24 | 25 | # setuptools - avoid that we end up in a recursion 26 | # inherit (basePythonPackages) setuptools; 27 | 28 | } 29 | -------------------------------------------------------------------------------- /nix/python-packages.nix: -------------------------------------------------------------------------------- 1 | # Generated by pip2nix 0.8.0.dev1 2 | # See https://github.com/nix-community/pip2nix 3 | 4 | { pkgs, fetchurl, fetchgit, fetchhg }: 5 | 6 | self: super: { 7 | "psycopg2" = super.buildPythonPackage rec { 8 | pname = "psycopg2"; 9 | version = "2.8.5"; 10 | src = fetchurl { 11 | url = "https://files.pythonhosted.org/packages/a8/8f/1c5690eebf148d1d1554fc00ccf9101e134636553dbb75bdfef4f85d7647/psycopg2-2.8.5.tar.gz"; 12 | sha256 = "06081jk9srkd4ra9j8b93x9ld3a2yxsbsf5bbbcivbm1yx065m7p"; 13 | }; 14 | format = "setuptools"; 15 | doCheck = false; 16 | buildInputs = []; 17 | checkInputs = []; 18 | nativeBuildInputs = []; 19 | propagatedBuildInputs = []; 20 | }; 21 | "tqdm" = super.buildPythonPackage rec { 22 | pname = "tqdm"; 23 | version = "4.45.0"; 24 | src = fetchurl { 25 | url = "https://files.pythonhosted.org/packages/4a/1c/6359be64e8301b84160f6f6f7936bbfaaa5e9a4eab6cbc681db07600b949/tqdm-4.45.0-py2.py3-none-any.whl"; 26 | sha256 = "155ghg31xd48civkw75xlqyq3ipkzb0w9gvm7mwfhdwsppb3z7pa"; 27 | }; 28 | format = "wheel"; 29 | doCheck = false; 30 | buildInputs = []; 31 | checkInputs = []; 32 | nativeBuildInputs = []; 33 | propagatedBuildInputs = []; 34 | }; 35 | } 36 | -------------------------------------------------------------------------------- /nix/sources.json: -------------------------------------------------------------------------------- 1 | { 2 | "gitignore": { 3 | "branch": "master", 4 | "description": "Nix function for filtering local git sources", 5 | "homepage": "", 6 | "owner": "hercules-ci", 7 | "repo": "gitignore", 8 | "rev": "2ced4519f865341adcb143c5d668f955a2cb997f", 9 | "sha256": "0fc5bgv9syfcblp23y05kkfnpgh3gssz6vn24frs8dzw39algk2z", 10 | "type": "tarball", 11 | "url": "https://github.com/hercules-ci/gitignore/archive/2ced4519f865341adcb143c5d668f955a2cb997f.tar.gz", 12 | "url_template": "https://github.com///archive/.tar.gz" 13 | }, 14 | "niv": { 15 | "branch": "master", 16 | "description": "Easy dependency management for Nix projects", 17 | "homepage": "https://github.com/nmattia/niv", 18 | "owner": "nmattia", 19 | "repo": "niv", 20 | "rev": "f73bf8d584148677b01859677a63191c31911eae", 21 | "sha256": "0jlmrx633jvqrqlyhlzpvdrnim128gc81q5psz2lpp2af8p8q9qs", 22 | "type": "tarball", 23 | "url": "https://github.com/nmattia/niv/archive/f73bf8d584148677b01859677a63191c31911eae.tar.gz", 24 | "url_template": "https://github.com///archive/.tar.gz" 25 | }, 26 | "nixpkgs": { 27 | "branch": "nixpkgs-unstable", 28 | "description": "A read-only mirror of NixOS/nixpkgs tracking the released channels. Send issues and PRs to", 29 | "homepage": "https://github.com/NixOS/nixpkgs", 30 | "owner": "NixOS", 31 | "repo": "nixpkgs-channels", 32 | "rev": "7c21abdf4ca3035b88e63a38e5620e56a5aeb9aa", 33 | "sha256": "008yvinqqlbw0zmn0wj028a4n58izgc0pm5nmfn7q9l4csxa6x0k", 34 | "type": "tarball", 35 | "url": "https://github.com/NixOS/nixpkgs-channels/archive/7c21abdf4ca3035b88e63a38e5620e56a5aeb9aa.tar.gz", 36 | "url_template": "https://github.com///archive/.tar.gz" 37 | }, 38 | "pip2nix": { 39 | "branch": "master", 40 | "description": "Freeze pip-installable packages into Nix expressions [maintainer=@datakurre]", 41 | "homepage": "", 42 | "owner": "nix-community", 43 | "repo": "pip2nix", 44 | "rev": "7557e61808bfb5724ccae035d38d385a3c8d4dba", 45 | "sha256": "0rwxkbih5ml2mgz6lx23p3jgb6v0wvslyvscki1vv4hl3pd6jcld", 46 | "type": "tarball", 47 | "url": "https://github.com/nix-community/pip2nix/archive/7557e61808bfb5724ccae035d38d385a3c8d4dba.tar.gz", 48 | "url_template": "https://github.com///archive/.tar.gz" 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /nix/sources.nix: -------------------------------------------------------------------------------- 1 | # This file has been generated by Niv. 2 | 3 | let 4 | 5 | # 6 | # The fetchers. fetch_ fetches specs of type . 7 | # 8 | 9 | fetch_file = pkgs: spec: 10 | if spec.builtin or true then 11 | builtins_fetchurl { inherit (spec) url sha256; } 12 | else 13 | pkgs.fetchurl { inherit (spec) url sha256; }; 14 | 15 | fetch_tarball = pkgs: name: spec: 16 | let 17 | ok = str: ! builtins.isNull (builtins.match "[a-zA-Z0-9+-._?=]" str); 18 | # sanitize the name, though nix will still fail if name starts with period 19 | name' = stringAsChars (x: if ! ok x then "-" else x) "${name}-src"; 20 | in 21 | if spec.builtin or true then 22 | builtins_fetchTarball { name = name'; inherit (spec) url sha256; } 23 | else 24 | pkgs.fetchzip { name = name'; inherit (spec) url sha256; }; 25 | 26 | fetch_git = spec: 27 | builtins.fetchGit { url = spec.repo; inherit (spec) rev ref; }; 28 | 29 | fetch_local = spec: spec.path; 30 | 31 | fetch_builtin-tarball = name: throw 32 | ''[${name}] The niv type "builtin-tarball" is deprecated. You should instead use `builtin = true`. 33 | $ niv modify ${name} -a type=tarball -a builtin=true''; 34 | 35 | fetch_builtin-url = name: throw 36 | ''[${name}] The niv type "builtin-url" will soon be deprecated. You should instead use `builtin = true`. 37 | $ niv modify ${name} -a type=file -a builtin=true''; 38 | 39 | # 40 | # Various helpers 41 | # 42 | 43 | # The set of packages used when specs are fetched using non-builtins. 44 | mkPkgs = sources: 45 | let 46 | sourcesNixpkgs = 47 | import (builtins_fetchTarball { inherit (sources.nixpkgs) url sha256; }) {}; 48 | hasNixpkgsPath = builtins.any (x: x.prefix == "nixpkgs") builtins.nixPath; 49 | hasThisAsNixpkgsPath = == ./.; 50 | in 51 | if builtins.hasAttr "nixpkgs" sources 52 | then sourcesNixpkgs 53 | else if hasNixpkgsPath && ! hasThisAsNixpkgsPath then 54 | import {} 55 | else 56 | abort 57 | '' 58 | Please specify either (through -I or NIX_PATH=nixpkgs=...) or 59 | add a package called "nixpkgs" to your sources.json. 60 | ''; 61 | 62 | # The actual fetching function. 63 | fetch = pkgs: name: spec: 64 | 65 | if ! builtins.hasAttr "type" spec then 66 | abort "ERROR: niv spec ${name} does not have a 'type' attribute" 67 | else if spec.type == "file" then fetch_file pkgs spec 68 | else if spec.type == "tarball" then fetch_tarball pkgs name spec 69 | else if spec.type == "git" then fetch_git spec 70 | else if spec.type == "local" then fetch_local spec 71 | else if spec.type == "builtin-tarball" then fetch_builtin-tarball name 72 | else if spec.type == "builtin-url" then fetch_builtin-url name 73 | else 74 | abort "ERROR: niv spec ${name} has unknown type ${builtins.toJSON spec.type}"; 75 | 76 | # Ports of functions for older nix versions 77 | 78 | # a Nix version of mapAttrs if the built-in doesn't exist 79 | mapAttrs = builtins.mapAttrs or ( 80 | f: set: with builtins; 81 | listToAttrs (map (attr: { name = attr; value = f attr set.${attr}; }) (attrNames set)) 82 | ); 83 | 84 | # https://github.com/NixOS/nixpkgs/blob/0258808f5744ca980b9a1f24fe0b1e6f0fecee9c/lib/lists.nix#L295 85 | range = first: last: if first > last then [] else builtins.genList (n: first + n) (last - first + 1); 86 | 87 | # https://github.com/NixOS/nixpkgs/blob/0258808f5744ca980b9a1f24fe0b1e6f0fecee9c/lib/strings.nix#L257 88 | stringToCharacters = s: map (p: builtins.substring p 1 s) (range 0 (builtins.stringLength s - 1)); 89 | 90 | # https://github.com/NixOS/nixpkgs/blob/0258808f5744ca980b9a1f24fe0b1e6f0fecee9c/lib/strings.nix#L269 91 | stringAsChars = f: s: concatStrings (map f (stringToCharacters s)); 92 | concatStrings = builtins.concatStringsSep ""; 93 | 94 | # fetchTarball version that is compatible between all the versions of Nix 95 | builtins_fetchTarball = { url, name, sha256 }@attrs: 96 | let 97 | inherit (builtins) lessThan nixVersion fetchTarball; 98 | in 99 | if lessThan nixVersion "1.12" then 100 | fetchTarball { inherit name url; } 101 | else 102 | fetchTarball attrs; 103 | 104 | # fetchurl version that is compatible between all the versions of Nix 105 | builtins_fetchurl = { url, sha256 }@attrs: 106 | let 107 | inherit (builtins) lessThan nixVersion fetchurl; 108 | in 109 | if lessThan nixVersion "1.12" then 110 | fetchurl { inherit url; } 111 | else 112 | fetchurl attrs; 113 | 114 | # Create the final "sources" from the config 115 | mkSources = config: 116 | mapAttrs ( 117 | name: spec: 118 | if builtins.hasAttr "outPath" spec 119 | then abort 120 | "The values in sources.json should not have an 'outPath' attribute" 121 | else 122 | spec // { outPath = fetch config.pkgs name spec; } 123 | ) config.sources; 124 | 125 | # The "config" used by the fetchers 126 | mkConfig = 127 | { sourcesFile ? ./sources.json 128 | , sources ? builtins.fromJSON (builtins.readFile sourcesFile) 129 | , pkgs ? mkPkgs sources 130 | }: rec { 131 | # The sources, i.e. the attribute set of spec name to spec 132 | inherit sources; 133 | 134 | # The "pkgs" (evaluated nixpkgs) to use for e.g. non-builtin fetchers 135 | inherit pkgs; 136 | }; 137 | in 138 | mkSources (mkConfig {}) // { __functor = _: settings: mkSources (mkConfig settings); } 139 | -------------------------------------------------------------------------------- /pip2nix.ini: -------------------------------------------------------------------------------- 1 | [pip2nix] 2 | requirements = -r ./requirements.txt 3 | output = ./nix/python-packages.nix 4 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # 2 | # This file is autogenerated by pip-compile 3 | # To update, run: 4 | # 5 | # pip-compile 6 | # 7 | psycopg2==2.8.5 # via hello (setup.py) 8 | tqdm==4.45.0 # via hello (setup.py) 9 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = hello 3 | version = 1.0 4 | description = My hello world app 5 | 6 | [options] 7 | py_modules = hello 8 | install_requires = 9 | tqdm ~= 4.45.0 10 | psycopg2 ~= 2.8.5 11 | 12 | [options.entry_points] 13 | console_scripts= 14 | hello = hello:main 15 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | from setuptools import setup 4 | 5 | setup() 6 | -------------------------------------------------------------------------------- /shell.nix: -------------------------------------------------------------------------------- 1 | let 2 | pkgs = import ./nix {}; 3 | app = import ./default.nix { inherit pkgs; }; 4 | in 5 | pkgs.mkShell { 6 | buildInputs = [ 7 | app 8 | pkgs.niv 9 | pkgs.pip2nix 10 | pkgs.python38Packages.pip-tools 11 | ]; 12 | } 13 | --------------------------------------------------------------------------------