├── .flake8 ├── .gitignore ├── LICENSE ├── README.md ├── build_order.d ├── 01-build.yaml ├── 10-arrow.yaml ├── 30-pydandic.yaml ├── 80-other.yaml ├── 80-skbeam.yaml ├── 85-nofreethread.yaml ├── 86-tiled.yaml ├── 87-qs.yaml └── 95-numba.yaml ├── build_py_env.xsh ├── build_stats.py ├── ensure_clones.xsh ├── extra_remotes.yaml ├── find_repos.xsh ├── make_bleeding.xsh ├── oci ├── 01-base.xsh ├── 02-deps.xsh └── 04-run_build.xsh ├── repo_report.xsh ├── setup_extra_remotes.xsh └── used_repos.yaml /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | exclude = 3 | .git, 4 | __pycache__, 5 | build, 6 | dist, 7 | max-line-length = 115 8 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | all_repos.yaml 2 | *~ 3 | logs/*.json 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2022 Thomas A Caswell 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this 7 | list of conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | 3. Neither the name of the copyright holder nor the names of its contributors 14 | may be used to endorse or promote products derived from this software without 15 | specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 18 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 19 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 20 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 21 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 22 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 23 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 24 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 25 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 26 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # 🏗️ Build the world 🌐 3 | 4 | This is a repository of Python and [xonsh](https://xon.sh) scripts that I 5 | use to build a the section Python / pydata / scipy stack that I use (and help 6 | maintain). 7 | 8 | ## Goals 9 | 10 | I have frequently been caught out by changes to my upstream dependencies 11 | breaking me. Sometimes the changes are things I just have to adapt to. But in 12 | other cases I have been told the changes were unintentional and if the impact 13 | had been known they would not have been released. Thus, I set out to try and 14 | find those issues as early as possible. 15 | 16 | The goals of this project are: 17 | 18 | 1. Build and make available in a venv the main branch of CPython. 19 | 2. Install the main / default branch of most of the Scientific Python ecosystem. 20 | 3. Incremental builds / ability to resume after debugging a package's install. 21 | 4. Be able to easily switch any package to a source install for development. 22 | 5. Be easy to re-build everything from scratch. 23 | 24 | 25 | ## Code quality 26 | 27 | This is 🚮 trash 🚮 code that as of the time of this writing has been used by 1 28 | (one) person on 3 (three) computers (but one of those is now de-commissioned). 29 | The unit testing is "can I rebuild the environment"; for a long while, the 30 | "continue" functionality was implemented by commenting out the already built 31 | projects... These tools have been slowly moving towards being proper CLI 32 | tools, but until then they get the job done. 33 | 34 | 35 | This code is offered in the fullest sense of "AS IS". However, I have been 36 | slowly adding quality of life features and am open to any suggestions and 37 | collaboration to improve it! 38 | 39 | ## Requirements 40 | 41 | I have not been carefully tracking what system dependencies these scripts rely 42 | on. At a minimum running these scripts will require: 43 | 44 | 1. xonsh 45 | 2. c, c++, rust, and fortran compilers 46 | 4. pyyaml 47 | 5. cmake 48 | 6. npm 49 | 7. git 50 | 8. find 51 | 9. make + autotools 52 | 10. libhdf5 + headers 53 | 11. all of the image libraries + headers supported by imagecodecs 54 | 12. meson 55 | 13. openblas 56 | 14. patchelf 57 | 15. a development version of librdkafka 58 | 59 | This has been run (mostly successfully) on: 60 | 61 | - an up-to-date [Arch Linux](https://archlinux.org/) machine with a fair number 62 | of AUR packages installed (mostly for imagecodecs). 63 | - an OSX 12.4 M1 machine with a fair number of brew packages installed. I have 64 | not gotten imagecodecs or cairocffi to build yet. 65 | - an up-to-date Fedora 41 machine with a few non-standard repositories (kafka 66 | not building) with a lot of -devel packages for imagecodecs 67 | - under Windows Subsystem for Linux (skipping all the kafka related packages and imagecodecs) 68 | 69 | 70 | ## Usage 71 | 72 | ### Set up source tree(s) 73 | 74 | To use project is currently a multi step process. The first step is to make 75 | sure all of the relevant projects are cloned locally. In principle there is 76 | enough information in `all_repos.yaml` and `build_order.yaml` to identify and 77 | clone any missing repositories. 78 | 79 | ```bash 80 | xonsh ensure_clones.xsh 81 | ``` 82 | 83 | will attempt clone most of the repositories. 84 | 85 | The second step is locate all of the checkouts. 86 | 87 | ```bash 88 | $ cd build_the_world 89 | $ xonsh find_repos.xsh path/to/source/directory 90 | ``` 91 | 92 | will find all of the git and hg checkouts under the given directory and will 93 | write out a file `all_repos.yaml` with information about all of the checkouts 94 | it found. While this is walking the repositories it will also change the url 95 | on any `git://` urls for fetch to `https://` as github has stopped supporting 96 | the unauthenticated git protocol for fetching repostiory data. 97 | 98 | A third step is move some projects to a particular tracking branch for bug-fixes or 99 | compatibility with non-released versions of Python: 100 | 101 | ```bash 102 | $ xonsh setup_extra_remotes.xsh # this will reset --hard to the tracking branch 103 | $ xonsh find_repos.xsh path/to/source/directory # make repo metadata is up-to-date 104 | ``` 105 | 106 | The non-standard branches are tracked in `extra_remotes.yaml`. 107 | 108 | ### Build everything 109 | 110 | Once all of the required repositories are checked out and found, run 111 | 112 | ```bash 113 | $ xonsh make_bleeding.xsh 114 | ``` 115 | 116 | which will start from CPython try to move to the tip of the current default 117 | branch and then build everything. If something goes wrong in the middle (which 118 | it often does), you can resolve the issue 119 | 120 | ```xonsh 121 | $ vox activate bleeding # or how ever you activate venvs in your shell 122 | $ # fix the problem 123 | $ xonsh build_py_env.xsh --continue 124 | $ # repeat as needed 125 | ``` 126 | 127 | To update the metadata about any non-default branches required to make the build run 128 | (or if any of the projects can move back to the default branch) run 129 | 130 | ```bash 131 | $ xonsh repo_reort.xsh 132 | ``` 133 | 134 | and commit the updated `extra_remotes.yaml` to the repo. 135 | 136 | Eventually you will have a virtual environment with the development branch of 137 | a large swath of the Scientific Python (and some web) ecosystem installed! 138 | 139 | ### Control CPython branch and free-threading 140 | 141 | If you want to build a different branch of CPython than `main`, you can use the 142 | `--branch` flag to select the branch and `--target` flag to control the venv name 143 | 144 | ```bash 145 | $ xonsh make_bleeding.xsh --branch=aardvark_special --target burrow 146 | ``` 147 | 148 | To build CPython 3.13 with free threading enabled: 149 | 150 | ```bash 151 | # xonsh make_bleeding.xsh --target py313t --branch 3.13 --freethread 152 | ``` 153 | 154 | 155 | To build CPython 3.13 with the jit enabled 156 | 157 | ```bash 158 | # xonsh make_bleeding.xsh --target py313 --branch 3.13 --jit 159 | ``` 160 | 161 | 162 | ### Start with existing CPython build 163 | 164 | If you only want to install the development versions of the downstream 165 | projects, but not CPython itself, you can do: 166 | 167 | ```xonsh 168 | $ python -m venv /tmp/clean 169 | $ vox activate /tmp/clean # or how ever you activate venvs in your shell 170 | $ xonsh build_py_env.xsh 171 | $ # fix as needed 172 | $ xonsh build_py_env.xsh --continue 173 | ``` 174 | 175 | ## Add new projects 176 | 177 | The build order is stored in the `build_order.d/*yaml` files which are run in 178 | alphabetic order in the order the projects are listed in the files. 179 | 180 | Each file is formatted as a list of dictionaries. 181 | 182 | ### Add new source install 183 | 184 | To add a new install from a git checkout, add a section like: 185 | 186 | ```yaml 187 | default_branch: main # the default branch for the project 188 | function: numpy_build # the function in build_py_env.xsh used to build it 189 | kind: source_install # must be "source_install" to do a source install 190 | name: numpy # name that shows up in the log 191 | proj_name: numpy # the name top level folder when cloned, matched against all_repos.yaml 192 | ``` 193 | 194 | to the correct place in one of the files in `build_order.d`. Most projects can 195 | use the function `main_build`, but some projects require custom steps to build 196 | (e.g. manually regenerating cython output, special steps to get the version 197 | right). 198 | 199 | ### Add package from pypi 200 | 201 | To add packages from pypi add an entry like: 202 | 203 | 204 | ```yaml 205 | flags: --pre --upgrade --no-build-isolation # flags to pass to the invocation of pip 206 | kind: pip # must be 'pip' 207 | packages: beniget gast ply # list of packages to install 208 | ``` 209 | 210 | The custom version of `pip` that is used (assuming you are using my branch) will output 211 | a copy-paste-able error message to add missing dependencies. 212 | 213 | For the build flags `--pre --upgrade` are to stay in the spirit of being on the 214 | bleeding edge without yet installing from git and are optional. 215 | 216 | The `--no-build-isolation` is required for almost all packages to avoid errors 217 | creating the captive virtual environment used when pip attempts isolated builds. 218 | 219 | ### Format yaml files 220 | 221 | You can use [`yq`](https://github.com/mikefarah/yq) to format the yaml. This 222 | takes care of wrapping long lines, sorting the keys and adding the start/send 223 | markup to the yaml. 224 | 225 | ```bash 226 | yq -iYS --explicit-start --explicit-end -w 100 . build_order.d/*.yaml 227 | ``` 228 | 229 | ## Containers 230 | 231 | There are scripts in `oci` to build [`buildah`](https://buildah.io) to build 232 | OCI images that run this build. It works up the first need for packages form 233 | AUR. 234 | 235 | This seemed like a good idea, but all of the checked out source results in a 236 | 9GB image. 237 | 238 | Eventually the goal is to get a version of these on a container registry and 239 | generate them on CI. This is part of this project I very much would like help 240 | with! 241 | 242 | ## FAQ 243 | 244 | 1. **Aren't you reinventing \?**: Yes, well no. While this 245 | code and packaging systems both build from source, a packaging system is 246 | trying to create distributable binary artifacts. This code is only trying 247 | to making the installs work on the machine the script is run on. I am 248 | solving a _much_ simpler problem than a packaging system. 249 | 250 | This code does implicitly rely on `pip`'s dependency resolution, but the 251 | build is ordered to be approximately right. 252 | 253 | 254 | 2. **What about [Spack](https://github.com/spack/spack)?** Spack has a 255 | "(re)build the world" approach, but keeps careful track of versions, 256 | provides tools to change versions and deterministically rebuild. This 257 | code's goal is to get a working environment with the development branch of a 258 | majority of the Scientific Python stack installed and working. Upgrading 259 | via this code is _very_ destructive: it deletes the old environment and 260 | replaces it! 261 | 262 | Again, I am solving a much simpler problem that Spack is trying to solve. 263 | 2. **Why xonsh?**: I wanted to learn xonsh and the shell/Python hybrid is 264 | really pleasant for this sort of work (even if I sometimes have to 265 | trial-and-error accessing variables between the two sides and with string escaping). 266 | 3. **Is this re-inventing pythonci?**: 267 | No. [pythonci](https://github.com/vstinner/pythonci/) is a Victor Stinner 268 | project with a more reasonable goal of building the stable release of 269 | projects against the default development branch of CPython. I am trying to 270 | build the development branch of 271 | everything. [pythonperformance](https://github.com/python/pyperformance) 272 | also tries to rebuild stable versions of the ecosystem on top of the 273 | development branch of CPython. 274 | 4. **Do you run the tests for everything?**: No, that would be interesting. I 275 | do regularly run the test suites of the projects I work on day-to-day 276 | (Matplotlib, h5py, and the [bluesky suite](https://blueskyproject.io)) which 277 | covers the parts of the upstream code _I_ care most about. 278 | 5. **Does this work on \?**: I have only ever run this on an 279 | up-to-date Arch Linux machine and an up-to-date OSX M1 machine, so I have no 280 | idea! Given the changes I had to make to get it to run on OSX, I would 281 | expect significant changes to work on Windows, but a fair chance of working 282 | on other *nix. 283 | 6. **Doesn't this break all the time?**: Yes. 284 | 7. **How long does this take to build?**: A while! It is about 15min with 285 | ccache (and 40min without) when it does not break. 286 | 8. **Could some of these build steps be done in parallel?**: Yes, but so far 287 | kicking this off and either doing other work or walking away has worked well 288 | enough for me. 289 | 9. **These answers all seem very selfish.**: That is more of a comment, but 290 | yes. This project is currently all about solving _my_ problems (and my own 291 | amusement). 292 | 10. **Do you actually want anyone else to use this project?**: Yes! That is why 293 | this is now coming off of my computer and out into the world. However, I 294 | am not sure if anyone else would _want_ to participate in this admittedly 295 | silly activity. I am being honest about my current ambitions for it and 296 | the history. If this seems interesting / fun / useful to you then lets be 297 | friends! 298 | 299 | ## History 300 | 301 | The first version of this was a bash script that complied CPython and created a 302 | virtual environment. Very quickly the script started to automate installing 303 | Python packages from source. The number of projects that I was installing from 304 | version control rapidly grew -- driven by needing to work around projects that 305 | ship pre-cythonized c files in their sdists, needing un-released bug-fixes to 306 | support the main branch of CPython, or just projects I was personally 307 | contributing to. Eventually a single `bash` script (without functions!) was 308 | refactored to a `bash` script _with_ functions, to a `xonsh` script with all of 309 | checkout locations hard-coded at the top of the file, to a handful of `xonsh` 310 | scripts and a `yaml` file to track where the checkouts are, to the current 311 | state of a handful of `xonsh` scripts and a small fleet of `yaml` files to 312 | track where the source is, what branches are being used, and the build order. 313 | -------------------------------------------------------------------------------- /build_order.d/01-build.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | default_branch: main 3 | function: flit_build 4 | kind: source_install 5 | name: flit 6 | proj_name: flit 7 | ... 8 | --- 9 | flags: --pre --upgrade --no-build-isolation 10 | kind: pip 11 | packages: setuptools 12 | ... 13 | --- 14 | default_branch: main 15 | function: main_build 16 | kind: source_install 17 | name: setuptools 18 | proj_name: setuptools 19 | ... 20 | --- 21 | default_branch: main 22 | function: main_build 23 | kind: source_install 24 | name: wheel 25 | proj_name: wheel 26 | ... 27 | --- 28 | default_branch: main 29 | function: main_build 30 | kind: source_install 31 | name: pip_src 32 | proj_name: pip 33 | ... 34 | --- 35 | default_branch: main 36 | function: main_build 37 | kind: source_install 38 | name: Pillow 39 | proj_name: Pillow 40 | ... 41 | --- 42 | flags: --pre --upgrade --no-build-isolation 43 | kind: pip 44 | packages: mccabe pycodestyle pyflakes 45 | ... 46 | --- 47 | default_branch: main 48 | function: main_build 49 | kind: source_install 50 | name: flake8 51 | proj_name: flake8 52 | ... 53 | --- 54 | flags: --pre --upgrade --no-build-isolation 55 | kind: pip 56 | packages: flake8-force 57 | ... 58 | --- 59 | flags: --pre --upgrade --no-build-isolation 60 | kind: pip 61 | packages: pathspec scikit-build-core 62 | ... 63 | --- 64 | default_branch: master 65 | function: main_build 66 | kind: source_install 67 | name: pybind11 68 | proj_name: pybind11 69 | ... 70 | --- 71 | flags: --pre --upgrade --no-build-isolation 72 | kind: pip 73 | packages: cmake ninja 74 | ... 75 | --- 76 | default_branch: master 77 | function: cython_build 78 | kind: source_install 79 | name: cython 80 | proj_name: cython 81 | ... 82 | --- 83 | flags: --pre --upgrade --no-build-isolation 84 | kind: pip 85 | packages: packaging pyproject-hooks 86 | ... 87 | --- 88 | default_branch: main 89 | function: main_build 90 | kind: source_install 91 | name: build 92 | proj_name: build 93 | ... 94 | --- 95 | flags: --no-build-isolation --pre --upgrade 96 | kind: pip 97 | packages: ninja pyparsing pyproject-metadata 98 | ... 99 | --- 100 | default_branch: master 101 | function: main_build 102 | kind: source_install 103 | name: meson 104 | proj_name: meson 105 | ... 106 | --- 107 | default_branch: main 108 | function: main_build 109 | kind: source_install 110 | name: meson-python 111 | proj_name: meson-python 112 | ... 113 | --- 114 | default_branch: main 115 | function: numpy_build 116 | kind: source_install 117 | name: numpy 118 | proj_name: numpy 119 | ... 120 | --- 121 | flags: --pre --upgrade --no-build-isolation 122 | kind: pip 123 | packages: beniget gast ply 124 | ... 125 | --- 126 | default_branch: master 127 | function: main_build 128 | kind: source_install 129 | name: pythran 130 | proj_name: pythran 131 | ... 132 | --- 133 | default_branch: main 134 | function: scipy_build 135 | kind: source_install 136 | name: scipy 137 | proj_name: scipy 138 | ... 139 | --- 140 | flags: --pre --upgrade --no-build-isolation 141 | kind: pip 142 | packages: pkgconfig six 143 | ... 144 | --- 145 | default_branch: master 146 | function: main_build 147 | kind: source_install 148 | name: h5py 149 | proj_name: h5py 150 | ... 151 | --- 152 | flags: --pre --upgrade --no-build-isolation 153 | kind: pip 154 | packages: ptyprocess 155 | ... 156 | --- 157 | default_branch: master 158 | function: main_build 159 | kind: source_install 160 | name: pexpect 161 | proj_name: pexpect 162 | ... 163 | --- 164 | default_branch: master 165 | function: main_build 166 | kind: source_install 167 | name: parso 168 | proj_name: parso 169 | ... 170 | --- 171 | flags: --pre --upgrade --no-build-isolation 172 | kind: pip 173 | packages: setuptools_scm[toml] typing-extensions 174 | ... 175 | --- 176 | flags: --pre --upgrade --no-build-isolation 177 | kind: pip 178 | packages: scikit-build distro 179 | ... 180 | --- 181 | default_branch: main 182 | function: main_build 183 | kind: source_install 184 | name: pyzmq 185 | proj_name: pyzmq 186 | ... 187 | --- 188 | default_branch: master 189 | function: main_build 190 | kind: source_install 191 | name: jedi 192 | proj_name: jedi 193 | ... 194 | --- 195 | flags: --pre --upgrade --no-build-isolation 196 | kind: pip 197 | packages: asttokens decorator executing matplotlib-inline prompt-toolkit pure-eval pygments stack-data 198 | traitlets wcwidth 199 | ... 200 | --- 201 | default_branch: master 202 | function: main_build 203 | kind: source_install 204 | name: executing 205 | proj_name: executing 206 | ... 207 | --- 208 | flags: --pre --upgrade --no-build-isolation 209 | kind: pip 210 | packages: ipython-pygments-lexers 211 | ... 212 | --- 213 | default_branch: main 214 | function: main_build 215 | kind: source_install 216 | name: IPython 217 | proj_name: ipython 218 | ... 219 | --- 220 | flags: --pre --upgrade --no-build-isolation 221 | kind: pip 222 | packages: pickleshare 223 | ... 224 | --- 225 | flags: --pre --upgrade --no-build-isolation 226 | kind: pip 227 | packages: hatchling hatch-nodejs-version editables pluggy trove-classifiers pathspec 228 | ... 229 | --- 230 | flags: --no-build-isolation --pre --upgrade 231 | kind: pip 232 | packages: maturin 233 | ... 234 | --- 235 | flags: --pre --upgrade --no-build-isolation 236 | kind: pip 237 | packages: jsonschema fastjsonschema jupyter-core attrs jsonschema-specifications referencing rpds-py platformdirs 238 | ... 239 | --- 240 | default_branch: main 241 | function: main_build 242 | kind: source_install 243 | name: nbformat 244 | proj_name: nbformat 245 | ... 246 | --- 247 | default_branch: master 248 | function: main_build 249 | kind: source_install 250 | name: tornado 251 | proj_name: tornado 252 | ... 253 | --- 254 | flags: --pre --upgrade --no-build-isolation 255 | kind: pip 256 | packages: bleach jinja2 markupsafe mistune nbclient pandocfilters beautifulsoup4 defusedxml jupyterlab-pygments 257 | tinycss2 jupyter-client soupsieve webencodings python-dateutil tornado 258 | ... 259 | --- 260 | flags: --pre --upgrade --no-build-isolation 261 | kind: pip 262 | packages: tinycss2<1.3 263 | ... 264 | --- 265 | default_branch: main 266 | function: main_build 267 | kind: source_install 268 | name: nbconvert 269 | proj_name: nbconvert 270 | ... 271 | --- 272 | default_branch: master 273 | function: main_build 274 | kind: source_install 275 | name: certifi 276 | proj_name: python-certifi 277 | ... 278 | --- 279 | flags: --no-build-isolation --pre --upgrade 280 | kind: pip 281 | packages: cppy 282 | ... 283 | --- 284 | default_branch: main 285 | function: main_build 286 | kind: source_install 287 | name: contourpy 288 | proj_name: contourpy 289 | ... 290 | --- 291 | default_branch: master 292 | function: main_build 293 | kind: source_install 294 | name: dateutil 295 | proj_name: dateutil 296 | ... 297 | --- 298 | flags: --pre --upgrade --no-build-isolation 299 | kind: pip 300 | packages: fonttools kiwisolver 301 | ... 302 | --- 303 | default_branch: main 304 | function: main_build 305 | kind: source_install 306 | name: cycler 307 | proj_name: cycler 308 | ... 309 | --- 310 | default_branch: main 311 | function: main_build 312 | kind: source_install 313 | name: matplotlib 314 | proj_name: matplotlib 315 | ... 316 | --- 317 | default_branch: main 318 | function: main_build 319 | kind: source_install 320 | name: mpl-gui 321 | proj_name: mpl-gui 322 | ... 323 | --- 324 | flags: --pre --upgrade --no-build-isolation 325 | kind: pip 326 | packages: versioneer 327 | ... 328 | --- 329 | flags: --pre --upgrade --no-build-isolation 330 | kind: pip 331 | packages: pytz tzdata 332 | ... 333 | --- 334 | default_branch: main 335 | function: pandas_build 336 | kind: source_install 337 | name: pandas 338 | proj_name: pandas 339 | ... 340 | --- 341 | default_branch: master 342 | function: main_build 343 | kind: source_install 344 | name: mplfinance 345 | proj_name: mplfinance 346 | ... 347 | --- 348 | default_branch: main 349 | function: main_build 350 | kind: source_install 351 | name: networkx 352 | proj_name: networkx 353 | ... 354 | --- 355 | default_branch: master 356 | function: main_build 357 | kind: source_install 358 | name: toolz 359 | proj_name: toolz 360 | ... 361 | --- 362 | default_branch: main 363 | function: main_build 364 | kind: source_install 365 | name: pyyaml 366 | proj_name: pyyaml 367 | ... 368 | --- 369 | default_branch: master 370 | function: main_build 371 | kind: source_install 372 | name: lxml 373 | proj_name: lxml 374 | ... 375 | --- 376 | flags: --pre --upgrade --no-build-isolation 377 | kind: pip 378 | packages: pygraphviz pydot 379 | ... 380 | --- 381 | flags: --pre --upgrade --no-build-isolation 382 | kind: pip 383 | packages: pre-commit cfgv identify nodeenv virtualenv distlib filelock 384 | ... 385 | --- 386 | flags: --pre --upgrade --no-build-isolation 387 | kind: pip 388 | packages: iniconfig 389 | ... 390 | --- 391 | default_branch: main 392 | function: main_build 393 | kind: source_install 394 | name: pytest 395 | proj_name: pytest 396 | ... 397 | --- 398 | flags: --pre --upgrade --no-build-isolation 399 | kind: pip 400 | packages: alabaster babel imagesize roman-numerals-py snowballstemmer sphinx sphinxcontrib-applehelp sphinxcontrib-devhelp sphinxcontrib-htmlhelp sphinxcontrib-jsmath sphinxcontrib-qthelp sphinxcontrib-serializinghtml sphinxcontrib-video 401 | ... 402 | --- 403 | flags: --pre --upgrade --no-build-isolation 404 | kind: pip 405 | packages: roman-numerals-py 406 | ... 407 | --- 408 | default_branch: master 409 | function: main_build 410 | kind: source_install 411 | name: sphinx 412 | proj_name: sphinx 413 | ... 414 | --- 415 | flags: --pre --upgrade --no-build-isolation 416 | kind: pip 417 | packages: accessible-pygments docutils mpl-sphinx-theme numpydoc pydata-sphinx-theme sphinx-copybutton tabulate 418 | ... 419 | --- 420 | flags: --pre --upgrade --no-build-isolation 421 | kind: pip 422 | packages: pydocstyle flake8-docstrings pytest-cov pytest-rerunfailures pytest-timeout pytest-xdist pytest-xvfb 423 | execnet pyvirtualdisplay tomli coverage 424 | ... 425 | --- 426 | default_branch: main 427 | function: main_build 428 | kind: source_install 429 | name: sphinx-design 430 | proj_name: sphinx-design 431 | ... 432 | --- 433 | flags: --pre --upgrade --no-build-isolation 434 | kind: pip 435 | packages: colorspacious comm ipywidgets jupyterlab-widgets sphinx-gallery sphinx-tags sphinxcontrib-svg2pdfconverter 436 | widgetsnbextension 437 | ... 438 | --- 439 | flags: '' 440 | kind: pip-remove 441 | packages: flake8-per-file-ignores 442 | ... 443 | --- 444 | flags: --pre --upgrade --no-build-isolation 445 | kind: pip 446 | packages: jupyter-packaging hatch-jupyter-builder deprecation tomlkit 447 | ... 448 | --- 449 | flags: --pre --upgrade --no-build-isolation 450 | kind: pip 451 | packages: pycparser 452 | ... 453 | --- 454 | default_branch: main 455 | function: main_build 456 | kind: source_install 457 | name: cffi 458 | proj_name: cffi 459 | ... 460 | --- 461 | default_branch: main 462 | function: main_build 463 | kind: source_install 464 | name: argon2-cffi-bindings 465 | proj_name: argon2-cffi-bindings 466 | ... 467 | 468 | --- 469 | flags: --pre --upgrade --no-build-isolation 470 | kind: pip 471 | packages: anyio anyioutils argon2-cffi arrow async-lru debugpy fqdn h11 httpcore httpx ipykernel isoduration json5 jsonpointer jupyter-events jupyter-lsp jupyter-server jupyter-server-terminals jupyterlab-server nest-asyncio notebook-shim outcome overrides prometheus-client psutil python-json-logger rfc3339-validator rfc3986-validator send2trash sniffio terminado types-python-dateutil uri-template webcolors websocket-client zmq-anyio 472 | ... 473 | --- 474 | flags: --pre --upgrade --no-build-isolation 475 | kind: pip 476 | packages: cloudpickle fsspec importlib-metadata partd zipp locket click 477 | ... 478 | --- 479 | default_branch: main 480 | function: main_build 481 | kind: source_install 482 | name: dask 483 | proj_name: dask 484 | ... 485 | --- 486 | default_branch: main 487 | function: msgpack_build 488 | kind: source_install 489 | name: msgpack-python 490 | proj_name: msgpack-python 491 | ... 492 | --- 493 | flags: --pre --upgrade --no-build-isolation 494 | kind: pip 495 | packages: sortedcontainers tblib zict 496 | ... 497 | --- 498 | default_branch: main 499 | function: main_build 500 | kind: source_install 501 | name: distributed 502 | proj_name: distributed 503 | ... 504 | --- 505 | default_branch: main 506 | function: main_build 507 | kind: source_install 508 | name: pywt 509 | proj_name: pywt 510 | ... 511 | --- 512 | flags: --no-build-isolation --pre --upgrade 513 | kind: pip 514 | packages: lazy_loader 515 | ... 516 | --- 517 | default_branch: master 518 | function: main_build 519 | kind: source_install 520 | name: tifffile 521 | proj_name: tifffile 522 | ... 523 | --- 524 | flags: --pre --upgrade --no-build-isolation 525 | kind: pip 526 | packages: imageio 527 | ... 528 | --- 529 | default_branch: main 530 | function: main_build 531 | kind: source_install 532 | name: scikit-image 533 | proj_name: scikit-image 534 | ... 535 | --- 536 | default_branch: main 537 | function: main_build 538 | kind: source_install 539 | name: uritemplate 540 | proj_name: uritemplate 541 | ... 542 | --- 543 | default_branch: main 544 | function: main_build 545 | kind: source_install 546 | name: sip 547 | proj_name: sip 548 | ... 549 | --- 550 | flags: --pre --upgrade --no-build-isolation 551 | kind: pip 552 | packages: packaging pyqt-builder 553 | ... 554 | --- 555 | flags: --pre --upgrade --no-build-isolation 556 | kind: pip 557 | packages: asv asv-runner coverage jaraco-classes jaraco-context jaraco-functools jeepney markdown-it-py 558 | mdurl more-itertools nh3 outcome pkginfo py-cpuinfo pympler pytest-benchmark qtpy readme-renderer requests-toolbelt 559 | rfc3986 rich trio 560 | ... 561 | --- 562 | default_branch: master 563 | function: main_build 564 | kind: source_install 565 | name: multidict 566 | proj_name: multidict 567 | ... 568 | --- 569 | flags: --pre --upgrade 570 | kind: pip 571 | packages: expandvars 572 | ... 573 | --- 574 | flags: --pre --upgrade --no-build-isolation 575 | kind: pip 576 | packages: propcache 577 | ... 578 | --- 579 | default_branch: master 580 | function: build_yarl 581 | kind: source_install 582 | kwargs: 583 | name: yarl 584 | name: yarl 585 | proj_name: yarl 586 | ... 587 | --- 588 | default_branch: master 589 | function: build_yarl 590 | kind: source_install 591 | kwargs: 592 | name: frozenlist 593 | name: frozenlist 594 | proj_name: frozenlist 595 | ... 596 | --- 597 | flags: --pre --upgrade --no-build-isolation 598 | kind: pip 599 | packages: aiosignal aiohappyeyeballs 600 | ... 601 | --- 602 | default_branch: master 603 | function: build_aiohttp 604 | kind: source_install 605 | name: aiohttp 606 | proj_name: aiohttp 607 | ... 608 | --- 609 | default_branch: master 610 | function: main_build 611 | kind: source_install 612 | name: cytoolz 613 | proj_name: cytoolz 614 | ... 615 | --- 616 | flags: --pre --upgrade --no-build-isolation 617 | kind: pip 618 | packages: colorlog click mypy-extensions packaging pathspec platformdirs black aiosignal attrs 619 | ... 620 | --- 621 | flags: --pre --upgrade --no-build-isolation 622 | kind: pip 623 | packages: appdirs entrypoints 624 | ... 625 | --- 626 | default_branch: master 627 | function: main_build 628 | kind: source_install 629 | name: intake 630 | proj_name: intake 631 | ... 632 | --- 633 | default_branch: main 634 | function: main_build 635 | kind: source_install 636 | name: historydict 637 | proj_name: historydict 638 | ... 639 | --- 640 | flags: --pre --upgrade --no-build-isolation 641 | kind: pip 642 | packages: deprecated wrapt 643 | ... 644 | --- 645 | default_branch: main 646 | function: numcodecs_build 647 | kind: source_install 648 | name: numcodecs 649 | proj_name: numcodecs 650 | ... 651 | --- 652 | flags: --upgrade --no-build-isolation --pre --no-binary cftime 653 | kind: pip 654 | packages: cftime 655 | ... 656 | --- 657 | default_branch: master 658 | function: main_build 659 | kind: source_install 660 | name: pycurl 661 | proj_name: pycurl 662 | ... 663 | --- 664 | flags: --pre --upgrade --no-build-isolation 665 | kind: pip 666 | packages: flexcache hatch-vcs 667 | ... 668 | --- 669 | default_branch: main 670 | function: main_build 671 | kind: source_install 672 | name: flexparser 673 | proj_name: flexparser 674 | ... 675 | --- 676 | default_branch: master 677 | function: main_build 678 | kind: source_install 679 | name: pint 680 | proj_name: pint 681 | ... 682 | --- 683 | default_branch: develop 684 | function: main_build 685 | kind: source_install 686 | name: wrapt 687 | proj_name: wrapt 688 | ... 689 | --- 690 | flags: --pre --upgrade --no-build-isolation 691 | kind: pip 692 | packages: deprecated 693 | ... 694 | --- 695 | flags: --pre --upgrade --no-build-isolation 696 | kind: pip 697 | packages: importlib-metadata 698 | ... 699 | --- 700 | default_branch: main 701 | function: opentelemetry_build 702 | kind: source_install 703 | name: opentelemetry-api 704 | proj_name: opentelemetry-python 705 | ... 706 | --- 707 | default_branch: master 708 | function: main_build 709 | kind: source_install 710 | name: ophyd 711 | proj_name: ophyd 712 | ... 713 | --- 714 | flags: --pre --upgrade --no-build-isolation 715 | kind: pip 716 | packages: importlib-resources 717 | ... 718 | --- 719 | default_branch: main 720 | function: main_build 721 | kind: source_install 722 | name: event-model 723 | proj_name: event-model 724 | ... 725 | --- 726 | default_branch: master 727 | function: main_build 728 | kind: source_install 729 | name: area-detector-handlers 730 | proj_name: area-detector-handlers 731 | ... 732 | --- 733 | flags: --pre --upgrade --no-build-isolation 734 | kind: pip 735 | packages: tqdm msgpack-numpy heapdict 736 | ... 737 | --- 738 | default_branch: main 739 | function: main_build 740 | kind: source_install 741 | name: bluesky 742 | proj_name: bluesky 743 | ... 744 | --- 745 | default_branch: main 746 | function: main_build 747 | kind: source_install 748 | name: python-blosc 749 | proj_name: python-blosc 750 | ... 751 | --- 752 | flags: --pre --upgrade --no-build-isolation 753 | kind: pip 754 | packages: toml 755 | ... 756 | --- 757 | flags: --pre --upgrade --no-build-isolation 758 | kind: pip 759 | packages: aiofiles aiosqlite anyio asciitree asgi-correlation-id canonicaljson crc32c donfig fasteners 760 | h5netcdf lz4 mongomock mongoquery ndindex parquet sentinels starlette thriftpy2 watchgod websockets 761 | ... 762 | --- 763 | flags: --pre --upgrade --no-build-isolation 764 | kind: pip 765 | packages: frozendict 766 | ... 767 | --- 768 | default_branch: main 769 | function: main_build 770 | kind: source_install 771 | name: bluesky-darkframes 772 | proj_name: bluesky-darkframes 773 | ... 774 | --- 775 | flags: --pre --upgrade --no-build-isolation 776 | kind: pip 777 | packages: openpyxl xlrd et-xmlfile 778 | ... 779 | --- 780 | default_branch: master 781 | function: main_build 782 | kind: source_install 783 | name: bluesky-spreadsheet 784 | proj_name: bluesky-spreadsheet 785 | ... 786 | --- 787 | flags: --pre --upgrade --no-build-isolation 788 | kind: pip 789 | packages: aiohttp-jinja2 mily 790 | ... 791 | --- 792 | default_branch: main 793 | function: main_build 794 | kind: source_install 795 | name: bluesky-ui 796 | proj_name: bluesky-ui 797 | ... 798 | --- 799 | flags: --pre --upgrade --no-build-isolation 800 | kind: pip 801 | packages: coloredlogs prettytable simplejson humanfriendly 802 | ... 803 | --- 804 | default_branch: master 805 | function: main_build 806 | kind: source_install 807 | name: happi 808 | proj_name: happi 809 | ... 810 | --- 811 | default_branch: master 812 | function: main_build 813 | kind: source_install 814 | name: suitcase-utils 815 | proj_name: suitcase-utils 816 | ... 817 | --- 818 | default_branch: master 819 | function: main_build 820 | kind: source_install 821 | name: suitcase-csv 822 | proj_name: suitcase-csv 823 | ... 824 | --- 825 | default_branch: master 826 | function: main_build 827 | kind: source_install 828 | name: suitcase-dataexchange 829 | proj_name: suitcase-dataexchange 830 | ... 831 | --- 832 | default_branch: master 833 | function: main_build 834 | kind: source_install 835 | name: suitcase-jsonl 836 | proj_name: suitcase-jsonl 837 | ... 838 | --- 839 | default_branch: master 840 | function: main_build 841 | kind: source_install 842 | name: suitcase-json-metadata 843 | proj_name: suitcase-json-metadata 844 | ... 845 | --- 846 | flags: --pre --upgrade --no-build-isolation 847 | kind: pip 848 | packages: hatch-requirements-txt 849 | ... 850 | --- 851 | flags: --pre --upgrade --no-build-isolation 852 | kind: pip 853 | packages: pymongo dnspython 854 | ... 855 | --- 856 | default_branch: master 857 | function: main_build 858 | kind: source_install 859 | name: suitcase-mongo 860 | proj_name: suitcase-mongo 861 | ... 862 | --- 863 | default_branch: master 864 | function: main_build 865 | kind: source_install 866 | name: suitcase-msgpack 867 | proj_name: suitcase-msgpack 868 | ... 869 | --- 870 | default_branch: master 871 | function: main_build 872 | kind: source_install 873 | name: suitcase-nxstxm 874 | proj_name: suitcase-nxstxm 875 | ... 876 | --- 877 | flags: --pre --upgrade --no-build-isolation 878 | kind: pip 879 | packages: sphinxcontrib-jquery 880 | ... 881 | --- 882 | default_branch: master 883 | function: main_build 884 | kind: source_install 885 | name: sphinx_rtd_theme 886 | proj_name: sphinx_rtd_theme 887 | ... 888 | --- 889 | flags: --pre --upgrade --no-build-isolation 890 | kind: pip 891 | packages: codecov 892 | ... 893 | --- 894 | default_branch: master 895 | function: main_build 896 | kind: source_install 897 | name: suitcase-server 898 | proj_name: suitcase-server 899 | ... 900 | --- 901 | default_branch: master 902 | function: main_build 903 | kind: source_install 904 | name: suitcase-specfile 905 | proj_name: suitcase-specfile 906 | ... 907 | --- 908 | default_branch: main 909 | function: main_build 910 | kind: source_install 911 | name: suitcase-tiff 912 | proj_name: suitcase-tiff 913 | ... 914 | --- 915 | flags: --pre --upgrade --no-build-isolation 916 | kind: pip 917 | packages: annotated-types anyio async-timeout hiredis json-rpc jupyter-console pyjwt python-multipart redis 918 | ... 919 | --- 920 | flags: --pre --upgrade --no-build-isolation 921 | kind: pip 922 | packages: apischema pyresttable 923 | ... 924 | --- 925 | default_branch: main 926 | function: main_build 927 | kind: source_install 928 | name: hklpy 929 | proj_name: hklpy 930 | ... 931 | --- 932 | default_branch: main 933 | function: main_build 934 | kind: source_install 935 | name: nsls2-detector-handlers 936 | proj_name: nsls2-detector-handlers 937 | ... 938 | --- 939 | flags: --pre --upgrade --no-build-isolation 940 | kind: pip 941 | packages: colorama 942 | ... 943 | --- 944 | default_branch: master 945 | function: main_build 946 | kind: source_install 947 | name: pyqtgraph 948 | proj_name: pyqtgraph 949 | ... 950 | --- 951 | default_branch: master 952 | function: main_build 953 | kind: source_install 954 | name: curio 955 | proj_name: curio 956 | ... 957 | --- 958 | default_branch: main 959 | function: main_build 960 | kind: source_install 961 | name: caproto 962 | proj_name: caproto 963 | ... 964 | --- 965 | default_branch: master 966 | function: main_build 967 | kind: source_install 968 | name: bcsio 969 | proj_name: bcsio 970 | ... 971 | --- 972 | default_branch: master 973 | function: main_build 974 | kind: source_install 975 | name: mily 976 | proj_name: mily 977 | ... 978 | --- 979 | flags: --pre --upgrade --no-build-isolation 980 | kind: pip 981 | packages: hatch-fancy-pypi-readme 982 | ... 983 | --- 984 | default_branch: main 985 | function: awkward_build 986 | kind: source_install 987 | name: awkward 988 | proj_name: awkward 989 | ... 990 | --- 991 | flags: --pre --upgrade --no-build-isolation 992 | kind: pip 993 | packages: signalslot contexter 994 | ... 995 | --- 996 | default_branch: master 997 | function: main_build 998 | kind: source_install 999 | name: numexpr 1000 | proj_name: numexpr 1001 | ... 1002 | --- 1003 | flags: --pre --upgrade --no-build-isolation 1004 | kind: pip 1005 | packages: extension_helpers 1006 | ... 1007 | --- 1008 | default_branch: main 1009 | function: pyerfa_build 1010 | kind: source_install 1011 | name: pyerfa 1012 | proj_name: pyerfa 1013 | ... 1014 | --- 1015 | flags: --pre --upgrade --no-build-isolation 1016 | kind: pip 1017 | packages: astropy-iers-data 1018 | ... 1019 | --- 1020 | flags: --pre --upgrade --no-build-isolation 1021 | kind: pip 1022 | packages: slicerator 1023 | ... 1024 | --- 1025 | default_branch: main 1026 | function: main_build 1027 | kind: source_install 1028 | name: pims 1029 | proj_name: pims 1030 | ... 1031 | --- 1032 | flags: --pre --upgrade --no-build-isolation 1033 | kind: pip 1034 | packages: dpkt netifaces sphinxcontrib-jquery ujson 1035 | ... 1036 | --- 1037 | flags: --pre --upgrade --no-build-isolation 1038 | kind: pip 1039 | packages: binaryornot boltons chardet cookiecutter doct httpie humanize pysocks pytest-instafail python-slugify 1040 | requests text-unidecode tzlocal vcrpy 1041 | ... 1042 | --- 1043 | flags: --no-build-isolation --pre --upgrade 1044 | kind: pip 1045 | packages: xarray 1046 | ... 1047 | --- 1048 | flags: --pre --upgrade --no-build-isolation 1049 | kind: pip 1050 | packages: cachetools 1051 | ... 1052 | --- 1053 | default_branch: main 1054 | function: main_build 1055 | kind: source_install 1056 | name: data-prototype 1057 | proj_name: data-prototype 1058 | ... 1059 | --- 1060 | flags: --pre --upgrade --no-build-isolation 1061 | kind: pip 1062 | packages: mdit-py-plugins 1063 | ... 1064 | --- 1065 | default_branch: master 1066 | function: main_build 1067 | kind: source_install 1068 | name: MyST-Parser 1069 | proj_name: MyST-Parser 1070 | ... 1071 | --- 1072 | flags: --pre --upgrade --no-build-isolation 1073 | kind: pip 1074 | packages: jupyterlab-git jupyterlab-spellchecker mdit-py-plugins nbdime gitpython jupyter-server-mathjax 1075 | colorama gitdb smmap jupyterlab 1076 | ... 1077 | --- 1078 | flags: -y 1079 | kind: pip-remove 1080 | packages: pyepics 1081 | ... 1082 | --- 1083 | flags: --pre --upgrade --no-build-isolation 1084 | kind: pip 1085 | packages: xcffib pytest-tornasync 1086 | ... 1087 | --- 1088 | flags: --pre --upgrade --no-build-isolation --no-cache-dir 1089 | kind: pip 1090 | packages: cairocffi 1091 | ... 1092 | --- 1093 | flags: --no-build-isolation --pre --upgrade 1094 | kind: pip 1095 | packages: pycairo 1096 | ... 1097 | --- 1098 | default_branch: main 1099 | function: main_build 1100 | kind: source_install 1101 | name: mplcairo 1102 | proj_name: mplcairo 1103 | ... 1104 | --- 1105 | default_branch: main 1106 | function: watchfiles_build 1107 | kind: source_install 1108 | name: watchfiles 1109 | proj_name: watchfiles 1110 | ... 1111 | --- 1112 | flags: --pre --upgrade --no-build-isolation 1113 | kind: pip 1114 | packages: blinker feedgenerator markdown ordered-set pelican[markdown] pygments pymdown-extensions unidecode 1115 | ... 1116 | --- 1117 | flags: --no-build-isolation --pre --upgrade 1118 | kind: pip 1119 | packages: mypy 1120 | ... 1121 | --- 1122 | flags: --pre --upgrade --no-build-isolation 1123 | kind: pip 1124 | packages: joblib threadpoolctl 1125 | ... 1126 | --- 1127 | default_branch: main 1128 | function: main_build 1129 | kind: source_install 1130 | name: scikit-learn 1131 | proj_name: scikit-learn 1132 | ... 1133 | --- 1134 | flags: --pre --upgrade --no-build-isolation 1135 | kind: pip 1136 | packages: mpmath 1137 | ... 1138 | --- 1139 | default_branch: master 1140 | function: main_build 1141 | kind: source_install 1142 | name: sympy 1143 | proj_name: sympy 1144 | ... 1145 | --- 1146 | default_branch: master 1147 | function: imagecodecs_build 1148 | kind: source_install 1149 | name: imagecodecs 1150 | proj_name: imagecodecs 1151 | ... 1152 | --- 1153 | flags: --pre --upgrade --no-build-isolation 1154 | kind: pip 1155 | packages: xkcdpass 1156 | ... 1157 | --- 1158 | flags: --pre --upgrade --no-build-isolation 1159 | kind: pip 1160 | packages: blosc2 1161 | ... 1162 | --- 1163 | default_branch: master 1164 | function: main_build 1165 | kind: source_install 1166 | name: pyopengl 1167 | proj_name: pyopengl 1168 | ... 1169 | --- 1170 | default_branch: master 1171 | function: pyopengl_accel_build 1172 | kind: source_install 1173 | name: pyopengl-accelerate 1174 | proj_name: pyopengl 1175 | ... 1176 | --- 1177 | flags: --pre --upgrade --no-build-isolation 1178 | kind: pip 1179 | packages: pytools nanobind siphash24 1180 | ... 1181 | --- 1182 | default_branch: main 1183 | function: main_build 1184 | kind: source_install 1185 | name: pyopencl 1186 | proj_name: pyopencl 1187 | ... 1188 | --- 1189 | default_branch: main 1190 | function: main_build 1191 | kind: source_install 1192 | name: multianalyzer 1193 | proj_name: multianalyzer 1194 | ... 1195 | -------------------------------------------------------------------------------- /build_order.d/10-arrow.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | default_branch: main 3 | function: build_pyarrow 4 | kind: source_install 5 | name: pyarrow 6 | proj_name: arrow 7 | ... 8 | -------------------------------------------------------------------------------- /build_order.d/30-pydandic.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | flags: --pre --upgrade --no-build-isolation 3 | kind: pip 4 | packages: cachey click jmespath pydantic pydantic-core pydantic-settings python-dotenv shellingham typer typing-inspection 5 | ... 6 | --- 7 | default_branch: master 8 | function: main_build 9 | kind: source_install 10 | name: greenlet 11 | proj_name: greenlet 12 | ... 13 | --- 14 | default_branch: master 15 | function: main_build 16 | kind: source_install 17 | name: uvloop 18 | proj_name: uvloop 19 | ... 20 | --- 21 | default_branch: master 22 | function: main_build 23 | kind: source_install 24 | name: httptools 25 | proj_name: httptools 26 | ... 27 | --- 28 | flags: --pre --upgrade --no-build-isolation 29 | kind: pip 30 | packages: dep-logic findpython hishel id installer pbs-installer pdm pdm-backend resolvelib socksio truststore unearth 31 | ... 32 | --- 33 | flags: --pre --upgrade --no-build-isolation 34 | kind: pip 35 | packages: email-validator uvicorn 36 | ... 37 | --- 38 | flags: --pre --upgrade --no-build-isolation --no-deps 39 | kind: pip 40 | packages: fastapi-cli 41 | ... 42 | --- 43 | default_branch: master 44 | function: main_build 45 | kind: source_install 46 | name: fastapi 47 | proj_name: fastapi 48 | ... 49 | --- 50 | flags: --pre --upgrade --no-build-isolation 51 | kind: pip 52 | packages: alembic ecdsa ldap3 mako pamela pyasn1 pycryptodomex python-jose rich-toolkit rsa sqlalchemy 53 | ... 54 | --- 55 | flags: --pre --upgrade --no-build-isolation 56 | kind: pip 57 | packages: copier dunamai funcy jinja2-ansible-filters plumbum questionary typing-inspection copier-templates-extensions 58 | ... 59 | -------------------------------------------------------------------------------- /build_order.d/80-other.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | flags: --pre --upgrade --no-build-isolation 3 | kind: pip 4 | packages: json-merge-patch jsonpatch 5 | ... 6 | -------------------------------------------------------------------------------- /build_order.d/80-skbeam.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | flags: --pre --upgrade --no-build-isolation 3 | kind: pip 4 | packages: fabio hdf5plugin 5 | ... 6 | --- 7 | default_branch: main 8 | function: main_build 9 | kind: source_install 10 | name: silx 11 | proj_name: silx 12 | ... 13 | --- 14 | flags: --pre --upgrade --no-build-isolation 15 | kind: pip 16 | packages: asteval dill lmfit pyfai uncertainties 17 | ... 18 | --- 19 | default_branch: master 20 | function: main_build 21 | kind: source_install 22 | name: scikit-beam 23 | proj_name: scikit-beam 24 | ... 25 | -------------------------------------------------------------------------------- /build_order.d/85-nofreethread.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | default_branch: main 3 | function: main_build 4 | kind: source_install 5 | name: zstandard 6 | proj_name: python-zstandard 7 | ... 8 | --- 9 | flags: --pre --upgrade --no-build-isolation 10 | kind: pip 11 | packages: zarr 12 | ... 13 | --- 14 | default_branch: main 15 | function: main_build 16 | kind: source_install 17 | name: msgspec 18 | proj_name: msgspec 19 | ... 20 | --- 21 | flags: --pre --upgrade --no-build-isolation 22 | kind: pip 23 | packages: cryptography secretstorage keyring 24 | ... 25 | --- 26 | default_branch: main 27 | function: main_build 28 | kind: source_install 29 | name: twine 30 | proj_name: twine 31 | ... 32 | --- 33 | flags: --pre --upgrade --no-build-isolation 34 | kind: pip 35 | packages: PyQt6 pyqt6-qt6 pyqt6-sip 36 | ... 37 | --- 38 | default_branch: main 39 | function: main_build 40 | kind: source_install 41 | name: mpl-qtthread 42 | proj_name: mpl-qtthread 43 | ... 44 | -------------------------------------------------------------------------------- /build_order.d/86-tiled.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | default_branch: master 3 | function: main_build 4 | kind: source_install 5 | name: orjson 6 | proj_name: orjson 7 | ... 8 | --- 9 | flags: --pre --upgrade --no-build-isolation 10 | kind: pip 11 | packages: tiled 12 | ... 13 | --- 14 | flags: --pre --upgrade --no-build-isolation 15 | kind: pip 16 | packages: stamina tenacity 17 | ... 18 | --- 19 | default_branch: main 20 | function: main_build 21 | kind: source_install 22 | name: databroker 23 | proj_name: databroker 24 | ... 25 | --- 26 | default_branch: master 27 | function: main_build 28 | kind: source_install 29 | name: databroker-pack 30 | proj_name: databroker-pack 31 | ... 32 | -------------------------------------------------------------------------------- /build_order.d/87-qs.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | default_branch: master 3 | function: main_build 4 | kind: source_install 5 | name: confluent-kafka-python 6 | proj_name: confluent-kafka-python 7 | ... 8 | --- 9 | flags: --pre --upgrade --no-build-isolation 10 | kind: pip 11 | packages: bluesky-kafka 12 | ... 13 | --- 14 | default_branch: main 15 | function: main_build 16 | kind: source_install 17 | name: bluesky-queueserver 18 | proj_name: bluesky-queueserver 19 | ... 20 | --- 21 | default_branch: main 22 | function: main_build 23 | kind: source_install 24 | name: bluesky-queueserver-api 25 | proj_name: bluesky-queueserver-api 26 | ... 27 | --- 28 | default_branch: main 29 | function: main_build 30 | kind: source_install 31 | name: bluesky-httpserver 32 | proj_name: bluesky-httpserver 33 | ... 34 | -------------------------------------------------------------------------------- /build_order.d/95-numba.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | flags: --pre --upgrade --no-build-isolation 3 | kind: pip 4 | packages: asyncpg dask-expr llvmlite numba sparse 5 | ... 6 | --- 7 | default_branch: main 8 | function: main_build 9 | kind: source_install 10 | name: bluesky-live 11 | proj_name: bluesky-live 12 | ... 13 | --- 14 | default_branch: master 15 | function: main_build 16 | kind: source_install 17 | name: bluesky-widgets 18 | proj_name: bluesky-widgets 19 | ... 20 | --- 21 | flags: --pre --upgrade --no-build-isolation 22 | kind: pip 23 | packages: suitcase-utils 24 | ... 25 | --- 26 | default_branch: main 27 | function: main_build 28 | kind: source_install 29 | name: bluesky-adaptive 30 | proj_name: bluesky-adaptive 31 | ... 32 | -------------------------------------------------------------------------------- /build_py_env.xsh: -------------------------------------------------------------------------------- 1 | import json 2 | import sys 3 | 4 | import time 5 | import uuid 6 | import yaml 7 | import datetime 8 | from pathlib import Path 9 | 10 | from subprocess import CalledProcessError 11 | 12 | from xonsh.dirstack import with_pushd 13 | 14 | if "/usr/bin/vendor_perl" not in $PATH: 15 | $PATH.append("/usr/bin/vendor_perl") 16 | 17 | 18 | $RAISE_SUBPROC_ERROR = False 19 | $XONSH_TRACE_SUBPROC = True 20 | $PIP_NO_BUILD_ISOLATION = 1 21 | # YOLO don't check if it is a supported version of Python 22 | $UNSAFE_PYO3_SKIP_VERSION_CHECK = 1 23 | $UNSAFE_PYO3_BUILD_FREE_THREADED = 1 24 | 25 | $CXXFLAGS = ' '.join(('-fpermissive', ${...}.get('CXXFLAGS', ''))) 26 | 27 | MESON_LAPACK = '' 28 | 29 | if sys.platform == 'darwin': 30 | # make sure we find openblas from homebrew 31 | $LDFLAGS = ' '.join( 32 | ( 33 | "-L/opt/homebrew/opt/openblas/lib", 34 | "-L/opt/homebrew/Cellar/libxcb/1.15/lib/", 35 | '-L/opt/homebrew/Cellar/libyaml/0.2.5/lib/', 36 | '-L/opt/homebrew/Cellar/librdkafka/2.6.0/lib', 37 | '-L/opt/homebrew/Cellar/libxcb/1.15/lib', 38 | '-L/opt/homebrew/Cellar/graphviz/12.1.2/lib' 39 | ) 40 | ) 41 | $LD_LIBRARY_PATH = '/opt/homebrew/Cellar/libxcb/1.15/lib/' 42 | $CPPFLAGS = "-I/opt/homebrew/opt/openblas/include" 43 | $PKG_CONFIG_PATH = "/opt/homebrew/opt/openblas/lib/pkgconfig" 44 | $CFLAGS = ' '.join( 45 | ( 46 | '-I/opt/homebrew/Cellar/libyaml/0.2.5/include/', 47 | '-I/opt/homebrew/Cellar/graphviz/12.1.2/include', 48 | '-I/opt/homebrew/Cellar/librdkafka/2.6.0/include', 49 | '-I/opt/homebrew/Cellar/libxcb/1.15/include', 50 | ${...}.get('CFLAGS', '') 51 | ) 52 | ) 53 | $HDF5_DIR = '/opt/homebrew/Cellar/hdf5/1.14.4.3' 54 | # un-comment these to build freetype with CF compilers 55 | # del $host_alias 56 | # del $build_alias 57 | os = 'OSX' 58 | else: 59 | os_release = {} 60 | with open('/etc/os-release') as fin: 61 | for ln in fin: 62 | k, _, v = ln.strip().partition('=') 63 | os_release[k] = v 64 | if os_release['ID'] == 'fedora': 65 | MESON_LAPACK = '-Csetup-args=-Dblas=flexiblas -Csetup-args=-Dlapack=flexiblas' 66 | $CFLAGS=' '.join(['-DCYTHON_FAST_THREAD_STATE=0', ${...}.get('CFLAGS', '')]) 67 | 68 | 69 | build_order = [] 70 | for order in sorted(Path('build_order.d').glob('[!.]*yaml')): 71 | with open(order) as fin: 72 | build_order += list(yaml.unsafe_load_all(fin)) 73 | 74 | with open('all_repos.yaml') as fin: 75 | checkouts = list(yaml.unsafe_load_all(fin)) 76 | 77 | local_checkouts = {co['name']: co for co in checkouts} 78 | 79 | for step in build_order: 80 | if step['kind'] != 'source_install': 81 | continue 82 | lc = local_checkouts[step['proj_name']] 83 | # get the working directory 84 | step['wd'] = lc['local_checkout'] 85 | 86 | step.setdefault('kwargs', {}) 87 | step['kwargs']['upstream_branch'] = step['default_branch'] 88 | 89 | step['kwargs']['upstream_remote'] = next( 90 | n for n, r in lc['remotes'].items() if r['url'] == lc['primary_remote']['url'] 91 | ) 92 | 93 | 94 | def extract_git_shas(): 95 | headsha = $(git rev-parse HEAD).strip() 96 | describe = $(git describe --tags --abbrev=11 --long --dirty --always).strip() 97 | return {'head': headsha, 'describe': describe} 98 | 99 | 100 | def auto_main(upstream_remote='origin', upstream_branch='master'): 101 | with ${...}.swap(RAISE_SUBPROC_ERROR=True): 102 | git remote update 103 | with ${...}.swap(RAISE_SUBPROC_ERROR=False): 104 | tracking = !(git rev-parse --abbrev-ref --symbolic-full-name '@{u}') 105 | has_tracking = bool(tracking) 106 | tracking_branch = tracking.output.strip() 107 | with ${...}.swap(RAISE_SUBPROC_ERROR=True): 108 | if has_tracking: 109 | git merge @(tracking_branch) 110 | git submodule update 111 | cur_branch = $(git branch --show-current).strip() 112 | upstream = f'{upstream_remote}/{upstream_branch}' 113 | # git fetch @(upstream_remote) 114 | if cur_branch in {'main', 'master', 'develop', upstream_branch}: 115 | git merge @(upstream) 116 | else: 117 | with ${...}.swap(RAISE_SUBPROC_ERROR=False): 118 | is_merged = bool(!(git merge-base --is-ancestor HEAD @(upstream))) 119 | if is_merged: 120 | print('Done') 121 | git checkout @(upstream_branch) 122 | git merge @(upstream) 123 | if len(cur_branch): 124 | git branch -d @(cur_branch) 125 | else: 126 | if "GITHUB_USER" in ${...} and $GITHUB_USER in $(git remote -v): 127 | git push $GITHUB_USER 128 | 129 | 130 | def cleanup_cython(): 131 | try: 132 | to_remove = [_ for _ in $(ack -l "Generated by Cython" --type=cc --ignore-dir=docs).split("\n") if _.strip()] 133 | except CalledProcessError: 134 | to_remove = [] 135 | if to_remove: 136 | rm @(to_remove) 137 | 138 | 139 | def setuptools_build(**kwargs): 140 | auto_main(**kwargs) 141 | git clean -xfd 142 | cleanup_cython() 143 | python bootstrap.py 144 | return !(pip install --no-build-isolation .) 145 | 146 | def git_cleanup(): 147 | 148 | meson_packagecache = gp`subprojects/packagecache/*` 149 | print(meson_packagecache) 150 | for p in meson_packagecache: 151 | cp @(p) /tmp/@(p.name) 152 | 153 | git clean -xfd 154 | git submodule init 155 | git submodule update 156 | cleanup_cython() 157 | if meson_packagecache: 158 | mkdir -p subprojects/packagecache 159 | for p in meson_packagecache: 160 | cp /tmp/@(p.name) @(p) 161 | 162 | 163 | 164 | def flit_build(**kwargs): 165 | auto_main(**kwargs) 166 | git_cleanup() 167 | with with_pushd('flit_core'): 168 | whl = $(python -m flit_core.wheel).split('\n')[1].split(' ')[-1] 169 | python bootstrap_install.py @(whl) 170 | return !(pip install --no-build-isolation .) 171 | 172 | 173 | def main_build(**kwargs): 174 | auto_main(**kwargs) 175 | git_cleanup() 176 | return !(pip install --no-build-isolation .) 177 | 178 | def pyerfa_build(**kwargs): 179 | auto_main(**kwargs) 180 | git_cleanup() 181 | with with_pushd('liberfa/erfa'): 182 | git_cleanup() 183 | return !(pip install --no-build-isolation .) 184 | 185 | 186 | def pyopengl_accel_build(**kwargs): 187 | auto_main(**kwargs) 188 | git_cleanup() 189 | with with_pushd('accelerate/'): 190 | b = !(pip install --no-build-isolation .) 191 | return b 192 | 193 | 194 | def msgpack_build(**kwargs): 195 | auto_main(**kwargs) 196 | git_cleanup() 197 | !(make cython) 198 | return !(pip install --no-build-isolation .) 199 | 200 | 201 | def watchfiles_build(**kwargs): 202 | auto_main(**kwargs) 203 | git_cleanup() 204 | git checkout Cargo.lock Cargo.toml 205 | with ${...}.swap(VERSION='-'.join($(git describe --tags).strip()[1:].split('-')[:2])): 206 | python .github/set_version.py 207 | ret = !(pip install --no-build-isolation .) 208 | ret.returncode 209 | git checkout Cargo.lock Cargo.toml 210 | return ret 211 | 212 | 213 | def opentelemetry_build(**kwargs): 214 | auto_main(**kwargs) 215 | git_cleanup() 216 | return !(pip install --no-build-isolation ./opentelemetry-api) 217 | 218 | 219 | def setup_py_build(**kwargs): 220 | auto_main(**kwargs) 221 | git_cleanup() 222 | return !(python setup.py install) 223 | 224 | 225 | def numcodecs_build(**kwargs): 226 | auto_main(**kwargs) 227 | git clean -xfd 228 | git submodule init 229 | git submodule update 230 | cleanup_cython() 231 | rm -rf numcodecs/*.c 232 | with ${...}.swap(DISABLE_NUMCODECS_AVX2=1): 233 | return !(pip install --no-build-isolation -v .) 234 | 235 | 236 | def cython_build(**kwargs): 237 | auto_main(**kwargs) 238 | git clean -xfd 239 | git submodule update 240 | return !(pip install --upgrade --no-use-pep517 --no-build-isolation .) 241 | 242 | 243 | def awkward_build(**kwargs): 244 | auto_main(**kwargs) 245 | git clean -xfd 246 | git submodule init 247 | git submodule update 248 | cleanup_cython() 249 | # this is what nox -s prepare does but without needing nox 250 | python dev/copy-cpp-headers.py 251 | python dev/generate-kernel-signatures.py 252 | python dev/generate-tests.py 253 | python dev/generate-kernel-docs.py 254 | return !(pip install -v ./awkward-cpp --no-build-isolation) and !(pip install --no-build-isolation .) 255 | 256 | 257 | def numpy_build(**kwargs): 258 | auto_main(**kwargs) 259 | git clean -xfd 260 | cleanup_cython() 261 | git submodule update 262 | 263 | CFLAGS = (" -Wall -O2 -pipe -fomit-frame-pointer " 264 | "-fno-strict-aliasing " 265 | # "-Wmaybe-uninitialized " 266 | "-Wdeprecated-declarations -Wimplicit-function-declaration " 267 | "-march=native " 268 | "-DCYTHON_FAST_THREAD_STATE=0" 269 | ) 270 | # CFLAGS = "" 271 | print($CFLAGS) 272 | with ${...}.swap(CFLAGS=CFLAGS): 273 | ret = !(python -m build --no-isolation @(MESON_LAPACK.split()) .) 274 | if ret: 275 | wheel_file, = g`dist/*.whl` 276 | ret2 = !(pip install @(wheel_file)) 277 | if not ret2: 278 | return ret2 279 | 280 | return ret 281 | 282 | 283 | 284 | def scipy_build(**kwargs): 285 | auto_main(**kwargs) 286 | git clean -xfd 287 | git submodule update 288 | cleanup_cython() 289 | 290 | ret = !(python -m build --no-isolation --skip-dependency-check @(MESON_LAPACK.split()) .) 291 | if ret: 292 | wheel_file, = g`dist/*.whl` 293 | pip install @(wheel_file) 294 | return ret 295 | 296 | def pandas_build(**kwargs): 297 | auto_main(**kwargs) 298 | git clean -xfd 299 | cleanup_cython() 300 | git submodule update 301 | ret = !(pip install -v --no-build-isolation .) 302 | bool(ret) 303 | return ret 304 | 305 | def build_pyarrow(**kwargs): 306 | patch = """diff --git a/cpp/cmake_modules/Findlz4Alt.cmake b/cpp/cmake_modules/Findlz4Alt.cmake 307 | index 77a22957f..ce35d81eb 100644 308 | --- a/cpp/cmake_modules/Findlz4Alt.cmake 309 | +++ b/cpp/cmake_modules/Findlz4Alt.cmake 310 | @@ -33,6 +33,9 @@ if(lz4_FOUND) 311 | if(NOT TARGET LZ4::lz4 AND TARGET lz4::lz4) 312 | add_library(LZ4::lz4 ALIAS lz4::lz4) 313 | endif() 314 | + if(NOT TARGET LZ4::lz4 AND TARGET LZ4::lz4_shared) 315 | + add_library(LZ4::lz4 ALIAS LZ4::lz4_shared) 316 | + endif() 317 | return() 318 | endif() 319 | 320 | """ 321 | auto_main(**kwargs) 322 | git stash 323 | git clean -xfd 324 | cleanup_cython() 325 | git stash pop 326 | $PARQUET_TEST_DATA=$PWD+"/cpp/submodules/parquet-testing/data" 327 | $PARQUET_TEST_DATA 328 | $ARROW_TEST_DATA=$PWD+"/testing/data" 329 | cd ../ 330 | mkdir dist 331 | $ARROW_HOME=$PWD +'/dist' 332 | $LD_LIBRARY_PATH= [$PWD+"/dist/lib"] 333 | $CMAKE_PREFIX_PATH=[$ARROW_HOME] 334 | # patch from Arch packages 335 | # patch < lz4-cmake.patch -p 1 336 | cmake -S arrow/cpp -B arrow/cpp/build \ 337 | -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ 338 | --preset ninja-release-python 339 | cmake --build arrow/cpp/build --target install -j 340 | pushd arrow/python 341 | git clean -xfd 342 | $PYARROW_WITH_PARQUET=1 343 | $PYARROW_WITH_DATASET=1 344 | $PYARROW_PARALLEL=25 345 | return !(pip install -v . --no-build-isolation) 346 | 347 | def build_yarl(name, **kwargs): 348 | auto_main(**kwargs) 349 | git clean -xfd 350 | cleanup_cython() 351 | return !(pip install --no-build-isolation .) 352 | 353 | def build_aiohttp(**kwargs): 354 | # aiohttp has a makefile, but it pins to specific versions of cython which 355 | # defeats the point here! 356 | auto_main(**kwargs) 357 | git clean -xfd 358 | cleanup_cython() 359 | pushd vendor/llhttp/ 360 | npm install 361 | make generate 362 | popd 363 | python tools/gen.py 364 | cython -3 aiohttp/*.pyx 365 | cython -3 aiohttp/_websocket/*.pyx 366 | # because why not 367 | cython -3 aiohttp/_websocket/reader_c.py 368 | return !(pip install --no-build-isolation .) 369 | 370 | 371 | def imagecodecs_build(upstream_branch, **kwargs): 372 | # nuke the c files to force cython to run 373 | git remote update 374 | auto_main(**kwargs) 375 | git clean -xfd 376 | cleanup_cython() 377 | # rm imagecodecs/_*.c || true 378 | helper = """ 379 | def customize_build(EXTENSIONS, OPTIONS): 380 | EXTENSIONS['jpeg8']['sources'] = [] 381 | del EXTENSIONS['apng'] 382 | del EXTENSIONS['jetraw'] 383 | del EXTENSIONS['jpegxr'] 384 | del EXTENSIONS['jpegxs'] 385 | del EXTENSIONS['jpeg2k'] 386 | EXTENSIONS['lzham']['libraries'] = ['lzhamdll', 'lzhamcomp', 'lzhamdecomp'] 387 | del EXTENSIONS['lzo'] 388 | del EXTENSIONS['mozjpeg'] 389 | del EXTENSIONS['pcodec'] 390 | del EXTENSIONS['sperr'] 391 | del EXTENSIONS['sz3'] 392 | del EXTENSIONS['ultrahdr'] 393 | 394 | 395 | """ 396 | with open("imagecodecs_distributor_setup.py", 'w') as fout: 397 | fout.write(helper) 398 | print(helper) 399 | ret = !(python setup.py bdist_wheel) 400 | if ret: 401 | wheel_file, = g`dist/*.whl` 402 | ret2 = !(pip install @(wheel_file)) 403 | if not ret2: 404 | return ret2 405 | 406 | return ret 407 | 408 | 409 | 410 | 411 | def build_cffi(): 412 | hg purge 413 | hg pull 414 | hg update 415 | cleanup_cython() 416 | return !(pip install --no-use-pep517 --no-build-isolation .) 417 | 418 | 419 | def build_scipp(**kwargs): 420 | auto_main(**kwargs) 421 | git clean -xfd 422 | install_dir = $(python -c 'import site; print(site.getsitepackages()[0].strip(), end="")') 423 | py_ex = $(which python) 424 | cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DPYTHON_EXECUTABLE=@(py_ex) -DCMAKE_INSTALL_PREFIX=@(install_dir) -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF -DDYNAMIC_LIB=ON .. 425 | 426 | return !(cmake --build . --target install) 427 | 428 | def get_python_fingerprint(): 429 | script = """ 430 | import json 431 | import platform 432 | import sysconfig 433 | import sys 434 | 435 | json.dump( 436 | { 437 | "build": platform.python_build(), 438 | "compiler": platform.python_compiler(), 439 | "vars": sysconfig.get_config_vars(), 440 | "paths": sysconfig.get_paths(), 441 | }, 442 | sys.stdout, 443 | indent=2, 444 | ) 445 | 446 | """ 447 | return json.loads($(python -c @(script))) 448 | 449 | class JsonBuildRecord: 450 | @classmethod 451 | def start_record(cls, *, directory='logs'): 452 | p = Path(directory) 453 | p.mkdir(exist_ok=True, parents=True) 454 | uid = str(uuid.uuid4()) 455 | now = datetime.datetime.now() 456 | fname = f'{now:%Y%m%dT%H%M}.json' 457 | 458 | record = {'start_time': now.isoformat(), 459 | 'build_steps': [], 460 | 'uid': uid, 461 | **get_python_fingerprint() 462 | } 463 | with open(p / fname, 'w') as fin: 464 | json.dump(record, fin) 465 | 466 | return cls(p / fname) 467 | 468 | def __init__(self, fname): 469 | self._fname = Path(fname).absolute() 470 | with open(fname, 'r') as fin: 471 | self._data = json.load(fin) 472 | 473 | def add_build_record(self, build_data): 474 | self._data['build_steps'].append(build_data) 475 | with open(self._fname, 'w') as fin: 476 | json.dump(self._data, fin, indent=2) 477 | 478 | @property 479 | def steps(self): 480 | for record in self._data['build_steps']: 481 | yield record 482 | 483 | 484 | if '--continue' in sys.argv: 485 | p, *_ = sorted(Path('logs').glob('*.json'), reverse=True) 486 | record = JsonBuildRecord(p) 487 | else: 488 | record = JsonBuildRecord.start_record() 489 | 490 | add_build_record = record.add_build_record 491 | 492 | 493 | pre_done = set() 494 | for build_step in record.steps: 495 | if build_step['returncode'] == 0: 496 | if build_step['name'] == 'pip': 497 | pre_done.add(build_step['packages']) 498 | else: 499 | pre_done.add(build_step['name']) 500 | 501 | print(pre_done) 502 | for j, step in enumerate(build_order): 503 | print('*'*45) 504 | print(f'{" " + step.get("name", step.get("packages", "")) + " ":*^45s}') 505 | print('*'*45) 506 | if step.get('name', 'pip') == 'pip': 507 | if step['packages'] in pre_done: 508 | continue 509 | elif step['name'] in pre_done: 510 | continue 511 | 512 | if step['kind'] == 'source_install': 513 | echo @(step['wd']) 514 | with ${...}.swap(RAISE_SUBPROC_ERROR=True): 515 | pushd @(step['wd']) 516 | try: 517 | echo @(step['function']) 518 | func_name = step['function'] 519 | func = locals()[func_name] 520 | if step.get('vc', 'git') == 'git': 521 | shas = extract_git_shas() 522 | else: 523 | shas = {} 524 | start_time = time.time() 525 | kwargs = step.get('kwargs', {}) 526 | branch = step.get('project', {} 527 | ).get('primary_remote', {} 528 | ).get('default_branch', None) 529 | if branch is not None: 530 | kwargs.setdefault('upstream_branch', branch) 531 | build_log = func(**kwargs) 532 | succcesss = bool(build_log) 533 | stop_time = time.time() 534 | pl = json.loads($(pip list --format json)) 535 | 536 | add_build_record( 537 | { 538 | 'stdout': build_log.lines, 539 | 'stderr': build_log.errors, 540 | 'pip list': pl, 541 | 'start_time': start_time, 542 | 'stop_time': stop_time, 543 | 'shas': shas, 544 | 'step_inedx': j, 545 | 'name': step['name'], 546 | 'returncode': build_log.returncode 547 | } 548 | ) 549 | finally: 550 | popd 551 | 552 | if not build_log: 553 | print(''.join(build_log.lines)) 554 | print(''.join(build_log.errors) if build_log.errors is not None else None) 555 | print(f"pushd {step['wd']}") 556 | raise Exception 557 | 558 | elif step['kind'] == 'pip' or step['kind'] == 'pip-remove': 559 | if step['kind'] == 'pip': 560 | cmd = 'install' 561 | elif step['kind'] == 'pip-remove': 562 | cmd ='uninstall' 563 | else: 564 | raise Exception('womp womp') 565 | build_log = !(pip @(cmd) @(step['flags'].split()) @(step['packages'].split())) 566 | succcesss = bool(build_log) 567 | pl = json.loads($(pip list --format json)) 568 | add_build_record( 569 | { 570 | 'stdout': build_log.lines, 571 | 'stderr': build_log.errors, 572 | 'pip list': pl, 573 | 'start_time': build_log.starttime, 574 | 'stop_time': build_log.endtime, 575 | 'packages': step['packages'], 576 | 'flags': step['flags'], 577 | 'step_inedx': j, 578 | 'name': step['packages'], 579 | 'returncode': build_log.returncode 580 | } 581 | ) 582 | if not build_log: 583 | print(''.join(build_log.lines)) 584 | print(''.join(build_log.errors) if build_log.errors is not None else None) 585 | raise Exception 586 | else: 587 | raise Exception("womp womp") 588 | -------------------------------------------------------------------------------- /build_stats.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | from dataclasses import dataclass 3 | from collections import defaultdict 4 | import json 5 | from pathlib import Path 6 | 7 | 8 | @dataclass 9 | class BuildStepTiming: 10 | name: str 11 | date: datetime.datetime 12 | delta: float 13 | 14 | 15 | def find_and_load_logs(path="logs"): 16 | out = [] 17 | for f in sorted(Path("logs").glob("*.json"), reverse=True): 18 | with open(f) as fin: 19 | out.append(json.load(fin)) 20 | return out 21 | 22 | 23 | def package_build_times(logs, skip_pip=True): 24 | out = defaultdict(list) 25 | for log in logs: 26 | for step in log["build_steps"]: 27 | if "packages" in step and skip_pip: 28 | continue 29 | if step["returncode"] != 0: 30 | continue 31 | name = step["name"] 32 | out[name].append( 33 | BuildStepTiming( 34 | name, 35 | datetime.datetime.fromtimestamp(step["start_time"]), 36 | delta=step["stop_time"] - step["start_time"], 37 | ) 38 | ) 39 | return dict(out) 40 | 41 | 42 | def plot_trace(ax, times, package, **kwargs): 43 | return ax.plot( 44 | *zip(*[(_.date, _.delta) for _ in times[package]]), 45 | "o", 46 | **{"label": package, **kwargs}, 47 | ) 48 | 49 | 50 | def compare_build_times(ax, times, packages, **kwargs): 51 | ax.set_ylabel("build time [s]") 52 | ax.set_xlabel("date") 53 | for p in packages: 54 | plot_trace(ax, times, p, **kwargs) 55 | ax.legend() 56 | -------------------------------------------------------------------------------- /ensure_clones.xsh: -------------------------------------------------------------------------------- 1 | $RAISE_SUBPROC_ERROR = True 2 | 3 | from pathlib import Path 4 | import yaml 5 | 6 | import argparse 7 | 8 | 9 | parser = argparse.ArgumentParser(description='Clone all of the needed source repos.') 10 | parser.add_argument("--target", help="Path to clone into", type=str, default='~/source') 11 | parser.add_argument("--caswell", help="If Caswell's sorting should be applied", action='store_true') 12 | args = parser.parse_args() 13 | 14 | 15 | # Handle git 16 | 17 | 18 | 19 | 20 | bnl_orgs = { 21 | "networkx", 22 | "soft-matter", 23 | "zarr-developers", 24 | "pcdshub", 25 | "nsls-ii", 26 | "silx-kit", 27 | "dask", 28 | "bluesky", 29 | "h5py", 30 | "intake", 31 | "scikit-hep", 32 | "cgohlke", 33 | "nikea", 34 | "astropy", 35 | } 36 | 37 | def source_clone(target, org, repo, dest='other_source'): 38 | target = target / dest / org 39 | target.mkdir(parents=True, exist_ok=True) 40 | cd @(target) 41 | if not (target / repo).exists(): 42 | git clone --recursive @('https://github.com/' + '/'.join((org, repo))) 43 | else: 44 | cd @(target / repo) 45 | # git remote update 46 | 47 | target = Path(args.target).expanduser() 48 | 49 | 50 | with open('used_repos.yaml') as fin: 51 | required_repos = list(yaml.unsafe_load_all(fin)) 52 | 53 | print(set(b["user"] for b in required_repos)) 54 | 55 | for remote in required_repos: 56 | if not args.caswell: 57 | source_clone(target, remote['user'], remote['repo_name'], dest='') 58 | elif remote["user"].lower() in bnl_orgs: 59 | source_clone(target, remote['user'], remote['repo_name'], dest='bnl') 60 | else: 61 | source_clone(target, remote['user'] ,remote['repo_name'], dest='p') 62 | -------------------------------------------------------------------------------- /extra_remotes.yaml: -------------------------------------------------------------------------------- 1 | branch: nogil 2 | proj_name: argon2-cffi-bindings 3 | remote: 4 | host: github.com 5 | protocol: https 6 | repo_name: argon2-cffi-bindings 7 | ssh_user: null 8 | url: https://github.com/hynek/argon2-cffi-bindings.git 9 | user: hynek 10 | vc: git 11 | remote_name: origin 12 | --- 13 | branch: fix_py312 14 | proj_name: hklpy 15 | remote: 16 | host: github.com 17 | protocol: https 18 | repo_name: hklpy 19 | ssh_user: null 20 | url: https://github.com/tacaswell/hklpy.git 21 | user: tacaswell 22 | vc: git 23 | remote_name: tacaswell 24 | --- 25 | branch: fix/deep_copy_recurse 26 | proj_name: matplotlib 27 | remote: 28 | host: github.com 29 | protocol: https 30 | repo_name: matplotlib 31 | ssh_user: null 32 | url: https://github.com/tacaswell/matplotlib.git 33 | user: tacaswell 34 | vc: git 35 | remote_name: tacaswell 36 | --- 37 | branch: api/bump_python_support 38 | proj_name: mpl-gui 39 | remote: 40 | host: github.com 41 | protocol: https 42 | repo_name: mpl-gui 43 | ssh_user: null 44 | url: https://github.com/tacaswell/mpl-gui.git 45 | user: tacaswell 46 | vc: git 47 | remote_name: tacaswell 48 | --- 49 | branch: fix_new_implementation 50 | proj_name: mpl-qtthread 51 | remote: 52 | host: github.com 53 | protocol: https 54 | repo_name: mpl-qtthread 55 | ssh_user: null 56 | url: https://github.com/tacaswell/mpl-qtthread.git 57 | user: tacaswell 58 | vc: git 59 | remote_name: tacaswell 60 | --- 61 | branch: mnt/new_shape_convention 62 | proj_name: ophyd 63 | remote: 64 | host: github.com 65 | protocol: https 66 | repo_name: ophyd 67 | ssh_user: null 68 | url: https://github.com/tacaswell/ophyd.git 69 | user: tacaswell 70 | vc: git 71 | remote_name: tacaswell 72 | --- 73 | branch: enh/py314 74 | proj_name: parso 75 | remote: 76 | host: github.com 77 | protocol: https 78 | repo_name: parso 79 | ssh_user: null 80 | url: https://github.com/tacaswell/parso.git 81 | user: tacaswell 82 | vc: git 83 | remote_name: tacaswell 84 | --- 85 | branch: fix/av14 86 | proj_name: pims 87 | remote: 88 | host: github.com 89 | protocol: https 90 | repo_name: pims 91 | ssh_user: null 92 | url: https://github.com/tacaswell/pims.git 93 | user: tacaswell 94 | vc: git 95 | remote_name: tacaswell 96 | --- 97 | branch: hacky_hack 98 | proj_name: pip 99 | remote: 100 | host: github.com 101 | protocol: https 102 | repo_name: pip 103 | ssh_user: null 104 | url: https://github.com/tacaswell/pip.git 105 | user: tacaswell 106 | vc: git 107 | remote_name: tacaswell 108 | --- 109 | branch: cython-modernization 110 | proj_name: pyopengl 111 | remote: 112 | host: github.com 113 | protocol: https 114 | repo_name: pyopengl 115 | ssh_user: null 116 | url: https://github.com/tacaswell/pyopengl 117 | user: tacaswell 118 | vc: git 119 | remote_name: tacaswell 120 | --- 121 | branch: isra/fix-pathlib-with-ns 122 | proj_name: pytest 123 | remote: 124 | host: github.com 125 | protocol: https 126 | repo_name: pytest 127 | ssh_user: null 128 | url: https://github.com/flared/pytest.git 129 | user: flared 130 | vc: git 131 | remote_name: flared 132 | --- 133 | branch: patch-1 134 | proj_name: pyyaml 135 | remote: 136 | host: github.com 137 | protocol: https 138 | repo_name: pyyaml 139 | ssh_user: null 140 | url: https://github.com/henryiii/pyyaml.git 141 | user: henryiii 142 | vc: git 143 | remote_name: henryiii 144 | --- 145 | branch: mnt/np_and_cython_updates 146 | proj_name: scikit-beam 147 | remote: 148 | host: github.com 149 | protocol: https 150 | repo_name: scikit-beam 151 | ssh_user: null 152 | url: https://github.com/tacaswell/scikit-beam.git 153 | user: tacaswell 154 | vc: git 155 | remote_name: tacaswell 156 | --- 157 | branch: mnt/uncap_sphinx 158 | proj_name: sphinx-design 159 | remote: 160 | host: github.com 161 | protocol: https 162 | repo_name: sphinx-design 163 | ssh_user: null 164 | url: https://github.com/tacaswell/sphinx-design.git 165 | user: tacaswell 166 | vc: git 167 | remote_name: tacaswell 168 | -------------------------------------------------------------------------------- /find_repos.xsh: -------------------------------------------------------------------------------- 1 | import argparse 2 | from typing import Dict, Optional 3 | from collections import defaultdict 4 | from pathlib import Path 5 | import sys 6 | from dataclasses import dataclass, asdict 7 | 8 | import yaml 9 | 10 | from xonsh.dirstack import with_pushd 11 | from xonsh.tools import XonshCalledProcessError 12 | 13 | parser = argparse.ArgumentParser(description='Find source repos.') 14 | parser.add_argument("path", help='Top path to start searching for repos in.', type=Path) 15 | parser.add_argument("--update-used", help="If the used repo yaml should be updated.", action='store_true') 16 | args = parser.parse_args() 17 | 18 | path = args.path 19 | 20 | 21 | def find_git_repos(path): 22 | for candidate in $(find @(path) -type d -name '.git').split(): 23 | candidate = candidate.strip() 24 | if '.tox' in str(candidate): 25 | continue 26 | yield Path(candidate).resolve().parent 27 | 28 | 29 | def find_hg_repos(path): 30 | for candidate in !(find @(path) -type d -name '.hg'): 31 | candidate = candidate.strip() 32 | yield Path(candidate).resolve().parent 33 | 34 | def fix_git_protcol_to_https(repo): 35 | with with_pushd(repo): 36 | for ln in !(git remote -v).itercheck(): 37 | if not ln.strip(): 38 | continue 39 | name, url, _ = ln.split() 40 | if url.startswith('git://') and 'github.com' in url: 41 | print(url, '->', url.replace('git://', 'https://')) 42 | git remote set-url @(name) @(url.replace('git://', 'https://')) 43 | 44 | def get_git_remotes(repo): 45 | """Given a path to a repository, 46 | """ 47 | remotes = defaultdict(dict) 48 | 49 | with with_pushd(repo): 50 | for remote in !(git remote -v).itercheck(): 51 | name, _, rest = remote.strip().partition('\t') 52 | url, _, direction = rest.partition(' (') 53 | direction = direction[:-1] 54 | remotes[name][direction] = url 55 | 56 | return dict(remotes) 57 | 58 | def get_work_trees(repo): 59 | with with_pushd(repo): 60 | primary, *worktrees = [Path(_.split()[0]) for _ in !(git worktree list).itercheck()] 61 | 62 | return primary, worktrees 63 | 64 | def get_hg_remotes(repo): 65 | """Given a path to a repository, 66 | """ 67 | remotes = {} 68 | 69 | with with_pushd(repo): 70 | for remote in !(hg paths).itercheck(): 71 | name, _, url = [_.strip() for _ in remote.strip().partition('=')] 72 | remotes[name] = url 73 | 74 | return dict(remotes) 75 | 76 | 77 | @dataclass 78 | class Remote: 79 | url: str 80 | protocol: str 81 | host: str 82 | user: Optional[str] 83 | repo_name: Optional[str] 84 | ssh_user: Optional[str] = None 85 | vc: str = "git" 86 | 87 | 88 | @dataclass 89 | class Project: 90 | name: str 91 | primary_remote: Remote 92 | remotes: Dict[str, Remote] 93 | local_checkout: str 94 | 95 | 96 | def strip_dotgit(repo_name): 97 | return repo_name[:-4] if repo_name.endswith(".git") else repo_name 98 | 99 | 100 | def parse_git_name(git_url): 101 | if git_url.startswith("git@"): 102 | _, _, rest = git_url.partition("@") 103 | host, _, rest = rest.partition(":") 104 | parts = rest.split("/") 105 | if len(parts) == 1: 106 | user = None 107 | (repo_name,) = parts 108 | elif len(parts) == 2: 109 | user, repo_name = parts 110 | else: 111 | user, *rest = parts 112 | repo_name = "/".join(rest) 113 | return Remote( 114 | url=git_url, 115 | host=host, 116 | user=user, 117 | repo_name=strip_dotgit(repo_name), 118 | ssh_user="git", 119 | protocol="ssh", 120 | ) 121 | elif git_url.startswith("git://"): 122 | rest = git_url[len("git://") :] 123 | parts = rest.split("/") 124 | if len(parts) == 1: 125 | raise ValueError(f"this should not happen {git_url} {parts}") 126 | elif len(parts) == 2: 127 | host, repo_name = parts 128 | user = None 129 | elif len(parts) == 3: 130 | host, user, repo_name = parts 131 | else: 132 | host, user, *rest = parts 133 | repo_name = "/".join(rest) 134 | return Remote( 135 | url=f'git@{host}:{user}/{repo_name}', 136 | host=host, 137 | user=user, 138 | repo_name=strip_dotgit(repo_name), 139 | ssh_user='git', 140 | protocol="ssh", 141 | ) 142 | elif git_url.startswith("ssh://") or git_url.startswith("ssh+git://"): 143 | _, _, rest = git_url.partition(":") 144 | rest = rest[2:] 145 | if "@" in rest: 146 | ssh_user, _, rest = git_url.partition("@") 147 | else: 148 | ssh_user = ${'USER'} 149 | host, _, repo_name = rest.partition("/") 150 | return Remote( 151 | url=git_url, 152 | host=host, 153 | user=None, 154 | repo_name=strip_dotgit(repo_name), 155 | ssh_user=ssh_user, 156 | protocol="ssh", 157 | ) 158 | elif git_url.startswith("https://"): 159 | rest = git_url[len("https://") :] 160 | parts = rest.split("/") 161 | if len(parts) >= 3: 162 | host, user, *rest = parts 163 | repo_name = "/".join(rest) 164 | elif len(parts) == 2: 165 | host, repo_name = parts 166 | user = None 167 | elif len(parts) < 2: 168 | raise ValueError(f"do not think this can happen {git_url} {parts}") 169 | return Remote( 170 | url=git_url, 171 | host=host, 172 | user=user, 173 | repo_name=strip_dotgit(repo_name), 174 | protocol="https", 175 | ) 176 | return "http" 177 | elif "@" in git_url: 178 | ssh_user, _, rest = git_url.partition("@") 179 | host, _, rest = rest.partition(":") 180 | user, _, repo_name = rest.partition("/") 181 | return Remote( 182 | url=git_url, 183 | host=host, 184 | user=user, 185 | repo_name=strip_dotgit(repo_name), 186 | ssh_user=ssh_user, 187 | protocol="ssh", 188 | ) 189 | elif git_url.startswith("/") or git_url.startswith("."): 190 | return Remote( 191 | url=git_url, host="localhost", user=None, repo_name=None, protocol="file" 192 | ) 193 | elif ":" in git_url: 194 | host, _, repo_name = git_url.partition(":") 195 | ssh_user = ${'USER'} 196 | return Remote( 197 | url=git_url, 198 | host=host, 199 | user=None, 200 | repo_name=strip_dotgit(repo_name), 201 | ssh_user=ssh_user, 202 | protocol="ssh", 203 | ) 204 | else: 205 | raise ValueError(f"unknown scheme: {git_url}") 206 | 207 | 208 | def parse_hg_name(hg_url): 209 | if hg_url.startswith("http"): 210 | proto, _, rest = hg_url.partition("://") 211 | host, *parts = [_ for _ in rest.split("/") if len(_)] 212 | if parts[0] == "hg": 213 | parts = parts[1:] 214 | if len(parts) == 1: 215 | (repo_name,) = parts 216 | user = None 217 | elif len(parts) == 2: 218 | user, repo_name = parts 219 | else: 220 | user, *rest = parts 221 | repo_name = "/".join(parts) 222 | return Remote( 223 | url=hg_url, 224 | host=host, 225 | user=user, 226 | repo_name=repo_name, 227 | ssh_user=None, 228 | protocol=proto, 229 | vc="hg", 230 | ) 231 | elif hg_url.startswith("ssh"): 232 | proto, _, rest = hg_url.partition("://") 233 | if "@" in rest: 234 | ssh_user, _, rest = rest.partition("@") 235 | else: 236 | ssh_user = ${'USER'} 237 | host, *parts = [_ for _ in rest.split("/") if len(_)] 238 | if parts[0] == "hg": 239 | parts = parts[1:] 240 | if len(parts) == 1: 241 | (repo_name,) = parts 242 | user = None 243 | elif len(parts) == 2: 244 | user, repo_name = parts 245 | else: 246 | user, *rest = parts 247 | repo_name = "/".join(parts) 248 | return Remote( 249 | url=hg_url, 250 | host=host, 251 | user=user, 252 | repo_name=repo_name, 253 | ssh_user=ssh_user, 254 | protocol=proto, 255 | vc="hg", 256 | ) 257 | 258 | 259 | print(sys.version_info) 260 | 261 | projects = [] 262 | 263 | for repo_path in find_git_repos(path): 264 | print(repo_path) 265 | remotes = {} 266 | fix_git_protcol_to_https(repo_path) 267 | for k, v in get_git_remotes(repo_path).items(): 268 | print(f"\t{k}") 269 | parsed = {direction: parse_git_name(url) for direction, url in v.items()} 270 | assert len(parsed) == 2 271 | remotes[k] = parsed["fetch"] 272 | if not len(remotes): 273 | continue 274 | for k in ["upstream", "origin"]: 275 | if k in remotes: 276 | primary_remote = remotes[k] 277 | break 278 | else: 279 | primary_remote = next(iter(remotes.values())) 280 | if primary_remote is None: 281 | continue 282 | projects.append( 283 | Project( 284 | name=primary_remote.repo_name, 285 | primary_remote=primary_remote, 286 | remotes=remotes, 287 | local_checkout=str(repo_path), 288 | ) 289 | ) 290 | 291 | 292 | for repo_path in find_hg_repos(path): 293 | print(repo_path) 294 | remotes = {} 295 | for k, url in get_hg_remotes(repo_path).items(): 296 | print(f"\t{k}") 297 | remotes[k] = parse_hg_name(url) 298 | if not len(remotes): 299 | continue 300 | for k in ["origin", "default"]: 301 | if k in remotes: 302 | primary_remote = remotes[k] 303 | break 304 | else: 305 | primary_remote = next(iter(remotes.values())) 306 | if primary_remote is None: 307 | continue 308 | projects.append( 309 | Project( 310 | name=primary_remote.repo_name, 311 | primary_remote=primary_remote, 312 | remotes=remotes, 313 | local_checkout=str(repo_path), 314 | ) 315 | ) 316 | 317 | 318 | with open("all_repos.yaml", "w") as fout: 319 | yaml.dump_all([asdict(_) for _ in projects], fout) 320 | 321 | if args.update_used: 322 | local_checkouts = {co.name: co for co in projects} 323 | 324 | repos = [] 325 | 326 | for order in sorted(Path('build_order.d').glob('[!.]*yaml')): 327 | with open(order) as fin: 328 | build_order = list(yaml.unsafe_load_all(fin)) 329 | 330 | for step in build_order: 331 | if step['kind'] != 'source_install': 332 | continue 333 | lc = local_checkouts[step['proj_name']] 334 | repos.append(asdict(lc.primary_remote)) 335 | repos.append(asdict(local_checkouts['cpython'].primary_remote)) 336 | repos = filter(lambda x: x['vc'] == 'git', repos) 337 | with open("used_repos.yaml", "w") as fout: 338 | yaml.dump_all(sorted(repos, key=lambda x: (x['user'], x['repo_name'])), fout) 339 | -------------------------------------------------------------------------------- /make_bleeding.xsh: -------------------------------------------------------------------------------- 1 | #! /usr/env xonsh 2 | import sys 3 | import argparse 4 | from pathlib import Path 5 | 6 | import yaml 7 | 8 | 9 | from xonsh.dirstack import with_pushd 10 | 11 | xonsh_abs_path = str(Path(sys.executable).parent / 'xonsh') 12 | 13 | $RAISE_SUBPROC_ERROR = True 14 | 15 | $XONSH_TRACE_SUBPROC = True 16 | $PIP_NO_BUILD_ISOLATION = 1 17 | 18 | parser = argparse.ArgumentParser(description='Build the world.') 19 | parser.add_argument("--target", help="name of env to create", type=str, default='bleeding') 20 | parser.add_argument("--branch", help="CPython branch to build", type=str, default=None) 21 | parser.add_argument("--no-pull", help="Do not try to pull before building cpython (if on a branch)", action='store_true') 22 | parser.add_argument("--clang", help="Use clang", action='store_true') 23 | parser.add_argument("--debug", help="Debug build of CPython", action='store_true') 24 | parser.add_argument("--freethread", help="Try to use freethreading (no GIL)", action='store_true') 25 | parser.add_argument("--jit", help="Try to use the experimental jit", action='store_true') 26 | args = parser.parse_args() 27 | 28 | 29 | 30 | 31 | with open('all_repos.yaml') as fin: 32 | checkouts = list(yaml.unsafe_load_all(fin)) 33 | wd_mapping = {co['name']: co['local_checkout'] for co in checkouts} 34 | 35 | wd = wd_mapping['cpython'] 36 | 37 | prefix = ${'HOME'}+f"/.pybuild/{args.target}" 38 | prefix_as_path = Path(prefix) 39 | if prefix_as_path.exists(): 40 | rm -rf @(prefix) 41 | prefix_as_path.mkdir(parents=True) 42 | 43 | with with_pushd(wd): 44 | if args.branch is not None: 45 | git checkout @(args.branch) 46 | cur_branch = $(git branch --show-current).strip() 47 | if not args.no_pull and len(cur_branch): 48 | git pull 49 | git clean -xfd 50 | if args.clang: 51 | $CC = 'clang' 52 | $CXX = 'clang++' 53 | ./configure \ 54 | --prefix=@(prefix) \ 55 | --enable-shared LDFLAGS=@(f"-Wl,-rpath,$HOME/.pybuild/{args.target}/lib") \ 56 | --enable-optimizations \ 57 | --with-lto \ 58 | @('--with-pydebug' if args.debug else '') \ 59 | @('--disable-gil' if args.freethread else '') \ 60 | @('--enable-experimental-jit' if args.jit else '') 61 | make -j 62 | make install 63 | 64 | $HOME/.pybuild/@(args.target)/bin/python3 -m venv --copies --clear ~/.virtualenvs/@(args.target) 65 | # the build package seems to require this? 66 | ln $HOME/.pybuild/@(args.target)/bin/python3 $HOME/.pybuild/@(args.target)/bin/python 67 | 68 | 69 | source-bash f'~/.virtualenvs/{args.target}/bin/activate' 70 | 71 | pip install --upgrade pip 72 | 73 | pip cache remove '*cp313?-linux*' || true 74 | pip cache remove '*cp314?-linux*' || true 75 | pip cache remove '*cp315?-linux*' || true 76 | pip cache remove '*cp316?-linux*' || true 77 | 78 | 79 | @(xonsh_abs_path) build_py_env.xsh 80 | -------------------------------------------------------------------------------- /oci/01-base.xsh: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | debug = False 4 | 5 | container=$(buildah from btw-source) 6 | # we do not have an image to start from, bootstram from archlinux 7 | if not container: 8 | container=$(buildah from archlinux) 9 | 10 | print(container) 11 | 12 | scratchmnt=$(buildah mount @(container)) 13 | print(scratchmnt) 14 | 15 | print($(ls @(scratchmnt))) 16 | 17 | btw = (Path(scratchmnt) / 'btw') 18 | 19 | if not btw.exists(): 20 | git clone https://github.com/tacaswell/build_the_world @(btw) 21 | else: 22 | buildah run --workingdir=btw @(container) -- git remote update 23 | buildah run --workingdir=btw @(container) -- git reset --hard origin/main 24 | 25 | xonsh ensure_clones.xsh --target @(scratchmnt)/src 26 | 27 | buildah run @(container) -- pacman -Syu --noconfirm 28 | buildah run @(container) -- pacman -Sy git xonsh python python-yaml --noconfirm 29 | 30 | buildah run --workingdir=btw @(container) -- xonsh find_repos.xsh ../src/ 31 | buildah run --workingdir=btw @(container) -- xonsh setup_extra_remotes.xsh 32 | buildah run --workingdir=btw @(container) -- xonsh find_repos.xsh ../src/ 33 | 34 | buildah unmount @(container) 35 | buildah commit @(container) btw-source 36 | 37 | 38 | if debug: 39 | buildah rm @(container) 40 | -------------------------------------------------------------------------------- /oci/02-deps.xsh: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | debug = False 4 | 5 | try_continue = False 6 | 7 | if try_continue: 8 | container=$(buildah from btw-deps) 9 | # we do not have an image to start from, bootstram from btw-source 10 | if not try_continue or not container: 11 | container=$(buildah from btw-source) 12 | 13 | print(container) 14 | 15 | scratchmnt=$(buildah mount @(container)) 16 | 17 | buildah config --workingdir=btw @(container) 18 | 19 | buildah run @(container) -- pacman -Syu --noconfirm 20 | buildah run @(container) -- pacman -Sy ack --noconfirm 21 | buildah run @(container) -- pacman -Sy base-devel libjpeg-turbo cmake ccache gcc-fortran blas-openblas openblas hdf5 libxml2 libxslt graphviz npm --noconfirm 22 | 23 | with open(Path(scratchmnt) / 'root' / '.bashrc', 'a') as fout: 24 | fout.write( 25 | ''' 26 | PATH=$PATH:/usr/bin/vendor_perl/ 27 | export PATH 28 | ''') 29 | 30 | buildah unmount @(container) 31 | buildah commit @(container) btw-deps 32 | 33 | 34 | if debug: 35 | buildah rm @(container) 36 | -------------------------------------------------------------------------------- /oci/04-run_build.xsh: -------------------------------------------------------------------------------- 1 | import sys 2 | from pathlib import Path 3 | 4 | debug = False 5 | 6 | container=$(buildah from btw-deps) 7 | if not container: 8 | sys.exit(1) 9 | 10 | buildah run @(container) -- git pull 11 | buildah copy @(container) extra_remotes.yaml extra_remotes.yaml 12 | buildah copy @(container) setup_extra_remotes.xsh setup_extra_remotes.xsh 13 | 14 | buildah run @(container) -- xonsh setup_extra_remotes.xsh 15 | 16 | # TODO control the args via cli input 17 | buildah run @(container) -- xonsh make_bleeding.xsh --target py313 --branch 3.13 18 | # TODO control the container name via cli input 19 | buildah commit @(container) btw-built 20 | -------------------------------------------------------------------------------- /repo_report.xsh: -------------------------------------------------------------------------------- 1 | import json 2 | import sys 3 | 4 | import time 5 | import yaml 6 | from pathlib import Path 7 | 8 | from subprocess import CalledProcessError 9 | 10 | from xonsh.dirstack import with_pushd 11 | import sys 12 | 13 | $RAISE_SUBPROC_ERROR = False 14 | $XONSH_TRACE_SUBPROC = False 15 | 16 | 17 | def extract_git_shas(): 18 | headsha = $(git rev-parse HEAD).strip() 19 | describe = $(git describe --tags --abbrev=11 --long --dirty --always).strip() 20 | return {'head': headsha, 'describe': describe} 21 | 22 | build_order = [] 23 | for order in sorted(Path('build_order.d').glob('[!.]*yaml')): 24 | with open(order) as fin: 25 | build_order += list(yaml.unsafe_load_all(fin)) 26 | 27 | with open('all_repos.yaml') as fin: 28 | checkouts = list(yaml.unsafe_load_all(fin)) 29 | 30 | local_checkouts = {co['name']: co for co in checkouts} 31 | 32 | out = {} 33 | 34 | 35 | 36 | for step in build_order: 37 | if step['kind'] != 'source_install': 38 | continue 39 | lc = local_checkouts[step['proj_name']] 40 | # get the working directory 41 | step['wd'] = lc['local_checkout'] 42 | # set the default branch 43 | if lc['primary_remote']['vc'] != 'git': 44 | continue 45 | 46 | upstream_remote = next( 47 | n for n, r in lc['remotes'].items() if r['url'] == lc['primary_remote']['url'] 48 | ) 49 | upstream_branch = step['default_branch'] 50 | 51 | def foo(upstream_branch, checkout): 52 | with with_pushd(checkout): 53 | with ${...}.swap(RAISE_SUBPROC_ERROR=False): 54 | tracking = !(git rev-parse --abbrev-ref --symbolic-full-name '@{u}') 55 | has_tracking = bool(tracking) 56 | tracking_branch = tracking.output.strip() 57 | del tracking 58 | with ${...}.swap(RAISE_SUBPROC_ERROR=True): 59 | cur_branch = $(git branch --show-current).strip() 60 | upstream = f'{upstream_remote}/{upstream_branch}' 61 | with ${...}.swap(RAISE_SUBPROC_ERROR=False): 62 | is_merged = bool(!(git merge-base --is-ancestor HEAD @(upstream))) 63 | shas = extract_git_shas() 64 | return locals() 65 | out[step['proj_name']] = foo(upstream_branch, lc['local_checkout']) 66 | 67 | extra_remotes = [] 68 | 69 | for k, v in out.items(): 70 | off_dflt_branch = v['cur_branch'] != v['upstream_branch'] 71 | dirty_src = 'dirty' in v['shas']['describe'] 72 | if off_dflt_branch or dirty_src: 73 | print(f"{k} ({v['checkout']})") 74 | if off_dflt_branch: 75 | print(f" default: {v['upstream_branch']}") 76 | print(f" on: {v['cur_branch']} [{v['shas']['describe']}]({v['tracking_branch'] if v['has_tracking'] else '-'})") 77 | if v['has_tracking']: 78 | remote_name, *junk = v['tracking_branch'].partition('/') 79 | extra_remotes.append( 80 | {'proj_name': k, 'branch': v['cur_branch'], 'remote': local_checkouts[k]['remotes'][remote_name], 'remote_name': remote_name} 81 | ) 82 | if dirty_src: 83 | print(" DIRTY") 84 | 85 | with open('extra_remotes.yaml', 'w') as fout: 86 | yaml.dump_all(sorted(extra_remotes, key=lambda x: x['proj_name']), fout) 87 | -------------------------------------------------------------------------------- /setup_extra_remotes.xsh: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | import re 3 | 4 | import yaml 5 | from collections import defaultdict 6 | 7 | from xonsh.dirstack import with_pushd 8 | 9 | 10 | $RAISE_SUBPROC_ERROR = False 11 | $XONSH_TRACE_SUBPROC = False 12 | $PIP_NO_BUILD_ISOLATION = 1 13 | 14 | def fix_remotes(): 15 | for j, r in enumerate($(git remote).strip().split('\n')): 16 | remote_url = $(git remote get-url @(r)).strip() 17 | if ret := re.search(r'git@github.com:(?P.+)/(?P.*)', remote_url): 18 | git remote set-url @(r) f'https://github.com/{ret["org"]}/{ret["repo"]}' 19 | git remote set-url @(r) --push f'git@github.com:{ret["org"]}/{ret["repo"]}' 20 | 21 | remote_url = $(git remote get-url @(r)).strip() 22 | if remote_url.startswith('https') and not remote_url.endswith('.git'): 23 | git remote set-url @(r) f'{remote_url}.git' 24 | 25 | def get_git_remotes(repo): 26 | """Given a path to a repository, get remotes 27 | """ 28 | remotes = defaultdict(dict) 29 | 30 | with with_pushd(repo): 31 | for remote in !(git remote -v).itercheck(): 32 | name, _, rest = remote.strip().partition('\t') 33 | url, _, direction = rest.partition(' (') 34 | direction = direction[:-1] 35 | remotes[name][direction] = url 36 | 37 | return dict(remotes) 38 | 39 | 40 | with open('all_repos.yaml') as fin: 41 | checkouts = list(yaml.unsafe_load_all(fin)) 42 | 43 | local_checkouts = {co['name']: co for co in checkouts} 44 | del checkouts 45 | 46 | with open('extra_remotes.yaml') as fin: 47 | extra_remotes = {r['proj_name']: r for r in yaml.unsafe_load_all(fin)} 48 | 49 | 50 | build_order = [] 51 | for order in sorted(Path('build_order.d').glob('[!.]*yaml')): 52 | with open(order) as fin: 53 | build_order += [ 54 | bs 55 | for bs in yaml.unsafe_load_all(fin) 56 | if bs['kind'] == 'source_install' 57 | ] 58 | 59 | for step in build_order: 60 | lc = local_checkouts[step['proj_name']] 61 | wd = lc['local_checkout'] 62 | print(step['proj_name'], wd) 63 | if project := extra_remotes.get(step['proj_name'], None): 64 | remotes = get_git_remotes(wd) 65 | with with_pushd(wd): 66 | fix_remotes() 67 | if project['remote_name'] not in remotes: 68 | git remote add @(project['remote_name']) @(project['remote']['url']) 69 | elif remotes[project['remote_name']]['push'] != project['remote']['url']: 70 | git remote set-url @(project['remote_name']) @(project['remote']['url']) 71 | git fetch @(project['remote_name']) 72 | if $(git branch --list @(project['branch'])): 73 | git reset --hard @(project['remote_name'])/@(project['branch']) 74 | else: 75 | git switch -c @(project['branch']) -t @(project['remote_name'])/@(project['branch']) 76 | git pull 77 | else: 78 | with with_pushd(wd): 79 | fix_remotes() 80 | git switch @(step['default_branch']) 81 | git pull 82 | -------------------------------------------------------------------------------- /used_repos.yaml: -------------------------------------------------------------------------------- 1 | host: github.com 2 | protocol: https 3 | repo_name: python-blosc 4 | ssh_user: null 5 | url: https://github.com/Blosc/python-blosc 6 | user: Blosc 7 | vc: git 8 | --- 9 | host: github.com 10 | protocol: https 11 | repo_name: wrapt 12 | ssh_user: null 13 | url: https://github.com/GrahamDumpleton/wrapt.git 14 | user: GrahamDumpleton 15 | vc: git 16 | --- 17 | host: github.com 18 | protocol: https 19 | repo_name: httptools 20 | ssh_user: null 21 | url: https://github.com/MagicStack/httptools.git 22 | user: MagicStack 23 | vc: git 24 | --- 25 | host: github.com 26 | protocol: https 27 | repo_name: uvloop 28 | ssh_user: null 29 | url: https://github.com/MagicStack/uvloop.git 30 | user: MagicStack 31 | vc: git 32 | --- 33 | host: github.com 34 | protocol: https 35 | repo_name: bcsio 36 | ssh_user: null 37 | url: https://github.com/NSLS-II/bcsio 38 | user: NSLS-II 39 | vc: git 40 | --- 41 | host: github.com 42 | protocol: https 43 | repo_name: mily 44 | ssh_user: null 45 | url: https://github.com/NSLS-II/mily 46 | user: NSLS-II 47 | vc: git 48 | --- 49 | host: github.com 50 | protocol: https 51 | repo_name: nsls2-detector-handlers 52 | ssh_user: null 53 | url: https://github.com/NSLS-II/nsls2-detector-handlers 54 | user: NSLS-II 55 | vc: git 56 | --- 57 | host: github.com 58 | protocol: https 59 | repo_name: historydict 60 | ssh_user: null 61 | url: https://github.com/Nikea/historydict 62 | user: Nikea 63 | vc: git 64 | --- 65 | host: github.com 66 | protocol: https 67 | repo_name: flake8 68 | ssh_user: null 69 | url: https://github.com/PyCQA/flake8 70 | user: PyCQA 71 | vc: git 72 | --- 73 | host: github.com 74 | protocol: https 75 | repo_name: pywt 76 | ssh_user: null 77 | url: https://github.com/PyWavelets/pywt.git 78 | user: PyWavelets 79 | vc: git 80 | --- 81 | host: github.com 82 | protocol: https 83 | repo_name: sip 84 | ssh_user: null 85 | url: https://github.com/Python-SIP/sip 86 | user: Python-SIP 87 | vc: git 88 | --- 89 | host: github.com 90 | protocol: https 91 | repo_name: aiohttp 92 | ssh_user: null 93 | url: https://github.com/aio-libs/aiohttp.git 94 | user: aio-libs 95 | vc: git 96 | --- 97 | host: github.com 98 | protocol: https 99 | repo_name: frozenlist 100 | ssh_user: null 101 | url: https://github.com/aio-libs/frozenlist.git 102 | user: aio-libs 103 | vc: git 104 | --- 105 | host: github.com 106 | protocol: https 107 | repo_name: multidict 108 | ssh_user: null 109 | url: https://github.com/aio-libs/multidict.git 110 | user: aio-libs 111 | vc: git 112 | --- 113 | host: github.com 114 | protocol: https 115 | repo_name: yarl 116 | ssh_user: null 117 | url: https://github.com/aio-libs/yarl.git 118 | user: aio-libs 119 | vc: git 120 | --- 121 | host: github.com 122 | protocol: https 123 | repo_name: executing 124 | ssh_user: null 125 | url: https://github.com/alexmojaki/executing.git 126 | user: alexmojaki 127 | vc: git 128 | --- 129 | host: github.com 130 | protocol: https 131 | repo_name: arrow 132 | ssh_user: null 133 | url: https://github.com/apache/arrow.git 134 | user: apache 135 | vc: git 136 | --- 137 | host: github.com 138 | protocol: https 139 | repo_name: area-detector-handlers 140 | ssh_user: null 141 | url: https://github.com/bluesky/area-detector-handlers.git 142 | user: bluesky 143 | vc: git 144 | --- 145 | host: github.com 146 | protocol: https 147 | repo_name: bluesky 148 | ssh_user: null 149 | url: https://github.com/bluesky/bluesky.git 150 | user: bluesky 151 | vc: git 152 | --- 153 | host: github.com 154 | protocol: https 155 | repo_name: bluesky-adaptive 156 | ssh_user: null 157 | url: https://github.com/bluesky/bluesky-adaptive.git 158 | user: bluesky 159 | vc: git 160 | --- 161 | host: github.com 162 | protocol: https 163 | repo_name: bluesky-darkframes 164 | ssh_user: null 165 | url: https://github.com/bluesky/bluesky-darkframes.git 166 | user: bluesky 167 | vc: git 168 | --- 169 | host: github.com 170 | protocol: https 171 | repo_name: bluesky-httpserver 172 | ssh_user: null 173 | url: https://github.com/bluesky/bluesky-httpserver.git 174 | user: bluesky 175 | vc: git 176 | --- 177 | host: github.com 178 | protocol: https 179 | repo_name: bluesky-live 180 | ssh_user: null 181 | url: https://github.com/bluesky/bluesky-live.git 182 | user: bluesky 183 | vc: git 184 | --- 185 | host: github.com 186 | protocol: https 187 | repo_name: bluesky-queueserver 188 | ssh_user: null 189 | url: https://github.com/bluesky/bluesky-queueserver.git 190 | user: bluesky 191 | vc: git 192 | --- 193 | host: github.com 194 | protocol: https 195 | repo_name: bluesky-queueserver-api 196 | ssh_user: null 197 | url: https://github.com/bluesky/bluesky-queueserver-api.git 198 | user: bluesky 199 | vc: git 200 | --- 201 | host: github.com 202 | protocol: https 203 | repo_name: bluesky-spreadsheet 204 | ssh_user: null 205 | url: https://github.com/bluesky/bluesky-spreadsheet.git 206 | user: bluesky 207 | vc: git 208 | --- 209 | host: github.com 210 | protocol: https 211 | repo_name: bluesky-ui 212 | ssh_user: null 213 | url: https://github.com/bluesky/bluesky-ui.git 214 | user: bluesky 215 | vc: git 216 | --- 217 | host: github.com 218 | protocol: https 219 | repo_name: bluesky-widgets 220 | ssh_user: null 221 | url: https://github.com/bluesky/bluesky-widgets.git 222 | user: bluesky 223 | vc: git 224 | --- 225 | host: github.com 226 | protocol: https 227 | repo_name: databroker 228 | ssh_user: null 229 | url: https://github.com/bluesky/databroker.git 230 | user: bluesky 231 | vc: git 232 | --- 233 | host: github.com 234 | protocol: https 235 | repo_name: databroker-pack 236 | ssh_user: null 237 | url: https://github.com/bluesky/databroker-pack.git 238 | user: bluesky 239 | vc: git 240 | --- 241 | host: github.com 242 | protocol: https 243 | repo_name: event-model 244 | ssh_user: null 245 | url: https://github.com/bluesky/event-model.git 246 | user: bluesky 247 | vc: git 248 | --- 249 | host: github.com 250 | protocol: https 251 | repo_name: hklpy 252 | ssh_user: null 253 | url: https://github.com/bluesky/hklpy.git 254 | user: bluesky 255 | vc: git 256 | --- 257 | host: github.com 258 | protocol: https 259 | repo_name: ophyd 260 | ssh_user: null 261 | url: https://github.com/bluesky/ophyd.git 262 | user: bluesky 263 | vc: git 264 | --- 265 | host: github.com 266 | protocol: https 267 | repo_name: suitcase-csv 268 | ssh_user: null 269 | url: https://github.com/bluesky/suitcase-csv.git 270 | user: bluesky 271 | vc: git 272 | --- 273 | host: github.com 274 | protocol: https 275 | repo_name: suitcase-dataexchange 276 | ssh_user: null 277 | url: https://github.com/bluesky/suitcase-dataexchange.git 278 | user: bluesky 279 | vc: git 280 | --- 281 | host: github.com 282 | protocol: https 283 | repo_name: suitcase-json-metadata 284 | ssh_user: null 285 | url: https://github.com/bluesky/suitcase-json-metadata.git 286 | user: bluesky 287 | vc: git 288 | --- 289 | host: github.com 290 | protocol: https 291 | repo_name: suitcase-jsonl 292 | ssh_user: null 293 | url: https://github.com/bluesky/suitcase-jsonl.git 294 | user: bluesky 295 | vc: git 296 | --- 297 | host: github.com 298 | protocol: https 299 | repo_name: suitcase-mongo 300 | ssh_user: null 301 | url: https://github.com/bluesky/suitcase-mongo.git 302 | user: bluesky 303 | vc: git 304 | --- 305 | host: github.com 306 | protocol: https 307 | repo_name: suitcase-msgpack 308 | ssh_user: null 309 | url: https://github.com/bluesky/suitcase-msgpack.git 310 | user: bluesky 311 | vc: git 312 | --- 313 | host: github.com 314 | protocol: https 315 | repo_name: suitcase-nxstxm 316 | ssh_user: null 317 | url: https://github.com/bluesky/suitcase-nxstxm.git 318 | user: bluesky 319 | vc: git 320 | --- 321 | host: github.com 322 | protocol: https 323 | repo_name: suitcase-server 324 | ssh_user: null 325 | url: https://github.com/bluesky/suitcase-server.git 326 | user: bluesky 327 | vc: git 328 | --- 329 | host: github.com 330 | protocol: https 331 | repo_name: suitcase-specfile 332 | ssh_user: null 333 | url: https://github.com/bluesky/suitcase-specfile.git 334 | user: bluesky 335 | vc: git 336 | --- 337 | host: github.com 338 | protocol: https 339 | repo_name: suitcase-tiff 340 | ssh_user: null 341 | url: https://github.com/bluesky/suitcase-tiff.git 342 | user: bluesky 343 | vc: git 344 | --- 345 | host: github.com 346 | protocol: https 347 | repo_name: suitcase-utils 348 | ssh_user: null 349 | url: https://github.com/bluesky/suitcase-utils.git 350 | user: bluesky 351 | vc: git 352 | --- 353 | host: github.com 354 | protocol: https 355 | repo_name: caproto 356 | ssh_user: null 357 | url: https://github.com/caproto/caproto.git 358 | user: caproto 359 | vc: git 360 | --- 361 | host: github.com 362 | protocol: https 363 | repo_name: python-certifi 364 | ssh_user: null 365 | url: https://github.com/certifi/python-certifi.git 366 | user: certifi 367 | vc: git 368 | --- 369 | host: github.com 370 | protocol: https 371 | repo_name: imagecodecs 372 | ssh_user: null 373 | url: https://github.com/cgohlke/imagecodecs.git 374 | user: cgohlke 375 | vc: git 376 | --- 377 | host: github.com 378 | protocol: https 379 | repo_name: tifffile 380 | ssh_user: null 381 | url: https://github.com/cgohlke/tifffile.git 382 | user: cgohlke 383 | vc: git 384 | --- 385 | host: github.com 386 | protocol: https 387 | repo_name: confluent-kafka-python 388 | ssh_user: null 389 | url: https://github.com/confluentinc/confluent-kafka-python.git 390 | user: confluentinc 391 | vc: git 392 | --- 393 | host: github.com 394 | protocol: https 395 | repo_name: contourpy 396 | ssh_user: null 397 | url: https://github.com/contourpy/contourpy.git 398 | user: contourpy 399 | vc: git 400 | --- 401 | host: github.com 402 | protocol: https 403 | repo_name: cython 404 | ssh_user: null 405 | url: https://github.com/cython/cython.git 406 | user: cython 407 | vc: git 408 | --- 409 | host: github.com 410 | protocol: https 411 | repo_name: curio 412 | ssh_user: null 413 | url: https://github.com/dabeaz/curio.git 414 | user: dabeaz 415 | vc: git 416 | --- 417 | host: github.com 418 | protocol: https 419 | repo_name: dask 420 | ssh_user: null 421 | url: https://github.com/dask/dask.git 422 | user: dask 423 | vc: git 424 | --- 425 | host: github.com 426 | protocol: https 427 | repo_name: distributed 428 | ssh_user: null 429 | url: https://github.com/dask/distributed.git 430 | user: dask 431 | vc: git 432 | --- 433 | host: github.com 434 | protocol: https 435 | repo_name: dateutil 436 | ssh_user: null 437 | url: https://github.com/dateutil/dateutil.git 438 | user: dateutil 439 | vc: git 440 | --- 441 | host: github.com 442 | protocol: https 443 | repo_name: jedi 444 | ssh_user: null 445 | url: https://github.com/davidhalter/jedi.git 446 | user: davidhalter 447 | vc: git 448 | --- 449 | host: github.com 450 | protocol: https 451 | repo_name: parso 452 | ssh_user: null 453 | url: https://github.com/davidhalter/parso.git 454 | user: davidhalter 455 | vc: git 456 | --- 457 | host: github.com 458 | protocol: https 459 | repo_name: MyST-Parser 460 | ssh_user: null 461 | url: https://github.com/executablebooks/MyST-Parser.git 462 | user: executablebooks 463 | vc: git 464 | --- 465 | host: github.com 466 | protocol: https 467 | repo_name: sphinx-design 468 | ssh_user: null 469 | url: https://github.com/executablebooks/sphinx-design.git 470 | user: executablebooks 471 | vc: git 472 | --- 473 | host: github.com 474 | protocol: https 475 | repo_name: h5py 476 | ssh_user: null 477 | url: https://github.com/h5py/h5py.git 478 | user: h5py 479 | vc: git 480 | --- 481 | host: github.com 482 | protocol: https 483 | repo_name: flexparser 484 | ssh_user: null 485 | url: https://github.com/hgrecco/flexparser.git 486 | user: hgrecco 487 | vc: git 488 | --- 489 | host: github.com 490 | protocol: https 491 | repo_name: pint 492 | ssh_user: null 493 | url: https://github.com/hgrecco/pint.git 494 | user: hgrecco 495 | vc: git 496 | --- 497 | host: github.com 498 | protocol: https 499 | repo_name: argon2-cffi-bindings 500 | ssh_user: null 501 | url: https://github.com/hynek/argon2-cffi-bindings.git 502 | user: hynek 503 | vc: git 504 | --- 505 | host: github.com 506 | protocol: https 507 | repo_name: orjson 508 | ssh_user: null 509 | url: https://github.com/ijl/orjson.git 510 | user: ijl 511 | vc: git 512 | --- 513 | host: github.com 514 | protocol: https 515 | repo_name: pyopencl 516 | ssh_user: null 517 | url: https://github.com/inducer/pyopencl.git 518 | user: inducer 519 | vc: git 520 | --- 521 | host: github.com 522 | protocol: https 523 | repo_name: python-zstandard 524 | ssh_user: null 525 | url: https://github.com/indygreg/python-zstandard.git 526 | user: indygreg 527 | vc: git 528 | --- 529 | host: github.com 530 | protocol: https 531 | repo_name: intake 532 | ssh_user: null 533 | url: https://github.com/intake/intake.git 534 | user: intake 535 | vc: git 536 | --- 537 | host: github.com 538 | protocol: https 539 | repo_name: ipython 540 | ssh_user: null 541 | url: https://github.com/ipython/ipython 542 | user: ipython 543 | vc: git 544 | --- 545 | host: github.com 546 | protocol: https 547 | repo_name: msgspec 548 | ssh_user: null 549 | url: https://github.com/jcrist/msgspec.git 550 | user: jcrist 551 | vc: git 552 | --- 553 | host: github.com 554 | protocol: https 555 | repo_name: nbconvert 556 | ssh_user: null 557 | url: https://github.com/jupyter/nbconvert.git 558 | user: jupyter 559 | vc: git 560 | --- 561 | host: github.com 562 | protocol: https 563 | repo_name: nbformat 564 | ssh_user: null 565 | url: https://github.com/jupyter/nbformat.git 566 | user: jupyter 567 | vc: git 568 | --- 569 | host: github.com 570 | protocol: ssh 571 | repo_name: multianalyzer 572 | ssh_user: git 573 | url: git@github.com:kif/multianalyzer.git 574 | user: kif 575 | vc: git 576 | --- 577 | host: github.com 578 | protocol: https 579 | repo_name: pyerfa 580 | ssh_user: null 581 | url: https://github.com/liberfa/pyerfa.git 582 | user: liberfa 583 | vc: git 584 | --- 585 | host: github.com 586 | protocol: https 587 | repo_name: lxml 588 | ssh_user: null 589 | url: https://github.com/lxml/lxml.git 590 | user: lxml 591 | vc: git 592 | --- 593 | host: github.com 594 | protocol: https 595 | repo_name: cycler 596 | ssh_user: null 597 | url: https://github.com/matplotlib/cycler.git 598 | user: matplotlib 599 | vc: git 600 | --- 601 | host: github.com 602 | protocol: https 603 | repo_name: data-prototype 604 | ssh_user: null 605 | url: https://github.com/matplotlib/data-prototype.git 606 | user: matplotlib 607 | vc: git 608 | --- 609 | host: github.com 610 | protocol: https 611 | repo_name: matplotlib 612 | ssh_user: null 613 | url: https://github.com/matplotlib/matplotlib.git 614 | user: matplotlib 615 | vc: git 616 | --- 617 | host: github.com 618 | protocol: https 619 | repo_name: mpl-gui 620 | ssh_user: null 621 | url: https://github.com/matplotlib/mpl-gui.git 622 | user: matplotlib 623 | vc: git 624 | --- 625 | host: github.com 626 | protocol: https 627 | repo_name: mplcairo 628 | ssh_user: null 629 | url: https://github.com/matplotlib/mplcairo.git 630 | user: matplotlib 631 | vc: git 632 | --- 633 | host: github.com 634 | protocol: https 635 | repo_name: mplfinance 636 | ssh_user: null 637 | url: https://github.com/matplotlib/mplfinance.git 638 | user: matplotlib 639 | vc: git 640 | --- 641 | host: github.com 642 | protocol: https 643 | repo_name: pyopengl 644 | ssh_user: null 645 | url: https://github.com/mcfletch/pyopengl 646 | user: mcfletch 647 | vc: git 648 | --- 649 | host: github.com 650 | protocol: https 651 | repo_name: pyopengl 652 | ssh_user: null 653 | url: https://github.com/mcfletch/pyopengl 654 | user: mcfletch 655 | vc: git 656 | --- 657 | host: github.com 658 | protocol: https 659 | repo_name: meson 660 | ssh_user: null 661 | url: https://github.com/mesonbuild/meson.git 662 | user: mesonbuild 663 | vc: git 664 | --- 665 | host: github.com 666 | protocol: https 667 | repo_name: meson-python 668 | ssh_user: null 669 | url: https://github.com/mesonbuild/meson-python 670 | user: mesonbuild 671 | vc: git 672 | --- 673 | host: github.com 674 | protocol: https 675 | repo_name: msgpack-python 676 | ssh_user: null 677 | url: https://github.com/msgpack/msgpack-python.git 678 | user: msgpack 679 | vc: git 680 | --- 681 | host: github.com 682 | protocol: https 683 | repo_name: networkx 684 | ssh_user: null 685 | url: https://github.com/networkx/networkx.git 686 | user: networkx 687 | vc: git 688 | --- 689 | host: github.com 690 | protocol: https 691 | repo_name: numpy 692 | ssh_user: null 693 | url: https://github.com/numpy/numpy.git 694 | user: numpy 695 | vc: git 696 | --- 697 | host: github.com 698 | protocol: https 699 | repo_name: opentelemetry-python 700 | ssh_user: null 701 | url: https://github.com/open-telemetry/opentelemetry-python.git 702 | user: open-telemetry 703 | vc: git 704 | --- 705 | host: github.com 706 | protocol: https 707 | repo_name: pandas 708 | ssh_user: null 709 | url: https://github.com/pandas-dev/pandas.git 710 | user: pandas-dev 711 | vc: git 712 | --- 713 | host: github.com 714 | protocol: https 715 | repo_name: happi 716 | ssh_user: null 717 | url: https://github.com/pcdshub/happi.git 718 | user: pcdshub 719 | vc: git 720 | --- 721 | host: github.com 722 | protocol: https 723 | repo_name: pexpect 724 | ssh_user: null 725 | url: https://github.com/pexpect/pexpect.git 726 | user: pexpect 727 | vc: git 728 | --- 729 | host: github.com 730 | protocol: https 731 | repo_name: pybind11 732 | ssh_user: null 733 | url: https://github.com/pybind/pybind11.git 734 | user: pybind 735 | vc: git 736 | --- 737 | host: github.com 738 | protocol: https 739 | repo_name: pycurl 740 | ssh_user: null 741 | url: https://github.com/pycurl/pycurl.git 742 | user: pycurl 743 | vc: git 744 | --- 745 | host: github.com 746 | protocol: https 747 | repo_name: numexpr 748 | ssh_user: null 749 | url: https://github.com/pydata/numexpr.git 750 | user: pydata 751 | vc: git 752 | --- 753 | host: github.com 754 | protocol: https 755 | repo_name: build 756 | ssh_user: null 757 | url: https://github.com/pypa/build.git 758 | user: pypa 759 | vc: git 760 | --- 761 | host: github.com 762 | protocol: https 763 | repo_name: flit 764 | ssh_user: null 765 | url: https://github.com/pypa/flit.git 766 | user: pypa 767 | vc: git 768 | --- 769 | host: github.com 770 | protocol: https 771 | repo_name: pip 772 | ssh_user: null 773 | url: https://github.com/pypa/pip.git 774 | user: pypa 775 | vc: git 776 | --- 777 | host: github.com 778 | protocol: https 779 | repo_name: setuptools 780 | ssh_user: null 781 | url: https://github.com/pypa/setuptools.git 782 | user: pypa 783 | vc: git 784 | --- 785 | host: github.com 786 | protocol: https 787 | repo_name: twine 788 | ssh_user: null 789 | url: https://github.com/pypa/twine.git 790 | user: pypa 791 | vc: git 792 | --- 793 | host: github.com 794 | protocol: https 795 | repo_name: wheel 796 | ssh_user: null 797 | url: https://github.com/pypa/wheel.git 798 | user: pypa 799 | vc: git 800 | --- 801 | host: github.com 802 | protocol: https 803 | repo_name: pyqtgraph 804 | ssh_user: null 805 | url: https://github.com/pyqtgraph/pyqtgraph.git 806 | user: pyqtgraph 807 | vc: git 808 | --- 809 | host: github.com 810 | protocol: https 811 | repo_name: pytest 812 | ssh_user: null 813 | url: https://github.com/pytest-dev/pytest.git 814 | user: pytest-dev 815 | vc: git 816 | --- 817 | host: github.com 818 | protocol: https 819 | repo_name: cpython 820 | ssh_user: null 821 | url: https://github.com/python/cpython.git 822 | user: python 823 | vc: git 824 | --- 825 | host: github.com 826 | protocol: https 827 | repo_name: cffi 828 | ssh_user: null 829 | url: https://github.com/python-cffi/cffi.git 830 | user: python-cffi 831 | vc: git 832 | --- 833 | host: github.com 834 | protocol: https 835 | repo_name: greenlet 836 | ssh_user: null 837 | url: https://github.com/python-greenlet/greenlet.git 838 | user: python-greenlet 839 | vc: git 840 | --- 841 | host: github.com 842 | protocol: https 843 | repo_name: uritemplate 844 | ssh_user: null 845 | url: https://github.com/python-hyper/uritemplate.git 846 | user: python-hyper 847 | vc: git 848 | --- 849 | host: github.com 850 | protocol: https 851 | repo_name: Pillow 852 | ssh_user: null 853 | url: https://github.com/python-pillow/Pillow.git 854 | user: python-pillow 855 | vc: git 856 | --- 857 | host: github.com 858 | protocol: https 859 | repo_name: cytoolz 860 | ssh_user: null 861 | url: https://github.com/pytoolz/cytoolz.git 862 | user: pytoolz 863 | vc: git 864 | --- 865 | host: github.com 866 | protocol: https 867 | repo_name: toolz 868 | ssh_user: null 869 | url: https://github.com/pytoolz/toolz.git 870 | user: pytoolz 871 | vc: git 872 | --- 873 | host: github.com 874 | protocol: https 875 | repo_name: sphinx_rtd_theme 876 | ssh_user: null 877 | url: https://github.com/readthedocs/sphinx_rtd_theme.git 878 | user: readthedocs 879 | vc: git 880 | --- 881 | host: github.com 882 | protocol: https 883 | repo_name: watchfiles 884 | ssh_user: null 885 | url: https://github.com/samuelcolvin/watchfiles.git 886 | user: samuelcolvin 887 | vc: git 888 | --- 889 | host: github.com 890 | protocol: https 891 | repo_name: scikit-beam 892 | ssh_user: null 893 | url: https://github.com/scikit-beam/scikit-beam.git 894 | user: scikit-beam 895 | vc: git 896 | --- 897 | host: github.com 898 | protocol: https 899 | repo_name: awkward 900 | ssh_user: null 901 | url: https://github.com/scikit-hep/awkward.git 902 | user: scikit-hep 903 | vc: git 904 | --- 905 | host: github.com 906 | protocol: https 907 | repo_name: scikit-image 908 | ssh_user: null 909 | url: https://github.com/scikit-image/scikit-image.git 910 | user: scikit-image 911 | vc: git 912 | --- 913 | host: github.com 914 | protocol: https 915 | repo_name: scikit-learn 916 | ssh_user: null 917 | url: https://github.com/scikit-learn/scikit-learn 918 | user: scikit-learn 919 | vc: git 920 | --- 921 | host: github.com 922 | protocol: https 923 | repo_name: scipy 924 | ssh_user: null 925 | url: https://github.com/scipy/scipy.git 926 | user: scipy 927 | vc: git 928 | --- 929 | host: github.com 930 | protocol: https 931 | repo_name: pythran 932 | ssh_user: null 933 | url: https://github.com/serge-sans-paille/pythran.git 934 | user: serge-sans-paille 935 | vc: git 936 | --- 937 | host: github.com 938 | protocol: https 939 | repo_name: silx 940 | ssh_user: null 941 | url: https://github.com/silx-kit/silx.git 942 | user: silx-kit 943 | vc: git 944 | --- 945 | host: github.com 946 | protocol: https 947 | repo_name: pims 948 | ssh_user: null 949 | url: https://github.com/soft-matter/pims.git 950 | user: soft-matter 951 | vc: git 952 | --- 953 | host: github.com 954 | protocol: https 955 | repo_name: sphinx 956 | ssh_user: null 957 | url: https://github.com/sphinx-doc/sphinx.git 958 | user: sphinx-doc 959 | vc: git 960 | --- 961 | host: github.com 962 | protocol: https 963 | repo_name: sympy 964 | ssh_user: null 965 | url: https://github.com/sympy/sympy 966 | user: sympy 967 | vc: git 968 | --- 969 | host: github.com 970 | protocol: https 971 | repo_name: mpl-qtthread 972 | ssh_user: null 973 | url: https://github.com/tacaswell/mpl-qtthread.git 974 | user: tacaswell 975 | vc: git 976 | --- 977 | host: github.com 978 | protocol: https 979 | repo_name: fastapi 980 | ssh_user: null 981 | url: https://github.com/tiangolo/fastapi.git 982 | user: tiangolo 983 | vc: git 984 | --- 985 | host: github.com 986 | protocol: https 987 | repo_name: tornado 988 | ssh_user: null 989 | url: https://github.com/tornadoweb/tornado.git 990 | user: tornadoweb 991 | vc: git 992 | --- 993 | host: github.com 994 | protocol: https 995 | repo_name: pyyaml 996 | ssh_user: null 997 | url: https://github.com/yaml/pyyaml.git 998 | user: yaml 999 | vc: git 1000 | --- 1001 | host: github.com 1002 | protocol: https 1003 | repo_name: numcodecs 1004 | ssh_user: null 1005 | url: https://github.com/zarr-developers/numcodecs.git 1006 | user: zarr-developers 1007 | vc: git 1008 | --- 1009 | host: github.com 1010 | protocol: https 1011 | repo_name: pyzmq 1012 | ssh_user: null 1013 | url: https://github.com/zeromq/pyzmq.git 1014 | user: zeromq 1015 | vc: git 1016 | --------------------------------------------------------------------------------