├── .gitignore ├── LICENSE ├── README.md ├── examples ├── evince.yml ├── files │ ├── 0001-Sample-patch-merely-for-ypkg-testing.patch │ ├── 0002-Sample-commit-2.patch │ └── series ├── gnome-maps.yml ├── mupen64plus.yml ├── nano.yml └── poppler.yml ├── gen_docs.sh ├── man ├── package.yml.5 ├── package.yml.5.html ├── package.yml.5.md ├── ypkg-build.1 ├── ypkg-build.1.html ├── ypkg-build.1.md ├── ypkg-install-deps.1 ├── ypkg-install-deps.1.html ├── ypkg-install-deps.1.md ├── ypkg.1 ├── ypkg.1.html └── ypkg.1.md ├── setup.py ├── test.sh ├── ybump ├── ypkg ├── ypkg-build ├── ypkg-gen-history ├── ypkg-install-deps ├── ypkg2 ├── __init__.py ├── dependencies.py ├── examine.py ├── main.py ├── metadata.py ├── packages.py ├── rc.yml ├── scripts.py ├── sources.py ├── stringglob.py ├── ui.py ├── yamlhelper.py ├── ypkgcontext.py └── ypkgspec.py └── yupdate /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ypkg2 2 | ----- 3 | 4 | **Modern, declarative, structured build format** 5 | 6 | 7 | ypkg is the build tool of choice for Solus. Simply put, it is a tool to convert a build process into a packaging operation. 8 | 9 | ypkg evolved from a basic set of ideas within the Solus project, that packaging should be simple. Given rules should be obeyed globally, and where appropriate the packager should be able to tweak the results to how they desire. 10 | 11 | As a result, `ypkg` provides a highly intuitive, simple, yet incredibly powerful package building system, which follows the Solus philosophy of sane defaults, and hidden power under the surface. `ypkg` is capable of many build types, multilib, automatic & intelligent package splitting, choice compiler optimisations, standardised `GDB`-compatible `debuginfo` packages, automatic dependency resolution, and much more. 12 | 13 | `ypkg` is a [Solus project](https://solus-project.com/). 14 | 15 | ![logo](https://build.solus-project.com/logo.png) 16 | 17 | **How Rules Work:** 18 | 19 | Everything in ypkg can be traced to a *pattern*. We use these to define what part of a package, or sub-package, a file should be placed in. To enforce consistency, we have our own in-built patterns, i.e.: 20 | 21 | /usr/lib64/lib*.so = devel subpackage 22 | /usr/lib64/lib*.so.* = "main" package 23 | 24 | There are many in-built patterns which designate how the subpackages should be emitted, allowing for self resolving dependency graphs: 25 | 26 | dev: depends on "main": 27 | /usr/lib64/pkgconfig/*.pc 28 | /usr/lib64/lib*.so 29 | dev: also depends on whatever the pkgconfig files depend on 30 | 31 | main: shared library dependencies: 32 | /usr/lib64/lib*.so.* 33 | 34 | However, sometime's it is desirable to split up a package. Consider this example: 35 | 36 | main: 37 | /usr/lib64/libpoppler-glib.so.* 38 | /usr/lib64/libpoppler-qt5*.so.* 39 | devel: 40 | /usr/lib64/lib*.so 41 | 42 | The automatic behaviour would by default make the main package gain a dependency on the `qt5` package, through direct binary dependencies. This can be very undesirable and would cause unnecessary bloat in lighter systems. 43 | 44 | The solution, therefore, is to do the following: 45 | 46 | main: 47 | /usr/lib64/lib*.so.* 48 | devel: 49 | /usr/lib64/lib*.so 50 | /usr/lib64/pkgconfig/*.pc 51 | 52 | qt5: 53 | /usr/lib64/libpoppler-qt5*.so.* 54 | qt5-devel: 55 | /usr/lib64/libpoppler-qt5*.so 56 | /usr/lib64/pkgconfig/poppler-qt5.pc 57 | 58 | Dependency Handling 59 | ------------------- 60 | 61 | ypkg formulates many of the dependencies on an automatic basis, even on your own custom patterns. Listed below are the supported types. 62 | 63 | **Direct Dependency** 64 | 65 | When building the package we first analyze all ELF dynamic objects to determine both exported SONAMEs and _direct dependencies_ on shared libraries. We initially filter out dependencies on internal sonames to ensure a package doesn't gain an unnecessary *external* dependency. 66 | 67 | These are validated against provided rpaths in the objects as well as against the global symbol table. 68 | 69 | In the instance the dependency is provided externally, it will be sought out externally via a combination of the linker paths and the soname. 70 | 71 | **SOLINK** 72 | 73 | Typically when splitting packages, we provide the `.so` symbolic link in the `-devel` pattern. To alleviate the need for manually describing this dependency graph, we automatically detect resolvable solinks in the current build tree in order to place a dependency on the providing package. 74 | 75 | In the instance of libpopplet-qt5.so, in the `qt5-devel` package, this resolves to `libpoppler-qt5.so.1.3.0` in the `-qt5` package, thus a solink dependency is added automatically. 76 | 77 | **PkgConfig** 78 | 79 | Ypkg exports two kinds of pkgconfig providers: `PkgConfig` and `PkgConfig32`. This allows for correct dependency chaining in `emul32` build scenarios. In short, a pkgconfig() provider becomes a pkgconfig32() provider when it is in the /usr/lib32 tree. 80 | 81 | Much as with soname resolution, we scan pkg-config dependencies in `-devel` type packages against a global provider table. Using again the `poppler-qt5` example, this pkgconfig file expresses a dependency on `poppler`, which is provided by the `poppler-devel` package. In turn, `poppler` pkgconfig has dependencies on `cairo`, which is resolved to an external package dependency. 82 | 83 | All in all, we formulate a self resolving dependency graph through entirely logical rules and splitting, without having to construct them ourselves. 84 | 85 | poppler-qt5-devel: 86 | poppler-qt5 -> 87 | poppler 88 | -> cairo, etc. 89 | poppler-devel -> 90 | poppler 91 | -> cairo, etc 92 | 93 | As such, this makes `pkgconfig()` build dependencies very powerful, as using `pkgconfig(poppler-qt5)` grabs *everything* necessary for use, and nothing more. Note also in Solus we implement the equivalent of `-Wl,--as-needed` at the toolchain level itself, in order to ensure we only need to deal with absolutely required dependencies. 94 | 95 | `emul32` packages 96 | ----------------- 97 | 98 | To build a package in a multilib fashion, simply set the `emul32` key to `yes`. This will cause the build steps to double up, the first run will build as a 32-bit build, and the second set will be the native 64-bit set. 99 | 100 | Build dependencies for 32-bit packages can expressly be 32-bit pkgconfig dependencies, i.e: 101 | 102 | pkgconfig32(zlib) 103 | 104 | `ypkg` will automatically add the following build dependencies for `emul32` builds, to save the developer extra effort: 105 | 106 | 107 | - glibc-32bit-devel 108 | - libgcc-32bit 109 | - libstdc++-32bit 110 | 111 | Note that 32-bit builds are done *first*, in a separate build root to the native 64-bit package. However they will both install to the same tree. Note that for `emul32` packages, we use `--prefix=/emul32 --libdir=/usr/lib32`. This avoids collisions of binaries and assets, but you may need to tweak to your requirements. 112 | 113 | Utilising Profile Guided Optimization 114 | ------------------------------------- 115 | 116 | Packages wishing to make use of PGO will require multiple build steps. In short, a package must provide a valid *workload* to generate PGO data. The steps are as follows: 117 | 118 | - setup: (unpack roots, clean trees, run `setup` step with PGO GEN flags) 119 | - build 120 | - profile (Run the actual `profile` step) 121 | - setup (unpack and clean again, use PGO USE flags) 122 | - build 123 | - install 124 | - check 125 | 126 | Basically, provide a `profile` step to run the workload after the *first* build, and `ypkg` will take care of the necessary ordering and environmental overrides. 127 | 128 | License 129 | ------- 130 | 131 | GPL-3.0 132 | 133 | Copyright © 2015-2017 Ikey Doherty 134 | 135 | Some portions: Copyright (C) 2016 Intel Corporation 136 | 137 | Some elements have been gratefully borrowed from autospec: 138 | 139 | https://github.com/clearlinux/autospec 140 | -------------------------------------------------------------------------------- /examples/evince.yml: -------------------------------------------------------------------------------- 1 | name : evince 2 | version : 3.14.1 3 | release : 1 4 | source : 5 | - http://ftp.gnome.org/pub/gnome/sources/evince/3.14/evince-3.14.1.tar.xz : 13ec728d6957aa18ba21a3a66504dd52b8607596337f30f0908b62b5fcc14507 6 | homepage : https://wiki.gnome.org/Apps/Evince 7 | license : GPL-2.0 8 | summary : GNOME Document Viewer 9 | clang : no 10 | builddeps : 11 | - pkgconfig(gtk+-3.0) 12 | - pkgconfig(gsettings-desktop-schemas) 13 | - pkgconfig(cairo-pdf) 14 | - pkgconfig(libsecret-1) 15 | - pkgconfig(poppler-glib) 16 | - pkgconfig(libnautilus-extension) 17 | - pkgconfig(libtiff-4) 18 | - pkgconfig(adwaita-icon-theme) 19 | - itstool 20 | description: | 21 | Evince is a document viewer for multiple document formats. The goal 22 | of evince is to replace the multiple document viewers that exist on 23 | the GNOME Desktop with a single simple application. 24 | Evince is specifically designed to support the file followin 25 | formats: PDF, Postscript, djvu, tiff, dvi, XPS, SyncTex support with 26 | gedit, comics books (cbr,cbz,cb7 and cbt). 27 | setup : | 28 | %configure --disable-static 29 | build : | 30 | %make 31 | install : | 32 | %make_install 33 | -------------------------------------------------------------------------------- /examples/files/0001-Sample-patch-merely-for-ypkg-testing.patch: -------------------------------------------------------------------------------- 1 | From 4a280db84b17fdb64d890808015942027fe93776 Mon Sep 17 00:00:00 2001 2 | From: Ikey Doherty 3 | Date: Thu, 23 Apr 2015 22:52:49 +0100 4 | Subject: [PATCH 1/2] Sample patch merely for ypkg testing 5 | 6 | --- 7 | doc/nanorc.sample.in | 8 ++++---- 8 | 1 file changed, 4 insertions(+), 4 deletions(-) 9 | 10 | diff --git a/doc/nanorc.sample.in b/doc/nanorc.sample.in 11 | index 192f9b4..83f9c4e 100644 12 | --- a/doc/nanorc.sample.in 13 | +++ b/doc/nanorc.sample.in 14 | @@ -242,10 +242,10 @@ 15 | # include "@PKGDATADIR@/nanorc.nanorc" 16 | 17 | ## C/C++ 18 | -# include "@PKGDATADIR@/c.nanorc" 19 | +include "@PKGDATADIR@/c.nanorc" 20 | 21 | ## Makefiles 22 | -# include "@PKGDATADIR@/makefile.nanorc" 23 | +include "@PKGDATADIR@/makefile.nanorc" 24 | 25 | ## Cascading Style Sheets 26 | # include "@PKGDATADIR@/css.nanorc" 27 | @@ -284,7 +284,7 @@ 28 | # include "@PKGDATADIR@/perl.nanorc" 29 | 30 | ## Python 31 | -# include "@PKGDATADIR@/python.nanorc" 32 | +include "@PKGDATADIR@/python.nanorc" 33 | 34 | ## Ruby 35 | # include "@PKGDATADIR@/ruby.nanorc" 36 | @@ -308,7 +308,7 @@ 37 | # include "@PKGDATADIR@/asm.nanorc" 38 | 39 | ## Bourne shell scripts 40 | -# include "@PKGDATADIR@/sh.nanorc" 41 | +include "@PKGDATADIR@/sh.nanorc" 42 | 43 | ## POV-Ray 44 | # include "@PKGDATADIR@/pov.nanorc" 45 | -- 46 | 2.3.2 47 | 48 | -------------------------------------------------------------------------------- /examples/files/0002-Sample-commit-2.patch: -------------------------------------------------------------------------------- 1 | From d13cefb9538e7f30568f5a8f363257b90f578e5e Mon Sep 17 00:00:00 2001 2 | From: Ikey Doherty 3 | Date: Thu, 23 Apr 2015 22:53:22 +0100 4 | Subject: [PATCH 2/2] Sample commit 2 5 | 6 | --- 7 | doc/nanorc.sample.in | 2 +- 8 | 1 file changed, 1 insertion(+), 1 deletion(-) 9 | 10 | diff --git a/doc/nanorc.sample.in b/doc/nanorc.sample.in 11 | index 83f9c4e..fa0cf57 100644 12 | --- a/doc/nanorc.sample.in 13 | +++ b/doc/nanorc.sample.in 14 | @@ -302,7 +302,7 @@ include "@PKGDATADIR@/python.nanorc" 15 | # include "@PKGDATADIR@/ocaml.nanorc" 16 | 17 | ## AWK 18 | -# include "@PKGDATADIR@/awk.nanorc" 19 | +include "@PKGDATADIR@/awk.nanorc" 20 | 21 | ## Assembler 22 | # include "@PKGDATADIR@/asm.nanorc" 23 | -- 24 | 2.3.2 25 | 26 | -------------------------------------------------------------------------------- /examples/files/series: -------------------------------------------------------------------------------- 1 | 0001-Sample-patch-merely-for-ypkg-testing.patch 2 | 0002-Sample-commit-2.patch -------------------------------------------------------------------------------- /examples/gnome-maps.yml: -------------------------------------------------------------------------------- 1 | name : gnome-maps 2 | version : 3.14.2 3 | release : 1 4 | source : 5 | - http://ftp.gnome.org/pub/GNOME/sources/gnome-maps/3.14/gnome-maps-3.14.2.tar.xz : 92a6488b2632da0d4f9b6c67b9462dfd2dc790617f8b472e46130b8909bc2ab5 6 | homepage : https://wiki.gnome.org/Apps/Maps 7 | license : GPL-2.0 8 | summary : GNOME Maps Application 9 | builddeps : 10 | - pkgconfig(gjs-1.0) 11 | description: | 12 | Maps is a map application for GNOME 3. 13 | setup : | 14 | %configure --disable-static 15 | build : | 16 | %make 17 | install : | 18 | %make_install 19 | -------------------------------------------------------------------------------- /examples/mupen64plus.yml: -------------------------------------------------------------------------------- 1 | name : mupen64plus 2 | version : 2.5 3 | release : 1 4 | source : 5 | - https://github.com/mupen64plus/mupen64plus-core/releases/download/2.5/mupen64plus-core-src-2.5.tar.gz : 59e99a79e7f862517232e3115bd1f72377e3e34da795c118ba617c478c839578 6 | - https://github.com/mupen64plus/mupen64plus-audio-sdl/releases/download/2.5/mupen64plus-audio-sdl-src-2.5.tar.gz : 1c59f8c2d70206350c0b9067a9722cf2dd67e5686994d5a55adbf51ee8812f59 7 | - https://github.com/mupen64plus/mupen64plus-input-sdl/releases/download/2.5/mupen64plus-input-sdl-src-2.5.tar.gz : 65e528667b7f57a307b92abba37a3601ee17e9dd9d303ba7e4163b27f3ef771b 8 | - https://github.com/mupen64plus/mupen64plus-rsp-hle/releases/download/2.5/mupen64plus-rsp-hle-src-2.5.tar.gz : 4b2e11193746e0fbe4dfa78426c1214cf2e42779132eb2f668bf88f498517703 9 | - https://github.com/mupen64plus/mupen64plus-video-glide64mk2/releases/download/2.5/mupen64plus-video-glide64mk2-src-2.5.tar.gz : ef3dae0084e078d843605abdf5039eb8b5dd68ff1410b4fc12bdf19592a9fcb6 10 | - https://github.com/mupen64plus/mupen64plus-video-rice/releases/download/2.5/mupen64plus-video-rice-src-2.5.tar.gz : 969f65b9f42f48bc7e840e4ed1342233e4371c2edca5aa8f36f69c4b20955828 11 | - https://github.com/mupen64plus/mupen64plus-ui-console/releases/download/2.5/mupen64plus-ui-console-src-2.5.tar.gz : 71fee012678ff88f18130e339afd8c5467a2646b7e50da75ba2d5fa342b74a67 12 | license : 13 | - GPL-2.0 14 | summary : N64 emulator 15 | builddeps : 16 | - pkgconfig(libpng) 17 | - pkgconfig(sdl2) 18 | - pkgconfig(gl) 19 | - pkgconfig(glu) 20 | - pkgconfig(speex) 21 | - pkgconfig(samplerate) 22 | - libboost-devel 23 | description : | 24 | Mupen64Plus is a cross-platform plugin-based N64 emulator which is capable of accurately playing many games. Included are four MIPS R4300 CPU emulators, with dynamic recompilers for 32-bit x86 and 64-bit amd64 systems, and necessary plugins for audio, graphical rendering (RDP), signal co-processor (RSP), and input. There is 1 included OpenGL video plugin, called RiceVideo. There are 3 other excellent video plugins being maintained by wahrhaft, called Arachnoid, Glide64, and Z64. 25 | build : | 26 | # Build core first. 27 | %make -C projects/unix all PREFIX=/usr LIBDIR=%libdir% 28 | %make_install -C projects/unix PREFIX=/usr LIBDIR=%libdir% 29 | 30 | # Build the remaining plugins 31 | mkdir plugin_build 32 | pushd plugin_build 33 | for plugin in audio-sdl input-sdl rsp-hle video-glide64mk2 video-rice ui-console; do 34 | tar xvf $sources/mupen64plus-${plugin}-src-${version}.tar.gz 35 | pushd "mupen64plus-${plugin}-src-${version}" 36 | CFLAGS="$CFLAGS -I${installdir}/usr/include/mupen64plus" %make -C projects/unix all PREFIX=/usr LIBDIR=%libdir% APIDIR="$installdir/usr/include" 37 | popd 38 | done 39 | popd 40 | install : | 41 | # Install all the plugins now 42 | pushd plugin_build 43 | for plugin in audio-sdl input-sdl rsp-hle video-glide64mk2 video-rice ui-console; do 44 | pushd "mupen64plus-${plugin}-src-${version}" 45 | %make_install -C projects/unix PREFIX=/usr LIBDIR=%libdir% APIDIR="$installdir/usr/include" 46 | popd 47 | done 48 | popd 49 | -------------------------------------------------------------------------------- /examples/nano.yml: -------------------------------------------------------------------------------- 1 | name : nano 2 | version : 2.3.2 3 | release : 1 4 | homepage : http://www.nano-editor.org 5 | source : 6 | - http://www.nano-editor.org/dist/v2.3/nano-2.3.2.tar.gz : ff309248071486445609ad4269b798262a1324d7503dc09dea289f5b60106be8 7 | license : GPL-2.0 8 | summary : GNU nano is an easy-to-use text editor 9 | builddeps : 10 | - ncurses-devel 11 | description: | 12 | GNU nano is an easy-to-use text editor originally designed 13 | as a replacement for Pico, the ncurses-based editor from the non-free mailer 14 | package Pine (itself now available under the Apache License as Alpine). 15 | setup : | 16 | %apply_patches 17 | %reconfigure --enable-utf8 18 | build : | 19 | %make 20 | install : | 21 | %make_install 22 | install -d %installroot%/etc 23 | install -m 00644 doc/nanorc.sample %installroot%/etc/nanorc 24 | -------------------------------------------------------------------------------- /examples/poppler.yml: -------------------------------------------------------------------------------- 1 | name : poppler 2 | version : 0.34.0 3 | release : 7 4 | source : 5 | - http://poppler.freedesktop.org/poppler-0.34.0.tar.xz : 1ba4ba9a2f9eb1e62ee6d736f4d82be4fc5f6dd177dc2b03febbe2ef2e515cb0 6 | homepage : http://poppler.freedesktop.org/ 7 | license : GPL-2.0 8 | summary : PDF Rendering Library 9 | description : | 10 | PDF Rendering Library 11 | builddeps : 12 | - pkgconfig(freetype2) 13 | - pkgconfig(fontconfig) 14 | - pkgconfig(gdk-pixbuf-2.0) 15 | - pkgconfig(gtk+-3.0) 16 | - pkgconfig(lcms2) 17 | - pkgconfig(libcurl) 18 | - pkgconfig(libpng) 19 | - pkgconfig(libtiff-4) 20 | - pkgconfig(Qt5Core) 21 | - libjpeg-turbo-devel 22 | setup : | 23 | %configure --disable-static \ 24 | --enable-libcurl \ 25 | --enable-libtiff \ 26 | --enable-libjpeg \ 27 | --enable-libpng \ 28 | --enable-poppler-qt5 \ 29 | --enable-poppler-qt4 \ 30 | --enable-xpdf-headers \ 31 | --disable-libopenjpeg 32 | patterns : 33 | - qt5-devel: 34 | - /usr/include/poppler/qt5/ 35 | - /usr/lib64/lib*qt5*.so 36 | - /usr/lib64/pkgconfig/*qt5*.pc 37 | - qt5: 38 | - /usr/lib64/lib*qt5*.so.* 39 | - qt4-devel: 40 | - /usr/include/poppler/qt4/ 41 | - /usr/lib64/lib*qt4*.so 42 | - /usr/lib64/pkgconfig/*qt4*.pc 43 | - qt4: 44 | - /usr/lib64/lib*qt4*.so.* 45 | build : | 46 | %make 47 | install : | 48 | %make_install 49 | -------------------------------------------------------------------------------- /gen_docs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Credit to swupd developers: https://github.com/clearlinux/swupd-client 4 | 5 | MANPAGES="man/ypkg.1 man/ypkg-install-deps.1 man/ypkg-build.1 man/package.yml.5" 6 | 7 | for MANPAGE in ${MANPAGES}; do \ 8 | ronn --roff < ${MANPAGE}.md > ${MANPAGE}; \ 9 | ronn --html < ${MANPAGE}.md > ${MANPAGE}.html; \ 10 | done 11 | -------------------------------------------------------------------------------- /man/ypkg-build.1: -------------------------------------------------------------------------------- 1 | .\" generated with Ronn/v0.7.3 2 | .\" http://github.com/rtomayko/ronn/tree/0.7.3 3 | . 4 | .TH "YPKG\-BUILD" "1" "March 2018" "" "" 5 | . 6 | .SH "NAME" 7 | \fBypkg\-build\fR \- Build Solus ypkg files 8 | . 9 | .SH "SYNOPSIS" 10 | \fBypkg\-build [package\.yml]\fR 11 | . 12 | .SH "DESCRIPTION" 13 | \fBypkg\-build\fR is the main component of the \fBypkg(1)\fR package\. Given a \fBpackage\.yml(5)\fR file, it will attempt to build the package according to the rules, patterns and steps set in the file\. 14 | . 15 | .P 16 | For details on the package format itself, please refer to the \fBpackage\.yml(5)\fR manpage, or the Solus wiki\. 17 | . 18 | .P 19 | Note that you should not use \fBypkg\-build(1)\fR directly unless completely unavoidable\. Instead, you should be using \fBsolbuild(1)\fR for isolated build environments\. 20 | . 21 | .SH "OPTIONS" 22 | The following options are applicable to \fBypkg\-build(1)\fR\. 23 | . 24 | .IP "\(bu" 4 25 | \fB\-h\fR, \fB\-\-help\fR 26 | . 27 | .IP 28 | Print the command line options for \fBypkg(1)\fR and exit\. 29 | . 30 | .IP "\(bu" 4 31 | \fB\-v\fR, \fB\-\-version\fR 32 | . 33 | .IP 34 | Print the \fBypkg(1)\fR version and exit\. 35 | . 36 | .IP "\(bu" 4 37 | \fB\-n\fR, \fB\-\-no\-colors\fR 38 | . 39 | .IP 40 | Disable text colourisation in the output from \fBypkg\fR and all child processes\. 41 | . 42 | .IP "\(bu" 4 43 | \fB\-t\fR, \fB\-\-timestamp\fR 44 | . 45 | .IP 46 | This argument should be a UNIX timestamp, and will be used to set the file timestamps inside the final \fB\.eopkg\fR archive, as well as the container files within that archive\. 47 | . 48 | .IP 49 | Using this option helps achieve a level of reproducability in builds, and this option is passed by \fBsolbuild(1)\fR automatically for ypkg builds\. It will examine the git history and use the UTC UNIX timestamp for the last tag, ensuring the package can be built by any machine using \fBsolbuild(1)\fR and result in an identical package, byte for byte\. 50 | . 51 | .IP "\(bu" 4 52 | \fB\-D\fR, \fB\-\-output\-dir\fR 53 | . 54 | .IP 55 | Set the output directory for \fBypkg\-build(1)\fR 56 | . 57 | .IP "" 0 58 | . 59 | .SH "EXIT STATUS" 60 | On success, 0 is returned\. A non\-zero return code signals a failure\. 61 | . 62 | .SH "COPYRIGHT" 63 | . 64 | .IP "\(bu" 4 65 | Copyright © 2016 Ikey Doherty, License: CC\-BY\-SA\-3\.0 66 | . 67 | .IP "" 0 68 | . 69 | .SH "SEE ALSO" 70 | \fBsolbuild(1)\fR, \fBypkg\-install\-deps(1)\fR, \fBypkg(1)\fR, \fBpackage\.yml(5)\fR 71 | . 72 | .IP "\(bu" 4 73 | https://github\.com/solus\-project/ypkg 74 | . 75 | .IP "\(bu" 4 76 | https://wiki\.solus\-project\.com/Packaging 77 | . 78 | .IP "" 0 79 | . 80 | .SH "NOTES" 81 | Creative Commons Attribution\-ShareAlike 3\.0 Unported 82 | . 83 | .IP "\(bu" 4 84 | http://creativecommons\.org/licenses/by\-sa/3\.0/ 85 | . 86 | .IP "" 0 87 | 88 | -------------------------------------------------------------------------------- /man/ypkg-build.1.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | ypkg-build(1) - Build Solus ypkg files 7 | 44 | 45 | 52 | 53 |
54 | 55 | 65 | 66 |
    67 |
  1. ypkg-build(1)
    2. 68 |
  2. 69 |
  3. ypkg-build(1)
    4. 70 |
71 | 72 |

NAME

73 |

74 | ypkg-build - Build Solus ypkg files 75 |

76 | 77 |

SYNOPSIS

78 | 79 |

ypkg-build <flags> [package.yml]

80 | 81 |

DESCRIPTION

82 | 83 |

ypkg-build is the main component of the ypkg(1) package. Given a package.yml(5) 84 | file, it will attempt to build the package according to the rules, patterns and 85 | steps set in the file.

86 | 87 |

For details on the package format itself, please refer to the package.yml(5) 88 | manpage, or the Solus wiki.

89 | 90 |

Note that you should not use ypkg-build(1) directly unless completely unavoidable. 91 | Instead, you should be using solbuild(1) for isolated build environments.

92 | 93 |

OPTIONS

94 | 95 |

The following options are applicable to ypkg-build(1).

96 | 97 |
    98 |

  • -h, --help

    99 | 100 |

    Print the command line options for ypkg(1) and exit.

    • 101 |

  • -v, --version

    102 | 103 |

    Print the ypkg(1) version and exit.

    • 104 |

  • -n, --no-colors

    105 | 106 |

    Disable text colourisation in the output from ypkg and all child 107 | processes.

    • 108 |

  • -t, --timestamp

    109 | 110 |

    This argument should be a UNIX timestamp, and will be used to set the file 111 | timestamps inside the final .eopkg archive, as well as the container files 112 | within that archive.

    113 | 114 |

    Using this option helps achieve a level of reproducability in builds, and 115 | this option is passed by solbuild(1) automatically for ypkg builds. It 116 | will examine the git history and use the UTC UNIX timestamp for the last 117 | tag, ensuring the package can be built by any machine using solbuild(1) 118 | and result in an identical package, byte for byte.

    • 119 |

  • -D, --output-dir

    120 | 121 |

    Set the output directory for ypkg-build(1)

    • 122 |
123 | 124 | 125 |

EXIT STATUS

126 | 127 |

On success, 0 is returned. A non-zero return code signals a failure.

128 | 129 | 130 | 131 |
    132 |
  • Copyright © 2016 Ikey Doherty, License: CC-BY-SA-3.0
    • 133 |
134 | 135 | 136 |

SEE ALSO

137 | 138 |

solbuild(1), ypkg-install-deps(1), ypkg(1), package.yml(5)

139 | 140 |
    141 |
  • https://github.com/solus-project/ypkg
    • 142 |
  • https://wiki.solus-project.com/Packaging
    • 143 |
144 | 145 | 146 |

NOTES

147 | 148 |

Creative Commons Attribution-ShareAlike 3.0 Unported

149 | 150 |
    151 |
  • http://creativecommons.org/licenses/by-sa/3.0/
    • 152 |
153 | 154 | 155 | 156 |
    157 |
  1. 158 |
  2. March 2018
    3. 159 |
  3. ypkg-build(1)
    4. 160 |
161 | 162 |
163 | 164 | 165 | -------------------------------------------------------------------------------- /man/ypkg-build.1.md: -------------------------------------------------------------------------------- 1 | ypkg-build(1) -- Build Solus ypkg files 2 | ======================================= 3 | 4 | 5 | ## SYNOPSIS 6 | 7 | `ypkg-build [package.yml]` 8 | 9 | 10 | ## DESCRIPTION 11 | 12 | `ypkg-build` is the main component of the `ypkg(1)` package. Given a `package.yml(5)` 13 | file, it will attempt to build the package according to the rules, patterns and 14 | steps set in the file. 15 | 16 | For details on the package format itself, please refer to the `package.yml(5)` 17 | manpage, or the Solus wiki. 18 | 19 | Note that you should not use `ypkg-build(1)` directly unless completely unavoidable. 20 | Instead, you should be using `solbuild(1)` for isolated build environments. 21 | 22 | ## OPTIONS 23 | 24 | The following options are applicable to `ypkg-build(1)`. 25 | 26 | * `-h`, `--help` 27 | 28 | Print the command line options for `ypkg(1)` and exit. 29 | 30 | * `-v`, `--version` 31 | 32 | Print the `ypkg(1)` version and exit. 33 | 34 | * `-n`, `--no-colors` 35 | 36 | Disable text colourisation in the output from `ypkg` and all child 37 | processes. 38 | 39 | * `-t`, `--timestamp` 40 | 41 | This argument should be a UNIX timestamp, and will be used to set the file 42 | timestamps inside the final `.eopkg` archive, as well as the container files 43 | within that archive. 44 | 45 | Using this option helps achieve a level of reproducability in builds, and 46 | this option is passed by `solbuild(1)` automatically for ypkg builds. It 47 | will examine the git history and use the UTC UNIX timestamp for the last 48 | tag, ensuring the package can be built by any machine using `solbuild(1)` 49 | and result in an identical package, byte for byte. 50 | 51 | * `-D`, `--output-dir` 52 | 53 | Set the output directory for `ypkg-build(1)` 54 | 55 | 56 | ## EXIT STATUS 57 | 58 | On success, 0 is returned. A non-zero return code signals a failure. 59 | 60 | 61 | ## COPYRIGHT 62 | 63 | * Copyright © 2016 Ikey Doherty, License: CC-BY-SA-3.0 64 | 65 | 66 | ## SEE ALSO 67 | 68 | `solbuild(1)`, `ypkg-install-deps(1)`, `ypkg(1)`, `package.yml(5)` 69 | 70 | * https://github.com/solus-project/ypkg 71 | * https://wiki.solus-project.com/Packaging 72 | 73 | 74 | ## NOTES 75 | 76 | Creative Commons Attribution-ShareAlike 3.0 Unported 77 | 78 | * http://creativecommons.org/licenses/by-sa/3.0/ 79 | -------------------------------------------------------------------------------- /man/ypkg-install-deps.1: -------------------------------------------------------------------------------- 1 | .\" generated with Ronn/v0.7.3 2 | .\" http://github.com/rtomayko/ronn/tree/0.7.3 3 | . 4 | .TH "YPKG\-INSTALL\-DEPS" "1" "March 2018" "" "" 5 | . 6 | .SH "NAME" 7 | \fBypkg\-install\-deps\fR \- Install build dependencies 8 | . 9 | .SH "SYNOPSIS" 10 | \fBypkg\-install\-deps [package\.yml]\fR 11 | . 12 | .SH "DESCRIPTION" 13 | \fBypkg\-install\-deps\fR will install all of the build dependencies listed in the given \fBpackage\.yml(5)\fR file\. Note that resolution of \fBpkgconfig\fR and \fBpkgconfig32\fR dependencies is handled automatically\. 14 | . 15 | .SH "OPTIONS" 16 | The following options are applicable to \fBypkg\-install\-deps(1)\fR\. 17 | . 18 | .IP "\(bu" 4 19 | \fB\-h\fR, \fB\-\-help\fR 20 | . 21 | .IP 22 | Print the command line options for \fBypkg(1)\fR and exit\. 23 | . 24 | .IP "\(bu" 4 25 | \fB\-v\fR, \fB\-\-version\fR 26 | . 27 | .IP 28 | Print the \fBypkg(1)\fR version and exit\. 29 | . 30 | .IP "\(bu" 4 31 | \fB\-n\fR, \fB\-\-no\-colors\fR 32 | . 33 | .IP 34 | Disable text colourisation in the output from \fBypkg\fR and all child processes\. 35 | . 36 | .IP "\(bu" 4 37 | \fB\-D\fR, \fB\-\-output\-dir\fR 38 | . 39 | .IP 40 | This option is ignored by \fBypkg\-install\-deps(1)\fR\. It is provided simply for compatibility in scripting to allow \fBypkg(1)\fR to pass arguments forward for the duration of the session\. 41 | . 42 | .IP "\(bu" 4 43 | \fB\-f\fR, \fB\-\-force\fR 44 | . 45 | .IP 46 | Force the installation of package dependencies, which will bypass any prompting by ypkg\. The default behaviour is to prompt before installing packages\. 47 | . 48 | .IP "" 0 49 | . 50 | .SH "EXIT STATUS" 51 | On success, 0 is returned\. A non\-zero return code signals a failure\. 52 | . 53 | .SH "COPYRIGHT" 54 | . 55 | .IP "\(bu" 4 56 | Copyright © 2016 Ikey Doherty, License: CC\-BY\-SA\-3\.0 57 | . 58 | .IP "" 0 59 | . 60 | .SH "SEE ALSO" 61 | \fBsolbuild(1)\fR, \fBypkg(1)\fR \fBypkg\-build(1)\fR, \fBpackage\.yml(5)\fR 62 | . 63 | .IP "\(bu" 4 64 | https://github\.com/solus\-project/ypkg 65 | . 66 | .IP "\(bu" 4 67 | https://wiki\.solus\-project\.com/Packaging 68 | . 69 | .IP "" 0 70 | . 71 | .SH "NOTES" 72 | Creative Commons Attribution\-ShareAlike 3\.0 Unported 73 | . 74 | .IP "\(bu" 4 75 | http://creativecommons\.org/licenses/by\-sa/3\.0/ 76 | . 77 | .IP "" 0 78 | 79 | -------------------------------------------------------------------------------- /man/ypkg-install-deps.1.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | ypkg-install-deps(1) - Install build dependencies 7 | 44 | 45 | 52 | 53 |
54 | 55 | 65 | 66 |
    67 |
  1. ypkg-install-deps(1)
    2. 68 |
  2. 69 |
  3. ypkg-install-deps(1)
    4. 70 |
71 | 72 |

NAME

73 |

74 | ypkg-install-deps - Install build dependencies 75 |

76 | 77 |

SYNOPSIS

78 | 79 |

ypkg-install-deps <flags> [package.yml]

80 | 81 |

DESCRIPTION

82 | 83 |

ypkg-install-deps will install all of the build dependencies listed in the 84 | given package.yml(5) file. Note that resolution of pkgconfig and pkgconfig32 85 | dependencies is handled automatically.

86 | 87 |

OPTIONS

88 | 89 |

The following options are applicable to ypkg-install-deps(1).

90 | 91 |
    92 |

  • -h, --help

    93 | 94 |

    Print the command line options for ypkg(1) and exit.

    • 95 |

  • -v, --version

    96 | 97 |

    Print the ypkg(1) version and exit.

    • 98 |

  • -n, --no-colors

    99 | 100 |

    Disable text colourisation in the output from ypkg and all child 101 | processes.

    • 102 |

  • -D, --output-dir

    103 | 104 |

    This option is ignored by ypkg-install-deps(1). It is provided simply 105 | for compatibility in scripting to allow ypkg(1) to pass arguments forward 106 | for the duration of the session.

    • 107 |

  • -f, --force

    108 | 109 |

    Force the installation of package dependencies, which will bypass any 110 | prompting by ypkg. The default behaviour is to prompt before installing 111 | packages.

    • 112 |
113 | 114 | 115 |

EXIT STATUS

116 | 117 |

On success, 0 is returned. A non-zero return code signals a failure.

118 | 119 | 120 | 121 |
    122 |
  • Copyright © 2016 Ikey Doherty, License: CC-BY-SA-3.0
    • 123 |
124 | 125 | 126 |

SEE ALSO

127 | 128 |

solbuild(1), ypkg(1) ypkg-build(1), package.yml(5)

129 | 130 |
    131 |
  • https://github.com/solus-project/ypkg
    • 132 |
  • https://wiki.solus-project.com/Packaging
    • 133 |
134 | 135 | 136 |

NOTES

137 | 138 |

Creative Commons Attribution-ShareAlike 3.0 Unported

139 | 140 |
    141 |
  • http://creativecommons.org/licenses/by-sa/3.0/
    • 142 |
143 | 144 | 145 | 146 |
    147 |
  1. 148 |
  2. March 2018
    3. 149 |
  3. ypkg-install-deps(1)
    4. 150 |
151 | 152 |
153 | 154 | 155 | -------------------------------------------------------------------------------- /man/ypkg-install-deps.1.md: -------------------------------------------------------------------------------- 1 | ypkg-install-deps(1) -- Install build dependencies 2 | ================================================== 3 | 4 | 5 | ## SYNOPSIS 6 | 7 | `ypkg-install-deps [package.yml]` 8 | 9 | 10 | ## DESCRIPTION 11 | 12 | `ypkg-install-deps` will install all of the build dependencies listed in the 13 | given `package.yml(5)` file. Note that resolution of `pkgconfig` and `pkgconfig32` 14 | dependencies is handled automatically. 15 | 16 | ## OPTIONS 17 | 18 | The following options are applicable to `ypkg-install-deps(1)`. 19 | 20 | * `-h`, `--help` 21 | 22 | Print the command line options for `ypkg(1)` and exit. 23 | 24 | * `-v`, `--version` 25 | 26 | Print the `ypkg(1)` version and exit. 27 | 28 | * `-n`, `--no-colors` 29 | 30 | Disable text colourisation in the output from `ypkg` and all child 31 | processes. 32 | 33 | * `-D`, `--output-dir` 34 | 35 | This option is ignored by `ypkg-install-deps(1)`. It is provided simply 36 | for compatibility in scripting to allow `ypkg(1)` to pass arguments forward 37 | for the duration of the session. 38 | 39 | * `-f`, `--force` 40 | 41 | Force the installation of package dependencies, which will bypass any 42 | prompting by ypkg. The default behaviour is to prompt before installing 43 | packages. 44 | 45 | 46 | ## EXIT STATUS 47 | 48 | On success, 0 is returned. A non-zero return code signals a failure. 49 | 50 | 51 | ## COPYRIGHT 52 | 53 | * Copyright © 2016 Ikey Doherty, License: CC-BY-SA-3.0 54 | 55 | 56 | ## SEE ALSO 57 | 58 | `solbuild(1)`, `ypkg(1)` `ypkg-build(1)`, `package.yml(5)` 59 | 60 | * https://github.com/solus-project/ypkg 61 | * https://wiki.solus-project.com/Packaging 62 | 63 | 64 | ## NOTES 65 | 66 | Creative Commons Attribution-ShareAlike 3.0 Unported 67 | 68 | * http://creativecommons.org/licenses/by-sa/3.0/ 69 | -------------------------------------------------------------------------------- /man/ypkg.1: -------------------------------------------------------------------------------- 1 | .\" generated with Ronn/v0.7.3 2 | .\" http://github.com/rtomayko/ronn/tree/0.7.3 3 | . 4 | .TH "YPKG" "1" "March 2018" "" "" 5 | . 6 | .SH "NAME" 7 | \fBypkg\fR \- Build Solus ypkg files 8 | . 9 | .SH "SYNOPSIS" 10 | \fBypkg [package\.yml]\fR 11 | . 12 | .SH "DESCRIPTION" 13 | \fBypkg\fR is the main entry point into \fBpackage\.yml(5)\fR program\. It is a stub that will first call out to \fBypkg\-install\-deps(1)\fR before passing off to \fBypkg\-build(1)\fR\. See those manpages for more details\. 14 | . 15 | .P 16 | Note that you should not use \fBypkg(1)\fR directly unless completely unavoidable\. Instead, you should be using \fBsolbuild(1)\fR for isolated build environments\. 17 | . 18 | .SH "OPTIONS" 19 | The following options are applicable to \fBypkg(1)\fR\. 20 | . 21 | .IP "\(bu" 4 22 | \fB\-h\fR, \fB\-\-help\fR 23 | . 24 | .IP 25 | Print the command line options for \fBypkg(1)\fR and exit\. 26 | . 27 | .IP "\(bu" 4 28 | \fB\-v\fR, \fB\-\-version\fR 29 | . 30 | .IP 31 | Print the \fBypkg(1)\fR version and exit\. 32 | . 33 | .IP "\(bu" 4 34 | \fB\-n\fR, \fB\-\-no\-colors\fR 35 | . 36 | .IP 37 | Disable text colourisation in the output from \fBypkg\fR and all child processes\. 38 | . 39 | .IP "\(bu" 4 40 | \fB\-D\fR, \fB\-\-output\-dir\fR 41 | . 42 | .IP 43 | Set the output directory for \fBypkg\-build(1)\fR 44 | . 45 | .IP "\(bu" 4 46 | \fB\-f\fR, \fB\-\-force\fR 47 | . 48 | .IP 49 | Force the installation of package dependencies, which will bypass any prompting by ypkg\. The default behaviour is to prompt before installing packages\. 50 | . 51 | .IP "" 0 52 | . 53 | .SH "EXIT STATUS" 54 | On success, 0 is returned\. A non\-zero return code signals a failure\. 55 | . 56 | .SH "COPYRIGHT" 57 | . 58 | .IP "\(bu" 4 59 | Copyright © 2016 Ikey Doherty, License: CC\-BY\-SA\-3\.0 60 | . 61 | .IP "" 0 62 | . 63 | .SH "SEE ALSO" 64 | \fBsolbuild(1)\fR, \fBypkg\-install\-deps(1)\fR, \fBypkg\-build(1)\fR, \fBpackage\.yml(5)\fR 65 | . 66 | .IP "\(bu" 4 67 | https://github\.com/solus\-project/ypkg 68 | . 69 | .IP "\(bu" 4 70 | https://wiki\.solus\-project\.com/Packaging 71 | . 72 | .IP "" 0 73 | . 74 | .SH "NOTES" 75 | Creative Commons Attribution\-ShareAlike 3\.0 Unported 76 | . 77 | .IP "\(bu" 4 78 | http://creativecommons\.org/licenses/by\-sa/3\.0/ 79 | . 80 | .IP "" 0 81 | 82 | -------------------------------------------------------------------------------- /man/ypkg.1.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | ypkg(1) - Build Solus ypkg files 7 | 44 | 45 | 52 | 53 |
54 | 55 | 65 | 66 |
    67 |
  1. ypkg(1)
    2. 68 |
  2. 69 |
  3. ypkg(1)
    4. 70 |
71 | 72 |

NAME

73 |

74 | ypkg - Build Solus ypkg files 75 |

76 | 77 |

SYNOPSIS

78 | 79 |

ypkg <flags> [package.yml]

80 | 81 |

DESCRIPTION

82 | 83 |

ypkg is the main entry point into package.yml(5) program. It is a stub that 84 | will first call out to ypkg-install-deps(1) before passing off to ypkg-build(1). 85 | See those manpages for more details.

86 | 87 |

Note that you should not use ypkg(1) directly unless completely unavoidable. 88 | Instead, you should be using solbuild(1) for isolated build environments.

89 | 90 |

OPTIONS

91 | 92 |

The following options are applicable to ypkg(1).

93 | 94 |
    95 |

  • -h, --help

    96 | 97 |

    Print the command line options for ypkg(1) and exit.

    • 98 |

  • -v, --version

    99 | 100 |

    Print the ypkg(1) version and exit.

    • 101 |

  • -n, --no-colors

    102 | 103 |

    Disable text colourisation in the output from ypkg and all child 104 | processes.

    • 105 |

  • -D, --output-dir

    106 | 107 |

    Set the output directory for ypkg-build(1)

    • 108 |

  • -f, --force

    109 | 110 |

    Force the installation of package dependencies, which will bypass any 111 | prompting by ypkg. The default behaviour is to prompt before installing 112 | packages.

    • 113 |
114 | 115 | 116 |

EXIT STATUS

117 | 118 |

On success, 0 is returned. A non-zero return code signals a failure.

119 | 120 | 121 | 122 |
    123 |
  • Copyright © 2016 Ikey Doherty, License: CC-BY-SA-3.0
    • 124 |
125 | 126 | 127 |

SEE ALSO

128 | 129 |

solbuild(1), ypkg-install-deps(1), ypkg-build(1), package.yml(5)

130 | 131 |
    132 |
  • https://github.com/solus-project/ypkg
    • 133 |
  • https://wiki.solus-project.com/Packaging
    • 134 |
135 | 136 | 137 |

NOTES

138 | 139 |

Creative Commons Attribution-ShareAlike 3.0 Unported

140 | 141 |
    142 |
  • http://creativecommons.org/licenses/by-sa/3.0/
    • 143 |
144 | 145 | 146 | 147 |
    148 |
  1. 149 |
  2. March 2018
    3. 150 |
  3. ypkg(1)
    4. 151 |
152 | 153 |
154 | 155 | 156 | -------------------------------------------------------------------------------- /man/ypkg.1.md: -------------------------------------------------------------------------------- 1 | ypkg(1) -- Build Solus ypkg files 2 | ================================= 3 | 4 | 5 | ## SYNOPSIS 6 | 7 | `ypkg [package.yml]` 8 | 9 | 10 | ## DESCRIPTION 11 | 12 | `ypkg` is the main entry point into `package.yml(5)` program. It is a stub that 13 | will first call out to `ypkg-install-deps(1)` before passing off to `ypkg-build(1)`. 14 | See those manpages for more details. 15 | 16 | Note that you should not use `ypkg(1)` directly unless completely unavoidable. 17 | Instead, you should be using `solbuild(1)` for isolated build environments. 18 | 19 | ## OPTIONS 20 | 21 | The following options are applicable to `ypkg(1)`. 22 | 23 | * `-h`, `--help` 24 | 25 | Print the command line options for `ypkg(1)` and exit. 26 | 27 | * `-v`, `--version` 28 | 29 | Print the `ypkg(1)` version and exit. 30 | 31 | * `-n`, `--no-colors` 32 | 33 | Disable text colourisation in the output from `ypkg` and all child 34 | processes. 35 | 36 | * `-D`, `--output-dir` 37 | 38 | Set the output directory for `ypkg-build(1)` 39 | 40 | * `-f`, `--force` 41 | 42 | Force the installation of package dependencies, which will bypass any 43 | prompting by ypkg. The default behaviour is to prompt before installing 44 | packages. 45 | 46 | 47 | ## EXIT STATUS 48 | 49 | On success, 0 is returned. A non-zero return code signals a failure. 50 | 51 | 52 | ## COPYRIGHT 53 | 54 | * Copyright © 2016 Ikey Doherty, License: CC-BY-SA-3.0 55 | 56 | 57 | ## SEE ALSO 58 | 59 | `solbuild(1)`, `ypkg-install-deps(1)`, `ypkg-build(1)`, `package.yml(5)` 60 | 61 | * https://github.com/solus-project/ypkg 62 | * https://wiki.solus-project.com/Packaging 63 | 64 | ## NOTES 65 | 66 | Creative Commons Attribution-ShareAlike 3.0 Unported 67 | 68 | * http://creativecommons.org/licenses/by-sa/3.0/ 69 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | import os 2 | from setuptools import setup 3 | 4 | setup( 5 | name = "ypkg2", 6 | version = "27.0.0", 7 | author = "Ikey Doherty", 8 | author_email = "ikey@solus-project.com", 9 | description = ("Solus YPKG build Tool"), 10 | license = "GPL-3.0", 11 | keywords = "example documentation tutorial", 12 | url = "https://github.com/solus-project/ypkg", 13 | packages=['ypkg2'], 14 | scripts=['ypkg', 'ypkg-install-deps', 'ypkg-gen-history', 'ypkg-build', 'ybump', 'yupdate'], 15 | classifiers=[ 16 | "License :: OSI Approved :: GPL-3.0 License", 17 | ], 18 | package_data={'ypkg2': ['rc.yml']}, 19 | data_files = [("/usr/share/man/man1", ["man/ypkg.1", "man/ypkg-build.1", "man/ypkg-install-deps.1"]), 20 | ("/usr/share/man/man5", ["man/package.yml.5"])] 21 | ) 22 | -------------------------------------------------------------------------------- /test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | pep8 ypkg-build ypkg2/*.py ypkg-gen-history ypkg-install-deps ypkg || exit 1 4 | #for item in examples/*.yml ; do 5 | # python -m ypkg2.main $item || exit 1 6 | #done 7 | 8 | fakeroot ./ypkg-build examples/nano.yml || exit 1 9 | -------------------------------------------------------------------------------- /ybump: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # -*- coding: utf-8 -*- 3 | # 4 | # ybump.py 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software; you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation; either version 2 of the License, or 11 | # (at your option) any later version. 12 | # 13 | import sys 14 | import ruamel.yaml 15 | 16 | 17 | def usage(msg=None, ex=1): 18 | if msg: 19 | print(msg) 20 | else: 21 | print("Usage: %s file.yml" % sys.argv[0]) 22 | sys.exit(ex) 23 | 24 | if __name__ == "__main__": 25 | if len(sys.argv) != 2: 26 | usage() 27 | 28 | with open(sys.argv[1]) as fp: 29 | data = ruamel.yaml.round_trip_load(fp) 30 | data['release'] += 1 31 | try: 32 | with open(sys.argv[1], 'w') as fp: 33 | ruamel.yaml.round_trip_dump( 34 | data, fp, indent=4, block_seq_indent=4, 35 | top_level_colon_align=True, prefix_colon=' ') 36 | except Exception as e: 37 | print("Error writing file, may need to reset it.") 38 | print(e) 39 | sys.exit(1) 40 | -------------------------------------------------------------------------------- /ypkg: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from ypkg2 import console_ui 15 | from ypkg2.main import show_version 16 | import subprocess 17 | import sys 18 | import os 19 | import argparse 20 | 21 | 22 | def main(): 23 | parser = argparse.ArgumentParser(description="Ypkg") 24 | parser.add_argument("-n", "--no-colors", help="Disable color output", 25 | action="store_true") 26 | parser.add_argument("-v", "--version", action="store_true", 27 | help="Show version information and exit") 28 | parser.add_argument("-f", "--force", help="Force install dependencies, " 29 | "i.e. no prompt", action="store_true") 30 | parser.add_argument("-D", "--output-dir", type=str, 31 | help="Set the output directory for resulting files") 32 | # Main file 33 | parser.add_argument("filename", help="Path to the ypkg YAML file", 34 | nargs='?') 35 | 36 | args = parser.parse_args() 37 | # Kill colors 38 | if args.no_colors: 39 | console_ui.allow_colors = False 40 | # Show version 41 | if args.version: 42 | show_version() 43 | 44 | # Grab filename 45 | if not args.filename: 46 | console_ui.emit_error("Error", "Please provide a filename") 47 | print("") 48 | parser.print_help() 49 | sys.exit(1) 50 | 51 | needFakeroot = True 52 | if os.geteuid() == 0: 53 | if "FAKED_MODE" not in os.environ: 54 | needFakeroot = False 55 | 56 | args = " ".join(sys.argv[1:]) 57 | vargs = sys.argv[1:] 58 | cargs = " ".join(filter(lambda x: x != "-f" and x != "--force", vargs)) 59 | try: 60 | subprocess.check_call("ypkg-install-deps {}".format(args), shell=True) 61 | if needFakeroot: 62 | sub = "fakeroot " 63 | else: 64 | sub = "" 65 | subprocess.check_call("{}ypkg-build {}".format(sub, cargs), shell=True) 66 | except Exception as e: 67 | print(e) 68 | sys.exit(1) 69 | 70 | 71 | if __name__ == "__main__": 72 | main() 73 | -------------------------------------------------------------------------------- /ypkg-build: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from ypkg2.main import main 15 | from ypkg2 import console_ui 16 | 17 | 18 | if __name__ == "__main__": 19 | main() 20 | -------------------------------------------------------------------------------- /ypkg-gen-history: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from ypkg2.ypkgspec import YpkgSpec, PackageHistory 15 | from ypkg2.main import show_version 16 | 17 | import subprocess 18 | import sys 19 | import os 20 | import pisi.specfile 21 | import re 22 | import argparse 23 | 24 | from yaml import load as yaml_load 25 | try: 26 | from yaml import CLoader as Loader 27 | except Exception as e: 28 | from yaml import Loader 29 | 30 | MAX_HISTORY_LEN = 10 31 | 32 | cve_hit = re.compile(r".*(CVE\-[0-9]+\-[0-9]+).*") 33 | 34 | 35 | class CommiterInfo: 36 | 37 | name = None 38 | email = None 39 | date = None 40 | subject = None 41 | body = None 42 | 43 | 44 | def get_git_tags(wdir): 45 | cmd = "git -C \"{}\" tag --sort=-refname".format(wdir) 46 | 47 | ret = set() 48 | 49 | try: 50 | out = subprocess.check_output(cmd, shell=True) 51 | except Exception as e: 52 | return None 53 | 54 | for i in out.split("\n"): 55 | i = i.strip() 56 | ret.add(i) 57 | return sorted(list(ret)) 58 | 59 | 60 | def get_yml_at_tag(wdir, tag): 61 | cmd = "git -C \"{}\" show {}:package.yml".format(wdir, tag) 62 | 63 | ret = None 64 | try: 65 | out = subprocess.check_output(cmd, shell=True) 66 | except Exception as e: 67 | return None 68 | yml = YpkgSpec() 69 | 70 | try: 71 | yaml_data = yaml_load(out, Loader=Loader) 72 | except Exception as e: 73 | return False 74 | 75 | if not yml.load_from_data(yaml_data): 76 | return None 77 | return yml 78 | 79 | 80 | def get_commiter_infos(wdir, tag): 81 | fmt = "%an\n%ae\n%ad\n%s\n%b" 82 | cmd = "git -C \"{}\" log --pretty=format:\"{}\" {} -1 --date=iso". \ 83 | format(wdir, fmt, tag) 84 | 85 | out = None 86 | try: 87 | out = subprocess.check_output(cmd, shell=True) 88 | except Exception as e: 89 | return None 90 | 91 | splits = out.split("\n") 92 | if len(splits) < 4: 93 | return None 94 | com = CommiterInfo() 95 | com.name = splits[0].strip() 96 | com.email = splits[1].strip() 97 | com.date = splits[2].split(" ")[0].strip() 98 | com.subject = splits[3].strip() 99 | if len(splits) > 4: 100 | com.body = "\n".join(splits[4:]) 101 | 102 | return com 103 | 104 | 105 | def main(): 106 | parser = argparse.ArgumentParser(description="Ypkg History Generator") 107 | parser.add_argument("-v", "--version", action="store_true", 108 | help="Show version information and exit") 109 | parser.add_argument("-D", "--output-dir", type=str, 110 | help="Set the output directory for resulting files") 111 | # Main file 112 | parser.add_argument("filename", help="Path to the ypkg YAML file", 113 | nargs='?') 114 | 115 | args = parser.parse_args() 116 | if args.version: 117 | show_version() 118 | 119 | if not args.filename: 120 | print("Fatal: No filename provided") 121 | sys.exit(1) 122 | 123 | yml = os.path.abspath(args.filename) 124 | wdir = os.path.dirname(yml) 125 | 126 | # check git exists 127 | fp = os.path.join(wdir, ".git") 128 | if not os.path.exists(fp): 129 | print("Debug: Skipping non git tree: {}".format(wdir)) 130 | sys.exit(0) 131 | 132 | outputDir = wdir 133 | if args.output_dir: 134 | od = args.output_dir 135 | if not os.path.exists(args.output_dir): 136 | print("{} does not exist".format(od)) 137 | sys.exit(1) 138 | outputDir = od 139 | outputDir = os.path.abspath(outputDir) 140 | 141 | tags = get_git_tags(wdir) 142 | 143 | history = list() 144 | 145 | for tag in tags: 146 | tag = tag.strip() 147 | if tag == "": 148 | continue 149 | spec = get_yml_at_tag(wdir, tag) 150 | if not spec: 151 | continue 152 | info = get_commiter_infos(wdir, tag) 153 | if not info: 154 | continue 155 | history.append((spec, info)) 156 | 157 | history = sorted(history, key=lambda x: x[0].pkg_release, reverse=True) 158 | 159 | if len(history) > MAX_HISTORY_LEN: 160 | history = history[0:MAX_HISTORY_LEN] 161 | 162 | hist = os.path.join(outputDir, "history.xml") 163 | 164 | hist_obj = PackageHistory() 165 | for i in history: 166 | com = i[1] 167 | spec = i[0] 168 | update = pisi.specfile.Update() 169 | update.name = unicode(com.name) 170 | update.email = com.email 171 | update.version = spec.pkg_version 172 | update.release = str(spec.pkg_release) 173 | update.date = com.date 174 | comment = com.subject 175 | if com.body: 176 | comment += "\n" + com.body 177 | update.comment = comment 178 | for word in comment.split(): 179 | if cve_hit.match(word): 180 | update.type = "security" 181 | break 182 | hist_obj.history.append(update) 183 | 184 | hist_obj.write(hist) 185 | 186 | if __name__ == "__main__": 187 | main() 188 | -------------------------------------------------------------------------------- /ypkg-install-deps: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from ypkg2 import console_ui 15 | from ypkg2.ypkgspec import YpkgSpec 16 | from pisi.db.filesdb import FilesDB 17 | from pisi.db.installdb import InstallDB 18 | from pisi.db.packagedb import PackageDB 19 | from ypkg2 import pkgconfig_dep, pkgconfig32_dep 20 | from ypkg2.main import show_version 21 | 22 | import sys 23 | import os 24 | import subprocess 25 | import argparse 26 | 27 | 28 | def main(): 29 | spec = YpkgSpec() 30 | 31 | parser = argparse.ArgumentParser(description="Ypkg Build Dep Installer") 32 | parser.add_argument("-n", "--no-colors", help="Disable color output", 33 | action="store_true") 34 | parser.add_argument("-v", "--version", action="store_true", 35 | help="Show version information and exit") 36 | parser.add_argument("-f", "--force", help="Force install dependencies, " 37 | "i.e. no prompt", action="store_true") 38 | parser.add_argument("-D", "--output-dir", type=str, 39 | help="Ignored in ypkg-install-deps") 40 | # Main file 41 | parser.add_argument("filename", help="Path to the ypkg YAML file") 42 | 43 | args = parser.parse_args() 44 | # Kill colors 45 | if args.no_colors: 46 | console_ui.allow_colors = False 47 | # Show version 48 | if args.version: 49 | show_version() 50 | 51 | # Grab filename 52 | if not args.filename: 53 | console_ui.emit_error("Error", "Please provide a filename") 54 | print("") 55 | parser.print_help() 56 | sys.exit(1) 57 | 58 | if not spec.load_from_path(args.filename): 59 | sys.exit(1) 60 | 61 | pc32deps = set() 62 | pcdeps = set() 63 | ndeps = set() 64 | 65 | idb = InstallDB() 66 | pdb = PackageDB() 67 | fdb = FilesDB() 68 | 69 | console_ui.emit_info("BuildDep", "Checking build-deps for {}-{}-{}". 70 | format(spec.pkg_name, spec.pkg_version, 71 | spec.pkg_release)) 72 | 73 | if spec.pkg_builddeps: 74 | for dep in spec.pkg_builddeps: 75 | em32 = pkgconfig32_dep.match(dep) 76 | if em32: 77 | pc32deps.add(em32.group(1)) 78 | continue 79 | em = pkgconfig_dep.match(dep) 80 | if em: 81 | pcdeps.add(em.group(1)) 82 | continue 83 | if not idb.has_package(dep): 84 | ndeps.add(dep) 85 | 86 | # Get the global known pkgconfig providers 87 | pkgConfigs, pkgConfigs32 = pdb.get_pkgconfig_providers() 88 | 89 | for i in pc32deps: 90 | local = False 91 | pkg = None 92 | 93 | # Try global pkgconfig names first. 94 | if i in pkgConfigs32: 95 | pkg = pdb.get_package(pkgConfigs32[i]) 96 | elif i in pkgConfigs: 97 | pkg = pdb.get_package(pkgConfigs[i]) 98 | 99 | # Try the filesdb 100 | if not pkg: 101 | local = True 102 | nom = fdb.get_pkgconfig32_provider(i) 103 | if nom: 104 | pkg = idb.get_package_by_pkgconfig32(nom[0]) 105 | if not pkg: 106 | nom = fdb.get_pkgconfig_provider(i) 107 | if nom: 108 | pkg = idb.get_package_by_pkgconfig(nom[0]) 109 | 110 | if local: 111 | console_ui.emit_warning("pkgconfig32:{}".format(i), 112 | "This dependency is not in any repo") 113 | if not pkg: 114 | console_ui.emit_error("BuildDep", "pkgconfig32({}) build dep " 115 | "doesn't exist in the repository.".format(i)) 116 | sys.exit(1) 117 | if not idb.has_package(pkg.name): 118 | ndeps.add(pkg.name) 119 | 120 | for i in pcdeps: 121 | local = False 122 | pkg = None 123 | if i in pkgConfigs: 124 | pkg = pdb.get_package(pkgConfigs[i]) 125 | if not pkg: 126 | nom = fdb.get_pkgconfig_provider(i) 127 | if nom: 128 | pkg = idb.get_package_by_pkgconfig(nom[0]) 129 | local = True 130 | if local: 131 | console_ui.emit_warning("pkgconfig:{}".format(i), 132 | "This dependency is not in any repo") 133 | if not pkg: 134 | console_ui.emit_error("BuildDep", "pkgconfig({}) build dep" 135 | " does not exist in the repository.". 136 | format(i)) 137 | sys.exit(1) 138 | if not idb.has_package(pkg.name): 139 | ndeps.add(pkg.name) 140 | 141 | if len(ndeps) < 1: 142 | console_ui.emit_success("BuildDep", "All build deps satisfied") 143 | sys.exit(0) 144 | 145 | if os.geteuid() != 0: 146 | cmd = "sudo eopkg install {}".format(" ".join(ndeps)) 147 | else: 148 | cmd = "eopkg install {}".format(" ".join(ndeps)) 149 | 150 | if args.force: 151 | cmd += " --yes-all" 152 | 153 | if args.no_colors: 154 | cmd += " -N" 155 | 156 | invalid = [x for x in ndeps if not pdb.has_package(x)] 157 | if len(invalid) > 0: 158 | console_ui.emit_error("BuildDep", "Unknown build deps: {}". 159 | format(" ".join(invalid))) 160 | sys.exit(1) 161 | 162 | console_ui.emit_info("BuildDep", "Requesting installation of: {}". 163 | format(", ".join(ndeps))) 164 | try: 165 | subprocess.check_call(cmd, shell=True) 166 | except Exception as e: 167 | console_ui.emit_error("BuildDep", "Failed to install build deps") 168 | sys.exit(1) 169 | 170 | sys.exit(0) 171 | 172 | if __name__ == "__main__": 173 | main() 174 | -------------------------------------------------------------------------------- /ypkg2/__init__.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from .ui import YpkgUI 15 | 16 | import re 17 | 18 | 19 | global console_ui 20 | 21 | console_ui = YpkgUI() 22 | 23 | 24 | def remove_prefix(fpath, prefix): 25 | if fpath.startswith(prefix): 26 | fpath = fpath[len(prefix)+1:] 27 | if fpath[0] != '/': 28 | fpath = "/" + fpath 29 | return fpath 30 | 31 | pkgconfig32_dep = re.compile("^pkgconfig32\((.*)\)$") 32 | pkgconfig_dep = re.compile("^pkgconfig\((.*)\)$") 33 | 34 | 35 | global packager_name 36 | global packager_email 37 | 38 | 39 | packager_name = "Automated Package Build" 40 | packager_email = "no.email.set.in.config" 41 | 42 | EMUL32PC = "/usr/lib32/pkgconfig:/usr/share/pkgconfig:/usr/lib/pkgconfig" 43 | -------------------------------------------------------------------------------- /ypkg2/dependencies.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | from pisi.db.installdb import InstallDB 16 | from pisi.db.packagedb import PackageDB 17 | from pisi.db.filesdb import FilesDB 18 | import os 19 | 20 | # Provided historically for our pre-glvnd architecture. 21 | # Technically speaking this isn't required anymore, but lets just 22 | # play it safe for those directly using ypkg on old NVIDIA drivers. 23 | ExceptionRules = [ 24 | "libEGL.so", 25 | "libEGL.so.1", 26 | "libEGL.so.1.0.0", 27 | "libGLESv1_CM.so", 28 | "libGLESv1_CM.so.1", 29 | "libGLESv1_CM.so.1.1.0", 30 | "libGLESv2.so", 31 | "libGLESv2.so.2", 32 | "libGLESv2.so.2.0.0", 33 | "libGL.so", 34 | "libGL.so.1", 35 | "libGL.so.1.0.0", 36 | "libGL.so.1.2.0", 37 | "libglx.so", 38 | "libglx.so.1", 39 | ] 40 | 41 | 42 | class DependencyResolver: 43 | 44 | idb = None 45 | pdb = None 46 | fdb = None 47 | 48 | global_rpaths = set() 49 | global_rpaths32 = set() 50 | global_sonames = dict() 51 | global_sonames32 = dict() 52 | global_pkgconfigs = dict() 53 | global_pkgconfig32s = dict() 54 | global_kernels = dict() 55 | gene = None 56 | 57 | bindeps_cache = dict() 58 | bindeps_emul32 = dict() 59 | 60 | pkgconfig_cache = dict() 61 | pkgconfig32_cache = dict() 62 | 63 | files_cache = dict() 64 | 65 | kernel_cache = dict() 66 | 67 | deadends = dict() 68 | 69 | # Cached from packagedb 70 | pkgConfigs = None 71 | pkgConfigs32 = None 72 | 73 | def search_file(self, fname): 74 | if fname[0] == '/': 75 | fname = fname[1:] 76 | if fname in self.deadends: 77 | return None 78 | if self.fdb.has_file(fname): 79 | return self.fdb.get_file(fname) 80 | # Nasty file conflict crap happened on update and the filesdb 81 | # is now inconsistent .. 82 | ret = self.fdb.search_file(fname) 83 | if len(ret) == 1: 84 | return ret[0] 85 | # Just blacklist further lookups here 86 | self.deadends[fname] = 0 87 | return None 88 | 89 | def __init__(self): 90 | """ Allows us to do look ups on all packages """ 91 | self.idb = InstallDB() 92 | self.pdb = PackageDB() 93 | self.fdb = FilesDB() 94 | 95 | # Cache the pkgconfigs known in the pdb 96 | self.pkgConfigs, self.pkgConfigs32 = self.pdb.get_pkgconfig_providers() 97 | 98 | def get_symbol_provider(self, info, symbol): 99 | """ Grab the symbol from the local packages """ 100 | if info.emul32: 101 | tgtMap = self.global_sonames32 102 | rPaths = self.global_rpaths32 103 | else: 104 | tgtMap = self.global_sonames 105 | rPaths = self.global_rpaths 106 | 107 | if symbol in tgtMap: 108 | pkgname = tgtMap[symbol] 109 | return self.ctx.spec.get_package_name(pkgname) 110 | 111 | # Check if its in any rpath 112 | for rpath in rPaths: 113 | fpath = os.path.join(rpath, symbol) 114 | pkg = self.gene.get_file_owner(fpath) 115 | if pkg: 116 | return self.ctx.spec.get_package_name(pkg.name) 117 | return None 118 | 119 | def get_symbol_external(self, info, symbol, paths=None): 120 | """ Get the provider of the required symbol from the files database, 121 | i.e. installed binary dependencies 122 | """ 123 | # Try a cached approach first. 124 | if info.emul32: 125 | if symbol in self.bindeps_emul32: 126 | return self.bindeps_emul32[symbol] 127 | else: 128 | if symbol in self.bindeps_cache: 129 | return self.bindeps_cache[symbol] 130 | 131 | if symbol in ExceptionRules: 132 | if info.emul32: 133 | return "libglvnd-32bit" 134 | else: 135 | return "libglvnd" 136 | 137 | if not paths: 138 | paths = ["/usr/lib64", "/usr/lib"] 139 | if info.emul32: 140 | paths = ["/usr/lib32", "/usr/lib", "/usr/lib64"] 141 | 142 | if info.rpaths: 143 | paths.extend(info.rpaths) 144 | 145 | pkg = None 146 | for path in paths: 147 | fpath = os.path.join(path, symbol) 148 | if not os.path.exists(fpath): 149 | continue 150 | lpkg = None 151 | if fpath in self.files_cache: 152 | lpkg = self.files_cache[fpath] 153 | else: 154 | pkg = self.search_file(fpath) 155 | if pkg: 156 | lpkg = pkg[0] 157 | if lpkg: 158 | if info.emul32: 159 | self.bindeps_emul32[symbol] = lpkg 160 | else: 161 | self.bindeps_cache[symbol] = lpkg 162 | console_ui.emit_info("Dependency", 163 | "{} adds dependency on {} from {}". 164 | format(info.pretty, symbol, lpkg)) 165 | 166 | # Populate a global files cache, basically there is a high 167 | # chance that each package depends on multiple things in a 168 | # single package. 169 | for file in self.idb.get_files(lpkg).list: 170 | self.files_cache["/" + file.path] = lpkg 171 | return lpkg 172 | return None 173 | 174 | def get_pkgconfig_provider(self, info, name): 175 | """ Get the internal provider for a pkgconfig name """ 176 | if info.emul32: 177 | if name in self.global_pkgconfig32s: 178 | pkg = self.global_pkgconfig32s[name] 179 | return self.ctx.spec.get_package_name(pkg) 180 | if name in self.global_pkgconfigs: 181 | pkg = self.global_pkgconfigs[name] 182 | return self.ctx.spec.get_package_name(pkg) 183 | return None 184 | 185 | def get_pkgconfig_external(self, info, name): 186 | """ Get the external provider of a pkgconfig name """ 187 | pkg = None 188 | 189 | if info.emul32: 190 | if name in self.pkgconfig32_cache: 191 | return self.pkgconfig32_cache[name] 192 | else: 193 | if name in self.pkgconfig_cache: 194 | return self.pkgconfig_cache[name] 195 | 196 | if info.emul32: 197 | # InstallDB set 198 | nom = self.fdb.get_pkgconfig32_provider(name) 199 | if nom: 200 | pkg = self.idb.get_package(nom[0]) 201 | if not pkg: 202 | nom = self.fdb.get_pkgconfig_provider(name) 203 | if nom: 204 | pkg = self.idb.get_package(nom[0]) 205 | 206 | # PackageDB set 207 | if not pkg: 208 | if name in self.pkgConfigs32: 209 | pkg = self.pdb.get_package(self.pkgConfigs32[name]) 210 | if not pkg: 211 | if name in self.pkgConfigs: 212 | pkg = self.pdb.get_package(self.pkgConfigs[name]) 213 | else: 214 | nom = self.fdb.get_pkgconfig_provider(name) 215 | if nom: 216 | pkg = self.idb.get_package(nom[0]) 217 | if not pkg: 218 | if name in self.pkgConfigs: 219 | pkg = self.pdb.get_package(self.pkgConfigs[name]) 220 | 221 | if not pkg: 222 | return None 223 | if info.emul32: 224 | self.pkgconfig32_cache[name] = pkg.name 225 | else: 226 | self.pkgconfig_cache[name] = pkg.name 227 | return pkg.name 228 | 229 | def handle_binary_deps(self, packageName, info): 230 | """ Handle direct binary dependencies """ 231 | pkgName = self.ctx.spec.get_package_name(packageName) 232 | 233 | for sym in info.symbol_deps: 234 | r = self.get_symbol_provider(info, sym) 235 | if not r: 236 | r = self.get_symbol_external(info, sym) 237 | if not r: 238 | print("Fatal: Unknown symbol: {}".format(sym)) 239 | continue 240 | # Don't self depend 241 | if pkgName == r: 242 | continue 243 | self.gene.packages[packageName].depend_packages.add(r) 244 | 245 | def handle_pkgconfig_deps(self, packageName, info): 246 | """ Handle pkgconfig dependencies """ 247 | pkgName = self.ctx.spec.get_package_name(packageName) 248 | 249 | for item in info.pkgconfig_deps: 250 | 251 | prov = self.get_pkgconfig_provider(info, item) 252 | if not prov: 253 | prov = self.get_pkgconfig_external(info, item) 254 | 255 | if not prov: 256 | console_ui.emit_warning("PKGCONFIG", "Not adding unknown" 257 | " dependency {} to {}". 258 | format(item, pkgName)) 259 | continue 260 | tgtPkg = self.gene.packages[packageName] 261 | 262 | # Yes, it's a set, but i dont want the ui emission spam 263 | if prov in tgtPkg.depend_packages: 264 | continue 265 | # Forbid circular dependencies 266 | if pkgName == prov: 267 | continue 268 | tgtPkg.depend_packages.add(prov) 269 | 270 | console_ui.emit_info("PKGCONFIG", "{} adds dependency on {}". 271 | format(pkgName, prov)) 272 | 273 | def handle_pkgconfig_provides(self, packageName, info): 274 | adder = None 275 | if info.emul32: 276 | adder = "pkgconfig32({})".format(info.pkgconfig_name) 277 | else: 278 | adder = "pkgconfig({})".format(info.pkgconfig_name) 279 | # TODO: Add versioning in examine.py .. ? 280 | self.gene.packages[packageName].provided_symbols.add(adder) 281 | pass 282 | 283 | def handle_soname_links(self, packageName, info): 284 | """ Add dependencies between packages due to a .so splitting """ 285 | ourName = self.ctx.spec.get_package_name(packageName) 286 | 287 | for link in info.soname_links: 288 | fi = self.gene.get_file_owner(link) 289 | if not fi: 290 | console_ui.emit_warning("SOLINK", "{} depends on non existing " 291 | "soname link: {}". 292 | format(packageName, link)) 293 | continue 294 | pkgName = self.ctx.spec.get_package_name(fi.name) 295 | if pkgName == ourName: 296 | continue 297 | self.gene.packages[packageName].depend_packages.add(pkgName) 298 | console_ui.emit_info("SOLINK", "{} depends on {} through .so link". 299 | format(ourName, pkgName)) 300 | 301 | def get_kernel_provider(self, info, version): 302 | """ i.e. self dependency situation """ 303 | if version in self.global_kernels: 304 | pkg = self.global_kernels[version] 305 | return self.ctx.spec.get_package_name(pkg) 306 | return None 307 | 308 | def get_kernel_external(self, info, version): 309 | """ Try to find the owning kernel for a version """ 310 | if version in self.kernel_cache: 311 | return self.kernel_cache[version] 312 | 313 | paths = [ 314 | "/usr/lib/kernel", 315 | "/usr/lib64/kernel" 316 | ] 317 | 318 | pkg = None 319 | for path in paths: 320 | # Special file in the main kernel package 321 | fpath = "{}/System.map-{}".format(path, version) 322 | if not os.path.exists(fpath): 323 | continue 324 | lpkg = None 325 | if fpath in self.files_cache: 326 | lpkg = self.files_cache[fpath] 327 | else: 328 | pkg = self.search_file(fpath) 329 | if pkg: 330 | lpkg = pkg[0] 331 | if lpkg: 332 | self.kernel_cache[version] = lpkg 333 | console_ui.emit_info("Kernel", 334 | "{} adds module dependency on {} from {}". 335 | format(info.pretty, version, lpkg)) 336 | 337 | # Populate a global files cache, basically there is a high 338 | # chance that each package depends on multiple things in a 339 | # single package. 340 | for file in self.idb.get_files(lpkg).list: 341 | self.files_cache["/" + file.path] = lpkg 342 | return lpkg 343 | return None 344 | 345 | def handle_kernel_deps(self, packageName, info): 346 | """ Add dependency between packages due to kernel version """ 347 | pkgName = self.ctx.spec.get_package_name(packageName) 348 | 349 | r = self.get_kernel_provider(info, info.dep_kernel) 350 | if not r: 351 | r = self.get_kernel_external(info, info.dep_kernel) 352 | if not r: 353 | print("Fatal: Unknown kernel: {}".format(sym)) 354 | return 355 | # Don't self depend 356 | if pkgName == r: 357 | return 358 | self.gene.packages[packageName].depend_packages.add(r) 359 | 360 | def compute_for_packages(self, context, gene, packageSet): 361 | """ packageSet is a dict mapping here. """ 362 | self.gene = gene 363 | self.packageSet = packageSet 364 | self.ctx = context 365 | 366 | # First iteration, collect the globals 367 | for packageName in packageSet: 368 | for info in packageSet[packageName]: 369 | if info.rpaths: 370 | if info.emul32: 371 | self.global_rpaths32.update(info.rpaths) 372 | else: 373 | self.global_rpaths.update(info.rpaths) 374 | if info.soname: 375 | if info.emul32: 376 | self.global_sonames32[info.soname] = packageName 377 | else: 378 | self.global_sonames[info.soname] = packageName 379 | if info.pkgconfig_name: 380 | pcName = info.pkgconfig_name 381 | if info.emul32: 382 | self.global_pkgconfig32s[pcName] = packageName 383 | else: 384 | self.global_pkgconfigs[pcName] = packageName 385 | if info.prov_kernel: 386 | self.global_kernels[info.prov_kernel] = packageName 387 | 388 | # Ok now find the dependencies 389 | for packageName in packageSet: 390 | for info in packageSet[packageName]: 391 | if info.symbol_deps: 392 | self.handle_binary_deps(packageName, info) 393 | 394 | if info.pkgconfig_deps: 395 | self.handle_pkgconfig_deps(packageName, info) 396 | 397 | if info.pkgconfig_name: 398 | self.handle_pkgconfig_provides(packageName, info) 399 | 400 | if info.soname_links: 401 | self.handle_soname_links(packageName, info) 402 | 403 | if info.dep_kernel: 404 | self.handle_kernel_deps(packageName, info) 405 | return True 406 | -------------------------------------------------------------------------------- /ypkg2/examine.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | from .metadata import readlink 16 | from . import remove_prefix 17 | from . import EMUL32PC 18 | import magic 19 | import re 20 | import os 21 | import subprocess 22 | import shutil 23 | import multiprocessing 24 | 25 | global share_ctx 26 | 27 | 28 | v_dyn = re.compile(r"ELF (64|32)\-bit LSB shared object,") 29 | v_bin = re.compile(r"ELF (64|32)\-bit LSB executable,") 30 | v_rel = re.compile(r"ELF (64|32)\-bit LSB relocatable,") 31 | shared_lib = re.compile(r".*Shared library: \[(.*)\].*") 32 | r_path = re.compile(r".*Library rpath: \[(.*)\].*") 33 | r_soname = re.compile(r".*Library soname: \[(.*)\].*") 34 | 35 | 36 | def is_pkgconfig_file(pretty, mgs): 37 | """ Simple as it sounds, work out if this is a pkgconfig file """ 38 | if pretty.endswith(".pc"): 39 | pname = os.path.basename(os.path.dirname(pretty)) 40 | if pname == "pkgconfig": 41 | return True 42 | return False 43 | 44 | 45 | def is_soname_link(file, mgs): 46 | """ Used to detect soname links """ 47 | if not file.endswith(".so"): 48 | return False 49 | 50 | if os.path.islink(file) and not os.path.isdir(file): 51 | return True 52 | return False 53 | 54 | 55 | def is_static_archive(file, mgs): 56 | """ Very trivially determine .a files """ 57 | if not file.endswith(".a"): 58 | return False 59 | 60 | if mgs != "current ar archive": 61 | return False 62 | 63 | if os.path.islink(file) or os.path.isdir(file): 64 | return False 65 | 66 | return True 67 | 68 | 69 | def is_system_map(file, mgs): 70 | """ Ensure we have a system map file """ 71 | if "kernel/System.map-" not in file: 72 | return False 73 | 74 | if mgs != "ASCII text": 75 | return False 76 | 77 | if os.path.islink(file) or os.path.isdir(file): 78 | return False 79 | 80 | return True 81 | 82 | 83 | class FileReport: 84 | 85 | pkgconfig_deps = None 86 | pkgconfig_name = None 87 | 88 | emul32 = False 89 | 90 | soname = None 91 | symbol_deps = None 92 | 93 | rpaths = None 94 | 95 | soname_links = None 96 | 97 | # Dependent kernel versions 98 | dep_kernel = None 99 | prov_kernel = None 100 | 101 | def scan_kernel(self, file): 102 | """ Scan a .ko file to figure out which kernel this depends on """ 103 | cmd = "LC_ALL=C /sbin/modinfo --field=vermagic \"{}\"".format(file) 104 | try: 105 | output = subprocess.check_output(cmd, shell=True) 106 | except Exception as e: 107 | console_ui.emit_warning("File", "Failed to scan kernel modules for" 108 | " path: {}".format(file)) 109 | return 110 | line = output.split("\n")[0] 111 | splits = line.strip().split(" ") 112 | if "modversions" not in splits: 113 | return 114 | if "mod_unload" not in splits: 115 | return 116 | self.dep_kernel = splits[0].strip() 117 | 118 | def scan_binary(self, file, check_soname=False): 119 | cmd = "LC_ALL=C /usr/bin/readelf -d \"{}\"".format(file) 120 | try: 121 | output = subprocess.check_output(cmd, shell=True) 122 | except Exception as e: 123 | console_ui.emit_warning("File", "Failed to scan binary deps for" 124 | " path: {}".format(file)) 125 | return 126 | 127 | for line in output.split("\n"): 128 | line = line.strip() 129 | 130 | # Match rpath 131 | r = r_path.match(line) 132 | if r: 133 | if self.rpaths is None: 134 | self.rpaths = set() 135 | self.rpaths.update(r.group(1).split(":")) 136 | continue 137 | 138 | # Match direct needed dependency 139 | m = shared_lib.match(line) 140 | if m: 141 | if self.symbol_deps is None: 142 | self.symbol_deps = set() 143 | self.symbol_deps.add(m.group(1)) 144 | continue 145 | 146 | # Check the soname for this binary file 147 | if check_soname: 148 | so = r_soname.match(line) 149 | if so: 150 | self.soname = so.group(1) 151 | 152 | def scan_pkgconfig(self, file): 153 | sub = "" 154 | pcDir = os.path.dirname(file) 155 | pcPaths = [] 156 | # Ensure we account for private pkgconfig deps too 157 | if self.emul32: 158 | pcPaths.append(os.path.join(pcDir, "../../lib32/pkgconfig")) 159 | pcPaths.append(EMUL32PC) 160 | else: 161 | pcPaths.append(os.path.join(pcDir, "../../lib64/pkgconfig")) 162 | pcPaths.append(os.path.join(pcDir, "../../lib/pkgconfig")) 163 | pcPaths.append(os.path.join(pcDir, "../../share/pkgconfig")) 164 | pkgConfigPaths = [] 165 | for path in pcPaths: 166 | p = os.path.abspath(path) 167 | if p and os.path.exists(p) and p not in pkgConfigPaths: 168 | pkgConfigPaths.append(p) 169 | 170 | if len(pkgConfigPaths) > 0: 171 | sub = "PKG_CONFIG_PATH=\"{}\" ".format(":".join(pkgConfigPaths)) 172 | 173 | cmds = [ 174 | "LC_ALL=C {}pkg-config --print-requires \"{}\"", 175 | "LC_ALL=C {}pkg-config --print-requires-private \"{}\"" 176 | ] 177 | 178 | pcname = os.path.basename(file).split(".pc")[0] 179 | self.pkgconfig_name = pcname 180 | 181 | if not share_ctx.spec.pkg_autodep: 182 | return 183 | for cmd in cmds: 184 | try: 185 | out = subprocess.check_output(cmd.format(sub, file), 186 | shell=True) 187 | except Exception as e: 188 | print(e) 189 | continue 190 | for line in out.split("\n"): 191 | line = line.strip() 192 | 193 | if line == "": 194 | continue 195 | name = None 196 | # In future we'll do something useful with versions 197 | if ">=" in line: 198 | name = line.split(">=")[0] 199 | elif "=" in line: 200 | # This is an internal dependency 201 | name = line.split("=")[0] 202 | else: 203 | name = line 204 | name = name.strip() 205 | 206 | if not self.pkgconfig_deps: 207 | self.pkgconfig_deps = set() 208 | self.pkgconfig_deps.add(name) 209 | 210 | def add_solink(self, file, pretty): 211 | """ .so links are almost always split into -devel subpackages in ypkg, 212 | unless explicitly overriden. However, they are useless without the 213 | actual versioned so they link to. Therefore, we add an automatic 214 | dependency to the hosting package when we find one of these, i.e: 215 | 216 | zlib: 217 | /usr/lib64/libz.so.1.2.8 218 | zlib-devel: 219 | /usr/lib64/libz.so -> libz.so.1.2.8 220 | 221 | zlib-devel -> zlib 222 | """ 223 | fpath = readlink(file) 224 | 225 | dirn = os.path.dirname(file) 226 | fobj = os.path.join(dirn, fpath) 227 | 228 | try: 229 | mg = magic.from_file(fobj) 230 | except Exception as e: 231 | return 232 | 233 | if not v_dyn.match(mg): 234 | return 235 | fpath = remove_prefix(fobj, share_ctx.get_install_dir()) 236 | if not self.soname_links: 237 | self.soname_links = set() 238 | self.soname_links.add(fpath) 239 | 240 | def add_kernel_prov(self, file): 241 | self.prov_kernel = str(file.split("System.map-")[1]) 242 | 243 | def __init__(self, pretty, file, mgs): 244 | global share_ctx 245 | self.pretty = pretty 246 | self.file = file 247 | 248 | if pretty.startswith("/usr/lib32/") or pretty.startswith("/lib32"): 249 | self.emul32 = True 250 | if is_pkgconfig_file(pretty, mgs): 251 | self.scan_pkgconfig(file) 252 | if is_system_map(pretty, mgs): 253 | self.add_kernel_prov(file) 254 | 255 | # Some things omit automatic dependencies 256 | if share_ctx.spec.pkg_autodep: 257 | if is_soname_link(file, mgs): 258 | self.add_solink(file, pretty) 259 | elif v_dyn.match(mgs): 260 | self.scan_binary(file, True) 261 | elif v_bin.match(mgs): 262 | self.scan_binary(file, False) 263 | elif v_rel.match(mgs) and file.endswith(".ko"): 264 | self.scan_kernel(file) 265 | 266 | 267 | def strip_file(context, pretty, file, magic_string, mode=None): 268 | """ Schedule a strip, basically. """ 269 | if not context.spec.pkg_strip: 270 | return 271 | exports = ["LC_ALL=C"] 272 | if context.spec.pkg_optimize and not context.spec.pkg_clang: 273 | if "thin-lto" in context.spec.pkg_optimize \ 274 | or "lto" in context.spec.pkg_optimize: 275 | exports.extend([ 276 | "AR=\"gcc-ar\"", 277 | "RANLIB=\"gcc-ranlib\"", 278 | "NM=\"gcc-nm\""]) 279 | 280 | cmd = "{} strip {} \"{}\"" 281 | flags = "" 282 | if mode == "shared": 283 | flags = "--strip-unneeded" 284 | elif mode == "ko": 285 | flags = "-g --strip-unneeded" 286 | elif mode == "ar": 287 | flags = "--strip-debug" 288 | try: 289 | s = " ".join(exports) 290 | subprocess.check_call(cmd.format(s, flags, file), shell=True) 291 | console_ui.emit_info("Stripped", pretty) 292 | except Exception as e: 293 | console_ui.emit_warning("Strip", "Failed to strip '{}'". 294 | format(pretty)) 295 | print(e) 296 | 297 | 298 | def get_debug_path(context, file, magic_string): 299 | """ Grab the NT_GNU_BUILD_ID """ 300 | cmd = "LC_ALL=C readelf -n \"{}\"".format(file) 301 | try: 302 | lines = subprocess.check_output(cmd, shell=True) 303 | except Exception as e: 304 | return None 305 | 306 | for line in lines.split("\n"): 307 | if "Build ID:" not in line: 308 | continue 309 | v = line.split(":")[1].strip() 310 | 311 | libdir = "/usr/lib" 312 | if "ELF 32" in magic_string: 313 | libdir = "/usr/lib32" 314 | 315 | path = os.path.join(libdir, "debug", ".build-id", v[0:2], v[2:]) 316 | return path + ".debug" 317 | return None 318 | 319 | 320 | def examine_file(*args): 321 | global share_ctx 322 | package = args[0] 323 | pretty = args[1] 324 | file = args[2] 325 | mgs = args[3] 326 | 327 | context = share_ctx 328 | 329 | if v_dyn.match(mgs): 330 | # Get soname, direct deps and strip 331 | store_debug(context, pretty, file, mgs) 332 | strip_file(context, pretty, file, mgs, mode="shared") 333 | elif v_bin.match(mgs): 334 | # Get direct deps, and strip 335 | store_debug(context, pretty, file, mgs) 336 | strip_file(context, pretty, file, mgs, mode="executable") 337 | elif v_rel.match(mgs): 338 | # Kernel object in all probability 339 | if file.endswith(".ko"): 340 | store_debug(context, pretty, file, mgs) 341 | strip_file(context, pretty, file, mgs, mode="ko") 342 | elif mgs == "current ar archive": 343 | # Strip only. 344 | strip_file(context, pretty, file, mgs, mode="ar") 345 | 346 | freport = FileReport(pretty, file, mgs) 347 | return freport 348 | 349 | 350 | def store_debug(context, pretty, file, magic_string): 351 | if not context.can_dbginfo: 352 | return 353 | if not context.spec.pkg_debug: 354 | return 355 | 356 | did = get_debug_path(context, file, magic_string) 357 | 358 | if did is None: 359 | if "ELF 32" in magic_string: 360 | did = "/usr/lib32/debug/{}.debug".format(pretty) 361 | else: 362 | did = "/usr/lib/debug/{}.debug".format(pretty) 363 | 364 | did_full = os.path.join(context.get_install_dir(), did[1:]) 365 | 366 | # Account for race condition in directory creation 367 | dirs = os.path.dirname(did_full) 368 | try: 369 | os.makedirs(dirs, mode=00755) 370 | except Exception as e: 371 | pass 372 | if not os.path.exists(dirs): 373 | console_ui.emit_error("Debug", "Failed to make directory") 374 | return 375 | 376 | cmd = "objcopy --only-keep-debug \"{}\" \"{}\"".format(file, did_full) 377 | try: 378 | subprocess.check_call(cmd, shell=True) 379 | except Exception as e: 380 | console_ui.emit_warning("objcopy", "Failed --only-keep-debug") 381 | return 382 | cmd = "objcopy --add-gnu-debuglink=\"{}\" \"{}\"".format(did_full, 383 | file) 384 | try: 385 | subprocess.check_call(cmd, shell=True) 386 | except Exception as e: 387 | console_ui.emit_warning("objcopy", "Failed --add-gnu-debuglink") 388 | return 389 | 390 | 391 | class PackageExaminer: 392 | """ Responsible for identifying files suitable for further examination, 393 | such as those that should be removed, checked for dependencies, 394 | providers, and even those that should be stripped 395 | """ 396 | 397 | def __init__(self): 398 | self.libtool_file = re.compile("libtool library file, ASCII text.*") 399 | self.can_kernel = True 400 | 401 | def should_nuke_file(self, context, pretty, file, mgs): 402 | # it's not that we hate.. Actually, no, we do. We hate you libtool. 403 | if context.spec.pkg_lastrip and self.libtool_file.match(mgs): 404 | return True 405 | if pretty == "/usr/share/info/dir": 406 | return True 407 | if pretty.startswith("/emul32"): 408 | return True 409 | # Nuke AVX2 dir .a files with no remorse 410 | if (pretty.startswith("/usr/lib64/haswell/") or 411 | pretty.startswith("/usr/lib32/haswell/")): 412 | if ".so" not in pretty: 413 | return True 414 | # Don't want .so links, they're useless. 415 | if pretty.endswith(".so") and os.path.islink(file): 416 | return True 417 | return False 418 | 419 | def file_is_of_interest(self, pretty, file, mgs): 420 | """ So we can keep our list of things to check low """ 421 | if v_dyn.match(mgs) or v_bin.match(mgs) or v_rel.match(mgs): 422 | if not self.can_kernel and file.endswith(".ko"): 423 | return False 424 | return True 425 | if is_pkgconfig_file(pretty, mgs): 426 | return True 427 | if is_soname_link(file, mgs): 428 | return True 429 | if is_static_archive(file, mgs): 430 | return True 431 | if self.can_kernel and is_system_map(file, mgs): 432 | return True 433 | return False 434 | 435 | def examine_package(self, context, package): 436 | """ Examine the given package and update symbols, etc. """ 437 | install_dir = context.get_install_dir() 438 | 439 | global share_ctx 440 | 441 | share_ctx = context 442 | 443 | # Right now we actually only care about magic matching 444 | removed = set() 445 | 446 | pool = multiprocessing.Pool() 447 | results = list() 448 | 449 | for file in package.emit_files(): 450 | if file[0] == '/': 451 | file = file[1:] 452 | fpath = os.path.join(install_dir, file) 453 | try: 454 | mgs = magic.from_file(fpath) 455 | except Exception as e: 456 | print(e) 457 | continue 458 | if self.should_nuke_file(context, "/" + file, fpath, mgs): 459 | try: 460 | if os.path.isfile(fpath): 461 | os.unlink(fpath) 462 | else: 463 | shutil.rmtree(fpath) 464 | except Exception as e: 465 | console_ui.emit_error("Clean", "Failed to remove unwanted" 466 | "file: {}".format(e)) 467 | return False 468 | console_ui.emit_info("Clean", "Removed unwanted file: {}". 469 | format("/" + file)) 470 | removed.add("/" + file) 471 | continue 472 | 473 | if not self.file_is_of_interest("/" + file, fpath, mgs): 474 | continue 475 | # Handle this asynchronously 476 | results.append(pool.apply_async(examine_file, [ 477 | package, "/" + file, fpath, mgs], 478 | callback=None)) 479 | 480 | pool.close() 481 | pool.join() 482 | 483 | infos = [x.get() for x in results] 484 | 485 | for r in removed: 486 | package.remove_file(r) 487 | return infos 488 | 489 | def examine_packages(self, context, packages): 490 | """ Examine all packages, in order to update dependencies, etc """ 491 | console_ui.emit_info("Examine", "Examining packages") 492 | 493 | examinations = dict() 494 | for package in packages: 495 | ir = self.examine_package(context, package) 496 | if not ir: 497 | continue 498 | examinations[package.name] = ir 499 | return examinations 500 | -------------------------------------------------------------------------------- /ypkg2/main.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | from . import remove_prefix 16 | from .ypkgspec import YpkgSpec 17 | from .sources import SourceManager 18 | from .ypkgcontext import YpkgContext 19 | from .scripts import ScriptGenerator 20 | from .packages import PackageGenerator, PRIORITY_USER 21 | from .examine import PackageExaminer 22 | from . import metadata 23 | from .dependencies import DependencyResolver 24 | from . import packager_name, packager_email 25 | from . import EMUL32PC 26 | 27 | import ypkg2 28 | 29 | import sys 30 | import argparse 31 | import os 32 | import shutil 33 | import tempfile 34 | import subprocess 35 | from configobj import ConfigObj 36 | 37 | 38 | def show_version(): 39 | print("Ypkg - Package Build Tool") 40 | print("\nCopyright (C) 2015-2017 Ikey Doherty\n") 41 | print("This program is free software; you are free to redistribute it\n" 42 | "and/or modify it under the terms of the GNU General Public License" 43 | "\nas published by the Free Software foundation; either version 3 of" 44 | "\nthe License, or (at your option) any later version.") 45 | sys.exit(0) 46 | 47 | 48 | def main(): 49 | parser = argparse.ArgumentParser(description="Ypkg Package Build Tool") 50 | parser.add_argument("-n", "--no-colors", help="Disable color output", 51 | action="store_true") 52 | parser.add_argument("-v", "--version", action="store_true", 53 | help="Show version information and exit") 54 | parser.add_argument("-t", "--timestamp", help="Unix timestamp for build", 55 | type=int, default=-1) 56 | parser.add_argument("-D", "--output-dir", type=str, 57 | help="Set the output directory for resulting files") 58 | # Main file 59 | parser.add_argument("filename", help="Path to the ypkg YAML file to build", 60 | nargs='?') 61 | 62 | outputDir = "." 63 | 64 | args = parser.parse_args() 65 | # Kill colors 66 | if args.no_colors: 67 | console_ui.allow_colors = False 68 | # Show version 69 | if args.version: 70 | show_version() 71 | if args.timestamp > 0: 72 | metadata.history_timestamp = args.timestamp 73 | 74 | if args.output_dir: 75 | od = args.output_dir 76 | if not os.path.exists(args.output_dir): 77 | console_ui.emit_error("Opt", "{} does not exist".format(od)) 78 | sys.exit(1) 79 | outputDir = od 80 | outputDir = os.path.abspath(outputDir) 81 | 82 | # Grab filename 83 | if not args.filename: 84 | console_ui.emit_error("Error", 85 | "Please provide a filename to ypkg-build") 86 | print("") 87 | parser.print_help() 88 | sys.exit(1) 89 | 90 | # Test who we are 91 | if os.geteuid() == 0: 92 | if "FAKED_MODE" not in os.environ: 93 | console_ui.emit_warning("Warning", "ypkg-build should be run via " 94 | "fakeroot, not as real root user") 95 | else: 96 | console_ui.emit_error("Fail", "ypkg-build must be run with fakeroot, " 97 | "or as the root user (not recommended)") 98 | sys.exit(1) 99 | 100 | build_package(args.filename, outputDir) 101 | 102 | 103 | def clean_build_dirs(context): 104 | if os.path.exists(context.get_build_dir()): 105 | try: 106 | shutil.rmtree(context.get_build_dir()) 107 | except Exception as e: 108 | console_ui.emit_error("BUILD", "Could not clean build directory") 109 | print(e) 110 | return False 111 | return True 112 | 113 | 114 | def execute_step(context, step, step_n, work_dir): 115 | script = ScriptGenerator(context, context.spec, work_dir) 116 | if context.emul32: 117 | script.define_export("EMUL32BUILD", "1") 118 | script.define_export("PKG_CONFIG_PATH", EMUL32PC) 119 | if context.avx2: 120 | script.define_export("AVX2BUILD", "1") 121 | extraScript = None 122 | 123 | # Allow GCC and such to pick up on our timestamp 124 | script.define_export("SOURCE_DATA_EPOCH", 125 | "{}".format(metadata.history_timestamp)) 126 | 127 | # Handle the anal nature of llvm profiling 128 | if context.gen_pgo and context.spec.pkg_clang: 129 | profileFile = os.path.join(context.get_pgo_dir(), 130 | "default-%m.profraw") 131 | script.define_export("LLVM_PROFILE_FILE", profileFile) 132 | script.define_export("YPKG_PGO_DIR", context.get_pgo_dir()) 133 | elif context.use_pgo and context.spec.pkg_clang: 134 | profileFile = os.path.join(context.get_pgo_dir(), 135 | "default.profdata") 136 | extraScript = "%llvm_profile_merge" 137 | script.define_export("LLVM_PROFILE_FILE", profileFile) 138 | script.define_export("YPKG_PGO_DIR", context.get_pgo_dir()) 139 | 140 | exports = script.emit_exports() 141 | 142 | # Run via bash with enable and error 143 | full_text = "#!/usr/bin/env -i /bin/bash --norc --noprofile\n" \ 144 | "set -e\nset -x\n" 145 | # cd to the given directory 146 | full_text += "\n\ncd \"%workdir%\"\n" 147 | 148 | # Add our exports 149 | full_text += "\n".join(exports) 150 | if extraScript: 151 | full_text += "\n\n{}\n".format(extraScript) 152 | full_text += "\n\n{}\n".format(step) 153 | output = script.escape_string(full_text) 154 | 155 | with tempfile.NamedTemporaryFile(prefix="ypkg-%s" % step_n) as script_ex: 156 | script_ex.write(output) 157 | script_ex.flush() 158 | 159 | cmd = ["/bin/bash", "--norc", "--noprofile", script_ex.name] 160 | try: 161 | subprocess.check_call(cmd, stdin=subprocess.PIPE) 162 | except KeyboardInterrupt: 163 | pass 164 | except Exception as e: 165 | print(e) 166 | return False 167 | return True 168 | 169 | 170 | def build_package(filename, outputDir): 171 | """ Will in future be moved to a separate part of the module """ 172 | spec = YpkgSpec() 173 | if not spec.load_from_path(filename): 174 | print("Unable to continue - aborting") 175 | sys.exit(1) 176 | 177 | possibles = ["{}/.solus/packager", "{}/.evolveos/packager"] 178 | 179 | packager_name = ypkg2.packager_name 180 | packager_email = ypkg2.packager_email 181 | 182 | dflt = True 183 | for item in possibles: 184 | fpath = item.format(os.path.expanduser("~")) 185 | if not os.path.exists(fpath): 186 | continue 187 | try: 188 | c = ConfigObj(fpath) 189 | pname = c["Packager"]["Name"] 190 | pemail = c["Packager"]["Email"] 191 | 192 | packager_name = pname 193 | packager_email = pemail 194 | dflt = False 195 | break 196 | except Exception as e: 197 | console_ui.emit_error("Config", "Error in packager config:") 198 | print(e) 199 | dflt = True 200 | break 201 | if dflt: 202 | packager_name = ypkg2.packager_name 203 | packager_email = ypkg2.packager_email 204 | console_ui.emit_warning("Config", "Using default packager values") 205 | print(" Name: {}".format(packager_name)) 206 | print(" Email: {}".format(packager_email)) 207 | 208 | spec.packager_name = packager_name 209 | spec.packager_email = packager_email 210 | # Try to load history 211 | dirn = os.path.dirname(filename) 212 | hist = os.path.join(dirn, "history.xml") 213 | if os.path.exists(hist): 214 | if not spec.load_history(hist): 215 | sys.exit(1) 216 | 217 | metadata.initialize_timestamp(spec) 218 | 219 | manager = SourceManager() 220 | if not manager.identify_sources(spec): 221 | print("Unable to continue - aborting") 222 | sys.exit(1) 223 | 224 | # Dummy content 225 | console_ui.emit_info("Info", "Building {}-{}". 226 | format(spec.pkg_name, spec.pkg_version)) 227 | 228 | ctx = YpkgContext(spec) 229 | 230 | need_verify = [] 231 | for src in manager.sources: 232 | if src.cached(ctx): 233 | need_verify.append(src) 234 | continue 235 | if not src.fetch(ctx): 236 | console_ui.emit_error("Source", "Cannot continue without sources") 237 | sys.exit(1) 238 | need_verify.append(src) 239 | 240 | for verify in need_verify: 241 | if not verify.verify(ctx): 242 | console_ui.emit_error("Source", "Cannot verify sources") 243 | sys.exit(1) 244 | 245 | steps = { 246 | 'setup': spec.step_setup, 247 | 'build': spec.step_build, 248 | 'install': spec.step_install, 249 | 'check': spec.step_check, 250 | 'profile': spec.step_profile, 251 | } 252 | 253 | r_runs = list() 254 | 255 | # Before we get started, ensure PGOs are cleaned 256 | if not ctx.clean_pgo(): 257 | console_ui.emit_error("Build", "Failed to clean PGO directories") 258 | sys.exit(1) 259 | 260 | if not ctx.clean_install(): 261 | console_ui.emit_error("Build", "Failed to clean install directory") 262 | sys.exit(1) 263 | if not ctx.clean_pkg(): 264 | console_ui.emit_error("Build", "Failed to clean pkg directory") 265 | 266 | possible_sets = [] 267 | # Emul32 is *always* first 268 | # AVX2 emul32 comes first too so "normal" emul32 can override it 269 | if spec.pkg_emul32: 270 | if spec.pkg_avx2: 271 | # Emul32, avx2 build 272 | possible_sets.append((True, True)) 273 | # Normal, no-avx2, emul32 build 274 | possible_sets.append((True, False)) 275 | 276 | # Build AVX2 before native, but after emul32 277 | if spec.pkg_avx2: 278 | possible_sets.append((False, True)) 279 | 280 | # Main step, always last 281 | possible_sets.append((False, False)) 282 | 283 | for emul32, avx2 in possible_sets: 284 | r_steps = list() 285 | c = YpkgContext(spec, emul32=emul32, avx2=avx2) 286 | if spec.step_profile is not None: 287 | c = YpkgContext(spec, emul32=emul32, avx2=avx2) 288 | c.enable_pgo_generate() 289 | r_steps.append(['setup', c]) 290 | r_steps.append(['build', c]) 291 | r_steps.append(['profile', c]) 292 | c = YpkgContext(spec, emul32=emul32, avx2=avx2) 293 | c.enable_pgo_use() 294 | r_steps.append(['setup', c]) 295 | r_steps.append(['build', c]) 296 | r_steps.append(['install', c]) 297 | r_steps.append(['check', c]) 298 | else: 299 | c = YpkgContext(spec, emul32=emul32, avx2=avx2) 300 | r_steps.append(['setup', c]) 301 | r_steps.append(['build', c]) 302 | r_steps.append(['install', c]) 303 | r_steps.append(['check', c]) 304 | r_runs.append((emul32, avx2, r_steps)) 305 | 306 | for emul32, avx2, run in r_runs: 307 | if emul32: 308 | console_ui.emit_info("Build", "Building for emul32") 309 | else: 310 | console_ui.emit_info("Build", "Building native package") 311 | if avx2: 312 | console_ui.emit_info("Build", "Building for AVX2 optimisations") 313 | 314 | for step, context in run: 315 | # When doing setup, always do pre-work by blasting away any 316 | # existing build directories for the current context and then 317 | # re-extracting sources 318 | if step == "setup": 319 | if not clean_build_dirs(context): 320 | sys.exit(1) 321 | 322 | # Only ever extract the primary source ourselves 323 | if spec.pkg_extract: 324 | src = manager.sources[0] 325 | console_ui.emit_info("Source", 326 | "Extracting source") 327 | if not src.extract(context): 328 | console_ui.emit_error("Source", 329 | "Cannot extract sources") 330 | sys.exit(1) 331 | 332 | if spec.step_profile: 333 | try: 334 | if not os.path.exists(context.get_pgo_dir()): 335 | os.makedirs(context.get_pgo_dir(), 00755) 336 | except Exception as e: 337 | console_ui.emit_error("Source", "Error creating dir") 338 | print(e) 339 | sys.exit(1) 340 | 341 | work_dir = manager.get_working_dir(context) 342 | if not os.path.exists(work_dir): 343 | try: 344 | os.makedirs(work_dir, mode=00755) 345 | except Exception as e: 346 | console_ui.emit_error("Source", "Error creating directory") 347 | print(e) 348 | sys.exit(1) 349 | 350 | r_step = steps[step] 351 | if not r_step: 352 | continue 353 | 354 | console_ui.emit_info("Build", "Running step: {}".format(step)) 355 | 356 | if execute_step(context, r_step, step, work_dir): 357 | console_ui.emit_success("Build", "{} successful". 358 | format(step)) 359 | continue 360 | console_ui.emit_error("Build", "{} failed".format(step)) 361 | sys.exit(1) 362 | 363 | # Add user patterns - each consecutive package has higher priority than the 364 | # package before it, ensuring correct levels of control 365 | gene = PackageGenerator(spec) 366 | count = 0 367 | for pkg in spec.patterns: 368 | for pt in spec.patterns[pkg]: 369 | gene.add_pattern(pt, pkg, priority=PRIORITY_USER + count) 370 | count += 1 371 | 372 | idir = ctx.get_install_dir() 373 | bad_dir = os.path.join(idir, "emul32") 374 | if os.path.exists(bad_dir): 375 | shutil.rmtree(bad_dir) 376 | 377 | for root, dirs, files in os.walk(idir): 378 | for f in files: 379 | fpath = os.path.join(root, f) 380 | 381 | localpath = remove_prefix(fpath, idir) 382 | gene.add_file(localpath) 383 | if len(dirs) == 0 and len(files) == 0: 384 | console_ui.emit_warning("Package", "Including empty directory: {}". 385 | format(remove_prefix(root, idir))) 386 | gene.add_file(remove_prefix(root, idir)) 387 | 388 | # Handle symlinks to directories. 389 | for d in dirs: 390 | fpath = os.path.join(root, d) 391 | if os.path.islink(fpath): 392 | gene.add_file(remove_prefix(fpath, idir)) 393 | 394 | if not os.path.exists(ctx.get_packaging_dir()): 395 | try: 396 | os.makedirs(ctx.get_packaging_dir(), mode=00755) 397 | except Exception as e: 398 | console_ui.emit_error("Package", "Failed to create pkg dir") 399 | print(e) 400 | sys.exit(1) 401 | 402 | exa = PackageExaminer() 403 | # Avoid expensive self calculations for kernels 404 | exa.can_kernel = True 405 | if spec.get_component("main") == "kernel.image": 406 | exa.can_kernel = False 407 | 408 | exaResults = exa.examine_packages(ctx, gene.packages.values()) 409 | if exaResults is None: 410 | console_ui.emit_error("Package", "Failed to correctly examine all " 411 | "packages.") 412 | sys.exit(1) 413 | 414 | deps = DependencyResolver() 415 | if not deps.compute_for_packages(ctx, gene, exaResults): 416 | console_ui.emit_error("Dependencies", "Failed to compute all" 417 | " dependencies") 418 | sys.exit(1) 419 | 420 | dbgs = ["/usr/lib64/debug", "/usr/lib/debug", "/usr/lib32/debug"] 421 | if ctx.can_dbginfo: 422 | for dbg in dbgs: 423 | fpath = os.path.join(ctx.get_install_dir(), dbg[1:]) 424 | if not os.path.exists(fpath): 425 | continue 426 | for root, dirs, files in os.walk(fpath): 427 | # Empty directories in dbginfo we don't care about. 428 | for f in files: 429 | fpath = os.path.join(root, f) 430 | 431 | localpath = remove_prefix(fpath, idir) 432 | 433 | gene.add_file(localpath) 434 | 435 | if len(gene.packages) == 0: 436 | console_ui.emit_error("Package", "No resulting packages found") 437 | w = "https://solus-project.com/articles/packaging/" 438 | print("Ensure your files end up in $installdir. Did you mean to " 439 | "use %make_install?\n\nPlease see the help center: {}".format(w)) 440 | sys.exit(1) 441 | 442 | gene.emit_packages() 443 | # TODO: Ensure main is always first 444 | for package in sorted(gene.packages): 445 | pkg = gene.packages[package] 446 | files = sorted(pkg.emit_files()) 447 | if len(files) == 0: 448 | console_ui.emit_info("Package", "Skipping empty package: {}". 449 | format(package)) 450 | continue 451 | metadata.create_eopkg(ctx, gene, pkg, outputDir) 452 | 453 | # Write out the final pspec 454 | metadata.write_spec(ctx, gene, outputDir) 455 | 456 | for pkg in spec.patterns: 457 | if pkg in gene.packages: 458 | continue 459 | nm = spec.get_package_name(pkg) 460 | console_ui.emit_warning("Package:{}".format(pkg), 461 | "Did not produce {} by any pattern".format(nm)) 462 | 463 | # TODO: Consider warning about unused patterns 464 | ctx.clean_pkg() 465 | console_ui.emit_success("Package", "Building complete") 466 | sys.exit(0) 467 | -------------------------------------------------------------------------------- /ypkg2/packages.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # Many portions of this are related to autospec, concepts included 9 | # 10 | # Copyright (C) 2016 Intel Corporation 11 | # 12 | # This program is free software: you can redistribute it and/or modify 13 | # it under the terms of the GNU General Public License as published by 14 | # the Free Software Foundation, either version 3 of the License, or 15 | # (at your option) any later version. 16 | 17 | from . import console_ui 18 | from .stringglob import StringPathGlob 19 | 20 | import os 21 | 22 | PRIORITY_DEFAULT = 0 # Standard internal priority for a pattern 23 | PRIORITY_USER = 100 # Priority for a user pattern, do what they say. 24 | DBG = 1000 # Never allow the user to override these guys. 25 | 26 | 27 | class DefaultPolicy(StringPathGlob): 28 | 29 | def __init__(self): 30 | StringPathGlob.__init__(self, "a") 31 | pass 32 | 33 | 34 | class Package: 35 | 36 | patterns = None 37 | files = None 38 | excludes = None 39 | 40 | # Symbols provided by this package, right now this is limited to 41 | # pkgconfig() and pkgconfig32() 42 | provided_symbols = None 43 | 44 | # Symbols depended upon by this package 45 | depend_packages = None 46 | 47 | # List of permanent files 48 | permanent = None 49 | 50 | def __init__(self, name): 51 | self.name = name 52 | self.patterns = dict() 53 | self.files = set() 54 | self.excludes = set() 55 | self.permanent = set() 56 | 57 | self.provided_symbols = set() 58 | self.depend_packages = set() 59 | 60 | self.default_policy = DefaultPolicy() 61 | 62 | def get_pattern(self, path): 63 | """ Return a matching pattern for the given path. 64 | This is ordered according to priority to enable 65 | multiple layers of priorities """ 66 | matches = [p for p in self.patterns if p.match(path)] 67 | if len(matches) == 0: 68 | return self.default_policy 69 | 70 | matches = sorted(matches, key=StringPathGlob.get_priority, 71 | reverse=True) 72 | return matches[0] 73 | 74 | def add_file(self, pattern, path, permanent): 75 | """ Add a file by a given pattern to this package """ 76 | if pattern is None: 77 | pattern = self.default_policy 78 | if pattern not in self.patterns: 79 | self.patterns[pattern] = set() 80 | self.patterns[pattern].add(path) 81 | self.files.add(path) 82 | if permanent: 83 | self.permanent.add(path) 84 | 85 | def remove_file(self, path): 86 | """ Remove a file from this package if it owns it """ 87 | pat = self.get_pattern(path) 88 | if not pat: 89 | return 90 | if path in self.patterns[pat]: 91 | self.patterns[pat].remove(path) 92 | if path in self.files: 93 | self.files.remove(path) 94 | 95 | def exclude_file(self, path): 96 | """ Exclude a file from this package if it captures it """ 97 | pat = self.get_pattern(path) 98 | if not pat: 99 | return 100 | if path in self.files: 101 | self.files.remove(path) 102 | self.excludes.add(path) 103 | 104 | def emit_files(self): 105 | """ Emit actual file lists, vs the globs we have """ 106 | ret = set() 107 | for pt in self.patterns: 108 | adds = [x for x in self.patterns[pt] if x not in self.excludes] 109 | ret.update(adds) 110 | return sorted(ret) 111 | 112 | def is_permanent(self, path): 113 | """ Determine if a path if a permanent path or not """ 114 | return path in self.permanent 115 | 116 | def emit_files_by_pattern(self): 117 | """ Emit file lists, using the globs though. Note that eopkg has no 118 | exclude concept, this is left for us to handle as we build the 119 | resulting eopkg ourselves """ 120 | ret = set() 121 | for pt in self.patterns: 122 | pat = self.patterns[pt] 123 | 124 | tmp = set([x for x in pat if x not in self.excludes]) 125 | if len(tmp) == 0: 126 | continue 127 | # Default policy, just list all the files 128 | if isinstance(pt, DefaultPolicy): 129 | ret.update(tmp) 130 | else: 131 | ret.add(str(pt)) 132 | return sorted(ret) 133 | 134 | 135 | class PackageGenerator: 136 | 137 | patterns = None 138 | packages = None 139 | permanent = None 140 | 141 | def __init__(self, spec): 142 | self.patterns = dict() 143 | self.packages = dict() 144 | self.permanent = set() 145 | 146 | if spec.pkg_permanent: 147 | for perm in spec.pkg_permanent: 148 | self.add_permanent_pattern(perm) 149 | 150 | self.add_pattern("/usr/bin", "main") 151 | self.add_pattern("/usr/sbin", "main") 152 | self.add_pattern("/sbin", "main") 153 | self.add_pattern("/bin", "main") 154 | self.add_pattern("/usr/share/info", "main") 155 | self.add_pattern("/usr/lib64/lib*.so.*", "main") 156 | self.add_pattern("/usr/lib64/haswell/*.so*", "main") 157 | self.add_pattern("/usr/lib/lib*.so.*", "main") 158 | self.add_pattern("/usr/lib32/", "32bit") 159 | self.add_pattern("/usr/lib32/lib*.so.*", "32bit", 160 | priority=PRIORITY_DEFAULT+1) 161 | self.add_pattern("/usr/lib32/haswell/*.so*", "32bit", 162 | priority=PRIORITY_DEFAULT+1) 163 | 164 | self.add_pattern("/usr/share/locale", "main") 165 | self.add_pattern("/usr/share/doc", "main") 166 | self.add_pattern("/usr/share/{}".format(spec.pkg_name), "main") 167 | 168 | if spec.pkg_libsplit: 169 | self.add_pattern("/usr/lib64/lib*.so", "devel") 170 | self.add_pattern("/usr/lib/lib*.so", "devel") 171 | self.add_pattern("/usr/lib32/lib*.so", "32bit-devel", 172 | priority=PRIORITY_DEFAULT+1) 173 | 174 | else: 175 | self.add_pattern("/usr/lib64/lib*.so", "main") 176 | self.add_pattern("/usr/lib/lib*.so", "main") 177 | self.add_pattern("/usr/lib32/lib*.so", "32bit", 178 | priority=PRIORITY_DEFAULT+1) 179 | 180 | self.add_pattern("/usr/lib64/lib*.a", "devel") 181 | self.add_pattern("/usr/lib/lib*.a", "devel") 182 | self.add_pattern("/usr/lib/pkgconfig/*.pc", "devel") 183 | self.add_pattern("/usr/lib64/pkgconfig/*.pc", "devel") 184 | self.add_pattern("/usr/share/pkgconfig/*.pc", "devel") 185 | self.add_pattern("/usr/include/", "devel") 186 | self.add_pattern("/usr/share/man3/", "devel", 187 | priority=PRIORITY_DEFAULT+1) 188 | self.add_pattern("/usr/share/man", "main") 189 | self.add_pattern("/usr/share/aclocal/*.m4", "devel") 190 | self.add_pattern("/usr/share/aclocal/*.ac", "devel") 191 | 192 | self.add_pattern("/usr/lib32/lib*.a", "32bit-devel") 193 | self.add_pattern("/usr/lib32/pkgconfig/*.pc", "32bit-devel") 194 | self.add_pattern("/usr/lib32/lib*.so.*", "32bit") 195 | 196 | # Debug infos get highest priority. you don't override these guys. 197 | self.add_pattern("/usr/lib64/debug/", "dbginfo", priority=DBG) 198 | self.add_pattern("/usr/lib/debug/", "dbginfo", priority=DBG) 199 | self.add_pattern("/usr/lib32/debug/", "32bit-dbginfo", priority=DBG) 200 | 201 | self.add_pattern("/usr/share/gtk-doc/html/", "docs") 202 | 203 | # cmake stuffs 204 | self.add_pattern("/usr/share/cmake/", "devel") 205 | self.add_pattern("/usr/lib64/cmake/", "devel") 206 | self.add_pattern("/usr/lib/cmake/", "devel") 207 | self.add_pattern("/usr/lib32/cmake/", "32bit-devel") 208 | 209 | # Haskell 210 | self.add_pattern("/usr/lib64/ghc-*/*/*.a", "devel") 211 | 212 | # Vala.. 213 | self.add_pattern("/usr/share/vala*/vapi/*", "devel") 214 | 215 | # KDE developer documentation 216 | self.add_pattern("/usr/share/doc/qt5/*.qch", "devel", 217 | priority=PRIORITY_DEFAULT+1) 218 | self.add_pattern("/usr/share/doc/qt5/*.tags", "devel", 219 | priority=PRIORITY_DEFAULT+1) 220 | 221 | def add_file(self, path): 222 | """ Add a file path to the owned list and place it into the correct 223 | package (main or named subpackage) according to the highest found 224 | priority pattern rule, otherwise it shall fallback under default 225 | policy into the main package itself. 226 | 227 | This enables a fallback approach, whereby subpackages "steal" from 228 | the main listing, and everything that is left is packaged into the 229 | main package (YpkgSpec::name), making "abandoned" files utterly 230 | impossible. """ 231 | 232 | target = "main" # default pattern name 233 | pattern = self.get_pattern(path) 234 | if pattern: 235 | target = self.patterns[pattern] 236 | 237 | permanent = False 238 | for perm in self.permanent: 239 | if perm.match(path): 240 | permanent = True 241 | break 242 | 243 | if target not in self.packages: 244 | self.packages[target] = Package(target) 245 | self.packages[target].add_file(pattern, path, permanent) 246 | 247 | def remove_file(self, path): 248 | """ Remove a file from our set, in any of our main or sub packages 249 | that may currently own it. """ 250 | 251 | for pkg in self.packages: 252 | self.packages[pkg].remove_file(path) 253 | 254 | def get_pattern(self, path): 255 | """ Return a matching pattern for the given path. 256 | This is ordered according to priority to enable 257 | multiple layers of priorities """ 258 | matches = [p for p in self.patterns if p.match(path)] 259 | if len(matches) == 0: 260 | return None 261 | 262 | matches = sorted(matches, key=StringPathGlob.get_priority, 263 | reverse=True) 264 | return matches[0] 265 | 266 | def add_pattern(self, pattern, pkgName, priority=PRIORITY_DEFAULT): 267 | """ Add a pattern to the internal map according to the 268 | given priority. """ 269 | 270 | obj = None 271 | is_prefix = False 272 | if pattern.endswith(os.sep): 273 | if not StringPathGlob.is_a_pattern(pattern): 274 | is_prefix = True 275 | 276 | obj = StringPathGlob(pattern, prefixMatch=is_prefix, priority=priority) 277 | self.patterns[obj] = pkgName 278 | 279 | def add_permanent_pattern(self, pattern): 280 | """ Add a pattern to our mapping of permanent paths. """ 281 | obj = None 282 | is_prefix = False 283 | if pattern.endswith(os.sep): 284 | if not StringPathGlob.is_a_pattern(pattern): 285 | is_prefix = True 286 | 287 | obj = StringPathGlob(pattern, prefixMatch=is_prefix) 288 | self.permanent.add(obj) 289 | 290 | def emit_packages(self): 291 | """ Ensure we've finalized our state, allowing proper theft and 292 | exclusion to take place, and then return all package objects 293 | that we've managed to generate. There is no gaurantee that 294 | a "main" package will be generated, as patterns may omit 295 | the production of one. """ 296 | 297 | for package in self.packages: 298 | for comparison in self.packages: 299 | if comparison == package: 300 | continue 301 | for file in self.packages[comparison].emit_files(): 302 | self.packages[package].exclude_file(file) 303 | 304 | def get_file_owner(self, file): 305 | """ Return the owning package for the specified file """ 306 | rname = os.path.realpath(file) 307 | for pkg in self.packages: 308 | package = self.packages[pkg] 309 | if file in package.files: 310 | return package 311 | elif rname in package.files: 312 | return package 313 | return None 314 | -------------------------------------------------------------------------------- /ypkg2/rc.yml: -------------------------------------------------------------------------------- 1 | actions: 2 | - configure: | 3 | ./configure %CONFOPTS% 4 | - reconfigure: | 5 | autoreconf -vfi 6 | %configure 7 | - autogen: | 8 | ./autogen.sh %CONFOPTS% 9 | %configure 10 | - make: | 11 | make %JOBS% 12 | - make_install: | 13 | %make install DESTDIR="%installroot%" 14 | - cmake: | 15 | cmake -DCMAKE_CFLAGS="${CFLAGS}" -DCMAKE_CXX_FLAGS="${CXXFLAGS}" \ 16 | -DCMAKE_LD_FLAGS="${LDFLAGS}" -DCMAKE_LIB_SUFFIX="%LIBSUFFIX%" \ 17 | -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_INSTALL_PREFIX=%PREFIX% 18 | - cmake_ninja: | 19 | function cmake_ninja() { 20 | mkdir solusBuildDir && pushd solusBuildDir 21 | cmake -G Ninja .. \ 22 | -DCMAKE_CFLAGS="${CFLAGS}" -DCMAKE_CXX_FLAGS="${CXXFLAGS}" \ 23 | -DCMAKE_LD_FLAGS="${LDFLAGS}" -DCMAKE_LIB_SUFFIX="%LIBSUFFIX%" \ 24 | -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_INSTALL_PREFIX=%PREFIX% $* || exit 1 25 | popd 26 | } 27 | cmake_ninja 28 | - meson_configure: | 29 | LC_ALL=en_US.utf8 CFLAGS="${CFLAGS}" CXXFLAGS="${CXXFLAGS}" LDFLAGS="${LDFLAGS}" meson --prefix %PREFIX% --buildtype=plain --libdir="lib%LIBSUFFIX%" --libexecdir="lib%LIBSUFFIX%/%PKGNAME%" --sysconfdir=/etc --localstatedir=/var solusBuildDir 30 | - ninja_build: &ninja_build | 31 | LC_ALL=en_US.utf8 ninja %JOBS% -C solusBuildDir 32 | - ninja_install: &ninja_install | 33 | LC_ALL=en_US.utf8 DESTDIR="%installroot%" ninja install %JOBS% -C solusBuildDir 34 | - ninja_check: &ninja_check | 35 | LC_ALL=en_US.utf8 ninja test %JOBS% -C solusBuildDir 36 | # These macros are for backward compatibility only. Please use the ninja ones 37 | - meson_build: *ninja_build 38 | - meson_install: *ninja_install 39 | - meson_check: *ninja_check 40 | - qmake: | 41 | qmake QMAKE_CFLAGS_RELEASE="${CFLAGS}" QMAKE_CXXFLAGS_RELEASE="${CXXFLAGS}" QMAKE_LFLAGS="${LDFLAGS}" 42 | - qmake4: | 43 | qmake-qt4 QMAKE_CFLAGS_RELEASE="${CFLAGS}" QMAKE_CXXFLAGS_RELEASE="${CXXFLAGS}" QMAKE_LFLAGS="${LDFLAGS}" QMAKE_LRELEASE=/usr/bin/lrelease-qt4 QMAKE_MOC=/usr/bin/moc-qt4 QMAKE_RCC=/usr/bin/rcc-qt4 QMAKE_UIC=/usr/bin/uic-qt4 44 | - qml_cache: | 45 | function generate_cache() { 46 | pushd $installdir 47 | for i in `find -type f -name "*.qml"`; do 48 | if ! [ -a "${i}"c ]; then 49 | qmlcachegen -o "${i}"c "${i}" $* 50 | fi 51 | done 52 | popd 53 | } 54 | generate_cache 55 | - patch: | 56 | patch -t -E --no-backup-if-mismatch -f 57 | # Only works if the user has a series file. They can provide the name to 58 | # override 'series' if needed 59 | - apply_patches: | 60 | function apply_patches() { 61 | if [[ ! -z "$1" ]]; then 62 | srs="$pkgfiles/$1" 63 | else 64 | srs="$pkgfiles/series" 65 | fi 66 | test -e "$srs" || exit 1 67 | while read -r pname ; do 68 | if [[ "$pname" == "" ]]; then 69 | continue 70 | fi 71 | %patch -p1 -i $pkgfiles/$pname 72 | done < $srs 73 | } 74 | apply_patches 75 | # Make life easier with Haskell 76 | - cabal_configure: | 77 | export GHCV=$(ghc --numeric-version) 78 | mkdir -p ~/.ghc/%ARCH%-linux-$GHCV/package.conf.d 79 | cp /usr/lib%LIBSUFFIX%/ghc-$GHCV/package.conf.d/* ~/.ghc/%ARCH%-linux-$GHCV/package.conf.d 80 | ghc-pkg recache --user 81 | cabal configure --prefix=%PREFIX% \ 82 | --libdir=%libdir% \ 83 | --libsubdir="\$compiler/\$pkgid" \ 84 | --sysconfdir=/etc \ 85 | --enable-executable-dynamic \ 86 | --enable-shared \ 87 | --ghc-options="-O2 -fPIC" 88 | - cabal_build: | 89 | cabal build %JOBS% 90 | - cabal_install: | 91 | cabal copy --destdir=$installdir 92 | - cabal_register: | 93 | export GHCV=$(ghc --numeric-version) 94 | cabal register --gen-pkg-config=$package-$version.conf 95 | install -D -m 00644 $package-$version.conf $installdir%libdir%/ghc-$GHCV/package.conf.d/$package-$version.conf 96 | # Make life easier with Perl 97 | - perl_setup: | 98 | function perl_setup() { 99 | if [[ -e Build.PL ]]; then 100 | perl Build.PL installdirs=vendor create_packlist=0 $* || exit 1 101 | else 102 | perl Makefile.PL PREFIX=%PREFIX% INSTALLDIRS=vendor DESTDIR="%installroot%" $* || exit 103 | fi 104 | } 105 | perl_setup 106 | - perl_build: | 107 | function perl_build() { 108 | if [[ -e Build.PL ]]; then 109 | perl Build installdirs=vendor create_packlist=0 $* || exit 1 110 | else 111 | %make $* || exit 1 112 | fi 113 | } 114 | perl_build 115 | # We need to nuke use of privlib because its always a perllocal.pod situation, 116 | # and these macros explicitly use vendor libs 117 | - perl_install: | 118 | function perl_install() { 119 | if [[ -e Build.PL ]]; then 120 | perl Build destdir="%installroot%" install $* || exit 1 121 | else 122 | %make_install $* || exit 1 123 | fi 124 | priv_lib="%perl_privlib%" 125 | if [[ -e "$installdir/$priv_lib" ]]; then 126 | rm -rfv "$installdir/$priv_lib" 127 | fi 128 | } 129 | perl_install 130 | - python_setup: | 131 | function python_setup() { 132 | if [[ -e $PKG_BUILD_DIR/.workdir ]]; then 133 | cd $(cat $PKG_BUILD_DIR/.workdir) 134 | else 135 | echo $workdir > $PKG_BUILD_DIR/.workdir 136 | fi 137 | 138 | instdir=`basename $PWD` 139 | pushd .. 140 | cp -a $instdir py2build && pushd py2build 141 | python2.7 setup.py build $* || exit 142 | popd 143 | popd 144 | } 145 | python_setup 146 | - python_install: | 147 | function python_install() { 148 | if [[ -e $PKG_BUILD_DIR/.workdir ]]; then 149 | cd $(cat $PKG_BUILD_DIR/.workdir) 150 | else 151 | echo $workdir > $PKG_BUILD_DIR/.workdir 152 | fi 153 | 154 | instdir=`basename $PWD` 155 | pushd .. 156 | if [[ ! -d py2build ]]; then 157 | cp -a $instdir py2build 158 | fi 159 | pushd py2build 160 | python2.7 setup.py install --root="%installroot%" $* || exit 161 | popd 162 | popd 163 | } 164 | python_install 165 | - python_compile: | 166 | function python_compile() { 167 | if [ -z "$1" ]; then 168 | python2 -m compileall -q $installdir || exit 1 169 | else 170 | python2 -m compileall -q $* || exit 1 171 | fi 172 | } 173 | python_compile 174 | - python3_setup: | 175 | function python3_setup() { 176 | if [[ -e $PKG_BUILD_DIR/.workdir ]]; then 177 | cd $(cat $PKG_BUILD_DIR/.workdir) 178 | else 179 | echo $workdir > $PKG_BUILD_DIR/.workdir 180 | fi 181 | 182 | instdir=`basename $PWD` 183 | pushd .. 184 | cp -a $instdir py3build && pushd py3build 185 | python3 setup.py build $* || exit 186 | popd 187 | popd 188 | } 189 | python3_setup 190 | - python3_install: | 191 | function python3_install() { 192 | if [[ -e $PKG_BUILD_DIR/.workdir ]]; then 193 | cd $(cat $PKG_BUILD_DIR/.workdir) 194 | else 195 | echo $workdir > $PKG_BUILD_DIR/.workdir 196 | fi 197 | 198 | instdir=`basename $PWD` 199 | pushd .. 200 | if [[ ! -d py3build ]]; then 201 | cp -a $instdir py3build 202 | fi 203 | pushd py3build 204 | python3 setup.py install --root="%installroot%" $* || exit 205 | popd 206 | popd 207 | } 208 | python3_install 209 | - python3_compile: | 210 | function python3_compile() { 211 | if [ -z "$1" ]; then 212 | python3 -m compileall -q $installdir || exit 1 213 | else 214 | python3 -m compileall -q $* || exit 1 215 | fi 216 | } 217 | python3_compile 218 | # Make life easier with Ruby gems 219 | - gem_build: | 220 | function gem_build() { 221 | if [ -z "$1" ]; then 222 | gem build *.gemspec || exit 1 223 | else 224 | gem build $* || exit 1 225 | fi 226 | } 227 | gem_build 228 | - gem_install: | 229 | function gem_install() { 230 | export geminstalldir=$(ruby -e'puts Gem.default_dir') 231 | export GEM_HOME=$geminstalldir 232 | export GEM_PATH=$geminstalldir 233 | if [ -a *.gem ]; then 234 | gem install --ignore-dependencies --no-user-install --no-document -i "$installdir/$geminstalldir" -n "$installdir/usr/bin" *.gem || exit 1 235 | else 236 | gem install --ignore-dependencies --no-user-install --no-document -i "$installdir/$geminstalldir" -n "$installdir/usr/bin" $sources/*.gem || exit 1 237 | fi 238 | if [[ -e "$installdir/$geminstalldir/cache" ]]; then 239 | rm -rfv $installdir/$geminstalldir/cache 240 | fi 241 | } 242 | gem_install 243 | - waf_configure: | 244 | ./waf configure --prefix=%PREFIX% 245 | - waf_build: | 246 | ./waf build --jobs="%YJOBS%" 247 | - waf_install: | 248 | ./waf install --jobs="%YJOBS%" --destdir="%installroot%" 249 | - llvm_profile_merge: | 250 | if [[ -d "$YPKG_PGO_DIR" ]]; then 251 | pushd "$YPKG_PGO_DIR" 252 | if [[ ! -e default.profdata ]]; then 253 | llvm-profdata merge -output=default.profdata default-*.profraw 254 | fi 255 | popd 256 | else 257 | echo "\n\nError: Profiling requested by ypkg but doesn't exist\n\n" 258 | fi 259 | defines: 260 | - CONFOPTS: | 261 | --prefix=%PREFIX% --build=%HOST% --libdir=%libdir% --mandir=/usr/share/man \ 262 | --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc \ 263 | --localstatedir=/var --libexecdir=%libdir%/%PKGNAME% 264 | # Inspired by the RPM guys :] 265 | - perl_privlib: | 266 | $(eval `perl -V:installprivlib`; echo $installprivlib) 267 | # Get LTS kernel version 268 | - kernel_version_lts: | 269 | $(echo "$(basename `readlink /usr/lib/kernel/default-lts` | cut -d '.' -f 4,5,6)".lts) 270 | # Get "Current" kernel version 271 | - kernel_version_current: | 272 | $(echo "$(basename `readlink /usr/lib/kernel/default-current` | cut -d '.' -f 4,5,6)".current) 273 | -------------------------------------------------------------------------------- /ypkg2/scripts.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | 16 | from collections import OrderedDict 17 | import re 18 | import os 19 | 20 | from yaml import load as yaml_load 21 | try: 22 | from yaml import CLoader as Loader 23 | except Exception as e: 24 | from yaml import Loader 25 | 26 | 27 | class ScriptGenerator: 28 | """ Generates build scripts on the fly by providing a default header 29 | tailored to the current build context and performing substitution 30 | on exported macros from this instance """ 31 | 32 | macros = None 33 | context = None 34 | spec = None 35 | exports = None 36 | unexports = None 37 | work_dir = None 38 | 39 | def __init__(self, context, spec, work_dir): 40 | self.work_dir = work_dir 41 | self.macros = OrderedDict() 42 | self.context = context 43 | self.spec = spec 44 | self.init_default_macros() 45 | self.load_system_macros() 46 | self.init_default_exports() 47 | 48 | def define_macro(self, key, value): 49 | """ Define a named macro. This will take the form %name% """ 50 | self.macros["%{}%".format(key)] = value 51 | 52 | def define_action_macro(self, key, value): 53 | """ Define an action macro. These take the form %action """ 54 | self.macros["%{}".format(key)] = value 55 | 56 | def define_export(self, key, value): 57 | """ Define a shell export for scripts """ 58 | self.exports[key] = value 59 | 60 | def define_unexport(self, key): 61 | """ Ensure key is unexported from shell script """ 62 | self.unexports[key] = (None,) 63 | 64 | def load_system_macros(self): 65 | path = os.path.join(os.path.dirname(__file__), "rc.yml") 66 | 67 | try: 68 | f = open(path, "r") 69 | yamlData = yaml_load(f, Loader=Loader) 70 | f.close() 71 | except Exception as e: 72 | console_ui.emit_error("SCRIPTS", "Cannot load system macros") 73 | print(e) 74 | return 75 | 76 | for section in ["defines", "actions"]: 77 | if section not in yamlData: 78 | continue 79 | v = yamlData[section] 80 | 81 | if not isinstance(v, list): 82 | console_ui.emit_error("rc.yml", 83 | "Expected list of defines in rc config") 84 | return 85 | for item in v: 86 | if not isinstance(item, dict): 87 | console_ui.emit_error("rc.yml", 88 | "Expected key:value mapping in list") 89 | return 90 | keys = item.keys() 91 | if len(keys) > 1: 92 | console_ui.emit_error("rc.yml", 93 | "Expected one key in key:value") 94 | return 95 | key = keys[0] 96 | value = item[key] 97 | if value.endswith("\n"): 98 | value = value[:-1] 99 | value = value.strip() 100 | if section == "defines": 101 | self.define_macro(key, unicode(value)) 102 | else: 103 | self.define_action_macro(key, unicode(value)) 104 | 105 | def init_default_macros(self): 106 | 107 | if self.context.emul32: 108 | if self.context.avx2: 109 | self.define_macro("libdir", "/usr/lib32/haswell") 110 | else: 111 | self.define_macro("libdir", "/usr/lib32") 112 | self.define_macro("LIBSUFFIX", "32") 113 | self.define_macro("PREFIX", "/usr") 114 | else: 115 | # 64-bit AVX2 build in subdirectory 116 | if self.context.avx2: 117 | self.define_macro("libdir", "/usr/lib64/haswell") 118 | else: 119 | self.define_macro("libdir", "/usr/lib64") 120 | self.define_macro("LIBSUFFIX", "64") 121 | self.define_macro("PREFIX", "/usr") 122 | 123 | self.define_macro("installroot", self.context.get_install_dir()) 124 | self.define_macro("workdir", self.work_dir) 125 | self.define_macro("JOBS", "-j{}".format(self.context.build.jobcount)) 126 | self.define_macro("YJOBS", "{}".format(self.context.build.jobcount)) 127 | 128 | # Consider moving this somewhere else 129 | self.define_macro("CFLAGS", " ".join(self.context.build.cflags)) 130 | self.define_macro("CXXFLAGS", " ".join(self.context.build.cxxflags)) 131 | self.define_macro("LDFLAGS", " ".join(self.context.build.ldflags)) 132 | 133 | self.define_macro("HOST", self.context.build.host) 134 | self.define_macro("ARCH", self.context.build.arch) 135 | self.define_macro("PKGNAME", self.spec.pkg_name) 136 | self.define_macro("PKGFILES", self.context.files_dir) 137 | 138 | self.define_macro("package", self.context.spec.pkg_name) 139 | self.define_macro("release", self.context.spec.pkg_release) 140 | self.define_macro("version", self.context.spec.pkg_version) 141 | self.define_macro("sources", self.context.get_sources_directory()) 142 | 143 | self.define_macro("rootdir", self.context.get_package_root_dir()) 144 | self.define_macro("builddir", self.context.get_build_dir()) 145 | 146 | def init_default_exports(self): 147 | """ Initialise our exports """ 148 | self.exports = OrderedDict() 149 | self.unexports = OrderedDict() 150 | 151 | self.define_export("CFLAGS", " ".join(self.context.build.cflags)) 152 | self.define_export("CXXFLAGS", " ".join(self.context.build.cxxflags)) 153 | self.define_export("LDFLAGS", " ".join(self.context.build.ldflags)) 154 | self.define_export("FFLAGS", " ".join(self.context.build.cxxflags)) 155 | self.define_export("FCFLAGS", " ".join(self.context.build.cxxflags)) 156 | self.define_export("PATH", self.context.get_path()) 157 | self.define_export("workdir", "%workdir%") 158 | self.define_export("package", "%package%") 159 | self.define_export("release", "%release%") 160 | self.define_export("version", "%version%") 161 | self.define_export("sources", "%sources%") 162 | self.define_export("pkgfiles", "%PKGFILES%") 163 | self.define_export("installdir", "%installroot%") 164 | self.define_export("PKG_ROOT_DIR", "%rootdir%") 165 | # Build dir, which is one level up from the source directory. 166 | self.define_export("PKG_BUILD_DIR", "%builddir%") 167 | self.define_export("CC", self.context.build.cc) 168 | self.define_export("CXX", self.context.build.cxx) 169 | if self.context.build.ld_as_needed: 170 | self.define_export("LD_AS_NEEDED", "1") 171 | 172 | # Handle lto correctly 173 | if self.context.spec.pkg_optimize and not self.context.spec.pkg_clang: 174 | if "thin-lto" in self.context.spec.pkg_optimize \ 175 | or "lto" in self.context.spec.pkg_optimize: 176 | self.define_export("AR", "gcc-ar") 177 | self.define_export("RANLIB", "gcc-ranlib") 178 | self.define_export("NM", "gcc-nm") 179 | 180 | if not console_ui.allow_colors: 181 | self.define_export("TERM", "dumb") 182 | 183 | # Mask display 184 | self.define_unexport("DISPLAY") 185 | # Mask sudo from anyone 186 | self.define_unexport("SUDO_USER") 187 | self.define_unexport("SUDO_GID") 188 | self.define_unexport("SUDO_UID") 189 | self.define_unexport("SUDO_COMMAND") 190 | self.define_unexport("CDPATH") 191 | 192 | def emit_exports(self): 193 | """ TODO: Grab known exports into an OrderedDict populated by an rc 194 | YAML file to allow easier manipulation """ 195 | ret = [] 196 | for key in self.exports: 197 | ret.append("export {}=\"{}\"".format(key, self.exports[key])) 198 | 199 | unset_line = "unset {} || :".format(" ".join(self.unexports.keys())) 200 | ret.append(unset_line) 201 | return ret 202 | 203 | def is_valid_macro_char(self, char): 204 | if char.isalpha() or char.isdigit(): 205 | return True 206 | if char == "_": 207 | return True 208 | 209 | def escape_single(self, line): 210 | offset = line.find('%') 211 | if offset < 0: 212 | return (line, False) 213 | 214 | tmp_name = "%" 215 | tmp_idx = 0 216 | for i in xrange(offset+1, len(line)): 217 | if line[i] == "%": 218 | tmp_name += "%" 219 | break 220 | if self.is_valid_macro_char(line[i]): 221 | tmp_name += line[i] 222 | else: 223 | break 224 | start = line[0:offset] 225 | remnant = line[offset+len(tmp_name):] 226 | # TODO: Change to is-valid-macro check and consume anyway 227 | if tmp_name in self.macros: 228 | mc = self.macros[tmp_name] 229 | if mc is None: 230 | mc = "" 231 | line = "%s%s%s" % (start, mc, remnant) 232 | return (line, True) 233 | else: 234 | line = "%s%s%s" % (start, tmp_name, remnant) 235 | return (line, False) 236 | 237 | def escape_string(self, input_string): 238 | """ Recursively escape our macros out of a string until no more of our 239 | macros appear in it """ 240 | ret = [] 241 | 242 | for line in input_string.split("\n"): 243 | while (True): 244 | (line, cont) = self.escape_single(line) 245 | if not cont: 246 | ret.append(line) 247 | break 248 | 249 | return "\n".join(ret) 250 | -------------------------------------------------------------------------------- /ypkg2/sources.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | 16 | import os 17 | import hashlib 18 | import subprocess 19 | import fnmatch 20 | import shutil 21 | 22 | KnownSourceTypes = { 23 | 'tar': [ 24 | '*.tar.*', 25 | '*.tgz', 26 | ], 27 | 'zip': [ 28 | '*.zip', 29 | ], 30 | } 31 | 32 | 33 | class YpkgSource: 34 | 35 | def __init__(self): 36 | pass 37 | 38 | def fetch(self, context): 39 | """ Fetch this source from it's given location """ 40 | return False 41 | 42 | def verify(self, context): 43 | """ Verify the locally obtained source """ 44 | return False 45 | 46 | def extract(self, context): 47 | """ Attempt extraction of this source type, if needed """ 48 | return False 49 | 50 | def remove(self, context): 51 | """ Attempt removal of this source type """ 52 | return False 53 | 54 | def cached(self, context): 55 | """ Report on whether this source is cached """ 56 | return False 57 | 58 | 59 | class GitSource(YpkgSource): 60 | """ Provides git source support to ypkg """ 61 | 62 | # Source URI 63 | uri = None 64 | 65 | # Tag or branch to check out 66 | tag = None 67 | 68 | def __init__(self, uri, tag): 69 | YpkgSource.__init__(self) 70 | self.uri = uri 71 | self.tag = tag 72 | self.filename = self.get_target_name() 73 | 74 | def __str__(self): 75 | return "{} ({})".format(self.uri, self.tag) 76 | 77 | def is_dumb_transport(self): 78 | """ Http depth cloning = no go """ 79 | if self.uri.startswith("http:") or self.uri.startswith("https:"): 80 | return True 81 | return False 82 | 83 | def get_target_name(self): 84 | """ Get the target directory base name after its fetched """ 85 | uri = str(self.uri) 86 | if uri.endswith(".git"): 87 | uri = uri[:-4] 88 | return os.path.basename(uri) + ".git" 89 | 90 | def get_full_path(self, context): 91 | """ Fully qualified target path """ 92 | return os.path.join(context.get_sources_directory(), 93 | self.get_target_name()) 94 | 95 | def fetch(self, context): 96 | """ Clone the actual git repo, favouring efficiency... """ 97 | source_dir = context.get_sources_directory() 98 | 99 | # Ensure source dir exists 100 | if not os.path.exists(source_dir): 101 | try: 102 | os.makedirs(source_dir, mode=00755) 103 | except Exception as e: 104 | console_ui.emit_error("Source", "Cannot create sources " 105 | "directory: {}".format(e)) 106 | return False 107 | 108 | cmd = "git -C \"{}\" clone \"{}\" {}".format( 109 | source_dir, self.uri, self.get_target_name()) 110 | 111 | console_ui.emit_info("Git", "Fetching: {}".format(self.uri)) 112 | try: 113 | r = subprocess.check_call(cmd, shell=True) 114 | except Exception as e: 115 | console_ui.emit_error("Git", "Failed to fetch {}".format( 116 | self.uri)) 117 | print("Error follows: {}".format(e)) 118 | return False 119 | 120 | console_ui.emit_info("Git", "Checking out: {}".format(self.tag)) 121 | cmd = "git -C \"{}\" checkout \"{}\"".format( 122 | os.path.join(source_dir, self.get_target_name()), 123 | self.tag) 124 | try: 125 | r = subprocess.check_call(cmd, shell=True) 126 | except Exception as e: 127 | console_ui.emit_error("Git", "Failed to checkout {}".format( 128 | self.tag)) 129 | return False 130 | 131 | ddir = os.path.join(source_dir, self.get_target_name()) 132 | if not os.path.exists(os.path.join(ddir, ".gitmodules")): 133 | return True 134 | 135 | cmd1 = "git -C \"{}\" submodule init".format(ddir) 136 | cmd2 = "git -C \"{}\" submodule update".format(ddir) 137 | 138 | try: 139 | r = subprocess.check_call(cmd1, shell=True) 140 | r = subprocess.check_call(cmd2, shell=True) 141 | except Exception as e: 142 | console_ui.emit_error("Git", "Failed to submodule init {}".format( 143 | e)) 144 | return False 145 | return True 146 | 147 | def verify(self, context): 148 | """ Verify source = good. """ 149 | bpath = self.get_full_path(context) 150 | 151 | status_cmd = "git -C {} diff --exit-code" 152 | try: 153 | subprocess.check_call(status_cmd.format(bpath), shell=True) 154 | except Exception: 155 | console_ui.emit_error("Git", "Unexpected diff in source") 156 | return False 157 | return True 158 | 159 | def extract(self, context): 160 | """ Extract ~= copy source into build area. Nasty but original source 161 | should not be tainted between runs. 162 | """ 163 | 164 | source = self.get_full_path(context) 165 | target = os.path.join(context.get_build_dir(), 166 | self.get_target_name()) 167 | 168 | if os.path.exists(target): 169 | try: 170 | shutil.rmtree(target) 171 | except Exception as e: 172 | console_ui.emit_error("Git", "Cannot remove stagnant tree") 173 | print(e) 174 | return False 175 | 176 | if not os.path.exists(context.get_build_dir()): 177 | try: 178 | os.makedirs(context.get_build_dir(), mode=00755) 179 | except Exception as e: 180 | console_ui.emit_error("Source", "Cannot create sources " 181 | "directory: {}".format(e)) 182 | return False 183 | try: 184 | cmd = "cp -Ra \"{}/\" \"{}\"".format(source, target) 185 | subprocess.check_call(cmd, shell=True) 186 | except Exception as e: 187 | console_ui.emit_error("Git", "Failed to copy source to build") 188 | print(e) 189 | return False 190 | return True 191 | 192 | def cached(self, context): 193 | bpath = self.get_full_path(context) 194 | return os.path.exists(bpath) 195 | 196 | 197 | class TarSource(YpkgSource): 198 | """ Represents a simple tarball source """ 199 | 200 | uri = None 201 | hash = None 202 | filename = None 203 | 204 | def __init__(self, uri, hash): 205 | YpkgSource.__init__(self) 206 | self.uri = uri 207 | self.filename = os.path.basename(uri) 208 | self.hash = hash 209 | 210 | def __str__(self): 211 | return "%s (%s)" % (self.uri, self.hash) 212 | 213 | def _get_full_path(self, context): 214 | bpath = os.path.join(context.get_sources_directory(), 215 | self.filename) 216 | return bpath 217 | 218 | def fetch(self, context): 219 | source_dir = context.get_sources_directory() 220 | 221 | # Ensure source dir exists 222 | if not os.path.exists(source_dir): 223 | try: 224 | os.makedirs(source_dir, mode=00755) 225 | except Exception as e: 226 | console_ui.emit_error("Source", "Cannot create sources " 227 | "directory: {}".format(e)) 228 | return False 229 | 230 | console_ui.emit_info("Source", "Fetching: {}".format(self.uri)) 231 | fpath = self._get_full_path(context) 232 | cmd = "curl -o \"{}\" --url \"{}\" --location".format( 233 | fpath, self.uri) 234 | try: 235 | r = subprocess.check_call(cmd, shell=True) 236 | except Exception as e: 237 | console_ui.emit_error("Source", "Failed to fetch {}".format( 238 | self.uri)) 239 | print("Error follows: {}".format(e)) 240 | return False 241 | 242 | return True 243 | 244 | def verify(self, context): 245 | bpath = self._get_full_path(context) 246 | 247 | hash = None 248 | 249 | with open(bpath, "r") as inp: 250 | h = hashlib.sha256() 251 | h.update(inp.read()) 252 | hash = h.hexdigest() 253 | if hash != self.hash: 254 | console_ui.emit_error("Source", "Incorrect hash for {}". 255 | format(self.filename)) 256 | print("Found hash : {}".format(hash)) 257 | print("Expected hash : {}".format(self.hash)) 258 | return False 259 | return True 260 | 261 | target = os.path.join(BallDir, os.path.basename(x)) 262 | ext = "unzip" if target.endswith(".zip") else "tar xf" 263 | diropt = "-d" if target.endswith(".zip") else "-C" 264 | cmd = "%s \"%s\" %s \"%s\"" % (ext, target, diropt, bd) 265 | 266 | def get_extract_command_zip(self, context, bpath): 267 | """ Get a command tailored for zip usage """ 268 | cmd = "unzip \"{}\" -d \"{}/\"".format(bpath, context.get_build_dir()) 269 | return cmd 270 | 271 | def get_extract_command_tar(self, context, bpath): 272 | """ Get a command tailored for tar usage """ 273 | if os.path.exists("/usr/bin/bsdtar"): 274 | frag = "bsdtar" 275 | else: 276 | frag = "tar" 277 | cmd = "{} xf \"{}\" -C \"{}/\"".format( 278 | frag, bpath, context.get_build_dir()) 279 | return cmd 280 | 281 | def extract(self, context): 282 | """ Extract an archive into the context.get_build_dir() """ 283 | bpath = self._get_full_path(context) 284 | 285 | # Grab the correct extraction command 286 | fileType = None 287 | for key in KnownSourceTypes: 288 | lglobs = KnownSourceTypes[key] 289 | for lglob in lglobs: 290 | if fnmatch.fnmatch(self.filename, lglob): 291 | fileType = key 292 | break 293 | if fileType: 294 | break 295 | 296 | if not fileType: 297 | console_ui.emit_warning("Source", "Type of file {} is unknown, " 298 | "falling back to tar handler". 299 | format(self.filename)) 300 | fileType = "tar" 301 | 302 | cmd_name = "get_extract_command_{}".format(fileType) 303 | if not hasattr(self, cmd_name): 304 | console_ui.emit_error("Source", "Fatal error: No handler for {}". 305 | format(fileType)) 306 | return False 307 | 308 | if not os.path.exists(context.get_build_dir()): 309 | try: 310 | os.makedirs(context.get_build_dir(), mode=00755) 311 | except Exception as e: 312 | console_ui.emit_error("Source", "Failed to construct build " 313 | "directory") 314 | print(e) 315 | return False 316 | 317 | cmd = getattr(self, cmd_name)(context, bpath) 318 | try: 319 | subprocess.check_call(cmd, shell=True) 320 | except Exception as e: 321 | console_ui.emit_error("Source", "Failed to extract {}". 322 | format(self.filename)) 323 | return False 324 | return True 325 | 326 | def remove(self, context): 327 | console_ui.emit_error("Source", "Remove not yet implemented") 328 | return False 329 | 330 | def cached(self, context): 331 | bpath = self._get_full_path(context) 332 | return os.path.exists(bpath) 333 | 334 | 335 | class SourceManager: 336 | """ Responsible for identifying, fetching, and verifying sources as listed 337 | within a YpkgSpec. """ 338 | 339 | sources = None 340 | 341 | def __init__(self): 342 | self.sources = list() 343 | 344 | def identify_sources(self, spec): 345 | if not spec: 346 | return False 347 | 348 | for source in spec.pkg_source: 349 | if not isinstance(source, dict): 350 | console_ui.emit_error("SOURCE", 351 | "Source lines must be of 'key : value' " 352 | "mapping type") 353 | print("Erronous line: {}".format(str(source))) 354 | return False 355 | 356 | if len(source.keys()) != 1: 357 | console_ui.emit_error("SOURCE", 358 | "Encountered too many keys in source") 359 | print("Erronous source: {}".format(str(source))) 360 | return False 361 | 362 | uri = source.keys()[0] 363 | hash = source[uri] 364 | # Check if its a namespaced support type 365 | if "|" in uri: 366 | brk = uri.split("|") 367 | # It's a git| prefix 368 | if brk[0] == 'git': 369 | uri = "|".join(brk[1:]) 370 | self.sources.append(GitSource(uri, hash)) 371 | continue 372 | self.sources.append(TarSource(uri, hash)) 373 | 374 | return True 375 | 376 | def _get_working_dir(self, context): 377 | """ Need to make this.. better. It's very tar-type now""" 378 | build_dir = context.get_build_dir() 379 | 380 | source0 = self.sources[0].filename 381 | if os.path.exists(build_dir): 382 | items = os.listdir(build_dir) 383 | if len(items) == 1: 384 | return os.path.join(build_dir, items[0]) 385 | for item in items: 386 | if source0.startswith(item): 387 | return os.path.join(build_dir, item) 388 | return build_dir 389 | else: 390 | return build_dir 391 | 392 | def get_working_dir(self, context): 393 | potential = self._get_working_dir(context) 394 | if not os.path.exists(potential) or not os.path.isdir(potential): 395 | return context.get_build_dir() 396 | return potential 397 | -------------------------------------------------------------------------------- /ypkg2/stringglob.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Re-adapted into ypkg2 from autospec 5 | # 6 | # Copyright (C) 2016-2017 Intel Corporation 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | import fnmatch 15 | import os 16 | 17 | 18 | class StringPathGlob: 19 | 20 | pattern = None 21 | prefixMatch = False 22 | priority = 0 23 | 24 | def __init__(self, pattern, prefixMatch=False, priority=0): 25 | self.pattern = pattern 26 | self.prefixMatch = prefixMatch 27 | self.priority = priority 28 | 29 | @staticmethod 30 | def is_a_pattern(item): 31 | if "[" in item or "?" in item or "*" in item: 32 | return True 33 | return False 34 | 35 | def match(self, path): 36 | if self.prefixMatch: 37 | if self.pattern.endswith(os.sep) and not \ 38 | StringPathGlob.is_a_pattern(self.pattern): 39 | if path.startswith(self.pattern): 40 | return True 41 | return False 42 | 43 | our_splits = self.pattern.split(os.sep) 44 | test_splits = path.split(os.sep) 45 | 46 | their_len = len(test_splits) 47 | our_len = len(our_splits) 48 | 49 | if our_len > their_len: 50 | return False 51 | 52 | for i in range(0, our_len): 53 | our_elem = our_splits[i] 54 | their_elem = test_splits[i] 55 | 56 | if our_elem == their_elem: 57 | continue 58 | 59 | if StringPathGlob.is_a_pattern(our_elem): 60 | if not fnmatch.fnmatchcase(their_elem, our_elem): 61 | return False 62 | else: 63 | return False 64 | 65 | return True 66 | 67 | def __str__(self): 68 | return str(self.pattern) 69 | 70 | def __eq__(self, obj2): 71 | return self.pattern == obj2.pattern 72 | 73 | def __ne__(self, obj): 74 | return not(self == obj) 75 | 76 | def __hash__(self): 77 | return hash((self.pattern, self.priority)) 78 | 79 | def get_pattern(self): 80 | return self.pattern 81 | 82 | def get_priority(self): 83 | return self.priority 84 | -------------------------------------------------------------------------------- /ypkg2/ui.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | 15 | class AnsiColors: 16 | """ ANSI sequences for color output """ 17 | 18 | RED = '\033[31m' 19 | GREEN = '\033[32m' 20 | YELLOW = '\033[33m' 21 | BLUE = '\033[34m' 22 | MAGENTA = '\033[35m' 23 | CYAN = '\033[36m' 24 | LGREY = '\033[37m' 25 | DGREY = '\033[90m' 26 | LRED = '\033[91m' 27 | LGREEN = '\033[92m' 28 | LYELLOW = '\033[93m' 29 | LBLUE = '\033[94m' 30 | LMAGENTA = '\033[95m' 31 | LCYAN = '\033[96m' 32 | WHITE = '\033[97m' 33 | 34 | RESET = '\033[0m' 35 | 36 | BOLD = '\033[1m' 37 | UNBOLD = '\033[21m' 38 | 39 | DIM = '\033[2m' 40 | UNDIM = '\033[22m' 41 | 42 | UNDERLINE = '\033[4m' 43 | UNUNDERLINE = '\033[24m' 44 | 45 | BLINK = '\033[5m' 46 | UNBLINK = '\033[25m' 47 | 48 | REVERSE = '\033[7m' 49 | UNREVERSE = '\033[27m' 50 | 51 | HIDEEN = '\033[8m' 52 | UNHIDDEN = '\033[28m' 53 | 54 | 55 | class YpkgUI: 56 | 57 | """ We must allow toggling of colors in the UI """ 58 | allow_colors = False 59 | 60 | def __init__(self): 61 | self.allow_colors = True 62 | 63 | def emit_error(self, key, error): 64 | """ Report an error to the user """ 65 | if not self.allow_colors: 66 | print("[{}] {}".format(key, error)) 67 | else: 68 | print("{}[{}]{} {}{}{}".format(AnsiColors.RED, key, 69 | AnsiColors.RESET, AnsiColors.BOLD, error, AnsiColors.RESET)) 70 | 71 | def emit_warning(self, key, warn): 72 | """ Report a warning to the user """ 73 | if not self.allow_colors: 74 | print("[{}] {}".format(key, warn)) 75 | else: 76 | print("{}[{}]{} {}{}{}".format(AnsiColors.YELLOW, key, 77 | AnsiColors.RESET, AnsiColors.BOLD, warn, AnsiColors.RESET)) 78 | 79 | def emit_info(self, key, info): 80 | """ Report information to the user """ 81 | if not self.allow_colors: 82 | print("[{}] {}".format(key, info)) 83 | else: 84 | print("{}[{}]{} {}".format(AnsiColors.BLUE, key, 85 | AnsiColors.RESET, info)) 86 | 87 | def emit_success(self, key, success): 88 | """ Report success to the user """ 89 | if not self.allow_colors: 90 | print("[{}] {}".format(key, success)) 91 | else: 92 | print("{}[{}]{} {}".format(AnsiColors.GREEN, key, 93 | AnsiColors.RESET, success)) 94 | -------------------------------------------------------------------------------- /ypkg2/yamlhelper.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | 16 | import yaml 17 | import os 18 | import sys 19 | import re 20 | from collections import OrderedDict 21 | 22 | iterable_types = [list, dict] 23 | 24 | 25 | class OneOrMoreString: 26 | """ Request one or more string """ 27 | def __init__(self): 28 | pass 29 | 30 | 31 | class MultimapFormat: 32 | """ Request items in a multimap format """ 33 | 34 | ref_object = None 35 | ref_function = None 36 | ref_default = None 37 | 38 | def __init__(self, ref_object, ref_function, ref_default): 39 | self.ref_object = ref_object 40 | self.ref_function = ref_function 41 | self.ref_default = unicode(ref_default) 42 | 43 | 44 | def _insert_helper(mapping, key, value): 45 | """ Helper to prevent repetetive code """ 46 | if key not in mapping: 47 | mapping[key] = list() 48 | mapping[key].append(value) 49 | 50 | 51 | def get_key_value_mapping(data, t): 52 | mapping = OrderedDict() 53 | 54 | dicts = filter(lambda s: isinstance(s, dict), data) 55 | no_keys = filter(lambda s: type(s) not in iterable_types, data) 56 | 57 | for key in no_keys: 58 | _insert_helper(mapping, t.ref_default, key) 59 | 60 | # Explicit key: to value mapping 61 | for mapp in dicts: 62 | keys = mapp.keys() 63 | if len(keys) > 1: 64 | console_ui.emit_error("YAML", 65 | "Encountered multiple keys") 66 | return False 67 | key = keys[0] 68 | val = mapp[key] 69 | 70 | if isinstance(val, list): 71 | bad = filter(lambda s: type(s) in iterable_types, val) 72 | if len(bad) > 0: 73 | console_ui.emit_error("YAML", 74 | "Multimap does not support inception...") 75 | return None 76 | for item in val: 77 | _insert_helper(mapping, key, unicode(item)) 78 | continue 79 | elif type(val) in iterable_types: 80 | # Illegal to have a secondary layer here!! 81 | console_ui.emit_error("YAML", "Expected a value here") 82 | print("Erronous line: {}".format(str(mapp))) 83 | return None 84 | else: 85 | # This is key->value mapping 86 | _insert_helper(mapping, key, unicode(val)) 87 | 88 | return mapping 89 | 90 | 91 | def assertMultimap(ymlFile, key, t): 92 | """ Perform multi-map operations in the given key """ 93 | 94 | if key not in ymlFile: 95 | console_ui.emit_error("YAML:{}".format(key), 96 | "Fatally requested a non-existent key!") 97 | return False 98 | 99 | val = ymlFile[key] 100 | if type(val) not in iterable_types: 101 | mapping = get_key_value_mapping([unicode(val)], t) 102 | else: 103 | mapping = get_key_value_mapping(ymlFile[key], t) 104 | 105 | if mapping is None: 106 | return False 107 | 108 | for key in mapping.keys(): 109 | dat = mapping[key] 110 | for val in dat: 111 | t.ref_function(unicode(key), unicode(val)) 112 | 113 | return True 114 | 115 | 116 | def assertGetType(ymlFile, key, t): 117 | """ Ensure a value of the given type exists """ 118 | if key not in ymlFile: 119 | console_ui.emit_error("YAML", 120 | "Mandatory token '{}' is missing".format(key)) 121 | return None 122 | val = ymlFile[key] 123 | if val is None: 124 | console_ui.emit_error("YAML:{}".format(key), 125 | "Mandatory token cannot be empty") 126 | return None 127 | val_type = type(val) 128 | # YAML might report integer when we want strings, which is OK. 129 | 130 | if t == OneOrMoreString: 131 | ret = list() 132 | if val_type == str or val_type == unicode: 133 | ret.append(val) 134 | return ret 135 | if val_type != list: 136 | console_ui.emit_error("YAML:{}".format(key), 137 | "Token must be a string or list of strings") 138 | return None 139 | for item in val: 140 | if type(item) in iterable_types: 141 | console_ui.emit_error("YAML:{}".format(key), 142 | "Found unexpected iterable type in list") 143 | console_ui.emit_error("YAML:{}".format(key), 144 | "Expected a string") 145 | return None 146 | ret.append(item) 147 | return ret 148 | 149 | if t == str: 150 | if type(val) not in iterable_types: 151 | val = str(val) 152 | elif t == unicode: 153 | if type(val) not in iterable_types: 154 | val = unicode(val) 155 | 156 | if not isinstance(val, t): 157 | j = t.__name__ 158 | f = type(val).__name__ 159 | console_ui.emit_error("YAML", 160 | "Token '{}' must be of type '{}'".format(key, j)) 161 | console_ui.emit_error("YAML:{}".format(key), 162 | "Token found was of type '{}".format(f)) 163 | return None 164 | return val 165 | -------------------------------------------------------------------------------- /ypkg2/ypkgcontext.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | 16 | import pisi.config 17 | import os 18 | import shutil 19 | import multiprocessing 20 | 21 | # This speed flag set was originally from autospec in 22 | # Clear Linux Project For Intel Architecture. 23 | SPEED_FLAGS = "-fno-semantic-interposition -O3 -falign-functions=32" 24 | 25 | # Clang defaults to -fno-semantic-interposition behaviour but doesn't have a 26 | # CLI flag to control it. It also does a better job on function alignment. 27 | SPEED_FLAGS_CLANG = "-O3" 28 | 29 | BIND_NOW_FLAGS = ["-Wl,-z,now", "-Wl,-z -Wl,relro", "-Wl,-z -Wl,now"] 30 | 31 | # Allow turning off the symbolic functions linker flag 32 | SYMBOLIC_FLAGS = ["-Wl,-Bsymbolic-functions"] 33 | 34 | # Allow optimizing for size 35 | SIZE_FLAGS = "-Os" 36 | 37 | # Unfortunately clang's LLVMGold will not accept -Os and is broken by design 38 | SIZE_FLAGS_CLANG = "-O2" 39 | 40 | # Allow optimizing for lto 41 | LTO_FLAGS = "-flto" 42 | 43 | # Use linker plugin when not compiling with Clang 44 | LTO_FLAGS_GCC = "-fuse-linker-plugin" 45 | 46 | # Use gold for thin LTO 47 | THIN_LTO_FLAGS = "-flto=thin -fuse-ld=gold" 48 | 49 | # Allow unrolling loops 50 | UNROLL_LOOPS_FLAGS = "-funroll-loops" 51 | 52 | # GCC PGO flags 53 | PGO_GEN_FLAGS = "-fprofile-generate -fprofile-dir=\"{}\" " 54 | PGO_USE_FLAGS = "-fprofile-use -fprofile-dir=\"{}\" -fprofile-correction" 55 | 56 | # Clang can handle parameters to the args unlike GCC 57 | PGO_GEN_FLAGS_CLANG = "-fprofile-instr-generate=\"{}/default-%m.profraw\"" 58 | PGO_USE_FLAGS_CLANG = "-fprofile-instr-use=\"{}/default.profdata\" " \ 59 | "-fprofile-correction" 60 | 61 | # AVX2 62 | AVX2_ARCH = "haswell" 63 | AVX2_TUNE = "haswell" 64 | 65 | 66 | class Flags: 67 | 68 | C = 0 69 | CXX = 1 70 | LD = 2 71 | 72 | @staticmethod 73 | def get_desc(f): 74 | ''' Get descriptor for flag type ''' 75 | if f == Flags.C: 76 | return "CFLAGS" 77 | elif f == Flags.CXX: 78 | return "CXXFLAGS" 79 | elif f == Flags.LD: 80 | return "LDFLAGS" 81 | else: 82 | return "UNKNOWN_FLAG_SET_CHECK_IT" 83 | 84 | @staticmethod 85 | def filter_flags(f, filters): 86 | """ Filter the flags from this set """ 87 | nflags = filter(lambda s: s not in filters, f) 88 | return nflags 89 | 90 | @staticmethod 91 | def optimize_flags(f, opt_type, clang=False): 92 | """ Optimize this flag set for a given optimisation type """ 93 | newflags = f 94 | if opt_type == "speed" or opt_type == "size": 95 | # Only filter optimisation levels when changing it 96 | optimisations = ["-O%s" % x for x in range(0, 4)] 97 | optimisations.extend("-Os") 98 | 99 | newflags = Flags.filter_flags(f, optimisations) 100 | if opt_type == "speed": 101 | if clang: 102 | newflags.extend(SPEED_FLAGS_CLANG.split(" ")) 103 | else: 104 | newflags.extend(SPEED_FLAGS.split(" ")) 105 | else: 106 | if clang: 107 | newflags.extend(SIZE_FLAGS_CLANG.split(" ")) 108 | else: 109 | newflags.extend(SIZE_FLAGS.split(" ")) 110 | elif opt_type == "lto": 111 | newflags.extend(LTO_FLAGS.split(" ")) 112 | if not clang: 113 | newflags.extend(LTO_FLAGS_GCC.split(" ")) 114 | elif opt_type == "unroll-loops": 115 | newflags.extend(UNROLL_LOOPS_FLAGS.split(" ")) 116 | elif opt_type == "no-bind-now": 117 | newflags = Flags.filter_flags(f, BIND_NOW_FLAGS) 118 | elif opt_type == "no-symbolic": 119 | newflags = Flags.filter_flags(f, SYMBOLIC_FLAGS) 120 | elif opt_type == "thin-lto": 121 | newflags.extend(THIN_LTO_FLAGS.split(" ")) 122 | if not clang: 123 | newflags.extend(LTO_FLAGS_GCC.split(" ")) 124 | else: 125 | console_ui.emit_warning("Flags", "Unknown optimization: {}". 126 | format(opt_type)) 127 | return f 128 | return newflags 129 | 130 | @staticmethod 131 | def pgo_gen_flags(f, d, clang=False): 132 | """ Update flags with PGO generator flags """ 133 | r = list(f) 134 | flagSet = PGO_GEN_FLAGS if not clang else PGO_GEN_FLAGS_CLANG 135 | r.extend((flagSet.format(d).split(" "))) 136 | return r 137 | 138 | @staticmethod 139 | def pgo_use_flags(f, d, clang=False): 140 | """ Update flags with PGO use flags """ 141 | r = list(f) 142 | flagSet = PGO_USE_FLAGS if not clang else PGO_USE_FLAGS_CLANG 143 | r.extend((flagSet.format(d).split(" "))) 144 | return r 145 | 146 | 147 | class BuildConfig: 148 | 149 | arch = None 150 | host = None 151 | ccache = None 152 | 153 | cflags = None 154 | cxxflags = None 155 | ldflags = None 156 | 157 | cc = None 158 | cxx = None 159 | 160 | ld_as_needed = True # Make this configurable at some point. 161 | 162 | jobcount = 4 163 | 164 | def get_flags(self, t): 165 | """ Simple switch to grab a set of flags by a type """ 166 | if t == Flags.C: 167 | return self.cflags 168 | if t == Flags.CXX: 169 | return self.cxxflags 170 | if t == Flags.LD: 171 | return self.ldflags 172 | return set([]) 173 | 174 | 175 | class YpkgContext: 176 | """ Base context for things like cflags, etc """ 177 | 178 | build = None 179 | 180 | global_archive_dir = None 181 | is_root = False 182 | spec = None 183 | emul32 = False 184 | files_dir = None 185 | pconfig = None 186 | avx2 = False 187 | use_pgo = None 188 | gen_pgo = None 189 | 190 | can_dbginfo = False 191 | 192 | def __init__(self, spec, emul32=False, avx2=False): 193 | self.spec = spec 194 | self.emul32 = emul32 195 | self.avx2 = avx2 196 | self.build = BuildConfig() 197 | self.init_config() 198 | if os.geteuid() == 0 and "FAKED_MODE" not in os.environ: 199 | self.is_root = True 200 | 201 | def get_path(self): 202 | """ Return the path, mutated to include ccache if needed """ 203 | default_path = "/usr/bin:/bin:/usr/sbin:/sbin" 204 | 205 | if not self.spec.pkg_ccache: 206 | return default_path 207 | if not self.build.ccache: 208 | return default_path 209 | 210 | ccaches = ["/usr/lib64/ccache/bin", "/usr/lib/ccache/bin"] 211 | for i in ccaches: 212 | if os.path.exists(i): 213 | console_ui.emit_info("Build", "Enabling ccache") 214 | return "{}:{}".format(i, default_path) 215 | return default_path 216 | 217 | def get_sources_directory(self): 218 | """ Get the configured source directory for fetching sources to """ 219 | if self.is_root: 220 | return self.global_archive_dir 221 | return os.path.join(self.get_build_prefix(), "sources") 222 | 223 | def get_build_prefix(self): 224 | """ Get the build prefix used by ypkg """ 225 | if self.is_root: 226 | return "/var/ypkg-root" 227 | return "{}/YPKG".format(os.path.expanduser("~")) 228 | 229 | def get_install_dir(self): 230 | """ Get the install directory for the given package """ 231 | return os.path.abspath("{}/root/{}/install".format( 232 | self.get_build_prefix(), 233 | self.spec.pkg_name)) 234 | 235 | def get_packaging_dir(self): 236 | """ The temporary packaging directory """ 237 | return os.path.abspath("{}/root/{}/pkg".format( 238 | self.get_build_prefix(), 239 | self.spec.pkg_name)) 240 | 241 | def get_build_dir(self): 242 | """ Get the build directory for the given package """ 243 | buildSuffix = "build" 244 | if self.avx2: 245 | if self.emul32: 246 | buildSuffix = "build-32-avx2" 247 | else: 248 | buildSuffix = "build-avx2" 249 | elif self.emul32: 250 | buildSuffix = "build-32" 251 | 252 | return os.path.abspath("{}/root/{}/{}".format( 253 | self.get_build_prefix(), 254 | self.spec.pkg_name, 255 | buildSuffix)) 256 | 257 | def get_package_root_dir(self): 258 | """ Return the root directory for the package """ 259 | return os.path.abspath("{}/root/{}".format( 260 | self.get_build_prefix(), 261 | self.spec.pkg_name)) 262 | 263 | def get_pgo_dir(self): 264 | """ Get the PGO data directory for the given package """ 265 | pgoSuffix = "pgo" 266 | if self.avx2: 267 | if self.emul32: 268 | pgoSuffix = "pgo-32-avx2" 269 | else: 270 | pgoSuffix = "pgo-avx2" 271 | elif self.emul32: 272 | pgoSuffix = "pgo-32" 273 | 274 | return os.path.abspath("{}/root/{}/{}".format( 275 | self.get_build_prefix(), 276 | self.spec.pkg_name, 277 | pgoSuffix)) 278 | 279 | def repl_flags_avx2(self, flags): 280 | """ Adjust flags to compensate for avx2 build """ 281 | ncflags = list() 282 | for flag in flags: 283 | if flag.startswith("-march="): 284 | flag = "-march={}".format(AVX2_ARCH) 285 | elif flag.startswith("-mtune="): 286 | flag = "-mtune={}".format(AVX2_TUNE) 287 | ncflags.append(flag) 288 | return ncflags 289 | 290 | def init_config(self): 291 | """ Initialise our configuration prior to building """ 292 | conf = pisi.config.Config() 293 | 294 | # For now follow the eopkg.conf.. 295 | self.build.host = conf.values.build.host 296 | self.build.arch = conf.values.general.architecture 297 | self.build.cflags = list(conf.values.build.cflags.split(" ")) 298 | self.build.cxxflags = list(conf.values.build.cxxflags.split(" ")) 299 | self.build.ldflags = list(conf.values.build.ldflags.split(" ")) 300 | if conf.values.build.buildhelper: 301 | self.build.ccache = "ccache" in conf.values.build.buildhelper 302 | else: 303 | self.build.ccache = None 304 | 305 | self.can_dbginfo = conf.values.build.generatedebug 306 | self.pconfig = conf 307 | 308 | self.init_compiler() 309 | 310 | # Set the $pkgfiles up properly 311 | spec_dir = os.path.dirname(os.path.abspath(self.spec.path)) 312 | self.files_dir = os.path.join(spec_dir, "files") 313 | 314 | # We'll export job count ourselves.. 315 | jobs = conf.values.build.jobs 316 | if "-j" in jobs: 317 | jobs = jobs.replace("-j", "") 318 | elif jobs == "auto": 319 | try: 320 | jobs = multiprocessing.cpu_count() 321 | except Exception as e: 322 | console_ui.emit_warning( 323 | "BUILD", "Failed to detect CPU count, defaulting to 4") 324 | try: 325 | jcount = int(jobs) 326 | self.build.jobcount = jcount 327 | except Exception as e: 328 | console_ui.emit_warning("BUILD", 329 | "Invalid job count of {}, defaulting to". 330 | format(jobs)) 331 | 332 | self.global_archive_dir = conf.values.dirs.archives_dir 333 | 334 | def init_compiler(self): 335 | if self.spec.pkg_clang: 336 | self.build.cc = "clang" 337 | self.build.cxx = "clang++" 338 | else: 339 | self.build.cc = "{}-gcc".format(self.pconfig.values.build.host) 340 | self.build.cxx = "{}-g++".format(self.pconfig.values.build.host) 341 | 342 | if self.spec.pkg_optimize: 343 | self.init_optimize() 344 | 345 | if self.emul32: 346 | self.init_emul32() 347 | 348 | if self.avx2: 349 | self.init_avx2() 350 | 351 | def init_optimize(self): 352 | """ Handle optimize settings within the spec """ 353 | for opt in self.spec.pkg_optimize: 354 | self.build.cflags = Flags.optimize_flags(self.build.cflags, 355 | opt, 356 | self.spec.pkg_clang) 357 | self.build.cxxflags = Flags.optimize_flags(self.build.cxxflags, 358 | opt, 359 | self.spec.pkg_clang) 360 | if opt == "no-bind-now" or opt == "no-symbolic": 361 | self.build.ldflags = Flags.optimize_flags(self.build.ldflags, 362 | opt, 363 | self.spec.pkg_clang) 364 | 365 | def init_emul32(self): 366 | """ Handle emul32 toolchain options """ 367 | ncflags = list() 368 | for flag in self.build.cflags: 369 | if flag.startswith("-march="): 370 | flag = "-march=i686" 371 | ncflags.append(flag) 372 | self.build.cflags = ncflags 373 | ncxxflags = list() 374 | for flag in self.build.cxxflags: 375 | if flag.startswith("-march="): 376 | flag = "-march=i686" 377 | ncxxflags.append(flag) 378 | self.build.cxxflags = ncxxflags 379 | 380 | self.build.host = "i686-pc-linux-gnu" 381 | 382 | # Get the multilib gcc stuff set up 383 | if self.spec.pkg_clang: 384 | self.build.cc = "clang -m32" 385 | self.build.cxx = "clang++ -m32" 386 | else: 387 | self.build.cc = "gcc -m32" 388 | self.build.cxx = "g++ -m32" 389 | 390 | def init_avx2(self): 391 | """ Adjust flags for AVX2 builds """ 392 | self.build.cflags = self.repl_flags_avx2(self.build.cflags) 393 | self.build.cxxflags = self.repl_flags_avx2(self.build.cxxflags) 394 | 395 | def enable_pgo_generate(self): 396 | """ Enable PGO generate step """ 397 | pgo_dir = self.get_pgo_dir() 398 | self.gen_pgo = True 399 | self.build.cflags = Flags.pgo_gen_flags(self.build.cflags, 400 | pgo_dir, 401 | self.spec.pkg_clang) 402 | self.build.cxxflags = Flags.pgo_gen_flags(self.build.cxxflags, 403 | pgo_dir, 404 | self.spec.pkg_clang) 405 | 406 | def enable_pgo_use(self): 407 | """ Enable PGO use step """ 408 | pgo_dir = self.get_pgo_dir() 409 | self.use_pgo = True 410 | self.build.cflags = Flags.pgo_use_flags(self.build.cflags, 411 | pgo_dir, 412 | self.spec.pkg_clang) 413 | self.build.cxxflags = Flags.pgo_use_flags(self.build.cxxflags, 414 | pgo_dir, 415 | self.spec.pkg_clang) 416 | 417 | def clean_pgo(self): 418 | suffixes = ["pgo", "pgo-avx2", "pgo-32", "pgo-32-avx2"] 419 | pgo_dirs = [os.path.abspath("{}/root/{}/{}".format( 420 | self.get_build_prefix(), 421 | self.spec.pkg_name, x)) for x in suffixes] 422 | 423 | for d in pgo_dirs: 424 | if not os.path.exists(d): 425 | continue 426 | try: 427 | shutil.rmtree(d) 428 | except Exception as e: 429 | console_ui.emit_error("Build", "Failed to clean PGO dir") 430 | print(e) 431 | return False 432 | return True 433 | 434 | def clean_install(self): 435 | """ Purge the install directory """ 436 | d = self.get_install_dir() 437 | if not os.path.exists(d): 438 | return True 439 | try: 440 | shutil.rmtree(d) 441 | except Exception as e: 442 | print(e) 443 | return False 444 | return True 445 | 446 | def clean_pkg(self): 447 | """ Purge the packing directory """ 448 | d = self.get_packaging_dir() 449 | if not os.path.exists(d): 450 | return True 451 | try: 452 | shutil.rmtree(d) 453 | except Exception as e: 454 | print(e) 455 | return False 456 | return True 457 | -------------------------------------------------------------------------------- /ypkg2/ypkgspec.py: -------------------------------------------------------------------------------- 1 | #!/bin/true 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of ypkg2 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software: you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation, either version 3 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | from . import console_ui 15 | from . import yamlhelper 16 | 17 | from .yamlhelper import OneOrMoreString, MultimapFormat 18 | from .sources import SourceManager, GitSource 19 | 20 | import os 21 | from collections import OrderedDict 22 | # Consider moving this into a stub 23 | import pisi.version 24 | import pisi.history 25 | import pisi.pxml.xmlfile as xmlfile 26 | import pisi.pxml.autoxml as autoxml 27 | 28 | from yaml import load as yaml_load 29 | try: 30 | from yaml import CLoader as Loader 31 | except Exception as e: 32 | console_ui.emit_warning("YAML", "Native YAML loader unavailable") 33 | from yaml import Loader 34 | 35 | 36 | class PackageSanity: 37 | 38 | @staticmethod 39 | def is_name_valid(name): 40 | """ Determine if a package name is actually valid. """ 41 | name = str(name.strip()) 42 | if len(name) < 1: 43 | console_ui.emit_error("YAML:name", 44 | "Package name cannot be empty") 45 | return False 46 | 47 | if " " in name: 48 | console_ui.emit_error("YAML:name", 49 | "Package names cannot contain whitespace") 50 | return False 51 | illegal = set() 52 | permitted = ['-', '_', '+', '.'] 53 | for char in name[0:]: 54 | if char in permitted: 55 | continue 56 | if not char.isalpha() and not char.isdigit(): 57 | illegal.add(char) 58 | if len(illegal) == 0: 59 | return True 60 | 61 | console_ui.emit_error("YAML:name", 62 | "Illegal characters in package name '{}' : {}". 63 | format(name, ", ".join(illegal))) 64 | return False 65 | 66 | @staticmethod 67 | def is_version_valid(version): 68 | """ Determine if the given version is valid """ 69 | try: 70 | v = pisi.version.make_version(version) 71 | except Exception as e: 72 | console_ui.emit_error("YAML", "Invalid version: {}".format( 73 | version)) 74 | return False 75 | return True 76 | 77 | 78 | class YpkgSpec: 79 | 80 | # Root meta information 81 | pkg_name = None 82 | pkg_version = None 83 | pkg_release = None 84 | pkg_homepage = None 85 | pkg_license = None 86 | pkg_source = None 87 | 88 | # Build control 89 | pkg_devel = False 90 | pkg_debug = True 91 | pkg_clang = False 92 | pkg_strip = True 93 | pkg_lastrip = True 94 | pkg_ccache = True 95 | pkg_emul32 = False 96 | pkg_avx2 = False 97 | pkg_autodep = True 98 | pkg_extract = True 99 | pkg_optimize = None 100 | pkg_libsplit = True 101 | 102 | # Only used by solbuild 103 | pkg_networking = False 104 | 105 | # Dependencies 106 | pkg_builddeps = None 107 | 108 | mandatory_tokens = None 109 | optional_tokens = None 110 | build_steps = None 111 | 112 | # Build steps 113 | step_setup = None 114 | step_build = None 115 | step_install = None 116 | step_check = None 117 | step_profile = None 118 | 119 | # More meta 120 | summaries = None 121 | descriptions = None 122 | rundeps = None 123 | components = None 124 | 125 | # Multimap: replaces/conflicts 126 | replaces = None 127 | conflicts = None 128 | 129 | # Path to filename 130 | path = None 131 | 132 | # Permanent paths 133 | pkg_permanent = None 134 | 135 | # Custom user provided patterns 136 | patterns = None 137 | 138 | history = None 139 | 140 | def add_summary(self, key, value): 141 | """ Add a summary to a package """ 142 | self.summaries[key] = value 143 | 144 | def add_desc(self, key, value): 145 | """ Add a description to a package """ 146 | self.descriptions[key] = value 147 | 148 | def add_rundep(self, key, val): 149 | if key not in self.rundeps: 150 | self.rundeps[key] = list() 151 | if val in self.rundeps[key]: 152 | console_ui.emit_warning("YAML", "Duplicate rundep: {}".format(val)) 153 | return 154 | self.rundeps[key].append(val) 155 | 156 | def add_component(self, key, value): 157 | """ Set the component for a package """ 158 | self.components[key] = value 159 | 160 | def add_pattern(self, key, pt): 161 | if key not in self.patterns: 162 | self.patterns[key] = list() 163 | if pt in self.patterns[key]: 164 | console_ui.emit_warning("YAML", "Duplicate pattern: {}".format(pt)) 165 | self.patterns[key].append(pt) 166 | 167 | def add_replace(self, key, val): 168 | """ Add a 'replaces:' to the package """ 169 | if key not in self.replaces: 170 | self.replaces[key] = list() 171 | # i.e. llvm-clang-devel replaces clang-devel 172 | if val in self.replaces and val != key: 173 | console_ui.emit_warning("YAML", 174 | "Duplicate replace: {}".format(val)) 175 | return 176 | self.replaces[key].append(val) 177 | 178 | def add_conflict(self, key, val): 179 | """ Add a 'conflicts:' to the package """ 180 | if key not in self.conflicts: 181 | self.conflicts[key] = list() 182 | if val in self.conflicts: 183 | console_ui.emit_warning("YAML", 184 | "Duplicate conflict: {}".format(val)) 185 | return 186 | self.conflicts[key].append(val) 187 | 188 | def __init__(self): 189 | # These tokens *must* exist 190 | self.mandatory_tokens = OrderedDict([ 191 | ("name", str), 192 | ("version", str), 193 | ("release", int), 194 | ("license", OneOrMoreString), 195 | ("summary", MultimapFormat(self, self.add_summary, "main")), 196 | ("description", MultimapFormat(self, self.add_desc, "main")), 197 | ("source", list), # We verify sources later 198 | ]) 199 | # These guys are optional 200 | self.optional_tokens = OrderedDict([ 201 | ("homepage", str), 202 | ("devel", bool), 203 | ("clang", bool), 204 | ("debug", bool), 205 | ("strip", bool), 206 | ("lastrip", bool), 207 | ("ccache", bool), 208 | ("emul32", bool), 209 | ("networking", bool), 210 | ("avx2", bool), 211 | ("autodep", bool), 212 | ("extract", bool), 213 | ("libsplit", bool), 214 | ("patterns", MultimapFormat(self, self.add_pattern, "main")), 215 | ("permanent", OneOrMoreString), 216 | ("builddeps", OneOrMoreString), 217 | ("rundeps", MultimapFormat(self, self.add_rundep, "main")), 218 | ("component", MultimapFormat(self, self.add_component, "main")), 219 | ("conflicts", MultimapFormat(self, self.add_conflict, "main")), 220 | ("replaces", MultimapFormat(self, self.add_replace, "main")), 221 | ("optimize", OneOrMoreString), 222 | ]) 223 | # Build steps are handled separately 224 | self.build_steps = OrderedDict([ 225 | ("setup", unicode), 226 | ("build", unicode), 227 | ("install", unicode), 228 | ("check", unicode), 229 | ("profile", unicode), 230 | ]) 231 | self.summaries = dict() 232 | self.descriptions = dict() 233 | self.rundeps = dict() 234 | self.components = dict() 235 | self.patterns = OrderedDict() 236 | self.replaces = dict() 237 | self.conflicts = dict() 238 | 239 | def init_defaults(self): 240 | # Add some sane defaults 241 | name = self.pkg_name 242 | if "devel" not in self.summaries: 243 | self.add_summary("devel", "Development files for {}".format(name)) 244 | if "32bit" not in self.summaries: 245 | self.add_summary("32bit", "32-bit libraries for {}".format(name)) 246 | if "32bit-devel" not in self.summaries: 247 | self.add_summary("32bit-devel", "Development files for 32-bit {}". 248 | format(name)) 249 | 250 | if "devel" not in self.components: 251 | if self.pkg_devel: 252 | self.add_component("devel", "system.devel") 253 | else: 254 | self.add_component("devel", "programming.devel") 255 | if "32bit" not in self.components: 256 | self.add_component("32bit", "emul32") 257 | if "32bit-devel" not in self.components: 258 | self.add_component("32bit-devel", "programming.devel") 259 | if "docs" not in self.components: 260 | self.add_component("docs", "programming.docs") 261 | 262 | # debuginfos 263 | if "dbginfo" not in self.components: 264 | self.add_component("dbginfo", "debug") 265 | if "dbginfo" not in self.summaries: 266 | self.add_summary("dbginfo", "Debug symbols for {}".format(name)) 267 | if "32bit-dbginfo" not in self.components: 268 | self.add_component("32bit-dbginfo", "debug") 269 | if "32bit-dbginfo" not in self.summaries: 270 | self.add_summary("32bit-dbginfo", "32-bit debug symbols for {}" 271 | .format(name)) 272 | if "docs" not in self.summaries: 273 | self.add_summary("docs", "Documentation for {}".format(name)) 274 | 275 | extras = [] 276 | if self.pkg_emul32: 277 | extras.extend(["glibc-32bit-devel", "libgcc-32bit", 278 | "libstdc++-32bit", "fakeroot-32bit"]) 279 | if self.pkg_clang: 280 | extras.extend(["llvm-clang-32bit", "llvm-clang-32bit-devel"]) 281 | if self.pkg_clang: 282 | extras.extend(["llvm-clang", "llvm-clang-devel"]) 283 | if self.pkg_optimize: 284 | if "thin-lto" in self.pkg_optimize: 285 | extras.extend(["binutils-gold"]) 286 | 287 | man = SourceManager() 288 | try: 289 | man.identify_sources(self) 290 | for x in man.sources: 291 | if isinstance(x, GitSource): 292 | extras.append("git") 293 | break 294 | except: 295 | pass 296 | for x in extras: 297 | if not self.pkg_builddeps: 298 | self.pkg_builddeps = list() 299 | if x not in self.pkg_builddeps: 300 | self.pkg_builddeps.append(x) 301 | 302 | def load_from_path(self, path): 303 | self.path = path 304 | if not os.path.exists(path): 305 | console_ui.emit_error("Error", "Path does not exist") 306 | return False 307 | if not os.path.isfile and not os.path.islink(path): 308 | console_ui.emit_error("Error", "File specified is not a file") 309 | return False 310 | 311 | filename = os.path.basename(path) 312 | 313 | # We'll get rid of this at some point :P 314 | if filename != "package.yml": 315 | console_ui.emit_warning("Unnecessarily Anal Warning", 316 | "File is not named package.yml") 317 | 318 | # Attempt to parse the input file 319 | with open(path, "r") as inpfile: 320 | try: 321 | yaml_data = yaml_load(inpfile, Loader=Loader) 322 | except Exception as e: 323 | console_ui.emit_error("YAML", "Failed to parse YAML file") 324 | print(e) 325 | return False 326 | 327 | b = self.load_from_data(yaml_data) 328 | if not b: 329 | return b 330 | return self.load_component() 331 | 332 | def load_from_data(self, yaml_data): 333 | # Grab the main root elements (k->v mapping) 334 | sets = [self.mandatory_tokens, self.optional_tokens, self.build_steps] 335 | for tk_set in sets: 336 | for token in tk_set.keys(): 337 | t = tk_set[token] 338 | 339 | if token not in yaml_data and tk_set != self.mandatory_tokens: 340 | # ok to skip optionals 341 | continue 342 | 343 | if isinstance(t, MultimapFormat): 344 | if not yamlhelper.assertMultimap(yaml_data, token, t): 345 | return False 346 | continue 347 | else: 348 | val = yamlhelper.assertGetType(yaml_data, token, t) 349 | if val is None: 350 | return False 351 | # Handle build steps differently to avoid collisions 352 | if tk_set == self.build_steps: 353 | instance_name = "step_{}".format(token) 354 | else: 355 | instance_name = "pkg_{}".format(token) 356 | if not hasattr(self, instance_name): 357 | console_ui.emit_error("YAML:{}".format(token), 358 | "Internal error for unknown token") 359 | return False 360 | setattr(self, instance_name, val) 361 | 362 | if "main" not in self.summaries: 363 | console_ui.emit_info("YAML", "Missing summary for package") 364 | return False 365 | if "main" not in self.descriptions: 366 | console_ui.emit_info("YAML", "Missing description for package") 367 | return False 368 | 369 | # Ensure this package would actually be able to build.. 370 | steps = [self.step_setup, self.step_build, self.step_install] 371 | steps = filter(lambda s: s, steps) 372 | if len(steps) == 0: 373 | console_ui.emit_error("YAML", "No functional build steps found") 374 | return False 375 | 376 | # Validate the names and version 377 | if not PackageSanity.is_version_valid(self.pkg_version): 378 | return False 379 | if not PackageSanity.is_name_valid(self.pkg_name): 380 | return False 381 | 382 | for name in self.patterns: 383 | fname = self.get_package_name(name) 384 | if not PackageSanity.is_name_valid(fname): 385 | return False 386 | 387 | self.init_defaults() 388 | 389 | return True 390 | 391 | def load_history(self, path): 392 | try: 393 | self.history = PackageHistory() 394 | self.history.read(path) 395 | up = self.history.history[0] 396 | release = int(up.release) 397 | version = up.version 398 | 399 | if release != self.pkg_release: 400 | console_ui.emit_warning("HISTORY", "Release not consistent") 401 | if version != self.pkg_version: 402 | console_ui.emit_warning("HISTORY", "Version not consistent..") 403 | except Exception as e: 404 | console_ui.emit_error("HISTORY", "Could not load history file") 405 | print(e) 406 | return False 407 | return True 408 | 409 | def load_component(self): 410 | if "main" in self.components: 411 | return True 412 | 413 | p = os.path.dirname(self.path) 414 | 415 | for i in ["component.xml", "../component.xml"]: 416 | fpath = os.path.join(p, i) 417 | if not os.path.exists(fpath): 418 | continue 419 | comp = pisi.component.CompatComponent() 420 | try: 421 | comp.read(fpath) 422 | console_ui.emit_error("Component", 423 | "Using legacy component.xml: {}". 424 | format(fpath)) 425 | print(" Please switch to using the 'component' key") 426 | return False 427 | except Exception as e: 428 | console_ui.emit_error("Component", "Error in legacy file") 429 | print(e) 430 | return False 431 | return True 432 | 433 | def get_package_name(self, name): 434 | if name == "main": 435 | return self.pkg_name 436 | if name.startswith('^'): 437 | return str(name[1:]) 438 | return "{}-{}".format(self.pkg_name, name) 439 | 440 | def get_component(self, name): 441 | if name in self.components: 442 | return self.components[name] 443 | return None 444 | 445 | def get_description(self, name): 446 | if name not in self.descriptions: 447 | return self.descriptions["main"] 448 | return self.descriptions[name] 449 | 450 | def get_summary(self, name): 451 | if name not in self.summaries: 452 | return self.summaries["main"] 453 | return self.summaries[name] 454 | 455 | 456 | class PackageHistory(xmlfile.XmlFile): 457 | 458 | __metaclass__ = autoxml.autoxml 459 | 460 | tag = "YPKG" 461 | 462 | t_History = [[pisi.specfile.Update], autoxml.mandatory] 463 | -------------------------------------------------------------------------------- /yupdate: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # -*- coding: utf-8 -*- 3 | # 4 | # yupdate.py 5 | # 6 | # Copyright 2015-2017 Ikey Doherty 7 | # 8 | # This program is free software; you can redistribute it and/or modify 9 | # it under the terms of the GNU General Public License as published by 10 | # the Free Software Foundation; either version 2 of the License, or 11 | # (at your option) any later version. 12 | # 13 | 14 | import sys 15 | import ruamel.yaml 16 | import os 17 | import subprocess 18 | import pisi.version 19 | 20 | 21 | def usage(msg=None, ex=1): 22 | if msg: 23 | print(msg) 24 | else: 25 | print("Usage: %s [version] [url]" % sys.argv[0]) 26 | sys.exit(ex) 27 | 28 | if __name__ == "__main__": 29 | if len(sys.argv) != 3: 30 | usage() 31 | 32 | ymlfile = "package.yml" 33 | if not os.path.exists(ymlfile): 34 | usage("Specified file does not exist") 35 | if not ymlfile.endswith(".yml"): 36 | usage("%s does not look like a valid package.yml file") 37 | 38 | newversion = sys.argv[1] 39 | try: 40 | d = pisi.version.Version(newversion) 41 | except Exception as e: 42 | print("Problematic version string: %s" % e) 43 | sys.exit(1) 44 | 45 | url = sys.argv[2] 46 | file = url.split("/")[-1] 47 | 48 | try: 49 | r = os.system("wget \"%s\"" % url) 50 | except: 51 | print("Failed to download file") 52 | sys.exit(1) 53 | if r != 0: 54 | print("Failed to download file") 55 | sys.exit(1) 56 | 57 | sha256 = subprocess.check_output(["sha256sum", file]).split()[0].strip() 58 | 59 | with open(ymlfile, "r") as infile: 60 | data = ruamel.yaml.round_trip_load(infile) 61 | data['source'] = sources = [] 62 | sources.append({url: sha256}) 63 | data['release'] += 1 64 | data['version'] = newversion 65 | 66 | os.unlink(file) 67 | 68 | try: 69 | with open(ymlfile, 'w') as fp: 70 | ruamel.yaml.round_trip_dump( 71 | data, fp, indent=4, block_seq_indent=4, width=200, 72 | top_level_colon_align=True, prefix_colon=' ') 73 | except Exception as e: 74 | print("Error writing file, may need to reset it.") 75 | print(e) 76 | sys.exit(1) 77 | --------------------------------------------------------------------------------