├── .gitignore ├── COPYING ├── MANIFEST.in ├── README.md ├── pipoe ├── __init__.py ├── licenses.py └── pipoe.py ├── requirements.txt └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | env 3 | build 4 | dist 5 | *.egg-info 6 | *~ 7 | *.pyc 8 | python-* 9 | python3-* 10 | .vscode/* 11 | -------------------------------------------------------------------------------- /COPYING: -------------------------------------------------------------------------------- 1 | Copyright 2019 Neil F Jones 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include requirements.txt 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # pipoe 2 | 3 | The objective of this project is to make creating OpenEmbedded python recipes just a bit easier. `pipoe` will take either a single package name or a requirements file and recursively generate bitbake recipes for every pypi package listed. It is not guaranteed that it will work for every package. Additionally, many recipes will still require additional modification after generation (patches, overrides, appends, etc.). In those cases it is recommended that the user add these modifications in a bbappend file. 4 | 5 | ## Install 6 | ``` 7 | > pip3 install pipoe 8 | ``` 9 | 10 | ## Licenses 11 | 12 | Licensing within OE is typically pretty strict. `pipoe` contains a license map which will attempt to map a packages license to one that will be accepted by the OE framework. If a license string is found which cannot be mapped, the user will be prompted to enter a valid license name. This mapping will be saved and the updated map will be saved to `./licenses.py` if the `--licenses` flag is provided. It is recommended that this file be PR'ed to this repository when generally useful changes are made. 13 | 14 | ## Extras 15 | `pipoe` supports generating "extra" recipes based on the extra feature declarations in the packages `requires_dist` field (i.e. urllib3\[secure\]). These recipes are generated as packagegroups which rdepend on the base package. 16 | 17 | 18 | ## Versions 19 | By default `pipoe` will generate a recipe for the newest version of a package. Supplying the `--version` argument will override this behavior. Additionally, `pipoe` will automatically parse versions from requirements files. 20 | 21 | ## Example 22 | 23 | ``` 24 | > pipoe --help 25 | usage: pipoe [-h] [--package PACKAGE] [--version VERSION] 26 | [--requirements REQUIREMENTS] [--extras] [--outdir OUTDIR] 27 | [--python {python,python3}] [--licenses] 28 | [--default-license DEFAULT_LICENSE] 29 | 30 | optional arguments: 31 | -h, --help show this help message and exit 32 | --package PACKAGE, -p PACKAGE 33 | The package to process. 34 | --version VERSION, -v VERSION 35 | The package version. 36 | --requirements REQUIREMENTS, -r REQUIREMENTS 37 | The pypi requirements file. 38 | --extras, -e Generate recipes for extras. 39 | --outdir OUTDIR, -o OUTDIR 40 | The recipe directory. 41 | --python {python,python3}, -y {python,python3} 42 | The python version to use. 43 | --licenses, -l Output an updated license map upon completion. 44 | --default-license DEFAULT_LICENSE, -d DEFAULT_LICENSE 45 | The default license to use when the package license 46 | cannot be mapped. 47 | --pypi, -s Use oe pypi class for recipe 48 | > pipoe -p requests 49 | Gathering info: 50 | requests 51 | | chardet 52 | | idna 53 | | urllib3 54 | | certifi 55 | Generating recipes: 56 | python-requests_2.21.0.bb 57 | python-chardet_3.0.4.bb 58 | python-idna_2.8.bb 59 | python-urllib3_1.24.1.bb 60 | python-certifi_2018.11.29.bb 61 | 62 | License mappings are available in: ./licenses.py 63 | PREFERRED_VERSIONS are available in: ./python-versions.inc 64 | ``` 65 | -------------------------------------------------------------------------------- /pipoe/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NFJones/pipoe/fbb0fa60034c00158c5a163234557b789de33dc7/pipoe/__init__.py -------------------------------------------------------------------------------- /pipoe/licenses.py: -------------------------------------------------------------------------------- 1 | LICENSES = { 2 | "AAL": "AAL", 3 | "AFL-1.2": "AFL-1.2", 4 | "AFL-2.0": "AFL-2.0", 5 | "AFL-2.1": "AFL-2.1", 6 | "AFL-3.0": "AFL-3.0", 7 | "AGPL-3.0": "AGPL-3.0", 8 | "ANTLR-PD": "ANTLR-PD", 9 | "APL-1.0": "APL-1.0", 10 | "APL2": "Apache-2.0", 11 | "APSL-1.0": "APSL-1.0", 12 | "APSL-1.1": "APSL-1.1", 13 | "APSL-1.2": "APSL-1.2", 14 | "APSL-2.0": "APSL-2.0", 15 | "ASL": "Apache-2.0", 16 | "Adobe": "Adobe", 17 | "Apache": "Apache-2.0", 18 | "Apache 2.0": "Apache-2.0", 19 | "Apache License 2.0": "Apache-2.0", 20 | "Apache License, Version 2.0": "Apache-2.0", 21 | "Apache Software License": "Apache-2.0", 22 | "Apache-1.0": "Apache-1.0", 23 | "Apache-1.1": "Apache-1.1", 24 | "Apache-2.0": "Apache-2.0", 25 | "Artistic-1.0": "Artistic-1.0", 26 | "Artistic-2.0": "Artistic-2.0", 27 | "BSD": "BSD", 28 | "BSD - See ndg/httpsclient/LICENCE file for details": "BSD", 29 | "BSD 3-Clause": "BSD-3-Clause", 30 | "BSD 3-Clause License": "BSD-3-Clause", 31 | "BSD License": "BSD", 32 | "BSD or Apache License, Version 2.0": "BSD", 33 | "BSD, Public Domain": "BSD", 34 | "BSD-2-Clause": "BSD-2-Clause", 35 | "BSD-3-Clause": "BSD-3-Clause", 36 | "BSD-4-Clause": "BSD-4-Clause", 37 | "BSD-derived (http://www.repoze.org/LICENSE.txt)": "BSD", 38 | "BSD-like": "BSD", 39 | "BSL-1.0": "BSL-1.0", 40 | "BitstreamVera": "BitstreamVera", 41 | "CATOSL-1.1": "CATOSL-1.1", 42 | "CC-BY-1.0": "CC-BY-1.0", 43 | "CC-BY-2.0": "CC-BY-2.0", 44 | "CC-BY-2.5": "CC-BY-2.5", 45 | "CC-BY-3.0": "CC-BY-3.0", 46 | "CC-BY-NC-1.0": "CC-BY-NC-1.0", 47 | "CC-BY-NC-2.0": "CC-BY-NC-2.0", 48 | "CC-BY-NC-2.5": "CC-BY-NC-2.5", 49 | "CC-BY-NC-3.0": "CC-BY-NC-3.0", 50 | "CC-BY-NC-ND-1.0": "CC-BY-NC-ND-1.0", 51 | "CC-BY-NC-ND-2.0": "CC-BY-NC-ND-2.0", 52 | "CC-BY-NC-ND-2.5": "CC-BY-NC-ND-2.5", 53 | "CC-BY-NC-ND-3.0": "CC-BY-NC-ND-3.0", 54 | "CC-BY-NC-SA-1.0": "CC-BY-NC-SA-1.0", 55 | "CC-BY-NC-SA-2.0": "CC-BY-NC-SA-2.0", 56 | "CC-BY-NC-SA-2.5": "CC-BY-NC-SA-2.5", 57 | "CC-BY-NC-SA-3.0": "CC-BY-NC-SA-3.0", 58 | "CC-BY-ND-1.0": "CC-BY-ND-1.0", 59 | "CC-BY-ND-2.0": "CC-BY-ND-2.0", 60 | "CC-BY-ND-2.5": "CC-BY-ND-2.5", 61 | "CC-BY-ND-3.0": "CC-BY-ND-3.0", 62 | "CC-BY-SA-1.0": "CC-BY-SA-1.0", 63 | "CC-BY-SA-2.0": "CC-BY-SA-2.0", 64 | "CC-BY-SA-2.5": "CC-BY-SA-2.5", 65 | "CC-BY-SA-3.0": "CC-BY-SA-3.0", 66 | "CC0-1.0": "CC0-1.0", 67 | "CDDL-1.0": "CDDL-1.0", 68 | "CECILL-1.0": "CECILL-1.0", 69 | "CECILL-2.0": "CECILL-2.0", 70 | "CECILL-B": "CECILL-B", 71 | "CECILL-C": "CECILL-C", 72 | "CPAL-1.0": "CPAL-1.0", 73 | "CPL-1.0": "CPL-1.0", 74 | "CUA-OPL-1.0": "CUA-OPL-1.0", 75 | "ClArtistic": "ClArtistic", 76 | "DSSSL": "DSSSL", 77 | "Dual License": "BSD", 78 | "ECL-1.0": "ECL-1.0", 79 | "ECL-2.0": "ECL-2.0", 80 | "EDL-1.0": "EDL-1.0", 81 | "EFL-1.0": "EFL-1.0", 82 | "EFL-2.0": "EFL-2.0", 83 | "EPL-1.0": "EPL-1.0", 84 | "EPL-2.0": "EPL-2.0", 85 | "EUDatagrid": "EUDatagrid", 86 | "EUPL-1.0": "EUPL-1.0", 87 | "EUPL-1.1": "EUPL-1.1", 88 | "Elfutils-Exception": "Elfutils-Exception", 89 | "Entessa": "Entessa", 90 | "ErlPL-1.1": "ErlPL-1.1", 91 | "Expat license": "MIT", 92 | "Expat (MIT/X11)": "MIT", 93 | "Fair": "Fair", 94 | "Frameworx-1.0": "Frameworx-1.0", 95 | "FreeType": "FreeType", 96 | "GFDL-1.1": "GFDL-1.1", 97 | "GFDL-1.2": "GFDL-1.2", 98 | "GFDL-1.3": "GFDL-1.3", 99 | "GNU GPLv3+": "GPL-3.0", 100 | "GNU General Public License Version 3": "GPL-3.0", 101 | "GNU LGPL": "LGPL-2.0", 102 | "GPL": "GPL-1.0", 103 | "GPL V2 or later": "GPL-2.0", 104 | "GPL-1.0": "GPL-1.0", 105 | "GPL-2-with-bison-exception": "GPL-2-with-bison-exception", 106 | "GPL-2.0": "GPL-2.0", 107 | "GPL-2.0-with-GCC-exception": "GPL-2.0-with-GCC-exception", 108 | "GPL-2.0-with-autoconf-exception": "GPL-2.0-with-autoconf-exception", 109 | "GPL-2.0-with-classpath-exception": "GPL-2.0-with-classpath-exception", 110 | "GPL-2.0-with-font-exception": "GPL-2.0-with-font-exception", 111 | "GPL-3.0": "GPL-3.0", 112 | "GPL-3.0-with-GCC-exception": "GPL-3.0-with-GCC-exception", 113 | "GPL-3.0-with-autoconf-exception": "GPL-3.0-with-autoconf-exception", 114 | "GPLv3+": "GPLv3", 115 | "HPND": "HPND", 116 | "IPA": "IPA", 117 | "IPL-1.0": "IPL-1.0", 118 | "ISC": "ISC", 119 | "ISC license": "ISC", 120 | "LGPL": "LGPL-2.0", 121 | "LGPL-2.0": "LGPL-2.0", 122 | "LGPL-2.1": "LGPL-2.1", 123 | "LGPL-3.0": "LGPL-3.0", 124 | "LGPLv2": "LGPL-2.0", 125 | "LGPLv2+": "LGPL-2.0", 126 | "LGPLv3": "LGPL-3.0", 127 | "LPGL, see LICENSE file.": "LGPL-1.0", 128 | "LPL-1.02": "LPL-1.02", 129 | "LPPL-1.0": "LPPL-1.0", 130 | "LPPL-1.1": "LPPL-1.1", 131 | "LPPL-1.2": "LPPL-1.2", 132 | "LPPL-1.3c": "LPPL-1.3c", 133 | "Libpng": "Libpng", 134 | "License :: OSI Approved :: Apache Software License": "Apache-2.0", 135 | "License :: OSI Approved :: MIT License (http://opensource.org/licenses/MIT)": "MIT", 136 | "MIT": "MIT", 137 | "MIT License": "MIT", 138 | "MIT license": "MIT", 139 | "MIT/X11": "MIT", 140 | "MPL v2": "MPL-2.0", 141 | "MPL-1.0": "MPL-1.0", 142 | "MPL-1.1": "MPL-1.1", 143 | "MPL-2.0": "MPL-2.0", 144 | "MPLv2.0, MIT Licences": "MIT", 145 | "MS-PL": "MS-PL", 146 | "MS-RL": "MS-RL", 147 | "MirOS": "MirOS", 148 | "Modified BSD License": "BSD", 149 | "Motosoto": "Motosoto", 150 | "Multics": "Multics", 151 | "NASA-1.3": "NASA-1.3", 152 | "NCSA": "NCSA", 153 | "NGPL": "NGPL", 154 | "NPOSL-3.0": "NPOSL-3.0", 155 | "NTP": "NTP", 156 | "Nauman": "Nauman", 157 | "New BSD": "BSD", 158 | "Nokia": "Nokia", 159 | "OASIS": "OASIS", 160 | "OCLC-2.0": "OCLC-2.0", 161 | "ODbL-1.0": "ODbL-1.0", 162 | "OFL-1.1": "OFL-1.1", 163 | "OGTSL": "OGTSL", 164 | "OLDAP-2.8": "OLDAP-2.8", 165 | "OSL-1.0": "OSL-1.0", 166 | "OSL-2.0": "OSL-2.0", 167 | "OSL-3.0": "OSL-3.0", 168 | "OpenSSL": "OpenSSL", 169 | "PD": "PD", 170 | "PHP-3.0": "PHP-3.0", 171 | "PSF": "Python-2.0", 172 | "PSF License": "Python-2.0", 173 | "PSF license": "Python-2.0", 174 | "PSFL": "Python-2.0", 175 | "PostgreSQL": "PostgreSQL", 176 | "Proprietary": "Proprietary", 177 | "Public Domain": "PD", 178 | "Public domain": "PD", 179 | "Python Software Foundation License": "Python-2.0", 180 | "Python style": "Python-2.0", 181 | "Python-2.0": "Python-2.0", 182 | "QPL-1.0": "QPL-1.0", 183 | "RHeCos-1": "RHeCos-1", 184 | "RHeCos-1.1": "RHeCos-1.1", 185 | "RPL-1.5": "RPL-1.5", 186 | "RPSL-1.0": "RPSL-1.0", 187 | "RSCPL": "RSCPL", 188 | "Ruby": "Ruby", 189 | "SAX-PD": "SAX-PD", 190 | "SGI-1": "SGI-1", 191 | "SPL-1.0": "SPL-1.0", 192 | "Simple-2.0": "Simple-2.0", 193 | "Sleepycat": "Sleepycat", 194 | "SugarCRM-1": "SugarCRM-1", 195 | "SugarCRM-1.1.3": "SugarCRM-1.1.3", 196 | "This software released into the public domain. Anyone is free to copy,": "PD", 197 | "Two-clause BSD license": "BSD-2-Clause", 198 | "UCB": "UCB", 199 | "UNKNOWN": "CLOSED", 200 | "VSL-1.0": "VSL-1.0", 201 | "W3C": "W3C", 202 | "WXwindows": "WXwindows", 203 | "Watcom-1.0": "Watcom-1.0", 204 | "XFree86-1.0": "XFree86-1.0", 205 | "XFree86-1.1": "XFree86-1.1", 206 | "XSL": "XSL", 207 | "Xnet": "Xnet", 208 | "YPL-1.1": "YPL-1.1", 209 | "ZPL 2.1": "ZPL-2.0", 210 | "ZPL-1.1": "ZPL-1.1", 211 | "ZPL-2.0": "ZPL-2.0", 212 | "ZPL-2.1": "ZPL-2.1", 213 | "Zimbra-1.3": "Zimbra-1.3", 214 | "Zlib": "Zlib", 215 | "eCos-2.0": "eCos-2.0", 216 | "gSOAP-1": "gSOAP-1", 217 | "gSOAP-1.3b": "gSOAP-1.3b", 218 | "http://creativecommons.org/publicdomain/zero/1.0/": "PD", 219 | "http://opensource.org/licenses/MIT": "MIT", 220 | "public domain, Python, 2-Clause BSD, GPL 3 (see COPYING.txt)": "PD", 221 | } 222 | 223 | -------------------------------------------------------------------------------- /pipoe/pipoe.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import os 5 | import os.path 6 | import re 7 | import sys 8 | import urllib.request 9 | import hashlib 10 | import shutil 11 | import json 12 | import tarfile 13 | import tempfile 14 | import zipfile 15 | import mmap 16 | from pep508_parser import parser 17 | from pipoe import licenses 18 | from functools import partial 19 | from collections import namedtuple 20 | from pprint import pformat 21 | 22 | import pkginfo 23 | 24 | BB_TEMPLATE = """ 25 | SUMMARY = "{summary}" 26 | HOMEPAGE = "{homepage}" 27 | AUTHOR = "{author} <{author_email}>" 28 | LICENSE = "{license}" 29 | LIC_FILES_CHKSUM = "file://{license_file};md5={license_md5}" 30 | 31 | inherit setuptools{setuptools} 32 | 33 | SRC_URI = "{src_uri}" 34 | SRC_URI[md5sum] = "{md5}" 35 | SRC_URI[sha256sum] = "{sha256}" 36 | 37 | S = "${{WORKDIR}}/{src_dir}" 38 | 39 | DEPENDS += " {build_dependencies}" 40 | RDEPENDS_${{PN}} = "{dependencies}" 41 | 42 | BBCLASSEXTEND = "native nativesdk" 43 | """ 44 | 45 | BB_TEMPLATE_PYPI = """ 46 | SUMMARY = "{summary}" 47 | HOMEPAGE = "{homepage}" 48 | AUTHOR = "{author} <{author_email}>" 49 | LICENSE = "{license}" 50 | LIC_FILES_CHKSUM = "file://{license_file};md5={license_md5}" 51 | 52 | inherit setuptools{setuptools} pypi 53 | 54 | SRC_URI[md5sum] = "{md5}" 55 | SRC_URI[sha256sum] = "{sha256}" 56 | 57 | PYPI_PACKAGE = "{pypi_package}"{pypi_package_ext} 58 | 59 | DEPENDS += " {build_dependencies}" 60 | RDEPENDS_${{PN}} = "{dependencies}" 61 | 62 | BBCLASSEXTEND = "native nativesdk" 63 | """ 64 | 65 | 66 | BB_EXTRA_TEMPLATE = """ 67 | SUMMARY = "{summary}" 68 | HOMEPAGE = "{homepage}" 69 | AUTHOR = "{author} <{author_email}>" 70 | 71 | RDEPENDS_${{PN}} = "{dependencies}" 72 | 73 | inherit packagegroup 74 | 75 | BBCLASSEXTEND = "native nativesdk" 76 | """ 77 | 78 | 79 | Package = namedtuple( 80 | "Package", 81 | [ 82 | "name", 83 | "version", 84 | "summary", 85 | "homepage", 86 | "author", 87 | "author_email", 88 | "license", 89 | "license_file", 90 | "license_md5", 91 | "src_dir", 92 | "src_uri", 93 | "src_md5", 94 | "src_sha256", 95 | "dependencies", 96 | "build_dependencies", 97 | ], 98 | ) 99 | 100 | Dependency = namedtuple("Dependency", ["name", "version", "extra"]) 101 | 102 | 103 | def md5sum(path): 104 | with open(path, mode="rb") as f: 105 | d = hashlib.md5() 106 | for buf in iter(partial(f.read, 128), b""): 107 | d.update(buf) 108 | return d.hexdigest() 109 | 110 | 111 | def sha256sum(path): 112 | with open(path, mode="rb") as f: 113 | d = hashlib.sha256() 114 | for buf in iter(partial(f.read, 128), b""): 115 | d.update(buf) 116 | return d.hexdigest() 117 | 118 | 119 | def package_to_bb_name(package): 120 | return package.lower().replace("_", "-").replace(".", "-") 121 | 122 | 123 | def translate_license(license, default_license): 124 | try: 125 | return licenses.LICENSES[license] 126 | except: 127 | if default_license: 128 | return default_license 129 | 130 | print("Failed to translate license: {}".format(license)) 131 | mapping = input("Please enter a valid license name: ") 132 | licenses.LICENSES[license] = mapping 133 | return mapping 134 | 135 | 136 | def unpack_package(file): 137 | tmpdir = "{}.d".format(file) 138 | 139 | if os.path.exists(tmpdir): 140 | shutil.rmtree(tmpdir) 141 | 142 | os.mkdir(tmpdir) 143 | shutil.unpack_archive(file, extract_dir=tmpdir) 144 | 145 | return tmpdir 146 | 147 | 148 | def get_file_extension(uri): 149 | extensions = ["tar", "tar.gz", "tar.bz2", "tar.xz", "zip"] 150 | for extension in extensions: 151 | if uri.endswith(extension): 152 | return extension 153 | raise Exception("Extension not supported: {}".format(uri)) 154 | 155 | def package_to_bb_build_depends(package_name): 156 | name = package_name.split('<')[0].split('>')[0].split('~')[0].split('=')[0].strip() 157 | return "${PYTHON_PN}-" + package_to_bb_name(name) + "-native" 158 | 159 | def gather_package_build_depends(name, data): 160 | build_deps = [] 161 | 162 | if re.match(b"^(\s*)$", name): 163 | return build_deps 164 | 165 | # Check if it's a variable 166 | match = re.search(name + b" = (.*)", data) 167 | if match: 168 | # This is a variable check his contents 169 | for bd in match.group(1).replace(b'[', b'').replace(b']', b'').split(b","): 170 | match = re.match('^\w\S+', bd.decode("utf-8").replace("'","").replace("\"", "").strip()) 171 | if match: 172 | build_deps.append(package_to_bb_build_depends(match.group(0))) 173 | else: 174 | # This is a regular field 175 | build_deps.append(package_to_bb_build_depends(name.decode("utf-8").replace("'","").replace("\"", ""))) 176 | 177 | 178 | return build_deps 179 | 180 | 181 | def get_package_file_info(package, version, uri): 182 | extension = get_file_extension(uri) 183 | with tempfile.TemporaryDirectory() as tmp: 184 | build_deps = [] 185 | output = os.path.join(str(tmp), "{}_{}.{}".format(package, version, extension)) 186 | 187 | if os.path.exists(output): 188 | os.remove(output) 189 | 190 | urllib.request.urlretrieve(uri, output) 191 | 192 | tmpdir = unpack_package(output) 193 | src_dir = os.listdir(tmpdir)[0] 194 | 195 | src_files = os.listdir("{}/{}".format(tmpdir, src_dir)) 196 | 197 | try: 198 | license_file = next( 199 | f 200 | for f in src_files 201 | if ("license" in f.lower() or "copying" in f.lower()) 202 | and not os.path.isdir(os.path.join(tmpdir, src_dir, f)) 203 | ) 204 | except: 205 | license_file = "setup.py" 206 | 207 | # Try to catch build depends into setup.py file 208 | with open(os.path.join(tmpdir, src_dir, "setup.py"), 'r+') as f: 209 | data = mmap.mmap(f.fileno(), 0) 210 | match = re.search(b'^(\s*)setup_requires( *)=( *)([\[|\(]*)(.*)([\]|\)]*)', data, re.MULTILINE) 211 | if match: 212 | for bd in match.group(5).replace(b'[', b'').replace(b']', b'').replace(b'(', b'').replace(b')', b'').strip().split(b","): 213 | build_deps.extend(gather_package_build_depends(bd, data)) 214 | 215 | 216 | license_path = os.path.join(tmpdir, src_dir, license_file) 217 | license_md5 = md5sum(license_path) 218 | src_md5 = md5sum(output) 219 | src_sha256 = sha256sum(output) 220 | 221 | os.remove(output) 222 | shutil.rmtree(tmpdir) 223 | 224 | return (src_md5, src_sha256, src_dir, license_file, license_md5, build_deps) 225 | 226 | 227 | def decide_version(spec): 228 | version = spec[2] 229 | if version: 230 | version = version[0] 231 | relation = version[0] 232 | version = version[1] 233 | 234 | if relation == "==": 235 | return version 236 | elif relation == ">=": 237 | return None 238 | elif relation == "<=": 239 | return version 240 | else: 241 | return None 242 | else: 243 | return None 244 | 245 | 246 | def decide_extra(spec): 247 | extra = spec[3] 248 | if extra: 249 | if extra[0] == "and": 250 | return extra[2][2] 251 | else: 252 | return extra[2] 253 | else: 254 | return None 255 | 256 | 257 | def parse_requires_dist(requires_dist): 258 | spec = parser.parse(requires_dist) 259 | ret = Dependency(spec[0], decide_version(spec), decide_extra(spec)) 260 | return ret 261 | 262 | def pkg_size(pkg): 263 | # whl is omitted as we prefer source package 264 | extensions = ["tar", "tar.gz", "tar.bz2", "tar.xz"] 265 | for extension in extensions: 266 | if pkg["url"].endswith(extension): 267 | return pkg["size"] 268 | if pkg["url"].endswith("zip"): 269 | return pkg["size"] * 10 270 | return pkg["size"] * 10000 271 | 272 | 273 | def fetch_requirements_from_remote_package(info, version): 274 | """ Looks up requires_dist from an actual package """ 275 | 276 | # Version must exists, otherwise previous steps failed 277 | pkg_versions = info["releases"][version] 278 | 279 | # If we must fetch a package, lets fetch the smallest one 280 | pkg_url = sorted(pkg_versions, key=pkg_size, reverse=False)[0]["url"] 281 | filename = pkg_url.split("/")[-1] 282 | 283 | # Select the appropriate parser from pkginfo based on the filename 284 | parse = None 285 | if filename.endswith(".tar.gz") or filename.endswith(".zip") or filename.endswith(".tar.xz") or filename.endswith(".tar.bz2") or filename.endswith(".tar"): 286 | parse = pkginfo.SDist 287 | elif filename.endswith(".whl"): 288 | parse = pkginfo.Wheel 289 | elif filename.endswith(".egg"): 290 | parse =pkginfo.BDist 291 | else: 292 | raise RuntimeError("Unsupported fileformat for package introspection: {}".format(filename)) 293 | 294 | # Download the package and read the MANIFEST 295 | with tempfile.TemporaryDirectory() as directory: 296 | path = os.path.join(directory, filename) 297 | urllib.request.urlretrieve(pkg_url, path) 298 | return parse(path).requires_dist 299 | 300 | 301 | def get_package_dependencies(requires_dist, follow_extras=False): 302 | deps = [] 303 | 304 | if requires_dist: 305 | for dep in requires_dist: 306 | d = parse_requires_dist(dep) 307 | if d.extra and not follow_extras: 308 | continue 309 | deps.append(d) 310 | 311 | return deps 312 | 313 | 314 | PROCESSED_PACKAGES = [] 315 | 316 | 317 | def get_package_info( 318 | package, 319 | version=None, 320 | packages=None, 321 | indent=0, 322 | extra=None, 323 | follow_extras=False, 324 | default_license=None, 325 | ): 326 | global PROCESSED_PACKAGES 327 | 328 | package_name = package.split('[')[0] 329 | # extra_needed = package.split('[')[1].replace("]", "") 330 | 331 | if not packages: 332 | packages = [[]] 333 | elif package_name in [package.name for package in PROCESSED_PACKAGES] or package_name in [ 334 | package.name for package in packages[0] 335 | ]: 336 | return packages[0] 337 | 338 | indent_str = "" 339 | if indent: 340 | indent_str = "|" + (indent - 2) * "-" + " " 341 | 342 | extra_str = "" 343 | if extra: 344 | extra_str = "[{}]".format(extra) 345 | 346 | print( 347 | " {}{}{}{}".format( 348 | indent_str, package, extra_str, "=={}".format(version) if version else "" 349 | ) 350 | ) 351 | 352 | try: 353 | if version: 354 | if re.search('\*', version): 355 | url = "https://pypi.org/pypi/{}/json".format(package_name) 356 | response = urllib.request.urlopen(url).read().decode(encoding="UTF-8") 357 | info = json.loads(response) 358 | pv = [] 359 | v = version.split('.') 360 | print("fuzzy version {} ".format(v)) 361 | for i in info["releases"]: 362 | tv = i.split('.') 363 | found=True 364 | for j in enumerate(v): 365 | if j[1] == '*': 366 | break; 367 | if j[1] != tv[j[0]]: 368 | found=False 369 | break 370 | if found: 371 | pv.append(i) 372 | version = pv[-1] 373 | 374 | 375 | url = "https://pypi.org/pypi/{}/{}/json".format(package_name, version) 376 | else: 377 | url = "https://pypi.org/pypi/{}/json".format(package_name) 378 | 379 | response = urllib.request.urlopen(url).read().decode(encoding="UTF-8") 380 | info = json.loads(response) 381 | 382 | name = package_name 383 | version = info["info"]["version"] 384 | summary = info["info"]["summary"].replace('\n', ' \\\n') 385 | homepage = info["info"]["home_page"] 386 | author = info["info"]["author"] 387 | author_email = info["info"]["author_email"] 388 | license = translate_license(info["info"]["license"], default_license) 389 | 390 | try: 391 | version_info = next( 392 | i for i in info["releases"][version] if i["packagetype"] == "sdist" 393 | ) 394 | except: 395 | raise Exception("No sdist package can be found.") 396 | 397 | src_uri = version_info["url"] 398 | src_md5, src_sha256, src_dir, license_file, license_md5, build_deps = get_package_file_info( 399 | package_name, version, src_uri 400 | ) 401 | 402 | requires_dist = info["info"]["requires_dist"] 403 | 404 | # Only parse if requires_dist is missing, e.g. sentry-sdk 405 | if requires_dist is None: 406 | requires_dist = fetch_requirements_from_remote_package(info, version) 407 | 408 | dependencies = get_package_dependencies(requires_dist, follow_extras=follow_extras) 409 | 410 | package = Package( 411 | name, 412 | version, 413 | summary, 414 | homepage, 415 | author, 416 | author_email, 417 | license, 418 | license_file, 419 | license_md5, 420 | src_dir, 421 | src_uri, 422 | src_md5, 423 | src_sha256, 424 | dependencies, 425 | build_deps, 426 | ) 427 | 428 | packages[0].append(package) 429 | PROCESSED_PACKAGES.append(package) 430 | 431 | for dependency in dependencies: 432 | get_package_info( 433 | dependency.name, 434 | version=dependency.version, 435 | packages=packages, 436 | indent=indent + 2, 437 | extra=dependency.extra, 438 | follow_extras=follow_extras, 439 | default_license=default_license, 440 | ) 441 | 442 | except Exception as e: 443 | print( 444 | " {} [ERROR] Failed to gather {} ({})".format(indent_str, package, str(e)) 445 | ) 446 | 447 | return packages[0] 448 | 449 | 450 | def generate_recipe(package, outdir, python, is_extra=False, use_pypi=False): 451 | basename = "{}-{}_{}.bb".format( 452 | python, package_to_bb_name(package.name), package.version 453 | ) 454 | bbfile = os.path.join(outdir, basename) 455 | 456 | print(" {}".format(basename)) 457 | 458 | if is_extra: 459 | output = BB_EXTRA_TEMPLATE.format( 460 | summary=package.summary, 461 | homepage=package.homepage, 462 | author=package.author, 463 | author_email=package.author_email, 464 | dependencies=" ".join( 465 | [ 466 | "{}-{}".format(python, package_to_bb_name(dep.name)) 467 | for dep in package.dependencies 468 | ] 469 | ), 470 | ) 471 | else: 472 | selected_template = BB_TEMPLATE_PYPI if use_pypi else BB_TEMPLATE 473 | output = selected_template.format( 474 | summary=package.summary, 475 | md5=package.src_md5, 476 | sha256=package.src_sha256, 477 | src_uri=package.src_uri, 478 | src_dir=package.src_dir, 479 | pypi_package=package.name, 480 | pypi_package_ext="\nPYPI_PACKAGE_EXT = \"" + get_file_extension(package.src_uri) + "\"" if not package.src_uri.endswith(".tar.gz") else "", 481 | license=package.license, 482 | license_file=package.license_file, 483 | license_md5=package.license_md5, 484 | homepage=package.homepage, 485 | author=package.author, 486 | author_email=package.author_email, 487 | build_dependencies=" ".join( 488 | [ 489 | dep 490 | for dep in package.build_dependencies 491 | ] 492 | ), 493 | dependencies=" ".join( 494 | [ 495 | "{}-{}".format(python, package_to_bb_name(dep.name)) 496 | for dep in package.dependencies 497 | ] 498 | ), 499 | setuptools="3" if python == "python3" else "", 500 | ) 501 | 502 | with open(bbfile, "w") as outfile: 503 | outfile.write(output) 504 | 505 | 506 | def parse_requirements(requirements_file, follow_extras=False, default_license=None): 507 | packages = [] 508 | 509 | with open(requirements_file, "r") as infile: 510 | for package in infile.read().split("\n"): 511 | package = package.strip() 512 | if package: 513 | if not (package.startswith("-e") or package.startswith(".")): 514 | parts = [part.strip() for part in package.split("==")] 515 | if len(parts) == 2: 516 | packages += get_package_info( 517 | parts[0], 518 | parts[1], 519 | follow_extras=follow_extras, 520 | default_license=default_license, 521 | ) 522 | elif len(parts) == 1: 523 | packages += get_package_info( 524 | parts[0], 525 | None, 526 | follow_extras=follow_extras, 527 | default_license=default_license, 528 | ) 529 | else: 530 | print(" Unparsed package: {}".format(package)) 531 | else: 532 | print(" Skipping: {}".format(package)) 533 | 534 | return packages 535 | 536 | 537 | def write_preferred_versions(packages, outfile, python): 538 | versions = [] 539 | for package in packages: 540 | versions.append( 541 | 'PREFERRED_VERSION_{}-{} = "{}"'.format( 542 | python, package_to_bb_name(package.name), package.version 543 | ) 544 | ) 545 | 546 | with open(outfile, "w") as outfile: 547 | outfile.write("\n".join(versions)) 548 | 549 | 550 | def generate_recipes(packages, outdir, python, follow_extras=False, pypi=False): 551 | for package in packages: 552 | generate_recipe(package, outdir, python, use_pypi=pypi) 553 | 554 | if follow_extras: 555 | extras = [dep for dep in package.dependencies if dep.extra] 556 | processed = [] 557 | for extra in extras: 558 | if extra.extra in processed: 559 | continue 560 | 561 | processed.append(extra.extra) 562 | extra_package = package 563 | extra_package = extra_package._replace( 564 | name=package.name + "-{}".format(extra.extra) 565 | ) 566 | extra_package = extra_package._replace( 567 | dependencies=[Dependency(package.name, package.version, None)] 568 | + [ 569 | Dependency(e.name, e.version, None) 570 | for e in extras 571 | if e.extra == extra.extra 572 | ] 573 | ) 574 | generate_recipe(extra_package, outdir, python, is_extra=True, use_pypi=pypi) 575 | 576 | 577 | def main(): 578 | try: 579 | parser = argparse.ArgumentParser() 580 | parser.add_argument("--package", "-p", help="The package to process.") 581 | parser.add_argument( 582 | "--version", "-v", help="The package version.", default=None 583 | ) 584 | parser.add_argument("--requirements", "-r", help="The pypi requirements file.") 585 | parser.add_argument( 586 | "--extras", "-e", action="store_true", help="Generate recipes for extras." 587 | ) 588 | parser.add_argument( 589 | "--outdir", "-o", help="The recipe directory.", default="./" 590 | ) 591 | parser.add_argument( 592 | "--python", 593 | "-y", 594 | help="The python version to use.", 595 | default="python", 596 | choices=["python", "python3"], 597 | ) 598 | parser.add_argument( 599 | "--licenses", 600 | "-l", 601 | action="store_true", 602 | help="Output an updated license map upon completion.", 603 | ) 604 | parser.add_argument( 605 | "--default-license", 606 | "-d", 607 | help="The default license to use when the package license cannot be mapped.", 608 | default=None, 609 | ) 610 | parser.add_argument( 611 | "--pypi", 612 | "-s", 613 | action="store_true", 614 | help="Use oe pypi class for recipe" 615 | ) 616 | args = parser.parse_args() 617 | 618 | print("Gathering info:") 619 | packages = [] 620 | if args.requirements: 621 | packages = parse_requirements( 622 | args.requirements, 623 | follow_extras=args.extras, 624 | default_license=args.default_license, 625 | ) 626 | elif args.package: 627 | packages = get_package_info( 628 | args.package, 629 | args.version, 630 | follow_extras=args.extras, 631 | default_license=args.default_license, 632 | ) 633 | else: 634 | raise Exception("No packages provided!") 635 | 636 | print("Generating recipes:") 637 | generate_recipes(packages, args.outdir, args.python, args.extras, args.pypi) 638 | 639 | version_file = os.path.join(args.outdir, "{}-versions.inc".format(args.python)) 640 | write_preferred_versions(packages, version_file, args.python) 641 | 642 | print() 643 | if args.licenses: 644 | license_file = os.path.join(args.outdir, "licenses.py") 645 | with open(license_file, "w") as outfile: 646 | outfile.write("LICENSES = " + pformat(licenses.LICENSES)) 647 | 648 | print("License mappings are available in: {}".format(license_file)) 649 | 650 | print("PREFERRED_VERSIONS are available in: {}".format(version_file)) 651 | 652 | except Exception as e: 653 | print(str(e)) 654 | sys.exit(1) 655 | except KeyboardInterrupt: 656 | os._exit(1) 657 | 658 | 659 | if __name__ == "__main__": 660 | main() 661 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Parsley==1.3 2 | pep508-parser==2019.3 3 | pkginfo==1.5.0.1 4 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | from setuptools import setup, find_packages 4 | from os import path 5 | from io import open 6 | 7 | here = path.abspath(path.dirname(__file__)) 8 | 9 | with open(path.join(here, "README.md"), encoding="utf-8") as f: 10 | long_description = f.read() 11 | 12 | install_requires = [] 13 | with open("requirements.txt", "r") as infile: 14 | install_requires = [r for r in infile.read().split("\n") if r] 15 | 16 | setup( 17 | name="pipoe", 18 | version="2020.10", 19 | description="Generate python bitbake recipes from pypi metadata.", 20 | long_description=long_description, 21 | long_description_content_type="text/markdown", 22 | url="https://github.com/NFJones/pipoe", 23 | author="Neil F Jones", 24 | classifiers=[ 25 | "Intended Audience :: Developers", 26 | "Topic :: Software Development :: Build Tools", 27 | "License :: OSI Approved :: MIT License", 28 | "Programming Language :: Python :: 3", 29 | ], 30 | keywords="yocto bitbake openembedded", 31 | packages=["pipoe"], 32 | python_requires="!=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4", 33 | install_requires=install_requires, 34 | entry_points={"console_scripts": ["pipoe = pipoe.pipoe:main"]}, 35 | ) 36 | --------------------------------------------------------------------------------