├── LICENSE ├── README.md └── etc └── portage ├── bashrc └── binhost └── gh-upload.py /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2002 JSON.org 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | The Software shall be used for Good, not Evil. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 20 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 21 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 22 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # gentoo binhost 2 | 3 | Providing [gentoo](https://gentoo.org/) binary packages using [github](https://github.com/) infrastructure. 4 | 5 |
gentoo-logo
6 | 7 | This repo provides various gentoo [binary packages](https://wiki.gentoo.org/wiki/Binary_package_guide) for a variety of different architectures (checkout branches for details). This branch contains the script that is used for GitHub upload. 8 | 9 | ## Concept 10 | 11 | The package upload is realized using a small upload script thats executed via portage [hooks](https://wiki.gentoo.org/wiki//etc/portage/bashrc). For every package that is being merged via portage the Gentoo *Packages* manifest file is committed to Git. The binary packages itself are not stored into repository there are uploaded as [GitHub release](https://developer.github.com/v3/repos/releases) artifacts. 12 | 13 | To make everything work the following nomenclature has to apply: 14 | 15 | Gentoo idiom|GitHub entity 16 | ------------|------------- 17 | [CATEGORY](https://wiki.gentoo.org/wiki//etc/portage/categories)|GitHub release 18 | [PF](https://devmanual.gentoo.org/ebuild-writing/variables/)|GitHub release asset 19 | [CHOST](https://wiki.gentoo.org/wiki/CHOST)|Git branch name 20 | [CHOST](https://wiki.gentoo.org/wiki/CHOST)/[CATEGORY](https://wiki.gentoo.org/wiki//etc/portage/categories)|Git release tag 21 | 22 | ## Usage 23 | 24 | Setup a gentoo binhost Github and provide the following. 25 | 26 | ### Dependencies 27 | 28 | The upload script uses Python3 and [PyGithub](https://github.com/PyGithub/PyGithub) module. 29 | 30 | ```shell 31 | emerge dev-python/PyGithub 32 | ``` 33 | 34 | ### Configuration 35 | 36 | github upload can be easily configured. 37 | 38 | #### make.conf 39 | 40 | Enable gentoo binhost by adding the following lines. 41 | ```python 42 | # enable binhost 43 | PORTAGE_BINHOST_HEADER_URI="https://github.com/coldnew/gentoo-binhost/releases/download/${CHOST}" 44 | FEATURES="${FEATURES} buildpkg" 45 | USE="${USE} bindist" 46 | ACCEPT_LICENSE="-* @BINARY-REDISTRIBUTABLE" 47 | ``` 48 | 49 | Since github releases are used to store the packages *PORTAGE_BINHOST_HEADER_URI* has to be set here. 50 | 51 | #### bashrc 52 | 53 | Add the [/etc/portage/bashrc ](https://wiki.gentoo.org/wiki//etc/portage/bashrc) file below, if you use your own file make sure to call the **gh-upload.py** script during **postinst** phase. 54 | 55 | ```bash 56 | #!/bin/env bash 57 | 58 | if [[ ${EBUILD_PHASE} == 'postinst' ]]; then 59 | # FIXME come up with a more sophisticated approach to detect if binary package build is actually requested 60 | # commandline args like -B or --buildpkg-exclude and other conditionals are not supported right now. 61 | grep -q 'buildpkg' <<< {$PORTAGE_FEATURES} 62 | if [ $? -eq 0 ]; then 63 | /etc/portage/binhost/gh-upload.py 64 | fi 65 | fi 66 | ``` 67 | 68 | #### gh-upload.py 69 | 70 | Add the [/etc/portage/binhost/gh-upload.py](/etc/portage/binhost/gh-upload.py) script and add your github settings accordingly. 71 | You need to create a [github access token](https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line) that is able to access repository and create releases. 72 | 73 | ```python 74 | gh_repo = 'coldnew/gentoo-binhost' 75 | gh_token = '' 76 | ``` 77 | 78 | ## Disclaimer 79 | 80 | Although this software is released under [JSON](/LICENSE) license, the binary packages come with their respective license according to *Packages* Manifest file. Refer to [gentoo license](https://devmanual.gentoo.org/general-concepts/licenses/index.html) for details. 81 | -------------------------------------------------------------------------------- /etc/portage/bashrc: -------------------------------------------------------------------------------- 1 | #!/bin/env bash 2 | 3 | if [[ ${EBUILD_PHASE} == 'postinst' ]]; then 4 | # FIXME come up with a more sophisticated approach to detect if binary package build is actually requested 5 | # commandline args like -B or --buildpkg-exclude and other conditionals are not supported right now. 6 | grep -q 'buildpkg' <<< {$PORTAGE_FEATURES} 7 | if [ $? -eq 0 ]; then 8 | /etc/portage/binhost/gh-upload.py 9 | fi 10 | fi 11 | -------------------------------------------------------------------------------- /etc/portage/binhost/gh-upload.py: -------------------------------------------------------------------------------- 1 | #!/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | # Copyright 2021-2022 by Yen-Chin, Lee 5 | # Copyright 2020 by generik at spreequalle.de. All rights reserved. 6 | # This file is released under the "JSON License Agreement". Please see the LICENSE 7 | # file that should have been included as part of this package. 8 | 9 | import os 10 | import socket 11 | import re 12 | import xml.etree.ElementTree as xml 13 | from pathlib import Path 14 | from github import Github, GithubException, UnknownObjectException, InputGitAuthor 15 | 16 | gh_repo = 'coldnew/gentoo-binhost' 17 | gh_token = '' 18 | gh_branch = os.environ['CHOST'] # use chost as git branch name 19 | gh_relName = gh_branch + '/' + os.environ['CATEGORY'] # create new github release for every category 20 | gh_author = InputGitAuthor(os.environ['PORTAGE_BUILD_USER'], os.environ['PORTAGE_BUILD_USER'] + '@' + socket.getfqdn()) 21 | g_header_uri = "https://github.com/{}/release/download/{}".format(gh_repo, gh_branch) 22 | 23 | g_pkgName = os.environ['PF'] # create a new github asset for every package 24 | g_pkgVersion = os.environ['PV'] 25 | g_cat = os.environ['CATEGORY'] 26 | 27 | # detect pkgdir layout 28 | # https://wiki.gentoo.org/wiki/Binary_package_guide 29 | g_pkgdirLayoutVersion = 2 if os.environ['PORTAGE_FEATURES'].__contains__('binpkg-multi-instance') else 1 30 | 31 | g_xpakExt = 'tbz2' # XPAK extension (chanding compression scheme $BINPKG_COMPRESS does not change the extenstion) 32 | g_xpak = os.environ['PF'] + '.' + g_xpakExt 33 | g_xpakPath = os.environ['PKGDIR'] + '/' + g_cat + '/' + g_xpak 34 | g_xpakStatus = ' added.' 35 | g_manifest = 'Packages' 36 | g_manifestPath = os.environ['PKGDIR'] + '/' + g_manifest 37 | 38 | if g_pkgdirLayoutVersion == 2: 39 | g_xpakExt = 'xpak' 40 | g_xpakDir = os.environ['PKGDIR'] + '/' + g_cat + '/' + os.environ['PN'] 41 | g_buildID = str(len([name for name in os.listdir(g_xpakDir) if os.path.isfile(os.path.join(g_xpakDir,name)) and name.startswith(os.environ['PF'] + '-')])) 42 | g_xpak = os.environ['PF'] + '-' + g_buildID + '.' + g_xpakExt 43 | g_xpakPath = os.environ['PKGDIR'] + '/' + g_cat + '/' + os.environ['PN'] + '/' + g_xpak 44 | # create new github release for every category 45 | g_cat = os.environ['CATEGORY'] + '/' + os.environ['PN'] 46 | gh_relName = gh_branch + '/' + g_cat # create new github release for every category 47 | 48 | # FIXME figure out how to do this right, will fail on custom repos 49 | def getXpakDesc(): 50 | try: 51 | # this has to be relative to the ebuild in case of different repos 52 | # custom repos have no metadata.xml for base categories like sys-apps 53 | # if packages from these there merged before github release create we don't get the description 54 | g_catMetadataFile = Path(os.environ['EBUILD']).parents[1] / 'metadata.xml' 55 | root = xml.parse(g_catMetadataFile) 56 | g_catDesc = root.findall('./longdescription[@lang="en"]') 57 | 58 | if len(g_catDesc) > 0: 59 | g_catDesc = g_catDesc[0].text.strip() 60 | g_catDesc = re.sub('^\s*', '', g_catDesc, flags=re.M) # strip leading spaces> 61 | g_catDesc = re.sub('\n', ' ', g_catDesc, flags=re.M) # convert to single lin> 62 | except: 63 | g_catDesc = 'custom category' 64 | 65 | return g_catDesc 66 | 67 | def getEbuildDesc(): 68 | """Get DESCRIPTION from ebuild""" 69 | try: 70 | g_catDesc = '' 71 | # read description fron ebuild 72 | ebuildPath = os.environ['EBUILD'] 73 | with open(ebuildPath, 'r', encoding='utf-8') as ebuildFile: 74 | for line in ebuildFile: 75 | line = line.strip() 76 | try: 77 | key, value = line.split('=', 1) 78 | except ValueError: 79 | continue 80 | if key == 'DESCRIPTION': 81 | g_catDesc = value 82 | # remove quotes at start and end 83 | g_catDesc = g_catDesc.strip('\"') 84 | except: 85 | g_catDesc = '' 86 | 87 | return g_catDesc 88 | 89 | g = Github(gh_token, timeout = 280) 90 | repo = g.get_repo(gh_repo) 91 | 92 | # make sure we are working on an existent branch 93 | try: 94 | branch = repo.get_branch(gh_branch) 95 | except GithubException: 96 | print("branch not found!\nCreate git branch: '%s' first!" % gh_branch) 97 | exit(1) 98 | 99 | # get release 100 | try: 101 | rel = repo.get_release(gh_relName) 102 | # create new release (gentoo category), read category description from gentoo metadata 103 | except UnknownObjectException: 104 | if g_pkgdirLayoutVersion == 2: 105 | g_catDesc = getEbuildDesc() 106 | else: 107 | g_catDesc = getXpakDesc() 108 | 109 | rel = repo.create_git_release(gh_relName, g_cat, g_catDesc, target_commitish=gh_branch) 110 | 111 | # upload packages as an gitlab asset 112 | assets = rel.get_assets() 113 | for asset in rel.get_assets(): 114 | if asset.name == g_xpak: 115 | g_xpakStatus = ' updated.' 116 | asset.delete_asset() 117 | 118 | asset = rel.upload_asset(path=g_xpakPath, content_type='application/x-tar', name=g_xpak) 119 | print('GIT ' + g_xpak + ' upload') 120 | 121 | # create/update Packages file 122 | try: 123 | commitMsg = g_cat + "-" + g_pkgVersion + g_xpakStatus 124 | with open(g_manifestPath, 'r') as file: 125 | g_manifestFile = file.read() 126 | 127 | # check if we need to insert PORTAGE_BINHOST_HEADER_URI in Packages 128 | # the URI: entry will always between PROFILE: and TIMESTAMP: 129 | def insertURI(match): 130 | return match.group(1) + "URI: {}\n".format(g_header_uri) + match.group(2) 131 | g_manifestFile = re.sub(r'(PROFILE:.*\n)(TIMESTAMP:.*\n)', insertURI, g_manifestFile) 132 | 133 | # receive git file/blob reference via git tree 134 | ref = repo.get_git_ref(f'heads/{gh_branch}') # get branch ref 135 | tree = repo.get_git_tree(ref.object.sha).tree # get git tree 136 | sha = [x.sha for x in tree if x.path == g_manifest] # get file sha 137 | 138 | if not sha: 139 | # create new file (Packages) 140 | repo.create_file(g_manifest, commitMsg, g_manifestFile, branch=gh_branch, committer=gh_author) 141 | else: 142 | repo.update_file(g_manifest, commitMsg, g_manifestFile, sha[0], branch=gh_branch, committer=gh_author) 143 | except Exception as e: 144 | print('error handling Manifest under: ' + g_manifestPath + ' Error: ' + str(e)) 145 | exit(1) 146 | print('GIT ' + g_manifest + ' commit') 147 | 148 | --------------------------------------------------------------------------------