├── README.md ├── _funcs.sh ├── deprecated ├── badstyle.sh ├── deadeclasses.sh ├── depcheck.sh ├── dupuse.sh ├── eapichecks.sh ├── eapistats.sh ├── eclassstats.sh ├── eclassusage.sh ├── patchcheck.sh ├── patchtest.sh ├── repomancheck.sh ├── simplechecks.sh ├── srctest.sh ├── trailwhite.sh ├── treehashgen.sh └── wwwtest.sh ├── exp ├── _old_funcs.sh ├── eclasscheck.sh ├── obsoletfuncs.sh └── patchcheckdir.sh ├── genlists.sh ├── gentoo-whitelist ├── gpl-2.0.txt ├── qa.sh ├── repochecks.sh ├── repostats.sh └── tools ├── dteclass.sh ├── eclassinfo.sh ├── maintainer.py └── whitecheck.sh /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Gentoo QA Scripts 3 | 4 | This repository contains a collection of various qa scripts for gentoo ebuild repositories. Results of the these scripts can be seen on https://gentooqa.levelnine.at/ 5 | 6 | ## repostats.sh 7 | Repostats generates statistics about repos. Following statistics are generated: 8 | * ebuild_eapi_statistics: `EAPI` usage in repo 9 | * ebuild_live_statistics: lists all live (`9999`) ebuilds 10 | * ebuild_eclass_statistics: eclass usage among ebuilds 11 | * ebuild_licenses_statistics: `LICENSE` usage among ebuilds 12 | * ebuild_keywords_statistics: keyword usage among ebuilds 13 | * ebuild_virtual_use_statistics: `virtual/*` usage among ebuilds 14 | * ebuild_obsolete_eapi: lists all ebuilds which have an obsolete `EAPI` 15 | * ebuild_cleanup_candidates: lists ebuilds which are candidates to cleanup 16 | * ebuild_stable_candidates: lists ebuilds which are candidates for stabilization 17 | * ebuild_glep81_group_statistics: `app-group/*` usage among ebuilds 18 | * ebuild_glep81_user_statistics: `app-user/*` usage among ebuilds 19 | 20 | ## repochecks.sh 21 | `repochecks` scan ebuilds, categories or a whole repository for various errors and mistakes. Nowadays many checks are already done by the the amazing tool `dev-util/pkgcheck`, which is why i've included the checks from `pkgcheck` into `repochecks`. 22 | Since `pkgcheck` already has mostly the same checks as `repochecks`, checks which are basically the same are going to be removed in further updates. 23 | Following a list of all the checks done with `repochecks` and (if available) the correspondig check in `pkgcheck`. 24 | * ebuild_install_worthless_file_install: checks for ebuilds which install `INSTALL` files 25 | * ebuild_trailing_whitespaces: checks for trailing whitespaces in ebuilds 26 | * **pkgcheck**: WhitespaceFound 27 | * ebuild_obsolete_dependency_tracking: checks if ebuilds have `--disable-dependency-tracking` enabled. Not needed since `EAPI4` 28 | * ebuild_obsolete_silent_rules: checks if ebuilds have `--disable-silent-rules` enabled. Not needed since `EAPI5` 29 | * ebuild_obsolete_disable_static: checks if ebuilds have `--disable-static` enabled. Not needed since `EAPI8` 30 | * ebuild_variable_missing_braces: Simple check to find variables in ebuilds which not use curly braces. 31 | * ebuild_epatch_in_eapi6: checks for `epatch` usage in `EAPI 6` ebuilds 32 | * going to be removed with the last EAPI6 ebuild 33 | * ebuild_dohtml_in_eapi6: checks for `dohtml` usage in `EAPI 6` ebuids 34 | * going to be removed with the last EAPI6 ebuild 35 | * ebuild_description_over_80: checks if `DESCRIPTION` is longer then 80 chars 36 | * **pkgcheck**: DescriptionCheck 37 | * ebuild_variables_in_homepages: checks if `HOMEPAGE` contains variables 38 | * **pkgcheck**: ReferenceInMetadataVar 39 | * ebuild_insecure_git_uri_usage: checks for `git://` usage 40 | * ebuild_deprecated_eclasses: lists ebuilds which uses deprecated eclasses 41 | * **pkgcheck**: DeprecatedEclass 42 | * ebuild_leading_trailing_whitespaces_in_variables: checks for leading / trailing whitespaces in certain variables 43 | * ebuild_multiple_deps_per_line: lists ebuilds which have multiple dependencies in on line 44 | * ebuild_nonexist_dependency: checks for dependencies which doesn't exists 45 | * ebuild_obsolete_virtual: lists virtuals which has effective only on consumer left 46 | * ebuild_missing_eclasses: lists ebuilds which are missing eclasses 47 | * **pkgcheck**: InheritsCheck 48 | * ebuild_unused_eclasses: lists ebuilds who `inherit` eclasses which are not needed 49 | * **pkgcheck**: InheritsCheck 50 | * ebuild_missing_eclasses_fatal: lists ebuilds which are missing eclasses and are not inherited indirectly 51 | * **pkgcheck**: InheritsCheck 52 | * ebuild_homepage_upstream_shutdown: lists ebuilds who have a `HOMEPAGE` to a know shutdown service 53 | * ebuild_homepage_unsync: lists packages who have different `HOMEPAGE` among ebuilds 54 | * ebuild_missing_zip_dependency: lists ebuilds which misses `app-arch/unzip` dependency 55 | * **pkgcheck**: MissingUnpackerDep 56 | * ebuild_src_uri_offline: lists ebuilds who's upstream `SRC_URI` is unavailable and are the ebuild has mirror restricted enabled 57 | * ebuild_src_uri_bad: This check uses wget's spider functionality to check if a ebuild's `SRC_URI` link still works 58 | * ebuild_unused_patches: Extensive check to find unused pachtes. Uses a `whitelist` for false-positives 59 | * ebuild_unused_patches_simple: simple check for unused patches 60 | * ebuild_insecure_pkg_post_config: lists ebuilds who use ch{mod,own} -R in `pkg_config` or `pkg_postinst` 61 | * ebuild_insecure_init_scripts: lists ebuilds who use ch{mod,own} -R in init scripts 62 | * ebuild_homepage_redirections: lists ebuilds with a Homepage which actually redirects to another sites 63 | * **pkgcheck**: RedirectedUrl 64 | * packages_pkgcheck_scan: runs `pkgcheck scan --net --keywords=-info -q` on packages 65 | * **pkgcheck**: simply use `pkgcheck` 66 | * metadata_mixed_indentation: Checks metadata files (`metadata.xml`) if it uses mixed tabs and whitespaces 67 | * **pkgcheck**: PkgMetadataXmlIndentation 68 | * metadata_missing_proxy_maintainer: Checks the `metadata.xml` of proxy maintained packages if it includes actually a non gentoo email address (address of proxy maintainer) 69 | * **pkgcheck**: MaintainerWithoutProxy 70 | * metadata_duplicate_useflag_description: Lists packages which define use flags locally in `metadata.xml` which already exists as a global use flag. 71 | * metadata_missing_remote_id: Lists packages which has a certain homepage (github, sourceforge) but doesn't set remote-id in `metadata.xml` 72 | * **pkgcheck**: MissingRemoteId 73 | 74 | ### Limitations for ebuild_unused_patches: 75 | Following Structures can't be found (yet) and will give a false positive: 76 | * Usage of special vars which are not included in the script: `epatch foo-bar-${MY_VERS}.patch` 77 | * Usage of patching via eclasses: In this case not the ebuild uses the file, but the eclass. For example: `apache-module` do that. However we exclude most of the beforehand. 78 | * Usage for loops like: 79 | ``` 80 | for i in name1 name2 name3; do 81 | epatch foo-bar-$i.patch 82 | done 83 | ``` 84 | * False negative can also occur if the filename is the same as the package name. 85 | * Directories under the `files` directory will be ignored (for now) 86 | 87 | 88 | For a detailed explanation of each check please refer to the descriptions in `repochecks` 89 | 90 | # Misc 91 | 92 | ## qa.sh 93 | This script is used in order to run multiple repositories. It support to run scripts in `diff` mode (git diffs) in order to only run on packages which were changes since last run. 94 | 95 | # Tools 96 | 97 | ## maintainer.py 98 | This python script doesn't take any arguments and just prints every gentoo project and it members. 99 | 100 | ## dteclasses.sh 101 | Simply prints the inheration tree of a particular eclass. 102 | 103 | ## whitecheck.sh 104 | Removes obsolete entries from whitelist files (for unused patches) 105 | 106 | ## eclassinfo.sh 107 | Prints functions which can be used by ebuilds. 108 | 109 | # Usage: 110 | 111 | Following usage will work for `repochecks` and `repostats`: 112 | 113 | * Scan the full portage tree: `./scriptname.sh full` 114 | * Scan a whole category: `./scriptname.sh app-admin` 115 | * Scan a single package: `./scriptname.sh app-admin/diradm` 116 | 117 | There are also a few Variables which can be set, but don't have too. Most importantly are: 118 | 119 | * `REPOTREE`: Set's the repository path, default is: `/usr/portage` 120 | * `FILERESULTS`: If this is set to `true` the script will save it's output in files. Default is `false` 121 | * `RESULTSDIR`: This is the directory were the files will be copied if `SCRIPT_MODE` is enabled. Default is set to `${HOME}/${scriptname}` 122 | 123 | ### Debugging 124 | Most script also have some debugging possibilities. Following variables can be configured: 125 | * `DEBUG`: if this is set to `true`, debugging is enabled 126 | * `DEBUGLEVEL`: if unset, the default is set to `1`. Changes in Levels: 127 | * `1` only print basic information, in this case script runs in parallel 128 | * `2` includes important messages, script runs not in parallel anymore 129 | * `3` includes non important messages 130 | * `<=4` includes messages from `_funcs.sh` as well. 131 | * `DEBUGFILE`: redirects the debug output into a file. 132 | 133 | # License: 134 | 135 | All scripts are free software. You can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. 136 | -------------------------------------------------------------------------------- /deprecated/badstyle.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: badstyle.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 14/01/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # checks for multiple package dependencies in one line 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export FILERESULTS=true 28 | #export RESULTSDIR="${HOME}/badstyle/" 29 | #export REPOTREE=/usr/portage/ 30 | 31 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 32 | if [ -e ${realdir}/_funcs.sh ]; then 33 | source ${realdir}/_funcs.sh 34 | else 35 | echo "Missing _funcs.sh" 36 | exit 1 37 | fi 38 | 39 | # 40 | ### IMPORTANT SETTINGS START ### 41 | # 42 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 43 | ${ENABLE_MD5} || exit 0 # only works with md5 cache 44 | #${ENABLE_GIT} || exit 0 # only works with git tree 45 | 46 | SCRIPT_TYPE="checks" 47 | WORKDIR="/tmp/badstyle-${RANDOM}" 48 | 49 | array_names(){ 50 | RUNNING_CHECKS=( 51 | "${WORKDIR}/ebuild_multiple_deps_per_line" #Index 0 52 | ) 53 | } 54 | output_format(){ 55 | index=( 56 | "${ebuild_eapi}${DL}${category}/${package}${DL}${filename}${DL}${maintainer}" 57 | ) 58 | echo "${index[$1]}" 59 | } 60 | data_descriptions(){ 61 | read -r -d '' info_index0 <<- EOM 62 | Ebuilds which have multiple dependencies written in one line like: || ( app-arch/foo app-arch/bar ) 63 | Should look like: || ( 64 | app-arch/foo 65 | app-arch/bar 66 | ) 67 | Also see at: Link 68 | 69 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|dev@gentoo.org:loper@foo.de ): 70 | 7 EAPI Version 71 | dev-libs/foo package category/name 72 | foo-1.12-r2.ebuild full filename 73 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 74 | EOM 75 | description=( "${info_index0}" ) 76 | echo "${description[$1]}" 77 | } 78 | # 79 | ### IMPORTANT SETTINGS END ### 80 | # 81 | 82 | main() { 83 | array_names 84 | repo_categories 85 | local absolute_path=${1} 86 | local category="$(echo ${absolute_path}|cut -d'/' -f1)" 87 | local package="$(echo ${absolute_path}|cut -d'/' -f2)" 88 | local filename="$(echo ${absolute_path}|cut -d'/' -f3)" 89 | local packagename="${filename%.*}" 90 | local maintainer="$(get_main_min "${category}/${package}")" 91 | local ebuild_eapi="$(get_eapi ${absolute_path})" 92 | 93 | local used_cats=( ) 94 | for cat in ${all_cat[@]}; do 95 | if $(grep DEPEND /${REPOTREE}/metadata/md5-cache/${category}/${packagename} | grep -q ${cat}); then 96 | used_cats+=( "${cat}" ) 97 | fi 98 | done 99 | 100 | output() { 101 | local id=${1} 102 | if ${FILERESULTS}; then 103 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 104 | else 105 | output_format ${id} 106 | fi 107 | } 108 | 109 | if [ -n "${used_cats}" ]; then 110 | #remove duplicates from found categories 111 | cat_to_check=($(printf "%s\n" "${used_cats[@]}" | sort -u)) 112 | 113 | x=0 114 | y="${#cat_to_check[@]}" 115 | z=( ) 116 | 117 | for a in ${cat_to_check[@]}; do 118 | for b in ${cat_to_check[@]:${x}:${y}}; do 119 | if [ "${a}" = "${b}" ]; then 120 | z+=( "${a}/.*${b}/.*" ) 121 | else 122 | z+=( "${a}/.*${b}/.*|${b}/.*${a}/.*" ) 123 | fi 124 | done 125 | # search the pattern 126 | x=$(expr ${x} + 1) 127 | done 128 | 129 | 130 | if $(grep "^[^#;]" ${absolute_path} | egrep -q "$(echo ${z[@]}|tr ' ' '|')" ); then 131 | output 0 132 | fi 133 | fi 134 | } 135 | 136 | # create a list of categories in ${REPOTREE} 137 | repo_categories(){ 138 | all_cat=( $(find ${REPOTREE} -mindepth 1 -maxdepth 1 -type d -regextype sed -regex "./*[a-z0-9].*-[a-z0-9].*" -printf '%P\n') ) 139 | [ -e ${REPOTREE}/virtual ] && all_cat+=( "virtual" ) 140 | } 141 | 142 | find_func(){ 143 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 144 | -type f -name "*.ebuild" -exec egrep -l "DEPEND" {} \; | parallel main {} 145 | } 146 | 147 | gen_results(){ 148 | if ${FILERESULTS}; then 149 | gen_descriptions 150 | sort_result_v2 151 | gen_sort_main_v3 152 | gen_sort_pak_v3 153 | 154 | copy_checks ${SCRIPT_TYPE} 155 | fi 156 | } 157 | 158 | array_names 159 | # switch to the REPOTREE dir 160 | cd ${REPOTREE} 161 | # export important variables and functions 162 | export WORKDIR 163 | export -f main array_names repo_categories output_format 164 | # create all folders 165 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 166 | # set the search depth 167 | depth_set_v2 ${1} 168 | # cleanup 169 | ${FILERESULTS} && rm -rf ${WORKDIR} 170 | -------------------------------------------------------------------------------- /deprecated/deadeclasses.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: deadeclasses.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 24/05/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # lists deprecated eclasses 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export REPOTREE=/usr/portage/ 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/deadeclasses/" 30 | 31 | # get dirpath and load funcs.sh 32 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 33 | if [ -e ${realdir}/_funcs.sh ]; then 34 | source ${realdir}/_funcs.sh 35 | else 36 | echo "Missing _funcs.sh" 37 | exit 1 38 | fi 39 | 40 | # 41 | ### IMPORTANT SETTINGS START ### 42 | # 43 | # feature requirements 44 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 45 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 46 | #${ENABLE_GIT} || exit 0 # only works with git tree 47 | 48 | WORKDIR="/tmp/deadeclasses-${RANDOM}" 49 | SCRIPT_TYPE="checks" 50 | 51 | array_names(){ 52 | RUNNING_CHECKS=( 53 | "${WORKDIR}/ebuild_deprecated_eclasses" #Index 0 54 | ) 55 | } 56 | output_format(){ 57 | index=( 58 | "$(get_eapi ${full_path_ebuild})${DL}${category}/${package}${DL}${filename}${DL}$(echo ${found_usage[@]}|tr ' ' ':')${DL}${maintainer}" 59 | ) 60 | echo "${index[$1]}" 61 | } 62 | data_descriptions(){ 63 | read -r -d '' info_index0 <<- EOM 64 | Lists ebuilds who use deprecated or obsolete eclasses. 65 | 66 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|user:cmake-utils|dev@gentoo.org:loper@foo.de ): 67 | 7 EAPI Version 68 | dev-libs/foo package category/name 69 | foo-1.12-r2.ebuild full filename 70 | user:cmake-utils list obsolete eclasse(s), seperated by ':' 71 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 72 | EOM 73 | description=( "${info_index0}" ) 74 | echo "${description[$1]}" 75 | } 76 | # 77 | ### IMPORTANT SETTINGS END ### 78 | # 79 | 80 | main() { 81 | array_names 82 | local relative_path=${1} 83 | local category="$(echo ${relative_path}|cut -d'/' -f1)" 84 | local package="$(echo ${relative_path}|cut -d'/' -f2)" 85 | local filename="$(echo ${relative_path}|cut -d'/' -f3)" 86 | local packagename="${filename%.*}" 87 | local full_path="${REPOTREE}/${category}/${package}" 88 | local full_path_ebuild="${REPOTREE}/${category}/${package}/${filename}" 89 | local maintainer="$(get_main_min "${category}/${package}")" 90 | # https://github.com/gentoo/portage/blob/master/repoman/lib/repoman/modules/linechecks/deprecated/inherit.py 91 | local dead_eclasses=( readme.gentoo base bash-completion boost-utils clutter cmake-utils confutils distutils epatch fdo-mime games gems git-2 gpe gst-plugins-bad gst-plugins-base gst-plugins-good gst-plugins-ugly gst-plugins10 ltprune mono python ruby user versionator x-modular xfconf ) 92 | local found_usage=( ) 93 | 94 | output() { 95 | local id=${1} 96 | if ${FILERESULTS}; then 97 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 98 | else 99 | output_format ${id} 100 | fi 101 | } 102 | 103 | for eclass in ${dead_eclasses[@]}; do 104 | if $(check_eclasses_usage ${full_path_ebuild} ${eclass}); then 105 | found_usage+=( ${eclass} ) 106 | fi 107 | done 108 | 109 | 110 | if [ -n "${found_usage}" ]; then 111 | output 0 112 | fi 113 | } 114 | 115 | find_func(){ 116 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 117 | -type f -name "*.ebuild" -exec egrep -l "inherit" {} \; | parallel main {} 118 | } 119 | 120 | gen_results(){ 121 | if ${FILERESULTS}; then 122 | gen_descriptions 123 | sort_result_v2 2 124 | for file in $(cat ${RUNNING_CHECKS[0]}/full.txt); do 125 | for ec in $(echo ${file}|cut -d'|' -f4|tr ':' ' '); do 126 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-filter/${ec}.eclass 127 | echo ${file} >> ${RUNNING_CHECKS[0]}/sort-by-filter/${ec}.eclass/full.txt 128 | done 129 | done 130 | 131 | for ecd in $(ls ${RUNNING_CHECKS[0]}/sort-by-filter/); do 132 | gen_sort_main_v3 ${RUNNING_CHECKS[0]}/sort-by-filter/${ecd} 133 | gen_sort_pak_v3 ${RUNNING_CHECKS[0]}/sort-by-filter/${ecd} 134 | done 135 | 136 | gen_sort_main_v3 137 | gen_sort_pak_v3 138 | 139 | copy_checks ${SCRIPT_TYPE} 140 | fi 141 | } 142 | 143 | array_names 144 | # switch to the REPOTREE dir 145 | cd ${REPOTREE} 146 | # export important variables 147 | export WORKDIR 148 | export -f main array_names output_format 149 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 150 | # set the search depth 151 | depth_set_v2 ${1} 152 | # cleanup 153 | ${FILERESULTS} && rm -rf ${WORKDIR} 154 | -------------------------------------------------------------------------------- /deprecated/depcheck.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: depcheck.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 28/03/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simply script to find packages in (R)DEPEND block which 25 | # doesn't exist anymore (mainly obsolete blocks via !category/package) 26 | 27 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 28 | #export REPOTREE=/usr/portage/ 29 | #export FILERESULTS=true 30 | #export RESULTSDIR="${HOME}/depcheck/" 31 | 32 | # get dirpath and load funcs.sh 33 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 34 | if [ -e ${realdir}/_funcs.sh ]; then 35 | source ${realdir}/_funcs.sh 36 | else 37 | echo "Missing _funcs.sh" 38 | exit 1 39 | fi 40 | 41 | # 42 | ### IMPORTANT SETTINGS START ### 43 | # 44 | # don't run on overlays because dependencies are most likely only 45 | # available at the main tree 46 | ${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 47 | ${ENABLE_MD5} || exit 0 # only works with md5 cache 48 | #${ENABLE_GIT} || exit 0 # only works with git tree 49 | 50 | WORKDIR="/tmp/depcheck-${RANDOM}" 51 | SCRIPT_TYPE="checks" 52 | 53 | array_names(){ 54 | RUNNING_CHECKS=( 55 | "${WORKDIR}/ebuild_nonexist_dependency" #Index 0 56 | "${WORKDIR}/ebuild_obsolete_virtual" #Index 1 57 | ) 58 | } 59 | output_format(){ 60 | index=( 61 | "$(get_eapi ${full_path_ebuild})${DL}${category}/${package}${DL}${filename}${DL}$(echo ${obsolete_dep[@]}|tr ' ' ':')${DL}${maintainer}" 62 | "$(get_eapi ${full_path_ebuild})${DL}${category}/${package}${DL}${filename}${DL}${maintainer}" 63 | ) 64 | echo "${index[$1]}" 65 | } 66 | data_descriptions(){ 67 | read -r -d '' info_index0 <<- EOM 68 | This checks the ebuilds *DEPEND* Blocks for packages which doesn't exist anymore. 69 | 70 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|sys-apps/bar:dev-libs/libdir(2015-08-13)|dev@gentoo.org:loper@foo.de ): 71 | 7 EAPI Version 72 | dev-libs/foo package category/name 73 | foo-1.12-r2.ebuild full filename 74 | sys-apps/bar:dev-libs/libdir(2015-08-13) non-existing package(s). If removed after git migration a removal date is shown. 75 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 76 | EOM 77 | read -r -d '' info_index1 <<- EOM 78 | Lists virtuals were only one provider is still available. 79 | 80 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|dev@gentoo.org:loper@foo.de ): 81 | 7 EAPI Version 82 | dev-libs/foo package category/name 83 | foo-1.12-r2.ebuild full filename 84 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 85 | EOM 86 | description=( "${info_index0}" "${info_index1}" ) 87 | echo "${description[$1]}" 88 | } 89 | # 90 | ### IMPORTANT SETTINGS END ### 91 | # 92 | 93 | main() { 94 | array_names 95 | local absolute_path=${1} 96 | local category="$(echo ${absolute_path}|cut -d'/' -f1)" 97 | local package="$(echo ${absolute_path}|cut -d'/' -f2)" 98 | local filename="$(echo ${absolute_path}|cut -d'/' -f3)" 99 | local packagename="${filename%.*}" 100 | local full_path="${REPOTREE}/${category}/${package}" 101 | local full_path_ebuild="${REPOTREE}/${category}/${package}/${filename}" 102 | local maintainer="$(get_main_min "${category}/${package}")" 103 | 104 | 105 | local found=false 106 | local obsolete_dep=() 107 | local dependencies=( $(grep DEPEND /${REPOTREE}/metadata/md5-cache/${category}/${packagename}|grep -oE "[a-zA-Z0-9-]{2,30}/[+a-zA-Z_0-9-]{2,80}"|sed 's/-[0-9].*//g'|sort -u) ) 108 | 109 | for dep in ${dependencies[@]}; do 110 | if $(grep ${dep} ${full_path_ebuild} >/dev/null 2>&1); then 111 | if ! [ -e "${REPOTREE}/${dep}" ]; then 112 | # provide gitage if git is available 113 | if ${ENABLE_GIT}; then 114 | local deadage="$(get_age_last "${dep}")" 115 | if [ -n "${deadage}" ]; then 116 | dep="${dep}(${deadage})" 117 | fi 118 | fi 119 | obsolete_dep+=( "${dep}" ) 120 | found=true 121 | fi 122 | fi 123 | done 124 | 125 | output() { 126 | local id=${1} 127 | if ${FILERESULTS}; then 128 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 129 | else 130 | output_format ${id} 131 | fi 132 | } 133 | 134 | if ${found} && [ "${category}" = "virtual" ]; then 135 | if [ $(expr ${#dependencies[@]}) -eq 1 ] && [ $(grep ${dependencies[0]} ${full_path_ebuild} | wc -l) -gt 1 ]; then 136 | continue 137 | else 138 | if [ $(expr ${#dependencies[@]} - ${#obsolete_dep[@]}) -le 1 ]; then 139 | output 1 140 | fi 141 | fi 142 | fi 143 | 144 | 145 | if ${found}; then 146 | output 0 147 | fi 148 | } 149 | 150 | find_func(){ 151 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 152 | -type f -name "*.ebuild" -exec egrep -l "DEPEND" {} \; | parallel main {} 153 | } 154 | 155 | gen_results() { 156 | if ${FILERESULTS}; then 157 | gen_descriptions 158 | sort_result_v2 2 159 | 160 | for file in $(cat ${RUNNING_CHECKS[0]}/full.txt); do 161 | for fp in $(echo ${file}|cut -d'|' -f4|tr ':' ' '); do 162 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-filter/$(echo ${fp}|tr '/' '_') 163 | echo ${file} >> ${RUNNING_CHECKS[0]}/sort-by-filter/$(echo ${fp}|tr '/' '_')/full.txt 164 | done 165 | done 166 | 167 | gen_sort_main_v3 168 | gen_sort_pak_v3 169 | 170 | copy_checks ${SCRIPT_TYPE} 171 | fi 172 | } 173 | 174 | array_names 175 | # switch to the REPOTREE dir 176 | cd ${REPOTREE} 177 | # export important variables 178 | export WORKDIR 179 | export -f main array_names output_format 180 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 181 | depth_set_v2 ${1} 182 | ${FILERESULTS} && rm -rf ${WORKDIR} 183 | -------------------------------------------------------------------------------- /deprecated/dupuse.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: dupuse.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 11/02/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # find duplicate use flag descriptions 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export REPOTREE=/usr/portage/ 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/dupuse/" 30 | 31 | # get dirpath and load funcs.sh 32 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 33 | if [ -e ${realdir}/_funcs.sh ]; then 34 | source ${realdir}/_funcs.sh 35 | else 36 | echo "Missing _funcs.sh" 37 | exit 1 38 | fi 39 | 40 | # 41 | ### IMPORTANT SETTINGS START ### 42 | # 43 | ${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 44 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 45 | #${ENABLE_GIT} || exit 0 # only works with git tree 46 | 47 | SCRIPT_TYPE="checks" 48 | WORKDIR="/tmp/dupuse-${RANDOM}" 49 | 50 | array_names(){ 51 | RUNNING_CHECKS=( 52 | "${WORKDIR}/metadata_duplicate_useflag_description" #Index 0 53 | ) 54 | } 55 | output_format(){ 56 | index=( 57 | "${category}/${package}${DL}${dupuse::-1}${DL}${maintainer}" 58 | ) 59 | echo "${index[$1]}" 60 | } 61 | data_descriptions(){ 62 | read -r -d '' info_index0 <<- EOM 63 | Lists packages which define use flags locally in metadata.xml, which already exists as 64 | a global use flag. 65 | 66 | Data Format ( dev-libs/foo|gtk[:X:qt:zlib]|dev@gentoo.org:loper@foo.de ): 67 | dev-libs/foo package category/name 68 | gtk[:X:qt:zlib] list of USE flags which already exists as a global flag. 69 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 70 | EOM 71 | description=( "${info_index0}" ) 72 | echo "${description[$1]}" 73 | } 74 | # 75 | ### IMPORTANT SETTINGS END ### 76 | # 77 | 78 | main() { 79 | array_names 80 | local absolute_path=${1} 81 | local category="$(echo ${absolute_path}|cut -d'/' -f1)" 82 | local package="$(echo ${absolute_path}|cut -d'/' -f2)" 83 | local maintainer="$(get_main_min "${category}/${package}")" 84 | 85 | output() { 86 | local id=${1} 87 | if ${FILERESULTS}; then 88 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 89 | else 90 | output_format ${id} 91 | fi 92 | } 93 | 94 | localuses="$(grep "flag name" ${absolute_path} | cut -d'"' -f2)" 95 | 96 | if [ -n "${localuses}" ]; then 97 | for use in ${localuses}; do 98 | if $(tail -n+6 ${REPOTREE}/profiles/use.desc|cut -d'-' -f1|grep "\<${use}\>" > /dev/null); then 99 | dupuse="${use}:${dupuse}" 100 | fi 101 | done 102 | fi 103 | 104 | if [ -n "${dupuse}" ]; then 105 | output 0 106 | fi 107 | } 108 | 109 | find_func(){ 110 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 111 | -type f -name "*.xml" -print | parallel main {} 112 | } 113 | 114 | gen_results() { 115 | if ${FILERESULTS}; then 116 | gen_descriptions 117 | sort_result_v2 118 | 119 | for file in $(cat ${RUNNING_CHECKS[0]}/full.txt); do 120 | for fp in $(echo ${file}|cut -d'|' -f2|tr ':' ' '); do 121 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-filter/${fp} 122 | echo ${file} >> ${RUNNING_CHECKS[0]}/sort-by-filter/${fp}/full.txt 123 | done 124 | done 125 | 126 | gen_sort_main_v3 127 | gen_sort_pak_v3 128 | 129 | copy_checks ${SCRIPT_TYPE} 130 | fi 131 | } 132 | 133 | array_names 134 | # switch to the REPOTREE dir 135 | cd ${REPOTREE} 136 | # export important variables 137 | export WORKDIR 138 | export -f main array_names output_format 139 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 140 | # set the search depth 141 | depth_set_v2 ${1} 142 | # cleanup 143 | ${FILERESULTS} && rm -rf ${WORKDIR} 144 | -------------------------------------------------------------------------------- /deprecated/eapichecks.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: eapichecks.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 20/11/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # This scripts checks eapi usage. it looks for ebuils with old eapi 25 | # and checks if there is a revision/version bump with a newer eapi 26 | 27 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/eapichecks/" 30 | #export REPOTREE="/usr/portage/" 31 | 32 | # get dirpath and load funcs.sh 33 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 34 | if [ -e ${realdir}/_funcs.sh ]; then 35 | source ${realdir}/_funcs.sh 36 | else 37 | echo "Missing _funcs.sh" 38 | exit 1 39 | fi 40 | 41 | # 42 | ### IMPORTANT SETTINGS START ### 43 | # 44 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 45 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 46 | #${ENABLE_GIT} || exit 0 # only works with git tree 47 | 48 | SCRIPT_TYPE="stats" 49 | WORKDIR="/tmp/eapichecks-${RANDOM}" 50 | 51 | array_names(){ 52 | RUNNING_CHECKS=( 53 | "${WORKDIR}/ebuild_cleanup_candidates" #Index 0 54 | "${WORKDIR}/ebuild_stable_candidates" #Index 1 55 | "${WORKDIR}/ebuild_obsolete_eapi" #Index 2 56 | ) 57 | } 58 | output_format(){ 59 | index=( 60 | "${ebuild_eapi}\ 61 | ${DL}${old_ebuild_created}${DL}${old_ebuild_last_modified}\ 62 | ${DL}${eapi_found_ebuild}\ 63 | ${DL}${new_ebuild_created}${DL}${new_ebuild_last_modified}\ 64 | ${DL}${package_bugs}${DL}${category}/${package}\ 65 | ${DL}${org_name}${DL}${name}-r${i}${DL}${maintainer}" 66 | "${ebuild_eapi}\ 67 | ${DL}${old_ebuild_created}${DL}${old_ebuild_last_modified}\ 68 | ${DL}${eapi_found_ebuild}\ 69 | ${DL}${new_ebuild_created}${DL}${new_ebuild_last_modified}\ 70 | ${DL}${package_bugs}${DL}${category}/${package}\ 71 | ${DL}${org_name}${DL}${name}-r${i}${DL}${maintainer}" 72 | "${ebuild_eapi}\ 73 | ${DL}__C2__${DL}__C3__${DL}${eapilist}${DL}${eapi_fileage}\ 74 | ${DL}${category}/${package}${DL}${org_name}${DL}${maintainer}" 75 | ) 76 | echo "${index[$1]}" 77 | } 78 | data_descriptions(){ 79 | read -r -d '' info_default0 <<- EOM 80 | Data Format ( 6|01865|01311|6|00649|-----|+|dev-libs/foo|foo-1.12-r1|foo-1.12-r2|dev@gentoo.org:loper@foo.de ): 81 | 6|01865|01311| EAPI Version (older) | days since created | days since last modified 82 | 6|00649|-----| EAPI Version (newer) | days since created | days since last modified 83 | + indicates if bugs are open to this package (+=yes, -=no) 84 | dev-libs/foo package category/name 85 | foo-1.12-r1.ebuild full filename (older) 86 | foo-1.12-r2.ebuild full filename (newer) 87 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 88 | EOM 89 | 90 | read -r -d '' info_index0 <<- EOM 91 | This script searches if there is a newer revision (-rX) available. In case a newer revision is found KEYWORDS are 92 | gonna be checked as well. If both keywords are the same for both ebuilds the older one is considered as a removal 93 | candidate. 94 | 95 | ${info_default0} 96 | EOM 97 | read -r -d '' info_index1 <<- EOM 98 | This script searches if there is a newer revision (-rX) available. In case a newer revision is found KEYWORDS are 99 | gonna be checked as well. In case keywords differ, the newer ebuild is considered as a candidate for 100 | stablization. 101 | 102 | ${info_default0} 103 | EOM 104 | read -r -d '' info_index2 <<- EOM 105 | This scirpt lists every ebuild with a EAPI <6. It list all other available ebuild EAPIs too which should make it easier 106 | to find packages which can be removed or need some attention. 107 | 108 | Data Format ( 5|-|000|0:0:0:0:0:1:0:0|02016|dev-libs/foo|foo-1.12-r2|dev@gentoo.org:loper@foo.de ): 109 | 5 EAPI Version 110 | - indicates if bugs are open to this package (+=yes, -=no) 111 | 000 number of bugs found on bugs.gentoo.org for this package 112 | 0:0:0:0:0:1:0:0 list of available EAPIs each number represents a EAPI version (first EAPI0, last EAPI7) 113 | 02016 days since created 114 | dev-libs/foo package category/name 115 | foo-1.12-r2.ebuild full filename 116 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 117 | EOM 118 | description=( "${info_index0}" "${info_index1}" "${info_index2}" ) 119 | echo "${description[$1]}" 120 | } 121 | # 122 | ### IMPORTANT SETTINGS END ### 123 | # 124 | 125 | main() { 126 | array_names 127 | local full_package=${1} 128 | local category="$(echo ${full_package}|cut -d'/' -f1)" 129 | local package="$(echo ${full_package}|cut -d'/' -f2)" 130 | local filename="$(echo ${full_package}|cut -d'/' -f3)" 131 | local name="${filename%.*}" 132 | local ebuild_eapi="$(get_eapi ${full_package})" 133 | 134 | local date_today="$(date '+%s' -d today)" 135 | local package_path="/${REPOTREE}/${category}/${package}" 136 | 137 | local maintainer="$(get_main_min "${category}/${package}")" 138 | 139 | if [ "${name: -3}" = "-r${name: -1}" ]; then 140 | start=$(expr ${name: -1} + 1) 141 | org_name=${name} 142 | name=${name::-3} 143 | else 144 | start=1 145 | org_name=${name} 146 | fi 147 | 148 | output(){ 149 | local checkid=${1} 150 | if ${FILERESULTS}; then 151 | output_format ${checkid} >> ${RUNNING_CHECKS[${checkid}]}/full.txt 152 | else 153 | echo "${RUNNING_CHECKS[${checkid}]##*/}${DL}$(output_format ${checkid})" 154 | fi 155 | } 156 | 157 | local i 158 | 159 | # check for maximal 9 reversion 160 | for i in $(seq $start 9); do 161 | if [ -e ${package_path}/${name}-r${i}.ebuild ]; then 162 | local found_ebuild="${category}/${name}-r${i}" 163 | local eapi_found_ebuild="$(get_eapi ${package_path}/${name}-r${i}.ebuild)" 164 | 165 | if [ ${eapi_found_ebuild} -ge 6 ] && ! $(check_mask ${found_ebuild}); then 166 | 167 | local old_ebuild="${category}/${package}/${org_name}.ebuild" 168 | local new_ebuild="${category}/${package}/${name}-r${i}.ebuild" 169 | 170 | if $(count_keywords "${org_name}" "${name}-r${i}" ${category} ${package}); then 171 | 172 | local old_ebuild_created="$(get_git_age ${old_ebuild} "ct" "A" "first")" 173 | local old_ebuild_last_modified="$(get_git_age ${old_ebuild} "ct" "M" "last")" 174 | local new_ebuild_created="$(get_git_age ${new_ebuild} "ct" "A" "first")" 175 | local new_ebuild_last_modified="$(get_git_age ${new_ebuild} "ct" "M" "last")" 176 | local package_bugs="__C6__" 177 | 178 | if $(compare_keywords "${org_name}" "${name}-r${i}" ${category} ${package}); then 179 | output 0 180 | else 181 | output 1 182 | fi 183 | fi 184 | break 2 185 | fi 186 | fi 187 | done 188 | # everything below EAPI=6 is considered obsolete 189 | if [ ${ebuild_eapi} -lt 6 ]; then 190 | local eapilist="$(get_eapi_list ${package_path})" 191 | local eapi_fileage="$(get_git_age ${full_package} "ct" "A" "first")" 192 | output 2 193 | fi 194 | } 195 | 196 | find_func(){ 197 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 198 | -type f -name "*.ebuild" -print | parallel main {} 199 | } 200 | 201 | upd_results(){ 202 | if ${FILERESULTS}; then 203 | for rcid in $(seq 0 1); do 204 | if [ -e ${RUNNING_CHECKS[${rcid}]}/full.txt ]; then 205 | #get time diff since last run 206 | local indexfile="${RESULTSDIR}/${SCRIPT_TYPE}/${RUNNING_CHECKS[${rcid}]/${WORKDIR}/}/index.html" 207 | local time_diff="$(get_time_diff "${indexfile}")" 208 | for id in $(cat "${RUNNING_CHECKS[${rcid}]}/full.txt"); do 209 | 210 | local old_ebuild_created="$(date_update "$(echo ${id}|cut -d'|' -f2)" "${time_diff}")" 211 | local old_ebuild_last_modified="$(date_update "$(echo ${id}|cut -d'|' -f3)" "${time_diff}")" 212 | local new_ebuild_created="$(date_update "$(echo ${id}|cut -d'|' -f5)" "${time_diff}")" 213 | local new_ebuild_last_modified="$(date_update "$(echo ${id}|cut -d'|' -f6)" "${time_diff}")" 214 | #local package_bugs="$(echo ${id}|cut -d'|' -f7)" 215 | local package="$(echo ${id}|cut -d'|' -f8)" 216 | $(get_bugs_bool "${package}") && local package_bugs="+" || local package_bugs="-" 217 | 218 | local new_id=$( 219 | echo "${id}" | gawk -F'|' '{$2=v2; $3=v3; $5=v5; $6=v6; $7=v7}1' \ 220 | v2="${old_ebuild_created}" \ 221 | v3="${old_ebuild_last_modified}" \ 222 | v5="${new_ebuild_created}" \ 223 | v6="${new_ebuild_last_modified}" \ 224 | v7="${package_bugs}" OFS='|' 225 | ) 226 | 227 | sed -i "s ${id} ${new_id} g" ${RUNNING_CHECKS[${rcid}]}/full.txt 228 | 229 | done 230 | fi 231 | done 232 | # obsolet eapi 233 | if [ -e ${RUNNING_CHECKS[2]}/full.txt ]; then 234 | #get time diff since last run 235 | local indexfile="${RESULTSDIR}/${SCRIPT_TYPE}/${RUNNING_CHECKS[2]/${WORKDIR}/}/index.html" 236 | local time_diff="$(get_time_diff "${indexfile}")" 237 | for id in $(cat "${RUNNING_CHECKS[2]}/full.txt"); do 238 | 239 | local package="$(echo ${id}|cut -d'|' -f6)" 240 | local old_ebuild_created="$(date_update "$(echo ${id}|cut -d'|' -f5)" "${time_diff}")" 241 | $(get_bugs_bool "${package}") && local package_bugs="+" || local package_bugs="-" 242 | local package_bugs_count="$(get_bugs_count ${package})" 243 | 244 | local new_id=$( 245 | echo "${id}" | gawk -F'|' '{$2=v2; $3=v3; $5=v5}1' \ 246 | v2="${package_bugs}" \ 247 | v3="${package_bugs_count}" \ 248 | v5="${old_ebuild_created}" OFS='|' 249 | ) 250 | 251 | sed -i "s ${id} ${new_id} g" ${RUNNING_CHECKS[2]}/full.txt 252 | done 253 | fi 254 | fi 255 | } 256 | 257 | gen_results(){ 258 | if ${FILERESULTS}; then 259 | gen_descriptions 260 | # update results with actual git age/bugs information 261 | upd_results 262 | # sort the results 263 | sort_result_v4 "1,1 -k8,8" 0 264 | sort_result_v4 "1,1 -k8,8" 1 265 | sort_result_v4 "1,1 -k6,6" 2 266 | # create maintainer/package listings 267 | gen_sort_main_v3 268 | gen_sort_pak_v3 269 | # copy results to sitedir 270 | copy_checks ${SCRIPT_TYPE} 271 | fi 272 | } 273 | 274 | array_names 275 | cd ${REPOTREE} 276 | export WORKDIR 277 | export -f main array_names output_format 278 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 279 | depth_set_v2 ${1} 280 | # cleanup tmp files 281 | ${FILERESULTS} && rm -rf ${WORKDIR} 282 | -------------------------------------------------------------------------------- /deprecated/eapistats.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: eapistats.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 07/09/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simple script for generating EAPI statistics 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export FILERESULTS=true 28 | #export RESULTSDIR="${HOME}/eapistats/" 29 | #export REPOTREE=/usr/portage/ 30 | 31 | # get dirpath and load funcs.sh 32 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 33 | if [ -e ${realdir}/_funcs.sh ]; then 34 | source ${realdir}/_funcs.sh 35 | else 36 | echo "Missing _funcs.sh" 37 | exit 1 38 | fi 39 | 40 | # 41 | ### IMPORTANT SETTINGS START ### 42 | # 43 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 44 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 45 | #${ENABLE_GIT} || exit 0 # only works with git tree 46 | 47 | WORKDIR="/tmp/eapistats-${RANDOM}" 48 | SCRIPT_TYPE="stats" 49 | 50 | array_names(){ 51 | RUNNING_CHECKS=( 52 | "${WORKDIR}/ebuild_eapi_statistics" #Index 0 53 | "${WORKDIR}/ebuild_eapi_live_statistics" #Index 1 54 | ) 55 | } 56 | output_format(){ 57 | index=( 58 | "${eapi}${DL}${category}/${package}${DL}${filename}${DL}${maintainer}${openbugs}" 59 | "${eapi}${DL}${category}/${package}${DL}${filename}${DL}${maintainer}${openbugs}" 60 | ) 61 | echo "${index[$1]}" 62 | } 63 | data_descriptions(){ 64 | read -r -d '' info_default0 <<- EOM 65 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|dev@gentoo.org:loper@foo.de ): 66 | 7 EAPI Version 67 | dev-libs/foo package category/name 68 | foo-1.12-r2.ebuild full filename 69 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 70 | EOM 71 | read -r -d '' info_index0 <<- EOM 72 | A simple list of all ebuilds with it's corresponding EAPI Version. Also includes all maintainers to the package 73 | EAPI Statistics 74 | 75 | ${info_default0} 76 | EOM 77 | read -r -d '' info_index1 <<- EOM 78 | A simple list of all live ebuilds and it's corresponding EAPI Version and maintainer(s). 79 | 80 | ${info_default0} 81 | EOM 82 | description=( "${info_index0}" "${info_index1}" ) 83 | echo "${description[$1]}" 84 | } 85 | # 86 | ### IMPORTANT SETTINGS END ### 87 | # 88 | 89 | main() { 90 | array_names 91 | local full_package=${1} 92 | local category="$(echo ${full_package}|cut -d'/' -f1)" 93 | local package="$(echo ${full_package}|cut -d'/' -f2)" 94 | local filename="$(echo ${full_package}|cut -d'/' -f3)" 95 | local packagename="${filename%.*}" 96 | local fileversion="${packagename/${package}-/}" 97 | local maintainer="$(get_main_min "${category}/${package}")" 98 | local openbugs="$(get_bugs ${category}/${package})" 99 | if ! [ -z "${openbugs}" ]; then 100 | openbugs="${DL}${openbugs}" 101 | fi 102 | local eapi="$(get_eapi ${full_package})" 103 | 104 | output() { 105 | local id=${1} 106 | if ${FILERESULTS}; then 107 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 108 | else 109 | echo "${RUNNING_CHECKS[${checkid}]##*/}${DL}$(output_format ${id})" 110 | fi 111 | } 112 | 113 | if $(echo ${fileversion}|grep -q 9999); then 114 | output 1 115 | fi 116 | 117 | output 0 118 | } 119 | 120 | 121 | find_func(){ 122 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 123 | -type f -name "*.ebuild" -print | parallel main {} 124 | } 125 | 126 | gen_results() { 127 | if ${FILERESULTS}; then 128 | gen_descriptions 129 | sort_result_v2 2 130 | # filter after EAPI 131 | for eapi in $(cut -c-1 ${RUNNING_CHECKS[0]}/full.txt|sort -u); do 132 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-eapi/EAPI${eapi} 133 | grep ^${eapi}${DL} ${RUNNING_CHECKS[0]}/full.txt > ${RUNNING_CHECKS[0]}/sort-by-eapi/EAPI${eapi}/full.txt 134 | 135 | gen_sort_main_v3 ${RUNNING_CHECKS[0]}/sort-by-eapi/EAPI${eapi}/full.txt 136 | gen_sort_pak_v3 ${RUNNING_CHECKS[0]}/sort-by-eapi/EAPI${eapi}/full.txt 137 | done 138 | gen_sort_main_v3 ${RUNNING_CHECKS[0]} 139 | gen_sort_pak_v3 ${RUNNING_CHECKS[0]} 140 | 141 | for eapi in $(cut -c-1 ${RUNNING_CHECKS[1]}/full.txt|sort -u); do 142 | mkdir -p ${RUNNING_CHECKS[1]}/sort-by-eapi/EAPI${eapi} 143 | grep ^${eapi}${DL} ${RUNNING_CHECKS[1]}/full.txt > ${RUNNING_CHECKS[1]}/sort-by-eapi/EAPI${eapi}/full.txt 144 | 145 | gen_sort_main_v3 ${RUNNING_CHECKS[1]}/sort-by-eapi/EAPI${eapi}/full.txt 146 | gen_sort_pak_v3 ${RUNNING_CHECKS[1]}/sort-by-eapi/EAPI${eapi}/full.txt 147 | done 148 | gen_sort_main_v3 ${RUNNING_CHECKS[1]} 149 | gen_sort_pak_v3 ${RUNNING_CHECKS[1]} 150 | 151 | copy_checks ${SCRIPT_TYPE} 152 | rm -rf ${WORKDIR} 153 | fi 154 | } 155 | 156 | array_names 157 | cd ${REPOTREE} 158 | export -f main array_names output_format 159 | export WORKDIR 160 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 161 | depth_set_v2 ${1} 162 | ${FILERESULTS} && rm -rf ${WORKDIR} 163 | -------------------------------------------------------------------------------- /deprecated/eclassstats.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: eclassstats.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 05/05/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # lists eclass uses of the tree 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export REPOTREE=/usr/portage/ 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/eclassstats/" 30 | 31 | # get dirpath and load funcs.sh 32 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 33 | if [ -e ${realdir}/_funcs.sh ]; then 34 | source ${realdir}/_funcs.sh 35 | else 36 | echo "Missing _funcs.sh" 37 | exit 1 38 | fi 39 | 40 | # 41 | ### IMPORTANT SETTINGS START ### 42 | # 43 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 44 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 45 | #${ENABLE_GIT} || exit 0 # only works with git tree 46 | 47 | SCRIPT_TYPE="stats" 48 | WORKDIR="/tmp/eclassstats-${RANDOM}" 49 | 50 | array_names(){ 51 | RUNNING_CHECKS=( 52 | "${WORKDIR}/ebuild_eclass_statistics" #Index 0 53 | ) 54 | } 55 | output_format(){ 56 | index=( 57 | "$(get_eapi ${full_path_ebuild})${DL}${category}/${package}${DL}${filename}${DL}${ebuild_eclass_file}${DL}${maintainer}" 58 | ) 59 | echo "${index[$1]}" 60 | } 61 | data_descriptions(){ 62 | read -r -d '' info_index0 <<- EOM 63 | Lists the eclasses used by every ebuild. 64 | Not including packages which don't inherit anything. Also not included are eclasses inherited by other eclasses. 65 | 66 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|eutils:elisp|dev@gentoo.org:loper@foo.de ): 67 | 7 EAPI Version 68 | dev-libs/foo package category/name 69 | foo-1.12-r2.ebuild full filename 70 | eutils:elisp eclasses which the ebuilds inherit (implicit inherit not included) 71 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 72 | EOM 73 | description=( "${info_index0}" ) 74 | echo "${description[$1]}" 75 | } 76 | # 77 | ### IMPORTANT SETTINGS END ### 78 | # 79 | 80 | main() { 81 | array_names 82 | local relative_path=${1} 83 | local category="$(echo ${relative_path}|cut -d'/' -f1)" 84 | local package="$(echo ${relative_path}|cut -d'/' -f2)" 85 | local filename="$(echo ${relative_path}|cut -d'/' -f3)" 86 | local packagename="${filename%.*}" 87 | local full_path="${REPOTREE}/${category}/${package}" 88 | local full_path_ebuild="${REPOTREE}/${category}/${package}/${filename}" 89 | local maintainer="$(get_main_min "${category}/${package}")" 90 | local full_md5path="${REPOTREE}/metadata/md5-cache/${category}/${packagename}" 91 | 92 | ebuild_eclass_file=$(get_eclasses_file ${full_md5path} ${full_path_ebuild}) 93 | #ebuild_eclass_real=$(get_eclasses_real ${full_md5path}) 94 | 95 | output() { 96 | local id=${1} 97 | if ${FILERESULTS}; then 98 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 99 | else 100 | output_format ${id} 101 | fi 102 | } 103 | 104 | if [ -n "${ebuild_eclass_file}" ]; then 105 | output 0 106 | fi 107 | } 108 | 109 | 110 | find_func(){ 111 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 112 | -type f -name "*.ebuild" -exec egrep -l "inherit" {} \; | parallel main {} 113 | } 114 | 115 | gen_results(){ 116 | if ${FILERESULTS}; then 117 | gen_descriptions 118 | sort_result_v2 2 119 | 120 | for file in $(cat ${RUNNING_CHECKS[0]}/full.txt); do 121 | for ec in $(echo ${file}|cut -d'|' -f4|tr ':' ' '); do 122 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-filter/${ec}.eclass 123 | echo ${file} >> ${RUNNING_CHECKS[0]}/sort-by-filter/${ec}.eclass/full.txt 124 | done 125 | done 126 | 127 | for ecd in $(ls ${RUNNING_CHECKS[0]}/sort-by-filter/); do 128 | gen_sort_main_v3 ${RUNNING_CHECKS[0]}/sort-by-filter/${ecd} 129 | gen_sort_pak_v3 ${RUNNING_CHECKS[0]}/sort-by-filter/${ecd} 130 | done 131 | 132 | gen_sort_main_v3 133 | gen_sort_pak_v3 134 | 135 | copy_checks ${SCRIPT_TYPE} 136 | fi 137 | } 138 | 139 | array_names 140 | # create a list and create corresponding folders and files of all available 141 | # eclasses before running the check. 142 | # this way we also see eclasses without customers 143 | if ${FILERESULTS}; then 144 | if ${TREE_IS_MASTER}; then 145 | eclass_list=( $(ls ${REPOTREE}/eclass/*.eclass) ) 146 | eclass_list=( ${eclass_list[@]##*/} ) 147 | for ecl in ${eclass_list[@]}; do 148 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-filter/${ecl} 149 | touch ${RUNNING_CHECKS[0]}/sort-by-filter/${ecl}/full.txt 150 | done 151 | fi 152 | fi 153 | # switch to the REPOTREE dir 154 | cd ${REPOTREE} 155 | # export important variables 156 | export WORKDIR 157 | export -f main array_names output_format 158 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 159 | # set the search depth 160 | depth_set_v2 ${1} 161 | ${FILERESULTS} && rm -rf ${WORKDIR} 162 | -------------------------------------------------------------------------------- /deprecated/eclassusage.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: eclassusage.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 26/05/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # checks correct usage of eclasses 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export REPOTREE=/usr/portage/ 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/eclassusage/" 30 | 31 | # get dirpath and load funcs.sh 32 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 33 | if [ -e ${realdir}/_funcs.sh ]; then 34 | source ${realdir}/_funcs.sh 35 | else 36 | echo "Missing _funcs.sh" 37 | exit 1 38 | fi 39 | 40 | # 41 | ### IMPORTANT SETTINGS START ### 42 | # 43 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 44 | ${ENABLE_MD5} || exit 0 # only works with md5 cache 45 | #${ENABLE_GIT} || exit 0 # only works with git tree 46 | 47 | # in order to make this script available for overlays we need to know the were 48 | # the main gentoo tree is because the eclass informations (functions available) 49 | # are generated with each run. 50 | TREEGENTOO="/tmp/repos/gentoo/" 51 | if ! [ -d "${TREEGENTOO}/eclass" ]; then 52 | if [ -d "/usr/portage/eclass" ]; then 53 | TREEGENTOO=/usr/portage/ 54 | else 55 | echo "ERR: gentoo tree not available" 56 | exit 1 57 | fi 58 | fi 59 | SCRIPT_TYPE="checks" 60 | WORKDIR="/tmp/eclassusage-${RANDOM}" 61 | 62 | array_names(){ 63 | RUNNING_CHECKS=( 64 | "${WORKDIR}/ebuild_missing_eclasses" #Index 0 65 | "${WORKDIR}/ebuild_unused_eclasses" #Index 1 66 | "${WORKDIR}/ebuild_missing_eclasses_fatal" #Index 2 67 | ) 68 | } 69 | output_format(){ 70 | index=( 71 | "${ebuild_eapi}${DL}${category}/${package}${DL}${filename}${DL}$(echo ${missing_ecl[@]}|tr ' ' ':')${DL}${maintainer}" 72 | "${ebuild_eapi}${DL}${category}/${package}${DL}${filename}${DL}$(echo ${obsol_ecl[@]}|tr ' ' ':')${DL}${maintainer}" 73 | "${ebuild_eapi}${DL}${category}/${package}${DL}${filename}${DL}$(echo ${missing_ecl_fatal[@]}|tr ' ' ':')${DL}${maintainer}" 74 | ) 75 | echo "${index[$1]}" 76 | } 77 | data_descriptions(){ 78 | read -r -d '' info_index0 <<- EOM 79 | Lists ebuilds which use functions of eclasses which are not directly inherited. (usually inherited implicit) 80 | Following eclasses are checked: 81 | ltprune, eutils, estack, preserve-libs, vcs-clean, epatch, 82 | desktop, versionator, user, eapi7-ver, flag-o-matic, libtool, pam, udev, xdg-utils 83 | 84 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|user(enewuser):udev(edev_get)|dev@gentoo.org:loper@foo.de ): 85 | 7 EAPI Version 86 | dev-libs/foo package category/name 87 | foo-1.12-r2.ebuild full filename 88 | user(enewuser):udev(edev_get) eclasse(s) and function name the ebuild uses but not inherits, seperated by ':' 89 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 90 | EOM 91 | read -r -d '' info_index1 <<- EOM 92 | Lists ebuilds which inherit eclasses but doesn't use their features. 93 | Following eclasses are checked: 94 | ltprune, eutils, estack, preserve-libs, vcs-clean, epatch, desktop, 95 | versionator, user, eapi7-ver, flag-o-matic, libtool, pam, udev, xdg-utils 96 | 97 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|user:udev|dev@gentoo.org:loper@foo.de ): 98 | 7 EAPI Version 99 | dev-libs/foo package category/name 100 | foo-1.12-r2.ebuild full filename 101 | user:udev eclasse(s) the ebuild inherits but not uses, seperated by ':' 102 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 103 | EOM 104 | read -r -d '' info_index2 <<- EOM 105 | Lists ebuilds which use functions of eclasses which are not directly or indirectly (implicit) inherited. 106 | This would be an fatal error since the ebuild would use a feature which it doesn't know. 107 | Following eclasses are checked: 108 | ltprune, eutils, estack, preserve-libs, vcs-clean, epatch, 109 | desktop, versionator, user, eapi7-ver, flag-o-matic, libtool, pam, udev, xdg-utils 110 | 111 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|user(enewuser):udev(edev_get)|dev@gentoo.org:loper@foo.de ): 112 | 7 EAPI Version 113 | dev-libs/foo package category/name 114 | foo-1.12-r2.ebuild full filename 115 | user(enewuser):udev(edev_get) eclasse(s) and function name the ebuild uses but not inherits, seperated by ':' 116 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 117 | EOM 118 | description=( "${info_index0}" "${info_index1}" "${info_index2}" ) 119 | echo "${description[$1]}" 120 | } 121 | # 122 | ### IMPORTANT SETTINGS END ### 123 | # 124 | 125 | gen_eclass_funcs(){ 126 | # a list of eclass which we going to check 127 | local etc=( optfeature wrapper edos2unix ltprune l10n eutils estack preserve-libs \ 128 | vcs-clean epatch desktop versionator user user-info flag-o-matic xdg-utils \ 129 | libtool udev eapi7-ver pam ssl-cert toolchain-funcs ) 130 | 131 | local eclasses_with_funcs=( ) 132 | 133 | local i x 134 | for i in ${etc[@]}; do 135 | # check if the eclass exports functions (these eclass cannot be checked) 136 | if ! $(grep -q "EXPORT_FUNCTIONS" /${TREEGENTOO}/eclass/${i}.eclass); then 137 | # get all functions of the eclass 138 | local efuncs="$(sed -n 's/# @FUNCTION: //p' "/${TREEGENTOO}/eclass/${i}.eclass" | sed ':a;N;$!ba;s/\n/ /g')" 139 | local f=( ) 140 | # only continue if we found functions 141 | if [ -n "${efuncs}" ]; then 142 | for x in ${efuncs}; do 143 | if ! $(grep "@FUNCTION: ${x}" -A3 -m1 /${TREEGENTOO}/eclass/${i}.eclass |grep -q "@INTERNAL"); then 144 | case "${i}" in 145 | eutils) 146 | case "${x}" in 147 | usex) continue ;; #available from EAPI6 148 | in_iuse) continue ;; #available from EAPI6 149 | eqawarn) continue ;; #ignore for now 150 | einstalldocs) continue ;; #available from EAPI6 151 | *) f+=( "${x}" ) 152 | esac ;; 153 | toolchain-funcs) 154 | case "${x}" in 155 | gen_usr_ldscript) continue ;; #deprecated, use it from usr-ldscriplt.eclass 156 | *) f+=( "${x}" ) 157 | esac ;; 158 | *) 159 | f+=( "${x}" ) ;; 160 | esac 161 | fi 162 | done 163 | eclasses_with_funcs+=( "$(echo ${i##*/}|cut -d '.' -f1);$(echo ${f[@]}|tr ' ' ':')" ) 164 | fi 165 | else 166 | echo "ERR: ${i} exports functions" 167 | fi 168 | done 169 | 170 | ECLASSES="$(echo ${eclasses_with_funcs[@]})" 171 | export ECLASSES 172 | } 173 | 174 | main() { 175 | array_names 176 | 177 | local relative_path=${1} 178 | local category="$(echo ${relative_path}|cut -d'/' -f1)" 179 | local package="$(echo ${relative_path}|cut -d'/' -f2)" 180 | local filename="$(echo ${relative_path}|cut -d'/' -f3)" 181 | local packagename="${filename%.*}" 182 | local full_path="${REPOTREE}/${category}/${package}" 183 | local maintainer="$(get_main_min "${category}/${package}")" 184 | local ebuild_eapi="$(get_eapi ${relative_path})" 185 | local full_md5path="${REPOTREE}/metadata/md5-cache/${category}/${packagename}" 186 | 187 | output(){ 188 | local checkid=${1} 189 | if ${FILERESULTS}; then 190 | output_format ${checkid} >> ${RUNNING_CHECKS[${checkid}]}/full.txt 191 | else 192 | echo "${RUNNING_CHECKS[${checkid}]##*/}${DL}$(output_format ${checkid})" 193 | fi 194 | } 195 | 196 | 197 | if [ "${ebuild_eapi}" = "6" ] || [ "${ebuild_eapi}" = "7" ]; then 198 | 199 | local obsol_ecl=( ) 200 | local missing_ecl=( ) 201 | local missing_ecl_fatal=( ) 202 | local func_in_use=( ) 203 | local func_in_use_fatal=( ) 204 | 205 | for echeck in ${ECLASSES[@]}; do 206 | local eclass="$(echo ${echeck}|cut -d';' -f1)" 207 | local eclass_funcs="$(echo ${echeck}|cut -d';' -f2|tr ':' ' ')" 208 | 209 | # don't check for eapi7-ver at EAPI=7 ebuilds 210 | if [ "${eclass}" = "eapi7-ver" ] && [ "${ebuild_eapi}" = "7" ]; then 211 | continue 212 | fi 213 | 214 | # check if ebuild uses ${eclass} 215 | if $(check_eclasses_usage ${relative_path} ${eclass}); then 216 | # check if ebuild uses one of the functions provided by the eclass 217 | local catch=false 218 | for i in ${eclass_funcs}; do 219 | if $(grep -qP "^(?!#).*(?> ${rc}/sort-by-filter/${ec}.eclass/full.txt 273 | done 274 | done 275 | done 276 | 277 | for rc2 in ${RUNNING_CHECKS[@]}; do 278 | for ecf in $(ls ${rc2}/sort-by-filter/); do 279 | gen_sort_main_v3 ${rc2}/sort-by-filter/${ecf} 280 | gen_sort_pak_v3 ${rc2}/sort-by-filter/${ecf} 281 | done 282 | done 283 | 284 | gen_sort_main_v3 285 | gen_sort_pak_v3 286 | 287 | copy_checks ${SCRIPT_TYPE} 288 | fi 289 | } 290 | 291 | array_names 292 | gen_eclass_funcs 293 | # switch to the REPOTREE dir 294 | cd ${REPOTREE} 295 | # export important variables 296 | export WORKDIR 297 | export -f main array_names output_format 298 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 299 | # set the search depth 300 | depth_set_v2 ${1} 301 | # cleanup tmp files 302 | ${FILERESULTS} && rm -rf ${WORKDIR} 303 | -------------------------------------------------------------------------------- /deprecated/patchcheck.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: patchtest 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 07/08/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simple scirpt to find unused scripts directories in the gentoo tree 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export FILERESULTS=true 28 | #export RESULTSDIR="${HOME}/patchcheck/" 29 | #export REPOTREE=/usr/portage/ 30 | 31 | # get dirpath and load funcs.sh 32 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 33 | if [ -e ${realdir}/_funcs.sh ]; then 34 | source ${realdir}/_funcs.sh 35 | else 36 | echo "Missing _funcs.sh" 37 | exit 1 38 | fi 39 | 40 | # 41 | ### IMPORTANT SETTINGS START ### 42 | # 43 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 44 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 45 | #${ENABLE_GIT} || exit 0 # only works with git tree 46 | 47 | if [ -z "${PT_WHITELIST}" ]; then 48 | WFILE="${realdir}/whitelist" 49 | else 50 | WFILE="${realdir}/${PT_WHITELIST}" 51 | fi 52 | 53 | SCRIPT_TYPE="checks" 54 | WORKDIR="/tmp/patchcheck-${RANDOM}/" 55 | 56 | array_names(){ 57 | RUNNING_CHECKS=( 58 | "${WORKDIR}/ebuild_unused_patches_simple" #Index 0 59 | ) 60 | } 61 | output_format(){ 62 | index=( 63 | "${category}/${package_name}${DL}${main}" 64 | ) 65 | echo "${index[$1]}" 66 | } 67 | data_descriptions(){ 68 | read -r -d '' info_index0 <<- EOM 69 | Very limited check to find unused patches, mostly without false positives 70 | 71 | Data Format ( dev-libs/foo|dev@gentoo.org:loper@foo.de ): 72 | dev-libs/foo package category/name 73 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 74 | EOM 75 | description=( "${info_index0}" ) 76 | echo "${description[$1]}" 77 | } 78 | # 79 | ### IMPORTANT SETTINGS END ### 80 | # 81 | 82 | _gen_whitelist(){ 83 | if [ -e ${WFILE} ]; then 84 | source ${WFILE} 85 | for i in ${white_list[@]}; do 86 | whitelist+=("$(echo ${i}|cut -d';' -f1)") 87 | done 88 | else 89 | whitelist=() 90 | fi 91 | # remove duplicates 92 | mapfile -t whitelist < <(printf '%s\n' "${whitelist[@]}"|sort -u) 93 | echo ${whitelist[@]} 94 | } 95 | 96 | main(){ 97 | array_names 98 | local eclasses="apache-module|elisp|vdr-plugin-2|ruby-ng|readme.gentoo|readme.gentoo-r1|java-vm-2|php-ext-source-r3|selinux-policy-2|toolchain-glibc" 99 | local full_package=${1} 100 | local category="$(echo ${full_package}|cut -d'/' -f1)" 101 | local package_name="$(echo ${full_package}|cut -d'/' -f2)" 102 | local fullpath="/${REPOTREE}/${full_package}" 103 | 104 | output() { 105 | local id=${1} 106 | if ${FILERESULTS}; then 107 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 108 | else 109 | output_format ${id} 110 | fi 111 | } 112 | 113 | # check if the patches folder exist 114 | if [ -e ${fullpath}/files ]; then 115 | if ! echo ${whitelist[@]}|grep "${category}/${package_name}" > /dev/null; then 116 | if ! grep -E ".diff|.patch|FILESDIR|${eclasses}" ${fullpath}/*.ebuild >/dev/null; then 117 | main=$(get_main_min "${category}/${package_name}") 118 | output 0 119 | fi 120 | fi 121 | fi 122 | } 123 | 124 | find_func(){ 125 | find ${searchp[@]} -mindepth ${MIND} -maxdepth ${MAXD} \ 126 | -type d -print | parallel main {} 127 | } 128 | 129 | gen_results(){ 130 | if ${FILERESULTS}; then 131 | gen_descriptions 132 | sort_result_v2 133 | gen_sort_main_v3 134 | gen_sort_pak_v3 135 | 136 | copy_checks ${SCRIPT_TYPE} 137 | fi 138 | } 139 | 140 | array_names 141 | cd ${REPOTREE} 142 | export WORKDIR 143 | export whitelist=$(_gen_whitelist) 144 | export -f main array_names output_format 145 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 146 | depth_set_v2 ${1} 147 | ${FILERESULTS} && rm -rf ${WORKDIR} 148 | -------------------------------------------------------------------------------- /deprecated/patchtest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: patchtest 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 01/08/2016 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simple scirpt to find unused patches in the gentoo portage tree 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export FILERESULTS=true 28 | #export RESULTSDIR="${HOME}/patchtest/" 29 | #export REPOTREE=/usr/portage/ 30 | # enabling debug output 31 | #export DEBUG=true 32 | #export DEBUGLEVEL=1 33 | #export DEBUGFILE=/tmp/patchtest.log 34 | 35 | # get dirpath and load funcs.sh 36 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 37 | if [ -e ${realdir}/_funcs.sh ]; then 38 | source ${realdir}/_funcs.sh 39 | else 40 | echo "Missing _funcs.sh" 41 | exit 1 42 | fi 43 | 44 | # 45 | ### IMPORTANT SETTINGS START ### 46 | # 47 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 48 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 49 | #${ENABLE_GIT} || exit 0 # only works with git tree 50 | 51 | if [ -z "${PT_WHITELIST}" ]; then 52 | WFILE="${realdir}/whitelist" 53 | else 54 | WFILE="${realdir}/${PT_WHITELIST}" 55 | fi 56 | 57 | SCRIPT_TYPE="checks" 58 | WORKDIR="/tmp/patchtest-${RANDOM}/" 59 | array_names(){ 60 | RUNNING_CHECKS=( 61 | "${WORKDIR}/ebuild_unused_patches" #Index 0 62 | ) 63 | } 64 | output_format(){ 65 | index=( 66 | "${category}/${package}${DL}${upatch}${DL}${main}" 67 | ) 68 | echo "${index[$1]}" 69 | } 70 | data_descriptions(){ 71 | read -r -d '' info_index0 <<- EOM 72 | Extensive check to find unused pachtes. In order to reduce flase positives it uses a whilelist to exclude them. 73 | 74 | Data Format ( dev-libs/foo|foo-fix-1.12.patch|dev@gentoo.org:loper@foo.de ): 75 | dev-libs/foo package category/name 76 | foo-fix-1.12.patch patch which is not used by any ebuild 77 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 78 | EOM 79 | description=( "${info_index0}" ) 80 | echo "${description[$1]}" 81 | } 82 | # 83 | ### IMPORTANT SETTINGS END ### 84 | # 85 | 86 | main(){ 87 | # output function 88 | output() { 89 | local id=${1} 90 | if ${FILERESULTS}; then 91 | for upatch in "${unused_patches[@]}"; do 92 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 93 | done 94 | else 95 | for upatch in "${unused_patches[@]}"; do 96 | output_format ${id} 97 | done 98 | fi 99 | } 100 | 101 | # main check 102 | check_ebuild(){ 103 | local patchfile=$1 104 | 105 | local pn='${PN}' 106 | local p='${P}' 107 | local pf='${PF}' 108 | local pv='${PV}' 109 | local pvr='${PVR}' 110 | local slot='${SLOT}' 111 | local var_my_pn='${MY_PN}' 112 | local var_my_p='${MY_P}' 113 | local var_my_pv='${MY_PV}' 114 | local var_mod_ver='${MODULE_VERSION}' 115 | local var_dist_ver='${DIST_VERSION}' 116 | local var_x509_ver='${X509_VER}' 117 | local var_hpn_ver='${HPN_VER}' 118 | 119 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 1-check_ebuild: checking: ${patchfile}" | (debug_output) 120 | 121 | for ebuild in ${fullpath}/*.ebuild; do 122 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 1-check_ebuild: looking into: ${ebuild}" | (debug_output) 123 | 124 | # get ebuild detail 125 | local ebuild_full=$(basename ${ebuild%.*}) 126 | local ebuild_version=$(echo ${ebuild_full/${package}}|cut -d'-' -f2) 127 | local ebuild_revision=$(echo ${ebuild_full/${package}}|cut -d'-' -f3) 128 | local ebuild_slot="$(grep ^SLOT $ebuild|cut -d'"' -f2)" 129 | local package_name_ver="${package}-${ebuild_version}" 130 | 131 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 1-check_ebuild: ebuild details: ver: $ebuild_version rever: $ebuild_revision slot: $ebuild_slot" | (debug_output) 132 | 133 | local cn_name_vers="${patchfile/${package}/${pn}}" 134 | local cn=( ) 135 | 136 | # create custom names to check 137 | cn+=("${patchfile}") 138 | cn+=("${patchfile/${package}/${pn}}") 139 | cn+=("${patchfile/${package}-${ebuild_version}/${p}}") 140 | cn+=("${patchfile/${ebuild_version}/${pv}}") 141 | 142 | cn+=("${cn_name_vers/${ebuild_version}/${pv}}") 143 | # add special naming if there is a revision 144 | if [ -n "${ebuild_revision}" ]; then 145 | cn+=("${patchfile/${package}-${ebuild_version}-${ebuild_revision}/${pf}}") 146 | cn+=("${patchfile/${ebuild_version}-${ebuild_revision}/${pvr}}") 147 | fi 148 | # looks for names with slotes, if slot is not 0 149 | if [ -n "${ebuild_slot}" ] && ! [ "${ebuild_slot}" = "0" ]; then 150 | cn+=("${patchfile/${ebuild_slot}/${slot}}") 151 | cn+=("${cn_name_vers/${ebuild_slot}/${slot}}") 152 | fi 153 | 154 | # special naming 155 | if $(grep -q -E "^MY_PN=|^MY_P=|^MY_PV=|^MODULE_VERSION=|^DIST_VERSION=|^X509_VER=|^HPN_VER=" ${ebuild}); then 156 | 157 | # get the variables from the ebuilds 158 | 159 | # MY_PN and other such variables often are constructed with the usage of 160 | # global variables like $PN and $PV. 161 | # With using eval these variables are replaces by it's real content, 162 | # which must be replaced by our variable names first 163 | my_pn_name="$(grep ^MY_PN\= ${ebuild} | cut -d' ' -f1 | cut -d'=' -f2 | sed -e "s|PN|package|g" -e 's|"||g')" 164 | [ -n "${my_pn_name}" ] && eval my_pn_name="$(echo ${my_pn_name})" >/dev/null 2>&1 165 | my_pv_name="$(grep ^MY_PV\= ${ebuild} | cut -d' ' -f1 | cut -d'=' -f2 | sed -e "s|PV|ebuild_version|g" -e 's|"||g')" 166 | [ -n "${my_pv_name}" ] && eval my_pv_name="$(echo ${my_pv_name})" >/dev/null 2>&1 167 | my_p_name="$(grep ^MY_P\= ${ebuild} | cut -d' ' -f1 | cut -d'=' -f2 | sed -e "s|P|package_name_ver|g" -e 's|"||g')" 168 | [ -n "${my_p_name}" ] && eval my_p_name="$(echo ${my_p_name})" >/dev/null 2>&1 169 | 170 | my_mod_ver="$(grep ^MODULE_VERSION\= ${ebuild} | cut -d' ' -f1 | cut -d'=' -f2 | sed -e 's|"||g')" 171 | my_dist_ver="$(grep ^DIST_VERSION\= ${ebuild} | cut -d' ' -f1 | cut -d'=' -f2 | sed -e 's|"||g')" 172 | my_x509_ver="$(grep ^X509_VER\= ${ebuild} | cut -d' ' -f1 | cut -d'=' -f2 | sed -e 's|"||g')" 173 | my_hpn_ver="$(grep ^HPN_VER\= ${ebuild} | cut -d' ' -f1 | cut -d'=' -f2 | sed -e 's|"||g')" 174 | 175 | 176 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 1-check_ebuild: Found special vars: $my_pv_name, $my_pn_name, $my_p_name, $my_mod_ver, $my_dist_ver, $my_x509_ver, $my_hpn_ver" | (debug_output) 177 | 178 | [ -n "${my_pn_name}" ] && cn+=("${patchfile/${my_pn_name}/${var_my_pn}}") 179 | [ -n "${my_pv_name}" ] && cn+=("${patchfile/${my_pv_name}/${var_my_pv}}") 180 | [ -n "${my_p_name}" ] && cn+=("${patchfile/${my_p_name}/${var_my_p}}") 181 | 182 | local tmpvar="" 183 | 184 | if [ -n "${my_mod_ver}" ]; then 185 | cn+=("${patchfile/${my_mod_ver}/${var_mod_ver}}") 186 | tmpvar="${patchfile/${my_mod_ver}/${var_mod_ver}}" 187 | cn+=("${tmpvar/${package}/${pn}}") 188 | fi 189 | 190 | if [ -n "${my_dist_ver}" ]; then 191 | cn+=("${patchfile/${my_dist_ver}/${var_dist_ver}}") 192 | tmpvar="${patchfile/${my_dist_ver}/${var_dist_ver}}" 193 | cn+=("${tmpvar/${package}/${pn}}") 194 | fi 195 | 196 | if [ -n "${my_x509_ver}" ]; then 197 | cn+=("${patchfile/${my_x509_ver}/${var_x509_ver}}") 198 | tmpvar="${patchfile/${my_x509_ver}/${var_x509_ver}}" 199 | cn+=("${tmpvar/${package}/${pn}}") 200 | cn+=("${tmpvar/${package}-${ebuild_version}/${p}}") 201 | fi 202 | 203 | if [ -n "${my_hpn_ver}" ]; then 204 | cn+=("${patchfile/${my_hpn_ver}/${var_hpn_ver}}") 205 | tmpvar="${patchfile/${my_hpn_ver}/${var_hpn_ver}}" 206 | cn+=("${tmpvar/${package}/${pn}}") 207 | cn+=("${tmpvar/${package}-${ebuild_version}/${p}}") 208 | fi 209 | fi 210 | 211 | # remove duplicates 212 | mapfile -t cn < <(printf '%s\n' "${cn[@]}"|sort -u) 213 | # replace list with newpackages 214 | local searchpattern="$(echo ${cn[@]}|tr ' ' '\n')" 215 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 1-check_ebuild: Custom names: $(echo ${cn[@]}|tr ' ' ':')" | (debug_output) 216 | 217 | # check ebuild for the custom names 218 | if $(sed 's|"||g' ${ebuild} | grep -q -F "${searchpattern}"); then 219 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 1-check_ebuild: FOUND: ${patchfile}" | (debug_output) 220 | return 0 221 | else 222 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 1-check_ebuild: NOT FOUND: ${patchfile}" | (debug_output) 223 | fi 224 | done 225 | 226 | return 1 227 | } 228 | 229 | # white list checking 230 | white_check() { 231 | local pfile=${1} 232 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 0-whitelist: checking ${pfile} in ${WFILE}" | (debug_output) 233 | 234 | if [ -e ${WFILE} ]; then 235 | source ${WFILE} 236 | else 237 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-whitelist: ${WFILE} - file not found" | (debug_output) 238 | return 1 239 | fi 240 | 241 | if $(echo ${white_list[@]} | grep -q "${category}/${package};${pfile}"); then 242 | # detailed output only if debugging is enabled 243 | if [ ${DEBUGLEVEL} -ge 2 ]; then 244 | for white in ${white_list[@]}; do 245 | local cat_pak="$(echo ${white}|cut -d';' -f1)" 246 | local white_file="$(echo ${white}|cut -d';' -f2)" 247 | local white_ebuild="$(echo ${white}|cut -d';' -f3)" 248 | if [ "${category}/${package};${pfile}" = "${cat_pak};${white_file}" ]; then 249 | if [ "${white_ebuild}" = "*" ]; then 250 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-whitelist: found patch ${pfile} in all ebuilds" | (debug_output) 251 | return 0 252 | else 253 | for wbuild in $(echo ${white_ebuild} | tr ':' ' '); do 254 | if [ -e "${REPOTREE}/${full_package}/${wbuild}" ]; then 255 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-whitelist: found patch ${pfile} in ${wbuild}" | (debug_output) 256 | return 0 257 | fi 258 | done 259 | fi 260 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-whitelist: error in whitecheck with ${pfile}" | (debug_output) 261 | return 1 262 | fi 263 | done 264 | fi 265 | return 0 266 | else 267 | return 1 268 | fi 269 | } 270 | 271 | eclass_prechecks() { 272 | local pfile="${1}" 273 | 274 | if [ "${pfile}" = "rc-addon.sh" ]; then 275 | if $(grep -q vdr-plugin-2 ${fullpath}/*.ebuild); then 276 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-prechecks: file is rc-addon.sh and used in vdr-plugin-2.eclass" | (debug_output) 277 | return 0 278 | else 279 | return 1 280 | fi 281 | # check for vdr-plugin-2 eclass which installs confd files if exists 282 | elif [ "${pfile}" = "confd" ]; then 283 | if $(grep -q vdr-plugin-2 ${fullpath}/*.ebuild); then 284 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-prechecks: file is confd and used in vdr-plugin-2.eclass" | (debug_output) 285 | return 0 286 | else 287 | return 1 288 | fi 289 | # check for apache-module eclass which installs conf files if a APACHE2_MOD_CONF is set 290 | elif [ "${pfile##*.}" = "conf" ]; then 291 | if $(grep -q apache-module ${fullpath}/*.ebuild) && \ 292 | $(grep -q APACHE2_MOD_CONF ${fullpath}/*.ebuild); then 293 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-prechecks: file is conf and used in apache-module.eclass" | (debug_output) 294 | return 0 295 | else 296 | return 1 297 | fi 298 | # check for elisp eclass which install el files if a SITEFILE is set 299 | elif [ "${pfile##*.}" = "el" ]; then 300 | if $(grep -q elisp ${fullpath}/*.ebuild) && \ 301 | $(grep -q SITEFILE ${fullpath}/*.ebuild); then 302 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-prechecks: file is el and used in elisp.eclass" | (debug_output) 303 | return 0 304 | else 305 | return 1 306 | fi 307 | # ignoring README.gentoo files 308 | elif $(echo ${pfile}|grep -i README.gentoo >/dev/null); then 309 | if $(grep -q readme.gentoo ${fullpath}/*.ebuild); then 310 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 0-prechecks: file is readme.gentoo and used in readme.gentoo-r1.eclass" | (debug_output) 311 | return 0 312 | else 313 | return 1 314 | fi 315 | else 316 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 0-prechecks: ${pfile} is not used in inherited eclasses!" | (debug_output) 317 | return 1 318 | fi 319 | } 320 | 321 | # find_patches_in_braces 322 | # this function takes a list of patchfiles, and looks if the names 323 | # are similar by cutting the names at '-' starting with the last position. 324 | # 325 | # The similar part to the files found are being removed and with the 326 | # remaning part of the name we generate verions of how the patches can 327 | # be called like: 328 | # $PN-{patch2,patch1} or $PN-{patch1,patch2} 329 | # 330 | # Lastly we check again with check_ebuild to look if the ebuild use one 331 | # of the version profided and - if found - remove them from the 332 | # $unused_patches variable 333 | find_patches_in_braces() { 334 | local work_list=( $(echo ${1}|tr ':' ' ') ) 335 | 336 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 2-brace_patches: checking ${work_list[@]}" | (debug_output) 337 | 338 | local common_pattern=( ) 339 | local patches_to_remove=( ) 340 | local patch 341 | for patch in "${work_list[@]}"; do 342 | # how often contains the name the seperator '-' 343 | local deli=$(echo ${patch%.patch}|grep -o '-'|wc -w) 344 | # try find duplicates, starting with the highest count of $deli 345 | for n in $(echo $(seq 1 ${deli})|rev); do 346 | if [ $(echo ${work_list[@]}|tr ' ' '\n'|grep $(echo ${patch}|cut -d'-' -f1-${n})|wc -l) -gt 1 ]; then 347 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 2-brace_patches: pattern candidate: $(echo ${patch}|cut -d'-' -f1-${n})" | (debug_output) 348 | common_pattern+=("$(echo ${patch}|cut -d'-' -f1-${n})") 349 | break 350 | fi 351 | done 352 | done 353 | # remove duplicates from array 354 | mapfile -t common_pattern < <(printf '%s\n' ${common_pattern[@]}|sort -u) 355 | 356 | if [ -n "${common_pattern}" ]; then 357 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 2-brace_patches: found patterns ${common_pattern[@]}" | (debug_output) 358 | local x p 359 | for x in ${common_pattern[@]}; do 360 | local braces_patches=( ) 361 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 2-brace_patches: checking pattern ${x}" | (debug_output) 362 | 363 | # find duplicates files by pattern $x and strip everything away 364 | local matching=( $(find ${fullpath}/files/ -type f -name "${x}*" -printf '%f\n'|sed -e 's/.patch//' -e "s/${x}-//") ) 365 | 366 | # do not make permutations with greater then 5 matchings 367 | if [ ${#matching[@]} -le 5 ]; then 368 | local permutations=$(get_perm "$(echo ${matching[@]})") 369 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 2-brace_patches: create permutations with $(echo ${matching[@]})" | (debug_output) 370 | for p in ${permutations}; do 371 | braces_patches+=("$(echo ${x}-{${p}}.patch)") 372 | done 373 | 374 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 2-brace_patches: found ${braces_patches[@]}" | (debug_output) 375 | local t u 376 | for t in ${braces_patches[@]}; do 377 | if $(check_ebuild ${t}); then 378 | for u in ${matching[@]}; do 379 | patches_to_remove+=( "${x}-${u}.patch" ) 380 | done 381 | break 382 | fi 383 | done 384 | else 385 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 2-brace_patches: to much candidates, skipping" | (debug_output) 386 | fi 387 | done 388 | echo "${patches_to_remove[@]}" 389 | else 390 | echo "" 391 | fi 392 | } 393 | 394 | # find_patches_with_asterisk 395 | # check for a certain text (FILESDIR.*\*) which is an indicater that files are 396 | # being used by using '*'. In this case we are checking what files are 397 | # matching and remove the from the unused_patches list 398 | find_patches_with_asterisk() { 399 | local work_list=( $(echo ${1}|tr ':' ' ') ) 400 | 401 | local ebuild 402 | local candidate 403 | local found_patches=( ) 404 | 405 | for ebuild in ${fullpath}/*.ebuild; do 406 | # look for matching FILESDIR.*\*, remove path part, replace some patterns 407 | # like ';', '"' and replace curly braces (and everthing between) with '*' 408 | for candidate in $(grep -oP "FILESDIR.*\*(\S+|$)" ${ebuild} | rev | cut -d'/' -f1 | rev | sort -u |sed -e 's/"//g' -e 's/;//g' -e 's/[{].*[}]/*/g'); do 409 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 3-asterixes: found asterixes in ebuild ${candidate[@]}" | (debug_output) 410 | 411 | local matching_asterisk=( $(find ${fullpath}/files/${candidate} -type f -printf '%f\n' 2>/dev/null) ) 412 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 3-asterixes: matching following files: ${matching_asterisk[@]}" | (debug_output) 413 | 414 | for p in ${matching_asterisk[@]}; do 415 | if $(echo ${work_list[@]} | grep -q ${p}); then 416 | [ ${DEBUGLEVEL} -ge 3 ] && echo " 3-asterixes: adding to remove list: ${p}" | (debug_output) 417 | found_patches+=( ${p} ) 418 | fi 419 | done 420 | done 421 | done 422 | 423 | if [ -n "${found_patches}" ]; then 424 | [ ${DEBUGLEVEL} -ge 2 ] && echo " 3-asterixes: removing from unused_patches: $(echo ${found_patches[@]}|tr ' ' '\n'|sort -u| tr '\n' ' ')" | (debug_output) 425 | echo ${found_patches[@]}|tr ' ' '\n'|sort -u 426 | else 427 | echo 428 | fi 429 | } 430 | 431 | local full_package=${1} 432 | local fullpath="/${REPOTREE}/${full_package}" 433 | 434 | [ ${DEBUGLEVEL} -ge 2 ] && echo "checking: ${category}/${package}" | (debug_output) 435 | # check if the patches folder exist 436 | if [ -e ${fullpath}/files ]; then 437 | 438 | local category="$(echo ${full_package}|cut -d'/' -f1)" 439 | local package="$(echo ${full_package}|cut -d'/' -f2)" 440 | local main="$(get_main_min "${category}/${package}")" 441 | local patch_list=() 442 | 443 | [ ${DEBUGLEVEL} -ge 3 ] && echo "found files dir in: ${category}/${package}/files" | (debug_output) 444 | 445 | 446 | # prechecks and patchlist generation 447 | # every file found will be checked against the whitelist (white_check) and 448 | # checked for the usage of certain eclasses (which are know to use certain 449 | # files from the FILESDIR directory directly. 450 | [ ${DEBUGLEVEL} -ge 3 ] && echo "prechecks: whitelist and special eclasses" | (debug_output) 451 | for file in ${fullpath}/files/*; do 452 | # ignore directories 453 | if ! [ -d ${file} ]; then 454 | file="${file##*/}" 455 | if ! $(white_check ${file}); then 456 | # elcass-prechecks 457 | if $(echo ${file##*/}|grep -q -E "\.el|conf|rc-addon.sh|README*"); then 458 | if ! $(eclass_prechecks ${file}); then 459 | patch_list+=("${file}") 460 | fi 461 | else 462 | patch_list+=("${file}") 463 | fi 464 | fi 465 | fi 466 | done 467 | [ ${DEBUGLEVEL} -ge 2 ] && echo "prechecks done: patchlist: ${patch_list[@]}" | (debug_output) 468 | 469 | 470 | # only continue if we found actually files to check 471 | if [ -n "${patch_list}" ]; then 472 | 473 | # first check 474 | # every patchfile from $patch_list gets passed to check_ebuild, which 475 | # replaces names and version with their corresponding ebuild name 476 | # ($PN, PV, ..) and grep's the ebuild with them. 477 | [ ${DEBUGLEVEL} -ge 3 ] && echo "starting basic check for: ${patch_list[@]}" | (debug_output) 478 | unused_patches=() 479 | for patchfile in "${patch_list[@]}"; do 480 | if ! $(check_ebuild "${patchfile}"); then 481 | unused_patches+=("${patchfile}") 482 | fi 483 | done 484 | [ ${DEBUGLEVEL} -ge 2 ] && echo "basic check done, unused patches are: ${unused_patches[@]}" | (debug_output) 485 | 486 | # second check 487 | # find patches in braces (works only with *.patch files) 488 | # examples: app-editors/zile, app-office/scribus, app-office/libreoffice, 489 | # dev-cpp/antlr-cpp, dev-qt/qtcore, games-arcade/supertux 490 | [ ${DEBUGLEVEL} -ge 3 ] && echo "starting second check for: ${unused_patches[@]}" | (debug_output) 491 | if [ ${#unused_patches[@]} -ge 1 ]; then 492 | for patchfile in $(find_patches_in_braces "$(echo ${unused_patches[@]}|tr ' ' ':')"); do 493 | [ ${DEBUGLEVEL} -ge 2 ] && echo "patch to remove: ${patchfile}" | (debug_output) 494 | for target in "${!unused_patches[@]}"; do 495 | if [ "${unused_patches[target]}" = "${patchfile}" ]; then 496 | unset 'unused_patches[target]' 497 | fi 498 | done 499 | done 500 | [ ${DEBUGLEVEL} -ge 2 ] && echo "finish second check, remaining patches: ${unused_patches[@]}" | (debug_output) 501 | fi 502 | 503 | # third check 504 | # find pachtes which are called with an asterix (*) 505 | # examples: net-misc/icaclient, app-admin/consul 506 | [ ${DEBUGLEVEL} -ge 3 ] && echo "starting third check for: ${unused_patches[@]}" | (debug_output) 507 | if [ -n "${unused_patches}" ]; then 508 | for patchfile in $(find_patches_with_asterisk "$(echo ${unused_patches[@]}|tr ' ' ':')"); do 509 | [ ${DEBUGLEVEL} -ge 2 ] && echo "patch to remove: ${patchfile}" | (debug_output) 510 | for target in "${!unused_patches[@]}"; do 511 | if [ "${unused_patches[target]}" = "${patchfile}" ]; then 512 | unset 'unused_patches[target]' 513 | fi 514 | done 515 | done 516 | [ ${DEBUGLEVEL} -ge 2 ] && echo "finish third check, remaining patches: ${unused_patches[@]}" | (debug_output) 517 | fi 518 | 519 | array_names 520 | if [ ${#unused_patches[@]} -gt 0 ]; then 521 | [ ${DEBUGLEVEL} -ge 2 ] && echo "found unused patches: ${unused_patches[@]}" | (debug_output) 522 | output 0 523 | else 524 | [ ${DEBUGLEVEL} -ge 3 ] && echo "found zero unused patches" | (debug_output) 525 | fi 526 | 527 | [ ${DEBUGLEVEL} -ge 2 ] && echo | (debug_output) 528 | [ ${DEBUGLEVEL} -ge 2 ] && echo | (debug_output) 529 | else 530 | [ ${DEBUGLEVEL} -ge 3 ] && echo "skipping: ${category}/${package} has files directory, but no there are no files to check" | (debug_output) 531 | fi 532 | else 533 | [ ${DEBUGLEVEL} -ge 3 ] && echo "skipping: ${category}/${package} has no files directory" | (debug_output) 534 | fi 535 | } 536 | 537 | find_func(){ 538 | # Dont use parallel if DEBUG is enabled 539 | if [ ${DEBUGLEVEL} -ge 2 ]; then 540 | [ ${DEBUGLEVEL} -ge 2 ] && echo "NORMAL run: searchpattern is ${searchp[@]}" | (debug_output) 541 | find ${searchp[@]} -mindepth ${MIND} -maxdepth ${MAXD} \ 542 | -type d -print 2>/dev/null | while read -r line; do 543 | main ${line} 544 | done 545 | else 546 | [ ${DEBUGLEVEL} -ge 1 ] && echo "PARALLEL run: searchpattern is ${searchp[@]}" | (debug_output) 547 | find ${searchp[@]} -mindepth ${MIND} -maxdepth ${MAXD} \ 548 | -type d -print 2>/dev/null | parallel main {} 549 | fi 550 | 551 | if ${FILERESULTS}; then 552 | gen_descriptions 553 | sort_result_v4 554 | gen_sort_main_v4 555 | gen_sort_pak_v4 556 | 557 | copy_checks ${SCRIPT_TYPE} 558 | fi 559 | } 560 | 561 | [ ${DEBUGLEVEL} -ge 1 ] && echo "*** starting patchtest" | (debug_output) 562 | 563 | cd ${REPOTREE} 564 | array_names 565 | export -f main array_names output_format 566 | export WORKDIR WFILE 567 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 568 | depth_set_v3 ${1} 569 | ${FILERESULTS} && rm -rf ${WORKDIR} 570 | 571 | [ ${DEBUGLEVEL} -ge 1 ] && echo "*** finished patchtest" | (debug_output) 572 | -------------------------------------------------------------------------------- /deprecated/repomancheck.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: repomancheck.sh 4 | # Autor: Michael Mair-Keimberger (mmk AT levelnine DOT at) 5 | # Date: 01/06/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simply runs repoman full on every package 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export REPOTREE=/usr/portage/ 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/repomancheck/" 30 | # enabling debug output 31 | #export DEBUG=true 32 | #export DEBUGLEVEL=1 33 | #export DEBUGFILE=/tmp/repostats.log 34 | 35 | # get dirpath and load funcs.sh 36 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 37 | if [ -e ${realdir}/_funcs.sh ]; then 38 | source ${realdir}/_funcs.sh 39 | else 40 | echo "Missing _funcs.sh" 41 | exit 1 42 | fi 43 | 44 | # 45 | ### IMPORTANT SETTINGS START ### 46 | # 47 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 48 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 49 | #${ENABLE_GIT} || exit 0 # only works with git tree 50 | 51 | SCRIPT_TYPE="checks" 52 | WORKDIR="/tmp/repomancheck-${RANDOM}" 53 | REPOCHECK=true 54 | 55 | array_names(){ 56 | RUNNING_CHECKS=( 57 | "${WORKDIR}/packages_full_repoman" #Index 0 58 | ) 59 | } 60 | output_format(){ 61 | index=( 62 | "${category}/${package}${DL}$(echo ${affected_checks[@]}|tr ' ' ':')${DL}${maintainer}" 63 | ) 64 | echo "${index[$1]}" 65 | } 66 | data_descriptions(){ 67 | read -r -d '' info_index0 <<- EOM 68 | A script which runs 'repoman full' on every package. The result is also filtered 69 | by repomans checks. 70 | 71 | Data Format ( dev-libs/foo|inherit.deprecated:uri.https|dev@gentoo.org:loper@foo.de ): 72 | dev-libs/foo package category/name 73 | inherit.deprecated:uri.https repoman problem(s), seperated by ':' 74 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 75 | EOM 76 | description=( "${info_index0}" "${info_index1}" ) 77 | echo "${description[$1]}" 78 | } 79 | # 80 | ### IMPORTANT SETTINGS END ### 81 | # 82 | 83 | main() { 84 | array_names 85 | local relative_path=${1} 86 | local category="$(echo ${relative_path}|cut -d'/' -f1)" 87 | local package="$(echo ${relative_path}|cut -d'/' -f2)" 88 | local full_path="${REPOTREE}/${category}/${package}" 89 | local maintainer="$(get_main_min "${category}/${package}")" 90 | 91 | cd ${full_path} 92 | local TMPFILE="/tmp/${category}-${package}-${RANDOM}.log" 93 | /usr/bin/repoman -q full > ${TMPFILE} 94 | 95 | local affected_checks=( $(grep '^ [a-zA-Z].*' ${TMPFILE} | cut -d' ' -f3 ) ) 96 | 97 | if ! [ "$(cat ${TMPFILE})" = "No QA issues found" ]; then 98 | if ${FILERESULTS}; then 99 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-package/${category}/ 100 | head -n-1 ${TMPFILE} > ${RUNNING_CHECKS[0]}/sort-by-package/${category}/${package}.txt 101 | output_format 0 >> ${RUNNING_CHECKS[0]}/full.txt 102 | else 103 | if [ "${1}" = "full" ]; then 104 | output_format 0 105 | else 106 | echo "${category}/${package}${DL}${maintainer}" 107 | head -n-1 ${TMPFILE} 108 | fi 109 | fi 110 | fi 111 | 112 | rm ${TMPFILE} 113 | } 114 | 115 | 116 | find_func(){ 117 | find ${searchp[@]} -mindepth ${MIND} -maxdepth ${MAXD} \ 118 | -type d -print 2>/dev/null | parallel main {} 119 | 120 | if ${FILERESULTS}; then 121 | gen_descriptions 122 | gen_sort_filter_v1 2 ${RUNNING_CHECKS[0]} 123 | 124 | gen_sort_main_v4 125 | for ffp in $(ls ${RUNNING_CHECKS[0]}/sort-by-filter/); do 126 | gen_sort_pak_v4 ${RUNNING_CHECKS[0]}/sort-by-filter/${ffp} 127 | done 128 | 129 | copy_checks ${SCRIPT_TYPE} 130 | fi 131 | } 132 | 133 | array_names 134 | # switch to the REPOTREE dir 135 | cd ${REPOTREE} 136 | # export important variables 137 | export WORKDIR REPOCHECK 138 | export -f main array_names output_format 139 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 140 | # set the search depth 141 | depth_set_v3 ${1} 142 | ${FILERESULTS} && rm -rf ${WORKDIR} 143 | -------------------------------------------------------------------------------- /deprecated/simplechecks.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: simplechecks.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 26/08/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # This script finds simple errors in ebuilds and other files. For now it can 25 | # ebuilds: check for trailing whitespaces 26 | # metadata: mixed indentation (mixed tabs & whitespaces) 27 | 28 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 29 | #export REPOTREE=/usr/portage/ 30 | #export FILERESULTS=true 31 | #export RESULTSDIR="${HOME}/simplechecks/" 32 | 33 | # get dirpath and load funcs.sh 34 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 35 | if [ -e ${realdir}/_funcs.sh ]; then 36 | source ${realdir}/_funcs.sh 37 | else 38 | echo "Missing _funcs.sh" 39 | exit 1 40 | fi 41 | 42 | # 43 | ### IMPORTANT SETTINGS START ### 44 | # 45 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 46 | ${ENABLE_MD5} || exit 0 # only works with md5 cache 47 | #${ENABLE_GIT} || exit 0 # only works with git tree 48 | 49 | SCRIPT_TYPE="checks" 50 | WORKDIR="/tmp/simplechecks-${RANDOM}" 51 | 52 | array_names(){ 53 | RUNNING_CHECKS=( 54 | "${WORKDIR}/ebuild_trailing_whitespaces" # Index 0 55 | "${WORKDIR}/metadata_mixed_indentation" # Index 1 56 | "${WORKDIR}/ebuild_obsolete_gentoo_mirror_usage" # Index 2 57 | "${WORKDIR}/ebuild_epatch_in_eapi6" # Index 3 58 | "${WORKDIR}/ebuild_dohtml_in_eapi6" # Index 4 59 | "${WORKDIR}/ebuild_description_over_80" # Index 5 60 | "${WORKDIR}/metadata_missing_proxy_maintainer" # Index 6 61 | "${WORKDIR}/ebuild_variables_in_homepages" # Index 7 62 | "${WORKDIR}/ebuild_insecure_git_uri_usage" # Index 8 63 | ) 64 | } 65 | output_format(){ 66 | index=( 67 | "${ebuild_eapi}${DL}${category}/${package}${DL}${filename}${DL}${maintainer}" 68 | "${category}/${package}${DL}${filename}${DL}${maintainer}" 69 | ) 70 | echo "${index[$1]}" 71 | } 72 | data_descriptions(){ 73 | read -r -d '' info_default0 <<- EOM 74 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|dev@gentoo.org:loper@foo.de ): 75 | 7 EAPI Version 76 | dev-libs/foo package category/name 77 | foo-1.12-r2.ebuild full filename 78 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 79 | EOM 80 | read -r -d '' info_default1 <<- EOM 81 | Data Format ( dev-libs/foo|metadata.xml|dev@gentoo.org:loper@foo.de ): 82 | dev-libs/foo package category/name 83 | metadata.xml metadata filename 84 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 85 | EOM 86 | 87 | read -r -d '' info_index0 <<- EOM 88 | Simple check to find leading or trailing whitespaces in a set of variables. 89 | For example: SRC_URI=" www.foo.com/bar.tar.gz " 90 | 91 | ${info_default0} 92 | EOM 93 | read -r -d '' info_index1 <<- EOM 94 | Checks metadata files (metadata.xml) if it uses mixed tabs and whitespaces. 95 | 96 | ${info_default1} 97 | EOM 98 | read -r -d '' info_index2 <<- EOM 99 | Ebuilds shouldn't use mirror://gentoo in SRC_URI because it's deprecated. 100 | Also see: Link 101 | 102 | ${info_default0} 103 | EOM 104 | read -r -d '' info_index3 <<- EOM 105 | 'epatch' is deprecated and should be replaced by 'eapply'. 106 | Also see: Link 107 | 108 | ${info_default0} 109 | EOM 110 | read -r -d '' info_index4 <<- EOM 111 | 'dohtml' is deprecated in EAPI6 and banned in EAPI7. 112 | This check lists EAPI6 ebuilds which still use 'dohtml' 113 | Also see: Link 114 | 115 | ${info_default0} 116 | EOM 117 | read -r -d '' info_index5 <<- EOM 118 | Checks ebuilds if the DESCRIPTION is longer than 80 characters. 119 | 120 | ${info_default0} 121 | EOM 122 | read -r -d '' info_index6 <<- EOM 123 | Checks the metadata.xml of proxy maintained packages if it includes actually a 124 | non gentoo email address (address of proxy maintainer). 125 | Reason: There can't be a proxy maintained package without a proxy maintainer in metadata.xml 126 | 127 | ${info_default1} 128 | EOM 129 | read -r -d '' info_index7 <<- EOM 130 | Simple check to find variables in HOMEPAGE. While not technically a bug, this shouldn't be used. 131 | See Tracker bug: Link 132 | Also see bug: Link 133 | 134 | ${info_default0} 135 | EOM 136 | read -r -d '' info_index8 <<- EOM 137 | Ebuilds shouldn't use git:// for git repos because its insecure. Should be replaced with https:// 138 | Also see: Link 139 | 140 | ${info_default0} 141 | EOM 142 | 143 | description=( "${info_index0}" "${info_index1}" "${info_index2}" \ 144 | "${info_index3}" "${info_index4}" "${info_index5}" "${info_index6}" \ 145 | "${info_index7}" "${info_index8}" \ 146 | ) 147 | echo "${description[$1]}" 148 | } 149 | # 150 | ### IMPORTANT SETTINGS END ### 151 | # 152 | 153 | main() { 154 | array_names 155 | local full_package=${1} 156 | local category="$(echo ${full_package}|cut -d'/' -f1)" 157 | local package="$(echo ${full_package}|cut -d'/' -f2)" 158 | local filename="$(echo ${full_package}|cut -d'/' -f3)" 159 | 160 | local maintainer="$(get_main_min "${category}/${package}")" 161 | local ebuild_eapi="$(get_eapi ${full_package})" 162 | 163 | output(){ 164 | local checkid=${1} 165 | if ${FILERESULTS}; then 166 | output_format 0 >> ${RUNNING_CHECKS[${checkid}]}/full.txt 167 | else 168 | echo "${RUNNING_CHECKS[${checkid}]##*/}${DL}$(output_format 0)" 169 | fi 170 | } 171 | 172 | # trailing whitespace 173 | if $(egrep -q " +$" ${full_package}); then 174 | output 0 175 | fi 176 | # mirror usage 177 | if $(grep -q 'mirror://gentoo' ${full_package}); then 178 | output 2 179 | fi 180 | 181 | if [ "${ebuild_eapi}" = "6" ]; then 182 | # epatch usage 183 | if $(grep -q "\" ${full_package}); then 184 | output 3 185 | fi 186 | # dohtml usage 187 | if $(grep -q "\" ${full_package}); then 188 | output 4 189 | fi 190 | fi 191 | # DESCRIPTION over 80 192 | if [ $(grep DESCRIPTION ${REPOTREE}/metadata/md5-cache/${category}/${filename%.*} | wc -m) -gt 95 ]; then 193 | output 5 194 | fi 195 | # HOMEPAGE with variables 196 | if $(grep -q "HOMEPAGE=.*\${" ${full_package}); then 197 | if ! $(grep -q 'HOMEPAGE=.*${HOMEPAGE}' ${full_package}); then 198 | output 7 199 | fi 200 | fi 201 | # insecure git usage 202 | if $(grep -q "EGIT_REPO_URI=\"git://" ${full_package}); then 203 | output 8 204 | fi 205 | } 206 | 207 | main-xml(){ 208 | array_names 209 | local full_package=${1} 210 | local category="$(echo ${full_package}|cut -d'/' -f1)" 211 | local package="$(echo ${full_package}|cut -d'/' -f2)" 212 | local filename="metadata.xml" 213 | 214 | if [ -e ${REPOTREE}/${category}/${package}/metadata.xml ]; then 215 | local maintainer="$(get_main_min "${category}/${package}")" 216 | fi 217 | 218 | output(){ 219 | local checkid=${1} 220 | if ${FILERESULTS}; then 221 | output_format 1 >> ${RUNNING_CHECKS[${checkid}]}/full.txt 222 | else 223 | echo "${RUNNING_CHECKS[${checkid}]##*/}${DL}$(output_format 1)" 224 | fi 225 | } 226 | 227 | # mixed indentation 228 | if $(grep -q "^ " ${full_package}); then 229 | if $(grep -q $'\t' ${full_package}); then 230 | output 1 231 | fi 232 | fi 233 | # missing proxy maintainer 234 | local ok=false 235 | if $(grep -q "proxy-maint@gentoo.org" ${full_package}); then 236 | local i 237 | for i in $(echo ${maintainer}|tr ':' '\n'); do 238 | if ! $(echo ${i} | grep -q "@gentoo.org"); then 239 | ok=true 240 | fi 241 | done 242 | 243 | if ! ${ok}; then 244 | output 6 245 | fi 246 | fi 247 | } 248 | 249 | 250 | find_func(){ 251 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 252 | -type f -name "*.ebuild" -print | parallel main {} 253 | 254 | find ${searchp[@]} -mindepth ${MIND} -maxdepth $(expr ${MAXD} + 1) \ 255 | -type f -name "*.xml" -print | parallel main-xml {} 256 | } 257 | 258 | gen_results(){ 259 | if ${FILERESULTS}; then 260 | gen_descriptions 261 | sort_result_v3 262 | gen_sort_main_v3 263 | gen_sort_pak_v3 264 | 265 | copy_checks ${SCRIPT_TYPE} 266 | fi 267 | } 268 | 269 | array_names 270 | cd ${REPOTREE} 271 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 272 | export -f main main-xml array_names output_format 273 | export WORKDIR 274 | depth_set_v2 ${1} 275 | ${FILERESULTS} && rm -rf ${WORKDIR} 276 | -------------------------------------------------------------------------------- /deprecated/srctest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: srctest 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 12/08/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simple scirpt to find broken SRC_URI links 25 | 26 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export REPOTREE=/usr/portage/ 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/srctest/" 30 | # enabling debug output 31 | #export DEBUG=true 32 | #export DEBUGLEVEL=1 33 | #export DEBUGFILE=/tmp/repostats.log 34 | 35 | # get dirpath and load funcs.sh 36 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 37 | if [ -e ${realdir}/_funcs.sh ]; then 38 | source ${realdir}/_funcs.sh 39 | else 40 | echo "Missing _funcs.sh" 41 | exit 1 42 | fi 43 | 44 | # 45 | ### IMPORTANT SETTINGS START ### 46 | # 47 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 48 | ${ENABLE_MD5} || exit 0 # only works with md5 cache 49 | #${ENABLE_GIT} || exit 0 # only works with git tree 50 | 51 | SCRIPT_TYPE="checks" 52 | WORKDIR="/tmp/srctest-${RANDOM}" 53 | TMPCHECK="/tmp/srctest-tmp-${RANDOM}.txt" 54 | JOBS="50" 55 | 56 | array_names(){ 57 | RUNNING_CHECKS=( 58 | "${WORKDIR}/ebuild_src_uri_check" #Index 0 59 | ) 60 | } 61 | output_format(){ 62 | index=( 63 | "${ebuild_eapi}${DL}${category}/${package}${DL}${ebuild}${DL}${srclink}${DL}${maintainer}${openbugs}" 64 | ) 65 | echo "${index[$1]}" 66 | } 67 | data_descriptions(){ 68 | read -r -d '' info_index0 <<- EOM 69 | This check uses wget's spider functionality to check if a ebuild's SRC_URI link still works. 70 | The timeout to try to get a file is 15 seconds. 71 | 72 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|https://foo.bar.com/bar.zip|dev@gentoo.org:loper@foo.de|754124:612230 ): 73 | 7 EAPI Version 74 | dev-libs/foo package category/name 75 | foo-1.12-r2.ebuild full filename 76 | https://foo.bar.com/bar.zip file which is not available 77 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 78 | 754124:612230 open bug ids related to this package, seperated by ':' 79 | EOM 80 | description=( "${info_index0}" ) 81 | echo "${description[$1]}" 82 | } 83 | # 84 | ### IMPORTANT SETTINGS END ### 85 | # 86 | 87 | main() { 88 | get_status() { 89 | local uri="${1}" 90 | local code="${2}" 91 | if $(timeout 15 wget -T 10 --no-check-certificate -S --spider ${uri} 2>&1 | grep -E "${code}" >/dev/null); then 92 | echo true 93 | else 94 | echo false 95 | fi 96 | } 97 | 98 | mode() { 99 | local id=${1} 100 | local status=${2} # available/maybe_available/not_available 101 | 102 | if ${FILERESULTS}; then 103 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full_${status}.txt 104 | echo "${status}${DL}$(output_format ${id})" >> "${RUNNING_CHECKS[${id}]}/full-unfiltered.txt" 105 | else 106 | echo "${status}${DL}$(output_format ${id})" 107 | fi 108 | } 109 | 110 | array_names 111 | 112 | local full_package=${1} 113 | local category="$(echo ${full_package}|cut -d'/' -f1)" 114 | local package="$(echo ${full_package}|cut -d'/' -f2)" 115 | local maintainer="$(get_main_min "${category}/${package}")" 116 | local openbugs="$(get_bugs "${category}/${package}")" 117 | if ! [ -z "${openbugs}" ]; then 118 | openbugs="${DL}${openbugs}" 119 | fi 120 | 121 | # ignore the following packagelist for now. 122 | # all inheriting the toolchain-{binutils,glibc} eclass which generates the 123 | # SRC_URI somehow and probably generates lots of false positives 124 | local _ignore_list=( 125 | dev-lang/gnat-gpl 126 | sys-devel/gcc 127 | sys-devel/kgcc64 128 | sys-devel/binutils 129 | sys-devel/binutils-hppa64 130 | sys-libs/glibc 131 | app-text/texlive-core 132 | dev-texlive/texlive-basic 133 | dev-texlive/texlive-bibtexextra 134 | dev-texlive/texlive-context 135 | dev-texlive/texlive-fontsextra 136 | dev-texlive/texlive-fontsrecommended 137 | dev-texlive/texlive-fontutils 138 | dev-texlive/texlive-formatsextra 139 | dev-texlive/texlive-games 140 | dev-texlive/texlive-humanities 141 | dev-texlive/texlive-langarabic 142 | dev-texlive/texlive-langchinese 143 | dev-texlive/texlive-langcjk 144 | dev-texlive/texlive-langcyrillic 145 | dev-texlive/texlive-langczechslovak 146 | dev-texlive/texlive-langenglish 147 | dev-texlive/texlive-langeuropean 148 | dev-texlive/texlive-langfrench 149 | dev-texlive/texlive-langgerman 150 | dev-texlive/texlive-langgreek 151 | dev-texlive/texlive-langitalian 152 | dev-texlive/texlive-langjapanese 153 | dev-texlive/texlive-langkorean 154 | dev-texlive/texlive-langother 155 | dev-texlive/texlive-langpolish 156 | dev-texlive/texlive-langportuguese 157 | dev-texlive/texlive-langspanish 158 | dev-texlive/texlive-latex 159 | dev-texlive/texlive-latexextra 160 | dev-texlive/texlive-latexrecommended 161 | dev-texlive/texlive-luatex 162 | dev-texlive/texlive-mathscience 163 | dev-texlive/texlive-metapost 164 | dev-texlive/texlive-music 165 | dev-texlive/texlive-pictures 166 | dev-texlive/texlive-plaingeneric 167 | dev-texlive/texlive-pstricks 168 | dev-texlive/texlive-publishers 169 | dev-texlive/texlive-xetex 170 | ) 171 | for iglist in ${_ignore_list[@]}; do 172 | if [ "${category}/${package}" = "${iglist}" ]; then 173 | return 0 174 | fi 175 | done 176 | 177 | # code_available='HTTP/1.0 200 OK|HTTP/1.1 200 OK' 178 | code_available='Remote file exists.' 179 | maybe_available='HTTP/1.0 403 Forbidden|HTTP/1.1 403 Forbidden' 180 | 181 | for eb in ${REPOTREE}/${full_package}/*.ebuild; do 182 | local ebuild_eapi="$(get_eapi ${eb})" 183 | local ebuild=$(basename ${eb%.*}) 184 | 185 | local _src="$(grep ^SRC_URI= ${REPOTREE}/metadata/md5-cache/${category}/${ebuild})" 186 | local _src=${_src:8} 187 | 188 | if [ -n "${_src}" ]; then 189 | # the variable SRC_URI sometimes has more data than just download links like 190 | # useflags or renamings, so just grep each text for http/https 191 | local u 192 | for u in ${_src}; do 193 | # add ^mirror:// to the grep, somehow we should be able to test them too 194 | local i 195 | for i in $(echo ${u} | grep -E "^http://|^https://"); do 196 | # check for zip dependecy first 197 | local srclink=${i} 198 | 199 | local _checktmp="$(grep -P "(^|\s)\K${srclink}(?=\s|$)" ${TMPCHECK}|sort -u)" 200 | if [ -n "${_checktmp}" ]; then 201 | srclink="$(echo ${_checktmp}| cut -d' ' -f2-)" 202 | mode 0 "$(echo ${_checktmp} | cut -d' ' -f1)" 203 | else 204 | if $(get_status ${srclink} "${code_available}"); then 205 | mode 0 available 206 | echo "available ${srclink}" >> ${TMPCHECK} 207 | elif $(get_status ${srclink} "${maybe_available}"); then 208 | mode 0 maybe_available 209 | echo "maybe_available ${srclink}" >> ${TMPCHECK} 210 | else 211 | mode 0 not_available 212 | echo "not_available ${srclink}" >> ${TMPCHECK} 213 | fi 214 | fi 215 | done 216 | done 217 | fi 218 | done 219 | } 220 | 221 | find_func() { 222 | find ${searchp[@]} -mindepth ${MIND} -maxdepth ${MAXD} \ 223 | -type d -print 2>/dev/null | parallel -j ${JOBS} main {} 224 | 225 | if ${FILERESULTS}; then 226 | gen_descriptions 227 | cp ${RUNNING_CHECKS[0]}/full_not_available.txt ${RUNNING_CHECKS[0]}/full.txt 2>/dev/null 228 | 229 | sort_result_v4 230 | gen_sort_main_v4 231 | gen_sort_pak_v4 232 | 233 | copy_checks ${SCRIPT_TYPE} 234 | fi 235 | } 236 | 237 | array_names 238 | cd ${REPOTREE} 239 | export -f main array_names output_format 240 | export WORKDIR TMPCHECK 241 | touch ${TMPCHECK} 242 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 243 | if [ "${1}" = "diff" ]; then 244 | depth_set_v3 full 245 | else 246 | depth_set_v3 ${1} 247 | fi 248 | ${FILERESULTS} && rm -rf ${WORKDIR} 249 | rm ${TMPCHECK} 250 | -------------------------------------------------------------------------------- /deprecated/trailwhite.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: trailwhite.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 15/04/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # a spinoff of simplechecks, only for finding trailing and leading whitespaces 25 | # in variables 26 | 27 | #override REPOTREE,FILERESULTS,RESULTSDIR settings 28 | #export REPOTREE=/usr/portage/ 29 | #export FILERESULTS=true 30 | #export RESULTSDIR="${HOME}/trailwhite/" 31 | 32 | # get dirpath and load funcs.sh 33 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 34 | if [ -e ${realdir}/_funcs.sh ]; then 35 | source ${realdir}/_funcs.sh 36 | else 37 | echo "Missing _funcs.sh" 38 | exit 1 39 | fi 40 | 41 | # 42 | ### IMPORTANT SETTINGS START ### 43 | # 44 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 45 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 46 | #${ENABLE_GIT} || exit 0 # only works with git tree 47 | 48 | SCRIPT_TYPE="checks" 49 | WORKDIR="/tmp/trailwhite-${RANDOM}" 50 | 51 | array_names(){ 52 | RUNNING_CHECKS=( 53 | "${WORKDIR}/ebuild_leading_trailing_whitespaces_in_variables" # Index 0 54 | ) 55 | } 56 | output_format(){ 57 | index=( 58 | "${category}/${package}${DL}${filename}${DL}$(echo ${_tl_vars[@]}|tr ' ' ':')${DL}${maintainer}" 59 | ) 60 | echo "${index[$1]}" 61 | } 62 | data_descriptions(){ 63 | read -r -d '' info_index0 <<- EOM 64 | Simple check to find leading or trailing whitespaces in a set of variables. 65 | For example: SRC_URI=" www.foo.com/bar.tar.gz " 66 | 67 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|DEPEND:SRC_URI|dev@gentoo.org:loper@foo.de ): 68 | 7 EAPI Version 69 | dev-libs/foo package category/name 70 | foo-1.12-r2.ebuild full filename 71 | DEPEND:SRC_URI list of variables which have unusual whitespaces, seperated by ':' 72 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 73 | EOM 74 | description=( "${info_index0}" ) 75 | echo "${description[$1]}" 76 | } 77 | # 78 | ### IMPORTANT SETTINGS END ### 79 | # 80 | 81 | main() { 82 | array_names 83 | 84 | local full_package=${1} 85 | local category="$(echo ${full_package}|cut -d'/' -f1)" 86 | local package="$(echo ${full_package}|cut -d'/' -f2)" 87 | local filename="$(echo ${full_package}|cut -d'/' -f3)" 88 | 89 | local maintainer="$(get_main_min "${category}/${package}")" 90 | 91 | local _varibales="DESCRIPTION LICENSE KEYWORDS IUSE RDEPEND DEPEND SRC_URI" 92 | local _tl_vars=( ) 93 | 94 | output() { 95 | local id=${1} 96 | if ${FILERESULTS}; then 97 | output_format ${id} >> ${RUNNING_CHECKS[${id}]}/full.txt 98 | else 99 | output_format ${id} 100 | fi 101 | } 102 | 103 | for var in ${_varibales}; do 104 | if $(egrep -q "^${var}=\" |^${var}=\".* \"$" ${full_package}); then 105 | _tl_vars+=( ${var} ) 106 | fi 107 | done 108 | 109 | if [ -n "${_tl_vars}" ]; then 110 | output 0 111 | fi 112 | } 113 | 114 | 115 | find_func(){ 116 | find ${searchp[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 117 | -type f -name "*.ebuild" -print | parallel main {} 118 | } 119 | 120 | gen_results(){ 121 | if ${FILERESULTS}; then 122 | gen_descriptions 123 | sort_result_v2 124 | gen_sort_main_v3 125 | gen_sort_pak_v3 126 | 127 | for file in $(cat ${RUNNING_CHECKS[0]}/full.txt); do 128 | for ec in $(echo ${file}|cut -d'|' -f3|tr ':' ' '); do 129 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-filter/${ec} 130 | echo ${file} >> ${RUNNING_CHECKS[0]}/sort-by-filter/${ec}/full.txt 131 | done 132 | done 133 | 134 | for ecd in $(ls ${RUNNING_CHECKS[0]}/sort-by-filter/); do 135 | gen_sort_main_v3 ${RUNNING_CHECKS[0]}/sort-by-filter/${ecd} 136 | gen_sort_pak_v3 ${RUNNING_CHECKS[0]}/sort-by-filter/${ecd} 137 | done 138 | 139 | copy_checks ${SCRIPT_TYPE} 140 | fi 141 | } 142 | 143 | array_names 144 | cd ${REPOTREE} 145 | export -f main array_names output_format 146 | export WORKDIR 147 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 148 | depth_set_v2 ${1} 149 | ${FILERESULTS} && rm -rf ${WORKDIR} 150 | -------------------------------------------------------------------------------- /deprecated/treehashgen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: treehashgen.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 23/05/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # script to generate hashes of the gentoo tree 25 | 26 | #override REPOTREE settings 27 | #REPOTREE=/usr/portage/ 28 | #REPO=gentoo 29 | #HASHTREE=/tmp/treehashtmp 30 | 31 | 32 | # check and set DEBUG/DRYRUN 33 | [ -z "${DEBUG}" ] && DEBUG=false 34 | [ -z "${DRYRUN}" ] && DRYRUN=false 35 | 36 | # check and set set the REPOTREE 37 | [ -z "${REPOTREE}" ] && DRYRUN=true 38 | [ -z "${REPO}" ] && DRYRUN=true 39 | [ -z "${HASHTREE}" ] && DRYRUN=true 40 | 41 | # 42 | ### IMPORTANT SETTINGS START ### 43 | # 44 | WORKDIR="/var/tmp/treehashgen/${REPO}" 45 | # 46 | ### IMPORTANT SETTINGS END ### 47 | # 48 | 49 | # DRYRUN - only print vars and exit 50 | if ${DRYRUN}; then 51 | echo treehashgen 52 | echo "Repo: ${REPO}" 53 | echo "Porttree: ${REPOTREE}" 54 | echo "Debug: ${DEBUG}" 55 | echo "Hashtree: ${HASHTREE}" 56 | exit 1 57 | fi 58 | 59 | # switch to the repotree first 60 | cd ${REPOTREE} 61 | 62 | hash_start(){ 63 | date_today="$(date -I)" 64 | mkdir -p ${HASHTREE} 65 | # only run if there doesn't exists a result for today 66 | if ! [ -e ${HASHTREE}/full-${date_today}.log ]; then 67 | mkdir -p ${WORKDIR} 68 | 69 | # generate hashes for every package 70 | # list every category 71 | local searchp="${REPOTREE}/*-*" 72 | local searchp=( $(find ${REPOTREE} -mindepth 1 -maxdepth 1 -type d -regextype sed -regex "./*[a-z0-9].*-[a-z0-9].*") ) 73 | [ -d "${REPOTREE}/virtual" ] && searchp+=( "${REPOTREE%/}/virtual" ) 74 | 75 | for cate in ${searchp[@]}; do 76 | for paka in $(find ${cate} -mindepth 1 -maxdepth 1 -type d); do 77 | mkdir -p ${WORKDIR}/${paka/${REPOTREE}/} 78 | # list all files in each directory and create hash 79 | find ${paka} -type f -exec xxh64sum {} \; > ${WORKDIR}/${paka/${REPOTREE}/}/package-xhash.xha 80 | #echo ${paka} >> /tmp/package-ng-new.log 81 | done 82 | done 83 | 84 | # generate hashes for every category based on the package hashes 85 | for cat in $(find ${WORKDIR} -mindepth 1 -maxdepth 1 -type d); do 86 | find ${cat} -mindepth 2 -type f -name *.xha -exec xxh64sum {} \; \ 87 | | tee -a ${cat}/category-xhash.xha ${HASHTREE}/full-${date_today}.log >/dev/null 88 | done 89 | # generate hash for the full tree 90 | find ${WORKDIR} -mindepth 2 -maxdepth 2 -type f -name *.xha -exec xxh64sum {} \; \ 91 | | tee -a ${WORKDIR}/tree-xhash.xha ${HASHTREE}/full-${date_today}.log >/dev/null 92 | # generate results file, based on the diffs from the full-last.log 93 | # if this file doesn't exist nothing happen. 94 | if [ -e ${HASHTREE}/full-last.log ]; then 95 | mkdir -p ${HASHTREE}/results 96 | touch ${HASHTREE}/results/results-${date_today}.log 97 | # list every category, save hashes of today and yesterday, compare and 98 | # if it doesn't match, check every package in that category 99 | for cat in $(find ${WORKDIR} -mindepth 1 -maxdepth 1 -type d); do 100 | cat_hash_last="$(grep ${cat}/category-xhash.xha ${HASHTREE}/full-last.log | cut -d ' ' -f1)" 101 | cat_hash_today="$(grep ${cat}/category-xhash.xha ${HASHTREE}/full-${date_today}.log| cut -d' ' -f1)" 102 | if ! [ "${cat_hash_last}" = "${cat_hash_today}" ]; then 103 | for pak in $(find ${cat} -mindepth 1 -maxdepth 1 -type d); do 104 | pak_hash_last="$(grep ${pak}/package-xhash.xha ${HASHTREE}/full-last.log | cut -d ' ' -f1)" 105 | pak_hash_today="$(grep ${pak}/package-xhash.xha ${HASHTREE}/full-${date_today}.log| cut -d' ' -f1)" 106 | if ! [ "${pak_hash_last}" = "${pak_hash_today}" ]; then 107 | echo "${pak/${WORKDIR}}" >> ${HASHTREE}/results/results-${date_today}.log 108 | fi 109 | done 110 | fi 111 | done 112 | fi 113 | rm -rf ${WORKDIR} 114 | fi 115 | } 116 | 117 | hash_start 118 | -------------------------------------------------------------------------------- /deprecated/wwwtest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: wwwtest 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 19/02/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simple scirpt to find broken websites 25 | 26 | #export FILERESULTS=true 27 | #export REPOTREE="/usr/portage/" 28 | #export RESULTSDIR="${HOME}/wwwtest/" 29 | # enabling debug output 30 | #export DEBUG=true 31 | #export DEBUGLEVEL=1 32 | #export DEBUGFILE=/tmp/repostats.log 33 | 34 | # get dirpath and load funcs.sh 35 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 36 | if [ -e ${realdir}/_funcs.sh ]; then 37 | source ${realdir}/_funcs.sh 38 | else 39 | echo "Missing _funcs.sh" 40 | exit 1 41 | fi 42 | 43 | # 44 | ### IMPORTANT SETTINGS START ### 45 | # 46 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 47 | #${ENABLE_MD5} || exit 0 # only works with md5 cache 48 | #${ENABLE_GIT} || exit 0 # only works with git tree 49 | 50 | SCRIPT_TYPE="checks" 51 | WORKDIR="/tmp/wwwtest-${RANDOM}" 52 | TMPFILE="/tmp/wwwtest-$(date +%y%m%d)-${RANDOM}.txt" 53 | TMPCHECK="/tmp/wwwtest-tmp-${RANDOM}.txt" 54 | JOBS="50" 55 | TIMEOUT="20" 56 | 57 | array_names(){ 58 | RUNNING_CHECKS=( 59 | "${WORKDIR}/ebuild_homepage_http_statuscode" #Index 0 60 | "${WORKDIR}/ebuild_homepage_301_redirections" #Index 1 61 | "${WORKDIR}/ebuild_homepage_redirection_missing_slash_www" #Index 2 62 | "${WORKDIR}/ebuild_homepage_redirection_http_to_https" #Index 3 63 | ) 64 | } 65 | output_format(){ 66 | index=( 67 | not_used0 68 | "${ebuild_eapi}${DL}${new_code}${DL}${cat}/${pak}${DL}${ebuild}${DL}${hp}${DL}${correct_site}${DL}${main}" 69 | "${ebuild_eapi}${DL}${cat}/${pak}${DL}${ebuild}${DL}${hp}${DL}${sitemut}${DL}${main}" 70 | "${ebuild_eapi}${DL}${cat}/${pak}${DL}${ebuild}${DL}${hp}${DL}${sitemut}${DL}${main}" 71 | ) 72 | echo "${index[$1]}" 73 | } 74 | data_descriptions(){ 75 | read -r -d '' info_index0 <<- EOM 76 | This checks tests every homepage and gets their http return code. The list contain packages with a bad returncode. 77 | Following statuscodes are ignored: VAR, FTP, 200, 301, 302, 307, 400, 503. 78 | Status Code History 79 | 80 | Data Format ( 7|404|dev-libs/foo|foo-1.12-r2.ebuild|https://foo.bar.com|dev@gentoo.org:loper@foo.de|754124:612230 ): 81 | 7 EAPI Version 82 | 404 http statuscode 83 | dev-libs/foo package category/name 84 | foo-1.12-r2.ebuild full filename 85 | https://foo.bar.com homepage corresponding to the statuscode 86 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 87 | 754124:612230 open bug ids related to this package 88 | EOM 89 | read -r -d '' info_index1 <<- EOM 90 | Lists ebuilds with a Homepage which actually redirects to another sites. 91 | 92 | Data Format ( 7|404|dev-libs/foo|foo-1.12-r2.ebuild|https://foo.bar.com|https://bar.foo.com|dev@gentoo.org:loper@foo.de ): 93 | 7 EAPI Version 94 | 404 http statuscode of redirected website 95 | dev-libs/foo package category/name 96 | foo-1.12-r2.ebuild full filename 97 | https://foo.bar.com original hommepage in ebuild 98 | https://bar.foo.com redirected homepage 99 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 100 | EOM 101 | read -r -d '' info_index2 <<- EOM 102 | Lists ebuild who's homepage redirects to the same site only including a "www" or a missing "/" at the end (or both) 103 | 104 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|https://foo.bar.com|https://foo.bar.com/|dev@gentoo.org:loper@foo.de ): 105 | 7 EAPI Version 106 | dev-libs/foo package category/name 107 | foo-1.12-r2.ebuild full filename 108 | https://foo.bar.com original hommepage in ebuild 109 | https://foo.bar.com/ same homepage, only with a slash at the end 110 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 111 | EOM 112 | read -r -d '' info_index3 <<- EOM 113 | Lists ebuids who's homepage redirects to the same site only via HTTPS. 114 | 115 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|http://foo.bar.com|https://foo.bar.com|dev@gentoo.org:loper@foo.de ): 116 | 7 EAPI Version 117 | dev-libs/foo package category/name 118 | foo-1.12-r2.ebuild full filename 119 | http://foo.bar.com original hommepage in ebuild 120 | https://foo.bar.com same homepage, only with https 121 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 122 | EOM 123 | 124 | description=( "${info_index0}" "${info_index1}" "${info_index2}" \ 125 | "${info_index3}" ) 126 | echo "${description[$1]}" 127 | } 128 | # 129 | ### IMPORTANT SETTINGS END ### 130 | # 131 | 132 | 133 | 301check() { 134 | # needed to get the names 135 | array_names 136 | 137 | local hp=${1} 138 | local cat=${2} 139 | local pak=${3} 140 | local main=${4} 141 | local ebuild="$(basename ${5})" 142 | local ebuild_eapi=${6} 143 | 144 | local found=false 145 | local lastchar="${hp: -1}" 146 | 147 | output(){ 148 | local checkid=${1} 149 | if ${FILERESULTS}; then 150 | output_format ${checkid} >> ${RUNNING_CHECKS[${checkid}]}/full.txt 151 | else 152 | output_format ${checkid} 153 | fi 154 | } 155 | 156 | if echo ${hp}|grep 'http://' > /dev/null; then 157 | 158 | _sitemuts=("${hp/http:\/\//https:\/\/}" \ 159 | "${hp/http:\/\//https:\/\/www.}") 160 | 161 | if ! [ "${lastchar}" = "/" ]; then 162 | _sitemuts+=("${hp/http:\/\//https:\/\/}/" \ 163 | "${hp/http:\/\//https:\/\/www.}/") 164 | fi 165 | 166 | for sitemut in ${_sitemuts[@]}; do 167 | local _code="$(get_code ${sitemut})" 168 | if [ ${_code} = 200 ]; then 169 | found=true 170 | output 3 171 | break 172 | fi 173 | done 174 | 175 | else 176 | 177 | _sitemuts_v2=("${hp/https:\/\//https:\/\/www.}" \ 178 | "${hp/http:\/\//http:\/\/www.}") 179 | 180 | if ! [ "${lastchar}" = "/" ]; then 181 | _sitemuts_v2+=("${hp}/" \ 182 | "${hp/https:\/\//https:\/\/www.}/" \ 183 | "${hp/http:\/\//http:\/\/www.}/") 184 | fi 185 | 186 | if ! ${found}; then 187 | for sitemut in ${_sitemuts_v2[@]}; do 188 | local _code="$(get_code ${sitemut})" 189 | if [ ${_code} = 200 ]; then 190 | found=true 191 | output 2 192 | break 193 | fi 194 | done 195 | fi 196 | fi 197 | 198 | if ! ${found}; then 199 | local correct_site="$(curl -Ls -o /dev/null --silent --max-time ${TIMEOUT} --head -w %{url_effective} ${hp})" 200 | new_code="$(get_code ${correct_site})" 201 | output 1 202 | fi 203 | } 204 | 205 | get_code() { 206 | local code="$(curl -o /dev/null --silent --max-time ${TIMEOUT} --head --write-out '%{http_code}\n' ${1})" 207 | echo ${code} 208 | } 209 | 210 | main() { 211 | mode() { 212 | local msg=${1} 213 | if ${FILERESULTS}; then 214 | echo "${msg}" >> ${TMPFILE} 215 | else 216 | echo "${msg}" 217 | fi 218 | } 219 | 220 | local full_package=${1} 221 | local category="$(echo ${full_package}|cut -d'/' -f1)" 222 | local package="$(echo ${full_package}|cut -d'/' -f2)" 223 | local maintainer="$(get_main_min "${category}/${package}")" 224 | local openbugs="$(get_bugs "${category}/${package}")" 225 | if ! [ -z "${openbugs}" ]; then 226 | openbugs="${DL}${openbugs}" 227 | fi 228 | 229 | for eb in ${REPOTREE}/${full_package}/*.ebuild; do 230 | 231 | local ebuild_eapi="$(get_eapi ${eb})" 232 | local ebuild=$(basename ${eb%.*}) 233 | 234 | if ${ENABLE_MD5}; then 235 | _hp="$(grep ^HOMEPAGE= ${REPOTREE}/metadata/md5-cache/${category}/${ebuild})" 236 | _hp="${_hp:9}" 237 | else 238 | _hp="$(grep ^HOMEPAGE= ${eb}|cut -d'"' -f2)" 239 | fi 240 | 241 | if [ -n "${_hp}" ]; then 242 | for i in ${_hp}; do 243 | local _checktmp="$(grep "${DL}${i}${DL}" ${TMPCHECK}|head -1)" 244 | 245 | if echo ${i}|grep ^ftp >/dev/null;then 246 | mode "${ebuild_eapi}${DL}FTP${DL}${category}/${package}${DL}${ebuild}${DL}${i}${DL}${maintainer}${openbugs}" 247 | elif echo ${i}|grep '${' >/dev/null; then 248 | mode "${ebuild_eapi}${DL}VAR${DL}${category}/${package}${DL}${ebuild}${DL}${i}${DL}${maintainer}${openbugs}" 249 | elif [ -n "${_checktmp}" ]; then 250 | # don't check again 251 | mode "${ebuild_eapi}${DL}${_checktmp:2:3}${DL}${category}/${package}${DL}${ebuild}${DL}${_checktmp:6:-1}${DL}${maintainer}${openbugs}" 252 | else 253 | # get http status code 254 | _code="$(get_code ${i})" 255 | mode "${ebuild_eapi}${DL}${_code}${DL}${category}/${package}${DL}${ebuild}${DL}${i}${DL}${maintainer}${openbugs}" 256 | echo "${ebuild_eapi}${DL}${_code}${DL}${i}${DL}" >> ${TMPCHECK} 257 | 258 | case ${_code} in 259 | 301) 260 | 301check "${i}" "${category}" "${package}" "${maintainer}" "${eb}" "${ebuild_eapi}" 261 | ;; 262 | esac 263 | 264 | fi 265 | done 266 | fi 267 | done 268 | } 269 | 270 | find_func() { 271 | find ${searchp[@]} -mindepth ${MIND} -maxdepth ${MAXD} \ 272 | -type d -print 2>/dev/null | parallel -j ${JOBS} main {} 273 | 274 | if ${FILERESULTS}; then 275 | gen_descriptions 276 | # sort after http codes (including all codes) 277 | for i in $(cat ${TMPFILE}|cut -d "${DL}" -f2|sort -u); do 278 | mkdir -p ${RUNNING_CHECKS[0]}/sort-by-filter/${i}/ 279 | grep "${DL}${i}${DL}" ${TMPFILE} > ${RUNNING_CHECKS[0]}/sort-by-filter/${i}/full.txt 280 | done 281 | 282 | # copy full log 283 | cp ${TMPFILE} ${RUNNING_CHECKS[0]}/full-unfiltered.txt 284 | 285 | # copy full log, ignoring "good" codes 286 | sed -i "/${DL}VAR${DL}/d; \ 287 | /${DL}FTP${DL}/d; \ 288 | /${DL}200${DL}/d; \ 289 | /${DL}301${DL}/d; \ 290 | /${DL}302${DL}/d; \ 291 | /${DL}307${DL}/d; \ 292 | /${DL}400${DL}/d; \ 293 | /${DL}503${DL}/d; \ 294 | /${DL}429${DL}/d; \ 295 | " ${TMPFILE} 296 | cp ${TMPFILE} ${RUNNING_CHECKS[0]}/full.txt 297 | 298 | sort_result_v4 299 | gen_sort_pak_v4 300 | gen_sort_main_v4 301 | 302 | copy_checks ${SCRIPT_TYPE} 303 | fi 304 | } 305 | 306 | array_names 307 | cd ${REPOTREE} 308 | # touch file first, otherwise the _checktmp could fail because of 309 | # the missing file 310 | touch ${TMPCHECK} 311 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 312 | # for parallel execution 313 | export -f main get_code 301check array_names output_format 314 | export TMPCHECK TMPFILE WORKDIR TIMEOUT 315 | if [ "${1}" = "diff" ]; then 316 | depth_set_v3 full 317 | else 318 | depth_set_v3 ${1} 319 | fi 320 | ${FILERESULTS} && rm ${TMPFILE} 321 | ${FILERESULTS} && rm -rf ${WORKDIR} 322 | rm ${TMPCHECK} 323 | -------------------------------------------------------------------------------- /exp/_old_funcs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: _funcs.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 03/06/2021 6 | 7 | # Copyright (C) 2021 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # these are functions which were used at some point but are not used anymore, 25 | # keep here for later usage. 26 | 27 | 28 | ### NOTUSED ### 29 | # returns the amout of bugs found - 3 digits long 30 | get_bugs_count(){ 31 | local value="${1}" 32 | local return="$(grep ${value} ${BUGTMPDIR}/full-$(date -I).txt | cut -d' ' -f2 | wc -l)" 33 | 34 | if [ -n "${return}" ]; then 35 | printf "%03d\n" "${return}" 36 | else 37 | echo "000" 38 | fi 39 | } 40 | 41 | ### NOTUSED ### 42 | # returns a list of Bug Numbers for a given ebuild 43 | get_bugs(){ 44 | local value="${1}" 45 | local return="$(grep ${value} ${BUGTMPDIR}/full-$(date -I).txt | cut -d' ' -f2 | tr '\n' ':')" 46 | 47 | [ -n "${return}" ] && echo "${return::-1}" 48 | } 49 | 50 | ### NOTUSED ### 51 | # return true or false if file exits remotly (or not) 52 | get_file_status(){ 53 | local uri="${1}" 54 | 55 | if $(timeout 15 wget -T 10 --no-check-certificate -q --method=HEAD ${uri}); then 56 | return 1 57 | else 58 | return 0 59 | fi 60 | } 61 | 62 | ### NOTUSED ### 63 | count_ebuilds(){ 64 | local epath="${1}" 65 | local return="$(find ${epath} -mindepth 1 -maxdepth 1 -type f -name "*.ebuild" | wc -l)" 66 | 67 | [ -n "${return}" ] && echo "${return}" 68 | } 69 | 70 | ### NOTUSED ### 71 | # this function get the age (file creation) of a particular ebuild file 72 | # depends on ${ENABLE_GIT} 73 | # returns the age in days 74 | get_age() { 75 | local file=${1} 76 | local date_today="$(date '+%s' -d today)" 77 | 78 | if ${ENABLE_GIT}; then 79 | fileage="$(expr \( "${date_today}" - \ 80 | "$(date '+%s' -d $(git -C ${REPOTREE} log --format="format:%ci" --name-only --diff-filter=A ${REPOTREE}/${file} \ 81 | | head -1|cut -d' ' -f1) 2>/dev/null )" \) / 86400 2>/dev/null)" 82 | printf "%05d\n" "${fileage}" 83 | else 84 | echo "-----" 85 | fi 86 | } 87 | 88 | ### NOTUSED ### 89 | get_age_v2() { 90 | local filedate="${1}" 91 | local date_today="$(date '+%s' -d today)" 92 | 93 | if ${ENABLE_GIT}; then 94 | if [ -n "${filedate}" ]; then 95 | fileage="$(expr \( "${date_today}" - "$(date '+%s' -d ${filedate})" \) / 86400 2>/dev/null)" 96 | printf "%05d\n" "${fileage}" 97 | else 98 | echo "-----" 99 | fi 100 | else 101 | echo "-----" 102 | fi 103 | } 104 | 105 | ### NOTUSED ### 106 | # list all eapi versions for a given package only showing used EAPIs. the list 107 | # looks like following: 108 | # 7(1):6(2):5(1) 109 | get_eapi_pak(){ 110 | local package=${1} 111 | local eapi_list=( $(grep EAPI ${package}/*.ebuild 2> /dev/null | cut -d'=' -f2 | cut -d' ' -f1 | grep -Eo '[0-9]' | sort | uniq -c | sed 's/^\s*//'|tr ' ' '_') ) 112 | 113 | local x 114 | local return_string=( ) 115 | for x in ${eapi_list[@]}; do 116 | local eapi=$(echo ${x}|rev|cut -d'_' -f1) 117 | local count=$(echo ${x}|cut -d'_' -f1) 118 | return_string+=( "${eapi}(${count})" ) 119 | done 120 | 121 | IFS=$'\n' return_string=($(sort -r <<<"${return_string[*]}")) 122 | 123 | echo "$(echo ${return_string[@]}|tr ' ' ':')" 124 | } 125 | 126 | ### NOTUSED ### 127 | # list all eapi's for a given package. the list looks like following: 128 | # EAPI Version: 0 1 2 3 4 5 6 7 (not outputed) 129 | # EAPI Count: 0:0:0:0:0:1:2:0 130 | get_eapi_list(){ 131 | local package=${1} 132 | local eapi_list=( ) 133 | for eapi in $(seq 0 7); do 134 | eapi_list+=( $(grep -h EAPI ${package}/*.ebuild |cut -d' ' -f1 | grep ${eapi}| wc -l) ) 135 | done 136 | echo "$(echo ${eapi_list[@]}|tr ' ' ':')" 137 | } 138 | 139 | ### NOTUSED ### 140 | # return all eclasses inherited by a ebuild 141 | # the list is generated from the md5-cache, which means it also includes 142 | # eclasses inherited by other eclasses 143 | get_eclasses_real() { 144 | local md5_file=${1} 145 | 146 | if ${ENABLE_MD5}; then 147 | local real_eclasses=( $(grep '_eclasses_=' ${md5_file}|cut -c12-|sed 's/\(\t[^\t]*\)\t/\1\n/g'|cut -d$'\t' -f1) ) 148 | echo ${real_eclasses[@]}|tr ' ' ':' 149 | fi 150 | } 151 | 152 | ### NOTUSED ### 153 | # list all eclasses used by a given ebuild file 154 | # returns the list as followed: 155 | # eclass1:eclass2 156 | # NOTE: metadata files also include a INHERIT variable, but for some reason not 157 | # for every ebuild. 158 | get_eclasses_file() { 159 | local md5_file=${1} 160 | local real_file=${2} 161 | 162 | if ${ENABLE_MD5}; then 163 | local real_eclasses=( $(grep '_eclasses_=' ${md5_file}|cut -c12-|sed 's/\(\t[^\t]*\)\t/\1\n/g'|cut -d$'\t' -f1) ) 164 | local file_eclasses=( ) 165 | local eclass_var="$(grep ^inherit ${real_file} |grep -o $\{.*\}|sed 's/${\(.*\)}/\1/')" 166 | if [ -n "${eclass_var}" ]; then 167 | eclass_in_var="$(grep -o "${eclass_var}=.*" ${real_file} | tail -n1 | tr -d '"' | cut -d '=' -f2 | cut -d ' ' -f1 )" 168 | if $(echo ${real_eclasses[@]}|grep -q ${eclass_in_var}); then 169 | file_eclasses+=( "${eclass_in_var}" ) 170 | fi 171 | fi 172 | for ecl in ${real_eclasses[@]}; do 173 | if $(sed -e :a -e '/\\$/N; s/\\\n//; s/\t/ /; ta' ${real_file} | grep inherit | grep -q " ${ecl} \\| ${ecl}\$"); then 174 | file_eclasses+=( ${ecl} ) 175 | fi 176 | done 177 | echo ${file_eclasses[@]}|tr ' ' ':' 178 | fi 179 | } 180 | -------------------------------------------------------------------------------- /exp/eclasscheck.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: patchtest 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 01/08/2016 6 | 7 | # Copyright (C) 2016 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simple scirpt to find unused patches in the gentoo portage tree 25 | 26 | if [ "$(hostname)" = methusalix ]; then 27 | PORTTREE="/usr/portage" 28 | script_mode=true 29 | script_mode_dir="/var/www/gentoo.levelnine.at/eclasscheck/" 30 | else 31 | PORTTREE="/mnt/data/gentoo/" 32 | script_mode=false 33 | fi 34 | cd ${PORTTREE} 35 | 36 | usage() { 37 | echo "You need an argument" 38 | } 39 | 40 | if [ -z "${1}" ]; then 41 | usage 42 | exit 1 43 | else 44 | if [ -d "${PORTTREE}/${1}" ]; then 45 | level="${1}" 46 | MAXD=0 47 | MIND=0 48 | _cat=${1%%/*} 49 | _pac=${1##*/} 50 | if [ -z "${_pac}" ] || [ "${_cat}" == "${_pac}" ]; then 51 | MAXD=1 52 | MIND=1 53 | fi 54 | elif [ "${1}" == "full" ]; then 55 | level="" 56 | MAXD=2 57 | MIND=2 58 | else 59 | echo "${PORTTREE}/${1}: Path not found" 60 | fi 61 | fi 62 | 63 | 64 | get_main_min(){ 65 | local ret=`/usr/bin/python3 - $1 </dev/null); then 91 | local _e_funcs="$(grep "@FUNCTION" ${ec} | cut -d ':' -f2|tr -d ' '|tr '\n' ' ')" 92 | for f in ${_e_funcs}; do 93 | if [ -n "$(grep "\<$f\>" $ebuild)" ]; then 94 | eclasses+=("$(basename ${ec%.eclass})") 95 | eclasses_ext+=("${f}:$(basename ${ec%.eclass})") 96 | #break 97 | echo " $package_name uses function $f from $(basename $ec)" 98 | fi 99 | done 100 | _e_vars="$(grep "@ECLASS-VARIABLE" ${ec} | cut -d ':' -f2|tr -d ' '|tr '\n' ' '|grep -v -E "${_filter_vars}")" 101 | for v in ${_e_vars}; do 102 | if [ -n "$(grep "\<$v\>" $ebuild)" ]; then 103 | eclasses+=("$(basename ${ec%.eclass})") 104 | eclasses_ext+=("${v}:$(basename ${ec%.eclass})") 105 | #break 106 | echo " $package_name uses variable $v from $(basename $ec)" 107 | fi 108 | done 109 | fi 110 | 111 | 112 | 113 | done 114 | # echo ${eclasses_ext[@]} 115 | 116 | m_ecl=() 117 | l_ecl=() 118 | for match in ${eclasses_ext[@]}; do 119 | func=${match%%:*} 120 | ecla=${match##*/} 121 | u=$(printf -- '%s\n' "${eclasses_ext[@]}"|grep "\<${func}\>:"|cut -d':' -f2) 122 | c=$(echo $u|sed 's/ /:/g') 123 | b=$(echo $u|sed 's/ /\|\|/g') 124 | # echo $u|sed 's/ /\|\|/g' 125 | m_ecl+=("${c}") 126 | l_ecl+=("${b}") 127 | #m_ecl+=("$(printf -- '%s\n' "${eclasses_ext[@]}"|grep "\<$func\>"|cut -d':' -f2)") 128 | # echo $m_ecl 129 | # echo 130 | done 131 | 132 | mapfile -t m_ecl < <(printf '%s\n' "${m_ecl[@]}"|sort -u) 133 | mapfile -t l_ecl < <(printf '%s\n' "${l_ecl[@]}"|sort -u) 134 | 135 | mapfile -t eclasses < <(printf '%s\n' "${eclasses[@]}"|sort -u) 136 | 137 | # ebuild_eclasses=("$(grep inherit ${ebuild}|cut -d ' ' -f2-)") 138 | ebuild_eclasses=() 139 | for i in $(grep inherit ${ebuild}|cut -d ' ' -f2-); do 140 | ebuild_eclasses+=("--${i}--") 141 | done 142 | #echo ${ebuild_eclasses[@]} 143 | tmp_ebuild_eclasses=("${ebuild_eclasses[@]}") 144 | missing_eclasses=() 145 | found=false 146 | for blub in ${m_ecl[@]}; do 147 | for e in $(echo $blub|tr ':' ' '); do 148 | #echo $e 149 | if $(echo ${ebuild_eclasses[@]}|grep "\-\-${e}\-\-" >/dev/null); then 150 | #echo "found $e" 151 | tmp_ebuild_eclasses=("${tmp_ebuild_eclasses[@]/\-\-$e\-\-}") 152 | found=true 153 | break 154 | fi 155 | done 156 | if ! $found; then 157 | #echo "not found ${blub}" 158 | missing_eclasses+=("${blub}") 159 | fi 160 | found=false 161 | done 162 | 163 | #echo " eclasses needed by functions: ${l_ecl[@]}" 164 | #echo " eclasses by ebuild: $(grep inherit ${ebuild}|cut -d ' ' -f2-)" 165 | # echo 166 | if [ -n "$(echo ${missing_eclasses[@]}|tr -d ' ')" ] || [ -n "$(echo ${tmp_ebuild_eclasses[@]}|tr -d ' ')" ]; then 167 | echo " eclasses missing: $(echo ${missing_eclasses[@]}|sed 's|:|\|\||g')" 168 | echo " eclasses to much: $(echo ${tmp_ebuild_eclasses[@]}|sed 's|--||g')" 169 | echo 170 | fi 171 | 172 | # echo " eclasses needed: ${eclasses[@]}" 173 | ebuild_eclasses=() 174 | tmp_ebuild_eclasses=() 175 | missing_eclasses=() 176 | eclasses=() 177 | } 178 | 179 | main() { 180 | 181 | category="$(echo ${package}|cut -d'/' -f2)" 182 | package_name=${line##*/} 183 | fullpath="/${PORTTREE}/${line}" 184 | 185 | for e in ${fullpath}/*.ebuild; do 186 | check_by_usage $e 187 | done 188 | 189 | } 190 | 191 | if $script_mode; then 192 | rm -rf ${script_mode_dir}/* 193 | fi 194 | 195 | find ./${level} -mindepth $MIND -maxdepth $MAXD \( \ 196 | -path ./scripts/\* -o \ 197 | -path ./profiles/\* -o \ 198 | -path ./packages/\* -o \ 199 | -path ./licenses/\* -o \ 200 | -path ./distfiles/\* -o \ 201 | -path ./metadata/\* -o \ 202 | -path ./eclass/\* -o \ 203 | -path ./.git/\* \) -prune -o -type d -print | while read -r line; do 204 | main ${line} 205 | done 206 | 207 | 208 | #ls -d ${cat}/${pac} |grep -E -v "distfiles|metadata|eclass" | while read -r line; do 209 | 210 | 211 | # # check if the patches folder exist 212 | # if [ -e ${fullpath}/files ]; then 213 | # if grep -E ".diff|.patch|FILESDIR|apache-module|elisp|vdr-plugin-2|games-mods|ruby-ng|readme.gentoo|readme.gentoo-r1|bzr|bitcoincore|gnatbuild|gnatbuild-r1|java-vm-2|mysql-cmake|mysql-multilib-r1|php-ext-source-r2|php-ext-source-r3|php-pear-r1|selinux-policy-2|toolchain-binutils|toolchain-glibc|x-modular" ${fullpath}/*.ebuild >/dev/null; then 214 | # continue 215 | # else 216 | # if $script_mode; then 217 | # main=$(get_main_min "${category}/${package_name}") 218 | # mkdir -p ${script_mode_dir}/sort-by-package/${category} 219 | # ls ${PORTTREE}/${category}/${package_name}/files/* > ${script_mode_dir}/sort-by-package/${category}/${package_name}.txt 220 | # echo "${category}/${package_name}" >> ${script_mode_dir}/full.txt 221 | # echo -e "${category}/${package_name}\t\t${main}" >> ${script_mode_dir}/full-with-maintainers.txt 222 | # 223 | # else 224 | # echo "${category}/${package_name}" 225 | # fi 226 | # fi 227 | # fi 228 | #done 229 | 230 | if $script_mode; then 231 | for a in $(cat ${script_mode_dir}/full-with-maintainers.txt |cut -d$'\t' -f3|tr ':' '\n'|tr ' ' '_'| grep -v "^[[:space:]]*$"|sort|uniq); do 232 | mkdir -p ${script_mode_dir}/sort-by-maintainer/ 233 | grep "${a}" ${script_mode_dir}/full-with-maintainers.txt > ${script_mode_dir}/sort-by-maintainer/"$(echo ${a}|sed "s|@|_at_|; s|gentoo.org|g.o|;")".txt 234 | done 235 | fi 236 | -------------------------------------------------------------------------------- /exp/obsoletfuncs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: obsoletfuncs.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 12/01/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # prototype script for new scripts 25 | 26 | #override PORTTREE,SCRIPT_MODE,SITEDIR settings 27 | #export PORTTREE=/usr/portage/ 28 | #export SCRIPT_MODE=true 29 | #export SITEDIR="${HOME}/obsoletfuncs/" 30 | 31 | # load repo specific settings 32 | startdir="$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" 33 | if [ -e ${startdir}/repo ]; then 34 | source ${startdir}/repo 35 | fi 36 | 37 | # get dirpath and load funcs.sh 38 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 39 | if [ -e ${realdir}/_funcs.sh ]; then 40 | source ${realdir}/_funcs.sh 41 | else 42 | echo "Missing _funcs.sh" 43 | exit 1 44 | fi 45 | 46 | # 47 | ### IMPORTANT SETTINGS START ### 48 | # 49 | SCRIPT_NAME="obsoletfuncs" 50 | SCRIPT_SHORT="OSF" 51 | WORKDIR="/tmp/${SCRIPT_NAME}-${RANDOM}" 52 | 53 | array_names(){ 54 | RUNNING_CHECKS=( 55 | "${WORKDIR}/${SCRIPT_SHORT}-BUG-ebuild_eapi6_funcs" #Index 0 56 | ) 57 | } 58 | array_names 59 | # 60 | ### IMPORTANT SETTINGS STOP ### 61 | # 62 | 63 | main() { 64 | array_names 65 | local relative_path=${1} 66 | local category="$(echo ${relative_path}|cut -d'/' -f2)" 67 | local package="$(echo ${relative_path}|cut -d'/' -f3)" 68 | local filename="$(echo ${relative_path}|cut -d'/' -f4)" 69 | local packagename="${filename%.*}" 70 | local full_path="${PORTTREE}/${category}/${package}" 71 | local full_path_ebuild="${PORTTREE}/${category}/${package}/${filename}" 72 | local maintainer="$(get_main_min "${category}/${package}")" 73 | 74 | if ${DEBUG}; then 75 | echo "relative path: ${relative_path}" # path relative to ${PORTTREE}: ./app-admin/salt/salt-0.5.2.ebuild 76 | echo "full path: ${full_path}" # full path: /usr/portage/app-admin/salt 77 | echo "full path ebuild: ${full_path_ebuild}" # full path ebuild: /usr/portage/app-admin/salt/salt-0.5.2.ebuild 78 | echo "category: ${category}" # package category: app-admin 79 | echo "package: ${package}" # package name: salt 80 | echo "filename: ${filename}" # package filename: salt-0.5.2.ebuild 81 | echo "packagename: ${packagename}" # package name-version: salt-0.5.2 82 | echo "fileage: $(get_age "${filename}")" # age of ebuild in days: 145 83 | echo "maintainer: ${maintainer}" # maintainer of package foo@gentoo.org:bar@gmail.com 84 | echo 85 | fi 86 | 87 | if [ "$(get_eapi ${full_path_ebuild})" = "6" ]; then 88 | if [ "$(sed -n '/src_install/,/^}/p' ${full_path_ebuild} | wc -l)" = 4 ]; then 89 | if $(sed -n '/src_install/,/^}/p' ${full_path_ebuild} | pcregrep -qM "emake.*DESTDIR.*\n.*default" ${full_path_ebuild}); then 90 | if ${SCRIPT_MODE}; then 91 | echo "${category}/${package}${DL}${filename}${DL}${maintainer}" >> ${RUNNING_CHECKS[0]}/full.txt 92 | else 93 | echo "${category}/${package}${DL}${filename}${DL}${maintainer}" 94 | fi 95 | fi 96 | fi 97 | fi 98 | } 99 | 100 | # set the search depth 101 | depth_set ${1} 102 | # switch to the PORTTREE dir 103 | cd ${PORTTREE} 104 | # export important variables 105 | export WORKDIR SCRIPT_SHORT 106 | export -f main array_names 107 | 108 | ${SCRIPT_MODE} && mkdir -p ${RUNNING_CHECKS[@]} 109 | 110 | find ./${level} \( \ 111 | -path ./scripts/\* -o \ 112 | -path ./profiles/\* -o \ 113 | -path ./packages/\* -o \ 114 | -path ./licenses/\* -o \ 115 | -path ./distfiles/\* -o \ 116 | -path ./metadata/\* -o \ 117 | -path ./eclass/\* -o \ 118 | -path ./.git/\* \) -prune -o -type f -name "*.ebuild" -exec egrep -l "\" {} \; | parallel main {} 119 | 120 | if ${SCRIPT_MODE}; then 121 | gen_sort_main_v2 ${RUNNING_CHECKS[0]} 5 122 | gen_sort_pak_v2 ${RUNNING_CHECKS[0]} 3 123 | 124 | copy_checks checks 125 | rm -rf ${WORKDIR} 126 | fi 127 | -------------------------------------------------------------------------------- /exp/patchcheckdir.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: patchcheckdir 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 07/08/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # simple scirpt to find unused scripts directories in the gentoo tree 25 | 26 | #override PORTTREE,SCRIPT_MODE,SITEDIR settings 27 | #export SCRIPT_MODE=true 28 | #export SITEDIR="${HOME}/patchcheckdir/" 29 | #export PORTTREE=/usr/portage/ 30 | 31 | # load repo specific settings 32 | startdir="$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)" 33 | if [ -e ${startdir}/repo ]; then 34 | source ${startdir}/repo 35 | fi 36 | 37 | # get dirpath and load funcs.sh 38 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 39 | if [ -e ${realdir}/_funcs.sh ]; then 40 | source ${realdir}/_funcs.sh 41 | else 42 | echo "Missing _funcs.sh" 43 | exit 1 44 | fi 45 | 46 | # 47 | ### IMPORTANT SETTINGS START ### 48 | # 49 | SCRIPT_NAME="patchcheckdir" 50 | SCRIPT_SHORT="PCD" 51 | WORKDIR="/tmp/${SCRIPT_NAME}-${RANDOM}/" 52 | 53 | array_names(){ 54 | RUNNING_CHECKS=( 55 | "${WORKDIR}/${SCRIPT_SHORT}-BUG-ebuild_unused_patches_folder" #Index 0 56 | ) 57 | } 58 | array_names 59 | # 60 | ### IMPORTANT SETTINGS STOP ### 61 | # 62 | 63 | _gen_whitelist(){ 64 | if [ -e ${startdir}/whitelist ]; then 65 | source ${startdir}/whitelist 66 | for i in ${white_list[@]}; do 67 | whitelist+=("$(echo ${i}|cut -d';' -f1)") 68 | done 69 | else 70 | whitelist=() 71 | fi 72 | # remove duplicates 73 | mapfile -t whitelist < <(printf '%s\n' "${whitelist[@]}"|sort -u) 74 | echo ${whitelist[@]} 75 | } 76 | 77 | main(){ 78 | array_names 79 | local absolute_path=${1} 80 | local category="$(echo ${absolute_path}|cut -d'/' -f2)" 81 | local package="$(echo ${absolute_path}|cut -d'/' -f3)" 82 | local filename="$(echo ${absolute_path}|cut -d'/' -f4)" 83 | local packagename="${filename%.*}" 84 | local full_path="${PORTTREE}/${category}/${package}" 85 | local full_path_ebuild="${PORTTREE}/${category}/${package}/${filename}" 86 | local maintainer="$(get_main_min "${category}/${package}")" 87 | 88 | # check if the patches folder exist 89 | if [ -e ${full_path}/files ]; then 90 | if ! $(echo ${whitelist[@]}|grep "${category}/${package}" > /dev/null); then 91 | for entry in ${full_path}/files/*; do 92 | if [ -d ${entry} ]; then 93 | echo ${entry} 94 | fi 95 | done 96 | fi 97 | fi 98 | } 99 | 100 | depth_set ${1} 101 | cd ${PORTTREE} 102 | export -f main get_main_min array_names 103 | export WORKDIR startdir SCRIPT_SHORT 104 | export whitelist=$(_gen_whitelist) 105 | ${SCRIPT_MODE} && mkdir -p ${RUNNING_CHECKS[@]} 106 | 107 | find ./${level} -mindepth ${MIND} -maxdepth ${MAXD} \( \ 108 | -path ./scripts/\* -o \ 109 | -path ./profiles/\* -o \ 110 | -path ./packages/\* -o \ 111 | -path ./licenses/\* -o \ 112 | -path ./distfiles/\* -o \ 113 | -path ./metadata/\* -o \ 114 | -path ./eclass/\* -o \ 115 | -path ./.git/\* \) -prune -o -type d -print | parallel main {} 116 | 117 | if ${SCRIPT_MODE}; then 118 | gen_sort_main_v2 ${RUNNING_CHECKS[0]} 2 119 | 120 | copy_checks checks 121 | rm -rf ${WORKDIR} 122 | fi 123 | -------------------------------------------------------------------------------- /genlists.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: genlists.sh 4 | # Autor: Michael Mair-Keimberger (mmk AT levelnine DOT at) 5 | # Date: 12/11/2017 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # create full lists per packages and maintainers, replaces full_gen 25 | 26 | #override RESULTSDIR settings 27 | #export RESULTSDIR="${HOME}/st/guru/" 28 | 29 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 30 | if [ -e ${realdir}/_funcs.sh ]; then 31 | source ${realdir}/_funcs.sh 32 | else 33 | echo "Missing _funcs{-httpgen}.sh" 34 | exit 1 35 | fi 36 | 37 | # 38 | ### IMPORTANT SETTINGS START ### 39 | # 40 | WORKDIR="/tmp/full-gen-${RANDOM}" 41 | # 42 | ### IMPORTANT SETTINGS END ### 43 | # 44 | 45 | # switch to RESULTSDIR to avoid access denied errors 46 | cd ${RESULTSDIR} 47 | 48 | mkdir -p ${WORKDIR}/{sort-by-package,sort-by-maintainer} 49 | 50 | ### full list generation start ### 51 | # 52 | # full list gen - maintainer sorting 53 | for check in $(find ${RESULTSDIR}/checks/ -mindepth 1 -maxdepth 1 -type d -print|sort); do 54 | if [ -d ${check}/sort-by-maintainer/ ]; then 55 | for main in $(ls ${check}/sort-by-maintainer/); do 56 | echo "<<< ${check##*/} >>>" >> ${WORKDIR}/sort-by-maintainer/${main} 57 | cat ${check}/sort-by-maintainer/${main} | sed 's/^/ /g' >> ${WORKDIR}/sort-by-maintainer/${main} 58 | done 59 | fi 60 | done 61 | 62 | # full list gen - package sorting 63 | for check in $(find ${RESULTSDIR}/checks/ -mindepth 1 -maxdepth 1 -type d -print|sort); do 64 | if [ -d ${check}/sort-by-package/ ]; then 65 | for cat in $(ls ${check}/sort-by-package/); do 66 | mkdir -p ${WORKDIR}/sort-by-package/${cat} 67 | for pack in $(ls ${check}/sort-by-package/${cat}/); do 68 | echo "<<< ${check##*/} >>>" >> ${WORKDIR}/sort-by-package/${cat}/${pack} 69 | cat ${check}/sort-by-package/${cat}/${pack} | sed 's/^/ /g' >> ${WORKDIR}/sort-by-package/${cat}/${pack} 70 | done 71 | done 72 | fi 73 | done 74 | 75 | # add bug information to the packages 76 | for cat in $(ls ${WORKDIR}/sort-by-package/); do 77 | for pack in $(ls ${WORKDIR}/sort-by-package/${cat}/); do 78 | openbugs="$(get_bugs_full "${cat}/${pack::-4}")" 79 | # only add openbugs information when they are any open bugs 80 | if [ -n "${openbugs}" ]; then 81 | echo "<<< open bugs >>>" >> ${WORKDIR}/sort-by-package/${cat}/${pack} 82 | echo "${openbugs}" | sed 's/^/ /g' >> ${WORKDIR}/sort-by-package/${cat}/${pack} 83 | fi 84 | done 85 | done 86 | ### full list generation end ### 87 | 88 | # remove old listings and replace it with the new one 89 | if [ -e "${RESULTSDIR}/listings/" ]; then 90 | rm -rf ${RESULTSDIR}/listings/* 91 | else 92 | mkdir -p "${RESULTSDIR}/listings/" 93 | fi 94 | cp -r ${WORKDIR}/* ${RESULTSDIR}/listings/ 95 | rm -rf ${WORKDIR} 96 | -------------------------------------------------------------------------------- /gpl-2.0.txt: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | 294 | Copyright (C) 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | , 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /qa.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # logfile 4 | LOGFILE="/tmp/qa-scripts.log" 5 | ERRFILE="/tmp/qa-scripts-err.log" 6 | logstd() { while IFS='' read -r line; do echo "($(date '+%Y-%m-%d %H:%M')) ${0##*/} Info: $line" >> ${LOGFILE}; done } 7 | logerr() { while IFS='' read -r line; do echo "($(date '+%Y-%m-%d %H:%M')) ${0##*/} Error: $line" >> ${ERRFILE}; done } 8 | 9 | # define repos which will be checked 10 | # the name correspond to the github/gentoo-mirror/${REPO} name and must exist. 11 | # the website correspond to the official/mirrored website and can be anything 12 | # (not needed for and script related doings) 13 | REPOSITORIES=( 14 | "gentoo|https://github.com/gentoo/gentoo" 15 | "kde|https://github.com/gentoo/kde" 16 | "guru|https://github.com/gentoo/guru" 17 | "science|https://github.com/gentoo/sci" 18 | "pentoo|https://github.com/pentoo/pentoo-overlay" 19 | ) 20 | 21 | # enable diffmode per default 22 | DIFFMODE=true 23 | # write DB entries 24 | DBWRITE=true 25 | # create website data 26 | SITEGEN=true 27 | # remove previous log file 28 | CLEANLOG=true 29 | # set directory were the scripts are 30 | SCRIPTDIR="/home/bob/qa/" 31 | # set directory were qa data resides 32 | QADIR="/mnt/data/qa/gentooqa.levelnine.at/" 33 | 34 | # 35 | ## important settings 36 | # global variables 37 | export DRYRUN=false 38 | export DEBUG=false 39 | export FILERESULTS=true 40 | export TIMELOG="/tmp/qa-time-$(date -I).log" 41 | export SITEDIR="${QADIR}/gentooqa/www/" 42 | # gentoo main tree directory, requried for certain checks 43 | export GTREE="${QADIR}/repos/gentoo/" 44 | export GITINFO="${SCRIPTDIR}/gitinfo" 45 | mkdir -p "${GITINFO}" 46 | # testvars 47 | #export SITEDIR="/tmp/wwwsite/" 48 | 49 | ${CLEANLOG} && rm ${LOGFILE} ${ERRFILE} 50 | cd ${SCRIPTDIR} 51 | 52 | for repodir in ${REPOSITORIES[@]}; do 53 | # set important variables 54 | export REPO="$(echo ${repodir%%|*})" 55 | REPOLINK="https://github.com/gentoo-mirror/${REPO}" 56 | export RESULTSDIR="/${SITEDIR}/results/${REPO}/" 57 | export REPOTREE="${QADIR}/repos/${REPO}/" 58 | export PT_WHITELIST="${REPO}-whitelist" 59 | # testvars 60 | #export RESULTSDIR="/${SITEDIR}/results/${REPO}/" 61 | #export REPOTREE="/${QADIR}/repos/${REPO}/" 62 | 63 | echo -e "\nChecking ${REPO}\n" | (logstd) 64 | # the repositories need to exists in order to be updated 65 | # check if directory exists 66 | # > if not, create dir, clone repo 67 | if ! [ -d "${REPOTREE}" ]; then 68 | mkdir -p "${REPOTREE}" 69 | git clone ${REPOLINK} ${REPOTREE} 1> /dev/null 2> >(logerr) 70 | # directory exists but is empty 71 | # > clone repo 72 | elif [ -z "$(ls -A ${REPOTREE})" ]; then 73 | git clone ${REPOLINK} ${REPOTREE} 1> /dev/null 2> >(logerr) 74 | # repo exists, sync it 75 | else 76 | if $(git -C ${REPOTREE} status 1> >(logstd) 2> >(logerr)); then 77 | # if the keephead file exists, don't update repo-head file 78 | # this way an older head (for example form the day before) could be used 79 | # to run the scripts. this file will be removed at the end. 80 | if [ -e "/tmp/keephead" ]; then 81 | git -C ${REPOTREE} pull 1> /dev/null 2> >(logerr) 82 | else 83 | git -C ${REPOTREE} rev-parse HEAD > ${GITINFO}/${REPO}-head 84 | git -C ${REPOTREE} pull 1> /dev/null 2> >(logerr) 85 | fi 86 | else 87 | echo "Error syncing ${REPO} git tree. Exiting" | (logstd) 88 | exit 89 | fi 90 | fi 91 | 92 | if [ -s "${GITINFO}/${REPO}-head" ]; then 93 | echo -e "\nFind changed packages for ${REPO}" | (logstd) 94 | git -C ${REPOTREE} diff --name-only $(<${GITINFO}/${REPO}-head) HEAD \ 95 | | cut -d'/' -f1,2|sort -u|grep -e '\([a-z0-9].*-[a-z0-9].*/\|virtual/\)' \ 96 | >> ${GITINFO}/${REPO}-catpak.log 97 | echo -e "\nFind removed packages for ${REPO}" | (logstd) 98 | git -C ${REPOTREE} diff --diff-filter=D --summary $(<${GITINFO}/${REPO}-head) HEAD \ 99 | | grep metadata.xml | cut -d' ' -f5 \ 100 | >> ${GITINFO}/${REPO}-catpak-rm.log 101 | else 102 | DIFFMODE=false 103 | fi 104 | 105 | echo -e "\nUpdate pkgcheck cache for ${REPO}" | (logstd) 106 | pkgcheck cache -r ${REPOTREE} -uf | (logstd) 107 | 108 | scripts_diff="repostats.sh repochecks.sh" 109 | for s_v2 in ${scripts_diff}; do 110 | printf "${s_v2}|" >> ${TIMELOG} 111 | echo "Processing script: ${s_v2}" | (logstd) 112 | export SCRIPT_NAME=${s_v2%%.*} 113 | # if /tmp/${SCRIPT_NAME} exist run in normal mode this way it's possible 114 | # to override the diff mode this is usefull when the script got updates 115 | # which should run on the whole tree 116 | if ${DIFFMODE} && ! [[ -e "/tmp/${SCRIPT_NAME}" ]]; then 117 | /usr/bin/time -q -f %e -a -o ${TIMELOG} ${SCRIPTDIR}/${s_v2} diff 1> >(logstd) 2> >(logerr) 118 | else 119 | /usr/bin/time -q -f %e -a -o ${TIMELOG} ${SCRIPTDIR}/${s_v2} full 1> >(logstd) 2> >(logerr) 120 | fi 121 | done 122 | 123 | echo -e "\nFinishing checks for ${REPO}\n" | (logstd) 124 | mv ${GITINFO}/${REPO}-catpak.log ${GITINFO}/${REPO}-changes-$(date -I).log 125 | mv ${GITINFO}/${REPO}-catpak-rm.log ${GITINFO}/${REPO}-deletes-$(date -I).log 126 | find ${GITINFO} -name "${REPO}-changes-*" -type f -printf '%T@ %p\n' \ 127 | | sort -k1 -n | head -n-7 | cut -d' ' -f2 | xargs -r rm 128 | find ${GITINFO} -name "${REPO}-deletes-*" -type f -printf '%T@ %p\n' \ 129 | | sort -k1 -n | head -n-7 | cut -d' ' -f2 | xargs -r rm 130 | 131 | # create full package/maintainer lists 132 | echo "Processing script: genlists" | (logstd) 133 | ${SCRIPTDIR}/genlists.sh 1> >(logstd) 2> >(logerr) 134 | 135 | # write results into database 136 | # must be done while the for loop since benchmark statistics are being 137 | # overwriten each run 138 | if ${DBWRITE}; then 139 | SITESCRIPTS=$(dirname ${SITEDIR}) 140 | echo -e "\nCreating Database Entries for ${REPO}\n" | (logstd) 141 | ${SITESCRIPTS}/dbinsert.sh 1> >(logstd) 2> >(logerr) 142 | fi 143 | 144 | rm ${TIMELOG} 145 | done 146 | 147 | echo -e "\nFinish with checking all repos\n" | (logstd) 148 | if ${SITEGEN}; then 149 | export REPOS="$(echo ${REPOSITORIES[@]})" 150 | SITESCRIPTS=$(dirname ${SITEDIR}) 151 | echo -e "Generating HTML output:\n" | (logstd) 152 | ${SITESCRIPTS}/sitegen.sh 1> >(logstd) 2> >(logerr) 153 | fi 154 | 155 | echo -e "Finish generating HTML output\n" | (logstd) 156 | 157 | # with /tmp/${scriptname} it's possible to override the default DIFFMODE to 158 | # force a full run. Since this should only be done once, we remove existings 159 | # files so that next time the default settings is used again 160 | for diff_s in ${scripts_diff}; do 161 | rm -f /tmp/${diff_s%.*} 162 | done 163 | # the same as with the temporay scriptname file, remove /tmp/keephead 164 | rm -f /tmp/keephead 165 | 166 | echo -e "\nDONE" | (logstd) 167 | -------------------------------------------------------------------------------- /repostats.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: repostats.sh 4 | # Autor: Michael Mair-Keimberger (mmk AT levelnine DOT at) 5 | # Date: 09/05/2021 6 | 7 | # Copyright (C) 2021 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # script for generating statistics regarding gentoo repositories 25 | 26 | # override REPOTREE,FILERESULTS,RESULTSDIR settings 27 | #export REPOTREE=/usr/portage/ 28 | #export FILERESULTS=true 29 | #export RESULTSDIR="${HOME}/$(basename ${0})/" 30 | # enabling debug output 31 | #export DEBUG=true 32 | #export DEBUGLEVEL=1 33 | #export DEBUGFILE=/tmp/$(basename ${0}).log 34 | 35 | # get dirpath and load funcs.sh 36 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 37 | if [ -e ${realdir}/_funcs.sh ]; then 38 | source ${realdir}/_funcs.sh 39 | else 40 | echo "Missing _funcs.sh" 41 | exit 1 42 | fi 43 | 44 | # 45 | ### IMPORTANT SETTINGS START ### 46 | # 47 | #${TREE_IS_MASTER} || exit 0 # only works with gentoo main tree 48 | ${ENABLE_MD5} || exit 0 # md5 cache is required 49 | ${ENABLE_GIT} || exit 0 # git tree is required 50 | 51 | SCRIPT_TYPE="stats" 52 | WORKDIR="/tmp/$(basename ${0})-${RANDOM}" 53 | 54 | array_names(){ 55 | SELECTED_CHECKS=( 56 | rs_eaps 57 | rs_livs 58 | rs_ecls 59 | rs_lics 60 | rs_keys 61 | rs_vius 62 | rs_clec rs_stac 63 | rs_ggrs 64 | rs_guss 65 | ) 66 | declare -gA FULL_CHECKS=( 67 | [rs_eaps]="${WORKDIR}/ebuild_eapi_statistics" 68 | [rs_livs]="${WORKDIR}/ebuild_live_statistics" 69 | [rs_ecls]="${WORKDIR}/ebuild_eclass_statistics" 70 | [rs_lics]="${WORKDIR}/ebuild_licenses_statistics" 71 | [rs_keys]="${WORKDIR}/ebuild_keywords_statistics" 72 | [rs_vius]="${WORKDIR}/ebuild_virtual_use_statistics" 73 | [rs_clec]="${WORKDIR}/ebuild_cleanup_candidates" 74 | [rs_stac]="${WORKDIR}/ebuild_stable_candidates" 75 | [rs_ggrs]="${WORKDIR}/ebuild_glep81_group_statistics" 76 | [rs_guss]="${WORKDIR}/ebuild_glep81_user_statistics" 77 | ) 78 | } 79 | 80 | var_descriptions(){ 81 | read -r -d '' info_default0 <<- EOM 82 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|dev@gentoo.org:loper@foo.de ): 83 | 7 EAPI Version 84 | dev-libs/foo package category/name 85 | foo-1.12-r2.ebuild full filename 86 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 87 | EOM 88 | read -r -d '' info_default1 <<- EOM 89 | Data Format ( 6|01865|01311|6|00649|-----|+|dev-libs/foo|foo-1.12-r1|foo-1.12-r2|dev@gentoo.org:loper@foo.de ): 90 | 6|01865|01311| EAPI Version (older) | days since created | days since last modified 91 | 6|00649|-----| EAPI Version (newer) | days since created | days since last modified 92 | + indicates if bugs are open to this package (+=yes, -=no) 93 | dev-libs/foo package category/name 94 | foo-1.12-r1.ebuild full filename (older) 95 | foo-1.12-r2.ebuild full filename (newer) 96 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 97 | EOM 98 | read -r -d '' rs_eaps <<- EOM 99 | A simple list of all ebuilds with it's corresponding EAPI Version. Also includes all maintainers to the package 100 | EAPI Statistics 101 | 102 | ${info_default0} 103 | EOM 104 | read -r -d '' rs_livs <<- EOM 105 | A simple list of all live ebuilds and it's corresponding EAPI Version and maintainer(s). 106 | 107 | ${info_default0} 108 | EOM 109 | read -r -d '' rs_ecls <<- EOM 110 | Lists the eclasses used by every ebuild. 111 | Not including packages which don't inherit anything. Also not included are eclasses inherited by other eclasses. 112 | 113 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|eutils:elisp|dev@gentoo.org:loper@foo.de ): 114 | 7 EAPI Version 115 | dev-libs/foo package category/name 116 | foo-1.12-r2.ebuild full filename 117 | eutils:elisp eclasses which the ebuild inherit (implicit inherit not included) 118 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 119 | EOM 120 | read -r -d '' rs_lics <<- EOM 121 | Lists the licenses used by every ebuild (not taking contional licenses into account). 122 | 123 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|GPL-2:BSD|dev@gentoo.org:loper@foo.de ): 124 | 7 EAPI Version 125 | dev-libs/foo package category/name 126 | foo-1.12-r2.ebuild full filename 127 | GPL-2:BSD licenses used the ebuild, seperated by ':' 128 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 129 | EOM 130 | read -r -d '' rs_keys <<- EOM 131 | Lists the keywords used by every ebuild. 132 | 133 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|~amd64:x86|dev@gentoo.org:loper@foo.de ): 134 | 7 EAPI Version 135 | dev-libs/foo package category/name 136 | foo-1.12-r2.ebuild full filename 137 | ~amd64:x86 keywords used by the ebuild, seperated by ':' 138 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 139 | EOM 140 | read -r -d '' rs_vius <<- EOM 141 | Lists virtual usage by ebuilds. 142 | 143 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|virtual/ooo|dev@gentoo.org:loper@foo.de ): 144 | 7 EAPI Version 145 | dev-libs/foo package category/name 146 | foo-1.12-r2.ebuild full filename 147 | virtual/ooo virtual(s) used by this ebuild, seperated by ':' 148 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 149 | EOM 150 | read -r -d '' rs_clec <<- EOM 151 | This script searches if there is a newer revision (-rX) available. In case a newer revision is found KEYWORDS are 152 | gonna be checked as well. If both keywords are the same for both ebuilds the older one is considered as a removal 153 | candidate. 154 | 155 | ${info_default1} 156 | EOM 157 | read -r -d '' rs_stac <<- EOM 158 | This script searches if there is a newer revision (-rX) available. In case a newer revision is found KEYWORDS are 159 | gonna be checked as well. In case keywords differ, the newer ebuild is considered as a candidate for 160 | stablization. 161 | 162 | ${info_default1} 163 | EOM 164 | read -r -d '' rs_ggrs <<- EOM 165 | Lists acct-group/* usage by ebuilds. 166 | 167 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|acct-group/gdm|dev@gentoo.org:loper@foo.de ): 168 | 7 EAPI Version 169 | dev-libs/foo package category/name 170 | foo-1.12-r2.ebuild full filename 171 | acct-group/gdm group(s) used by this ebuild, seperated by ':' 172 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 173 | EOM 174 | read -r -d '' rs_guss <<- EOM 175 | Lists acct-user/* usage by ebuilds. 176 | 177 | Data Format ( 7|dev-libs/foo|foo-1.12-r2.ebuild|acct-user/gdm|dev@gentoo.org:loper@foo.de ): 178 | 7 EAPI Version 179 | dev-libs/foo package category/name 180 | foo-1.12-r2.ebuild full filename 181 | acct-user/gdm user(s) used by this ebuild, seperated by ':' 182 | dev@gentoo.org:loper@foo.de maintainer(s), seperated by ':' 183 | EOM 184 | } 185 | # 186 | ### IMPORTANT SETTINGS END ### 187 | # 188 | 189 | main() { 190 | array_names 191 | [ ${DEBUGLEVEL} -ge 2 ] && echo "generating standard information for ${1}" | (debug_output) 192 | 193 | # everything below $min_allow_eapi is considered deprecated 194 | local min_allow_eapi=6 195 | 196 | local rel_path=${1} # path relative to ${REPOTREE}: ./app-admin/salt/salt-0.5.2.ebuild 197 | local cat="$(echo ${rel_path}|cut -d'/' -f1)" # package category: app-admin 198 | local pak="$(echo ${rel_path}|cut -d'/' -f2)" # package name: salt 199 | local filename="$(echo ${rel_path}|cut -d'/' -f3)" # package filename: salt-0.5.2.ebuild 200 | local pakname="${filename%.*}" # package name-version: salt-0.5.2 201 | local pakver="${pakname/${pak}-/}" # package version 0.5.2 202 | local abs_path="${REPOTREE}/${cat}/${pak}" # full path: /usr/portage/app-admin/salt 203 | local abs_path_ebuild="${REPOTREE}/${cat}/${pak}/${filename}" # full path ebuild: /usr/portage/app-admin/salt/salt-0.5.2.ebuild 204 | 205 | [ ${DEBUGLEVEL} -ge 2 ] && echo "generating detailed information for ${1}" | (debug_output) 206 | local maintainer="$(get_main_min "${cat}/${pak}")" # maintainer of package: foo@gentoo.org:bar@gmail.com 207 | local ebuild_eapi="$(get_eapi ${rel_path})" # eapi of ebuild: 6 208 | local ebuild_eclasses="$(get_eclasses "${cat}/${pak}/${pakname}")" # elasses inherited by ebuild: pam:udev 209 | local ebuild_licenses="$(get_licenses "${cat}/${pakname}")" # licenses set by ebuild: GPL2+:BSD-2 210 | local ebuild_keywords="$(get_keywords_v2 "${cat}/${pakname}")" # keywords set by ebuild: amd64:x86 211 | local ebuild_depend="$(get_depend "${cat}/${pak}/${pakname}")" # dependencies set by ebuild: dev-libs/gdl:app-admin/diradm 212 | 213 | if [ "${pakname: -3}" = "-r${pakname: -1}" ]; then 214 | start=$(expr ${pakname: -1} + 1) 215 | local org_name=${pakname} 216 | local norm_name=${pakname::-3} 217 | else 218 | start=1 219 | local org_name=${pakname} 220 | local norm_name=${pakname} 221 | fi 222 | 223 | output_formats(){ 224 | declare -gA array_formats=( 225 | [def0]="${ebuild_eapi}${DL}${cat}/${pak}${DL}${filename}${DL}${maintainer}" 226 | [def1]="${ebuild_eapi}${DL}${cat}/${pak}${DL}${filename}${DL}${results1}${DL}${maintainer}" 227 | [def2]="${ebuild_eapi}${DL}${oec}${DL}${oelm}${DL}${eapi_found_ebuild}${DL}${nec}${DL}${nelm}${DL}${package_bugs}${DL}${cat}/${pak}${DL}${org_name}${DL}${norm_name}-r${i}${DL}${maintainer}" 228 | ) 229 | echo "${array_formats[${1}]}" 230 | } 231 | 232 | output(){ 233 | local output="${1}" 234 | local file="${FULL_CHECKS[${2}]}" 235 | if ${FILERESULTS}; then 236 | output_formats ${output} >> ${file}/full.txt 237 | else 238 | echo "${file##*/}${DL}$(output_formats ${output})" 239 | fi 240 | } 241 | 242 | # eapi statistics [rs_eaps] 243 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_eaps " ]]; then 244 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_eaps]/${WORKDIR}\/}" | (debug_output) 245 | output def0 rs_eaps 246 | fi 247 | 248 | # keywords statistics [rs_keys] 249 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_keys " ]]; then 250 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_keys]/${WORKDIR}\/}" | (debug_output) 251 | results1="${ebuild_keywords}" 252 | output def1 rs_keys 253 | fi 254 | 255 | # license statistics [rs_lics] 256 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_lics " ]]; then 257 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_lics]/${WORKDIR}\/}" | (debug_output) 258 | local results1="${ebuild_licenses}" 259 | [[ -n "${results1}" ]] && output def1 rs_lics 260 | fi 261 | 262 | # inherited eclasses statistics [rs_ecls] 263 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_ecls " ]]; then 264 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_ecls]/${WORKDIR}\/}" | (debug_output) 265 | results1="${ebuild_eclasses}" 266 | [[ -n "${results1}" ]] && output def1 rs_ecls 267 | fi 268 | 269 | # live ebuilds [rs_livs] 270 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_livs " ]]; then 271 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_livs]/${WORKDIR}\/}" | (debug_output) 272 | if $(echo ${pakver}|grep -q 9999) && [[ "${ebuild_keywords}" = "none" ]]; then 273 | output def0 rs_livs 274 | fi 275 | fi 276 | 277 | if [ -n "${ebuild_depend}" ]; then 278 | # virtual usage statistics [rs_vius] 279 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_vius " ]]; then 280 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_vius]/${WORKDIR}\/}" | (debug_output) 281 | local vir_tmp=( $(echo ${ebuild_depend}|tr ':' '\n'|grep "virtual/"|cut -d'/' -f2) ) 282 | local results1="$(echo ${vir_tmp[@]}|tr ' ' ':')" 283 | [ -n "${results1}" ] && output def1 rs_vius 284 | fi 285 | # acct group statistics [rs_ggrs] 286 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_ggrs " ]]; then 287 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_ggrs]/${WORKDIR}\/}" | (debug_output) 288 | local group_tmp=( $(echo ${ebuild_depend}|tr ':' '\n'|grep "acct-group/"|cut -d'/' -f2) ) 289 | local results1="$(echo ${group_tmp[@]}|tr ' ' ':')" 290 | [ -n "${results1}" ] && output def1 rs_ggrs 291 | fi 292 | # acct user statistics [rs_guss] 293 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_guss " ]]; then 294 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_guss]/${WORKDIR}\/}" | (debug_output) 295 | local user_tmp=( $(echo ${ebuild_depend}|tr ':' '\n'|grep "acct-user/"|cut -d'/' -f2) ) 296 | local results1="$(echo ${user_tmp[@]}|tr ' ' ':')" 297 | [ -n "${results1}" ] && output def1 rs_guss 298 | fi 299 | fi 300 | 301 | # cleanup/stable candidates [rs_clec & rs_stac] 302 | if [[ " ${SELECTED_CHECKS[*]} " =~ " rs_clec " ]] || [[ " ${SELECTED_CHECKS[*]} " =~ " rs_stac " ]]; then 303 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "checking for ${FULL_CHECKS[rs_clec]/${WORKDIR}\/} and ${FULL_CHECKS[rs_clec]/${WORKDIR}\/}" | (debug_output) 304 | local i 305 | # check for maximal 9 reversion 306 | for i in $(seq $start 9); do 307 | if [ -e ${abs_path}/${norm_name}-r${i}.ebuild ]; then 308 | local found_ebuild="${cat}/${norm_name}-r${i}" 309 | local eapi_found_ebuild="$(get_eapi ${abs_path}/${norm_name}-r${i}.ebuild)" 310 | # only ebuild with a greater eapi then $min_allow_eapi are considered 311 | # candidates for stable requests/cleanup 312 | # also check if ebuild is masked 313 | if [ ${eapi_found_ebuild} -ge ${min_allow_eapi} ] && ! $(check_mask ${found_ebuild}); then 314 | 315 | local old_ebuild="${cat}/${pak}/${org_name}.ebuild" 316 | local new_ebuild="${cat}/${pak}/${norm_name}-r${i}.ebuild" 317 | 318 | if $(count_keywords "${org_name}" "${norm_name}-r${i}" ${cat} ${pak}); then 319 | 320 | local oec="$(get_git_age ${old_ebuild} "ct" "A" "first")" 321 | local oelm="$(get_git_age ${old_ebuild} "ct" "M" "last")" 322 | local nec="$(get_git_age ${new_ebuild} "ct" "A" "first")" 323 | local nelm="$(get_git_age ${new_ebuild} "ct" "M" "last")" 324 | local package_bugs="__C6__" 325 | 326 | if $(compare_keywords "${org_name}" "${norm_name}-r${i}" ${cat} ${pak}); then 327 | output def2 rs_clec 328 | else 329 | output def2 rs_stac 330 | fi 331 | fi 332 | break 2 333 | fi 334 | fi 335 | done 336 | fi 337 | 338 | [ ${DEBUGLEVEL} -ge 2 ] && echo "finished with ${1}" | (debug_output) 339 | } 340 | 341 | upd_results(){ 342 | if ${FILERESULTS}; then 343 | # only update rs_clec and rs_stac 344 | for rcid in rs_clec rs_stac; do 345 | # check if results were generated 346 | if [[ -e ${FULL_CHECKS[${rcid}]}/full.txt ]]; then 347 | local file="${FULL_CHECKS[${rcid}]}/full.txt" 348 | # otherwise check if old results exists 349 | elif [[ -e ${RESULTSDIR}/${SCRIPT_TYPE}/${FULL_CHECKS[${rcid}]/${WORKDIR}/}/full.txt ]]; then 350 | local file="${RESULTSDIR}/${SCRIPT_TYPE}/${FULL_CHECKS[${rcid}]/${WORKDIR}/}/full.txt" 351 | # unlikly case, do nothing 352 | else 353 | local file="" 354 | fi 355 | 356 | if [[ -n "${file}" ]]; then 357 | #get time diff since last run 358 | local indexfile="${RESULTSDIR}/${SCRIPT_TYPE}/${FULL_CHECKS[${rcid}]/${WORKDIR}/}/index.html" 359 | local time_diff="$(get_time_diff "${indexfile}")" 360 | for id in $(cat "${file}"); do 361 | 362 | # old ebuild created 363 | local oec="$(date_update "$(echo ${id}|cut -d'|' -f2)" "${time_diff}")" 364 | # old ebuild last modified 365 | local oelm="$(date_update "$(echo ${id}|cut -d'|' -f3)" "${time_diff}")" 366 | # new ebuild created 367 | local nec="$(date_update "$(echo ${id}|cut -d'|' -f5)" "${time_diff}")" 368 | # new ebuild last modified 369 | local nelm="$(date_update "$(echo ${id}|cut -d'|' -f6)" "${time_diff}")" 370 | local package="$(echo ${id}|cut -d'|' -f8)" 371 | $(get_bugs_bool "${package}") && local package_bugs="+" || local package_bugs="-" 372 | 373 | local new_id=$( 374 | echo "${id}" | gawk -F'|' '{$2=v2; $3=v3; $5=v5; $6=v6; $7=v7}1' \ 375 | v2="${oec}" \ 376 | v3="${oelm}" \ 377 | v5="${nec}" \ 378 | v6="${nelm}" \ 379 | v7="${package_bugs}" OFS='|' 380 | ) 381 | 382 | sed -i "s ${id} ${new_id} g" ${file} 383 | done 384 | fi 385 | done 386 | fi 387 | } 388 | 389 | find_func(){ 390 | [[ ${DEBUGLEVEL} -ge 1 ]] && echo ">>> calling ${FUNCNAME[0]} (MIND:${MIND} MAXD:${MAXD})" | (debug_output) 391 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "*** searchpattern is: ${SEARCHPATTERN[@]}" | (debug_output) 392 | 393 | if [ ${DEBUGLEVEL} -ge 2 ]; then 394 | find ${SEARCHPATTERN[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 395 | -type f -name "*.ebuild" 2>/dev/null | while read -r line; do main ${line}; done 396 | else 397 | find ${SEARCHPATTERN[@]} -mindepth $(expr ${MIND} + 1) -maxdepth $(expr ${MAXD} + 1) \ 398 | -type f -name "*.ebuild" 2>/dev/null | parallel main {} 399 | fi 400 | 401 | 402 | # check for empty results and remove them 403 | clean_results 404 | 405 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "*** fileresults enabled: ${FILERESULTS}" | (debug_output) 406 | if ${FILERESULTS}; then 407 | var_descriptions 408 | for s in ${SELECTED_CHECKS[@]}; do 409 | echo "${!s}" >> ${FULL_CHECKS[${s}]}/description.txt 410 | done 411 | upd_results 412 | 413 | # sort the results 414 | sort_result_v5 415 | 416 | # special sorting for stable/cleanup candidates 417 | sort_result_column_v1 "1,1 -k8,8" ${FULL_CHECKS[rs_clec]} 418 | sort_result_column_v1 "1,1 -k8,8" ${FULL_CHECKS[rs_stac]} 419 | 420 | # filter after EAPI/filter 421 | gen_sort_eapi_v1 ${FULL_CHECKS[rs_eaps]} 422 | gen_sort_eapi_v1 ${FULL_CHECKS[rs_livs]} 423 | gen_sort_eapi_v1 ${FULL_CHECKS[rs_obse]} 424 | 425 | gen_sort_filter_v2 4 ${FULL_CHECKS[rs_ecls]} 426 | gen_sort_filter_v2 4 ${FULL_CHECKS[rs_lics]} 427 | gen_sort_filter_v2 4 ${FULL_CHECKS[rs_keys]} 428 | gen_sort_filter_v2 4 ${FULL_CHECKS[rs_vius]} 429 | gen_sort_filter_v2 4 ${FULL_CHECKS[rs_ggrs]} 430 | gen_sort_filter_v2 4 ${FULL_CHECKS[rs_guss]} 431 | 432 | gen_sort_main_v5 433 | gen_sort_pak_v5 434 | 435 | post_checks ${SCRIPT_TYPE} 436 | fi 437 | } 438 | 439 | if [[ ${DEBUGLEVEL} -ge 2 ]]; then 440 | [[ ${DEBUGLEVEL} -ge 2 ]] && echo "*** starting ${0} (NON-PARALLEL)" | (debug_output) 441 | else 442 | [[ ${DEBUGLEVEL} -ge 1 ]] && echo "*** starting ${0} (PARALLEL)" | (debug_output) 443 | fi 444 | cd ${REPOTREE} 445 | 446 | array_names 447 | RUNNING_CHECKS=( ) 448 | for s in ${SELECTED_CHECKS[@]}; do 449 | RUNNING_CHECKS+=(${FULL_CHECKS[${s}]}) 450 | done 451 | 452 | export WORKDIR 453 | export -f main array_names 454 | if ${FILERESULTS}; then 455 | mkdir -p ${RUNNING_CHECKS[@]} 456 | # create a list and create corresponding folders and files of all available 457 | # eclasses and licenses before running the check - this way we also see 458 | # eclasses and licenses without customers. 459 | [[ ${DEBUGLEVEL} -ge 1 ]] && echo "*** generating various directories" | (debug_output) 460 | if ${TREE_IS_MASTER}; then 461 | eclass_list=( $(find ${REPOTREE}/eclass/*.eclass -maxdepth 1 -type f) ) 462 | eclass_list=( ${eclass_list[@]##*/} ) 463 | for ecl in ${eclass_list[@]}; do 464 | mkdir -p ${FULL_CHECKS[rs_ecls]}/sort-by-filter/${ecl%.*} 465 | touch ${FULL_CHECKS[rs_ecls]}/sort-by-filter/${ecl%.*}/full.txt 466 | done 467 | if [ -d "${REPOTREE}/licenses/" ]; then 468 | licenses_list=( $(find ${REPOTREE}/licenses/* -maxdepth 1 -type f) ) 469 | licenses_list=( ${licenses_list[@]##*/} ) 470 | for lic in ${licenses_list[@]}; do 471 | mkdir -p ${FULL_CHECKS[rs_lics]}/sort-by-filter/${lic} 472 | touch ${FULL_CHECKS[rs_lics]}/sort-by-filter/${lic}/full.txt 473 | done 474 | fi 475 | fi 476 | if [ -d "${REPOTREE}/virtual/" ]; then 477 | virtual_list=( $(find ${REPOTREE}/virtual/* -maxdepth 1 -type d) ) 478 | virtual_list=( ${virtual_list[@]##*/} ) 479 | for vir in ${virtual_list[@]}; do 480 | mkdir -p ${FULL_CHECKS[rs_vius]}/sort-by-filter/${vir} 481 | touch ${FULL_CHECKS[rs_vius]}/sort-by-filter/${vir}/full.txt 482 | done 483 | fi 484 | if [ -d "${REPOTREE}/acct-group/" ]; then 485 | group_list=( $(find ${REPOTREE}/acct-group/* -maxdepth 1 -type d) ) 486 | group_list=( ${group_list[@]##*/} ) 487 | for grp in ${group_list[@]}; do 488 | mkdir -p ${FULL_CHECKS[rs_ggrs]}/sort-by-filter/${grp} 489 | touch ${FULL_CHECKS[rs_ggrs]}/sort-by-filter/${grp}/full.txt 490 | done 491 | fi 492 | if [ -d "${REPOTREE}/acct-user/" ]; then 493 | user_list=( $(find ${REPOTREE}/acct-user/* -maxdepth 1 -type d) ) 494 | user_list=( ${user_list[@]##*/} ) 495 | for user in ${user_list[@]}; do 496 | mkdir -p ${FULL_CHECKS[rs_guss]}/sort-by-filter/${user} 497 | touch ${FULL_CHECKS[rs_guss]}/sort-by-filter/${user}/full.txt 498 | done 499 | fi 500 | fi 501 | depth_set_v4 ${1} 502 | ${FILERESULTS} && rm -rf ${WORKDIR} 503 | 504 | [[ ${DEBUGLEVEL} -ge 1 ]] && echo "*** finished ${0}" | (debug_output) 505 | -------------------------------------------------------------------------------- /tools/dteclass.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: dteclass.sh 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 21/07/2018 6 | 7 | # Copyright (C) 2018 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # this script creates a deptree of all eclasses 25 | 26 | #override REPOTREE,FILERESULTS,SITEDIR settings 27 | #export REPOTREE=/mnt/data/gentoo/ 28 | #export FILERESULTS=true 29 | #export SITEDIR="${HOME}/tmpcheck/" 30 | 31 | # get dirpath and load funcs.sh 32 | realdir="$(dirname $(readlink -f $BASH_SOURCE))" 33 | if [ -e ${realdir}/_funcs.sh ]; then 34 | source ${realdir}/_funcs.sh 35 | else 36 | echo "Missing _funcs.sh" 37 | exit 1 38 | fi 39 | 40 | # 41 | ### IMPORTANT SETTINGS START ### 42 | # 43 | WORKDIR="/tmp/dteclass-${RANDOM}" 44 | 45 | array_names(){ 46 | RUNNING_CHECKS=( 47 | "${WORKDIR}/deptree_eclasses" #Index 0 48 | ) 49 | } 50 | array_names 51 | # 52 | ### IMPORTANT SETTINGS STOP ### 53 | # 54 | 55 | get_inherits(){ 56 | local eclass=${1} 57 | # get the eclasses inherited by a eclass: 58 | # 1(sed): catch eclasses which are cut over multiple lines (remove newline) 59 | # 2(sed): remove leading spaces 60 | # 2(grep): remove all lines starting with '#' (comments) 61 | # 3(grep): search for lines starting with 'inherit ' or lines containing '&& inherit ' 62 | # 4(cut): remove inherit from result 63 | # 5(sed): remove comments at the end of lines (see darcs.eclass) 64 | local eclass_inherit=( $( sed -e :a -e '/\\$/N; s/\\\n//; ta' ${1} \ 65 | | sed "s/^[ \t]*//" \ 66 | | grep "^[^#]" \ 67 | | grep "^inherit \|&& inherit \|*) inherit " \ 68 | | sed 's/^.*\(inherit .*\).*$/\1/' \ 69 | | cut -c9- \ 70 | | sed 's/\#.*$//' \ 71 | | sed 's/\;\;//') ) 72 | 73 | echo "${eclass_inherit[@]}" 74 | } 75 | 76 | rec_dt_eclass(){ 77 | local tabdeep="${1}" 78 | local ecl="${2}" 79 | local dupcheck="${3}" 80 | 81 | tabdeep+="\t" 82 | dupcheck+=" ${ecl}" 83 | 84 | if [ -e "${REPOTREE}/eclass/${ecl}.eclass" ]; then 85 | local ec_in=( $(get_inherits "${REPOTREE}/eclass/${ecl}.eclass") ) 86 | if [ -n "${ec_in}" ]; then 87 | for tec in ${dupcheck}; do 88 | if $(echo ${ec_in[@]}|tr ' ' ':'|grep -q -P -o "${tec}(?=:|$)"); then 89 | echo -e "!C${tabdeep}| CIRCULAR DEEP with ${tec}" 90 | return 91 | fi 92 | done 93 | for e in $(echo ${ec_in[@]}); do 94 | if [ -e "${REPOTREE}/eclass/${e}.eclass" ]; then 95 | echo -e "${tabdeep}| ${e}" 96 | rec_dt_eclass ${tabdeep} "${e}" "${dupcheck}" 97 | else 98 | echo -e "!N${tabdeep}| ${e}.eclass doesn't exist" 99 | fi 100 | done 101 | fi 102 | else 103 | echo -e "!N${tabdeep}| ${ecl}.eclass doesn't exist" 104 | fi 105 | } 106 | 107 | main() { 108 | array_names 109 | local relative_path=${1} # path relative to ${REPOTREE}: ./app-admin/salt/salt-0.5.2.ebuild 110 | local eclassfile="$(echo ${relative_path}|cut -d'/' -f3)" # package filename: salt-0.5.2.ebuild 111 | local eclass="${eclassfile%.*}" # package name-version: salt-0.5.2 112 | 113 | # get a list of all inherits, looks like: "git-r3 eutils user" 114 | local ec_inherit=( $(get_inherits ${REPOTREE}/${relative_path}) ) 115 | 116 | if ${FILERESULTS}; then 117 | echo "${eclass}" >> ${RUNNING_CHECKS[0]}/full.txt 118 | else 119 | echo "${eclassfile}" 120 | if [ -n "${ec_inherit}" ]; then 121 | rec_dt_eclass "" "${eclass}" "" 122 | fi 123 | fi 124 | } 125 | 126 | if [ -n "${1}" ]; then 127 | # check if eclass exists 128 | if ! [ -e "${REPOTREE}/eclass/${1}.eclass" ]; then 129 | echo "Error: ${1}.eclass not found" 130 | exit 1 131 | fi 132 | fi 133 | # switch to the REPOTREE dir 134 | cd ${REPOTREE} 135 | # export important variables 136 | export WORKDIR 137 | export -f main array_names get_inherits rec_dt_eclass 138 | 139 | ${FILERESULTS} && mkdir -p ${RUNNING_CHECKS[@]} 140 | 141 | if [ -n "${1}" ]; then 142 | main "/eclass/${1}.eclass" 143 | else 144 | find ./eclass \ 145 | -type f -name "*.eclass" -print | sort | while read -r line; do main ${line}; done 146 | fi 147 | 148 | if ${FILERESULTS}; then 149 | gen_sort_main_v3 ${RUNNING_CHECKS[0]} 5 150 | gen_sort_pak_v3 ${RUNNING_CHECKS[0]} 3 151 | 152 | copy_checks checks 153 | rm -rf ${WORKDIR} 154 | fi 155 | -------------------------------------------------------------------------------- /tools/eclassinfo.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | 4 | main() { 5 | local ec=${1} 6 | local efuncs=( ) 7 | local f=( ) 8 | 9 | for i in ${ec}; do 10 | if ! $(grep -q "EXPORT_FUNCTIONS" /${GENTOOTREE}/eclass/${i}.eclass); then 11 | efuncs=( $(sed -n 's/# @FUNCTION: //p' "/${GENTOOTREE}/eclass/${i}.eclass" | sed ':a;N;$!ba;s/\n/ /g') ) 12 | if [ -n "${efuncs}" ]; then 13 | for x in ${efuncs[@]}; do 14 | if ! $(grep "@FUNCTION: ${x}" -A3 -m1 /${GENTOOTREE}/eclass/${i}.eclass |grep -q "@INTERNAL"); then 15 | f+=( "${x}" ) 16 | fi 17 | done 18 | fi 19 | else 20 | echo "ERR: ${i} exports functions" 21 | exit 1 22 | fi 23 | done 24 | echo "--<${ec}>--" 25 | echo "ALL(${#efuncs[@]}): $(echo ${efuncs[@]}|tr ' ' ':')" 26 | echo "FIL(${#f[@]}): $(echo ${f[@]}|tr ' ' ':')" 27 | echo 28 | } 29 | 30 | full() { 31 | for i in $(find /${GENTOOTREE}/eclass/ -maxdepth 1 -mindepth 1 -type f -name *.eclass); do 32 | if ! $(grep -q "EXPORT_FUNCTIONS" ${i}); then 33 | main "$(echo ${i##*/}|rev|cut -d '.' -f2-|rev)" 34 | fi 35 | done 36 | } 37 | 38 | def(){ 39 | ecl=( optfeature wrapper edos2unix ltprune l10n eutils estack preserve-libs \ 40 | vcs-clean epatch desktop versionator user user-info flag-o-matic xdg-utils \ 41 | libtool udev eapi7-ver pam ssl-cert ) 42 | 43 | for i in ${ecl[@]}; do 44 | main ${i} 45 | done 46 | } 47 | 48 | GENTOOTREE=/usr/portage/ 49 | if [ -n "${1}" ]; then 50 | if [ -e /${GENTOOTREE}/eclass/${1}.eclass ]; then 51 | main ${1} 52 | elif [ "${1}" == "all" ]; then 53 | full 54 | elif [ "${1}" == "def" ]; then 55 | def 56 | fi 57 | fi 58 | -------------------------------------------------------------------------------- /tools/maintainer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # Filename: maintainer.py 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 10/09/2016 6 | 7 | # Copyright (C) 2016 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: simple python script to get all gentoo projects 24 | # and its members (can be filtered with grep) 25 | 26 | projxml = "/usr/portage/metadata/projects.xml" 27 | 28 | import xml.etree.ElementTree 29 | e = xml.etree.ElementTree.parse(projxml).getroot() 30 | 31 | for i in e: 32 | for v in i.iter('member'): 33 | print(i[1].text,'(',i[0].text,') << ',v[1].text,'(',v[0].text,')') 34 | -------------------------------------------------------------------------------- /tools/whitecheck.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Filename: whitecheck 4 | # Autor: Michael Mair-Keimberger (m DOT mairkeimberger AT gmail DOT com) 5 | # Date: 02/02/2018 6 | 7 | # Copyright (C) 2017 Michael Mair-Keimberger 8 | # 9 | # This program is free software; you can redistribute it and/or 10 | # modify it under the terms of the GNU General Public License 11 | # as published by the Free Software Foundation; either version 2 12 | # of the License, or (at your option) any later version. 13 | # 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | # 19 | # You should have received a copy of the GNU General Public License 20 | # along with this program; if not, write to the Free Software 21 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. 22 | 23 | # Discription: 24 | # script for checking the whitelist for obsolete entries 25 | 26 | PORTTREE="/mnt/data/git/gentoo/" 27 | 28 | if [ -z "${1}" ]; then 29 | echo "You need to provide a whitelist file" 30 | exit 1 31 | else 32 | if ! [ -e ${1} ]; then 33 | echo "Whitelist file doesn't exist" 34 | exit 1 35 | else 36 | source ${1} 37 | fi 38 | fi 39 | 40 | for white in ${white_list[@]}; do 41 | cat_pak="$(echo ${white}|cut -d';' -f1)" 42 | white_file="$(echo ${white}|cut -d';' -f2)" 43 | white_ebuild="$(echo ${white}|cut -d';' -f3)" 44 | 45 | if ! [ -e ${PORTTREE}/${cat_pak}/files/${white_file} ]; then 46 | echo "removing ${cat_pak};${white_file}" 47 | sed -i "/${white_file}/d" ${1} 48 | fi 49 | done 50 | --------------------------------------------------------------------------------