├── .gitignore ├── COPYING ├── README.md ├── dave-script.sh ├── scripts └── tatt ├── setup.py ├── tatt.1 ├── tatt.5 ├── tatt ├── __init__.py ├── dot-tatt-spec ├── gentooPackage.py ├── job.py ├── packageFinder.py ├── scriptwriter.py ├── tattConfig.py ├── tinderbox.py ├── tool.py └── usecombis.py └── templates ├── cleanup ├── commit-footer ├── commit-header ├── commit-snippet ├── commit-snippet-2 ├── revdep-footer ├── revdep-header ├── revdep-snippet ├── tatt_functions.sh ├── updatebug ├── use-footer ├── use-header ├── use-loop ├── use-snippet └── use-test-snippet /.gitignore: -------------------------------------------------------------------------------- 1 | tatt.out 2 | .ropeproject 3 | *.pyc 4 | TAGS 5 | manual.html 6 | -------------------------------------------------------------------------------- /COPYING: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | 294 | Copyright (C) 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | , 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | tatt (is an) arch testing tool 2 | 3 | Introduction 4 | ============ 5 | 6 | Arch testing includes many boring tasks and tatt tries to automate 7 | them. It can be configured through the file ~/.tatt, an example is 8 | given below. 9 | 10 | tatt uses a template system. Basically it fills in small bash scripts 11 | using data scraped off bugzilla. You can look at the default 12 | templates which live in /usr/share/tatt/templates/ 13 | 14 | tatt uses 'bugz' from www-client/pybugz. You may want to configure an 15 | alias for 'bugz' to contain your login-credentials or you will have to 16 | type them everytime you use tatt. 17 | 18 | tatt lives on GitHub. Forks and pull requests are welcome: 19 | https://github.com/gentoo/tatt 20 | 21 | Ways to use tatt 22 | ================ 23 | 24 | Work on a stable bug no 300000 25 | ------------------------------ 26 | 27 | This will unmask the package and create five shell scripts. One is 28 | for automated testing of USE-flag combinations, one is for testing of 29 | reverse dependencies, one is for committing the new keywords to CVS, one 30 | is for leaving a message on the bug, and finally one is for cleaning up. 31 | 32 | tatt -b300000 -j myjob 33 | 34 | -j specifies a jobname which will be a prefix for the scripts that 35 | tatt produces, if it is left empty the bugnumber will be used 36 | 37 | Work on multiple packages 38 | ------------------------- 39 | 40 | If a whole list of packages should be tested, they can be specified 41 | in a file 42 | 43 | tatt -f myPackageFile -b bugnumber 44 | 45 | This will open the file myPackageFile, look for all atoms in it, and 46 | write scripts that test/commit all packages. If -j is omitted the 47 | filename is used. The bugnumber is necessary for the commit script. 48 | 49 | Resolving a bug 50 | --------------- 51 | 52 | Assume everything was committed and we want to resolve the bug. 53 | 54 | tatt -r bugnum -m "x86 stable, Thanks xyz" 55 | 56 | removes your arch from the CC field of the bug and adds the message. 57 | 58 | tatt -cr bugnum -m "x86 stable, Thanks, closing" 59 | 60 | Does the things -r does and additionally closes the bug. 61 | 62 | Running individual parts of tatt 63 | -------------------------------- 64 | 65 | - Open a bug and leave a message (for instance after successfull 66 | testing) 67 | 68 | tatt -s300000 69 | 70 | - Create only the test script for reverse dependencies of foo: 71 | 72 | tatt -d app-bar/foo 73 | 74 | - Create only the USE-flag testing script of foo 75 | 76 | tatt -u app-bar/foo 77 | 78 | - Show help 79 | 80 | tatt -h 81 | 82 | * Configuring tatt via ~/.tatt 83 | The specification of the configuration file can be found in dot-tatt-spec which usually resides 84 | /usr/lib/${python}/site-packages/tatt 85 | 86 | ```shell 87 | ####### EXAMPLE ~/.tatt ############ 88 | # Here we show the possible options together with their default values 89 | 90 | # Message for the success script @@ARCH@@ will be replaced by arch 91 | # successmessage='Archtested on @@ARCH@@: Everything fine' 92 | 93 | # ignoreprefix contains a list of use flag prefixes to be ignored 94 | # ignoreprefix="elibc_","video_cards_","linguas_","python_targets_","python_single_target_","kdeenablefinal","test","debug" 95 | 96 | # The arch you are working on (be careful, only tested with x86) 97 | # arch=x86 98 | 99 | # Directory where your script templates are (normally you don't need 100 | # to change this) 101 | # template-dir="/usr/share/tatt/templates/" 102 | 103 | # Where do you want tatt to put unmasked packages. Writes one file per 104 | # job in this directory. 105 | # unmaskdir="/etc/portage/package.accept_keywords" 106 | 107 | # You can customize the maximal number of rdeps to be tested as follows: 108 | # rdeps=3 109 | 110 | # You can customize the maximal number USE combis to be tested as follows: 111 | # usecombis=3 112 | # Note that All USE-flags on and all USE-flags off will always be tested. 113 | 114 | # Location of a checked out CVS Gentoo tree for repoman checks and commit scripts 115 | # repodir="./gentoo-x86" 116 | 117 | # Url where the pre-generated rindex is to be found 118 | # tinderbox-url="https://qa-reports.gentoo.org/output/genrdeps/rindex/" 119 | 120 | # If this is set, then tatt will refuse to run in a directory that does not 121 | # match this string. Use it as a safety measure against creating tatt-scripts 122 | # in random places of you filesystem 123 | # safedir=string(default="") 124 | 125 | # All emerge runs in the generated scripts are automatically passed 126 | # the -1 option. Here you can specify additional options. 127 | # emergeopts="-v" 128 | 129 | # directory where logs of failed builds will be stored 130 | # the exact name of the log will be shown in the report file 131 | # buildlogdir="./tatt/logs" 132 | ``` 133 | -------------------------------------------------------------------------------- /dave-script.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | . /etc/init.d/functions.sh 4 | 5 | if [ "$1" != "" ]; then 6 | ebegin "running repoman and searching for bugs for package $1" 7 | eend $? 8 | 9 | cd /usr/portage/"$1" 10 | repoman full 11 | einfo "repoman completed" 12 | eend $? 13 | 14 | cd 15 | 16 | bugz search -s all "$1" 17 | einfo "bugz completed" 18 | eend $? 19 | 20 | else 21 | echo "Enter a package!" 22 | fi 23 | 24 | read -p "Enter the bug number: " bug_number 25 | einfo "You entered $bug_number" 26 | mkdir $bug_number 27 | cd $bug_number 28 | tatt -b$bug_number 29 | for file in * 30 | do 31 | if [ -f "$file" ]; 32 | then 33 | chmod 777 $file 34 | fi 35 | done 36 | einfo "All good starting the test" 37 | ./*-rdeps.sh && ./*-useflags.sh 38 | 39 | 40 | printf "\033[32m\nComment success on Bug? Enter: 0\033[m\n" 41 | printf "\033[31m\nExit? Enter: 1\033[m\n" 42 | 43 | read -p "Enter >> " comment 44 | if [ $comment = "0" ]; then 45 | ./*-success.sh 46 | else 47 | exit 1 48 | fi -------------------------------------------------------------------------------- /scripts/tatt: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Global Modules 4 | from subprocess import * 5 | import sys 6 | import re 7 | import os 8 | import portage 9 | from portage.dep import dep_getcpv 10 | import base64 11 | import requests 12 | 13 | from tatt.gentooPackage import gentooPackage as gP 14 | import tatt.packageFinder as packageFinder 15 | from tatt.scriptwriter import writeusecombiscript as writeUSE 16 | from tatt.scriptwriter import writerdepscript as writeRdeps 17 | from tatt.scriptwriter import writesucessreportscript as writeSuccess 18 | from tatt.scriptwriter import writecommitscript as writeCommit 19 | from tatt.scriptwriter import writeCleanUpScript as writeCleanup 20 | from tatt.tattConfig import tattConfig as tattConfig 21 | from tatt.job import job as job 22 | from tatt.tool import get_repo_dir 23 | 24 | ##### Generate a global config obj, reading from ~/.tatt ##### 25 | config = tattConfig() 26 | session = requests.Session() 27 | session.params.update({'Bugzilla_api_key': config['bugzilla-key']}) 28 | 29 | ######### Main program starts here ############### 30 | 31 | ### USAGE and OPTIONS ### 32 | from optparse import OptionParser 33 | 34 | parser=OptionParser() 35 | parser.add_option("-d", "--depend", 36 | help="Determine stable rdeps", 37 | dest="depend", 38 | action="store_true", 39 | default = False) 40 | parser.add_option("-u", "--use", "--usecombis", 41 | help="Determine use flag combinations", 42 | dest="usecombi", 43 | action="store_true", 44 | default = False) 45 | parser.add_option("-f", "--file", 46 | help="Input File containing packages", 47 | dest="infile", 48 | action="store" 49 | ) 50 | parser.add_option("-j", "--jobname", 51 | help="name for the job, prefix of output files", 52 | dest="jobname", 53 | action="store") 54 | parser.add_option("-b", "--bug", 55 | help="do the full program for a given stable request bug", 56 | dest="bugnum", 57 | # type="int", 58 | # We could test this here, but in most cases the bugnumber 59 | # is actually used as a string, so we validate further down 60 | # and leave it as a string 61 | action="store") 62 | parser.add_option("-s", "--success", 63 | help="Comment that the program was successfully tested", 64 | dest="succbugnum", 65 | action="store") 66 | parser.add_option("-r", "--resolve", 67 | help="Resolve the given bugnumber, needs a message", 68 | dest="resolvenum", 69 | action="store") 70 | parser.add_option("-c", "--close", 71 | help="Resolve the given bugnumber with closing it, needs to be combined with -r", 72 | dest="close", 73 | action="store_true") 74 | parser.add_option("-k", "--keywording", 75 | help="search for keywording packages, needs to be combined with -f", 76 | dest="keywording", 77 | action="store_true", 78 | default = False) 79 | parser.add_option("-m", "--message", 80 | help="Message for bug resolution.", 81 | dest="resolvemessage", 82 | action="store") 83 | parser.add_option("-v", "--verbose", 84 | help="Print informative output.", 85 | dest="verbose", 86 | action="store_true", 87 | default = False) 88 | 89 | (options,args) = parser.parse_args() 90 | 91 | ## Messing with the configuration: 92 | # Save verbosity level 93 | config['verbose']=options.verbose 94 | # Normalize the template dir: 95 | config['template-dir']=os.path.abspath(config['template-dir'])+os.sep 96 | # If given, test if the bugnumber represents an int 97 | try: 98 | int(options.bugnum) 99 | except ValueError: 100 | print ("The bugnumber you gave with -b should be an integer.") 101 | sys.exit(1) 102 | except TypeError: 103 | # This occurs if bugnum is None, that is, -b was not given 104 | pass 105 | 106 | ## If safedir is set, check for the current directory 107 | if config['safedir'] != "": 108 | if os.getcwd().find(config['safedir']) == -1: 109 | # Safedir not found 110 | print ("Your safedir variable is set to '" + config['safedir'] + "',") 111 | print ("but you are in " + os.getcwd()) 112 | print ("Exiting.") 113 | sys.exit (1) 114 | 115 | ## -s and a bugnumber was given ? 116 | if options.succbugnum: 117 | print("Reporting success for bug number " + options.succbugnum) 118 | retcode = call(['bugz', 'modify', options.succbugnum, '-c', config['successmessage']]) 119 | if retcode == 0: 120 | print("Success!"); 121 | sys.exit (0) 122 | else: 123 | print("Failure commenting on Bugzilla") 124 | sys.exit(1) 125 | 126 | # get a job object to save things to 127 | myJob = job() 128 | 129 | ## If -f and a filename have been given: 130 | if options.infile: 131 | try: 132 | packfile=open(options.infile, 'r') 133 | except IOError: 134 | print("Given filename not found !") 135 | sys.exit(1) 136 | packraw = packfile.read() 137 | packfile.close() 138 | targetarch = config['arch'] 139 | if options.keywording: 140 | targetarch = '~' + targetarch 141 | myJob.type="keyword" 142 | else: 143 | myJob.type="stable" 144 | 145 | myJob.packageList = packageFinder.findPackages(packraw, targetarch, get_repo_dir(config['repodir'])) 146 | ## -b and a bugnumber was given ? 147 | if options.bugnum: 148 | print("Bugnumber: " + options.bugnum) 149 | myJob.bugnumber=options.bugnum 150 | params = {"id": options.bugnum} 151 | response = session.get(config["bugzilla-url"] + "/rest/bug", params=params).json() 152 | if "message" in response: 153 | print(response["message"]) 154 | sys.exit(1) 155 | if len(response["bugs"]) == 0: 156 | print("bug " + options.bugnum + " not found in bugzilla") 157 | sys.exit(1) 158 | 159 | response = response["bugs"][0] 160 | if "KEYWORDREQ" in response["keywords"] or response["component"] == "Keywording": 161 | # This is a keywording bug: 162 | print ("Keywording bug detected.") 163 | myJob.type="keyword" 164 | elif "STABLEREQ" in response["keywords"] or response["component"] == "Stabilization" or response["component"] == "Vulnerabilities": 165 | # Stablebug 166 | print ("Stabilization bug detected.") 167 | myJob.type="stable" 168 | else: 169 | print ("Could not detect bug's type, is the 'Keywords' field set?") 170 | sys.exit(1) 171 | if myJob.packageList==None: 172 | if response["cf_stabilisation_atoms"]: 173 | myJob.packageList = packageFinder.findPackages(response["cf_stabilisation_atoms"], config['arch'], get_repo_dir(config['repodir']), options.bugnum) 174 | if len(myJob.packageList) == 0 and ("KEYWORDREQ" in response["keywords"] or response["component"] == "Keywording"): 175 | myJob.packageList = packageFinder.findPackages(response["cf_stabilisation_atoms"], config['arch'], get_repo_dir(config['repodir']), options.bugnum) 176 | else: 177 | response = session.get(config["bugzilla-url"] + "/rest/bug/{}/attachment".format(options.bugnum), params=params).json()["bugs"][str(options.bugnum)] 178 | for attachment in response: 179 | if attachment["is_obsolete"] == 1: 180 | continue 181 | for flag in attachment['flags']: 182 | if flag["name"] == "stabilization-list" and flag["status"] == '+': 183 | myJob.packageList = packageFinder.findPackages(base64.b64decode(attachment["data"]).decode("utf8"), config['arch'], get_repo_dir(config['repodir']), options.bugnum) 184 | 185 | 186 | # joint code for -f and -b 187 | ########################## 188 | 189 | if myJob.packageList is not None and len(myJob.packageList) > 0: 190 | ## Assigning jobname 191 | if options.jobname: 192 | myJob.name = options.jobname 193 | elif options.infile: 194 | myJob.name = options.infile 195 | elif options.bugnum: 196 | myJob.name = myJob.packageList[0].packageName() + '-' + options.bugnum 197 | else: 198 | myJob.name = myJob.packageList[0].packageName() 199 | print ("Jobname: " + myJob.name) 200 | ## Determine jobtype 201 | 202 | port = portage.db[portage.root]["porttree"].dbapi 203 | 204 | filteredPackages = [] 205 | # for keywording bugs the packages that already have the keyword still need 206 | # to be unmasked so they can be used by the other packages that still need work 207 | kwPackages = [] 208 | 209 | if config['arch']: 210 | targetarch = config['arch'] 211 | if myJob.type == "keyword": 212 | targetarch = '~' + targetarch 213 | 214 | for p in myJob.packageList: 215 | print("Found the following package atom : " + p.packageString()) 216 | 217 | # check if the package already has the needed keywords 218 | kw = port.aux_get(dep_getcpv(p.packageString()), ["KEYWORDS"]) 219 | if len(kw) > 0: 220 | kwl = kw[0].split() 221 | try: 222 | kwl.index(config['arch']) 223 | # the list of keywords in portage already contains the arch 224 | # as stable, skip this package 225 | print("\talready stable for " + config['arch']) 226 | continue 227 | except ValueError: 228 | pass 229 | 230 | try: 231 | kwl.index(targetarch) 232 | # the list of keywords in portage already contains the target keyword, 233 | # skip this package from building, but remember it for unmasking 234 | print("\talready keyworded as " + targetarch) 235 | kwPackages.append(p) 236 | continue 237 | except ValueError: 238 | filteredPackages.append(p) 239 | 240 | if len(filteredPackages) == 0: 241 | print("\nno packages left") 242 | sys.exit(0) 243 | 244 | myJob.packageList = filteredPackages 245 | 246 | # Unmasking: 247 | unmaskname=config['unmaskdir'] 248 | if os.path.exists(unmaskname) and not os.path.isdir(unmaskname): 249 | print ("unmaskdir '", unmaskname, "' exists and is no directory") 250 | sys.exit(1) 251 | elif not os.path.exists(unmaskname): 252 | os.mkdir(unmaskname, 0o755) 253 | unmaskname=unmaskname+"/tatt_"+myJob.name 254 | 255 | try: 256 | unmaskfile=open(unmaskname, 'r+') 257 | except IOError: 258 | try: 259 | unmaskfile=open(unmaskname, 'w') 260 | unmaskfile.write(" ") 261 | unmaskfile.close() 262 | except IOError: 263 | # If we can't write to the file, then it should be configured differently 264 | print (" ".join(["Can not write to ",unmaskname])) 265 | print ("Maybe you don't have permission or the location is invalid.") 266 | print (" ".join(["Is",config['unmaskdir'],"a writeable directory?"])) 267 | print ("Probably you want to configure a different unmaskfile") 268 | print ("in your ~/.tatt. Exiting") 269 | sys.exit(1) 270 | unmaskfile=open(unmaskname, 'r+') 271 | 272 | unmaskfileContent = unmaskfile.read() 273 | for p in myJob.packageList: 274 | # Test if unmaskfile already contains the atom 275 | if re.search(re.escape(p.packageString()), unmaskfileContent): 276 | print (p.packageString() + " already in "+unmaskname) 277 | else: 278 | unmaskfile.write(p.packageString()) 279 | if myJob.type=="stable": 280 | pass 281 | elif myJob.type=="keyword": 282 | unmaskfile.write(" ** ") 283 | else: 284 | print ("Uh Oh, no job.type? Tell tomka@gentoo.org to fix this!") 285 | unmaskfile.write(" # Job " + myJob.name + "\n") 286 | print ("Unmasked " + p.packageString()+ " in "+unmaskname) 287 | 288 | # now write the remaining packages for keywording 289 | for p in kwPackages: 290 | # Test if unmaskfile already contains the atom 291 | if re.search(re.escape(p.packageString()), unmaskfileContent): 292 | print (p.packageString() + " already in "+unmaskname) 293 | else: 294 | unmaskfile.write(p.packageString() + " # Job " + myJob.name + "\n") 295 | print ("Unmasked " + p.packageString() + " in " + unmaskname) 296 | 297 | unmaskfile.close() 298 | ## Write the scripts 299 | writeUSE(myJob, config) 300 | writeRdeps(myJob, config) 301 | writeCleanup (myJob, config, unmaskname) 302 | ## Successscript can only be written if we have a bugnumber 303 | if myJob.bugnumber: 304 | writeSuccess(myJob, config) 305 | writeCommit(myJob, config) 306 | sys.exit (0) 307 | 308 | # Code for resolving bugs (-r and -m) 309 | ##################################### 310 | if options.resolvenum: 311 | if not options.resolvemessage: 312 | print("Please call with a message per -m") 313 | sys.exit (1) 314 | print("Resolving bug number " + options.resolvenum) 315 | calllist = ['bugz', 'modify', options.resolvenum, '-c', options.resolvemessage, '--remove-cc', config['arch']+"@gentoo.org"] 316 | if options.close: 317 | calllist = calllist + ['--fixed'] 318 | retcode = call(calllist) 319 | if retcode == 0: 320 | print("Success!"); 321 | sys.exit (0) 322 | else: 323 | print("Failure accessing bugzilla.") 324 | sys.exit(1) 325 | 326 | 327 | ## If we arrive here then a package atom should be given 328 | try: 329 | myJob.packageList=[gP(args[0])] 330 | myJob.name=myJob.packageList[0].packageName() 331 | except IndexError: 332 | print("Please call with complete package atom (including version) as argument.") 333 | sys.exit (1) 334 | 335 | if options.depend: 336 | writeRdeps(myJob, config) 337 | 338 | if options.usecombi: 339 | writeUSE(myJob, config) 340 | 341 | ## That's all folks ## 342 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | setup(name = "tatt", 4 | version = "0.11", 5 | packages = find_packages(), 6 | scripts = ['scripts/tatt'], 7 | package_data = { 8 | 'tatt': ['dot-tatt-spec'] 9 | }, 10 | 11 | install_requires = ['configobj>=4.6'], 12 | 13 | # metadata for upload to PyPI 14 | author = "Thomas Kahle", 15 | author_email = "tom111@gmx.de", 16 | description = "tatt is an arch testing tool", 17 | license = "GPL-2", 18 | keywords = "gentoo arch testing", 19 | url = "http://github.com/gentoo/tatt", # project home page 20 | ) 21 | -------------------------------------------------------------------------------- /tatt.1: -------------------------------------------------------------------------------- 1 | .TH TATT 1 2 | .SH NAME 3 | tatt (is an) arch testing tool. 4 | .SH SYNOPSIS 5 | .B tatt 6 | \fI [suboptions]\fB 7 | 8 | .SH DESCRIPTION 9 | Tatt is an arch-testing tool written for automation of frequently repeated tasks. 10 | 11 | .SH OPTIONS 12 | .TP 13 | \fB-h, --help\fI 14 | Show this help message and exit. 15 | .TP 16 | \fB-d, --depend\fI 17 | Determine stable rdeps. 18 | .TP 19 | \fB-u, --use, --usecombis\fI 20 | Determine use flag combinations. 21 | .TP 22 | \fB-f INFILE, --file=INFILE\fI 23 | Input File containing packages. 24 | .TP 25 | \fB-j JOBNAME, --jobname=JOBNAME\fI 26 | Name for the job, prefix of output files. 27 | .TP 28 | \fB-b BUGNUM, --bug=BUGNUM\fI 29 | Do the full program for a given stable request bug 30 | .TP 31 | \fB-s SUCCBUGNUM, --success=SUCCBUGNUM\fI 32 | Comment that the program was successfully tested 33 | .TP 34 | \fB-r RESOLVENUM, --resolve=RESOLVENUM\fI 35 | Resolve the given bugnumber, needs a message 36 | .TP 37 | \fB-c, --close\fI 38 | Resolve the given bugnumber with closing it, needs to be combined with -r 39 | .TP 40 | \fB-m RESOLVEMESSAGE, --message=RESOLVEMESSAGE\fI 41 | Message for bug resolution. 42 | .TP 43 | \fB-k, --keywording\fI 44 | Search for keywording packages, needs to be combined with -f 45 | .TP 46 | \fB-v, --verbose\fI 47 | Print informative output. 48 | 49 | .SH USAGE 50 | .TP 51 | tatt -b300000 -j myjob 52 | Work on a stable bug no 300000. This will unmask the package and 53 | create five shell scripts. One is for automated testing of USE-flag 54 | combinations, one is for testing of reverse dependencies, one is for 55 | committing the new keywords to CVS, one is for leaving a message on 56 | the bug, and finally one is for cleaning up. 57 | 58 | -j specifies a jobname which will be a prefix for the scripts that 59 | tatt produces, if it is left empty the bugnumber will be used 60 | 61 | .TP 62 | tatt -f myPackageFile -b bugnumber 63 | This will open the file myPackageFile, look for all atoms in it, and 64 | write scripts that test/commit all packages. If -j is omitted the 65 | filename is used. The bugnumber is necessary for the commit script. 66 | 67 | .TP 68 | tatt -r bugnum -m "x86 stable, Thanks xyz" 69 | Removes your arch from the CC field of the bug and adds the message (assuming everything was committed and we want to resolve the bug). 70 | .br 71 | -c switch additionally closes the bug. 72 | .SH FILES 73 | ~/.tatt 74 | .br 75 | /usr/share/tatt/templates/ 76 | 77 | .SH SEE ALSO 78 | gatt(8) 79 | .br 80 | https://github.com/gentoo/tatt 81 | 82 | .SH COPYRIGHT 83 | This software is licensed under the GNU General Public License, version 2 (GPLv2). A copy of this license is available at http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt. 84 | 85 | .SH BUGS 86 | Forks and pull requests are welcome on GitHub. https://github.com/gentoo/tatt 87 | 88 | .SH AUTHOR 89 | tatt is written and maintained by Thomas Kahle 90 | -------------------------------------------------------------------------------- /tatt.5: -------------------------------------------------------------------------------- 1 | .TH TATT 5 2 | .SH NAME 3 | .TP 4 | tatt configuration file ~/.tatt 5 | .SH SYNOPSIS 6 | .B tatt 7 | ~/.tatt 8 | .SH DESCRIPTION 9 | The specification of the configuration file can be found in dot-tatt-spec 10 | which usually resides \fI /usr/lib/${python}/site-packages/tatt \fI 11 | 12 | .SH EXAMPLE ~/.tatt 13 | .br 14 | # Message for the success script @@ARCH@@ will be replaced by arch 15 | .br 16 | #successmessage='Archtested on @@ARCH@@: Everything fine' 17 | 18 | .br 19 | # ignoreprefix contains a list of use flag prefixes to be ignored 20 | .br 21 | #ignoreprefix="elibc_","video_cards_","linguas_","kdeenablefinal","test","debug" 22 | 23 | .br 24 | # The arch you are working on (be careful, only tested with x86) 25 | .br 26 | #arch=x86 27 | 28 | .br 29 | # Directory where your script templates are (normally you don't need to change this) 30 | .br 31 | #template-dir="/usr/share/tatt/templates/" 32 | 33 | .br 34 | # Where do you want tatt to put unmasked packages. Writes one file per 35 | # job in this directory. 36 | .br 37 | #unmaskdir="/etc/portage/package.accept_keywords" 38 | 39 | .br 40 | # You can customize the maximal number of rdeps to be tested as follows: 41 | .br 42 | #rdeps=3 43 | 44 | .br 45 | # You can customize the maximal number USE combis to be tested as follows: 46 | .br 47 | # Note that All USE-flags on and all USE-flags off will always be tested. 48 | .br 49 | #usecombis=3 50 | 51 | .br 52 | # Location of a checked out CVS Gentoo tree for repoman checks and 53 | # commit scripts. 54 | .br 55 | #repodir="./gentoo-x86" 56 | 57 | .br 58 | # Url where the pre-generated rindex is to be found 59 | .br 60 | #tinderbox-url="\fIhttps://qa-reports.gentoo.org/output/genrdeps/rindex/\fP" 61 | 62 | .br 63 | # If this is set, then tatt will refuse to run in a directory that 64 | .br 65 | # does not match this string. 66 | .br 67 | # Use it as a safety measure against creating tatt-scripts in random 68 | .br 69 | # places of you filesystem. 70 | .br 71 | #safedir=string(default="") 72 | 73 | .br 74 | # All emerge runs in the generated scripts are automatically passed 75 | .br 76 | # the -1 option. Here you can specify additional options. 77 | .br 78 | #emergeopts="-v" 79 | 80 | .br 81 | # Bugzilla API key, generated at \fIhttps://bugs.gentoo.org/userprefs.cgi?tab=apikey\fP 82 | .br 83 | # Used to modify keywording annd stabilization bugs. 84 | .br 85 | #bugzilla-key="" 86 | 87 | .SH FILES 88 | ~/.tatt 89 | .br 90 | /usr/lib/${python}/site-packages/tatt 91 | .br 92 | /usr/share/tatt/templates/ 93 | 94 | .SH SEE ALSO 95 | tatt(1) 96 | .br 97 | https://github.com/gentoo/tatt 98 | 99 | .SH COPYRIGHT 100 | This software is licensed under the GNU General Public License, version 2 (GPLv2). A copy of this license is available at http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt. 101 | 102 | .SH AUTHOR 103 | tatt is written and maintained by Thomas Kahle 104 | -------------------------------------------------------------------------------- /tatt/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gentoo/tatt/8313ab7e2acaf36fcdf2e6381aabfe60ddb306f9/tatt/__init__.py -------------------------------------------------------------------------------- /tatt/dot-tatt-spec: -------------------------------------------------------------------------------- 1 | successmessage=string(default="Archtested on @@ARCH@@: Everything fine") 2 | ignoreprefix=string_list(default=list("elibc_","video_cards_","linguas_","python_targets_","python_single_target_","kdeenablefinal","test","debug")) 3 | template-dir=string(default="/usr/share/tatt/templates/") 4 | unmaskdir=string(default="/etc/portage/package.accept_keywords") 5 | arch=string(default="amd64") 6 | defaultopts=string(default="") 7 | emergeopts=string(default="") 8 | rdeps=integer(0,512,default=10) 9 | usecombis=integer(0,512,default=12) 10 | repodir=string(default="") 11 | tinderbox-url=string(default="https://qa-reports.gentoo.org/output/genrdeps/rindex/") 12 | safedir=string(default="") 13 | bugzilla-url=string(default="https://bugs.gentoo.org") 14 | bugzilla-key=string(default="") 15 | buildlogdir=string(default="") 16 | -------------------------------------------------------------------------------- /tatt/gentooPackage.py: -------------------------------------------------------------------------------- 1 | """Module for easy access to portage's atom syntax """ 2 | 3 | import re 4 | from portage.dep import dep_getcpv, dep_getkey, isvalidatom 5 | 6 | class gentooPackage(object): 7 | """A Gentoo package consists of: 8 | category 9 | name 10 | version 11 | of a Gentoo package""" 12 | 13 | def __init__(self, st): 14 | """An atom is initialized from an atom string""" 15 | if not isvalidatom(st): 16 | st = '=' + st 17 | cp = dep_getkey(st) 18 | self.ver = dep_getcpv(st)[len(cp) + 1:] # +1 to strip the leading '-' 19 | slashparts = cp.split("/") 20 | self.category = slashparts[0] 21 | self.name = slashparts[1] 22 | 23 | def packageName(self): 24 | """Returns the package name without category""" 25 | return self.name 26 | 27 | def packageVersion(self): 28 | """Returns the package version""" 29 | return self.ver 30 | 31 | def packageCategory(self): 32 | """Returns the package category without name""" 33 | return self.category 34 | 35 | def packageCatName(self): 36 | """Returns the package category and name without version""" 37 | return "/".join([self.category, self.name]) 38 | 39 | def packageString(self): 40 | """ Returns a portage compatible string representation""" 41 | if self.ver == '': 42 | return "/".join([self.category, self.name]) 43 | else: 44 | return ( "=" + "/".join([self.category, "-".join([self.name, self.ver])])) 45 | -------------------------------------------------------------------------------- /tatt/job.py: -------------------------------------------------------------------------------- 1 | """Abstraction of a tatt job""" 2 | 3 | class job(object): 4 | """A tatt job can have a 5 | - bugnumber 6 | - name 7 | - type (either 'stable', 'keyword') 8 | - list of packages 9 | """ 10 | 11 | def __init__(self, 12 | name="", 13 | bugnumber=0, 14 | type="stable", 15 | packageList=None): 16 | self.name=name 17 | self.bugnumber=bugnumber 18 | self.type=type 19 | self.packageList=packageList 20 | -------------------------------------------------------------------------------- /tatt/packageFinder.py: -------------------------------------------------------------------------------- 1 | """module for extracting packages from a package/architecture list """ 2 | import subprocess 3 | from .gentooPackage import gentooPackage as gP 4 | 5 | def findPackages (s, arch, repo, bugnum=False): 6 | """ Given a string s, a string arch, a string path to the repo, and 7 | an integer bugnum, return all gentooPackages from that string 8 | that need actioning on that arch """ 9 | 10 | packages = [] 11 | 12 | if bugnum: 13 | print("Using Nattka to process the bug") 14 | output = subprocess.check_output(['nattka', '--repo', repo, 'apply', '-a', arch.replace("~", ""), '-n', bugnum, '--ignore-sanity-check', '--ignore-dependencies']) 15 | output = output.decode("utf8").split("\n") 16 | output = [line for line in output if not line.startswith("#")] 17 | output = [line.split(" ")[0] for line in output] 18 | else: 19 | print("Manually processing") 20 | output = s.splitlines() 21 | 22 | for line in output: 23 | if not line: 24 | continue 25 | 26 | atom, _, arches = line.replace('\t', ' ').partition(' ') 27 | archlist = arches.split(' ') 28 | if not arches or arch in archlist or ('~' + arch) in archlist: 29 | packages.append(gP(atom)) 30 | 31 | return(packages) 32 | -------------------------------------------------------------------------------- /tatt/scriptwriter.py: -------------------------------------------------------------------------------- 1 | """ Filling script templates """ 2 | 3 | import random 4 | import os 5 | import portage 6 | import sys 7 | 8 | from .gentooPackage import gentooPackage as gP 9 | from .usecombis import findUseFlagCombis 10 | from .tinderbox import stablerdeps 11 | from .tool import unique 12 | from portage.dep import dep_getkey 13 | 14 | #### USE-COMBIS ######## 15 | 16 | def scriptTemplate(job, config, filename): 17 | """ open snippet file and replace common placeholders """ 18 | try: 19 | snippetfile=open(config['template-dir'] + filename, 'r') 20 | except IOError: 21 | print("template " + filename + " not found in " + config['template-dir']) 22 | sys.exit(1) 23 | 24 | reportname = job.name + ".report" 25 | if job.type == "stable": 26 | newkeyword = config['arch'] 27 | elif job.type == "keyword": 28 | newkeyword = "~" + config['arch'] 29 | else: 30 | print ("No job type? Can't continue. This is a bug") 31 | sys.exit(1) 32 | 33 | snippet = snippetfile.read() 34 | snippet = snippet.replace("@@EMERGEOPTS@@", config['emergeopts']) 35 | if job.bugnumber: 36 | snippet = snippet.replace("@@BUG@@", job.bugnumber) 37 | else: 38 | snippet = snippet.replace("@@BUG@@", '') 39 | snippet = snippet.replace("@@JOB@@", job.name) 40 | snippet = snippet.replace("@@ARCH@@", config['arch']) 41 | snippet = snippet.replace("@@REPODIR@@", config['repodir']) 42 | snippet = snippet.replace("@@REPORTFILE@@", reportname) 43 | snippet = snippet.replace("@@BUILDLOGDIR@@", config['buildlogdir']) 44 | snippet = snippet.replace("@@NEWKEYWORD@@", newkeyword) 45 | snippet = snippet.replace("@@TEMPLATEDIR@@", config['template-dir']) 46 | return snippet 47 | 48 | def useCombiTestString(job, pack, config, port): 49 | """ Build with diffent useflag combis """ 50 | usesnippet = scriptTemplate(job, config, "use-snippet") 51 | usesnippet = usesnippet.replace("@@CPV@@", pack.packageString() ) 52 | 53 | # test once with tests and users flags 54 | # do this first to trigger bugs in some packages where the test suite relies on 55 | # the package being already installed 56 | usetestsnippet = scriptTemplate(job, config, "use-test-snippet") 57 | usetestsnippet = usetestsnippet.replace("@@CPV@@", pack.packageString() ) 58 | s = usetestsnippet.replace("@@USE@@", "") 59 | 60 | usecombis = findUseFlagCombis (pack, config, port) 61 | for uc in usecombis: 62 | localsnippet = usesnippet.replace("@@USE@@", uc) 63 | s = s + localsnippet 64 | return s 65 | 66 | def writeusecombiscript(job, config): 67 | # job is a tatt job object 68 | # config is a tatt configuration 69 | useheader = scriptTemplate(job, config, "use-header") 70 | if os.path.exists(config['template-dir'] + "use-loop"): 71 | useloop = scriptTemplate(job, config, "use-loop") 72 | else: 73 | useloop = "@@LOOP_BODY@@" 74 | 75 | outfilename = (job.name + "-useflags.sh") 76 | reportname = (job.name + ".report") 77 | if os.path.isfile(outfilename): 78 | print("WARNING: Will overwrite " + outfilename) 79 | outfile = open(outfilename, 'w') 80 | outfile.write(useheader) 81 | port = portage.db[portage.root]["porttree"].dbapi 82 | for p in job.packageList: 83 | loop = useloop.replace("@@LOOP_BODY@@", useCombiTestString(job, p, config, port)) 84 | loop = loop.replace("@@CPV@@", p.packageString()) 85 | outfile.write(loop) 86 | if os.path.exists(config['template-dir'] + "use-footer"): 87 | footer = scriptTemplate(job, config, "use-footer") 88 | outfile.write(footer) 89 | # Note: fchmod needs the filedescriptor which is an internal 90 | # integer retrieved by fileno(). 91 | os.fchmod(outfile.fileno(), 0o744) # rwxr--r-- 92 | outfile.close() 93 | 94 | ###################################### 95 | 96 | ############ RDEPS ################ 97 | def rdepTestString(job, rdep, config): 98 | snip = scriptTemplate(job, config, "revdep-snippet") 99 | 100 | uflags = [] 101 | for st in rdep[1]: 102 | st = st.strip() 103 | if len(st) == 0: 104 | continue 105 | if st[0] == "!": 106 | uflags.append("-" + st[1:]) 107 | else: 108 | uflags.append(st) 109 | ustring = "USE=\'" + " ".join(uflags) + "\'" 110 | snip = snip.replace("@@USE@@", ustring) 111 | snip = snip.replace("@@CPV@@", rdep[0] ) 112 | snip = snip.replace("@@EMERGEOPTS@@", config['emergeopts']) 113 | return snip 114 | 115 | def writerdepscript(job, config): 116 | # Populate the list of rdeps 117 | # while at it also create a list of only the package names 118 | rdeps = [] 119 | pkgs = [] 120 | for p in job.packageList: 121 | atom = p.packageCatName() 122 | pkgs.append(atom) 123 | if config['rdeps'] > 0: 124 | rdeps = rdeps + stablerdeps (atom, config) 125 | if len(rdeps) == 0: 126 | print("No stable rdeps for " + job.name) 127 | return 128 | 129 | # now clean the list 130 | # first find all those entries that have no useflags and main packages of this job 131 | for i in range(len(rdeps) - 1, 0, -1): 132 | r = rdeps[i] 133 | hasU = False 134 | for st in r[1]: 135 | if len(st.strip()) > 0: 136 | hasU = True 137 | break 138 | if hasU: 139 | continue 140 | if r[0] in pkgs: 141 | rdeps.pop(i) 142 | 143 | # If there are rdeps, write the script 144 | rdepheader = scriptTemplate(job, config, "revdep-header") 145 | outfilename = (job.name + "-rdeps.sh") 146 | if os.path.isfile(outfilename): 147 | print("WARNING: Will overwrite " + outfilename) 148 | outfile = open(outfilename,'w') 149 | outfile.write(rdepheader) 150 | 151 | for r in rdeps: 152 | # Todo: remove duplicates 153 | outfile.write(rdepTestString(job, r, config)) 154 | os.fchmod(outfile.fileno(), 0o744) 155 | 156 | if os.path.exists(config['template-dir'] + "revdep-footer"): 157 | footer = scriptTemplate(job, config, "revdep-footer") 158 | outfile.write(footer) 159 | 160 | outfile.close() 161 | 162 | 163 | #######Write report script############ 164 | def writesucessreportscript (job, config): 165 | outfilename = (job.name + "-success.sh") 166 | if os.path.isfile(outfilename): 167 | print("WARNING: Will overwrite " + outfilename) 168 | updatebug = scriptTemplate(job, config, "updatebug") 169 | outfile = open(outfilename,'w') 170 | outfile.write(updatebug) 171 | os.fchmod(outfile.fileno(), 0o744) 172 | outfile.close() 173 | print("Success Report script written to " + outfilename) 174 | 175 | 176 | ####### Write the commit script ######### 177 | def writecommitscript (job, config): 178 | cheader = scriptTemplate(job, config, "commit-header") 179 | csnippet = scriptTemplate(job, config, "commit-snippet") 180 | csnippet2 = scriptTemplate(job, config, "commit-snippet-2") 181 | cfooter = scriptTemplate(job, config, "commit-footer") 182 | 183 | outfilename = (job.name + "-commit.sh") 184 | if os.path.isfile(outfilename): 185 | print("WARNING: Will overwrite " + outfilename) 186 | outfile = open(outfilename,'w') 187 | outfile.write (cheader) 188 | # Here's a catch: If there are multiple versions of the same package to be 189 | # stabilized, then we want only one keywording block and one commit block 190 | # for them. Therefore we split up the loop by sorting job.packlist 191 | # accordingly, saving them in a hash-table with the package names as keys 192 | # and the packages as values. 193 | packageHash = dict(); 194 | for pack in job.packageList: 195 | if pack.packageCatName() in packageHash: 196 | packageHash[pack.packageCatName()] = packageHash[pack.packageCatName()] + [pack] 197 | else: 198 | packageHash[pack.packageCatName()] = [pack] 199 | # First round (ekeyword) 200 | for pack in packageHash.keys(): 201 | # Prepare a list of ebuild names strings 202 | ebuilds = [p.packageName()+"-"+p.packageVersion()+".ebuild" for p in packageHash[pack]] 203 | s = csnippet.replace("@@EBUILD@@", " ".join(ebuilds)) 204 | s = s.replace("@@CP@@", pack) 205 | outfile.write(s) 206 | # Second round: repoman -d full checks and commit, should be done once per 207 | # key of packageHash 208 | for pack in packageHash.keys(): 209 | # Prepare a list of ebuild names strings 210 | ebuilds = [p.packageName()+"-"+p.packageVersion()+".ebuild" for p in packageHash[pack]] 211 | s = csnippet2.replace("@@EBUILD@@", " ".join(ebuilds)) 212 | s = s.replace("@@CP@@", pack) 213 | outfile.write(s) 214 | # Footer (committing) 215 | outfile.write(cfooter) 216 | os.fchmod(outfile.fileno(), 0o744) 217 | outfile.close() 218 | print("Commit script written to " + outfilename) 219 | 220 | 221 | ######## Write clean-up script ############## 222 | def writeCleanUpScript (job, config, unmaskname): 223 | script = scriptTemplate(job, config, "cleanup") 224 | script = script.replace("@@KEYWORDFILE@@", unmaskname) 225 | outfilename = (job.name + "-cleanup.sh") 226 | if os.path.isfile(outfilename): 227 | print("WARNING: Will overwrite " + outfilename) 228 | outfile = open(outfilename,'w') 229 | outfile.write(script) 230 | os.fchmod(outfile.fileno(), 0o744) 231 | outfile.close() 232 | -------------------------------------------------------------------------------- /tatt/tattConfig.py: -------------------------------------------------------------------------------- 1 | """Abstraction of a tatt configuration""" 2 | 3 | import os 4 | import sys 5 | 6 | # from configobj 7 | from configobj import ConfigObj 8 | from validate import Validator 9 | # To access the specfile: 10 | from pkg_resources import resource_filename 11 | 12 | # resource_filename will give us platform-independent access to the specfile 13 | specfile = resource_filename('tatt', 'dot-tatt-spec') 14 | 15 | # this validator will also do type conversion according to the spec file! 16 | class tattConfig (ConfigObj): 17 | """Nothing special here, just a checked ConfigObj""" 18 | def __init__(self): 19 | # Read the config from ~/.tatt and create ConfigObj 20 | ConfigObj.__init__(self, os.path.join(os.path.expanduser("~"), ".tatt"), configspec=specfile) 21 | 22 | # Validate against the specfile 23 | validator = Validator() 24 | result = self.validate(validator) 25 | 26 | if result != True: 27 | print ("Config file validation failed!") 28 | print ("The following items could not be parsed") 29 | for k in result.keys(): 30 | if result[k] == False: 31 | print (k) 32 | sys.exit(1) 33 | -------------------------------------------------------------------------------- /tatt/tinderbox.py: -------------------------------------------------------------------------------- 1 | """acessing the tinderbox at http://qa-reports.gentoo.org/output/genrdeps/rindex/ """ 2 | 3 | import socket # For setting a global timeout 4 | # Support python2 and python3 versions of urllib: 5 | try: 6 | from urllib2 import urlopen, HTTPError 7 | except: 8 | from urllib.request import urlopen 9 | from urllib.error import HTTPError 10 | import sys 11 | from subprocess import * 12 | import random 13 | 14 | from .gentooPackage import gentooPackage as gP 15 | from portage.dep import isvalidatom 16 | 17 | ## TODO: Make the number of rdeps to sample a config option 18 | ## Pass the config on to this function: 19 | 20 | ## Generate stable rdeps ### 21 | def stablerdeps (atom, config): 22 | """ 23 | Find packages with stable versions which depend on atom 24 | We query the tinderbox at http://qa-reports.gentoo.org/output/genrdeps/rindex/ 25 | for this purpose. 26 | The result is a list of pairs of package atoms and a list of necessary useflags 27 | """ 28 | tinderbox = config['tinderbox-url'] 29 | # File structure on this tinderbox equals that in the tree 30 | # Problem: The rdeps can be version dependent 31 | # nothing we can do about this here... 32 | 33 | socket.setdefaulttimeout(45) 34 | try: 35 | download = urlopen(tinderbox + atom).read().decode('utf-8') 36 | except HTTPError as e: 37 | # Cleanup the timeout: 38 | socket.setdefaulttimeout(None) 39 | if e.code == 404: 40 | # 404 is OK, the package has no stable rdeps 41 | return [] 42 | else: 43 | # Some other error should not occur: 44 | print("Non 404 Error on accessing the tinderbox") 45 | sys.exit (1) 46 | # If we are here everything is fine, cleanup the timeout: 47 | socket.setdefaulttimeout(None) 48 | # The result is a "\n" separated list of packages : useflags 49 | packlist = download.rstrip().split("\n") 50 | # Split at : to see if useflags are necessary 51 | splitlist2 = [p.split(":") for p in packlist] 52 | # Fill with empty useflags if nothing is given: 53 | splitlist = [] 54 | for s in splitlist2: 55 | if len(s) == 1: 56 | splitlist.append([s[0],[" "]]) 57 | else: 58 | splitlist.append([s[0],s[1].split("+")]) 59 | d = dict([]) 60 | for s in splitlist: 61 | # Saves necessary useflags under package names, removing duplicates. 62 | if isvalidatom('=' + s[0]): 63 | d[gP(s[0]).packageCatName()] = s[1] 64 | outlist2 = [[k, d[k]] for k in list(d.keys())] 65 | outlist = [] 66 | # outlist2 is set up at this point. It contains all candidates. To cut it down we sample 67 | # randomly without replacement until the list is empty or we have config['rdeps'] many. 68 | 69 | # We are calling eix for each package to work around issues with --stable: 70 | # What we should do with a future version of eix is to do this in a single run 71 | # or fork multiple eix instances 72 | 73 | while ((len (outlist2) > 0) and (len(outlist) < config['rdeps'])): 74 | # Warning: sample returns a list, even if only one sample 75 | [samp]=random.sample(outlist2, 1) 76 | # Drop the one we selected 77 | outlist2.remove(samp) 78 | eixcall = ["eix", "--stable", "--only-names", "--exact", samp[0]] 79 | p2 = Popen(eixcall, stdout=PIPE) 80 | out = p2.communicate()[0].decode('utf-8') 81 | if out == '': continue 82 | else : outlist.append(samp) 83 | 84 | if len(outlist2) > 0: 85 | print("More than " + str(config['rdeps']) + " stable rdeps for " + atom + ", took a sample. \n") 86 | return outlist 87 | ############################# 88 | -------------------------------------------------------------------------------- /tatt/tool.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | """ Helper functions used in tatt""" 4 | 5 | ## Getting unique elements of a list ## 6 | def unique(seq, idfun=None): 7 | """Returns the unique elements in a list 8 | order preserving""" 9 | if idfun is None: 10 | def idfun(x): return x 11 | seen = {} 12 | result = [] 13 | for item in seq: 14 | marker = idfun(item) 15 | # in old Python versions: 16 | # if seen.has_key(marker) 17 | # but in new ones: 18 | if marker in seen: continue 19 | seen[marker] = 1 20 | result.append(item) 21 | return result 22 | 23 | def get_repo_dir(repodir): 24 | # Prefer the repo dir in the config 25 | if repodir: 26 | if os.path.isdir(repodir): 27 | return repodir 28 | else: 29 | raise ValueError("Repo dir does not seem to be a directory") 30 | 31 | # No path given in config 32 | if os.path.isdir("/var/db/repos/gentoo/"): 33 | print("Using /var/db/repos/gentoo/ as fallback") 34 | return "/var/db/repos/gentoo" 35 | elif os.path.isdir("/usr/portage/"): 36 | print("Using /usr/portage/ as fallback") 37 | return "/usr/portage/" 38 | 39 | raise ValueError("Repo dir not given and fallbacks failed") 40 | -------------------------------------------------------------------------------- /tatt/usecombis.py: -------------------------------------------------------------------------------- 1 | """Use Flag Mechanics """ 2 | 3 | import random 4 | import re 5 | import math 6 | from portage.dep import check_required_use, dep_getcpv 7 | from subprocess import * 8 | 9 | from .tool import unique 10 | from gentoolkit.flag import get_flags, reduce_flags 11 | 12 | def all_valid_flags(flag): 13 | return True 14 | 15 | def check_uses(ruse, uselist, sw, package): 16 | act = [] # check_required_use doesn't like -flag entries 17 | for pos in range(len(uselist)): 18 | if ((2**pos) & sw): 19 | act.append(uselist[pos]) 20 | if bool(check_required_use(ruse, " ".join(act), all_valid_flags)): 21 | return True 22 | else: 23 | print(" " + package.packageString() + ": ignoring invalid USE flag combination", act) 24 | return False 25 | 26 | ## Useflag Combis ## 27 | def findUseFlagCombis (package, config, port): 28 | """ 29 | Generate combinations of use flags to test 30 | The output will be a list each containing a ready to use USE=... string 31 | """ 32 | uselist = sorted(reduce_flags(get_flags(dep_getcpv(package.packageString())))) 33 | # The uselist could have duplicates due to slot-conditional 34 | # output of equery 35 | uselist=unique(uselist) 36 | for i in config['ignoreprefix']: 37 | uselist=[u for u in uselist if not re.match(i,u)] 38 | 39 | ruse = " ".join(port.aux_get(dep_getcpv(package.packageString()), ["REQUIRED_USE"])) 40 | swlist = [] 41 | if config['usecombis'] == 0: 42 | # Do only all and nothing: 43 | if check_uses(ruse, uselist, 0, package): 44 | swlist.append(0) 45 | if check_uses(ruse, uselist, 2**len(uselist) - 1): 46 | swlist.append(2**len(uselist) - 1) 47 | # Test if we can exhaust all USE-combis by computing the binary logarithm. 48 | elif len(uselist) > math.log(config['usecombis'],2): 49 | 50 | # Generate a sample of USE combis 51 | s = 2**(len (uselist)) 52 | rnds = set() 53 | random.seed() 54 | 55 | attempts = 0 56 | ignore_use_expand = False 57 | 58 | while len(swlist) < config['usecombis'] and len(rnds) < s: 59 | if attempts >= 10000: 60 | if not ignore_use_expand: 61 | # After 10000 attempts, let's give up on USE_EXPAND. 62 | ignore_use_expand = True 63 | attempts = 0 64 | 65 | print("Giving up on USE_EXPAND after {0} tries to find valid USE combinations".format(attempts)) 66 | print("We've found {0} USE combinations so far".format(len(swlist))) 67 | uselist = [use for use in uselist if not "_" in use] 68 | else: 69 | # Let's give up entirely and use what we have. 70 | print("Giving up on finding more USE combinations after {0} further attempts".format(attempts)) 71 | print("We've found {0} USE combinations".format(len(swlist))) 72 | break 73 | 74 | r = random.randint(0, s-1) 75 | if r in rnds: 76 | # already checked 77 | continue 78 | rnds.add(r) 79 | 80 | if not check_uses(ruse, uselist, r, package): 81 | attempts += 1 82 | # invalid combination 83 | continue 84 | 85 | swlist.append(r) 86 | 87 | swlist.sort() 88 | else: 89 | # Yes we can: generate all combinations 90 | for pos in range(2**len(uselist)): 91 | if check_uses(ruse, uselist, pos, package): 92 | swlist.append(pos) 93 | 94 | usecombis=[] 95 | for sw in swlist: 96 | mod = [] 97 | for pos in range(len(uselist)): 98 | if ((2**pos) & sw): 99 | mod.append("") 100 | else: 101 | mod.append("-") 102 | usecombis.append(" ".join(["".join(uf) for uf in list(zip(mod, uselist))])) 103 | 104 | # Merge everything to a USE="" string 105 | return ["USE=\'" + uc + "\'" for uc in usecombis] 106 | -------------------------------------------------------------------------------- /templates/cleanup: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ## Clean-up the keywordfile 3 | rm @@KEYWORDFILE@@ 4 | 5 | # Remove all files associated to the job: 6 | rm -f @@JOB@@-* 7 | -------------------------------------------------------------------------------- /templates/commit-footer: -------------------------------------------------------------------------------- 1 | # Anything to be done here? 2 | -------------------------------------------------------------------------------- /templates/commit-header: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | pushd @@REPODIR@@ > /dev/null 3 | 4 | if [ "@@NEWKEYWORD@@" = "@@ARCH@@" ]; then 5 | DESCR="Stabilize @@ARCH@@" 6 | else 7 | DESCR="Keyword @@ARCH@@" 8 | fi 9 | 10 | if [ -n "@@BUG@@" ]; then 11 | DESCR="${DESCR}, #@@BUG@@" 12 | fi 13 | -------------------------------------------------------------------------------- /templates/commit-snippet: -------------------------------------------------------------------------------- 1 | pushd @@CP@@ > /dev/null 2 | 3 | ekeyword @@NEWKEYWORD@@ @@EBUILD@@ 4 | 5 | popd > /dev/null 6 | -------------------------------------------------------------------------------- /templates/commit-snippet-2: -------------------------------------------------------------------------------- 1 | # Code for checking consistency of @@EBUILD@@ 2 | pushd @@CP@@ > /dev/null 3 | 4 | pkgcheck scan || exit 1 5 | git add @@EBUILD@@ || exit 1 6 | pkgdev commit -m "@@CP@@: ${DESCR}" --signoff || exit 1 7 | 8 | popd > /dev/null 9 | -------------------------------------------------------------------------------- /templates/revdep-footer: -------------------------------------------------------------------------------- 1 | 2 | exit ${test_ret} 3 | -------------------------------------------------------------------------------- /templates/revdep-header: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Reverse dependency testing for @@JOB@@ 3 | 4 | trap "echo 'signal captured, exiting the entire script...'; exit" SIGHUP SIGINT SIGTERM 5 | 6 | export TATT_TEST_TYPE="rdep" 7 | export TATT_REPORTFILE="@@REPORTFILE@@" 8 | export TATT_BUILDLOGDIR="@@BUILDLOGDIR@@" 9 | export TATT_EMERGEOPTS="@@EMERGEOPTS@@" 10 | 11 | source "@@TEMPLATEDIR@@tatt_functions.sh" 12 | 13 | test_ret=0 14 | 15 | export USE 16 | 17 | echo -e "revdep tests started on $(date)\n" >> @@REPORTFILE@@ 18 | -------------------------------------------------------------------------------- /templates/revdep-snippet: -------------------------------------------------------------------------------- 1 | @@USE@@ tatt_test_pkg --test "@@CPV@@" || test_ret=1 2 | -------------------------------------------------------------------------------- /templates/tatt_functions.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | function tatt_pkg_error 4 | { 5 | local eout=${2} 6 | 7 | echo "${eout}" 8 | 9 | if [ -n "${USE}" ]; then 10 | echo -n "USE='${USE}'" >> "${TATT_REPORTFILE}" 11 | fi 12 | if [ -n "${FEATURES}" ]; then 13 | echo -n " FEATURES='${FEATURES}'" >> "${TATT_REPORTFILE}" 14 | fi 15 | 16 | if [[ "${eout}" =~ REQUIRED_USE ]] ; then 17 | echo " : REQUIRED_USE not satisfied (probably) for ${1:?}" >> "${TATT_REPORTFILE}" 18 | elif [[ "${eout}" =~ USE\ changes ]] ; then 19 | echo " : USE dependencies not satisfied (probably) for ${1:?}" >> "${TATT_REPORTFILE}" 20 | elif [[ "${eout}" =~ keyword\ changes ]]; then 21 | echo " : unkeyworded dependencies (probably) for ${1:?}" >> "${TATT_REPORTFILE}" 22 | elif [[ "${eout}" =~ Error:\ circular\ dependencies: ]]; then 23 | echo " : circular dependencies (probably) for ${1:?}" >> "${TATT_REPORTFILE}" 24 | elif [[ "${eout}" =~ Blocked\ Packages ]]; then 25 | echo " : blocked packages (probably) for ${1:?}" >> "${TATT_REPORTFILE}" 26 | else 27 | echo " failed for ${1:?}" >> "${TATT_REPORTFILE}" 28 | fi 29 | 30 | CP=${1#=} 31 | BUILDDIR=/var/tmp/portage/${CP} 32 | BUILDLOG=${BUILDDIR}/temp/build.log 33 | if [[ -n "${TATT_BUILDLOGDIR}" && -s "${BUILDLOG}" ]]; then 34 | mkdir -p "${TATT_BUILDLOGDIR}" 35 | LOGNAME=$(mktemp -p "${TATT_BUILDLOGDIR}" "${CP/\//_}_${TATT_TEST_TYPE}_XXXXX") 36 | mv "${BUILDLOG}" "${LOGNAME}" 37 | echo " log has been saved as ${LOGNAME}" >> "${TATT_REPORTFILE}" 38 | TESTLOGS=($(find ${BUILDDIR}/work -iname '*test*log*')) 39 | if [ ${#TESTLOGS[@]} -gt 0 ]; then 40 | tar cf ${LOGNAME}.tar ${TESTLOGS[@]} 41 | echo " testsuite logs have been saved as ${LOGNAME}.tar" >> "${TATT_REPORTFILE}" 42 | fi 43 | fi 44 | } 45 | 46 | function tatt_test_pkg 47 | { 48 | if [ "${1:?}" == "--test" ]; then 49 | shift 50 | 51 | # Do a first pass to avoid circular dependencies 52 | # --onlydeps should mean we're avoiding (too much) duplicate work 53 | USE="${USE} minimal -doc" emerge --onlydeps -q1 --with-test-deps ${TATT_EMERGEOPTS} "${1:?}" 54 | 55 | if ! emerge --onlydeps -q1 --with-test-deps ${TATT_EMERGEOPTS} "${1:?}"; then 56 | echo "merging test dependencies of ${1} failed" >> "${TATT_REPORTFILE}" 57 | return 1 58 | fi 59 | TFEATURES="${FEATURES} test" 60 | else 61 | TFEATURES="${FEATURES}" 62 | fi 63 | 64 | # --usepkg-exclude needs the package name, so let's extract it 65 | # from the atom we have 66 | local name=$(portageq pquery "${1:?}" -n) 67 | 68 | eout=$( FEATURES="${TFEATURES}" emerge -1 --getbinpkg=n --usepkg-exclude="${name}" ${TATT_EMERGEOPTS} "${1:?}" 2>&1 1>/dev/tty ) 69 | if [[ $? == 0 ]] ; then 70 | if [ -n "${TFEATURES}" ]; then 71 | echo -n "FEATURES='${TFEATURES}' " >> "${TATT_REPORTFILE}" 72 | fi 73 | echo "USE='${USE}' succeeded for ${1:?}" >> "${TATT_REPORTFILE}" 74 | else 75 | FEATURES="${TFEATURES}" tatt_pkg_error "${1:?}" "${eout}" 76 | return 1 77 | fi 78 | } 79 | -------------------------------------------------------------------------------- /templates/updatebug: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright 2016 Gentoo Foundation 3 | # Distributed under the terms of the GNU General Public License v2 4 | 5 | import portage.versions 6 | import requests 7 | import sys 8 | from tatt.tattConfig import tattConfig as tattConfig 9 | 10 | 11 | def main(): 12 | tattconfig = tattConfig() 13 | session = requests.Session() 14 | session.params.update({'Bugzilla_api_key': tattconfig['bugzilla-key']}) 15 | 16 | bug_url = tattconfig['bugzilla-url'] 17 | bug_id = str(@@BUG@@) 18 | 19 | params = {'id': bug_id} 20 | response = session.get(tattconfig['bugzilla-url'] + '/rest/bug', params=params).json() 21 | 22 | if 'message' in response: 23 | print(response['message']) 24 | sys.exit(1) 25 | 26 | response = response['bugs'][0] 27 | 28 | has_my_arch = False 29 | has_other_arches = False 30 | for cc in response['cc']: 31 | body, domain = cc.split('@', 1) 32 | if domain == 'gentoo.org': 33 | if body == '@@ARCH@@': 34 | has_my_arch = True 35 | elif body in portage.archlist: 36 | has_other_arches = True 37 | 38 | # We don't close bugs which still have other arches for obvious reasons, 39 | # and security bugs because stabilization is not the last step for them. 40 | comment = "@@ARCH@@ done" 41 | params['cc'] = {} 42 | if has_my_arch: 43 | params['cc']['remove'] = ['@@ARCH@@@gentoo.org'] 44 | params['comment'] = {} 45 | if has_other_arches or 'Security' in response['product']: 46 | params['comment']['body'] = comment 47 | else: 48 | params['comment']['body'] = comment + '\r\n\r\nall arches done' 49 | params['status'] = 'RESOLVED' 50 | params['resolution'] = 'FIXED' 51 | 52 | session.put(tattconfig['bugzilla-url'] + '/rest/bug/' + bug_id, json=params) 53 | 54 | if __name__ == '__main__': 55 | main() 56 | -------------------------------------------------------------------------------- /templates/use-footer: -------------------------------------------------------------------------------- 1 | 2 | exit ${test_ret} 3 | -------------------------------------------------------------------------------- /templates/use-header: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #USE-Flag build tests for job @@JOB@@ 3 | 4 | trap "echo 'signal captured, exiting the entire script...'; exit" SIGHUP SIGINT SIGTERM 5 | 6 | export TATT_TEST_TYPE="use" 7 | export TATT_REPORTFILE="@@REPORTFILE@@" 8 | export TATT_BUILDLOGDIR="@@BUILDLOGDIR@@" 9 | export TATT_EMERGEOPTS="@@EMERGEOPTS@@" 10 | 11 | source "@@TEMPLATEDIR@@tatt_functions.sh" 12 | 13 | echo -e "USE tests started on $(date)\n" >> @@REPORTFILE@@ 14 | 15 | test_ret=0 16 | -------------------------------------------------------------------------------- /templates/use-loop: -------------------------------------------------------------------------------- 1 | 2 | # Code for @@CPV@@ 3 | @@LOOP_BODY@@ 4 | echo >> @@REPORTFILE@@ 5 | -------------------------------------------------------------------------------- /templates/use-snippet: -------------------------------------------------------------------------------- 1 | @@USE@@ tatt_test_pkg "@@CPV@@" || test_ret=1 2 | -------------------------------------------------------------------------------- /templates/use-test-snippet: -------------------------------------------------------------------------------- 1 | @@USE@@ tatt_test_pkg --test "@@CPV@@" || test_ret=1 2 | --------------------------------------------------------------------------------