├── .gitignore ├── INSTALL.md ├── LICENSE ├── LICENSE.LESSER ├── OLD_BRANCHES ├── README.md ├── analyses ├── __init__.py ├── derivative.py ├── discipline.py ├── featurelocations.py ├── general.py ├── generalvalues.py └── interaction.py ├── cppstats.sh ├── cppstats ├── __init__.py ├── analysis.py ├── cli.py ├── cppstats.py └── preparation.py ├── cppstats_input.txt ├── lib ├── __init__.py └── cpplib.py ├── preparations ├── __init__.py ├── deleteComments.xsl ├── deleteIncludeGuards.py ├── rewriteIfdefs.py └── rewriteMultilineMacros.py ├── scripts ├── __init__.py ├── ascope.py ├── convert2xml.sh ├── disable_errorneous_files.sh ├── ifdefendifratio.py ├── partial_preprocessor.py └── reversecpp.py └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | ### cppstats folders 2 | ## log folder 3 | log/ 4 | ## temporary files folder 5 | tmp/ 6 | ## folder of sample projects used locally 7 | projects/ 8 | ## virtual environment 9 | virtualenv/ 10 | 11 | 12 | ### cppstats temporary files 13 | ## sample cppstats list-input 14 | cppstats_input.txt 15 | 16 | ## backup or compiled files 17 | *.pyc 18 | *.orig 19 | *.bak 20 | *.swp 21 | *.swo 22 | 23 | 24 | ### Python template 25 | # Distribution / packaging 26 | *.egg-info/ 27 | .eggs/ 28 | build/ 29 | dist/ 30 | 31 | ### JetBrains files 32 | # Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm 33 | .idea/ 34 | 35 | 36 | ### TeX files 37 | ## Core latex/pdflatex auxiliary files: 38 | *.aux 39 | *.lof 40 | *.log 41 | *.lot 42 | *.fls 43 | *.out 44 | *.toc 45 | 46 | ## Bibliography auxiliary files (bibtex/biblatex/biber): 47 | *.bbl 48 | *.bcf 49 | *.blg 50 | *-blx.aux 51 | *-blx.bib 52 | *.brf 53 | *.run.xml 54 | 55 | ## Build tool auxiliary files: 56 | *.fdb_latexmk 57 | *.synctex 58 | *.synctex.gz 59 | *.synctex.gz(busy) 60 | *.pdfsync 61 | 62 | ## Auxiliary and intermediate files from other packages: 63 | 64 | # beamer 65 | *.nav 66 | *.snm 67 | *.vrb 68 | 69 | # todonotes 70 | *.tdo 71 | -------------------------------------------------------------------------------- /INSTALL.md: -------------------------------------------------------------------------------- 1 | # cppstats installation 2 | 3 | 4 | ## Overview 5 | 6 | cppstats should be runnable under following systems: 7 | 8 | * Linux/Ubuntu, 9 | * Mac OS X, and 10 | * Cygwin. 11 | 12 | In detail, cppstats was successfully tested under: 13 | 14 | - Ubuntu 12.04 64-bit, Python 2.7.3, 15 | - Ubuntu 14.04 64-bit, Python 2.7.6, and 16 | - Cygwin 64-bit, Python 2.7.3 and Python 2.10.3. 17 | 18 | Right now, Python 3.x is NOT supported. 19 | 20 | Current tested version of srcML: 21 | `srcML Beta (v0.9.5)` 22 | 23 | 24 | ## xUBUNTU 25 | 26 | 1. checkout repo 27 | 28 | ```bash 29 | git clone https://github.com/clhunsen/cppstats.git 30 | ``` 31 | 32 | 2. install needed libraries (+ all dependencies) 33 | 34 | ```bash 35 | sudo apt-get install astyle # artistic style (code formatter) 36 | sudo apt-get install xsltproc # XSLT 1.0 command line processor 37 | sudo apt-get install libxml2 libxml2-dev # library for processing XML 38 | sudo apt-get install gcc # GNU compiler collection 39 | sudo apt-get install python-dev libxml2-dev libxslt-dev zlib1g-dev # cppstats setuptools builds some dependencies from scratch, so these development packages are required 40 | ``` 41 | 42 | - download and install srcML libraries 43 | - download the deb binary package that is sufficient for your platform from: http://www.srcml.org/#download 44 | - install the deb package; srcml is available from the command line via 'srcml' 45 | 46 | 3. install Python package for cppstats and defined Python dependencies 47 | 48 | ```bash 49 | sudo python setup.py install 50 | ``` 51 | 52 | Optionally, you can install it for the current user by appending `--user` to the command. 53 | 54 | If you want to install the package in *development mode*, substitute `install` with `develop`. 55 | 56 | Run `cppstats --help` for further instructions. 57 | 58 | 4. supply cppstats with the appropriate paths in `cppstats_input.txt` 59 | 60 | * use full paths like `/local/repos/mpsolve/mpsolve-2.2` 61 | * each project folder given in the file has to be structured as follows: 62 | 63 | ``` 64 | > /local/repos/cpp-history/mpsolve/mpsolve-2.2/ 65 | > source/ (here are the C source files) 66 | ``` 67 | 68 | 69 | 70 | CYGWIN 71 | ------ 72 | 73 | 1. checkout repo 74 | 75 | ```bash 76 | git clone https://github.com/clhunsen/cppstats.git 77 | ``` 78 | 79 | * **NOTE:** The old branch 'cygwin' is discontinued after version 0.8. 80 | * Git for Windows: 81 | * https://github.com/msysgit/msysgit/releases/ 82 | * git bash only 83 | * checkout as-is, commit as-is 84 | 85 | 2. install needed libraries (+ all dependencies) 86 | 87 | - https://cygwin.com/install.html 88 | - install from internet 89 | - select any mirror 90 | - do not install any packages yet 91 | - wait setup to finish 92 | 93 | - **NOTE:** your Windows drives are located at `/cygdrive/`! 94 | 95 | - run following command within cygwin terminal: 96 | 97 | ```bash 98 | cygstart -- /path/to/cygwin-setup.exe -K http://cygwinports.org/ports.gpg 99 | ``` 100 | 101 | - go through installation process as before, **but** add and select following download site: 102 | 103 | ``` 104 | ftp://ftp.cygwinports.org/pub/cygwinports 105 | ``` 106 | 107 | - install following packages (version number, newer versions should 108 | also work, except for Python 3.x): 109 | 110 | ``` 111 | - All/Python/ 112 | python (2.7.3-1) 113 | python-setuptools (15.2-1) 114 | - All/Libs/ 115 | libxml2 (2.9.1-1) 116 | libxml2-devel (2.9.1-1) 117 | libxslt (1.1.27-2) 118 | libxslt-devel (1.1.27-2) 119 | zlib (1.2.8-3) 120 | zlib-devel (1.2.8-3) 121 | - All/Utils/ 122 | astyle (2.03-1) 123 | - All/Devel/ 124 | gcc (4.7.3-1) 125 | + all dependencies you are prompted for 126 | ``` 127 | 128 | - download and install srcML libraries 129 | - download a binary package that is sufficient for you and your Windows version from: http://www.srcml.org/#download 130 | - extract into your `$PATH`, so that binaries are available directly 131 | - e.g., `C:/Program Files/Windows/system32/srcml.exe` 132 | 133 | 3. install Python package for cppstats and defined Python dependencies 134 | 135 | ```bash 136 | python setup.py develop --user 137 | ``` 138 | 139 | 4. supply cppstats with the appropriate paths in `cppstats_input.txt` 140 | 141 | * use relative paths like `data/mpsolve/mpsolve-2.2` and call cppstats relation to these given paths 142 | * each project folder given in the file has to be structured as follows: 143 | 144 | ``` 145 | > data/mpsolve/mpsolve-2.2 146 | > source/ (here are the C source files) 147 | ``` 148 | -------------------------------------------------------------------------------- /LICENSE.LESSER: -------------------------------------------------------------------------------- 1 | GNU LESSER GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | 9 | This version of the GNU Lesser General Public License incorporates 10 | the terms and conditions of version 3 of the GNU General Public 11 | License, supplemented by the additional permissions listed below. 12 | 13 | 0. Additional Definitions. 14 | 15 | As used herein, "this License" refers to version 3 of the GNU Lesser 16 | General Public License, and the "GNU GPL" refers to version 3 of the GNU 17 | General Public License. 18 | 19 | "The Library" refers to a covered work governed by this License, 20 | other than an Application or a Combined Work as defined below. 21 | 22 | An "Application" is any work that makes use of an interface provided 23 | by the Library, but which is not otherwise based on the Library. 24 | Defining a subclass of a class defined by the Library is deemed a mode 25 | of using an interface provided by the Library. 26 | 27 | A "Combined Work" is a work produced by combining or linking an 28 | Application with the Library. The particular version of the Library 29 | with which the Combined Work was made is also called the "Linked 30 | Version". 31 | 32 | The "Minimal Corresponding Source" for a Combined Work means the 33 | Corresponding Source for the Combined Work, excluding any source code 34 | for portions of the Combined Work that, considered in isolation, are 35 | based on the Application, and not on the Linked Version. 36 | 37 | The "Corresponding Application Code" for a Combined Work means the 38 | object code and/or source code for the Application, including any data 39 | and utility programs needed for reproducing the Combined Work from the 40 | Application, but excluding the System Libraries of the Combined Work. 41 | 42 | 1. Exception to Section 3 of the GNU GPL. 43 | 44 | You may convey a covered work under sections 3 and 4 of this License 45 | without being bound by section 3 of the GNU GPL. 46 | 47 | 2. Conveying Modified Versions. 48 | 49 | If you modify a copy of the Library, and, in your modifications, a 50 | facility refers to a function or data to be supplied by an Application 51 | that uses the facility (other than as an argument passed when the 52 | facility is invoked), then you may convey a copy of the modified 53 | version: 54 | 55 | a) under this License, provided that you make a good faith effort to 56 | ensure that, in the event an Application does not supply the 57 | function or data, the facility still operates, and performs 58 | whatever part of its purpose remains meaningful, or 59 | 60 | b) under the GNU GPL, with none of the additional permissions of 61 | this License applicable to that copy. 62 | 63 | 3. Object Code Incorporating Material from Library Header Files. 64 | 65 | The object code form of an Application may incorporate material from 66 | a header file that is part of the Library. You may convey such object 67 | code under terms of your choice, provided that, if the incorporated 68 | material is not limited to numerical parameters, data structure 69 | layouts and accessors, or small macros, inline functions and templates 70 | (ten or fewer lines in length), you do both of the following: 71 | 72 | a) Give prominent notice with each copy of the object code that the 73 | Library is used in it and that the Library and its use are 74 | covered by this License. 75 | 76 | b) Accompany the object code with a copy of the GNU GPL and this license 77 | document. 78 | 79 | 4. Combined Works. 80 | 81 | You may convey a Combined Work under terms of your choice that, 82 | taken together, effectively do not restrict modification of the 83 | portions of the Library contained in the Combined Work and reverse 84 | engineering for debugging such modifications, if you also do each of 85 | the following: 86 | 87 | a) Give prominent notice with each copy of the Combined Work that 88 | the Library is used in it and that the Library and its use are 89 | covered by this License. 90 | 91 | b) Accompany the Combined Work with a copy of the GNU GPL and this license 92 | document. 93 | 94 | c) For a Combined Work that displays copyright notices during 95 | execution, include the copyright notice for the Library among 96 | these notices, as well as a reference directing the user to the 97 | copies of the GNU GPL and this license document. 98 | 99 | d) Do one of the following: 100 | 101 | 0) Convey the Minimal Corresponding Source under the terms of this 102 | License, and the Corresponding Application Code in a form 103 | suitable for, and under terms that permit, the user to 104 | recombine or relink the Application with a modified version of 105 | the Linked Version to produce a modified Combined Work, in the 106 | manner specified by section 6 of the GNU GPL for conveying 107 | Corresponding Source. 108 | 109 | 1) Use a suitable shared library mechanism for linking with the 110 | Library. A suitable mechanism is one that (a) uses at run time 111 | a copy of the Library already present on the user's computer 112 | system, and (b) will operate properly with a modified version 113 | of the Library that is interface-compatible with the Linked 114 | Version. 115 | 116 | e) Provide Installation Information, but only if you would otherwise 117 | be required to provide such information under section 6 of the 118 | GNU GPL, and only to the extent that such information is 119 | necessary to install and execute a modified version of the 120 | Combined Work produced by recombining or relinking the 121 | Application with a modified version of the Linked Version. (If 122 | you use option 4d0, the Installation Information must accompany 123 | the Minimal Corresponding Source and Corresponding Application 124 | Code. If you use option 4d1, you must provide the Installation 125 | Information in the manner specified by section 6 of the GNU GPL 126 | for conveying Corresponding Source.) 127 | 128 | 5. Combined Libraries. 129 | 130 | You may place library facilities that are a work based on the 131 | Library side by side in a single library together with other library 132 | facilities that are not Applications and are not covered by this 133 | License, and convey such a combined library under terms of your 134 | choice, if you do both of the following: 135 | 136 | a) Accompany the combined library with a copy of the same work based 137 | on the Library, uncombined with any other library facilities, 138 | conveyed under the terms of this License. 139 | 140 | b) Give prominent notice with the combined library that part of it 141 | is a work based on the Library, and explaining where to find the 142 | accompanying uncombined form of the same work. 143 | 144 | 6. Revised Versions of the GNU Lesser General Public License. 145 | 146 | The Free Software Foundation may publish revised and/or new versions 147 | of the GNU Lesser General Public License from time to time. Such new 148 | versions will be similar in spirit to the present version, but may 149 | differ in detail to address new problems or concerns. 150 | 151 | Each version is given a distinguishing version number. If the 152 | Library as you received it specifies that a certain numbered version 153 | of the GNU Lesser General Public License "or any later version" 154 | applies to it, you have the option of following the terms and 155 | conditions either of that published version or of any later version 156 | published by the Free Software Foundation. If the Library as you 157 | received it does not specify a version number of the GNU Lesser 158 | General Public License, you may choose any version of the GNU Lesser 159 | General Public License ever published by the Free Software Foundation. 160 | 161 | If the Library as you received it specifies that a proxy can decide 162 | whether future versions of the GNU Lesser General Public License shall 163 | apply, that proxy's public statement of acceptance of any version is 164 | permanent authorization for you to choose that version for the 165 | Library. 166 | -------------------------------------------------------------------------------- /OLD_BRANCHES: -------------------------------------------------------------------------------- 1 | cygwin was a36de1e 2 | derivan was c9c73d0 3 | interan was acdeafe 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cppstats 2 | 3 | 4 | ## License & Copyright 5 | * Copyright (C) 2009-2015 University of Passau, Germany 6 | 7 | All rights reserved. 8 | 9 | cppstats is covered by the GNU Lesser General Public License. 10 | The full license text is distributed with this software. See the `LICENSE.LESSER` file. 11 | 12 | ### Main Developers 13 | 14 | * Jörg Liebig , University of Passau 15 | * Claus Hunsen , University of Passau 16 | 17 | ### Further Contributors 18 | 19 | * Andreas Ringlstetter , OTH Regensburg 20 | 21 | ## What is it? 22 | 23 | cppstats is a suite of analyses for measuring C preprocessor-based variability in software product lines. 24 | 25 | Currently, cppstats supports following analyses: 26 | 27 | * `general`, 28 | * `generalvalues`, 29 | * `discipline`, 30 | * `featurelocations`, 31 | * `derivative`, and 32 | * `interaction`. 33 | 34 | For detailed information on each kind of analysis, please refer to the corresponding paragraph below in this file. 35 | 36 | For further information, please see the tool's homepage at: http://www.fosd.net/cppstats 37 | 38 | Details of the latest version can be found on the cppstats project site at GitHub under https://github.com/clhunsen/cppstats/. 39 | 40 | 41 | ## System Requirements 42 | 43 | * srcML (http://www.srcml.org/) 44 | * astyle (http://astyle.sourceforge.net/) 45 | * libxml2 (http://www.xmlsoft.org/) 46 | * xsltproc (http://xmlsoft.org/XSLT/xsltproc2.html) 47 | * gcc (https://gcc.gnu.org/) 48 | * Python requirements from `setup.py` 49 | 50 | 51 | ## Installation 52 | 53 | cppstats should be runnable under following systems: 54 | 55 | * Linux/Ubuntu, 56 | * Mac OS X, and 57 | * Cygwin. 58 | 59 | Please see the file called `INSTALL.md` for detailed instructions for each 60 | system. 61 | 62 | In detail, cppstats was successfully tested under: 63 | 64 | * Ubuntu 12.04, Python 2.7.*, and 65 | * Cygwin, Python 2.7.*. 66 | 67 | Right now, Python 3.x is **NOT** supported. 68 | 69 | 70 | ## Quick Start 71 | 72 | - Install cppstats using `sudo python setup.py install`. 73 | 74 | - Supply cppstats with the appropriate paths in `cppstats_input.txt` 75 | 76 | * use full paths like `/local/repos/mpsolve/mpsolve-2.2` 77 | * each project folder given in the file has to be structured as follows: 78 | 79 | ``` 80 | > /local/repos/cpp-history/mpsolve/mpsolve-2.2/ 81 | > source/ (here are the C source files) 82 | ``` 83 | 84 | - Then run: 85 | ``` 86 | $ cppstats --kind 87 | ``` 88 | 89 | `` must be one of the analyses listed in the introduction. Also, have a look on `cppstats --help` for further command line options. 90 | 91 | - The output files for each analysis are written to the folders given in the file `cppstats_input.txt`. 92 | 93 | 94 | ## Analyses 95 | 96 | * `GENERAL` 97 | - Measurement of CPP-related metrics regarding scattering, 98 | tangling, and nesting 99 | - returns a list of metrics for each file, and a list of metric 100 | values for the whole project folder 101 | 102 | * `GENERALVALUES` 103 | - Calculation of scattering, tangling, and nesting values 104 | - allows deactivation of the rewriting of `#ifdefs` to get a 'pure' result 105 | - rewriting changes `#else` branches from no used constant to the 106 | negation of the corresponding `#if` clause 107 | - returns a list for each characteristic, which `#ifdef` or configuration 108 | constant has which value (merged and unmerged) 109 | - unmerged: each `#ifdef` expression is counted once per file 110 | - merged: each `#ifdef` expression is counted once per project 111 | 112 | * `DISCIPLINE` 113 | - Analysis of the discipline of used CPP annotations 114 | - returns the number of occurences for the categories listed below 115 | - (1) check top level siblings (compilation unit)' 116 | - (2) check sibling (excludes check top level siblings; NOT CLASSIFIED)' 117 | - (4) check if-then enframement (wrapper)' 118 | - (8) check case enframement (conditional)' 119 | - (16) check else-if enframement (conditional) 120 | - (32) check param/argument enframement (parameter) 121 | - (64) check expression enframement (expression) 122 | - (128) check else enframement (NOT CLASSIFIED) 123 | 124 | * `FEATURELOCATIONS` 125 | - Analysis of the locations of CPP annotation blocks in the given 126 | file or project folder 127 | - returns a table with the following headers: 128 | - file 129 | - starting line 130 | - end line 131 | - type of the annotation (`#if`, `#elif`, `#else`) 132 | - the `#ifdef` expression 133 | -- involved configuration constants 134 | 135 | * `DERIVATIVE` 136 | - Analysis of all derivatives in the given project folder 137 | - returns all CPP annotations that involve shared code (expression 138 | contains `&&`) 139 | 140 | * `INTERACTION` 141 | - Analysis of pair-wise interactions of configurations constants 142 | that have been used alltogether in one expression (# of constants 143 | involved >= 3) 144 | - (A, B, C) -> |(A, B)? (A, C)? (B, C)? ...| 145 | 146 | ## General Notes 147 | 148 | * When cppstats computes general stats (`--kind general` parameter), the reported granularity 149 | function level (GRANFL) also accounts for conditional elements within an array initialization 150 | or conditional field initializations when creating a struct variable. Example (for array): 151 | 152 | ```c 153 | static const struct squashfs_decompressor *decompressor[] = { 154 | &squashfs_zlib_comp_ops, 155 | &squashfs_lzma_unsupported_comp_ops, 156 | #if defined(CONFIG_SQUASHFS_LZO) 157 | &squashfs_lzo_comp_ops, 158 | #else 159 | &squashfs_lzo_unsupported_comp_ops, 160 | #endif 161 | &squashfs_unknown_comp_ops 162 | }; 163 | ``` 164 | 165 | The rationale behind such decision is that array/struct instance initializations can be interpreted 166 | as constructor procedure calls. 167 | -------------------------------------------------------------------------------- /analyses/__init__.py: -------------------------------------------------------------------------------- 1 | # cppstats is a suite of analyses for measuring C preprocessor-based 2 | # variability in software product lines. 3 | # Copyright (C) 2014-2015 University of Passau, Germany 4 | # 5 | # This program is free software: you can redistribute it and/or modify it 6 | # under the terms of the GNU Lesser General Public License as published 7 | # by the Free Software Foundation, either version 3 of the License, or 8 | # (at your option) any later version. 9 | # 10 | # This program is distributed in the hope that it will be useful, 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # Lesser General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU Lesser General Public 16 | # License along with this program. If not, see 17 | # . 18 | # 19 | # Contributors: 20 | # Claus Hunsen 21 | -------------------------------------------------------------------------------- /analyses/discipline.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2011-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Jörg Liebig 23 | # Claus Hunsen 24 | 25 | 26 | # modules from the std-library 27 | import os 28 | import re 29 | import sys 30 | from argparse import ArgumentParser, RawTextHelpFormatter 31 | 32 | 33 | # ################################################# 34 | # external modules 35 | 36 | # python-lxml module 37 | from lxml import etree 38 | 39 | 40 | def returnFileNames(folder, extfilt = ['.xml']): 41 | '''This function returns all files of the input folder 42 | and its subfolders.''' 43 | filesfound = list() 44 | 45 | if os.path.isdir(folder): 46 | wqueue = [os.path.abspath(folder)] 47 | 48 | while wqueue: 49 | currentfolder = wqueue[0] 50 | wqueue = wqueue[1:] 51 | foldercontent = os.listdir(currentfolder) 52 | tmpfiles = filter(lambda n: os.path.isfile( 53 | os.path.join(currentfolder, n)), foldercontent) 54 | tmpfiles = filter(lambda n: os.path.splitext(n)[1] in extfilt, 55 | tmpfiles) 56 | tmpfiles = map(lambda n: os.path.join(currentfolder, n), 57 | tmpfiles) 58 | filesfound += tmpfiles 59 | tmpfolders = filter(lambda n: os.path.isdir( 60 | os.path.join(currentfolder, n)), foldercontent) 61 | tmpfolders = map(lambda n: os.path.join(currentfolder, n), 62 | tmpfolders) 63 | wqueue += tmpfolders 64 | 65 | return filesfound 66 | 67 | 68 | class DisciplinedAnnotations: 69 | ################################################## 70 | # constants: 71 | __cppnscpp = 'http://www.srcML.org/srcML/cpp' 72 | __cppnsdef = 'http://www.srcML.org/srcML/src' 73 | __cpprens = re.compile('{(.+)}(.+)') 74 | __conditionals = ['if', 'ifdef', 'ifndef', 'else', 'elif', 'endif'] 75 | __conditions = ['if', 'ifdef', 'ifndef'] 76 | outputfile = "cppstats_discipline.csv" 77 | ################################################## 78 | 79 | def __init__(self, folder, options): 80 | 81 | self.opts = options 82 | self.opts.dir = os.path.abspath(folder) 83 | 84 | print self.opts.dir 85 | 86 | if not self.opts.dir: 87 | oparser.print_help() 88 | sys.exit(-1) 89 | 90 | self.overallblocks = 0 91 | self.disciplined = 0 92 | self.undisciplinedknown = 0 93 | self.undisciplinedunknown = 0 94 | self.compilationunit = 0 95 | self.functiontype = 0 96 | self.siblings = 0 97 | self.wrapperif = 0 98 | self.wrapperfor = 0 99 | self.wrapperwhile = 0 100 | self.conditionalcase = 0 101 | self.conditionalelif = 0 102 | self.parameter = 0 103 | self.expression = 0 104 | self.loc = 0 105 | self.checkFiles() 106 | 107 | def __getIfdefAnnotations__(self, root): 108 | '''This method returns all nodes of the xml which are ifdef 109 | annotations in the source code.''' 110 | treeifdefs = list() 111 | 112 | for _, elem in etree.iterwalk(root): 113 | ns, tag = DisciplinedAnnotations.__cpprens.match(elem.tag).\ 114 | groups() 115 | 116 | if ns == DisciplinedAnnotations.__cppnscpp \ 117 | and tag in DisciplinedAnnotations.__conditionals: 118 | treeifdefs.append(elem) 119 | 120 | return treeifdefs 121 | 122 | def __createListFromTreeifdefs__(self, treeifdefs): 123 | '''This method returns a list representation for the input treeifdefs 124 | (xml-objects). Corresponding #ifdef elements are in one sublist.''' 125 | 126 | if not treeifdefs: return [] 127 | 128 | listifdefs = list() 129 | workerlist = list() 130 | for nifdef in treeifdefs: 131 | tag = nifdef.tag.split('}')[1] 132 | if tag in ['if', 'ifdef', 'ifndef']: 133 | workerlist.append(list()) 134 | workerlist[-1].append(nifdef) 135 | elif tag in ['elif', 'else']: 136 | workerlist[-1].append(nifdef) 137 | elif tag in ['endif']: 138 | if not workerlist: 139 | return -1 140 | workerlist[-1].append(nifdef) 141 | last = workerlist[-1] 142 | getpairs = zip(last,last[1:]) 143 | map(lambda i: listifdefs.append(list(i)), getpairs) 144 | workerlist = workerlist[:-1] 145 | else: 146 | print('[ERROR] ill-formed tag (%s) occured in line (%4s).' % (tag, nifdef.sourceline)) 147 | 148 | if workerlist: 149 | return -2 150 | 151 | return listifdefs 152 | 153 | def __filterConditionalPreprocessorDirectives(self, listifdefs): 154 | '''This method filters out all ifdef-endif pairs that annotate only preprocessor directives.''' 155 | # iterate annotated blocks by determining all siblings of the #ifdef and filter out preprocessor 156 | # annotated elements 157 | resultlist = filter(lambda (ifdef, endif): ifdef.getnext() != endif, listifdefs) 158 | print('[INFO] before after: %s <-> %s' % (str(len(listifdefs)), str(len(resultlist)))) 159 | return resultlist 160 | 161 | 162 | PATTLS = 0 # 1 << 0 => 1 163 | def __checkStrictTLSFDPattern__(self, listifdefs): 164 | '''like sibling pattern, but only top level and statement elements are 165 | considered disciplined''' 166 | listundisciplinedknown = list() 167 | listundisciplinedunknown = list() 168 | 169 | for listcorifdef in listifdefs: 170 | nodeifdef = listcorifdef[0] 171 | nodeifdefsibs = [sib for sib in nodeifdef.itersiblings()] 172 | 173 | error=0 174 | for corifdef in listcorifdef[1:]: 175 | if not corifdef in nodeifdefsibs: 176 | error=1 177 | 178 | if error==0: 179 | parenttag = self.__getParentTag__(nodeifdef) 180 | if not parenttag in ['block','public']: 181 | error=1 182 | if error==1: 183 | listundisciplinedunknown.append(listcorifdef) 184 | else: 185 | listundisciplinedknown.append(listcorifdef) 186 | 187 | return (listundisciplinedknown, listundisciplinedunknown) 188 | 189 | 190 | def __checkStrictTLSCUPattern__(self, listifdefs): 191 | '''This method checks all patterns, if they occur right under the root element 192 | of the grammer, here unit.''' 193 | listundisciplinedknown = list() 194 | listundisciplinedunknown = list() 195 | 196 | for listcorifdef in listifdefs: 197 | nodeifdef = listcorifdef[0] 198 | nodeifdefsibs = [sib for sib in nodeifdef.itersiblings()] 199 | 200 | error=0 201 | for corifdef in listcorifdef[1:]: 202 | if not corifdef in nodeifdefsibs: 203 | error=1 204 | 205 | if error==0: 206 | parenttag = self.__getParentTag__(nodeifdef) 207 | if not parenttag in ['unit']: 208 | error=1 209 | if error==1: 210 | listundisciplinedunknown.append(listcorifdef) 211 | else: 212 | listundisciplinedknown.append(listcorifdef) 213 | 214 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 215 | return (listundisciplinedknown, listundisciplinedunknown) 216 | 217 | 218 | def __checkStrictPattern__(self, listifdefs): 219 | '''This pattern checks the annotation of functions, where the XML markup 220 | of src2srcml is ill-formed. TODO might be fixed in future versions of 221 | src2srcml. Example is: 222 | void foo(k) 223 | int k; 224 | { 225 | // some lines of code 226 | } 227 | ''' 228 | listundisciplinedknown = list() 229 | listundisciplinedunknown = list() 230 | 231 | for listcorifdef in listifdefs: 232 | if len(listcorifdef) != 2: 233 | listundisciplinedunknown.append(listcorifdef) 234 | continue 235 | 236 | nodeifdef = listcorifdef[0] 237 | nodeendif = listcorifdef[1] 238 | func = nodeendif.getparent() 239 | 240 | if func != None and func.tag.split('}')[1] == 'function': 241 | nodefuncsibs = [sib for sib in func.itersiblings(preceding=True)] 242 | if nodeifdef == nodefuncsibs[0]: 243 | if self.opts.verbose: 244 | print('[INFO] ill-formed compilation unit pattern occured in line (%4s).' % nodeifdef.sourceline) 245 | listundisciplinedknown.append(listcorifdef) 246 | continue 247 | 248 | listundisciplinedunknown.append(listcorifdef) 249 | 250 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 251 | return (listundisciplinedknown, listundisciplinedunknown) 252 | 253 | 254 | PATSIB = 1 # 1 << 1 => 2 255 | def __checkSiblingPattern__(self, listifdefs): 256 | '''This method returns a tuple with (listdisciplined, 257 | listundisciplined) #ifdef elements. The pattern works on the basis 258 | of the sibling pattern. If the xml elements of #if-#elif-#else-#endif 259 | are siblings, we determine them as disciplined.''' 260 | listundisciplinedknown = list() 261 | listundisciplinedunknown = list() 262 | 263 | for listcorifdef in listifdefs: 264 | nodeifdef = listcorifdef[0] 265 | nodeifdefsibs = [sib for sib in nodeifdef.itersiblings()] 266 | 267 | error=0; 268 | for corifdef in listcorifdef[1:]: 269 | if not corifdef in nodeifdefsibs: 270 | error=1 271 | if error==1: 272 | listundisciplinedunknown.append(listcorifdef) 273 | else: 274 | listundisciplinedknown.append(listcorifdef) 275 | 276 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 277 | return (listundisciplinedknown, listundisciplinedunknown) 278 | 279 | 280 | def __getParentTag__(self, tag): 281 | parent = tag.getparent() 282 | return parent.tag.split('}')[1] 283 | 284 | 285 | PATIFTHEN = 2 # 1 << 2 => 4 286 | def __checkIfThenPattern__(self, listifdefs): 287 | '''This method returns a tuple with (listdisciplined, 288 | listundisciplined) #ifdef elements. The pattern matches the following 289 | situation. The if-then in C is enframed by #if-#endif. The else part of 290 | the if-then in C is not enframed. The sibling pattern does not work here 291 | since the annatation cannot work properly here.''' 292 | listundisciplinedknown = list() 293 | listundisciplinedunknown = list() 294 | 295 | for listcorifdef in listifdefs: 296 | if len(listcorifdef) != 2: 297 | listundisciplinedunknown.append(listcorifdef) 298 | continue 299 | 300 | # check that the endif is the first child of its parent and the parent 301 | # is an else 302 | ifdef = listcorifdef[0] 303 | ifdefsibs = [sib for sib in ifdef.itersiblings()] 304 | 305 | # first sibling of starting ifdef must be an if 306 | if len(ifdefsibs) == 0 or ifdefsibs[0].tag.split('}')[1] != 'if': 307 | listundisciplinedunknown.append(listcorifdef) 308 | continue 309 | 310 | # parent of endif must be either an else or an then (if) 311 | endif = listcorifdef[1] 312 | poselse = endif.getparent() 313 | poselsetag = poselse.tag.split('}')[1] 314 | if poselsetag in ['else', 'then']: 315 | if self.opts.verbose: 316 | print('[INFO] if-then pattern occured in line (%4s).' % poselse.sourceline) 317 | listundisciplinedknown.append(listcorifdef) 318 | else: 319 | listundisciplinedunknown.append(listcorifdef) 320 | 321 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 322 | return (listundisciplinedknown, listundisciplinedunknown) 323 | 324 | #TODO 325 | def __checkForWrapperPattern__(self, listifdefs): 326 | '''This method returns a tuple with (listdisciplined, 327 | listundisciplined) #ifdef elements. The pattern matches the following 328 | situation. The for in C is enframed by #if-#endif.''' 329 | listundisciplinedknown = list() 330 | listundisciplinedunknown = list() 331 | 332 | for listcorifdef in listifdefs: 333 | if len(listcorifdef) != 2: 334 | listundisciplinedunknown.append(listcorifdef) 335 | continue 336 | 337 | # check that the endif is the first child of its parent and the parent 338 | # is an else 339 | ifdef = listcorifdef[0] 340 | ifdefsibs = [sib for sib in ifdef.itersiblings()] 341 | 342 | # first sibling of starting ifdef must be an for 343 | if len(ifdefsibs) == 0 or ifdefsibs[0].tag.split('}')[1] != 'for': 344 | listundisciplinedunknown.append(listcorifdef) 345 | continue 346 | 347 | # parent of endif must be either an else or an then (if) 348 | endif = listcorifdef[1] 349 | poselse = endif.getparent() 350 | poselsetag = poselse.tag.split('}')[1] 351 | if poselsetag in ['else', 'then']: 352 | if self.opts.verbose: 353 | print('[INFO] if-then pattern occured in line (%4s).' % poselse.sourceline) 354 | listundisciplinedknown.append(listcorifdef) 355 | else: 356 | listundisciplinedunknown.append(listcorifdef) 357 | 358 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 359 | return (listundisciplinedknown, listundisciplinedunknown) 360 | 361 | 362 | PATCASE = 3 # 1 << 3 => 8 363 | def __checkCasePattern__(self, listifdefs): 364 | '''The method checks the case-block pattern; the #ifdef enframes a case block 365 | of a switch case.''' 366 | listundisciplinedknown = list() 367 | listundisciplinedunknown = list() 368 | 369 | for listcorifdef in listifdefs: 370 | # pattern works only for #if-#endif combinations 371 | if len(listcorifdef) > 2: 372 | listundisciplinedunknown.append(listcorifdef) 373 | continue 374 | 375 | # get endif and check whether parent is a case 376 | nodeendif = listcorifdef[-1] 377 | parenttag = self.__getParentTag__(nodeendif) 378 | if parenttag in ['case']: 379 | if self.opts.verbose: 380 | print('[INFO] case pattern occured in line (%4s).' % nodeendif.sourceline) 381 | listundisciplinedknown.append(listcorifdef) 382 | else: 383 | listundisciplinedunknown.append(listcorifdef) 384 | 385 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 386 | return (listundisciplinedknown, listundisciplinedunknown) 387 | 388 | PATELSEIF = 4 # 1 << 4 => 16 389 | def __checkElseIfPattern__(self, listifdefs): 390 | '''The method check the elseif-block pattern; the #ifdef enframes an elseif 391 | block in an if-then-else.''' 392 | listundisciplinedknown = list() 393 | listundisciplinedunknown = list() 394 | 395 | for listcorifdef in listifdefs: 396 | # pattern works only for #if-#endif combinations 397 | if len(listcorifdef) > 2: 398 | listundisciplinedunknown.append(listcorifdef) 399 | continue 400 | 401 | # get the endif 402 | # endif parent -> then 403 | # then parent -> if 404 | # if parent -> else 405 | # else parent -> #ifdef 406 | nodeendif = listcorifdef[-1] 407 | thensib = nodeendif.getprevious() 408 | if thensib == None: 409 | listundisciplinedunknown.append(listcorifdef) 410 | continue 411 | if thensib.tag.split('}')[1] not in ['then']: 412 | listundisciplinedunknown.append(listcorifdef) 413 | continue 414 | ifparent = thensib.getparent() 415 | if ifparent.tag.split('}')[1] not in ['if']: 416 | listundisciplinedunknown.append(listcorifdef) 417 | continue 418 | elseparent = ifparent.getparent() 419 | if elseparent.tag.split('}')[1] not in ['else']: 420 | listundisciplinedunknown.append(listcorifdef) 421 | continue 422 | ifdefsib = elseparent.getprevious() 423 | 424 | if ifdefsib != listcorifdef[0]: 425 | if self.opts.verbose: 426 | print('[INFO] else-if pattern occured in line (%4s).' % ifdefsib.sourceline) 427 | listundisciplinedunknown.append(listcorifdef) 428 | else: 429 | listundisciplinedknown.append(listcorifdef) 430 | 431 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 432 | return (listundisciplinedknown, listundisciplinedunknown) 433 | 434 | PATPARAM = 5 # 1 << 5 => 32 435 | def __checkParameter__(self, listifdefs): 436 | '''The method checks whether an #ifdef enframes a parameter of a function; 437 | includes function definitions and function calls.''' 438 | listundisciplinedknown = list() 439 | listundisciplinedunknown = list() 440 | 441 | for listcorifdef in listifdefs: 442 | # pattern works only for #if-#endif combinations 443 | if len(listcorifdef) > 2: 444 | listundisciplinedunknown.append(listcorifdef) 445 | continue 446 | 447 | nodeifdef = listcorifdef[0] 448 | nodeifdefsibs = [sib for sib in nodeifdef.itersiblings()] 449 | 450 | error = 0 451 | for corifdef in listcorifdef[1:]: 452 | if not corifdef in nodeifdefsibs: 453 | error = 1 454 | 455 | if error == 0: 456 | # check whether node is an argument or parameter 457 | parenttag = self.__getParentTag__(nodeifdef) 458 | if not parenttag in ['argument_list','parameter_list']: 459 | error = 1 460 | 461 | firstsib = nodeifdefsibs[0] 462 | if firstsib.tag.split('}')[1] not in ['argument', 'param']: 463 | error = 1 464 | if error == 1: 465 | listundisciplinedunknown.append(listcorifdef) 466 | else: 467 | if self.opts.verbose: 468 | print('[INFO] param/argument pattern occured in line (%4s).' % nodeifdef.sourceline) 469 | listundisciplinedknown.append(listcorifdef) 470 | 471 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 472 | return (listundisciplinedknown, listundisciplinedunknown) 473 | 474 | PATEXP = 6 # 1 << 5 => 64 475 | def __checkExpression__(self, listifdefs): 476 | '''The method checks whether an #ifdef enframes an expression of a condition.''' 477 | listundisciplinedknown = list() 478 | listundisciplinedunknown = list() 479 | 480 | for listcorifdef in listifdefs: 481 | # pattern works only for #if-#endif combinations 482 | if len(listcorifdef) > 2: 483 | listundisciplinedunknown.append(listcorifdef) 484 | continue 485 | 486 | error = 0 487 | nodeifdef = listcorifdef[0] 488 | 489 | # get parent and check whether its tag is expr 490 | exppar = nodeifdef.getparent() 491 | exppartag = exppar.tag.split('}')[1] 492 | 493 | if not exppartag == 'expr': 494 | error = 1 495 | 496 | if error == 0: 497 | conpar = exppar.getparent() 498 | conpartag = conpar.tag.split('}')[1] 499 | 500 | if not conpartag == 'condition': 501 | error = 1 502 | 503 | if error == 1: 504 | listundisciplinedunknown.append(listcorifdef) 505 | else: 506 | if self.opts.verbose: 507 | print('[INFO] expression pattern occured in line (%4s).' % nodeifdef.sourceline) 508 | listundisciplinedknown.append(listcorifdef) 509 | 510 | assert len(listifdefs) == len(listundisciplinedknown)+len(listundisciplinedunknown) 511 | return (listundisciplinedknown, listundisciplinedunknown) 512 | 513 | 514 | def __iterateUnknownPatterns__(self, listifdefs, file): 515 | '''This method iterates of the unknown patterns and prints out information 516 | about the pattern: file and line.''' 517 | for ifdef in listifdefs: 518 | if self.opts.log: 519 | print('[INFO] Unknown pattern in file (%s) and line (%s)' % \ 520 | (file, ifdef[0].sourceline)) 521 | 522 | def __checkDiscipline__(self, treeifdefs, file): 523 | '''This method checks a number of patterns in the given treeifdefs. 524 | The checks are in that order, that ifdef patterns not recognized 525 | are passed to the next pattern.''' 526 | listundisciplined = self.__createListFromTreeifdefs__(treeifdefs) 527 | listundisciplined = self.__filterConditionalPreprocessorDirectives(listundisciplined) 528 | if (listundisciplined == -1): 529 | print('[ERROR] Too many #endifs in file (%s)' % file) 530 | return 531 | if (listundisciplined == -2): 532 | print('[ERROR] Not enough #endifs in file (%s)' % file) 533 | return 534 | self.overallblocks += len(listundisciplined) 535 | 536 | # check TLS pattern, subset of sibling pattern 537 | if (self.opts.disc_all or self.opts.check & (1 << DisciplinedAnnotations.PATTLS)): 538 | listifdefs = list(listundisciplined) 539 | (listdisciplined, listundisciplined) = \ 540 | self.__checkStrictTLSCUPattern__(listifdefs) 541 | self.compilationunit += len(listdisciplined) 542 | self.disciplined += len(listdisciplined) 543 | 544 | # checking fd pattern (part of tls) 545 | listifdefs = list(listundisciplined) 546 | (listdisciplined, listundisciplined) = \ 547 | self.__checkStrictTLSFDPattern__(listifdefs) 548 | self.functiontype += len(listdisciplined) 549 | self.disciplined += len(listdisciplined) 550 | 551 | # checking ill-formed compilation unit pattern 552 | listifdefs = list(listundisciplined) 553 | (listdisciplined, listundisciplined) = \ 554 | self.__checkStrictPattern__(listifdefs) 555 | self.compilationunit += len(listdisciplined) 556 | self.disciplined += len(listdisciplined) 557 | 558 | # check if-then pattern 559 | if (self.opts.disc_all or self.opts.check & (1 << DisciplinedAnnotations.PATIFTHEN)): 560 | listifdefs = list(listundisciplined) 561 | (listdisciplined, listundisciplined) = \ 562 | self.__checkIfThenPattern__(listifdefs) 563 | self.wrapperif += len(listdisciplined) 564 | self.undisciplinedknown += len(listdisciplined) 565 | 566 | # check case pattern 567 | if (self.opts.disc_all or self.opts.check & (1 << DisciplinedAnnotations.PATCASE)): 568 | listifdefs = list(listundisciplined) 569 | (listdisciplined, listundisciplined) = \ 570 | self.__checkCasePattern__(listifdefs) 571 | self.conditionalcase += len(listdisciplined) 572 | self.undisciplinedknown += len(listdisciplined) 573 | 574 | # check else-if pattern 575 | if (self.opts.disc_all or self.opts.check & (1 << DisciplinedAnnotations.PATELSEIF)): 576 | listifdefs = list(listundisciplined) 577 | (listdisciplined, listundisciplined) = \ 578 | self.__checkElseIfPattern__(listifdefs) 579 | self.conditionalelif += len(listdisciplined) 580 | self.undisciplinedknown += len(listdisciplined) 581 | 582 | # check param pattern 583 | if (self.opts.disc_all or self.opts.check & (1 << DisciplinedAnnotations.PATPARAM)): 584 | listifdefs = list(listundisciplined) 585 | (listdisciplined, listundisciplined) = \ 586 | self.__checkParameter__(listifdefs) 587 | self.parameter += len(listdisciplined) 588 | self.undisciplinedknown += len(listdisciplined) 589 | 590 | # check expression pattern 591 | if (self.opts.disc_all or self.opts.check & (1 << DisciplinedAnnotations.PATEXP)): 592 | listifdefs = list(listundisciplined) 593 | (listdisciplined, listundisciplined) = \ 594 | self.__checkExpression__(listifdefs) 595 | self.expression += len(listdisciplined) 596 | self.undisciplinedknown += len(listdisciplined) 597 | 598 | # check sibling pattern; check this late because pattern might match for others as well 599 | if (self.opts.disc_all or self.opts.check & (1 << DisciplinedAnnotations.PATSIB)): 600 | listifdefs = list(listundisciplined) 601 | (listdisciplined, listundisciplined) = \ 602 | self.__checkSiblingPattern__(listifdefs) 603 | self.siblings += len(listdisciplined) 604 | self.disciplined += len(listdisciplined) 605 | 606 | # wrap up listundisciplined 607 | self.__iterateUnknownPatterns__(listundisciplined, file) 608 | self.undisciplinedunknown += len(listundisciplined) 609 | 610 | def checkFile(self, file): 611 | try: 612 | tree = etree.parse(file) 613 | f = open(file, 'r') 614 | except etree.XMLSyntaxError: 615 | print('ERROR: file (%s) is not valid. Skipping it.' % file) 616 | return 617 | 618 | # get LOC 619 | self.loc += len(f.readlines())-2; 620 | 621 | # get root of the xml and iterate over it 622 | root = tree.getroot() 623 | treeifdefs = self.__getIfdefAnnotations__(root) 624 | try: 625 | self.__checkDiscipline__(treeifdefs, file) 626 | except: 627 | print('[ERROR]: file (%s) is not valid. Skipping it.' % file) 628 | return 629 | 630 | def checkFiles(self): 631 | xmlfiles = returnFileNames(self.opts.dir, ['.xml']) 632 | for xmlfile in xmlfiles: 633 | print('[INFO] checking file %s' % xmlfile) 634 | self.checkFile(xmlfile) 635 | projectpath = os.path.dirname(self.opts.dir) 636 | projectname = os.path.basename(projectpath) 637 | fd = open( 638 | os.path.join( 639 | projectpath, 640 | DisciplinedAnnotations.outputfile 641 | ), 'w') 642 | 643 | ratio = 0 644 | if (self.overallblocks > 0): 645 | ratio = self.disciplined/(0.0 + self.overallblocks) 646 | fd.write("projectname" 647 | +";"+"loc" 648 | +";"+"compilationunit" 649 | +";"+"functiontype" 650 | +";"+"siblings" 651 | +";"+"wrapperif" 652 | +";"+"conditionalcase" 653 | +";"+"conditionalelif" 654 | +";"+"parameter" 655 | +";"+"expression" 656 | +";"+"undisciplinedknown" 657 | +";"+"undisciplinedunknown" 658 | +";"+"disciplined/overallblocks" 659 | +";"+"overallblocks"+"\n") 660 | fd.write(projectname 661 | +";"+str(self.loc) 662 | +";"+str(self.compilationunit) 663 | +";"+str(self.functiontype) 664 | +";"+str(self.siblings) 665 | +";"+str(self.wrapperif) 666 | +";"+str(self.conditionalcase) 667 | +";"+str(self.conditionalelif) 668 | +";"+str(self.parameter) 669 | +";"+str(self.expression) 670 | +";"+str(self.undisciplinedknown) 671 | +";"+str(self.undisciplinedunknown) 672 | +";"+str(ratio) 673 | +";"+str(self.overallblocks)+"\n") 674 | 675 | 676 | # ################################################## 677 | # add command line options 678 | 679 | def addCommandLineOptionsMain(optionparser): 680 | ''' add command line options for a direct call of this script''' 681 | optionparser.add_argument('-d', '--dir', '--folder', dest='dir', 682 | help='input directory (mandatory)') 683 | 684 | 685 | def addCommandLineOptions(optionparser) : 686 | optionparser.description = 'This analysis counts the number of the disciplined CPP usage in software projects. \n' \ 687 | 'To this end, it checks xml representations of header and source ' \ 688 | 'files and returns the number of disciplined ifdefs in those. \n' 689 | # TODO what are the disciplined ones 690 | #'Disciplined annotations are:' 691 | 692 | optionparser.add_argument('-l', '--log', dest='log', action="store_true", 693 | default=True, help='log to stdout [default=%(default)s]') 694 | optionparser.add_argument('-v', '--verbose', dest='verbose', action="store_true", 695 | default=False, help='verbose output [default=%(default)s]') 696 | optionparser.add_argument('--check', dest='check', type=int, 697 | default=1, help='CHECK sets the patterns that are checked [default=%(default)s].\n' 698 | 'Supply sum of wanted patterns:\n' 699 | '(1) check top level siblings (compilation unit) \n' 700 | '(2) check sibling (excludes check top level siblings; NOT CLASSIFIED) \n' 701 | '(4) check if-then enframement (wrapper) \n' 702 | '(8) check case enframement (conditional) \n' 703 | '(16) check else-if enframement (conditional) \n' 704 | '(32) check param/argument enframement (parameter) \n' 705 | '(64) check expression enframement (expression) \n' 706 | '(128) check else enframement (NOT CLASSIFIED) \n' 707 | ) 708 | optionparser.add_argument('--dall', dest='disc_all', action="store_true", 709 | default=True, help='check all patterns [default=%(default)s] \n(overrides --check)') 710 | 711 | 712 | # ################################################ 713 | # path of the main output file 714 | 715 | def getResultsFile(): 716 | return DisciplinedAnnotations.outputfile 717 | 718 | 719 | ################################################## 720 | if __name__ == '__main__': 721 | 722 | ################################################## 723 | # options parsing 724 | parser = ArgumentParser(formatter_class=RawTextHelpFormatter) 725 | addCommandLineOptionsMain(parser) 726 | addCommandLineOptions(parser) 727 | 728 | options = parser.parse_args() 729 | 730 | # main 731 | DisciplinedAnnotations(options.dir, options) 732 | -------------------------------------------------------------------------------- /analyses/featurelocations.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2014-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Claus Hunsen 23 | 24 | 25 | # modules from the std-library 26 | import csv 27 | import os 28 | import re 29 | import sys 30 | import xmlrpclib 31 | from argparse import ArgumentParser, RawTextHelpFormatter 32 | 33 | 34 | # ################################################# 35 | # path adjustments, so that all imports can be done relative to these paths 36 | 37 | __lib_subfolder = "lib" 38 | sys.path.append(os.path.abspath(__lib_subfolder)) # lib subfolder 39 | 40 | 41 | # ################################################# 42 | # external modules 43 | 44 | # enums 45 | from enum import Enum 46 | # python-lxml module 47 | from lxml import etree 48 | # pyparsing module 49 | import pyparsing as pypa 50 | pypa.ParserElement.enablePackrat() # speed up parsing 51 | sys.setrecursionlimit(8000) # handle larger expressions 52 | 53 | 54 | # ################################################# 55 | # config: 56 | __outputfile = "cppstats_featurelocations.csv" 57 | __listoffeaturesfile = "listoffeatures.csv" 58 | 59 | 60 | # ################################################# 61 | # constants: 62 | 63 | # namespace-constant for src2srcml 64 | _cppnscpp = 'http://www.srcML.org/srcML/cpp' 65 | __cppnsdef = 'http://www.srcML.org/srcML/src' 66 | __cpprens = re.compile('{(.+)}(.+)') 67 | 68 | # conditionals - necessary for parsing the right tags 69 | __conditionals = ['if', 'ifdef', 'ifndef'] 70 | __conditionals_elif = ['elif'] 71 | __conditionals_else = ['else'] 72 | __conditionals_endif = ['endif'] 73 | __conditionals_all = __conditionals + __conditionals_elif + \ 74 | __conditionals_else 75 | __macro_define = ['define'] 76 | __macrofuncs = {} # functional macros like: "GLIBVERSION(2,3,4)", 77 | # used as "GLIBVERSION(x,y,z) 100*x+10*y+z" 78 | __curfile = '' # current processed xml-file 79 | __defset = set() # macro-objects 80 | __defsetf = dict() # macro-objects per file 81 | 82 | # collected statistics 83 | class __statsorder(Enum): 84 | FILENAME = 0 # name of the file 85 | LINE_START = 1 # starting line of an #ifdef block 86 | LINE_END = 2 # ending line of an #ifdef block (ends either at #else, 87 | # #elif, or #endif on same level) 88 | TYPE = 3 # either #if, #elif, or #else 89 | EXPRESSION = 4 # the presence condition stated in the #ifdef 90 | CONSTANTS = 5 # all configuration constants used in the presence condition 91 | 92 | 93 | ################################################## 94 | # class FeatureLocation 95 | 96 | 97 | class FeatureLocation: 98 | ''' A feature location consists of a filename, a start and end line, 99 | the type of #ifdef used (#if, #else, #elif), the presence condition 100 | as well as all used configuration constants.''' 101 | 102 | def __init__(self, filename, startline, endline, type, expression): 103 | global _cppnscpp # scrml namespace tag 104 | namespace = '{' + _cppnscpp + '}' 105 | typeWithoutNamespace = '#' + type.replace(namespace, "") 106 | 107 | self.filename = filename # TODO use relative paths here! 108 | self.startline = startline 109 | self.endline = endline 110 | self.type = typeWithoutNamespace 111 | self.expression = expression 112 | self.constants = set() 113 | 114 | def __str__(self): 115 | constants = ";".join(sorted(self.constants)) 116 | outList = (self.filename, self.startline, self.endline, self.type, 117 | self.expression, constants) 118 | return ",".join(map(str, outList)) 119 | 120 | def __repr__(self): 121 | return str(self) 122 | 123 | def __hash__(self): 124 | return hash(self.__repr__()) 125 | 126 | def __eq__(self, other): 127 | if isinstance(other, FeatureLocation): 128 | return self.__hash__() == other.__hash__() 129 | else: 130 | return False 131 | 132 | def __ne__(self, other): 133 | return (not self.__eq__(other)) 134 | 135 | def getCSVList(self): 136 | returnList = [] 137 | 138 | returnList.append(self.filename) 139 | returnList.append(self.startline) 140 | returnList.append(self.endline) 141 | returnList.append(self.type) 142 | returnList.append(self.expression) 143 | 144 | constants = ";".join(sorted(self.constants)) 145 | returnList.append(constants) 146 | 147 | return returnList 148 | 149 | 150 | ################################################## 151 | # helper functions, constants and errors 152 | 153 | 154 | def returnFileNames(folder, extfilt=['.xml']): 155 | '''This function returns all files of the input folder 156 | and its subfolders.''' 157 | filesfound = list() 158 | 159 | if os.path.isdir(folder): 160 | wqueue = [os.path.abspath(folder)] 161 | 162 | while wqueue: 163 | currentfolder = wqueue[0] 164 | wqueue = wqueue[1:] 165 | foldercontent = os.listdir(currentfolder) 166 | tmpfiles = filter(lambda n: os.path.isfile( 167 | os.path.join(currentfolder, n)), foldercontent) 168 | tmpfiles = filter(lambda n: os.path.splitext(n)[1] in extfilt, 169 | tmpfiles) 170 | tmpfiles = map(lambda n: os.path.join(currentfolder, n), 171 | tmpfiles) 172 | filesfound += tmpfiles 173 | tmpfolders = filter(lambda n: os.path.isdir( 174 | os.path.join(currentfolder, n)), foldercontent) 175 | tmpfolders = map(lambda n: os.path.join(currentfolder, n), 176 | tmpfolders) 177 | wqueue += tmpfolders 178 | 179 | return filesfound 180 | 181 | 182 | ################################################## 183 | # parsing methods 184 | 185 | 186 | def _collectDefines(d): 187 | """This functions adds all defines to a set. 188 | e.g. #define FEAT_WIN 189 | also #define FEAT_WIN 12 190 | but not #define GLIBCVER(x,y,z) ... 191 | """ 192 | __defset.add(d[0]) 193 | if __defsetf.has_key(__curfile): 194 | __defsetf[__curfile].add(d[0]) 195 | else: 196 | __defsetf[__curfile] = set([d[0]]) 197 | return d 198 | 199 | 200 | # possible operands: 201 | # - hexadecimal number 202 | # - decimal number 203 | # - identifier 204 | # - macro function, which is basically expanded via #define 205 | # to an expression 206 | __numlitl = pypa.Literal('l').suppress() | pypa.Literal('L').suppress() 207 | __numlitu = pypa.Literal('u').suppress() | pypa.Literal('U').suppress() 208 | 209 | __string = pypa.QuotedString('\'', '\\') 210 | 211 | __hexadec = \ 212 | pypa.Literal('0x').suppress() + \ 213 | pypa.Word(pypa.hexnums). \ 214 | setParseAction(lambda t: str(int(t[0], 16))) + \ 215 | pypa.Optional(__numlitu) + \ 216 | pypa.Optional(__numlitl) + \ 217 | pypa.Optional(__numlitl) 218 | 219 | __integer = \ 220 | pypa.Optional('~') + \ 221 | pypa.Word(pypa.nums + '-').setParseAction(lambda t: str(int(t[0]))) + \ 222 | pypa.Optional(pypa.Suppress(pypa.Literal('U'))) + \ 223 | pypa.Optional(pypa.Suppress(pypa.Literal('L'))) + \ 224 | pypa.Optional(pypa.Suppress(pypa.Literal('L'))) 225 | 226 | __identifier = \ 227 | pypa.Word(pypa.alphanums + '_' + '-' + '@' + '$').setParseAction(_collectDefines) 228 | __arg = pypa.Word(pypa.alphanums + '_') 229 | __args = __arg + pypa.ZeroOrMore(pypa.Literal(',').suppress() + \ 230 | __arg) 231 | __fname = pypa.Word(pypa.alphas, pypa.alphanums + '_') 232 | __function = pypa.Group(__fname + pypa.Literal('(').suppress() + \ 233 | __args + pypa.Literal(')').suppress()) 234 | 235 | 236 | class NoEquivalentSigError(Exception): 237 | def __init__(self): 238 | pass 239 | 240 | def __str__(self): 241 | return ("No equivalent signature found!") 242 | 243 | 244 | class IfdefEndifMismatchError(Exception): 245 | def __str__(self): 246 | return ("Ifdef and endif do not match!") 247 | 248 | 249 | def _parseFeatureSignatureAndRewrite(sig): 250 | """This function parses a given feature-signature and rewrites 251 | the signature according to the given __pt mapping. 252 | """ 253 | # this dictionary holds all transformations of operators from 254 | # the origin (cpp) to the compare (language) 255 | # e.g. in cpp && stands for the 'and'-operator. 256 | # the equivalent in maple (which is used for comparison) 257 | # is '&and' 258 | # if no equivalence can be found a name rewriting is done 259 | # e.g. 'defined' 260 | __pt = { 261 | #'defined' : 'defined_', 262 | 'defined': '', 263 | '!': '¬', 264 | '&&': '&and', 265 | '||': '&or', 266 | '<': '<', 267 | '>': '>', 268 | '<=': '<=', 269 | '>=': '>=', 270 | '==': '=', 271 | '!=': '!=', 272 | '*': '*', # needs rewriting with parenthesis 273 | '/': '/', 274 | '%': '', # needs rewriting a % b => modp(a, b) 275 | '+': '+', 276 | '-': '-', 277 | '&': '', # needs rewriting a & b => BitAnd(a, b) 278 | '|': '', # needs rewriting a | b => BitOr(a, b) 279 | '>>': '>>', # needs rewriting a >> b => a / (2^b) 280 | '<<': '<<', # needs rewriting a << b => a * (2^b) 281 | } 282 | 283 | def _rewriteOne(param): 284 | """This function returns each one parameter function 285 | representation for maple.""" 286 | if param[0][0] == '!': 287 | ret = __pt[param[0][0]] + '(' + str(param[0][1]) + ')' 288 | if param[0][0] == 'defined': 289 | ret = __pt[param[0][0]] + str(param[0][1]) 290 | return ret 291 | 292 | 293 | def _rewriteTwo(param): 294 | """This function returns each two parameter function 295 | representation for maple.""" 296 | # rewriting rules 297 | if param[0][1] == '%': 298 | return 'modp(' + param[0][0] + ',' + param[0][2] + ')' 299 | 300 | ret = ' ' + __pt[param[0][1]] + ' ' 301 | ret = '(' + ret.join(map(str, param[0][0::2])) + ')' 302 | 303 | if param[0][1] in ['<', '>', '<=', '>=', '!=', '==']: 304 | ret = '(true &and ' + ret + ')' 305 | return ret 306 | 307 | operand = __string | __hexadec | __integer | \ 308 | __function | __identifier 309 | compoperator = pypa.oneOf('< > <= >= == !=') 310 | calcoperator = pypa.oneOf('+ - * / & | << >> %') 311 | expr = pypa.operatorPrecedence(operand, [ 312 | ('defined', 1, pypa.opAssoc.RIGHT, _rewriteOne), 313 | ('!', 1, pypa.opAssoc.RIGHT, _rewriteOne), 314 | (calcoperator, 2, pypa.opAssoc.LEFT, _rewriteTwo), 315 | (compoperator, 2, pypa.opAssoc.LEFT, _rewriteTwo), 316 | ('&&', 2, pypa.opAssoc.LEFT, _rewriteTwo), 317 | ('||', 2, pypa.opAssoc.LEFT, _rewriteTwo), 318 | ]) 319 | 320 | try: 321 | rsig = expr.parseString(sig)[0] 322 | except pypa.ParseException, e: 323 | print('ERROR (parse): cannot parse sig (%s) -- (%s)' % 324 | (sig, e.col)) 325 | return sig 326 | except RuntimeError: 327 | print('ERROR (time): cannot parse sig (%s)' % (sig)) 328 | return sig 329 | except ValueError, e: 330 | print('ERROR (parse): cannot parse sig (%s) ~~ (%s)' % 331 | (sig, e)) 332 | return sig 333 | return ''.join(rsig) 334 | 335 | 336 | def _collapseSubElementsToList(node): 337 | """This function collapses all subelements of the given element 338 | into a list used for getting the signature out of an #ifdef-node.""" 339 | # get all descendants - recursive - children, children of children ... 340 | itdesc = node.itertext() 341 | 342 | # iterate over the elemtents and add them to a list 343 | return ''.join([it for it in itdesc]) 344 | 345 | 346 | def _parseAndAddDefine(node): 347 | """This function extracts the identifier and the corresponding 348 | expansion from define macros. Later on these are used in conditionals 349 | in order to make them comparable.""" 350 | 351 | define = _collapseSubElementsToList(node) 352 | 353 | # match only macro functions, no macro objects 354 | anytext = pypa.Word(pypa.printables) 355 | macrodef = pypa.Literal('#define').suppress() + __function + anytext 356 | 357 | try: 358 | res = macrodef.parseString(define) 359 | except pypa.ParseException: 360 | return 361 | 362 | iden = ''.join(map(str, res[0])) 363 | expn = res[-1] 364 | para = res[1:-1] 365 | __macrofuncs[iden] = (para, expn) 366 | 367 | 368 | ################################################## 369 | # #ifdef-related functions 370 | 371 | 372 | def _getMacroSignature(ifdefnode): 373 | """This function gets the signature of an ifdef or corresponding macro 374 | out of the xml-element and its descendants. Since the macros are held 375 | inside the xml-representation in an own namespace, all descendants 376 | and their text corresponds to the macro-signature. 377 | """ 378 | # get either way the expr-tag for if and elif 379 | # or the name-tag for ifdef and ifndef, 380 | # which are both the starting point for signature 381 | # see the srcml.dtd for more information 382 | nexpr = [] 383 | res = '' 384 | _, tag = __cpprens.match(ifdefnode.tag).groups() 385 | 386 | # get either the expr or the name tag, 387 | # which is always the second descendant 388 | if (tag in ['if', 'elif', 'ifdef', 'ifndef']): 389 | nexpr = [itex for itex in ifdefnode.iterdescendants()] 390 | if (len(nexpr) == 1): 391 | res = nexpr[0].tail 392 | else: 393 | nexpr = nexpr[1] 394 | res = ''.join([token for token in nexpr.itertext()]) 395 | return res 396 | 397 | 398 | def _getFeatureSignature(condinhist): 399 | """This method returns a feature signature that belongs to the 400 | current history of conditional inclusions held in condinhist.""" 401 | # we need to rewrite the elements before joining them to one 402 | # signature; reason is elements like else or elif, which mean 403 | # basically invert the fname found before 404 | # rewritelist = [(tag, fname, )] 405 | rewritelist = [None] * len(condinhist) 406 | cur = -1 407 | 408 | for tag, fname in condinhist: 409 | cur += 1 410 | if tag == 'if': 411 | rewritelist[cur] = (tag, fname, False) 412 | if tag in ['elif', 'else']: 413 | (t, f, _) = rewritelist[cur - 1] 414 | rewritelist[cur - 1] = (t, f, True) 415 | rewritelist[cur] = (tag, fname, False) 416 | 417 | fsig = '' 418 | 419 | for (tag, fname, invert) in rewritelist: 420 | if invert: 421 | fname = '!(' + fname + ')' 422 | if fsig == '': 423 | fsig = fname 424 | continue 425 | if tag == 'else': 426 | continue 427 | if tag in ['if', 'elif']: 428 | fsig = '(' + fsig + ') && (' + fname + ')' 429 | continue 430 | return fsig 431 | 432 | 433 | def _getFeatures(root, featlocations): 434 | """This function returns all features in the source-file. 435 | A feature is defined as an enframement of soure-code. The frame 436 | consists of an ifdef (conditional) and an endif-macro. The function 437 | returns a tuple with the following format: 438 | ({: (, [])}, 439 | {: []}, 440 | [(, (, ))]) 441 | 442 | feature elements: Every feature element reflects one part of a 443 | feature withing the whole source-code, that is framed by contional 444 | and endif-macros. 445 | 446 | featuresgrinner: All tags from the feature elements (see above). 447 | featuresgrouter: All tags from the elements arround the feature. 448 | """ 449 | 450 | def _wrapGrOuterUp(fouter, featuresgrouter, eelem): 451 | itouter = fouter[-1] # feature surround tags 452 | fouter = fouter[:-1] 453 | 454 | for i in xrange(0, len(itouter)): 455 | sig = itouter[i][0] 456 | elem = itouter[i][1] 457 | 458 | # start of next location is the end of the current one, 459 | # or else the #endif in eelem 460 | end = itouter[i + 1][1] if (i < len(itouter) - 1) else eelem 461 | 462 | featuresgrouter.append((sig, elem, end)) 463 | return (fouter, featuresgrouter) 464 | 465 | 466 | def _wrapFeatureUp(features, featuresgrinner, fcode, flist, finner): 467 | # wrap up the feature 468 | if (not flist): 469 | raise IfdefEndifMismatchError() 470 | itsig = flist[-1] # feature signature 471 | flist = flist[:-1] 472 | itcode = fcode[-1] # feature code 473 | itcode = itcode.replace('\n\n', '\n') 474 | itcode = itcode[1:] # itcode starts with '\n'; del 475 | fcode = fcode[:-1] 476 | itinner = finner[-1] # feature enclosed tags 477 | finner = finner[:-1] 478 | 479 | # handle the feature code 480 | if (features.has_key(itsig)): 481 | features[itsig][1].append(itcode) 482 | else: 483 | features[itsig] = (len(flist) + 1, [itcode]) 484 | 485 | # handle the inner granularity 486 | featuresgrinner.append((itsig, itinner)) 487 | return (features, featuresgrinner, fcode, flist, finner) 488 | 489 | features = {} # see above; return value 490 | featuresgrinner = [] # see above; return value 491 | featuresgrouter = [] # see above; return value 492 | flist = [] # holds the features in order 493 | # list empty -> no features to parse 494 | # list used as a stack 495 | # last element = top of stack; 496 | # and the element we currently 497 | # collecting source-code lines for 498 | fouter = [] # holds the xml-nodes of the ifdefs/endifs 499 | # in order like flist 500 | fcode = [] # holds the code of the features in 501 | # order like flist 502 | finner = [] # holds the tags of the features in 503 | # order like flist 504 | condinhist = [] # order of the conditional includes 505 | # with feature names 506 | parcon = False # parse-conditional-flag 507 | parend = False # parse-endif-flag 508 | _ = 0 # else and elif depth 509 | 510 | # iterate over all tags separately - and -tag 511 | for event, elem in etree.iterwalk(root, events=("start", "end")): 512 | ns, tag = __cpprens.match(elem.tag).groups() 513 | 514 | # handling conditionals 515 | # hitting on conditional-macro 516 | if ((tag in __conditionals_all) 517 | and (event == 'start') 518 | and (ns == _cppnscpp)): # check the cpp:namespace 519 | parcon = True 520 | 521 | # hitting next conditional macro; any of ifdef, else or elif 522 | if ((tag in __conditionals_all) 523 | and (event == 'end') 524 | and (ns == _cppnscpp)): # check the cpp:namespace 525 | parcon = False 526 | 527 | # with else or elif we finish up the last if, therefor 528 | # we can applicate the wrapup 529 | if ((tag in __conditionals_else) 530 | or (tag in __conditionals_elif)): 531 | (features, featuresgrinner, 532 | fcode, flist, finner) = _wrapFeatureUp(features, 533 | featuresgrinner, fcode, flist, finner) 534 | 535 | fname = _getMacroSignature(elem) 536 | if fname: 537 | condinhist.append((tag, fname)) 538 | else: 539 | condinhist.append((tag, '')) 540 | 541 | fsig = _getFeatureSignature(condinhist) 542 | if (tag in __conditionals): fouter.append([]) 543 | fouter[-1] += ([(fsig, elem)]) 544 | flist.append(fsig) 545 | fcode.append('') 546 | finner.append([]) 547 | 548 | # hitting end-tag of elif-macro 549 | if ((tag in __conditionals_elif) 550 | and (event == 'end') 551 | and (ns == _cppnscpp)): 552 | parcon = False 553 | 554 | # hitting end-tag of define-macro 555 | if ((tag in __macro_define) \ 556 | and (event == 'end') \ 557 | and (ns == _cppnscpp)): 558 | _parseAndAddDefine(elem) 559 | 560 | # iterateting in subtree of conditional-node 561 | if parcon: 562 | continue 563 | 564 | # handling endif-macro 565 | # hitting an endif-macro start-tag 566 | if ((tag in __conditionals_endif) \ 567 | and (event == "start") \ 568 | and (ns == _cppnscpp)): # check the cpp:namespace 569 | parend = True 570 | 571 | # hitting the endif-macro end-tag 572 | if ((tag in __conditionals_endif) \ 573 | and (event == "end") \ 574 | and (ns == _cppnscpp)): # check the cpp:namespace 575 | parend = False 576 | 577 | (features, featuresgrinner, fcode, flist, finner) = \ 578 | _wrapFeatureUp(features, featuresgrinner, 579 | fcode, flist, finner) 580 | (fouter, featuresgrouter) = _wrapGrOuterUp(fouter, 581 | featuresgrouter, elem) 582 | 583 | # transform feature locations and append them to global list 584 | for (asig, aselem, aeelem) in featuresgrouter: 585 | floc = FeatureLocation(__curfile, aselem.sourceline - 1, aeelem.sourceline - 1, 586 | aselem.tag, asig) 587 | featlocations.add(floc) 588 | 589 | while (condinhist[-1][0] != 'if'): 590 | condinhist = condinhist[:-1] 591 | condinhist = condinhist[:-1] 592 | 593 | # iterating the endif-node subtree 594 | if parend: 595 | continue 596 | 597 | # collect the source-code of the feature 598 | if (len(flist)): 599 | if ((event == "start") and (elem.text)): 600 | fcode[-1] += elem.text 601 | if ((event == "end") and (elem.tail)): 602 | fcode[-1] += elem.tail 603 | 604 | if (ns == __cppnsdef or tag not in __conditionals_all): 605 | finner[-1].append((tag, event, elem.sourceline)) 606 | 607 | if (flist): 608 | raise IfdefEndifMismatchError() 609 | 610 | return (features, featuresgrinner, featuresgrouter) 611 | 612 | 613 | def _getFeaturesAtLocations(flocations, defines): 614 | """TODO""" 615 | 616 | sigs = [x.expression for x in flocations] 617 | 618 | for d in defines: 619 | dre = re.compile(r'\b' + d + r'\b') # using word boundaries 620 | vec = map(lambda s: not dre.search(s) is None, sigs) 621 | 622 | for floc, contained in zip(flocations, vec): 623 | if (contained): 624 | floc.constants.add(d) 625 | 626 | 627 | ################################################## 628 | # output file 629 | 630 | 631 | def _prologCSV(folder, file, headings): 632 | """prolog of the CSV-output file 633 | no corresponding _epilogCSV.""" 634 | fd = open(os.path.join(folder, file), 'w') 635 | fdcsv = csv.writer(fd, delimiter=',') 636 | fdcsv.writerow(["sep=,"]) 637 | fdcsv.writerow(headings) 638 | return (fd, fdcsv) 639 | 640 | 641 | ################################################## 642 | # main method 643 | 644 | def resetModule() : 645 | global __macrofuncs, __defset, __defsetf 646 | __macrofuncs = {} # functional macros like: "GLIBVERSION(2,3,4)", 647 | # used as "GLIBVERSION(x,y,z) 100*x+10*y+z" 648 | __defset = set() # macro-objects 649 | __defsetf = dict() # macro-objects per file 650 | 651 | 652 | def apply(folder, options): 653 | """This function applies the analysis to all xml-files in that 654 | directory and take the results and joins them together. Results 655 | are getting written into the fdcsv-file.""" 656 | # overall status variables 657 | resetModule() 658 | 659 | featlocations = set() # list of feature locations of class FeatureLocation 660 | 661 | # outputfile 662 | fd, fdcsv = _prologCSV(os.path.join(folder, os.pardir), __outputfile, __statsorder.__members__.keys()) 663 | 664 | # list-of-features file 665 | loffheadings = ['FILENAME', 'CONSTANTS'] 666 | loffrow = [None]*len(loffheadings) 667 | loffhandle, loffwriter = _prologCSV(os.path.join(folder, os.pardir), __listoffeaturesfile, loffheadings) 668 | 669 | # preparations for file-loop 670 | global __curfile 671 | files = returnFileNames(folder, ['.xml']) 672 | files.sort() 673 | fcount = 0 674 | ftotal = len(files) 675 | 676 | #TODO rewrite comment! get statistics for all files; write results into csv 677 | # and merge the features 678 | for file in files: 679 | __curfile = file 680 | 681 | try: 682 | tree = etree.parse(file) 683 | except etree.XMLSyntaxError: 684 | print("ERROR: cannot parse (%s). Skipping this file." % os.path.join(folder, file)) 685 | continue 686 | 687 | root = tree.getroot() 688 | try: 689 | (features, _, _) = _getFeatures(root, featlocations) 690 | except IfdefEndifMismatchError: 691 | print("ERROR: ifdef-endif mismatch in file (%s)" % (os.path.join(folder, file))) 692 | continue 693 | 694 | # parse features and get all defined configuration constants 695 | for (sig, (depth, code)) in features.iteritems(): 696 | psig = _parseFeatureSignatureAndRewrite(sig) 697 | 698 | # file successfully parsed 699 | fcount += 1 700 | print('INFO: parsing file (%5d) of (%5d) -- (%s).' % (fcount, ftotal, os.path.join(folder, file))) 701 | 702 | # print features for this file to list-of-features file 703 | featureslist = list(__defsetf[__curfile]) \ 704 | if __defsetf.has_key(__curfile) else '' # list of features within the current file 705 | listoffeaturesstring = ';'.join(sorted(featureslist)) # sort and join 706 | loffwriter.writerow([__curfile, listoffeaturesstring]) # write row to file 707 | 708 | 709 | # collect feature locations and consisting used features 710 | featurelocations = list(featlocations) 711 | featurelocations.sort(key=lambda x: (x.filename, x.startline)) 712 | _getFeaturesAtLocations(featurelocations, list(__defset)) 713 | 714 | # print each feature location as one line into output file 715 | for floc in featurelocations: 716 | 717 | #adjust file name if wanted 718 | if options.filenamesRelative : # relative file name (root is project folder (not included in path)) 719 | floc.filename = os.path.relpath(floc.filename, folder) 720 | 721 | if options.filenames == options.FILENAME_SRCML : # cppstats file names 722 | pass # nothing to do here, as the file path is the cppstats path by default 723 | if options.filenames == options.FILENAME_SOURCE : # source file name 724 | floc.filename = floc.filename.replace(".xml", "").replace("/_cppstats/", "/source/", 1) 725 | 726 | # print floc information to CSV file 727 | row = floc.getCSVList() 728 | fdcsv.writerow(row) 729 | 730 | # close output files 731 | fd.close() # __outputfile 732 | loffhandle.close() # __listoffeaturesfile 733 | 734 | 735 | 736 | # ################################################## 737 | # add command line options 738 | 739 | def addCommandLineOptionsMain(optionparser): 740 | ''' add command line options for a direct call of this script''' 741 | optionparser.add_argument("--folder", dest="folder", 742 | help="input folder [default=.]", default=".") 743 | 744 | 745 | def addCommandLineOptions(optionparser): 746 | pass 747 | 748 | 749 | # ################################################ 750 | # path of the main output file 751 | 752 | def getResultsFile(): 753 | return __outputfile 754 | 755 | 756 | ################################################## 757 | if __name__ == '__main__': 758 | 759 | ################################################## 760 | # options parsing 761 | parser = ArgumentParser(formatter_class=RawTextHelpFormatter) 762 | addCommandLineOptionsMain(parser) 763 | addCommandLineOptions(parser) 764 | 765 | options = parser.parse_args() 766 | 767 | folder = os.path.abspath(options.folder) 768 | 769 | if (os.path.isdir(folder)): 770 | apply(folder) 771 | else: 772 | sys.exit(-1) 773 | -------------------------------------------------------------------------------- /analyses/generalvalues.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2014-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Claus Hunsen 23 | 24 | 25 | # modules from the std-library 26 | import csv 27 | import os 28 | import re 29 | import sys 30 | import xmlrpclib 31 | from argparse import ArgumentParser, RawTextHelpFormatter 32 | from collections import OrderedDict 33 | 34 | 35 | # ################################################# 36 | # path adjustments, so that all imports can be done relative to these paths 37 | 38 | __lib_subfolder = "lib" 39 | sys.path.append(os.path.abspath(__lib_subfolder)) # lib subfolder 40 | 41 | 42 | # ################################################# 43 | # external modules 44 | 45 | # python-lxml module 46 | from lxml import etree 47 | # statistics module 48 | from statlib import pstat 49 | # pyparsing module 50 | import pyparsing as pypa 51 | pypa.ParserElement.enablePackrat() # speed up parsing 52 | sys.setrecursionlimit(8000) # handle larger expressions 53 | 54 | 55 | ################################################## 56 | # config: 57 | __outputfile = "cppstats.csv" 58 | __metricvaluesfile = "metric_values.csv" 59 | 60 | # error numbers: 61 | __errorfexp = 0 62 | __errormatch = [] 63 | ################################################## 64 | 65 | 66 | ################################################## 67 | # constants: 68 | # namespace-constant for src2srcml 69 | __cppnscpp = 'http://www.srcML.org/srcML/cpp' 70 | __cppnsdef = 'http://www.srcML.org/srcML/src' 71 | __cpprens = re.compile('{(.+)}(.+)') 72 | 73 | # conditionals - necessary for parsing the right tags 74 | __conditionals = ['if', 'ifdef', 'ifndef'] 75 | __conditionals_elif = ['elif'] 76 | __conditionals_else = ['else'] 77 | __conditionals_endif = ['endif'] 78 | __conditionals_all = __conditionals + __conditionals_elif + \ 79 | __conditionals_else 80 | __conditionals_ending = __conditionals_elif + __conditionals_else + \ 81 | __conditionals_endif 82 | __macro_define = ['define'] 83 | __macrofuncs = {} # functional macros like: "GLIBVERSION(2,3,4)", 84 | # used as "GLIBVERSION(x,y,z) 100*x+10*y+z" 85 | __curfile = '' # current processed xml-file 86 | __defset = set() # macro-objects 87 | __defsetf = dict() # macro-objects per file 88 | 89 | ################################################## 90 | 91 | 92 | 93 | ################################################## 94 | # helper functions, constants and errors 95 | def returnFileNames(folder, extfilt = ['.xml']): 96 | '''This function returns all files of the input folder 97 | and its subfolders.''' 98 | filesfound = list() 99 | 100 | if os.path.isdir(folder): 101 | wqueue = [os.path.abspath(folder)] 102 | 103 | while wqueue: 104 | currentfolder = wqueue[0] 105 | wqueue = wqueue[1:] 106 | foldercontent = os.listdir(currentfolder) 107 | tmpfiles = filter(lambda n: os.path.isfile( 108 | os.path.join(currentfolder, n)), foldercontent) 109 | tmpfiles = filter(lambda n: os.path.splitext(n)[1] in extfilt, 110 | tmpfiles) 111 | tmpfiles = map(lambda n: os.path.join(currentfolder, n), 112 | tmpfiles) 113 | filesfound += tmpfiles 114 | tmpfolders = filter(lambda n: os.path.isdir( 115 | os.path.join(currentfolder, n)), foldercontent) 116 | tmpfolders = map(lambda n: os.path.join(currentfolder, n), 117 | tmpfolders) 118 | wqueue += tmpfolders 119 | 120 | return filesfound 121 | 122 | 123 | def _prologCSV(folder, file, headings, delimiter = ","): 124 | """prolog of the CSV-output file 125 | no corresponding _epilogCSV.""" 126 | fd = open(os.path.join(folder, file), 'w') 127 | fdcsv = csv.writer(fd, delimiter=delimiter) 128 | fdcsv.writerow(["sep=" + delimiter]) 129 | fdcsv.writerow(headings) 130 | return (fd, fdcsv) 131 | 132 | 133 | def _flatten(l): 134 | """This function takes a list as input and returns a flatten version 135 | of the list. So all nested lists are unpacked and moved up to the 136 | level of the list.""" 137 | i = 0 138 | while i < len(l): 139 | while isinstance(l[i], list): 140 | if not l[i]: 141 | l.pop(i) 142 | i -= 1 143 | break 144 | else: 145 | l[i:i+1] = l[i] 146 | i += 1 147 | return l 148 | 149 | 150 | def _collectDefines(d): 151 | """This functions adds all defines to a set. 152 | e.g. #define FEAT_WIN 153 | also #define FEAT_WIN 12 154 | but not #define GLIBCVER(x,y,z) ... 155 | """ 156 | __defset.add(d[0]) 157 | if __defsetf.has_key(__curfile): 158 | __defsetf[__curfile].add(d[0]) 159 | else: 160 | __defsetf[__curfile] = set([d[0]]) 161 | return d 162 | 163 | 164 | # possible operands: 165 | # - hexadecimal number 166 | # - decimal number 167 | # - identifier 168 | # - macro function, which is basically expanded via #define 169 | # to an expression 170 | __numlitl = pypa.Literal('l').suppress() | pypa.Literal('L').suppress() 171 | __numlitu = pypa.Literal('u').suppress() | pypa.Literal('U').suppress() 172 | 173 | __string = pypa.QuotedString('\'', '\\') 174 | 175 | __hexadec = \ 176 | pypa.Literal('0x').suppress() + \ 177 | pypa.Word(pypa.hexnums).\ 178 | setParseAction(lambda t: str(int(t[0], 16))) + \ 179 | pypa.Optional(__numlitu) + \ 180 | pypa.Optional(__numlitl) + \ 181 | pypa.Optional(__numlitl) 182 | 183 | __integer = \ 184 | pypa.Optional('~') + \ 185 | pypa.Word(pypa.nums+'-').setParseAction(lambda t: str(int(t[0]))) + \ 186 | pypa.Optional(pypa.Suppress(pypa.Literal('U'))) + \ 187 | pypa.Optional(pypa.Suppress(pypa.Literal('L'))) + \ 188 | pypa.Optional(pypa.Suppress(pypa.Literal('L'))) 189 | 190 | __identifier = \ 191 | pypa.Word(pypa.alphanums+'_'+'-'+'@'+'$').setParseAction(_collectDefines) 192 | __arg = pypa.Word(pypa.alphanums+'_') 193 | __args = __arg + pypa.ZeroOrMore(pypa.Literal(',').suppress() + \ 194 | __arg) 195 | __fname = pypa.Word(pypa.alphas, pypa.alphanums + '_') 196 | __function = pypa.Group(__fname + pypa.Literal('(').suppress() + \ 197 | __args + pypa.Literal(')').suppress()) 198 | 199 | 200 | class NoEquivalentSigError(Exception): 201 | def __init__(self): 202 | pass 203 | def __str__(self): 204 | return ("No equivalent signature found!") 205 | 206 | class IfdefEndifMismatchError(Exception): 207 | def __init__(self): 208 | pass 209 | def __str__(self): 210 | return ("Ifdef and endif do not match!") 211 | 212 | ################################################## 213 | 214 | 215 | def _collapseSubElementsToList(node): 216 | """This function collapses all subelements of the given element 217 | into a list used for getting the signature out of an #ifdef-node.""" 218 | # get all descendants - recursive - children, children of children ... 219 | itdesc = node.itertext() 220 | 221 | # iterate over the elemtents and add them to a list 222 | return ''.join([it for it in itdesc]) 223 | 224 | 225 | def _parseFeatureSignatureAndRewrite(sig): 226 | """This function parses a given feature-signature and rewrites 227 | the signature according to the given __pt mapping. 228 | """ 229 | # this dictionary holds all transformations of operators from 230 | # the origin (cpp) to the compare (language) 231 | # e.g. in cpp && stands for the 'and'-operator. 232 | # the equivalent in maple (which is used for comparison) 233 | # is '&and' 234 | # if no equivalence can be found a name rewriting is done 235 | # e.g. 'defined' 236 | __pt = { 237 | #'defined' : 'defined_', 238 | 'defined' : '', 239 | '!' : '¬', 240 | '&&': '&and', 241 | '||': '&or', 242 | '<' : '<', 243 | '>' : '>', 244 | '<=': '<=', 245 | '>=': '>=', 246 | '==': '=', 247 | '!=': '!=', 248 | '*' : '*', # needs rewriting with parenthesis 249 | '/' : '/', 250 | '%' : '', # needs rewriting a % b => modp(a, b) 251 | '+' : '+', 252 | '-' : '-', 253 | '&' : '', # needs rewriting a & b => BitAnd(a, b) 254 | '|' : '', # needs rewriting a | b => BitOr(a, b) 255 | '>>': '>>', # needs rewriting a >> b => a / (2^b) 256 | '<<': '<<', # needs rewriting a << b => a * (2^b) 257 | } 258 | 259 | def _rewriteOne(param): 260 | """This function returns each one parameter function 261 | representation for maple.""" 262 | if param[0][0] == '!': 263 | ret = __pt[param[0][0]] + '(' + str(param[0][1]) + ')' 264 | if param[0][0] == 'defined': 265 | ret = __pt[param[0][0]] + str(param[0][1]) 266 | return ret 267 | 268 | 269 | def _rewriteTwo(param): 270 | """This function returns each two parameter function 271 | representation for maple.""" 272 | # rewriting rules 273 | if param[0][1] == '%': 274 | return 'modp(' + param[0][0] + ',' + param[0][2] + ')' 275 | 276 | ret = ' ' + __pt[param[0][1]] + ' ' 277 | ret = '(' + ret.join(map(str, param[0][0::2])) + ')' 278 | 279 | if param[0][1] in ['<', '>', '<=', '>=', '!=', '==']: 280 | ret = '(true &and ' + ret + ')' 281 | return ret 282 | 283 | operand = __string | __hexadec | __integer | \ 284 | __function | __identifier 285 | compoperator = pypa.oneOf('< > <= >= == !=') 286 | calcoperator = pypa.oneOf('+ - * / & | << >> %') 287 | expr = pypa.operatorPrecedence(operand, [ 288 | ('defined', 1, pypa.opAssoc.RIGHT, _rewriteOne), 289 | ('!', 1, pypa.opAssoc.RIGHT, _rewriteOne), 290 | (calcoperator, 2, pypa.opAssoc.LEFT, _rewriteTwo), 291 | (compoperator, 2, pypa.opAssoc.LEFT, _rewriteTwo), 292 | ('&&', 2, pypa.opAssoc.LEFT, _rewriteTwo), 293 | ('||', 2, pypa.opAssoc.LEFT, _rewriteTwo), 294 | ]) 295 | 296 | try: 297 | rsig = expr.parseString(sig)[0] 298 | except pypa.ParseException, e: 299 | print('ERROR (parse): cannot parse sig (%s) -- (%s)' % 300 | (sig, e.col)) 301 | return sig 302 | except RuntimeError: 303 | print('ERROR (time): cannot parse sig (%s)' % (sig)) 304 | return sig 305 | except ValueError, e: 306 | print('ERROR (parse): cannot parse sig (%s) ~~ (%s)' % 307 | (sig, e)) 308 | return sig 309 | return ''.join(rsig) 310 | 311 | 312 | def _getMacroSignature(ifdefnode): 313 | """This function gets the signature of an ifdef or corresponding macro 314 | out of the xml-element and its descendants. Since the macros are held 315 | inside the xml-representation in an own namespace, all descendants 316 | and their text corresponds to the macro-signature. 317 | """ 318 | # get either way the expr-tag for if and elif 319 | # or the name-tag for ifdef and ifndef, 320 | # which are both the starting point for signature 321 | # see the srcml.dtd for more information 322 | nexpr = [] 323 | res = '' 324 | _, tag = __cpprens.match(ifdefnode.tag).groups() 325 | 326 | # get either the expr or the name tag, 327 | # which is always the second descendant 328 | if (tag in ['if', 'elif', 'ifdef', 'ifndef']): 329 | nexpr = [itex for itex in ifdefnode.iterdescendants()] 330 | if (len(nexpr) == 1): 331 | res = nexpr[0].tail 332 | else: 333 | nexpr = nexpr[1] 334 | res = ''.join([token for token in nexpr.itertext()]) 335 | return res 336 | 337 | 338 | _elsePrefix = "###" 339 | def _getFeatureSignature(condinhist, options): 340 | """This method returns a feature signature that belongs to the 341 | current history of conditional inclusions held in condinhist.""" 342 | # we need to rewrite the elements before joining them to one 343 | # signature; reason is elements like else or elif, which mean 344 | # basically invert the fname found before 345 | # rewritelist = [(tag, fname, )] 346 | rewritelist = [None]*len(condinhist) 347 | cur = -1 348 | 349 | for tag, fname in condinhist: 350 | cur += 1 351 | if tag == 'if': 352 | rewritelist[cur] = (tag, fname, False) 353 | if tag in ['elif', 'else']: 354 | (t, f, _) = rewritelist[cur-1] 355 | rewritelist[cur-1] = (t, f, True) 356 | rewritelist[cur] = (tag, fname, False) 357 | 358 | fsig = '' 359 | 360 | for (tag, fname, invert) in rewritelist: 361 | if invert: 362 | if (options.rewriteifdefs): 363 | fname = '!(' + fname + ')' 364 | else: 365 | fname = _elsePrefix + '!(' + fname + ')' 366 | 367 | if fsig == '': 368 | fsig = fname 369 | continue 370 | if tag == 'else': 371 | continue 372 | if tag in [ 'if', 'elif']: 373 | if (options.rewriteifdefs): 374 | fsig = '(' + fsig + ') && (' + fname + ')' 375 | else: 376 | fsig = fname 377 | continue 378 | return fsig 379 | 380 | 381 | def _parseAndAddDefine(node): 382 | """This function extracts the identifier and the corresponding 383 | expansion from define macros. Later on these are used in conditionals 384 | in order to make them comparable.""" 385 | define = _collapseSubElementsToList(node) 386 | 387 | # match only macro functions, no macro objects 388 | anytext = pypa.Word(pypa.printables) 389 | macrodef = pypa.Literal('#define').suppress() + __function + anytext 390 | 391 | try: 392 | res = macrodef.parseString(define) 393 | except pypa.ParseException: 394 | return 395 | 396 | iden = ''.join(map(str, res[0])) 397 | expn = res[-1] 398 | para = res[1:-1] 399 | __macrofuncs[iden] = (para, expn) 400 | 401 | 402 | def _getFeatures(root, options): 403 | """This function returns all features in the source-file. 404 | A feature is defined as an enframement of soure-code. The frame 405 | consists of an ifdef (conditional) and an endif-macro. The function 406 | returns a tuple with the following format: 407 | ({: (, [])}, 408 | {: []}, 409 | [(, (, ))]) 410 | 411 | feature elements: Every feature element reflects one part of a 412 | feature withing the whole source-code, that is framed by contional 413 | and endif-macros. 414 | 415 | featuresgrinner: All tags from the feature elements (see above). 416 | featuresgrouter: All tags from the elements arround the feature. 417 | """ 418 | 419 | def _wrapGrOuterUp(fouter, featuresgrouter, eelem): 420 | itouter = fouter[-1] # feature surround tags 421 | fouter = fouter[:-1] 422 | 423 | selem = itouter[0][1] 424 | for (sig, _) in itouter: 425 | featuresgrouter.append((sig, selem, eelem)) 426 | return (fouter, featuresgrouter) 427 | 428 | 429 | def _wrapFeatureUp(features, featuresgrinner, fcode, flist, finner): 430 | # wrap up the feature 431 | if (not flist): 432 | raise IfdefEndifMismatchError() 433 | itsig = flist[-1] # feature signature 434 | flist = flist[:-1] 435 | itcode = fcode[-1] # feature code 436 | itcode = itcode.replace('\n\n', '\n') 437 | itcode = itcode[1:] # itcode starts with '\n'; del 438 | fcode = fcode[:-1] 439 | itinner = finner[-1] # feature enclosed tags 440 | finner = finner[:-1] 441 | 442 | # handle the feature code 443 | if (features.has_key(itsig)): 444 | features[itsig][1].append(itcode) 445 | else: 446 | features[itsig] = (len(flist)+1, [itcode]) 447 | 448 | # handle the inner granularity 449 | featuresgrinner.append((itsig, itinner)) 450 | return (features, featuresgrinner, fcode, flist, finner) 451 | 452 | from collections import OrderedDict 453 | features = OrderedDict({}) # see above; return value 454 | featuresgrinner = [] # see above; return value 455 | featuresgrouter = [] # see above; return value 456 | flist = [] # holds the features in order 457 | # list empty -> no features to parse 458 | # list used as a stack 459 | # last element = top of stack; 460 | # and the element we currently 461 | # collecting source-code lines for 462 | fouter = [] # holds the xml-nodes of the ifdefs/endifs 463 | # in order like flist 464 | fcode = [] # holds the code of the features in 465 | # order like flist 466 | finner = [] # holds the tags of the features in 467 | # order like flist 468 | condinhist = [] # order of the conditional includes 469 | # with feature names 470 | parcon = False # parse-conditional-flag 471 | parend = False # parse-endif-flag 472 | _ = 0 # else and elif depth 473 | elses = [] 474 | ifdef_number = 0 475 | 476 | # iterate over all tags separately - and -tag 477 | for event, elem in etree.iterwalk(root, events=("start", "end")): 478 | ns, tag = __cpprens.match(elem.tag).groups() 479 | 480 | # handling conditionals 481 | # hitting on conditional-macro 482 | if ((tag in __conditionals_all) 483 | and (event == 'start') 484 | and (ns == __cppnscpp)): # check the cpp:namespace 485 | parcon = True 486 | 487 | # hitting on conditional-macro else or elif 488 | if (((tag in __conditionals_else) or (tag in __conditionals_elif)) 489 | and (event == 'start') 490 | and (ns == __cppnscpp)): # check the cpp:namespace 491 | ifdef_number += 1 492 | 493 | 494 | # hitting next conditional macro; any of ifdef, else or elif 495 | if ((tag in __conditionals_all) 496 | and (event == 'end') 497 | and (ns == __cppnscpp)): # check the cpp:namespace 498 | parcon = False 499 | 500 | # with else or elif we finish up the last if, therefor 501 | # we can applicate the wrapup 502 | if ((tag in __conditionals_else) 503 | or (tag in __conditionals_elif)): 504 | (features, featuresgrinner, 505 | fcode, flist, finner) = _wrapFeatureUp(features, 506 | featuresgrinner, fcode, flist, finner) 507 | 508 | fname = _getMacroSignature(elem) 509 | if fname: condinhist.append((tag, fname)) 510 | else: condinhist.append((tag, '')) 511 | 512 | fsig = _getFeatureSignature(condinhist, options) 513 | if (tag in __conditionals): fouter.append([]) 514 | fouter[-1] += ([(fsig, elem)]) 515 | flist.append(fsig) 516 | fcode.append('') 517 | finner.append([]) 518 | 519 | # hitting end-tag of elif-macro 520 | if ((tag in __conditionals_elif) 521 | and (event == 'end') 522 | and (ns == __cppnscpp)): 523 | parcon = False 524 | 525 | # hitting end-tag of define-macro 526 | if ((tag in __macro_define) \ 527 | and (event == 'end') \ 528 | and (ns == __cppnscpp)): 529 | _parseAndAddDefine(elem) 530 | 531 | # iterateting in subtree of conditional-node 532 | if parcon: 533 | continue 534 | 535 | # handling endif-macro 536 | # hitting an endif-macro start-tag 537 | if ((tag in __conditionals_endif) \ 538 | and (event == "start") \ 539 | and (ns == __cppnscpp)): # check the cpp:namespace 540 | parend = True 541 | 542 | # hitting the endif-macro end-tag 543 | if ((tag in __conditionals_endif) \ 544 | and (event == "end") \ 545 | and (ns == __cppnscpp)): # check the cpp:namespace 546 | parend = False 547 | 548 | (features, featuresgrinner, fcode, flist, finner) = \ 549 | _wrapFeatureUp(features, featuresgrinner, 550 | fcode, flist, finner) 551 | (fouter, featuresgrouter) = _wrapGrOuterUp(fouter, 552 | featuresgrouter, elem) 553 | 554 | while (condinhist[-1][0] != 'if'): 555 | if condinhist[-1][0] == 'else': 556 | elses.append(ifdef_number) 557 | condinhist = condinhist[:-1] 558 | 559 | condinhist = condinhist[:-1] 560 | ifdef_number += 1 561 | 562 | # iterating the endif-node subtree 563 | if parend: 564 | continue 565 | 566 | # collect the source-code of the feature 567 | if (len(flist)): 568 | if ((event == "start") and (elem.text)): 569 | fcode[-1] += elem.text 570 | if ((event == "end") and (elem.tail)): 571 | fcode[-1] += elem.tail 572 | 573 | if (ns == __cppnsdef or tag not in __conditionals_all): 574 | finner[-1].append((tag, event, elem.sourceline)) 575 | 576 | if (flist): 577 | raise IfdefEndifMismatchError() 578 | return (features, featuresgrinner, featuresgrouter, elses) 579 | 580 | 581 | __nestedIfdefsLevels = [] 582 | __nestingDepthsOfBranches = [] 583 | def _getNestingDepths(root): 584 | """This function counts the number of nested ifdefs (conditionals) 585 | within the source-file in two different ways. 586 | 1) the nesting depth of each #ifdef block (nested or not nested, one value for #if/#elif/#else branches) 587 | __nestedIfdefsLevels = [int] 588 | 2) the nesting depth of each top-level (non-nested) branch (#if/#elif/#else) separately 589 | __nestingDepthsOfBranches = [(file path, xml element, feature signature, maximum nesting of this element)] 590 | """ 591 | 592 | global __curfile, __nestedIfdefsLevels, __nestingDepthsOfBranches 593 | 594 | elements = [it for it in root.iterdescendants()] 595 | 596 | cncur = 0 597 | cnmax = -1 598 | sigblockhist = [] 599 | 600 | # [] of integers 601 | cnlist = [] 602 | # [(file path, xml element, feature signature, maximum nesting of this element)] 603 | sighist = [] 604 | 605 | for elem in elements: 606 | ns, tag = __cpprens.match(elem.tag).groups() 607 | 608 | # if a branch ends somehow 609 | if ((tag in __conditionals_ending) 610 | and (ns == __cppnscpp)): 611 | 612 | # reduce nesting level 613 | cncur -= 1 614 | 615 | # if we are back at top-level 616 | if cncur == 0: 617 | 618 | # insert max-nesting value to top-level element 619 | (xfile, xelem, xsig, xdepth) = sigblockhist[-1] 620 | sigblockhist[-1] = (xfile, xelem, xsig, cnmax) 621 | 622 | # reset value, since branch is finished 623 | cnmax = -1 624 | 625 | # if an #endif is reached, a whole block of #if/#elif/#else is finished 626 | if tag in __conditionals_endif: 627 | sighist += sigblockhist 628 | sigblockhist = [] 629 | 630 | # if hitting the next conditional 631 | if ((tag in __conditionals_all) 632 | and (ns == __cppnscpp)): 633 | 634 | # increase nesting level 635 | cncur += 1 636 | 637 | # gather the nesting depth of each #ifdef block (incl. #else/#elif branches) 638 | if (tag in __conditionals): 639 | cnlist.append(cncur) 640 | 641 | # add top-level signatures to history 642 | if cncur == 1: 643 | 644 | # if #else is reached, its empty signature must be rewritten as 645 | # negation of previous signatures within this #ifdef block 646 | if tag in __conditionals_else: 647 | #FIXME how to do this if rewriting is enabled?! 648 | 649 | newsig = ['!(' + xsig + ')' for (_, _, xsig, _) in sigblockhist] 650 | sigblockhist.append((__curfile, elem, " && ".join(newsig), -1)) 651 | 652 | else: 653 | 654 | sigblockhist.append((__curfile, elem, _getMacroSignature(elem), -1)) 655 | 656 | # calculate current max of this branch 657 | cnmax = max(cnmax, cncur) 658 | 659 | # # DEBUG 660 | # print "%s %s: %s (max: %s)" % (tag, _getMacroSignature(elem), cncur, cnmax) 661 | 662 | if (len(cnlist) > 0): 663 | nnitmp = filter(lambda n: n > 0, cnlist) 664 | __nestedIfdefsLevels += nnitmp 665 | 666 | __nestingDepthsOfBranches += sighist 667 | 668 | 669 | def _getScatteringTanglingValues(sigs, defines): 670 | """This method returns the scattering and tangling VALUES of 671 | defines according to the given mapping of a define to occurances 672 | in the signatures. The input is all feature-signatures and 673 | all defines.""" 674 | #TODO insert tuples into description! 675 | 676 | def __add(x, y): 677 | """This method is a helper function to add values 678 | of lists pairwise. See below for more information.""" 679 | return x+y 680 | 681 | scat = list() # relation define to signatures 682 | tang = [0]*len(sigs) # signatures overall 683 | for d in defines: 684 | dre = re.compile(r'\b'+d+r'\b') # using word boundaries 685 | vec = map(lambda s: not dre.search(s) is None, sigs) 686 | scat.append(vec.count(True)) 687 | tang = map(__add, tang, vec) 688 | 689 | # create dictionaries from sigs and defines and corresponding 690 | # scattering and tangling values 691 | scatdict = zip(defines, scat) 692 | tangdict = zip(sigs, tang) 693 | 694 | return (scatdict, tangdict) 695 | 696 | 697 | def _checkForEquivalentSig(l, sig): 698 | """This method takes a list of signatures and checks sig for an 699 | equivalent signature. If no equivalent signature is found this 700 | method raises an error.""" 701 | 702 | def _checkSigEquivalence(sig1, sig2): 703 | """This function checks the equivalence of two signatures. 704 | It uses an xmlrpc-call on troi.fim.uni-passau.de.""" 705 | global __errorfexp 706 | global __errormatch 707 | if not (sig1 and sig2): # ommit empty signatures 708 | return False 709 | 710 | # if options.str: 711 | return sig1 == sig2 712 | 713 | # _checkForEquivalentSig 714 | for it in l: 715 | if _checkSigEquivalence(it, sig): 716 | return it 717 | raise NoEquivalentSigError() 718 | 719 | 720 | def resetModule() : 721 | global __macrofuncs, __defset, __defsetf, __nestedIfdefsLevels, __nestingDepthsOfBranches 722 | __macrofuncs = {} # functional macros like: "GLIBVERSION(2,3,4)", 723 | # used as "GLIBVERSION(x,y,z) 100*x+10*y+z" 724 | __defset = set() # macro-objects 725 | __defsetf = dict() # macro-objects per file 726 | __nestedIfdefsLevels = [] 727 | __nestingDepthsOfBranches = [] 728 | 729 | 730 | def apply(folder, options): 731 | 732 | """This function applies the analysis to all xml-files in that 733 | directory and take the results and joins them together. Results 734 | are getting written into the fdcsv-file.""" 735 | # overall status variables 736 | resetModule() 737 | 738 | sigmap = {} # {: []} 739 | afeatures = {} # identified features; {: (depth, [code])} 740 | 741 | def _mergeFeatures(ffeatures): 742 | """This function merges the, with the parameter given 743 | dictionary (ffeatures) to the afeatures (overall-features).""" 744 | for (sig, (depth, code)) in ffeatures.iteritems(): 745 | psig = _parseFeatureSignatureAndRewrite(sig) 746 | 747 | try: 748 | sigmatch = _checkForEquivalentSig(sigmap.keys(), psig) 749 | (tmpdepth, tmpcode) = afeatures[sigmap[sigmatch][0]] 750 | # if (tmpdepth != depth): 751 | # print("INFO: depths of feature fragments do not" + 752 | # " match (%s, %s)!" % (str(tmpdepth), str(depth))) 753 | tmpdepth = min(tmpdepth, depth) 754 | tmpcode += code 755 | afeatures[sigmap[sigmatch][0]] = (tmpdepth, tmpcode) 756 | sigmap[sigmatch].append(sig) 757 | except NoEquivalentSigError: 758 | # mergedfeatures get the depth of minus one 759 | # so this way need to make less amount of changes here 760 | afeatures[sig] = (depth, list(code)) 761 | sigmap[psig] = [sig] 762 | 763 | 764 | global __curfile 765 | fcount = 0 766 | files = returnFileNames(folder, ['.xml']) 767 | files.sort() 768 | ftotal = len(files) 769 | 770 | # get statistics for all files 771 | # and merge the features 772 | for file in files: 773 | __curfile = file 774 | 775 | try: 776 | tree = etree.parse(file) 777 | except etree.XMLSyntaxError: 778 | print("ERROR: cannot parse (%s). Skipping this file." % os.path.join(folder, file)) 779 | continue 780 | 781 | root = tree.getroot() 782 | try: 783 | (features, _, featuresgrouter, elses) = _getFeatures(root, options) 784 | except IfdefEndifMismatchError: 785 | print("ERROR: ifdef-endif mismatch in file (%s)" % (os.path.join(folder, file))) 786 | continue 787 | 788 | # remove #else branches from list of features as there is no existing signature in the source code! 789 | if not options.rewriteifdefs: 790 | features = OrderedDict((sig, value) 791 | for sig, value in features.iteritems() 792 | if not sig.startswith(_elsePrefix)) 793 | 794 | # merge features of file in the global list of features 795 | _mergeFeatures(features) 796 | 797 | # calculate nesting depths (per block and per branch) 798 | _getNestingDepths(root) 799 | 800 | # file successfully parsed 801 | fcount += 1 802 | print('INFO: parsing file (%5d) of (%5d) -- (%s).' % (fcount, ftotal, os.path.join(folder, file))) 803 | 804 | # get signatures and defines 805 | sigs = _flatten(sigmap.values()) 806 | defs = list(__defset) 807 | 808 | # preparation: opn file for writing 809 | stfheadings = ['name', 'values'] 810 | stfrow = [None]*len(stfheadings) 811 | stfhandle, stfwriter = _prologCSV(os.path.join(folder, os.pardir), __metricvaluesfile, stfheadings) 812 | 813 | # scattering and tangling values 814 | # each signature is used once per file 815 | 816 | (scatvalues, tangvalues) = _getScatteringTanglingValues(sigs, defs) 817 | scats = sorted([x[1] for x in scatvalues]) 818 | tangs = sorted([x[1] for x in tangvalues]) 819 | 820 | stfrow[0] = "tangling" 821 | tanglingstring = ';'.join(map(str, tangs)) 822 | stfrow[1] = tanglingstring 823 | stfwriter.writerow(stfrow) 824 | 825 | stfrow[0] = "scattering" 826 | scatteringstring = ';'.join(map(str, scats)) 827 | stfrow[1] = scatteringstring 828 | stfwriter.writerow(stfrow) 829 | 830 | # nesting values 831 | 832 | stfrow[0] = "nestedIfdefsLevels" 833 | ndstring = ';'.join(map(str, __nestedIfdefsLevels)) 834 | stfrow[1] = ndstring 835 | stfwriter.writerow(stfrow) 836 | 837 | stfhandle.close() 838 | 839 | # MERGED VALUES 840 | 841 | # scattering + tangling (merged) 842 | # each signature is used only once per project (string equality) 843 | (scatvalues_merged, tangvalues_merged) = _getScatteringTanglingValues(list(set(sigs)), defs) 844 | 845 | sd, sdcsv = _prologCSV(os.path.join(folder, os.pardir), "merged_scattering_degrees.csv", ["define","SD"], delimiter=",") 846 | for (define, scat) in scatvalues_merged: 847 | sdcsv.writerow([define,scat]) 848 | sd.close() 849 | 850 | td, tdcsv = _prologCSV(os.path.join(folder, os.pardir), "merged_tangling_degrees.csv", ["signature","TD"], delimiter=",") 851 | for (sig, tang) in tangvalues_merged: 852 | tdcsv.writerow([sig,tang]) 853 | td.close() 854 | 855 | nd, ndcsv = _prologCSV(os.path.join(folder, os.pardir), "nesting_degrees_toplevel_branches.csv", ["file", "signature", "ND"], delimiter=",") # , "linenumber" 856 | for (file, elem, sig, depth) in __nestingDepthsOfBranches: 857 | 858 | #adjust file name if wanted 859 | if options.filenamesRelative : # relative file name (root is project folder (not included in path)) 860 | file = os.path.relpath(file, folder) 861 | 862 | if options.filenames == options.FILENAME_SRCML : # cppstats file names 863 | pass # nothing to do here, as the file path is the cppstats path by default 864 | if options.filenames == options.FILENAME_SOURCE : # source file name 865 | file = file.replace(".xml", "").replace("/_cppstats/", "/source/", 1) 866 | 867 | # print information to file 868 | ndcsv.writerow([file, sig, depth]) # , elem.sourceline - 1 869 | nd.close() 870 | 871 | 872 | # ################################################## 873 | # add command line options 874 | 875 | def addCommandLineOptionsMain(optionparser): 876 | ''' add command line options for a direct call of this script''' 877 | optionparser.add_argument("--folder", dest="folder", 878 | help="input folder [default=%(default)s]", default=".") 879 | 880 | 881 | def addCommandLineOptions(optionparser) : 882 | # TODO implement CSP solving? 883 | # optionparser.add_option("--csp", dest="csp", action="store_true", 884 | # default=False, help="make use of csp solver to check " \ 885 | # "feature expression equality [default=False]") 886 | # optionparser.add_option("--str", dest="str", action="store_true", 887 | # default=True, help="make use of simple string comparision " \ 888 | # "for checking feature expression equality [default=True]") 889 | optionparser.add_argument("--norewriteifdefs", dest="rewriteifdefs", 890 | action="store_false", default=True, 891 | help="rewrite nested #ifdefs and #elifs as a conjunction of " 892 | "inner and outer expressions [default=%(default)s]\n" 893 | "(exception are #else tags, which ARE rewritten as " 894 | "negation of the #if branch! see also --norewriteelse " 895 | "of analysis GENERALVALUES)") 896 | #FIXME add command line function to remove #else too! 897 | 898 | 899 | # ################################################ 900 | # path of the main output file 901 | 902 | def getResultsFile(): 903 | return __outputfile 904 | 905 | 906 | ################################################## 907 | if __name__ == '__main__': 908 | 909 | 910 | ################################################## 911 | # options parsing 912 | parser = ArgumentParser(formatter_class=RawTextHelpFormatter) 913 | addCommandLineOptionsMain(parser) 914 | addCommandLineOptions(parser) 915 | 916 | options = parser.parse_args() 917 | 918 | # #### 919 | # main 920 | 921 | folder = os.path.abspath(options.folder) 922 | if (os.path.isdir(folder)): 923 | apply(folder, options) 924 | else: 925 | sys.exit(-1) 926 | -------------------------------------------------------------------------------- /cppstats.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # cppstats is a suite of analyses for measuring C preprocessor-based 3 | # variability in software product lines. 4 | # Copyright (C) 2015 University of Passau, Germany 5 | # 6 | # This program is free software: you can redistribute it and/or modify it 7 | # under the terms of the GNU Lesser General Public License as published 8 | # by the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # This program is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14 | # Lesser General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU Lesser General Public 17 | # License along with this program. If not, see 18 | # . 19 | # 20 | # Contributors: 21 | # Claus Hunsen 22 | # Andreas Ringlstetter 23 | 24 | 25 | pushd `dirname $0` > /dev/null 26 | CPPSTATS=`pwd` 27 | popd > /dev/null 28 | 29 | cd ${CPPSTATS} 30 | PYTHONPATH="$PYTHONPATH:$CPPSTATS/lib" 31 | ./cppstats/cppstats.py "$@" 32 | -------------------------------------------------------------------------------- /cppstats/__init__.py: -------------------------------------------------------------------------------- 1 | # cppstats is a suite of analyses for measuring C preprocessor-based 2 | # variability in software product lines. 3 | # Copyright (C) 2014-2015 University of Passau, Germany 4 | # 5 | # This program is free software: you can redistribute it and/or modify it 6 | # under the terms of the GNU Lesser General Public License as published 7 | # by the Free Software Foundation, either version 3 of the License, or 8 | # (at your option) any later version. 9 | # 10 | # This program is distributed in the hope that it will be useful, 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # Lesser General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU Lesser General Public 16 | # License along with this program. If not, see 17 | # . 18 | # 19 | # Contributors: 20 | # Claus Hunsen 21 | # Andreas Ringlstetter 22 | 23 | 24 | # ################################################# 25 | # make version number available in setup.py 26 | 27 | from cppstats import version 28 | -------------------------------------------------------------------------------- /cppstats/analysis.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2014-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Claus Hunsen 23 | # Andreas Ringlstetter 24 | 25 | 26 | # ################################################# 27 | # imports from the std-library 28 | 29 | import os 30 | import sys 31 | import shutil # for copying files and folders 32 | import errno # for error/exception handling 33 | import threading # for parallelism 34 | import subprocess # for calling other commands 35 | import re # for regular expressions 36 | from abc import ABCMeta, abstractmethod # abstract classes 37 | from argparse import ArgumentParser, RawTextHelpFormatter # for parameters to this script 38 | from collections import OrderedDict # for ordered dictionaries 39 | 40 | # ################################################# 41 | # imports from subfolders 42 | 43 | import cppstats, cli 44 | 45 | # import different kinds of analyses 46 | from analyses import general, generalvalues, discipline, featurelocations, derivative, interaction 47 | 48 | 49 | # ################################################# 50 | # global constants 51 | 52 | 53 | 54 | # ################################################# 55 | # platform specific preliminaries 56 | 57 | # cf. https://docs.python.org/2/library/sys.html#sys.platform 58 | __platform = sys.platform.lower() 59 | 60 | __iscygwin = False 61 | if (__platform.startswith("cygwin")): 62 | __iscygwin = True 63 | elif (__platform.startswith("darwin") or __platform.startswith("linux")): 64 | pass 65 | else: 66 | print "Your system '" + __platform + "' is not supported right now." 67 | 68 | 69 | # ################################################# 70 | # helper functions 71 | 72 | def notify(message): 73 | if (__iscygwin): 74 | return 75 | 76 | # FIXME enable notifications again! 77 | # import pynotify # for system notifications 78 | # 79 | # pynotify.init("cppstats") 80 | # notice = pynotify.Notification(message) 81 | # notice.show() 82 | 83 | 84 | # ################################################# 85 | # abstract analysis thread 86 | 87 | class AbstractAnalysisThread(object): 88 | '''This class analyzes a whole project according to the given kind of analysis in an independent thread.''' 89 | __metaclass__ = ABCMeta 90 | 91 | def __init__(self, options, inputfolder=None, inputfile=None): 92 | self.options = options 93 | self.notrunnable = False 94 | 95 | if (inputfolder): 96 | self.file = None 97 | self.folder = os.path.join(inputfolder, self.getPreparationFolder()) 98 | self.project = os.path.basename(self.folder) 99 | 100 | elif (inputfile): 101 | self.file = inputfile 102 | self.outfile = self.options.outfile 103 | self.project = os.path.basename(self.file) 104 | 105 | # get full path of temp folder for 106 | import tempfile 107 | tmpfolder = tempfile.mkdtemp(suffix=self.getPreparationFolder()) 108 | self.tmpfolder = tmpfolder 109 | self.folder = os.path.join(tmpfolder, self.getPreparationFolder()) 110 | os.makedirs(self.folder) # create the folder actually 111 | 112 | self.resultsfile = os.path.join(self.tmpfolder, self.getResultsFile()) 113 | 114 | else: 115 | self.notrunnable = True 116 | 117 | 118 | def startup(self): 119 | # LOGGING 120 | notify("starting '" + self.getName() + "' analysis:\n " + self.project) 121 | print "# starting '" + self.getName() + "' analysis: " + self.project 122 | 123 | def teardown(self): 124 | 125 | # delete temp folder for file-based preparation 126 | if (self.file): 127 | shutil.rmtree(self.tmpfolder) 128 | 129 | # LOGGING 130 | notify("finished '" + self.getName() + "' analysis:\n " + self.project) 131 | print "# finished '" + self.getName() + "' analysis: " + self.project 132 | 133 | def run(self): 134 | 135 | if (self.notrunnable): 136 | print "ERROR: No single file or input list of projects given!" 137 | return 138 | 139 | self.startup() 140 | 141 | # copy srcml inputfile to tmp folder again and analyze project there! 142 | if (self.file): 143 | currentFile = os.path.join(self.folder, self.project) 144 | if (not currentFile.endswith(".xml")): 145 | currentFile += ".xml" 146 | shutil.copyfile(self.file, currentFile) 147 | 148 | # for all files in the self.folder (only C and H files) 149 | self.analyze(self.folder) 150 | 151 | # copy main results file from tmp folder to destination, if given 152 | if (self.file and self.resultsfile != self.outfile): 153 | shutil.copyfile(self.resultsfile, self.outfile) 154 | 155 | self.teardown() 156 | 157 | @classmethod 158 | @abstractmethod 159 | def getName(cls): 160 | pass 161 | 162 | @classmethod 163 | @abstractmethod 164 | def getPreparationFolder(self): 165 | pass 166 | 167 | @classmethod 168 | @abstractmethod 169 | def getResultsFile(self): 170 | pass 171 | 172 | @classmethod 173 | @abstractmethod 174 | def addCommandLineOptions(cls, optionParser): 175 | pass 176 | 177 | @abstractmethod 178 | def analyze(self): 179 | pass 180 | 181 | 182 | # ################################################# 183 | # analysis-thread implementations 184 | 185 | class GeneralAnalysisThread(AbstractAnalysisThread): 186 | @classmethod 187 | def getName(cls): 188 | return "general" 189 | 190 | @classmethod 191 | def getPreparationFolder(self): 192 | return "_cppstats" 193 | 194 | @classmethod 195 | def getResultsFile(self): 196 | return general.getResultsFile() 197 | 198 | @classmethod 199 | def addCommandLineOptions(cls, optionParser): 200 | title = "Options for analysis '" + cls.getName() + "'" 201 | group = optionParser.add_argument_group(title.upper()) 202 | general.addCommandLineOptions(group) 203 | 204 | def analyze(self, folder): 205 | general.apply(folder, self.options) 206 | 207 | 208 | class GeneralValuesAnalysisThread(AbstractAnalysisThread): 209 | @classmethod 210 | def getName(cls): 211 | return "generalvalues" 212 | 213 | @classmethod 214 | def getPreparationFolder(self): 215 | return "_cppstats" 216 | 217 | @classmethod 218 | def getResultsFile(self): 219 | return generalvalues.getResultsFile() 220 | 221 | @classmethod 222 | def addCommandLineOptions(cls, optionParser): 223 | title = "Options for analysis '" + cls.getName() + "'" 224 | group = optionParser.add_argument_group(title.upper()) 225 | generalvalues.addCommandLineOptions(group) 226 | 227 | def analyze(self, folder): 228 | generalvalues.apply(folder, self.options) 229 | 230 | 231 | class DisciplineAnalysisThread(AbstractAnalysisThread): 232 | @classmethod 233 | def getName(cls): 234 | return "discipline" 235 | 236 | @classmethod 237 | def getPreparationFolder(self): 238 | return "_cppstats_discipline" 239 | 240 | @classmethod 241 | def getResultsFile(self): 242 | return discipline.getResultsFile() 243 | 244 | @classmethod 245 | def addCommandLineOptions(cls, optionParser): 246 | title = "Options for analysis '" + cls.getName() + "'" 247 | group = optionParser.add_argument_group(title.upper()) 248 | discipline.addCommandLineOptions(group) 249 | 250 | def analyze(self, folder): 251 | discipline.DisciplinedAnnotations(folder, self.options) 252 | 253 | 254 | class FeatureLocationsAnalysisThread(AbstractAnalysisThread): 255 | @classmethod 256 | def getName(cls): 257 | return "featurelocations" 258 | 259 | @classmethod 260 | def getPreparationFolder(self): 261 | return "_cppstats_featurelocations" 262 | 263 | @classmethod 264 | def getResultsFile(self): 265 | return featurelocations.getResultsFile() 266 | 267 | @classmethod 268 | def addCommandLineOptions(cls, optionParser): 269 | title = "Options for analysis '" + cls.getName() + "'" 270 | group = optionParser.add_argument_group(title.upper()) 271 | featurelocations.addCommandLineOptions(group) 272 | 273 | def analyze(self, folder): 274 | featurelocations.apply(folder, self.options) 275 | 276 | 277 | class DerivativeAnalysisThread(AbstractAnalysisThread): 278 | @classmethod 279 | def getName(cls): 280 | return "derivative" 281 | 282 | @classmethod 283 | def getPreparationFolder(self): 284 | return "_cppstats_discipline" 285 | 286 | @classmethod 287 | def getResultsFile(self): 288 | return derivative.getResultsFile() 289 | 290 | @classmethod 291 | def addCommandLineOptions(cls, optionParser): 292 | title = "Options for analysis '" + cls.getName() + "'" 293 | group = optionParser.add_argument_group(title.upper()) 294 | derivative.addCommandLineOptions(group) 295 | 296 | def analyze(self, folder): 297 | derivative.apply(folder) 298 | 299 | 300 | class InteractionAnalysisThread(AbstractAnalysisThread): 301 | @classmethod 302 | def getName(cls): 303 | return "interaction" 304 | 305 | @classmethod 306 | def getPreparationFolder(self): 307 | return "_cppstats_discipline" 308 | 309 | @classmethod 310 | def getResultsFile(self): 311 | return interaction.getResultsFile() 312 | 313 | @classmethod 314 | def addCommandLineOptions(cls, optionParser): 315 | title = "Options for analyses '" + cls.getName() + "'" 316 | group = optionParser.add_argument_group(title.upper()) 317 | interaction.addCommandLineOptions(group) 318 | 319 | def analyze(self, folder): 320 | interaction.apply(folder, self.options) 321 | 322 | 323 | # ################################################# 324 | # collection of analysis threads 325 | 326 | # add all subclass of AbstractAnalysisThread as available analysis kinds 327 | __analysiskinds = [] 328 | for cls in AbstractAnalysisThread.__subclasses__(): 329 | entry = (cls.getName(), cls) 330 | __analysiskinds.append(entry) 331 | 332 | # exit, if there are no analysis threads available 333 | if (len(__analysiskinds) == 0): 334 | print "ERROR: No analysis tasks found! Revert your changes or call the maintainer." 335 | print "Exiting now..." 336 | sys.exit(1) 337 | 338 | __analysiskinds = OrderedDict(__analysiskinds) 339 | 340 | 341 | def getKinds(): 342 | return __analysiskinds 343 | 344 | 345 | # ################################################# 346 | # main method 347 | 348 | 349 | def applyFile(kind, inputfile, options): 350 | kinds = getKinds() 351 | 352 | # get proper preparation thread and call it 353 | threadClass = kinds[kind] 354 | thread = threadClass(options, inputfile=inputfile) 355 | thread.run() 356 | 357 | 358 | def getFoldersFromInputListFile(inputlist): 359 | ''' This method reads the given inputfile line-wise and returns the read lines without line breaks.''' 360 | 361 | file = open(inputlist, 'r') # open input file 362 | folders = file.read().splitlines() # read lines from file without line breaks 363 | file.close() # close file 364 | 365 | folders = filter(lambda f: not f.startswith("#"), folders) # remove commented lines 366 | folders = filter(os.path.isdir, folders) # remove all non-directories 367 | folders = map(os.path.normpath, folders) # normalize paths for easier transformations 368 | 369 | #TODO log removed folders 370 | 371 | return folders 372 | 373 | 374 | def applyFolders(kind, inputlist, options): 375 | kinds = getKinds() 376 | 377 | # get the list of projects/folders to process 378 | folders = getFoldersFromInputListFile(inputlist) 379 | 380 | # for each folder: 381 | for folder in folders: 382 | # start preparations for this single folder 383 | 384 | # get proper preparation thread and call it 385 | threadClass = kinds[kind] 386 | thread = threadClass(options, inputfolder=folder) 387 | thread.run() 388 | 389 | 390 | def applyFoldersAll(inputlist, options): 391 | kinds = getKinds() 392 | for kind in kinds.keys(): 393 | applyFolders(kind, inputlist, options) 394 | 395 | 396 | def main(): 397 | kinds = getKinds() 398 | 399 | # ################################################# 400 | # options parsing 401 | 402 | options = cli.getOptions(kinds, step=cli.steps.ANALYSIS) 403 | 404 | # ################################################# 405 | # main 406 | 407 | if (options.inputfile): 408 | 409 | # split --file argument 410 | options.infile = os.path.normpath(os.path.abspath(options.inputfile[0])) # IN 411 | options.outfile = os.path.normpath(os.path.abspath(options.inputfile[1])) # OUT 412 | 413 | # check if inputfile exists 414 | if (not os.path.isfile(options.infile)): 415 | print "ERROR: input file '{}' cannot be found!".format(options.infile) 416 | sys.exit(1) 417 | 418 | applyFile(options.kind, options.infile, options) 419 | 420 | elif (options.inputlist): 421 | # handle --list argument 422 | options.inputlist = os.path.normpath(os.path.abspath(options.inputlist)) # LIST 423 | 424 | # check if list file exists 425 | if (not os.path.isfile(options.inputlist)): 426 | print "ERROR: input file '{}' cannot be found!".format(options.inputlist) 427 | sys.exit(1) 428 | 429 | if (options.allkinds): 430 | applyFoldersAll(options.inputlist, options) 431 | else: 432 | applyFolders(options.kind, options.inputlist, options) 433 | 434 | else: 435 | print "This should not happen! No input file or list of projects given!" 436 | sys.exit(1) 437 | 438 | if __name__ == '__main__': 439 | main() 440 | -------------------------------------------------------------------------------- /cppstats/cli.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # cppstats is a suite of analyses for measuring C preprocessor-based 3 | # variability in software product lines. 4 | # Copyright (C) 2014-2015 University of Passau, Germany 5 | # 6 | # This program is free software: you can redistribute it and/or modify it 7 | # under the terms of the GNU Lesser General Public License as published 8 | # by the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # This program is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14 | # Lesser General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU Lesser General Public 17 | # License along with this program. If not, see 18 | # . 19 | # 20 | # Contributors: 21 | # Claus Hunsen 22 | 23 | 24 | # ################################################# 25 | # imports from the std-library 26 | 27 | import sys 28 | from argparse import ArgumentParser, RawTextHelpFormatter, _VersionAction # for parameters to this script 29 | 30 | # ################################################# 31 | # imports from subfolders 32 | 33 | import preparation, analysis 34 | import cppstats as cstats # import cppstats.py and avoid confusion with module 35 | 36 | 37 | # ################################################# 38 | # external modules 39 | 40 | # enums 41 | from enum import Enum 42 | 43 | 44 | # ################################################# 45 | # global constants 46 | 47 | __inputlist_default = "cppstats_input.txt" 48 | 49 | 50 | # ################################################# 51 | # definition of cppstats steps 52 | class steps(Enum): 53 | ALL = 0 54 | PREPARATION = 1 55 | ANALYSIS = 2 56 | 57 | 58 | # ################################################# 59 | # custom actions 60 | 61 | class CppstatsVersionAction(_VersionAction): 62 | # FIXME not needed for Python 3.4+! see http://bugs.python.org/issue18920 63 | ''' 64 | subclass to _VersionAction as that class prints version information 65 | to ``stderr``, but this subclass prints to ``stdout`` instead. 66 | 67 | Note: This is default in Python 3.4+. 68 | ''' 69 | 70 | def __init__(self, *args, **kwargs): 71 | """Initialisation method for the _VersionAction class""" 72 | _VersionAction.__init__(self, *args, **kwargs) 73 | 74 | def __call__(self, parser, namespace, values, option_string=None): 75 | version = self.version 76 | if version is None: 77 | version = parser.version 78 | formatter = parser._get_formatter() 79 | formatter.add_text(version) 80 | # parser.exit(message=formatter.format_help()) 81 | 82 | # change output to sys.stdout and exit then without a message 83 | parser._print_message(message=formatter.format_help(), file=sys.stdout) 84 | parser.exit(status=0) 85 | 86 | 87 | # ################################################# 88 | # construct ArgumentParser 89 | 90 | def getOptions(kinds, step=steps.ALL): 91 | """ 92 | Constructs the parser needed for cppstats. Includes following procedure: 93 | * addition of step-specific arguments 94 | * parsing 95 | * addition of constants 96 | * checking of constraints 97 | 98 | :arg kinds : the list of preperation/analysis kinds, dependent on the step parameter 99 | :arg step : the variant of cppstats to execute: one of cli.steps (ALL, PREPARATION, ANALYSIS) 100 | :rtype : the resulting options 101 | """ 102 | 103 | parser = ArgumentParser(formatter_class=RawTextHelpFormatter) 104 | 105 | # version (uses CppstatsVersionAction instead of 'version' as action) 106 | parser.add_argument('--version', action=CppstatsVersionAction, version=cstats.version()) 107 | 108 | 109 | # ADD KIND ARGUMENT 110 | 111 | # kinds 112 | kindgroup = parser.add_mutually_exclusive_group(required=False) 113 | kindgroup.add_argument("--kind", choices=kinds.keys(), dest="kind", 114 | default=kinds.keys()[0], metavar="", 115 | help="the preparation to be performed [default: %(default)s]") 116 | kindgroup.add_argument("-a", "--all", action="store_true", dest="allkinds", default=False, 117 | help="perform all available kinds of preparation/analysis [default: %(default)s]") 118 | 119 | 120 | # ADD INPUT TYPE (list or file) 121 | 122 | # input 1 123 | inputgroup = parser.add_mutually_exclusive_group(required=False) # TODO check if True is possible some time... 124 | inputgroup.add_argument("--list", type=str, dest="inputlist", metavar="LIST", 125 | nargs="?", default=__inputlist_default, const=__inputlist_default, 126 | help="a file that contains the list of input projects/folders [default: %(default)s]") 127 | # input 2 128 | if step == steps.ALL: 129 | inputgroup.add_argument("--file", type=str, dest="inputfile", nargs=2, metavar=("IN", "OUT"), 130 | help="a source file IN that is prepared and analyzed, the analysis results are written to OUT" 131 | "\n(--list is the default)") 132 | elif step == steps.PREPARATION: 133 | inputgroup.add_argument("--file", type=str, dest="inputfile", nargs=2, metavar=("IN", "OUT"), 134 | help="a source file IN that is prepared, the preparation result is written to OUT" 135 | "\n(--list is the default)") 136 | elif step == steps.ANALYSIS: 137 | inputgroup.add_argument("--file", type=str, dest="inputfile", nargs=2, metavar=("IN", "OUT"), 138 | help="a srcML file IN that is analyzed, the analysis results are written to OUT" 139 | "\n(--list is the default)") 140 | 141 | 142 | # ADD VARIOUS STEP-DEPENDENT ARGUMENTS 143 | 144 | # no backup files 145 | if step == steps.ALL or step == steps.PREPARATION: 146 | parser.add_argument("--nobak", action="store_true", dest="nobak", default=False, 147 | help="do not backup files during preparation [default: %(default)s]") 148 | 149 | # add general CLI options applying for all or several analyses 150 | if step == steps.ALL or step == steps.ANALYSIS: 151 | # constants for the choices of '--filenames' are added in method 'addConstants' 152 | parser.add_argument("--filenames", choices=[0, 1], dest="filenames", default=0, 153 | help="determines the file paths to print [default: %(default)s]\n" 154 | "(0=paths to srcML files, 1=paths to source files)") 155 | parser.add_argument("--filenamesRelative", action="store_true", dest="filenamesRelative", default=False, 156 | help="print relative file names [default: %(default)s]\n" 157 | "e.g., '/projects/apache/_cppstats/afile.c.xml' gets 'afile.c.xml'.") 158 | 159 | 160 | # ADD POSSIBLE PREPARATION/ANALYSIS KINDS AND THEIR COMMAND-LINE ARGUMENTS 161 | 162 | if step == steps.ALL: 163 | parser.add_argument_group("Possible Kinds of Analyses ".upper(), ", ".join(kinds.keys())) 164 | 165 | # add options for each analysis kind 166 | for kind in kinds.values(): 167 | analysisPart = kind[1] 168 | analysisThread = analysis.getKinds().get(analysisPart) 169 | analysisThread.addCommandLineOptions(parser) 170 | 171 | elif step == steps.PREPARATION: 172 | parser.add_argument_group("Possible Kinds of Preparation ".upper(), ", ".join(kinds.keys())) 173 | 174 | elif step == steps.ANALYSIS: 175 | parser.add_argument_group("Possible Kinds of Analyses ".upper(), ", ".join(kinds.keys())) 176 | 177 | # add options for each analysis kind 178 | for cls in kinds.values(): 179 | cls.addCommandLineOptions(parser) 180 | 181 | 182 | # PARSE OPTIONS 183 | 184 | options = parser.parse_args() 185 | 186 | 187 | # ADD CONSTANTS TO OPTIONS 188 | 189 | addConstants(options) 190 | 191 | 192 | # CHECK CONSTRAINTS ON OPTIONS 193 | 194 | checkConstraints(options) 195 | 196 | 197 | # RETURN 198 | 199 | return options 200 | 201 | 202 | def addConstants(options): 203 | # add option constants 204 | # --filenames 205 | options.FILENAME_SRCML = 0 206 | options.FILENAME_SOURCE = 1 207 | 208 | 209 | def checkConstraints(options): 210 | # constraints 211 | if (options.allkinds == True and options.inputfile): 212 | print "Using all kinds of preparation for a single input and output file is weird!" 213 | sys.exit(1) 214 | -------------------------------------------------------------------------------- /cppstats/cppstats.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2014-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Claus Hunsen 23 | # Andreas Ringlstetter 24 | 25 | 26 | # ################################################# 27 | # imports from the std-library 28 | 29 | import os 30 | import sys 31 | import shutil # for copying files and folders 32 | import errno # for error/exception handling 33 | import threading # for parallelism 34 | import subprocess # for calling other commands 35 | import re # for regular expressions 36 | from abc import ABCMeta, abstractmethod # abstract classes 37 | #import pynotify # for system notifications 38 | from argparse import ArgumentParser, RawTextHelpFormatter # for parameters to this script 39 | from collections import OrderedDict # for ordered dictionaries 40 | import tempfile # for temporary files 41 | 42 | # ################################################# 43 | # imports from subfolders 44 | 45 | # import different kinds of analyses 46 | import cli, preparation, analysis 47 | 48 | 49 | # ################################################# 50 | # version number 51 | 52 | __version__ = "v0.9.4" 53 | 54 | def version() : 55 | return "cppstats " + __version__ 56 | 57 | 58 | # ################################################# 59 | # collection of analyses 60 | 61 | # add all kinds of analyses: (name -> (preparation, analysis)) 62 | __kinds = [] 63 | __kinds.append(('general', ('general', 'general'))) 64 | __kinds.append(('generalvalues', ('general', 'generalvalues'))) 65 | __kinds.append(('discipline', ('discipline', 'discipline'))) 66 | __kinds.append(('featurelocations', ('featurelocations', 'featurelocations'))) 67 | __kinds.append(('derivative', ('discipline', 'derivative'))) 68 | __kinds.append(('interaction', ('discipline', 'interaction'))) 69 | 70 | 71 | # exit, if there are no analysis threads available 72 | if (len(__kinds) == 0) : 73 | print "ERROR: No analyses available! Revert your changes or call the maintainer." 74 | print "Exiting now..." 75 | sys.exit(1) 76 | 77 | __kinds = OrderedDict(__kinds) 78 | 79 | 80 | # ################################################# 81 | # main method 82 | 83 | 84 | def applyFile(kind, infile, outfile, options): 85 | 86 | tmpfile = tempfile.mkstemp(suffix=".xml")[1] # temporary srcML file 87 | 88 | # preparation 89 | options.infile = infile 90 | options.outfile = tmpfile 91 | preparation.applyFile(kind, options.infile, options) 92 | 93 | # analysis 94 | options.infile = tmpfile 95 | options.outfile = outfile 96 | analysis.applyFile(kind, options.infile, options) 97 | 98 | # delete temp file 99 | os.remove(tmpfile) 100 | 101 | def applyFolders(option_kind, inputlist, options): 102 | kind = __kinds.get(option_kind) 103 | preparationKind = kind[0] 104 | analysisKind = kind[1] 105 | 106 | preparation.applyFolders(preparationKind, inputlist, options) 107 | analysis.applyFolders(analysisKind, inputlist, options) 108 | 109 | def applyFoldersAll(inputlist, options): 110 | for kind in __kinds.keys(): 111 | applyFolders(kind, inputlist, options) 112 | 113 | 114 | def main(): 115 | # ################################################# 116 | # options parsing 117 | 118 | options = cli.getOptions(__kinds, step = cli.steps.ALL) 119 | 120 | # ################################################# 121 | # main 122 | 123 | if (options.inputfile): 124 | 125 | # split --file argument 126 | options.infile = os.path.normpath(os.path.abspath(options.inputfile[0])) # IN 127 | options.outfile = os.path.normpath(os.path.abspath(options.inputfile[1])) # OUT 128 | 129 | # check if inputfile exists 130 | if (not os.path.isfile(options.infile)): 131 | print "ERROR: input file '{}' cannot be found!".format(options.infile) 132 | sys.exit(1) 133 | 134 | applyFile(options.kind, options.infile, options.outfile, options) 135 | 136 | elif (options.inputlist): 137 | # handle --list argument 138 | options.inputlist = os.path.normpath(os.path.abspath(options.inputlist)) # LIST 139 | 140 | # check if list file exists 141 | if (not os.path.isfile(options.inputlist)): 142 | print "ERROR: input file '{}' cannot be found!".format(options.inputlist) 143 | sys.exit(1) 144 | 145 | if (options.allkinds): 146 | applyFoldersAll(options.inputlist, options) 147 | else: 148 | applyFolders(options.kind, options.inputlist, options) 149 | 150 | else: 151 | print "This should not happen! No input file or list of projects given!" 152 | sys.exit(1) 153 | 154 | if __name__ == '__main__': 155 | main() 156 | -------------------------------------------------------------------------------- /cppstats/preparation.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2014-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Claus Hunsen 23 | # Andreas Ringlstetter 24 | 25 | 26 | # ################################################# 27 | # imports from the std-library 28 | 29 | import os 30 | import sys 31 | import shutil # for copying files and folders 32 | import errno # for error/exception handling 33 | import subprocess # for calling other commands 34 | import re # for regular expressions 35 | from abc import ABCMeta, abstractmethod # abstract classes 36 | from collections import OrderedDict 37 | 38 | 39 | # ################################################# 40 | # paths 41 | 42 | import preparations 43 | 44 | def getPreparationScript(filename): 45 | return os.path.join(os.path.dirname(preparations.__file__), filename) 46 | 47 | 48 | # ################################################# 49 | # imports from subfolders 50 | 51 | import cli 52 | 53 | # for rewriting of #ifdefs to "if defined(..)" 54 | # for turning multiline macros to oneliners 55 | # for deletion of include guards in H files 56 | from preparations import rewriteIfdefs, rewriteMultilineMacros, deleteIncludeGuards 57 | 58 | from lib import cpplib 59 | 60 | # ################################################# 61 | # global constants 62 | 63 | _filepattern_c = ('.c', '.C') 64 | _filepattern_h = ('.h', '.H') 65 | _filepattern = _filepattern_c + _filepattern_h 66 | 67 | _cvs_pattern = (".git", ".cvs", ".svn") 68 | 69 | 70 | # ################################################# 71 | # helper functions 72 | 73 | def notify(message): 74 | pass 75 | 76 | # import pynotify # for system notifications 77 | # 78 | # pynotify.init("cppstats") 79 | # notice = pynotify.Notification(message) 80 | # notice.show() 81 | 82 | 83 | # function for ignore pattern 84 | def filterForFiles(dirpath, contents, pattern=_filepattern): 85 | filesToIgnore = [filename for filename in contents if 86 | not filename.endswith(pattern) and 87 | not os.path.isdir(os.path.join(dirpath, filename)) 88 | ] 89 | foldersToIgnore = [dir for dir in contents if 90 | dir in _cvs_pattern and 91 | os.path.isdir(os.path.join(dirpath, dir)) 92 | ] 93 | return filesToIgnore + foldersToIgnore 94 | 95 | 96 | def runBashCommand(command, shell=False, stdin=None, stdout=None): 97 | # split command if not a list/tuple is given already 98 | if type(command) is str: 99 | command = command.split() 100 | 101 | process = subprocess.Popen(command, shell=shell, stdin=stdin, stdout=stdout, stderr=stdout) 102 | out, err = process.communicate() # TODO do something with the output 103 | process.wait() 104 | 105 | # FIXME do something with return value of process.wait()! 106 | # if ret is not 0: 107 | # print "#### " + " ".join(command) + " returned " + str(ret) 108 | 109 | 110 | def replaceMultiplePatterns(replacements, infile, outfile): 111 | with open(infile, "rb") as source: 112 | with open(outfile, "w") as target: 113 | data = source.read() 114 | for pattern, replacement in replacements.iteritems(): 115 | data = re.sub(pattern, replacement, data, flags=re.MULTILINE) 116 | target.write(data) 117 | 118 | 119 | def stripEmptyLinesFromFile(infile, outfile): 120 | with open(infile, "rb") as source: 121 | with open(outfile, "w") as target: 122 | for line in source: 123 | if line.strip(): 124 | target.write(line) 125 | 126 | 127 | def silentlyRemoveFile(filename): 128 | try: 129 | os.remove(filename) 130 | except OSError as e: # this would be "except OSError, e:" before Python 2.6 131 | if e.errno != errno.ENOENT: # errno.ENOENT = no such file or directory 132 | raise # re-raise exception if a different error occured 133 | 134 | 135 | def src2srcml(src, srcml): 136 | __s2sml = "srcml" 137 | runBashCommand([__s2sml, src, "--language=C"], stdout=open(srcml, 'w+')) # + " -o " + srcml) 138 | # FIXME incorporate "|| rm ${f}.xml" from bash 139 | 140 | 141 | def srcml2src(srcml, src): 142 | __sml2s = "srcml" 143 | runBashCommand([__sml2s, srcml], stdout=open(src, 'w+')) # + " -o " + src) 144 | 145 | 146 | # ################################################# 147 | # abstract preparation thread 148 | 149 | class AbstractPreparationThread(object): 150 | '''This class prepares a single folder according to the given kind of preparations in an independent thread.''' 151 | __metaclass__ = ABCMeta 152 | sourcefolder = "source" 153 | 154 | def __init__(self, options, inputfolder=None, inputfile=None): 155 | self.options = options 156 | self.notrunnable = False 157 | 158 | if (inputfolder): 159 | self.file = None 160 | self.folder = inputfolder 161 | self.source = os.path.join(self.folder, self.sourcefolder) 162 | 163 | self.project = os.path.basename(self.folder) 164 | 165 | # get full path of subfolder "_cppstats" 166 | self.subfolder = os.path.join(self.folder, self.getSubfolder()) 167 | 168 | elif (inputfile): 169 | self.file = inputfile 170 | self.outfile = self.options.outfile 171 | self.folder = os.path.dirname(self.file) 172 | 173 | self.project = os.path.basename(self.file) 174 | 175 | # get full path of temp folder for 176 | import tempfile 177 | self.subfolder = tempfile.mkdtemp(suffix=self.getSubfolder()) 178 | 179 | 180 | else: 181 | self.notrunnable = True 182 | 183 | def startup(self): 184 | # LOGGING 185 | notify("starting '" + self.getPreparationName() + "' preparations:\n " + self.project) 186 | print "# starting '" + self.getPreparationName() + "' preparations: " + self.project 187 | 188 | def teardown(self): 189 | 190 | # delete temp folder for file-based preparation 191 | if (self.file): 192 | shutil.rmtree(self.subfolder) 193 | 194 | # LOGGING 195 | notify("finished '" + self.getPreparationName() + "' preparations:\n " + self.project) 196 | print "# finished '" + self.getPreparationName() + "' preparations: " + self.project 197 | 198 | def run(self): 199 | 200 | if (self.notrunnable): 201 | print "ERROR: No single file or input list of projects given!" 202 | return 203 | 204 | self.startup() 205 | 206 | if (self.file): 207 | 208 | self.currentFile = os.path.join(self.subfolder, self.project) 209 | shutil.copyfile(self.file, self.currentFile) 210 | 211 | self.backupCounter = 0 212 | self.prepareFile() 213 | 214 | shutil.copyfile(self.currentFile + ".xml", self.outfile) 215 | else: 216 | # copy C and H files to self.subfolder 217 | self.copyToSubfolder() 218 | # preparation for all files in the self.subfolder (only C and H files) 219 | for root, subFolders, files in os.walk(self.subfolder): 220 | for file in files: 221 | f = os.path.join(root, file) 222 | self.currentFile = f 223 | 224 | self.backupCounter = 0 225 | self.prepareFile() 226 | 227 | self.teardown() 228 | 229 | def copyToSubfolder(self): 230 | 231 | # TODO debug 232 | # echo '### preparing sources ...' 233 | # echo '### copying all-files to one folder ...' 234 | 235 | # delete folder if already existing 236 | if os.path.isdir(self.subfolder): 237 | shutil.rmtree(self.subfolder) 238 | 239 | # copy all C and H files recursively to the subfolder 240 | shutil.copytree(self.source, self.subfolder, ignore=filterForFiles) 241 | 242 | def backupCurrentFile(self): 243 | '''# backup file''' 244 | if (not self.options.nobak): 245 | bak = self.currentFile + ".bak" + str(self.backupCounter) 246 | shutil.copyfile(self.currentFile, bak) 247 | self.backupCounter += 1 248 | 249 | @classmethod 250 | @abstractmethod 251 | def getPreparationName(cls): 252 | pass 253 | 254 | @abstractmethod 255 | def getSubfolder(self): 256 | pass 257 | 258 | @abstractmethod 259 | def prepareFile(self): 260 | pass 261 | 262 | # TODO refactor such that file has not be opened several times! (__currentfile) 263 | def rewriteMultilineMacros(self): 264 | tmp = self.currentFile + "tmp.txt" 265 | 266 | self.backupCurrentFile() # backup file 267 | 268 | # turn multiline macros to oneliners 269 | shutil.move(self.currentFile, tmp) # move for script 270 | rewriteMultilineMacros.translate(tmp, self.currentFile) # call function 271 | 272 | os.remove(tmp) # remove temp file 273 | 274 | def formatCode(self): 275 | tmp = self.currentFile + "tmp.txt" 276 | 277 | self.backupCurrentFile() # backup file 278 | 279 | # call astyle to format file in Java-style 280 | shutil.move(self.currentFile, tmp) # move for script 281 | runBashCommand(["astyle", "--style=java"], stdin=open(tmp, 'r'), stdout=open(self.currentFile, 'w+')) 282 | 283 | os.remove(tmp) # remove temp file 284 | 285 | def deleteComments(self): 286 | tmp = self.currentFile + "tmp.xml" 287 | tmp_out = self.currentFile + "tmp_out.xml" 288 | 289 | self.backupCurrentFile() # backup file 290 | 291 | # call src2srcml to transform code to xml 292 | src2srcml(self.currentFile, tmp) 293 | 294 | # delete all comments in the xml and write to another file 295 | runBashCommand(["xsltproc", getPreparationScript("deleteComments.xsl"), tmp], stdout=open(tmp_out, 'w+')) 296 | 297 | # re-transform the xml to a normal source file 298 | srcml2src(tmp_out, self.currentFile) 299 | 300 | # delete temp files 301 | silentlyRemoveFile(tmp) 302 | silentlyRemoveFile(tmp_out) 303 | 304 | def deleteWhitespace(self): 305 | """deletes leading, trailing and inter (# ... if) whitespaces, 306 | replaces multiple whitespace with a single space""" 307 | tmp = self.currentFile + "tmp.txt" 308 | 309 | self.backupCurrentFile() # backup file 310 | 311 | # replace patterns with replacements 312 | replacements = { 313 | '^[ \t]+': '', # leading whitespaces 314 | '[ \t]+$': '', # trailing whitespaces 315 | '#[ \t]+': '#', # inter (# ... if) whitespaces # TODO '^#[ \t]+' or '#[ \t]+' 316 | '\t': ' ', # tab to space 317 | '[ \t]{2,}': ' ' # multiple whitespace to one space 318 | 319 | } 320 | replaceMultiplePatterns(replacements, self.currentFile, tmp) 321 | 322 | # move temp file to output file 323 | shutil.move(tmp, self.currentFile) 324 | 325 | def rewriteIfdefsAndIfndefs(self): 326 | tmp = self.currentFile + "tmp.txt" 327 | 328 | self.backupCurrentFile() # backup file 329 | 330 | # rewrite #if(n)def ... to #if (!)defined(...) 331 | d = rewriteIfdefs.rewriteFile(self.currentFile, open(tmp, 'w')) 332 | 333 | # move temp file to output file 334 | shutil.move(tmp, self.currentFile) 335 | 336 | def removeIncludeGuards(self): 337 | # include guards only exist in H files, otherwise return 338 | _, extension = os.path.splitext(self.currentFile) 339 | if (extension not in _filepattern_h): 340 | return 341 | 342 | tmp = self.currentFile + "tmp.txt" 343 | 344 | self.backupCurrentFile() # backup file 345 | 346 | # delete include guards 347 | deleteIncludeGuards.apply(self.currentFile, open(tmp, 'w')) 348 | 349 | # move temp file to output file 350 | shutil.move(tmp, self.currentFile) 351 | 352 | def removeOtherPreprocessor(self): 353 | tmp = self.currentFile + "tmp.txt" 354 | 355 | self.backupCurrentFile() # backup file 356 | 357 | # delete other preprocessor statements than #ifdefs 358 | cpplib._filterAnnotatedIfdefs(self.currentFile, tmp) 359 | 360 | # move temp file to output file 361 | shutil.copyfile(tmp, self.currentFile) 362 | 363 | def deleteEmptyLines(self): 364 | tmp = self.currentFile + "tmp.txt" 365 | 366 | self.backupCurrentFile() # backup file 367 | 368 | # remove empty lines 369 | stripEmptyLinesFromFile(self.currentFile, tmp) 370 | 371 | # move temp file to output file 372 | shutil.move(tmp, self.currentFile) 373 | 374 | def transformFileToSrcml(self): 375 | source = self.currentFile 376 | dest = self.currentFile + ".xml" 377 | 378 | # transform to srcml 379 | src2srcml(source, dest) 380 | 381 | 382 | # ################################################# 383 | # preparation-thread implementations 384 | 385 | class GeneralPreparationThread(AbstractPreparationThread): 386 | @classmethod 387 | def getPreparationName(cls): 388 | return "general" 389 | 390 | def getSubfolder(self): 391 | return "_cppstats" 392 | 393 | def prepareFile(self): 394 | # multiline macros 395 | self.rewriteMultilineMacros() 396 | 397 | # delete comments 398 | self.deleteComments() 399 | 400 | # delete leading, trailing and inter (# ... if) whitespaces 401 | self.deleteWhitespace() 402 | 403 | # rewrite #if(n)def ... to #if (!)defined(...) 404 | self.rewriteIfdefsAndIfndefs() 405 | 406 | # removes include guards from H files 407 | self.removeIncludeGuards() 408 | 409 | # delete empty lines 410 | self.deleteEmptyLines() 411 | 412 | # transform file to srcml 413 | self.transformFileToSrcml() 414 | 415 | 416 | class DisciplinePreparationThread(AbstractPreparationThread): 417 | @classmethod 418 | def getPreparationName(cls): 419 | return "discipline" 420 | 421 | def getSubfolder(self): 422 | return "_cppstats_discipline" 423 | 424 | def prepareFile(self): 425 | # multiline macros 426 | self.rewriteMultilineMacros() 427 | 428 | # delete comments 429 | self.deleteComments() 430 | 431 | # delete leading, trailing and inter (# ... if) whitespaces 432 | self.deleteWhitespace() 433 | 434 | # rewrite #if(n)def ... to #if (!)defined(...) 435 | self.rewriteIfdefsAndIfndefs() 436 | 437 | # removes include guards from H files 438 | self.removeIncludeGuards() 439 | 440 | # removes other preprocessor than #ifdefs 441 | self.removeOtherPreprocessor() 442 | 443 | # delete empty lines 444 | self.deleteEmptyLines() 445 | 446 | # transform file to srcml 447 | self.transformFileToSrcml() 448 | 449 | 450 | class FeatureLocationsPreparationThread(AbstractPreparationThread): 451 | @classmethod 452 | def getPreparationName(cls): 453 | return "featurelocations" 454 | 455 | def getSubfolder(self): 456 | return "_cppstats_featurelocations" 457 | 458 | def prepareFile(self): 459 | # multiline macros 460 | self.rewriteMultilineMacros() 461 | 462 | # delete comments 463 | self.deleteComments() 464 | 465 | # delete leading, trailing and inter (# ... if) whitespaces 466 | self.deleteWhitespace() 467 | 468 | # FIXME remove include guards?! 469 | 470 | # rewrite #if(n)def ... to #if (!)defined(...) 471 | self.rewriteIfdefsAndIfndefs() 472 | 473 | # transform file to srcml 474 | self.transformFileToSrcml() 475 | 476 | 477 | class PrettyPreparationThread(AbstractPreparationThread): 478 | @classmethod 479 | def getPreparationName(cls): 480 | return "pretty" 481 | 482 | def getSubfolder(self): 483 | return "_cppstats_pretty" 484 | 485 | def prepareFile(self): 486 | # multiline macros 487 | self.rewriteMultilineMacros() 488 | 489 | # format the code 490 | self.formatCode() 491 | 492 | # # delete comments 493 | # self.deleteComments() 494 | # 495 | # # delete empty lines 496 | # self.deleteEmptyLines() 497 | 498 | 499 | # ################################################# 500 | # collection of preparation threads 501 | 502 | # add all subclass of AbstractPreparationThread as available preparation kinds 503 | __preparationkinds = [] 504 | for cls in AbstractPreparationThread.__subclasses__(): 505 | entry = (cls.getPreparationName(), cls) 506 | __preparationkinds.append(entry) 507 | 508 | # exit, if there are no preparation threads available 509 | if (len(__preparationkinds) == 0): 510 | print "ERROR: No preparation tasks found! Revert your changes or call the maintainer." 511 | print "Exiting now..." 512 | sys.exit(1) 513 | __preparationkinds = OrderedDict(__preparationkinds) 514 | 515 | 516 | def getKinds(): 517 | return __preparationkinds 518 | 519 | 520 | # ################################################# 521 | # main method 522 | 523 | 524 | def applyFile(kind, inputfile, options): 525 | kinds = getKinds() 526 | 527 | # get proper preparation thread and call it 528 | threadClass = kinds[kind] 529 | thread = threadClass(options, inputfile=inputfile) 530 | thread.run() 531 | 532 | 533 | def getFoldersFromInputListFile(inputlist): 534 | ''' This method reads the given inputfile line-wise and returns the read lines without line breaks.''' 535 | 536 | file = open(inputlist, 'r') # open input file 537 | folders = file.read().splitlines() # read lines from file without line breaks 538 | file.close() # close file 539 | 540 | folders = filter(lambda f: not f.startswith("#"), folders) # remove commented lines 541 | folders = filter(os.path.isdir, folders) # remove all non-directories 542 | folders = map(os.path.normpath, folders) # normalize paths for easier transformations 543 | 544 | # TODO log removed folders 545 | 546 | return folders 547 | 548 | 549 | def applyFolders(kind, inputlist, options): 550 | kinds = getKinds() 551 | 552 | # get the list of projects/folders to process 553 | folders = getFoldersFromInputListFile(inputlist) 554 | 555 | # for each folder: 556 | for folder in folders: 557 | # start preparations for this single folder 558 | 559 | # get proper preparation thread and call it 560 | threadClass = kinds[kind] 561 | thread = threadClass(options, inputfolder=folder) 562 | thread.run() 563 | 564 | 565 | def applyFoldersAll(inputlist, options): 566 | kinds = getKinds() 567 | for kind in kinds.keys(): 568 | applyFolders(kind, inputlist, options) 569 | 570 | 571 | def main(): 572 | kinds = getKinds() 573 | 574 | # ################################################# 575 | # options parsing 576 | 577 | options = cli.getOptions(kinds, step=cli.steps.PREPARATION) 578 | 579 | # ################################################# 580 | # main 581 | 582 | if (options.inputfile): 583 | 584 | # split --file argument 585 | options.infile = os.path.normpath(os.path.abspath(options.inputfile[0])) # IN 586 | options.outfile = os.path.normpath(os.path.abspath(options.inputfile[1])) # OUT 587 | 588 | # check if inputfile exists 589 | if (not os.path.isfile(options.infile)): 590 | print "ERROR: input file '{}' cannot be found!".format(options.infile) 591 | sys.exit(1) 592 | 593 | applyFile(options.kind, options.infile, options) 594 | 595 | elif (options.inputlist): 596 | # handle --list argument 597 | options.inputlist = os.path.normpath(os.path.abspath(options.inputlist)) # LIST 598 | 599 | # check if list file exists 600 | if (not os.path.isfile(options.inputlist)): 601 | print "ERROR: input file '{}' cannot be found!".format(options.inputlist) 602 | sys.exit(1) 603 | 604 | if (options.allkinds): 605 | applyFoldersAll(options.inputlist, options) 606 | else: 607 | applyFolders(options.kind, options.inputlist, options) 608 | 609 | else: 610 | print "This should not happen! No input file or list of projects given!" 611 | sys.exit(1) 612 | 613 | 614 | if __name__ == '__main__': 615 | main() 616 | -------------------------------------------------------------------------------- /cppstats_input.txt: -------------------------------------------------------------------------------- 1 | /home/joliebig/workspace/cppstats/test/drivers_test 2 | -------------------------------------------------------------------------------- /lib/__init__.py: -------------------------------------------------------------------------------- 1 | # cppstats is a suite of analyses for measuring C preprocessor-based 2 | # variability in software product lines. 3 | # Copyright (C) 2014-2015 University of Passau, Germany 4 | # 5 | # This program is free software: you can redistribute it and/or modify it 6 | # under the terms of the GNU Lesser General Public License as published 7 | # by the Free Software Foundation, either version 3 of the License, or 8 | # (at your option) any later version. 9 | # 10 | # This program is distributed in the hope that it will be useful, 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # Lesser General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU Lesser General Public 16 | # License along with this program. If not, see 17 | # . 18 | # 19 | # Contributors: 20 | # Claus Hunsen 21 | -------------------------------------------------------------------------------- /lib/cpplib.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # cppstats is a suite of analyses for measuring C preprocessor-based 3 | # variability in software product lines. 4 | # Copyright (C) 2010-2015 University of Passau, Germany 5 | # 6 | # This program is free software: you can redistribute it and/or modify it 7 | # under the terms of the GNU Lesser General Public License as published 8 | # by the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # This program is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14 | # Lesser General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU Lesser General Public 17 | # License along with this program. If not, see 18 | # . 19 | # 20 | # Contributors: 21 | # Jörg Liebig 22 | # Claus Hunsen 23 | 24 | 25 | import sys 26 | 27 | # pyparsing module 28 | import pyparsing as pypa 29 | pypa.ParserElement.enablePackrat() # speed up parsing 30 | sys.setrecursionlimit(2000) # handle larger expressions 31 | 32 | # possible operands: 33 | # - hexadecimal number 34 | # - decimal number 35 | # - identifier 36 | # - macro function, which is basically expanded via #define 37 | # to an expression 38 | __string = \ 39 | pypa.Literal('\'').suppress() + \ 40 | pypa.Word(pypa.alphanums+'_\\') + \ 41 | pypa.Literal('\'').suppress() 42 | 43 | __hexadec = \ 44 | pypa.Literal('0x').suppress() + \ 45 | pypa.Word(pypa.hexnums).\ 46 | setParseAction(lambda t: str(int(t[0], 16))) + \ 47 | pypa.Optional(pypa.Suppress(pypa.Literal('L'))) 48 | 49 | __integer = \ 50 | pypa.Optional('~') + \ 51 | pypa.Word(pypa.nums+'-').setParseAction(lambda t: str(int(t[0]))) + \ 52 | pypa.Optional(pypa.Suppress(pypa.Literal('U'))) + \ 53 | pypa.Optional(pypa.Suppress(pypa.Literal('L'))) + \ 54 | pypa.Optional(pypa.Suppress(pypa.Literal('L'))) 55 | 56 | __identifier = \ 57 | pypa.Word(pypa.alphanums+'_'+'-') 58 | __arg = pypa.Word(pypa.alphanums+'_') 59 | __args = __arg + pypa.ZeroOrMore(pypa.Literal(',').suppress() + \ 60 | __arg) 61 | __fname = pypa.Word(pypa.alphas, pypa.alphanums + '_') 62 | __function = pypa.Group(__fname + pypa.Literal('(').suppress() + \ 63 | __args + pypa.Literal(')').suppress()) 64 | 65 | 66 | def _parseIfDefExpression(ifdefexp): 67 | """This function parses a given ifdef-expression and 68 | rewrites the expression according to the given __pt mapping. 69 | This one is used to make use of a csp solver without using 70 | a predicate.""" 71 | mal = list() 72 | 73 | def _rewriteOne(param): 74 | """This function returns each one parameter function 75 | representation for csp.""" 76 | op, ma = param[0] 77 | mal.append(ma) 78 | if op == '!': ret = op + '(' + ma + ')' 79 | if op == 'defined': ret = ma 80 | return ret 81 | 82 | def _rewriteTwo(param): 83 | """This function returns each two parameter function 84 | representation for csp.""" 85 | mal.extend(param[0][0::2]) 86 | ret = param[0][1] 87 | ret = '(' + ret.join(map(str, param[0][0::2])) + ')' 88 | return ret 89 | 90 | operand = __string | __hexadec | __integer | \ 91 | __function | __identifier 92 | operators = pypa.oneOf('&& ||') # extend with furhter operators 93 | expr = pypa.operatorPrecedence(operand, [ 94 | ('defined', 1, pypa.opAssoc.RIGHT, _rewriteOne), 95 | ('!', 1, pypa.opAssoc.RIGHT, _rewriteOne), 96 | (operators, 2, pypa.opAssoc.LEFT, _rewriteTwo), 97 | ]) + pypa.StringEnd() 98 | 99 | try: 100 | rsig = expr.parseString(ifdefexp)[0] 101 | except pypa.ParseException, e: 102 | print('ERROR (parse): cannot parse sig (%s) -- (%s)' % 103 | (ifdefexp, e.col)) 104 | return ifdefexp 105 | except RuntimeError: 106 | print('ERROR (time): cannot parse sig (%s)' % (ifdefexp)) 107 | return ifdefexp 108 | return (mal, ''.join(rsig)) 109 | 110 | 111 | __ifdefexplist = [] 112 | def _collectIfdefExpressions(fname): 113 | ''' 114 | This method filters all ifdef expressions out of a file and returns them as a list. 115 | ''' 116 | 117 | def _extractIfdefExpression(tokens): 118 | global __ifdefexplist 119 | __ifdefexplist += tokens 120 | 121 | __macro = pypa.Literal('#') \ 122 | + pypa.oneOf("if ifdef ifndef elif") \ 123 | + pypa.OneOrMore(pypa.Word(pypa.alphanums+"&|><=^")) \ 124 | .setParseAction(_extractIfdefExpression) \ 125 | + pypa.StringEnd() 126 | 127 | with open(fname, 'r') as fd: 128 | for line in fd.xreadlines(): 129 | try: 130 | print(__macro.parseString(line)) 131 | except pypa.ParseException: 132 | pass 133 | return __ifdefexplist 134 | 135 | def _filterAnnotatedIfdefs(fnamein, fnameout): 136 | ''' 137 | This method removes all preprocessor annotated lines from the input. 138 | ''' 139 | inifdef = 0 140 | 141 | with open(fnameout, 'w') as fdout: 142 | with open(fnamein, 'r') as fdin: 143 | for line in fdin.readlines(): 144 | # line is a preprocessor directive; determine weather its a conditional inclusion 145 | # directive or something else 146 | if line.startswith('#'): 147 | parseddirective = line.split(' ', 1) 148 | directive = parseddirective[0].strip() 149 | if directive in ['#if', '#ifdef', '#ifndef']: 150 | inifdef += 1 151 | elif directive in ['#else', '#elif']: 152 | pass 153 | elif directive == '#endif': 154 | inifdef -= 1 155 | elif directive in ['#line', '#error', '#pragma', '#define', '#undef', '#include', '#ident', '#warning', '#include_next']: 156 | if inifdef: 157 | fdout.write('\n') 158 | continue 159 | else: 160 | print("ERROR: directive (%s) not processed!" % parseddirective) 161 | fdout.write(line) 162 | # found regular C code 163 | else: 164 | fdout.write(line) 165 | 166 | 167 | ################################################## 168 | if __name__ == '__main__': 169 | symbols, expression = _parseIfDefExpression('AA && BB') 170 | print(symbols, expression) 171 | print(_collectIfdefExpressions('/home/joliebig/workspace/reverse_cpp/test/test.c')) 172 | _filterAnnotatedIfdefs('/home/joliebig/workspace/reverse_cpp/test/filterannotateddirectives.h', 173 | '/home/joliebig/workspace/reverse_cpp/test/filterannotateddirectives_out.h') 174 | -------------------------------------------------------------------------------- /preparations/__init__.py: -------------------------------------------------------------------------------- 1 | # cppstats is a suite of analyses for measuring C preprocessor-based 2 | # variability in software product lines. 3 | # Copyright (C) 2014-2015 University of Passau, Germany 4 | # 5 | # This program is free software: you can redistribute it and/or modify it 6 | # under the terms of the GNU Lesser General Public License as published 7 | # by the Free Software Foundation, either version 3 of the License, or 8 | # (at your option) any later version. 9 | # 10 | # This program is distributed in the hope that it will be useful, 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # Lesser General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU Lesser General Public 16 | # License along with this program. If not, see 17 | # . 18 | # 19 | # Contributors: 20 | # Claus Hunsen 21 | -------------------------------------------------------------------------------- /preparations/deleteComments.xsl: -------------------------------------------------------------------------------- 1 | 24 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | -------------------------------------------------------------------------------- /preparations/deleteIncludeGuards.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2011-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Jörg Liebig 23 | # Claus Hunsen 24 | 25 | 26 | # this script checks include guards of c, cpp - header files 27 | # and removes them: 28 | # e.g.: test.h 29 | # #if !defined(foo) 30 | # #define foo ... 31 | # #endif // foo 32 | # 33 | # parser works with following characteristics: 34 | # - the include guard doesn't have to start at the first line 35 | # and doesn't have to finish at the last line 36 | # - only one guard per file 37 | # - deleting first include-guard of following syntax 38 | # - include is formated like: 39 | # line x : #if !defined(foo) 40 | # line x+1: #define foo 41 | # line x+2: ... 42 | # line x+y: #endif 43 | 44 | import os, re, sys 45 | 46 | __debug = False 47 | 48 | 49 | def apply(fname, out=sys.stdout): 50 | fname = os.path.abspath(fname) 51 | 52 | # check for include-guard 53 | rg = re.compile('#if\s+!defined\((\S+)\)') 54 | rd = re.compile('#define\s+(\S+)') 55 | sourcecode = list() 56 | 57 | with open(fname, 'r') as fd: 58 | for line in fd.readlines(): 59 | sourcecode.append(line) 60 | 61 | def _findCorrespondingItems(sourcecode): 62 | '''This method returns a tuple with the include guard elements to cut of the source. 63 | return of (-1, -1), means no proper include guard found. 64 | the following rules apply: 65 | - include guard has to be the first one of the occuring #ifdefs 66 | - no alternatives, i.e. occuring #else or #elif, allowed 67 | ''' 68 | ifdefpos = -1 69 | ifdef = '' 70 | sifdef = '' # stripped #if 71 | currentitem = -1 72 | guardname = '' 73 | taillist = list() 74 | 75 | # processing ifdefs 76 | for item in sourcecode: 77 | sitem = item.strip() 78 | currentitem += 1 79 | 80 | # line is empty (except for whitespace, probably) 81 | if not sitem: 82 | continue 83 | 84 | if sitem.startswith('#if'): 85 | ifdef = item 86 | sifdef = sitem 87 | ifdefpos = currentitem 88 | taillist = list(sourcecode[currentitem:]) 89 | break # search for #if 90 | 91 | # end of code reached and nothing found so far 92 | if currentitem == len(sourcecode): 93 | # no include guard found 94 | return (-1, -1) 95 | 96 | # processing ifdef and define 97 | regres = rg.match(sifdef) 98 | if (regres): 99 | guardname = regres.groups()[0] 100 | else: 101 | return (-1, -1) 102 | 103 | define = taillist[1] 104 | regres = rd.match(define.strip()) 105 | if (regres): 106 | if guardname != regres.groups()[0]: 107 | return (-1, -1) 108 | else: 109 | return (-1, -1) 110 | 111 | # process taillist for else and endif 112 | ifcount = 1 113 | currentitem = 1 # go two steps ahead as we jump over #if and #define 114 | for item in taillist[2:]: 115 | currentitem += 1 116 | if item.startswith('#else') and ifcount == 1: 117 | return (-1, -1) # we do not support alternative include guards 118 | if item.startswith('#elif') and ifcount == 1: 119 | return (-1, -1) # we do not support alternative include guards 120 | if item.startswith('#if'): 121 | ifcount += 1 122 | continue 123 | if item.startswith('#endif'): 124 | ifcount -= 1 125 | if ifcount == 0: 126 | return (ifdefpos, currentitem + ifdefpos) 127 | continue 128 | return (-1, -1) 129 | 130 | (ifdef, endif) = _findCorrespondingItems(list(sourcecode)) 131 | if (ifdef == -1 or endif == -1): 132 | pass 133 | else: 134 | # concat source code again and replace include guard with empty lines 135 | sourcecode = sourcecode[:max(0, ifdef)] + \ 136 | ["", ""] + \ 137 | sourcecode[ifdef + 2:endif] + \ 138 | [""] + \ 139 | sourcecode[endif + 1:] 140 | 141 | for item in sourcecode: 142 | out.write(item.rstrip('\n') + '\n') 143 | 144 | 145 | def usage(): 146 | print(sys.argv[0] + ' filename') 147 | print('programm writes results to stdout') 148 | 149 | 150 | ################################################## 151 | if __name__ == '__main__': 152 | if len(sys.argv) < 2: 153 | usage() 154 | else: 155 | apply(sys.argv[1]) 156 | -------------------------------------------------------------------------------- /preparations/rewriteIfdefs.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2011-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Jörg Liebig 23 | # Claus Hunsen 24 | 25 | import sys, os 26 | 27 | class WrongIfdefError(Exception): 28 | def __init__(self): 29 | pass 30 | def __str__(self): 31 | return ("Didn't find \"ifdef\" or \"ifndef\" as macro") 32 | 33 | def rewriteFile(fname, out = sys.stdout): 34 | fd = open(fname, 'r') 35 | 36 | for line in fd: 37 | if line.startswith('#ifdef') or line.startswith('#ifndef'): 38 | ifdef, identifier = line.split(None, 1) # FIXME if there is a comment after the constant, it is incorporated into the brackets! this may lead to errors. 39 | identifier = identifier.strip() 40 | 41 | if ifdef == '#ifdef': 42 | out.write('#if defined(' + identifier + ')' + '\n') 43 | continue 44 | if ifdef == '#ifndef': 45 | out.write('#if !defined(' + identifier + ')' + '\n') 46 | continue 47 | raise WrongIfdefError() 48 | else: 49 | out.write(line) 50 | 51 | fd.close() 52 | 53 | 54 | ################################################## 55 | if __name__ == '__main__': 56 | if (len(sys.argv) != 2): 57 | print("usage: " + sys.argv[0] + " ") 58 | else: 59 | rewriteFile(sys.argv[1]) 60 | -------------------------------------------------------------------------------- /preparations/rewriteMultilineMacros.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2011-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distribin the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Jörg Liebig 23 | # Claus Hunsen 24 | 25 | # this script is made for changing macros that span over 26 | # multiple lines (examples see below) to one line for 27 | # better parsing 28 | # e.g.: 29 | # #define FillGroup1 \ 30 | # "mov r11,r2 \n\t" \ 31 | # "mov r10,r2 \n\t" \ ... 32 | # 33 | # effects all kind of macros #defines, conditionals, ... 34 | 35 | import os, sys 36 | 37 | # translates macros (s.o. or usage) 38 | def translate(infile, outfile): 39 | fdin = open(infile) 40 | fdout = open(outfile, 'w') 41 | curline = '' 42 | numlines = 0 43 | 44 | for line in fdin: 45 | sline = line.strip() 46 | 47 | # macro found 48 | if len(curline): 49 | # multi-line macro 50 | if sline.endswith('\\'): 51 | curline += sline[:-1] 52 | numlines += 1 53 | else: 54 | curline += sline 55 | #TODO fix line endings 56 | fdout.write(curline+'\n') 57 | fdout.write('\n' * numlines) 58 | curline = '' 59 | numlines = 0 60 | 61 | continue 62 | 63 | # found a new macro 64 | if (sline.startswith('#') and sline.endswith('\\')): 65 | curline += sline[:-1] 66 | numlines += 1 67 | continue 68 | 69 | # normal line 70 | fdout.write(line) 71 | 72 | # closeup 73 | fdin.close() 74 | fdout.close() 75 | 76 | 77 | # usage 78 | def usage(): 79 | print('usage: ' + sys.argv[0] + ' ') 80 | print('') 81 | print('Translates multiple macros in the source-code of the infile') 82 | print('to a oneliner-macro in the outfile.') 83 | 84 | ################################################## 85 | if __name__ == '__main__': 86 | 87 | if (len(sys.argv) < 3): 88 | usage() 89 | sys.exit(-1) 90 | 91 | infile = os.path.abspath(sys.argv[1]) 92 | outfile = os.path.abspath(sys.argv[2]) 93 | 94 | translate(infile, outfile) 95 | 96 | -------------------------------------------------------------------------------- /scripts/__init__.py: -------------------------------------------------------------------------------- 1 | # cppstats is a suite of analyses for measuring C preprocessor-based 2 | # variability in software product lines. 3 | # Copyright (C) 2014-2015 University of Passau, Germany 4 | # 5 | # This program is free software: you can redistribute it and/or modify it 6 | # under the terms of the GNU Lesser General Public License as published 7 | # by the Free Software Foundation, either version 3 of the License, or 8 | # (at your option) any later version. 9 | # 10 | # This program is distributed in the hope that it will be useful, 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 13 | # Lesser General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU Lesser General Public 16 | # License along with this program. If not, see 17 | # . 18 | # 19 | # Contributors: 20 | # Claus Hunsen 21 | -------------------------------------------------------------------------------- /scripts/ascope.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2011-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Jörg Liebig 23 | # Claus Hunsen 24 | 25 | 26 | # modules from the std-library 27 | import os 28 | import re 29 | import sys 30 | from optparse import OptionParser 31 | 32 | # external libs 33 | # python-lxml module 34 | try: 35 | from lxml import etree 36 | except ImportError: 37 | print("python-lxml module not found! (python-lxml)") 38 | print("see http://codespeak.net/lxml/") 39 | print("programm terminating ...!") 40 | sys.exit(-1) 41 | 42 | 43 | def returnFileNames(folder, extfilt = ['.xml']): 44 | '''This function returns all files of the input folder 45 | and its subfolders.''' 46 | filesfound = list() 47 | 48 | if os.path.isdir(folder): 49 | wqueue = [os.path.abspath(folder)] 50 | 51 | while wqueue: 52 | currentfolder = wqueue[0] 53 | wqueue = wqueue[1:] 54 | foldercontent = os.listdir(currentfolder) 55 | tmpfiles = filter(lambda n: os.path.isfile( 56 | os.path.join(currentfolder, n)), foldercontent) 57 | tmpfiles = filter(lambda n: os.path.splitext(n)[1] in extfilt, 58 | tmpfiles) 59 | tmpfiles = map(lambda n: os.path.join(currentfolder, n), 60 | tmpfiles) 61 | filesfound += tmpfiles 62 | tmpfolders = filter(lambda n: os.path.isdir( 63 | os.path.join(currentfolder, n)), foldercontent) 64 | tmpfolders = map(lambda n: os.path.join(currentfolder, n), 65 | tmpfolders) 66 | wqueue += tmpfolders 67 | 68 | return filesfound 69 | 70 | 71 | class Ascope: 72 | ################################################## 73 | # constants: 74 | __cppnscpp = 'http://www.srcML.org/srcML/cpp' 75 | __cppnsdef = 'http://www.srcML.org/srcML/src' 76 | __cpprens = re.compile('{(.+)}(.+)') 77 | __conditionals = ['if', 'ifdef', 'ifndef', 'else', 'elif', 'endif'] 78 | __conditions = ['if', 'ifdef', 'ifndef'] 79 | __screensize = 50 80 | __depthannotation = 60 81 | ################################################## 82 | 83 | def __init__(self): 84 | oparser = OptionParser() 85 | oparser.add_option('-d', '--dir', dest='dir', 86 | help='input directory (mandatory)') 87 | (self.opts, self.args) = oparser.parse_args() 88 | 89 | if not self.opts.dir: 90 | oparser.print_help() 91 | sys.exit(-1) 92 | 93 | self.loc=0 94 | self.checkFiles() 95 | 96 | def __getIfdefAnnotations__(self, root): 97 | '''This method returns all nodes of the xml which are ifdef 98 | annotations in the source code.''' 99 | treeifdefs = list() 100 | 101 | for _, elem in etree.iterwalk(root): 102 | ns, tag = Ascope.__cpprens.match(elem.tag).\ 103 | groups() 104 | 105 | if ns == Ascope.__cppnscpp \ 106 | and tag in Ascope.__conditionals: 107 | treeifdefs.append(elem) 108 | 109 | return treeifdefs 110 | 111 | def __createListFromTreeifdefs__(self, treeifdefs): 112 | '''This method returns a list representation for the input treeifdefs 113 | (xml-objects). Corresponding #ifdef elements are in one sublist.''' 114 | try: 115 | if not treeifdefs: return [] 116 | 117 | listifdefs = list() 118 | workerlist = list() 119 | for nifdef in treeifdefs: 120 | tag = nifdef.tag.split('}')[1] 121 | if tag in ['if', 'ifdef', 'ifndef']: 122 | workerlist.append(list()) 123 | workerlist[-1].append(nifdef) 124 | elif tag in ['elif', 'else']: 125 | workerlist[-1].append(nifdef) 126 | elif tag in ['endif']: 127 | workerlist[-1].append(nifdef) 128 | listifdefs.append(workerlist[-1]) 129 | workerlist = workerlist[:-1] 130 | else: 131 | print('ERROR: tag (%s) unknown!' % tag) 132 | 133 | return listifdefs 134 | except IndexError: 135 | return [] 136 | 137 | def __getParentTag__(self, tag): 138 | parent = tag.getparent() 139 | return parent.tag.split('}')[1] 140 | 141 | 142 | def __checkDiscipline__(self, treeifdefs, loc, stats, statsU): 143 | listundisciplined = self.__createListFromTreeifdefs__(treeifdefs) 144 | # print('INFO: %s annotations to check' % len(listundisciplined)) 145 | 146 | allannotations=[] 147 | for ifdef in listundisciplined: 148 | for i in range(len(ifdef)-1): 149 | allannotations.append([ifdef[i].sourceline,ifdef[i+1].sourceline,self.__findFeatures__(ifdef,i)]); 150 | 151 | for screen in range(0, max(1,min(65000,loc-Ascope.__screensize/2)), Ascope.__screensize/2): 152 | screenend=min(loc, screen+Ascope.__screensize) 153 | annotationsOnScreen=set() 154 | annotationsOnScreenCount=0 155 | for annotation in allannotations: 156 | if annotation[0]<=screenend: 157 | if annotation[1]>screen: 158 | annotationsOnScreen.add(annotation[2]) 159 | annotationsOnScreenCount=annotationsOnScreenCount+1 160 | try: 161 | stats[annotationsOnScreenCount]=stats[annotationsOnScreenCount]+1 162 | statsU[len(annotationsOnScreen)]=statsU[len(annotationsOnScreen)]+1 163 | except IndexError: 164 | print(annotationsOnScreenCount) 165 | sys.exit(-1) 166 | 167 | # print(stats) 168 | # print(statsU) 169 | 170 | def __findFeatures__(self, ifdef, idx): 171 | result="" 172 | if ifdef[idx].tag.split('}')[1]=='else': 173 | idx=0 174 | result="!" 175 | if ifdef[idx].tag.split('}')[1]=='ifndef': 176 | if (result=="!"): 177 | result="" 178 | else: 179 | result="!" 180 | 181 | context = etree.iterwalk(ifdef[idx]) 182 | for action, elem in context: 183 | if action=="end": 184 | if elem.tag.split('}')[1]=="name": 185 | result=result+elem.text 186 | # print result; 187 | return result 188 | 189 | def checkFile(self, file, stats, statsU): 190 | # print('INFO: processing (%s)' % file) 191 | 192 | try: 193 | tree = etree.parse(file) 194 | 195 | f = open(file, 'r') 196 | except etree.XMLSyntaxError: 197 | print('ERROR: file (%s) is not valid. Skipping it.' % file) 198 | return 199 | 200 | #get LOC 201 | thisloc=len(f.readlines())-2 202 | if (thisloc > 65000): 203 | print('INFO: file (%s) not fully processed!' % file) 204 | 205 | # get root of the xml and iterate over it 206 | root = tree.getroot() 207 | treeifdefs = self.__getIfdefAnnotations__(root) 208 | self.__checkDiscipline__(treeifdefs, thisloc, stats, statsU) 209 | 210 | 211 | def checkFiles(self): 212 | xmlfiles = returnFileNames(self.opts.dir, ['.xml']) 213 | stats=[0]*Ascope.__depthannotation 214 | statsU=[0]*Ascope.__depthannotation 215 | for xmlfile in xmlfiles: 216 | self.checkFile(xmlfile, stats, statsU) 217 | f = open("count.csv","a") 218 | f.write(self.opts.dir+";"+str(Ascope.__screensize)+";") 219 | for i in stats: 220 | f.write(str(i)+";") 221 | for i in statsU: 222 | f.write(str(i)+";") 223 | f.write("\n") 224 | 225 | 226 | ################################################## 227 | if __name__ == '__main__': 228 | Ascope() 229 | -------------------------------------------------------------------------------- /scripts/convert2xml.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # cppstats is a suite of analyses for measuring C preprocessor-based 3 | # variability in software product lines. 4 | # Copyright (C) 2011-2015 University of Passau, Germany 5 | # 6 | # This program is free software: you can redistribute it and/or modify it 7 | # under the terms of the GNU Lesser General Public License as published 8 | # by the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # This program is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14 | # Lesser General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU Lesser General Public 17 | # License along with this program. If not, see 18 | # . 19 | # 20 | # Contributors: 21 | # Jörg Liebig 22 | # Claus Hunsen 23 | 24 | 25 | prog=/home/joliebig/workspace/reverse_cpp/src/src2srcml2009 26 | progargs='--language=C ' 27 | 28 | while read dir; do 29 | cd ${dir} 30 | echo $PWD 31 | rm *.xml 32 | for i in `ls *.c`; 33 | do 34 | ${prog} ${progargs} ${i} ${i}.xml 35 | done 36 | for i in `ls *.h`; 37 | do 38 | ${prog} ${progargs} ${i} ${i}.xml 39 | done 40 | cd ../.. 41 | done < ./projects.txt 42 | -------------------------------------------------------------------------------- /scripts/disable_errorneous_files.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # cppstats is a suite of analyses for measuring C preprocessor-based 3 | # variability in software product lines. 4 | # Copyright (C) 2011-2015 University of Passau, Germany 5 | # 6 | # This program is free software: you can redistribute it and/or modify it 7 | # under the terms of the GNU Lesser General Public License as published 8 | # by the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # This program is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14 | # Lesser General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU Lesser General Public 17 | # License along with this program. If not, see 18 | # . 19 | # 20 | # Contributors: 21 | # Jörg Liebig 22 | # Claus Hunsen 23 | 24 | 25 | INPUTFILE="cppstats_errorneous_files.txt" 26 | 27 | while read file; 28 | do 29 | basefile=`dirname ${file}`/`basename ${file} .xml` 30 | if [ ! -e "${basefile}" ]; then 31 | echo "ERROR: basefile ($basefile)" not available! 32 | continue 33 | fi 34 | if [ -e "${file}.disabled" ]; then 35 | echo "INFO: file ($file) already disabled" 36 | continue 37 | fi 38 | if [ -e "${file}" ]; then 39 | echo "INFO: moving file ($file)" 40 | mv ${file} ${file}.disabled 41 | else 42 | echo "INFO: file ($file) not available" 43 | fi 44 | done < ${INPUTFILE} 45 | -------------------------------------------------------------------------------- /scripts/ifdefendifratio.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2009-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Jörg Liebig 23 | 24 | 25 | # This tool determines, whether the ration of conditionals (ifdef) to 26 | # endif-macros holds. The reason for this check is basically the effect, 27 | # when running the pxml.py against a xml-file and an error occurs because 28 | # the lists in _getFeatures are empty. 29 | 30 | 31 | # modules from the std-library 32 | import os, re, sys 33 | 34 | try: 35 | from lxml import etree 36 | except ImportError: 37 | print("python-lxml module not found! (python-lxml)") 38 | print("see http://codespeak.net/lxml/") 39 | print("programm terminating ...!") 40 | sys.exit(-1) 41 | 42 | 43 | ################################################## 44 | # constants: 45 | # namespace-constant for src2srcml 46 | __cppnscpp = 'http://www.srcML.org/srcML/cpp' 47 | __cppnsdef = 'http://www.srcML.org/srcML/src' 48 | __cpprens = re.compile('{(.+)}(.+)') 49 | 50 | __conditionals = ['if', 'ifdef', 'ifndef'] 51 | __conditionals_endif = ['endif'] 52 | ################################################## 53 | 54 | def _getIfdefEndifRatio(root): 55 | """This function determines all conditionals and their corresponding 56 | endifs and returns a counter for each of them.""" 57 | ifdef = 0 58 | endif = 0 59 | 60 | # get all nodes 61 | allnodes = [node for node in root.iterdescendants()] 62 | 63 | for node in allnodes: 64 | ns, tag = __cpprens.match(node.tag).groups() 65 | 66 | if ((tag in __conditionals) \ 67 | and (ns == __cppnscpp)): 68 | ifdef += 1 69 | if ((tag in __conditionals_endif) \ 70 | and (ns == __cppnscpp)): 71 | endif += 1 72 | 73 | return (ifdef, endif) 74 | 75 | 76 | def apply(folder): 77 | """This function applies the determination function (getIfdefEndifRation) 78 | to each file and prints out the differance in case there is one.""" 79 | folder = os.path.abspath(folder) 80 | files = os.listdir(folder) 81 | files = filter(lambda n: os.path.splitext(n)[1] == ".xml", files) 82 | 83 | for file in files: 84 | 85 | try: 86 | tree = etree.parse(file) 87 | except etree.XMLSyntaxError: 88 | print("ERROR: cannot parse (%s). Skipping this file!." % file) 89 | 90 | root = tree.getroot() 91 | ifdef, endif = _getIfdefEndifRatio(root) 92 | 93 | if (ifdef != endif): 94 | print("INFO: (%30s) ifdef : endif is %5s : %5s" % (file, str(ifdef), str(endif))) 95 | 96 | 97 | def usage(): 98 | """This function prints usage-informations to stdout.""" 99 | print('usage:') 100 | print(' ' + sys.argv[0] + ' ') 101 | 102 | 103 | ################################################## 104 | if __name__ == '__main__': 105 | 106 | if (len(sys.argv) < 2): 107 | usage() 108 | sys.exit(-1) 109 | 110 | folder = sys.argv[1] 111 | if (os.path.isdir(folder)): 112 | apply(folder) 113 | else: 114 | usage() 115 | sys.exit(-1) 116 | -------------------------------------------------------------------------------- /scripts/partial_preprocessor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # cppstats is a suite of analyses for measuring C preprocessor-based 4 | # variability in software product lines. 5 | # Copyright (C) 2010-2015 University of Passau, Germany 6 | # 7 | # This program is free software: you can redistribute it and/or modify it 8 | # under the terms of the GNU Lesser General Public License as published 9 | # by the Free Software Foundation, either version 3 of the License, or 10 | # (at your option) any later version. 11 | # 12 | # This program is distributed in the hope that it will be useful, 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 15 | # Lesser General Public License for more details. 16 | # 17 | # You should have received a copy of the GNU Lesser General Public 18 | # License along with this program. If not, see 19 | # . 20 | # 21 | # Contributors: 22 | # Jörg Liebig 23 | # Claus Hunsen 24 | 25 | 26 | #FIXME move this script to another folder! 27 | from optparse import OptionParser 28 | import sys 29 | 30 | from lib.cpplib import _filterAnnotatedIfdefs 31 | 32 | 33 | class PartialPreprocessor: 34 | def __init__(self): 35 | oparser = OptionParser() 36 | oparser.add_option('-i', '--inputfile', dest='ifile', 37 | help='input file (mandatory)') 38 | oparser.add_option('-o', '--outputfile', dest='ofile', 39 | help='output file (mandatory)') 40 | (self.opts, self.args) = oparser.parse_args() 41 | 42 | if not self.opts.ifile or not self.opts.ofile: 43 | oparser.print_help() 44 | sys.exit(-1) 45 | 46 | _filterAnnotatedIfdefs(self.opts.ifile, self.opts.ofile) 47 | 48 | 49 | ################################################## 50 | if __name__ == '__main__': 51 | PartialPreprocessor() 52 | -------------------------------------------------------------------------------- /scripts/reversecpp.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # cppstats is a suite of analyses for measuring C preprocessor-based 3 | # variability in software product lines. 4 | # Copyright (C) 2011 University of Passau, Germany 5 | # 6 | # This program is free software: you can redistribute it and/or modify it 7 | # under the terms of the GNU Lesser General Public License as published 8 | # by the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # This program is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14 | # Lesser General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU Lesser General Public 17 | # License along with this program. If not, see 18 | # . 19 | # 20 | # Contributors: 21 | # Jörg Liebig 22 | 23 | 24 | ''' 25 | this collection of functions is dedicated to discipline undisciplined 26 | preprocessor annotations in the source code. the annotations covered are 27 | those of conditional compilation (#if, #ifdef, #ifndef, #elif, #else, and 28 | #endif). 29 | 30 | basic strategy is: 31 | 1. adding line markers to the orginial source code file; these markers are 32 | necessary in the following steps for moving undisciplined annotations and 33 | merging different source code files 34 | 2. extract all configuration paramters from #ifdef expressions and generate 35 | all possible variants 36 | 3. create an xml representation for each of the variants 37 | 4. based on the line markers move undisciplined annotations to disciplined ones 38 | in each of the variants 39 | 5. merge the different variants and create a single source code file 40 | ''' 41 | 42 | import itertools 43 | import os 44 | import subprocess 45 | import sys 46 | import tempfile 47 | from cpplib import _collectIfdefExpressions, _parseIfDefExpression 48 | from optparse import OptionParser 49 | 50 | ############################################################ 51 | # config 52 | cpptool = '/usr/bin/cpp' 53 | tmpfolder = './tmp_reversecpp/' 54 | src2srcml = os.path.join(os.path.expanduser('~'), 'bin', 'src2srcml2009') 55 | src2srcmloptions = ['--language', 'C'] 56 | srcml2src = os.path.join(os.path.expanduser('~'), 'bin', 'srcml2src2009') 57 | ############################################################ 58 | 59 | class Util: 60 | @classmethod 61 | def returnFileNames(folder, extfilt = ['.xml']): 62 | ''' 63 | This function returns all files of the input folder 64 | and its subfolders. 65 | ''' 66 | filesfound = list() 67 | 68 | if os.path.isdir(folder): 69 | wqueue = [os.path.abspath(folder)] 70 | 71 | while wqueue: 72 | currentfolder = wqueue[0] 73 | wqueue = wqueue[1:] 74 | foldercontent = os.listdir(currentfolder) 75 | tmpfiles = filter(lambda n: os.path.isfile( 76 | os.path.join(currentfolder, n)), foldercontent) 77 | tmpfiles = filter(lambda n: os.path.splitext(n)[1] in extfilt, 78 | tmpfiles) 79 | tmpfiles = map(lambda n: os.path.join(currentfolder, n), 80 | tmpfiles) 81 | filesfound += tmpfiles 82 | tmpfolders = filter(lambda n: os.path.isdir( 83 | os.path.join(currentfolder, n)), foldercontent) 84 | tmpfolders = map(lambda n: os.path.join(currentfolder, n), 85 | tmpfolders) 86 | wqueue += tmpfolders 87 | 88 | return filesfound 89 | 90 | 91 | class ReverseCPP: 92 | 93 | def __init__(self): 94 | oparser = OptionParser() 95 | oparser.add_option("--ifolder", dest="ifolder", 96 | help="input folder") 97 | oparser.add_option("--ofolder", dest="ofolder", 98 | help="output folder") 99 | oparser.add_option("--debug", dest="debug", 100 | help="print out debug information") 101 | self.opts, self.args = oparser.parse_args() 102 | 103 | def setup(self): 104 | if not os.path.exists(tmpfolder): 105 | if self.opts.debug: 106 | print('INFO: tmpfolder (%s) does not exist; creating it' % tmpfolder) 107 | os.mkdir(tmpfolder) 108 | if not os.path.exists(src2srcml): 109 | print('ERROR: src2srcml tool is not available under path (%s)' % src2srcml) 110 | print('ERROR: program terminating ...!') 111 | sys.exit(-1) 112 | if not os.path.exists(srcml2src): 113 | print('ERROR: srcml2src tool is not available under path (%s)' % srcml2src) 114 | print('ERROR: program terminating ...!') 115 | sys.exit(-1) 116 | 117 | def addingLineMarkersToFile(self, infile, outfile): 118 | ''' 119 | This method adds line markers (comments) to the source code file (infile) and 120 | writes the result to the output file (outfile). Three different markers are added: 121 | 1. a comment containing the conditional compilation macro is added before each 122 | macro 123 | 2. each line gets a comment with the lineid at the end 124 | 3. include macros are turned into comments so that that preprocessor omits file 125 | inclusion during the preprocessing step 126 | ''' 127 | fdin = open(os.path.abspath(infile), 'r') 128 | fdout = open(os.path.abspath(outfile), 'w') 129 | lineid = 0 130 | 131 | for line in fdin.xreadlines(): 132 | # found #if{ndef|def||} or #e{ndif|lse|lif} 133 | if line.startswith('#if') or line.startswith('#e'): 134 | fdout.write('//'+ line + str(lineid)) 135 | lineid += 1 136 | if line.startswith('#include'): 137 | fdout.write('//'+ line + str(lineid)) 138 | lineid += 1 139 | continue 140 | fdout.write(line.strip() + '/* lineid=' + str(lineid) + ' */\n') 141 | lineid += 1 142 | 143 | print('processed file ' + os.path.abspath(infile)) 144 | 145 | def createVariants(self, symbols, fname): 146 | ''' 147 | Generate for each combination of symbols a variant for the inputfile fname 148 | and return the list of generated files. 149 | ''' 150 | generatedfiles = [] 151 | for configuration in itertools.product(range(2), repeat=len(symbols)): 152 | configuration = list(configuration) 153 | pairs = zip(configuration, symbols) 154 | validpairs = filter(lambda (m, n): m != 0, pairs) 155 | 156 | if len(validpairs): validdefines = list(zip(*validpairs)[1]) 157 | else: validdefines = [] 158 | 159 | validdefines = map(lambda n: '-D'+n, validdefines) 160 | cppinvocation = [cpptool] 161 | cppinvocation += validdefines 162 | cppinvocation += [fname] 163 | 164 | # create call-string and generate a variant 165 | extension = os.path.splitext(fname)[1] 166 | tmpfile = tempfile.NamedTemporaryFile(suffix=extension, dir=tmpfolder, delete=False) 167 | cppinvocation += ['-C'] 168 | cppinvocation += [tmpfile.name] 169 | print(cppinvocation) 170 | subprocess.call(cppinvocation) 171 | generatedfiles.append(tmpfile.name) 172 | return generatedfiles 173 | 174 | def createXMLRepresentation(self, fname): 175 | ''' 176 | This method creates an xml representation from the input file using the src2srcml 177 | tool (http://www.srcML.org/). After the successful generation of the 178 | xml representation the method returns the filename of the xml file. 179 | ''' 180 | src2srcmlinvocation = [src2srcml] 181 | src2srcmlinvocation += src2srcmloptions 182 | src2srcmlinvocation += [fname] # input file 183 | src2srcmlinvocation += [fname+'.xml'] # output file 184 | print(src2srcmlinvocation) 185 | subprocess.call(src2srcmlinvocation) 186 | return fname+'.xml' 187 | 188 | def createXMLRepresenations(self, flist): 189 | ''' 190 | This method creates an xml representation of each file in the input list (flist) 191 | and returns a list of the generated xml files. 192 | ''' 193 | generatedlist = [] 194 | for file in flist: 195 | generatedlist.append(self.createXMLRepresentation(file)) 196 | return generatedlist 197 | 198 | 199 | def apply(self): 200 | self.setup() 201 | symbols, _ = _parseIfDefExpression('AA && BB') 202 | flist = self.createVariants(symbols, '/home/joliebig/workspace/reverse_cpp/test/test.c') 203 | flist = self.createXMLRepresenations(flist) 204 | print(flist) 205 | print(_collectIfdefExpressions('/home/joliebig/workspace/reverse_cpp/test/test.c')) 206 | 207 | ################################################## 208 | if __name__ == '__main__': 209 | r = ReverseCPP() 210 | r.apply() 211 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # cppstats is a suite of analyses for measuring C preprocessor-based 3 | # variability in software product lines. 4 | # Copyright (C) 2015 University of Passau, Germany 5 | # 6 | # This program is free software: you can redistribute it and/or modify it 7 | # under the terms of the GNU Lesser General Public License as published 8 | # by the Free Software Foundation, either version 3 of the License, or 9 | # (at your option) any later version. 10 | # 11 | # This program is distributed in the hope that it will be useful, 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 14 | # Lesser General Public License for more details. 15 | # 16 | # You should have received a copy of the GNU Lesser General Public 17 | # License along with this program. If not, see 18 | # . 19 | # 20 | # Contributors: 21 | # Claus Hunsen 22 | # Andreas Ringlstetter 23 | 24 | 25 | from setuptools import setup, find_packages 26 | 27 | setup( 28 | name='cppstats', 29 | version="0.9.4", 30 | packages=find_packages(exclude=['scripts']), 31 | url='http://www.fosd.net/cppstats', 32 | license='LGPLv3', 33 | author='Claus Hunsen', 34 | author_email='hunsen@fim.uni-passau.de', 35 | description='toolsuite for analyzing preprocessor-based software product lines', 36 | 37 | package_data={ 38 | 'scripts' : ['*.sh'], 39 | 'preparations' : ['*.xsl'] 40 | }, 41 | 42 | install_requires=[ 43 | 'statlib==1.2', 44 | 'pyparsing==2.*', 45 | 'enum34', 46 | 'lxml>=3.4' 47 | ], 48 | 49 | dependency_links=[ 50 | 'https://github.com/clhunsen/python-statlib/archive/release-1.2.tar.gz#egg=statlib-1.2' 51 | ], 52 | 53 | entry_points={'console_scripts': [ 54 | 'cppstats = cppstats.cppstats:main', 55 | 'cppstats.analysis = cppstats.analysis:main', 56 | 'cppstats.preparation = cppstats.preparation:main' 57 | ]} 58 | ) 59 | --------------------------------------------------------------------------------