├── .github └── workflows │ └── build.yml ├── .gitignore ├── COPYING ├── DEVELOPERS.md ├── ENVIRONMENTS.md ├── INSTALL.md ├── README.md ├── bin ├── .pylintrc ├── jens-config ├── jens-gc ├── jens-gitlab-producer-runner ├── jens-purge-queue ├── jens-reset ├── jens-stats └── jens-update ├── conf └── main.conf ├── examples ├── environments │ └── production.yaml ├── example-repositories │ ├── common-hieradata │ │ └── data │ │ │ ├── common.yaml │ │ │ ├── environments │ │ │ ├── production.yaml │ │ │ └── qa.yaml │ │ │ ├── hardware │ │ │ └── vendor │ │ │ │ └── sinclair.yaml │ │ │ └── operatingsystems │ │ │ └── RedHat │ │ │ ├── 5.yaml │ │ │ ├── 6.yaml │ │ │ └── 7.yaml │ ├── hostgroup-myapp │ │ ├── code │ │ │ ├── files │ │ │ │ └── superscript.sh │ │ │ ├── manifests │ │ │ │ ├── frontend.pp │ │ │ │ └── init.pp │ │ │ └── templates │ │ │ │ └── someotherconfig.erb │ │ └── data │ │ │ ├── fqdns │ │ │ └── myapp-node1.example.org.yaml │ │ │ └── hostgroup │ │ │ ├── myapp.yaml │ │ │ └── myapp │ │ │ └── frontend.yaml │ ├── module-dummy │ │ ├── code │ │ │ ├── README.md │ │ │ ├── manifests │ │ │ │ ├── init.pp │ │ │ │ └── install.pp │ │ │ └── templates │ │ │ │ └── config.erb │ │ └── data │ │ │ └── jens.yaml │ └── site │ │ └── code │ │ └── site.pp ├── hiera.yaml └── repositories │ └── repositories.yaml ├── jens-tmpfiles.conf ├── jens ├── __init__.py ├── configfile.py ├── decorators.py ├── environments.py ├── errors.py ├── git_wrapper.py ├── locks.py ├── maintenance.py ├── messaging.py ├── repos.py ├── reposinventory.py ├── settings.py ├── test │ ├── __init__.py │ ├── test_git_wrapper.py │ ├── test_maintenance.py │ ├── test_messaging.py │ ├── test_metadata.py │ ├── test_update.py │ ├── testcases.py │ └── tools.py ├── tools.py └── webapps │ ├── __init__.py │ ├── gitlabproducer.py │ └── test │ ├── __init__.py │ └── test_gitlabproducer.py ├── man ├── jens-gc.1 ├── jens-purge-queue.1 ├── jens-reset.1 ├── jens-stats.1 └── jens-update.1 ├── puppet-jens.spec ├── pytest.ini ├── requirements.txt ├── scripts └── compare_instances.sh ├── setup.py ├── systemd ├── jens-purge-queue.service └── jens-update.service └── wsgi └── gitlab-producer.wsgi /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | name: Build 2 | 3 | on: [push] 4 | 5 | jobs: 6 | build: 7 | runs-on: ${{ matrix.os }} 8 | 9 | strategy: 10 | matrix: 11 | python-version: [3.7, 3.8, 3.9] 12 | os: [ubuntu-latest] 13 | include: 14 | - python-version: 3.6 15 | os: ubuntu-20.04 16 | 17 | steps: 18 | - uses: actions/checkout@v2 19 | - name: Set up Python ${{ matrix.python-version }} 20 | uses: actions/setup-python@v2 21 | with: 22 | python-version: ${{ matrix.python-version }} 23 | - name: Install dependencies 24 | run: | 25 | python -m pip install --upgrade pip 26 | if [ -f requirements.txt ]; then pip install -r requirements.txt; fi 27 | - name: Configure git 28 | run: | 29 | git config --global user.email "noreply@cern.ch" 30 | git config --global user.name "Github actions" 31 | - name: Test 32 | run: | 33 | python -m unittest -v 34 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.swp 3 | build 4 | -------------------------------------------------------------------------------- /DEVELOPERS.md: -------------------------------------------------------------------------------- 1 | # How to run the testsuite 2 | 3 | Jens is shipped with a bunch of functional tests which live in 4 | `jens/test`. Make sure that all the run-time dependencies declared in 5 | `requirements.txt` are installed before running them. The 6 | tests can also be run using `pytest`. 7 | 8 | ## Running the tests with a human-readable output 9 | 10 | ``` 11 | $ python -m unittest -v 12 | ``` 13 | 14 | ### Just a single test 15 | 16 | ``` 17 | $ python -m unittest jens.test.test_update.UpdateTest.test_base -v 18 | ``` 19 | 20 | ### And keeping the sandbox in disk 21 | 22 | ``` 23 | $ JENS_TEST_KEEP_SANDBOX=1 python -m unittest jens.test.test_update.UpdateTest.test_base -v 24 | ``` 25 | -------------------------------------------------------------------------------- /ENVIRONMENTS.md: -------------------------------------------------------------------------------- 1 | ## Disclaimer 2 | 3 | This documentation has been based on internal instructions available for CERN 4 | users so if there's something odd/unclear/stupid please report back to us. 5 | 6 | It's recommended to read the README first as it contains more generic 7 | information about Jens that might be interesting to digest before reading this. 8 | 9 | ## Introduction 10 | 11 | Environments are collections of modules and hostgroups at different development 12 | levels. They are defined in YAML files living in a Git repository. Jens uses 13 | this repository to check out the correct modules and hostgroups for each 14 | environment. 15 | 16 | With these files you can basically specify which is the default branch that 17 | must be used (normally _master_ or _qa_), who to inform in case of problems and 18 | any modules which must be overridden. The name of the Puppet environment much 19 | match the name of the file (without the .yaml extension). 20 | 21 | ## How to create/edit/delete environments 22 | 23 | Environment definions are plain text files. To do CUD operations on them, just 24 | add/edit/delete the file defining it and publish the change. 25 | 26 | ## Dynamic environments 27 | 28 | Dynamic environments give you a reasonable list of defaults (that will be 29 | dynamically updated) and the possibility to define overrides for specific modules or 30 | hostgroups. For instance, to create an environment named _ai321_ with all the 31 | modules/hostgroups pointing to the QA branch except from the module 'sssd' which 32 | will use the 'ai321' branch instead, just create a file looking like: 33 | 34 | ``` 35 | $ cat ai321.yaml 36 | --- 37 | default: qa 38 | notifications: bob@cern.ch 39 | overrides: 40 | modules: 41 | sssd: ai456 42 | ``` 43 | 44 | This kind of environment is the normal one, where all the included components 45 | follow the corresponding HEADs and update automatically (in the above example, 46 | every time a new commit is pushed to the QA branch of whatever module and Jens 47 | runs, the change is visible to all machines on environment ai321). Also, new 48 | modules and hostgroups that get added to the library after the environment has 49 | been created are automatically included following the default rule. 50 | 51 | Dynamic environments are meant to be used for development (essentially to have 52 | a sandbox to test a new configuration change without affecting any production 53 | service), whereas production and QA machines (CERNism warning) should live in 54 | the corresponding supported and long-lived "golden environments" with the same 55 | name. 56 | 57 | ``` 58 | $ cat production.yaml 59 | --- 60 | default: master 61 | notitications: bob@cern.ch 62 | 63 | $ cat qa.yaml 64 | --- 65 | default: qa 66 | notitications: higgs@cern.ch 67 | ``` 68 | 69 | ## Static (snapshot) environments 70 | 71 | It is also possible (but not recommended) to create configuration 72 | snapshots. 73 | 74 | A static environment is one that doesn't update dynamically, and is normally 75 | generated based on the state of an already existing dynamic environment. 76 | Nothing will change in a snapshot environment unless the environment definition 77 | is tweaked by hand. This type of environment is called a **snapshot** or an 78 | **environment with no default**. To create a static environment, just don't set 79 | any default and specify the refs you want to be expanded for each 80 | module/hostgroup, for instance: 81 | 82 | ``` 83 | $ cat snap1.yaml 84 | # Snapshot created on 2014-03-03 14:25:37.150312 based on production 85 | --- 86 | notifications: bob@example.org 87 | overrides: 88 | common: 89 | hieradata: commit/fb96070c9c77cc442ac60ba273768f547d376c17 90 | site: commit/fb96070c9c77cc442ac60ba273768f547d376c17 91 | hostgroups: 92 | adcmon: commit/8bf3ca9fe39a6f354dfc70377205ed806d6ae540 93 | foo: master 94 | ... 95 | modules: 96 | abrt: commit/580cdbcf154dec2fa9ae717f2f55a18abbaebd72 97 | ... 98 | ``` 99 | 100 | Internally, snapshots look a bit like dynamic environments but with some 101 | exceptions: 102 | 103 | * There's no default, therefore only modules/hostgroups specified in the list 104 | of overrides are included. 105 | * New modules/hostgroups cannot be sensibly added automatically, so a snapshot 106 | will not include any modules/hostgroups which are added after the snapshot 107 | is created. 108 | * Overrides point normally to commit hashes instead of branches, although 109 | branch names are supported. 110 | 111 | However, the same way as dynamic environments: 112 | 113 | * If a module/hostgroup is removed from the library, it will be removed from 114 | all the snapshots too, as if they were dynamic environments. 115 | 116 | ## Parser type 117 | 118 | When declaring an environment, use the _parser_ option to set the parser type 119 | that you want to have enabled in that particular Puppet environment. Example: 120 | 121 | ``` 122 | $ cat future.yaml 123 | --- 124 | default: qa 125 | notitications: higgs@cern.ch 126 | parser: future 127 | ``` 128 | 129 | The allowed values are: unset (parser key not declared), current or future. 130 | 131 | See [PuppetLabs' documentation](http://cern.ch/go/mb6h) for more information. 132 | 133 | ## Which environment should I use for my production service? 134 | 135 | Disclaimer: The following tips are inevitably coupled to CERN IT policies so 136 | it can be safely ignored. They're kept here as they might be useful for the 137 | general public to understand a bit more what dynamic and static environments 138 | are. 139 | 140 | You should use the _production_ environment. 141 | 142 | Reasons to use the default production dynamic environment: 143 | 144 | * You will get configuration fully aligned to infrastructure changes for free, 145 | guaranteeing that your machine works with the latest components. 146 | 147 | Reasons not to use snapshots: 148 | 149 | * You will get off the train of changes very quickly. 150 | * They are difficult to maintain, as once something is broken, mangling the 151 | overrides to make it work again by trying to get a newer version of 152 | several configuration components can be very tricky and potentially dangerous. 153 | * Make Jens slower and fatter. 154 | 155 | If you really need snapshots: 156 | 157 | * Make the lifetime of them as short as you can (i.e. don't stay on them 158 | for any long length of time). Your risk of divergence from the infrastructure 159 | increases with time. 160 | * When you want to advance, make a new snapshot from the current 161 | __production__ environment. 162 | * Delete them as soon as you don't need them anymore. 163 | -------------------------------------------------------------------------------- /INSTALL.md: -------------------------------------------------------------------------------- 1 | ## Building and installation 2 | 3 | Use the shipped spec file to create an RPM. The package will install an example 4 | configuration file and a series of skeletons that can be used to generate 5 | example repositories. These will be used in the next section to help you to get 6 | started with the tool. 7 | 8 | You can also use distutils and copy the example configuration file by hand. 9 | 10 | ## Configuration 11 | 12 | Jens consumes a single configuration file that is located by default in 13 | `/etc/jens/main.conf`. Apart from tweaking this file, it's also necessary to 14 | initialise and make available the metadata repositories that are needed by 15 | Jens. The RPM ships examples for all the bits that are required to get 16 | started with the tool, so in this section we will use all of them to get 17 | a basic working configuration where you can start from. 18 | 19 | The RPM creates a system user called `jens` and sets the permissions of 20 | the directories specified in the default configuration file for you. 21 | 22 | So, let's put our hands on it. Firstly, we're going to initialise in `/tmp` a 23 | bunch of Git repositories for which, as mentioned previously, there are example 24 | skeletons shipped by the package. This should be enough to get a clean 25 | jens-update run that does something useful. However, if you already know how 26 | Jens works you can safely forget about all these dummy repositories and plug-in 27 | existing stuff. 28 | 29 | The example configuration file installed by the package in `/etc/jens` should 30 | suffice for the time being. So, let's initialise the metadata repositories, a 31 | module, a hostgroup and some Hiera data, based on the examples provided by the 32 | package. It's recommended to run the commands below as `jens` as the rest of the 33 | tutorial relies on this account, however feel free to proceed as you see fit as 34 | long as everything is consistent :) 35 | 36 | ``` 37 | # sudo -u jens bash 38 | $ cp -r /usr/share/doc/puppet-jens-0.10/examples/example-repositories /tmp 39 | $ cd /tmp/example-repositories 40 | $ cp -r /usr/share/doc/puppet-jens-0.10/examples/environments /tmp/example-repositories 41 | $ cp -r /usr/share/doc/puppet-jens-0.10/examples/repositories /tmp/example-repositories 42 | $ ls | xargs -i bash -c "cd {} && git init && git add * && git commit -m 'Init' && cd .." 43 | $ cd /var/lib/jens/metadata 44 | $ git clone file:///tmp/example-repositories/environments environments 45 | $ git clone file:///tmp/example-repositories/repositories repository 46 | ``` 47 | 48 | ## Operation 49 | 50 | ### Triggering an update cycle 51 | 52 | So now everything should be in place to start doing something interesting. 53 | Let's then trigger the first Jens run: 54 | 55 | ``` 56 | # cd /var/lib/jens 57 | # sudo -u jens jens-update 58 | ``` 59 | 60 | Now take a look to `/var/log/jens/jens-update.log`. It should look like: 61 | 62 | ``` 63 | INFO Obtaining lock 'jens' (attempt: 1)... 64 | INFO Refreshing metadata... 65 | INFO Refreshing repositories... 66 | INFO Fetching repositories inventory... 67 | WARNING Inventory on disk not found or corrupt, generating... 68 | INFO Generating inventory of bares and clones... 69 | INFO Refreshing bare repositories (modules) 70 | INFO New repositories: set(['dummy']) 71 | INFO Deleted repositories: set([]) 72 | INFO Cloning and expanding NEW bare repositories... 73 | INFO Cloning and expanding modules/dummy... 74 | INFO Populating new ref '/var/lib/jens/clone/modules/dummy/master' 75 | INFO Expanding EXISTING bare repositories... 76 | INFO Purging REMOVED bare repositories... 77 | INFO Refreshing bare repositories (hostgroups) 78 | INFO New repositories: set(['myapp']) 79 | INFO Deleted repositories: set([]) 80 | INFO Cloning and expanding NEW bare repositories... 81 | INFO Cloning and expanding hostgroups/myapp... 82 | INFO Populating new ref '/var/lib/jens/clone/hostgroups/myapp/master' 83 | INFO Expanding EXISTING bare repositories... 84 | INFO Purging REMOVED bare repositories... 85 | INFO Refreshing bare repositories (common) 86 | INFO New repositories: set(['hieradata', 'site']) 87 | INFO Deleted repositories: set([]) 88 | INFO Cloning and expanding NEW bare repositories... 89 | INFO Cloning and expanding common/hieradata... 90 | INFO Populating new ref '/var/lib/jens/clone/common/hieradata/master' 91 | INFO Cloning and expanding common/site... 92 | INFO Populating new ref '/var/lib/jens/clone/common/site/master' 93 | INFO Expanding EXISTING bare repositories... 94 | INFO Purging REMOVED bare repositories... 95 | INFO Persisting repositories inventory... 96 | INFO Executed 'refresh_repositories' in 1405.49 ms 97 | INFO Refreshing environments... 98 | INFO New environments: set(['production']) 99 | INFO Existing and changed environments: [] 100 | INFO Deleted environments: set([]) 101 | INFO Creating new environments... 102 | INFO Creating new environment 'production' 103 | INFO Processing modules... 104 | INFO Processing hostgroups... 105 | INFO Processing site... 106 | INFO Processing common Hiera data... 107 | INFO Purging deleted environments... 108 | INFO Recreating changed environments... 109 | INFO Refreshing not changed environments... 110 | INFO Executed 'refresh_environments' in 11.53 ms 111 | INFO Releasing lock 'jens'... 112 | INFO Done 113 | ``` 114 | 115 | Also, keep an eye on `/var/lib/jens/environments` and `/var/lib/jens/clone` to 116 | see what actually happened. 117 | 118 | ``` 119 | environments 120 | └── production 121 | ├── hieradata 122 | │   ├── common.yaml -> ../../../clone/common/hieradata/master/data/common.yaml 123 | │   ├── environments -> ../../../clone/common/hieradata/master/data/environments 124 | │   ├── fqdns 125 | │   │   └── myapp -> ../../../../clone/hostgroups/myapp/master/data/fqdns 126 | │   ├── hardware -> ../../../clone/common/hieradata/master/data/hardware 127 | │   ├── hostgroups 128 | │   │   └── myapp -> ../../../../clone/hostgroups/myapp/master/data/hostgroup 129 | │   ├── module_names 130 | │   │   └── dummy -> ../../../../clone/modules/dummy/master/data 131 | │   └── operatingsystems -> ../../../clone/common/hieradata/master/data/operatingsystems 132 | ├── hostgroups 133 | │   └── hg_myapp -> ../../../clone/hostgroups/myapp/master/code 134 | ├── modules 135 | │   └── dummy -> ../../../clone/modules/dummy/master/code 136 | └── site -> ../../clone/common/site/master/code 137 | ``` 138 | 139 | ``` 140 | clone 141 | ├── common 142 | │   ├── hieradata 143 | │   │   └── master 144 | │   │   └── data 145 | │   │   ├── common.yaml 146 | │   │   ├── environments 147 | │   │   │   ├── production.yaml 148 | │   │   │   └── qa.yaml 149 | │   │   ├── hardware 150 | │   │   │   └── vendor 151 | │   │   │   └── sinclair.yaml 152 | │   │   └── operatingsystems 153 | │   │   └── RedHat 154 | │   │   ├── 5.yaml 155 | │   │   ├── 6.yaml 156 | │   │   └── 7.yaml 157 | │   └── site 158 | │   └── master 159 | │   └── code 160 | │   └── site.pp 161 | ├── hostgroups 162 | │   └── myapp 163 | │   └── master 164 | │   ├── code 165 | │   │   ├── files 166 | │   │   │   └── superscript.sh 167 | │   │   ├── manifests 168 | │   │   │   ├── frontend.pp 169 | │   │   │   └── init.pp 170 | │   │   └── templates 171 | │   │   └── someotherconfig.erb 172 | │   └── data 173 | │   ├── fqdns 174 | │   │   └── myapp-node1.example.org.yaml 175 | │   └── hostgroup 176 | │   ├── myapp 177 | │   │   └── frontend.yaml 178 | │   └── myapp.yaml 179 | └── modules 180 | └── dummy 181 | └── master 182 | ├── code 183 | │   ├── manifests 184 | │   │   ├── init.pp 185 | │   │   └── install.pp 186 | │   ├── README.md 187 | │   └── templates 188 | │   └── config.erb 189 | └── data 190 | └── jens.yaml 191 | ``` 192 | 193 | As expected, one new environment and a few repositories were expanded as 194 | required by the environment (all master branches, basically). 195 | 196 | Let's now add something to the dummy module in a separate branch, create a new 197 | environment using it and run jens-update again to see what it does: 198 | 199 | ``` 200 | # sudo -u jens bash 201 | $ cd /tmp/example-repositories/module-dummy 202 | $ git checkout -b test 203 | Switched to a new branch 'test' 204 | $ touch code/manifests/foo.pp 205 | $ git add code/manifests/foo.pp 206 | $ git commit -m 'foo' 207 | [test 052c5b4] foo 208 | 0 files changed, 0 insertions(+), 0 deletions(-) 209 | create mode 100644 code/manifests/foo.pp 210 | $ cd /tmp/example-repositories/environments/ 211 | $ cp production.yaml test.yaml 212 | ## Edit the file so it looks like: 213 | $ cat test.yaml 214 | --- 215 | default: master 216 | notifications: admins@example.org 217 | overrides: 218 | modules: 219 | dummy: test 220 | $ git add test.yaml 221 | $ git commit -m 'Add test environment' 222 | [master 1d78cd5] Add test environment 223 | 1 files changed, 6 insertions(+), 0 deletions(-) 224 | create mode 100644 test.yaml 225 | $ cd /var/lib/jens/ 226 | $ jens-update 227 | ## A new branch is necessary so it's expanded 228 | $ tree -L 1 /var/lib/jens/clone/modules/dummy/ 229 | /var/lib/jens/clone/modules/dummy/ 230 | ├── master 231 | └── test 232 | ## And the override is in place in the newly created environment 233 | $ tree /var/lib/jens/environments/test/modules/ 234 | /var/lib/jens/environments/test/modules/ 235 | └── dummy -> ../../../clone/modules/dummy/test/code 236 | $ tree /var/lib/jens/environments/test/hostgroups 237 | /var/lib/jens/environments/test/hostgroups 238 | └── hg_myapp -> ../../../clone/hostgroups/myapp/master/code 239 | $ ls /var/lib/jens/environments/test/modules/dummy/manifests/foo.pp 240 | /var/lib/jens/environments/test/modules/dummy/manifests/foo.pp 241 | $ ls /var/lib/jens/environments/production/modules/dummy/manifests/foo.pp 242 | ls: cannot access /var/lib/jens/environments/production/modules/dummy/manifests/foo.pp: No such file or directory 243 | ``` 244 | 245 | Log file wise this is what's printed: 246 | 247 | ``` 248 | INFO Obtaining lock 'jens' (attempt: 1)... 249 | INFO Refreshing metadata... 250 | INFO Refreshing repositories... 251 | INFO Fetching repositories inventory... 252 | INFO Refreshing bare repositories (modules) 253 | INFO New repositories: set([]) 254 | INFO Deleted repositories: set([]) 255 | INFO Cloning and expanding NEW bare repositories... 256 | INFO Expanding EXISTING bare repositories... 257 | INFO Populating new ref '/var/lib/jens/clone/modules/dummy/test' 258 | INFO Purging REMOVED bare repositories... 259 | INFO Refreshing bare repositories (hostgroups) 260 | INFO New repositories: set([]) 261 | INFO Deleted repositories: set([]) 262 | INFO Cloning and expanding NEW bare repositories... 263 | INFO Expanding EXISTING bare repositories... 264 | INFO Purging REMOVED bare repositories... 265 | INFO Refreshing bare repositories (common) 266 | INFO New repositories: set([]) 267 | INFO Deleted repositories: set([]) 268 | INFO Cloning and expanding NEW bare repositories... 269 | INFO Expanding EXISTING bare repositories... 270 | INFO Purging REMOVED bare repositories... 271 | INFO Persisting repositories inventory... 272 | INFO Executed 'refresh_repositories' in 691.18 ms 273 | INFO Refreshing environments... 274 | INFO New environments: set(['test']) 275 | INFO Existing and changed environments: [] 276 | INFO Deleted environments: set([]) 277 | INFO Creating new environments... 278 | INFO Creating new environment 'test' 279 | INFO Processing modules... 280 | INFO modules 'dummy' overridden to use treeish 'test' 281 | INFO Processing hostgroups... 282 | INFO Processing site... 283 | INFO Processing common Hiera data... 284 | INFO Purging deleted environments... 285 | INFO Recreating changed environments... 286 | INFO Refreshing not changed environments... 287 | INFO Executed 'refresh_environments' in 19.11 ms 288 | INFO Releasing lock 'jens'... 289 | INFO Done 290 | ``` 291 | 292 | Now it's your turn to play with it! Commit things, create environments, add 293 | more modules... :) 294 | 295 | ## Running modes: Polling or on-demand? 296 | 297 | Jens has two running modes that can be selected using the `mode` key available 298 | in the configuration file. By default Jens will run in polling mode 299 | (`mode=POLL`), meaning that all the repositories that Jens is aware of will be 300 | polled (git-fetched) on every run. This is generally slow and not very 301 | efficient but, on the other hand, simpler. 302 | 303 | However, in deployments where notifications can be sent when new code is 304 | available (for instance, via push webhooks emitted by Gitlab or Github), it's 305 | recommended to run Jens in on-demand mode instead (`mode=ONDEMAND`). These 306 | notifications can be received by a listener (read below), transformed, and 307 | handed over to Jens. 308 | 309 | When this mode is enabled, every Jens run will first get "update hints" from a 310 | local python-dirq queue (path set in the configuration file, being 311 | `/var/spool/jens-update` the default value) and only bug the servers when 312 | there's actually something new to retrieve. This is much more efficient and it 313 | allows running Jens more often as it's faster and more lightweight for the 314 | server. 315 | 316 | The format of the messages that Jens expects can be explored in detail by 317 | reading `messaging.py` but in short the schema is composed by two keys: a 318 | timestamp in ISO format (time key) which is a string and a pickled binary 319 | payload (data key) specifying what modules or hostgroups have changed. For 320 | example: 321 | 322 | ``` 323 | {'time': '2015-12-10T14:06:35.339550', 324 | 'data': pickle.dumps({'modules': ['m1'], 'hostgroups': ['h1', 'h2']})} 325 | ``` 326 | 327 | The idea then is to have something producing this type of message. This suite 328 | also ships a Gitlab listener that understands the payload contained in the 329 | requests made via [Gitlab push 330 | webhooks](https://docs.gitlab.com/ce/user/project/integrations/webhooks.html#push-events) 331 | and translates it into the format used internally, producing this way the 332 | messages required by Jens. This listener (and producer) can be run standalone 333 | for testing purposes via `jens-gitlab-producer-runner` or, much better, on top 334 | of a web server talking WSGI. For this purpose an example WSGI file is also 335 | shipped along the software. The producer has to run on the same host so it 336 | has access to the local queue. 337 | 338 | When a producer and `jens-update` are cooperating and the on-demand mode is 339 | enabled, information regarding the update hints consumed and the actions taken 340 | is present in the usual log file, for example: 341 | 342 | ``` 343 | ... 344 | INFO Getting and processing hints... 345 | INFO 1 messages found 346 | INFO Executed 'fetch_update_hints' in 2.31 ms 347 | ... 348 | INFO Fetching hostgroups/foo upon demand... 349 | INFO Updating ref '/mnt/puppet/aijens-3afegt67.cern.ch/clone/hostgroups/foo/qa' 350 | ``` 351 | 352 | Also, `jens-gitlab-producer.log` is populated as notifications come in and hints 353 | are enqueued: 354 | 355 | ``` 356 | INFO 2015-12-10T14:43:01.705468 - hostgroups/foo - '0000003c/56698165ac7909' added to the queue 357 | ``` 358 | 359 | Requests can be authenticated using a [secret 360 | token](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#validate-payloads-by-using-a-secret-token) 361 | which has to be configured both on the Gitlab side (via the group or 362 | repository settings) and on the Jens side via: 363 | 364 | ```ini 365 | [gitlabproducer] 366 | secret_token = your_token_here 367 | ``` 368 | 369 | If there's no token set (default) then all requests will be accepted 370 | regardless of the presence or the value of the header 371 | `X-Gitlab-Token`. 372 | 373 | ### Setting a timeout for Git operations via SSH 374 | 375 | To protect Jens from hanging indefinitely in case of a lack of response from 376 | the remote, it's possible to override the SSH command and put a timeout in 377 | between so the Git operations don't run forever. This only applies if the 378 | traffic to the remotes is transported via SSH. 379 | 380 | The idea is to write a file wrapping the call to ssh in a timeout. Example: 381 | 382 | ```shell 383 | # cat /usr/local/bin/timeoutssh.sh 384 | timeout 10 ssh $@ 385 | ``` 386 | 387 | And then tell Jens to use that wrapper via the configuration file: 388 | 389 | ``` 390 | [git] 391 | ssh_cmd_path = /usr/local/bin/timeoutssh.sh 392 | ``` 393 | 394 | If the configuration option is not set the environment variable GIT_SSH won't be internally set by Jens. 395 | 396 | ### Getting statistics about the number of modules, hostgroups and environments 397 | 398 | ``` 399 | # cd /var/lib/jens 400 | # sudo -u jens jens-stats -a 401 | There are 1 modules: 402 | - dummy (master,test) [17.5KiB] 403 | There are 1 hostgroups: 404 | - myapp (master) [18.2KiB] 405 | There are 2 cached environments 406 | - production 407 | - test 408 | There are 2 declared environments 409 | - production.yaml 410 | - test.yaml 411 | There are 2 synchronized environments 412 | - production 413 | - test 414 | Fetching repositories inventory... 415 | {'common': {'hieradata': ['master'], 'site': ['master']}, 416 | 'hostgroups': {'myapp': ['master']}, 417 | 'modules': {'dummy': ['master', 'test']}} 418 | ``` 419 | 420 | ### Resetting everything 421 | 422 | The following command will remove all generated environments, caches and all 423 | repository clones, taking Jens to an initial state: 424 | 425 | ``` 426 | # cd /var/lib/jens 427 | # sudo -u jens jens-reset --yes 428 | ... 429 | Done -- Jens is sad now 430 | # sudo -u jens jens-stats -a 431 | There are 0 modules: 432 | There are 0 hostgroups: 433 | There are 0 cached environments 434 | There are 2 declared environments 435 | - production.yaml 436 | - test.yaml 437 | There are 0 synchronized environments 438 | Fetching repositories inventory... 439 | Inventory on disk not found or corrupt, generating... 440 | Generating inventory of bares and clones... 441 | {'common': {}, 'hostgroups': {}, 'modules': {}} 442 | ``` 443 | 444 | ## Few comments on deployment and locking 445 | 446 | Jens implements one local locking mechanism based on a fcntl.flock so when run 447 | via a cronjob subsequent runs can't overlap. This is because of our deployment, 448 | detailed in the following paragraphs. 449 | 450 | Currently, the deployment at CERN relies on several Jens instances on 451 | top of different virtual machines running AlmaLinux 9 with different 452 | update frequencies writing the `clone` and `environments` data to the 453 | same NFS share but on different directories (whose name is based on 454 | the FQDN of the Jens node in question). One is the primary instance 455 | that runs jens-update every minute and the rest are satellite 456 | instances that do it less often (frequently enough to not to overload 457 | our internal Git service and to have a relatively up-to-date tree of 458 | environments that could be taken to the "HEAD" state quickly if a node 459 | had to take over). 460 | 461 | This allows a relatively quick but manual fail over mechanism in case of 462 | primary node failure, which consists of: electing a new primary node so the new 463 | node has a higher update frequency than before, running jens-update by hand on 464 | that node to make sure that there's no configuration flip-flop and flipping a 465 | symlink so the Puppet masters (which are also reading manifests from the same 466 | NFS share) start consuming data from another Jens instance. 467 | 468 | This is basically how our NFS share looks like: 469 | 470 | ``` 471 | /mnt/puppet 472 | ├── aijens -> aijens-3afegt67.cern.ch/ 473 | ├── aijens-3afegt67.cern.ch 474 | │   ├── clone 475 | │   └── environments 476 | ├── aijens-ty5ee527.cern.ch 477 | │   ├── clone 478 | │   └── environments 479 | ├── aijens-ior59ff6.cern.ch 480 | │   ├── clone 481 | │   └── environments 482 | ├── aijens-dev.cern.ch 483 | │   ├── clone 484 | │   └── environments 485 | └── environments -> aijens/environments/ 486 | ``` 487 | 488 | 11 directories, 0 files 489 | 490 | So, in our case, the masters' modulepath is something like: 491 | 492 | ``` 493 | /mnt/puppetdata/aijens/environments/$environment/hostgroups: 494 | /mnt/puppetdata/aijens/environments/$environment/modules 495 | ``` 496 | 497 | Where `/mnt/puppetdata` is the mountpoint of the NFS share. 498 | 499 | At the time of writing, our Jens instances are taking care of 500 | 260 modules, 150 hostgroups and 160 environments :) 501 | 502 | ## Protected environments 503 | 504 | It's possible to set a list of environments that won't ever be deleted from 505 | disk, even if the declaration is removed from the metadata. This is useful to 506 | protect delicate environments like production. To achieve this, set the 507 | following option in the configuration file: 508 | 509 | ``` 510 | [main] 511 | protectedenvironments = production, qa 512 | ``` 513 | 514 | It's not recommended to add environments that have overrides to the list as 515 | they might end up incomplete if they're removed. 516 | 517 | We recommend though to use this only as an extra safety feature and make sure 518 | that changes to environments are validated either manually or using a CI 519 | process. 520 | 521 | The default value is an empty list, which means that no environment is 522 | protected. 523 | 524 | ## Miscellaneous 525 | 526 | If you wanted to know in detail what Jens does in every run, changing the debug 527 | level to DEBUG (in `/etc/jens/main.conf`) might be a good idea :) 528 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![](https://github.com/cernops/jens/workflows/Build/badge.svg)](https://github.com/cernops/jens/actions?query=workflow%3ABuild) 2 | 3 | ## What's Jens? 4 | 5 | Jens is the Puppet modules/hostgroups librarian used by the [CERN IT 6 | department](https://cern.ch/it). It is basically a Python toolkit that 7 | generates [Puppet environments]( 8 | https://docs.puppetlabs.com/puppet/latest/reference/environments.html) 9 | dynamically based on some input metadata. Jens is useful in sites where there 10 | are several administrators taking care of siloed services (mapped to what we 11 | call top-level "hostgroups", see below) with very service-specific 12 | configuration but sharing configuration logic via modules. 13 | 14 | This tool covers the need of several roles that might show up in a typical 15 | shared Puppet infrastructure: 16 | 17 | * Developers writing manifests who want an environment to test new code: Jens 18 | provides dynamic environments that automatically update with overrides for 19 | the modules being developed that point to development branches. 20 | * Administrators who don't care: Jens supports simple dynamic environments 21 | that default to the production branch of all modules and that only update 22 | automatically when there's new production code. 23 | * Administrators looking for extra stability who are reluctant to do rolling 24 | updates: Jens implements snapshot environments that are completely static and 25 | don't update unless redefined, as all modules are pinned by commit 26 | identifier. 27 | 28 | Right now, the functionality is quite tailored to CERN IT's needs, however 29 | all contributions to make it more generic and therefore more useful for the 30 | community are more than welcome. 31 | 32 | This program has been used as the production Puppet librarian at CERN IT since 33 | August 2013. 34 | 35 | ## Introduction 36 | 37 | In Jens' realm, Puppet environments are basically a collection of modules, 38 | hostgroups, hierarchies of Hiera data and a site.pp. These environments 39 | are defined in environment definition files which are stored in a separate 40 | repository that's known to the program. Also, Jens makes use of a second 41 | metadata repository to know what modules and hostgroups are part of the library 42 | and are therefore available to generate environments. 43 | 44 | With all this information, Jens produces a set of environments that can be used 45 | by Puppet masters to compile Puppet catalogs. Two types of environments are 46 | supported: dynamic and static. The former update automatically as new commits 47 | arrive to the concerned repositories whereas the latter remain static pointing 48 | to the specified commits to implement the concept of "configuration snapshot" 49 | (read the environments section for more information). 50 | 51 | Jens is composed by several CLIs: jens-config, jens-gc, jens-reset, jens-stats 52 | and jens-update to perform different tasks. Manual pages are shipped for all of 53 | them. 54 | 55 | Basically, the input data that's necessary for an execution of jens-update (the 56 | core tool provided by this toolset) is two Git repositories: 57 | 58 | * The repository metadata repository (or _the library_) 59 | * The environment definitions repository (or _the environments_) 60 | 61 | More details about these are given in the following sections. 62 | 63 | ## Repository metadata 64 | 65 | Jens uses a single YAML file stored in a Git repository to know what are the 66 | modules and hostgroups available to generate environments. Apart from that, 67 | it's also used to define the paths to two special Git repositories containing 68 | what's called around here _the common Hiera data_ and the site manifest. 69 | 70 | This is all set up via two configuration keys: `repositorymetadatadir` (which is 71 | the directory containing a clone of the repository) and `repositorymetadata` 72 | (the file itself). 73 | 74 | The following is how a skeleton of the file looks like: 75 | 76 | ``` 77 | --- 78 | repositories: 79 | common: 80 | hieradata: http://git.example.org/pub/it-puppet-common-hieradata 81 | site: http://git.example.org/pub/it-puppet-site 82 | hostgroups: 83 | ... 84 | aimon: http://git.example.org/pub/it-puppet-hostgroup-aimon 85 | cloud: http://git.example.org/pub/it-puppet-hostgroup-cloud 86 | ... 87 | modules: 88 | ... 89 | apache: http://git.example.org/pub/it-puppet-module-apache 90 | bcache: http://git.example.org/pub/it-puppet-module-bcache 91 | ... 92 | ``` 93 | 94 | The idea is that when a new top-level hostgroup is added or a new module 95 | is needed this file gets populated with the corresponding clone URLs of 96 | the repositories. Jens will add new elements to all the environments 97 | that are entitled to get them during the next run of jens-update. 98 | 99 | Another example is available in `examples/repositories/repositories.yaml`. 100 | 101 | ### Common Hiera data and Site 102 | 103 | There are two bits that are declared via the library file that require some 104 | extra clarifications, especially because they are fundamentally traversal to 105 | the rest of the declarations and are maybe a bit hardcoded to how our Puppet 106 | infrastructure is designed. 107 | 108 | The repository pointed to by _site_ must contain a single manifest called 109 | site.pp that serves as the catalog compilation entrypoint and therefore where 110 | all the hostgroup autoloading (explained later) takes place. 111 | 112 | ``` 113 | it-puppet-site/ 114 | ├── code 115 | │   └── site.pp 116 | └── README 117 | ``` 118 | 119 | OTOH, the common hieradata is a special repository that hosts different types 120 | of Hiera data to fill the gaps that can't be defined at hostgroup or module 121 | level (operating system, hardware vendor, datacentre location and environment 122 | dependent keys). The list of these items is configurable and can be set by using 123 | the configuration key `common_hieradata_items`. The following is an example of 124 | how the hierarchy in there should look like. 125 | 126 | ``` 127 | it-puppet-common-hieradata/ 128 | ├── data 129 | │   ├── common.yaml 130 | │   ├── environments 131 | │   │   ├── production.yaml 132 | │   │   └── qa.yaml 133 | │   ├── datacentres 134 | │   │   ├── europe.yaml 135 | │   │   ├── usa.yaml 136 | │   │   └── ... 137 | │   ├── hardware 138 | │   │   └── vendor 139 | │   │   ├── foovendor.yaml 140 | │   │   └── ... 141 | │   └── operatingsystems 142 | │   └── RedHat 143 | │   ├── 5.yaml 144 | │   ├── 6.yaml 145 | │   └── 7.yaml 146 | └── README 147 | ``` 148 | 149 | common.yaml is the most generic Hiera data YAML file of all the hierarchy as 150 | it's visible for all nodes regardless of their hostgroup, environment, hardware 151 | type, operatingsystem and datacentre. It's useful to define very top-level keys. 152 | 153 | Working examples of both repositories (used during the installation tutorial 154 | later on) can be found in the following locations 155 | 156 | * `examples/example-repositories/common-hieradata` 157 | * `examples/example-repositories/site` 158 | 159 | Also, an example of a Hiera hierarchy configuration file that matches this 160 | structure is available on `examples/hiera.yaml`. 161 | 162 | ### Modules: Code and data directories 163 | 164 | Each module/hostgroup lives in a separate Git repository, which contains two 165 | top-level directories: code and data. 166 | 167 | * code: this is where the actual Puppet code resides, basically where the 168 | manifests, lib, files and templates directories live. 169 | * data: all the relevant Hiera data is stored here. For modules, 170 | there's only one YAML file named after the module. 171 | 172 | Example: 173 | 174 | ``` 175 | it-puppet-module-lemon/ 176 | ├── code 177 | │   ├── lib 178 | │   │   └── facter 179 | │   │   ├── configured_kernel.rb 180 | │   │   ├── lemon_exceptions.rb 181 | │   │   └── ... 182 | │   ├── manifests 183 | │   │   ├── config.pp 184 | │   │   ├── init.pp 185 | │   │   ├── install.pp 186 | │   │   ├── klogd.pp 187 | │   │   ├── las.pp 188 | │   │   └── ... 189 | │   ├── Modulefile 190 | │   ├── README 191 | │   └── templates 192 | │   └── metric.conf.rb 193 | └── data 194 | └── lemon.yaml 195 | ``` 196 | 197 | For those already wondering how we manage to keep track of upstream modules 198 | with this strutucture: Git subtree :) 199 | 200 | ### Hostgroups: What they are and why they're useful 201 | 202 | Hostgroups are just Puppet modules that are a bit special, allowing us to 203 | automatically load Puppet classes based on the hostgroup a given host belongs 204 | to (information which is fetched at compilation time from an ENC). 205 | 206 | This is a CERNism and unfortunately we're not aware of anybody in the Puppet 207 | community doing something similar. However, we found this idea very useful to 208 | classify IT services, grouping machines belonging to a given service in the 209 | same top-level hostgroup. Modules are normally included in the hostgroup 210 | manifests (along the hierarchy) and configured via Hiera. 211 | 212 | In short, hostgroups represent the service-specific configuration and modules 213 | are reusable "blocks" of code that abstract certain recurrent configuration 214 | tasks which are typically used by several hostgroups. 215 | 216 | Getting back to the structure itself, the code directory serves the same 217 | purpose as the one for modules, however the data one is slightly different, as 218 | it contains FQDN-specific Hiera data for hosts belonging to this hostgroup and 219 | data that applies at different levels of the hostgroup hierarchy. 220 | 221 | Next, a partial example of a real-life hostgroup and its subhostgroups with 222 | the corresponding manifests and Hiera data: 223 | 224 | ``` 225 | it-puppet-hostgroup-punch/code/manifests/ 226 | ├── aijens 227 | │   ├── app 228 | │   │   └── live.pp 229 | │   ├── app.pp 230 | ... 231 | ├── init.pp 232 | ``` 233 | 234 | ``` 235 | it-puppet-hostgroup-punch/data/hostgroup 236 | ├── punch 237 | │   ├── aijens 238 | │   │   ├── app 239 | │   │   │   ├── live 240 | │   │   │   │   └── primary.yaml 241 | │   │   │   └── live.yaml 242 | │   │   └── app.yaml 243 | │   ├── aijens.yaml 244 | ... 245 | └── punch.yaml 246 | ``` 247 | 248 | ``` 249 | it-puppet-hostgroup-punch/data/fqdns/ 250 | ├── foo1.cern.ch.yaml 251 | ``` 252 | 253 | For instance, if **foo1.cern.ch** belonged to **punch/aijens/app/live**, it'd 254 | be entitled to automatically include init.pp, aijens.pp (which does no exist in 255 | this case), app.pp and live.pp. Also, Hiera keys will be looked up using files 256 | foo1.cern.ch.yaml, punch.yaml, aijens.yaml, app.yaml and live.yaml 257 | 258 | To avoid clashes during the autoloading with modules that might have 259 | the same name, the top-most class of the hostgroup is prefixed with **hg_**. 260 | 261 | ``` 262 | ~ $ grep ^class it-puppet-hostgroup-punch/code/manifests/aijens/app.pp 263 | class hg_punch::aijens::app { 264 | ~ $ grep ^class it-puppet-hostgroup-punch/code/manifests/init.pp 265 | class hg_punch { 266 | ``` 267 | 268 | There's more information about how this all works filesystem-wise below. An 269 | example of the autoloading mechanism can be found in the example site.pp 270 | mentioned above. 271 | 272 | ## An introduction to Jens environments 273 | 274 | One of the parameters that can be configured via the configuration file is the 275 | path to a clone of the Git repository containing the environment definitions 276 | (configuration key `environmentsmetadatadir`). Each file with .yaml extension 277 | is a candidate environment and will be considered by Jens during the update cycle. 278 | 279 | An example of a very simple one can be located in `examples/environments`. 280 | 281 | That said, environments are mostly useful for development, for instance to 282 | validate the effect of a change in an specific module using the stable version 283 | of the remaining configuration. This is accomplished by using environment 284 | overrides: 285 | 286 | ``` 287 | ~/it-puppet-environments $ cat devticket34.yaml 288 | --- 289 | default: master 290 | notifications: higgs@example.org 291 | overrides: 292 | modules: 293 | apache: dev34 294 | ``` 295 | 296 | See [ENVIRONMENTS.md](ENVIRONMENTS.md) for further details on this 297 | subject, including how to create configuration snapshots instead of 298 | dynamic environments :) 299 | 300 | ### "Golden" environments 301 | 302 | Environments are cool to quickly get a development sandbox. They go in and out 303 | in a very rapid fashion, as new features are needed or finished. However, these 304 | are indeed not ideal for production nodes. Because of this, it's also 305 | interesting to have some kind of special environments and use them as the place 306 | where all the code converges at some point after development/testing/QA and 307 | place the bulk of the service there. 308 | 309 | This section is just a recommendation that describes how things are done over 310 | here, however Jens does not enforce the existence or the structure of any 311 | environment at all. 312 | 313 | In our site there's the concept of **golden environment**. These are 314 | long-lived, simple and unmodifiable dynamic environments used for production 315 | and QA (mandatory for all changes in modules impacting several unrelated 316 | services). The following are the definitions: 317 | 318 | ``` 319 | ~/it-puppet-environments $ cat production.yaml qa.yaml 320 | --- 321 | default: master 322 | notifications: higgs@example.org 323 | --- 324 | default: qa 325 | notifications: higgs@example.org 326 | ``` 327 | 328 | As you can see, these are very simple environments that collect all the master 329 | and qa branches of all modules and hostgroups available in the library. This 330 | of course relies on one of our internal policies that say that all the 331 | repositories containing Puppet code must have at least two branches: `master` 332 | and `qa`, meaning that they will always be expanded by Jens. This behaviour can 333 | be configured by the `mandatorybranches` configuration key. 334 | 335 | As environment definitions are just Yaml files, the files can be 336 | easily protected from unauthorised/accidental modification via, for 337 | instance, Gitolite rules. Another option is to receive changes via 338 | merge requests that will be validated by an external continuous 339 | integration job under the administrator's control. 340 | 341 | ### How does an environment look like on disk? 342 | 343 | Internally, Jens uses different directories to keep track of what's going on 344 | with the Git repositories it has to be aware of. There's one called `clone` 345 | that contains a checked out working tree of all the branches that are necessary 346 | by environments for a each module/hostgroup. This way, an environment is just a 347 | collection of symbolic links to these directories, which targets depend on the 348 | environment definition. As an example, this is how the whole tree for the 349 | hypothetical environment _devticket34_ declared previously would look like: 350 | 351 | ``` 352 | environments/devticket34/ 353 | ├── hieradata 354 | │   ├── common.yaml -> ../../../clone/common/hieradata/master/data/common.yaml 355 | │   ├── environments -> ../../../clone/common/hieradata/master/data/environments 356 | │   ├── fqdns 357 | │   │   ├── aimon -> ../../../../clone/hostgroups/aimon/master/data/fqdns 358 | │   │   ├── cloud -> ../../../../clone/hostgroups/cloud/master/data/fqdns 359 | │   │   └── ... 360 | │   ├── hardware -> ../../../clone/common/hieradata/master/data/hardware 361 | │   ├── hostgroups 362 | │   │   ├── aimon -> ../../../../clone/hostgroups/aimon/master/data/hostgroup 363 | │   │   ├── cloud -> ../../../../clone/hostgroups/cloud/master/data/hostgroup 364 | │   │   └── ... 365 | │   ├── module_names 366 | │   │   ├── apache -> ../../../../clone/modules/apache/dev34/data 367 | │   │   ├── bcache -> ../../../../clone/modules/apache/bcache/data 368 | │   │   └── ... 369 | │   └── operatingsystems -> ../../../clone/common/hieradata/master/data/operatingsystems 370 | ├── hostgroups 371 | │   ├── hg_aimon -> ../../../clone/hostgroups/aimon/master/code 372 | │   ├── hg_cloud -> ../../../clone/hostgroups/cloud/master/code 373 | │   └── hg_... 374 | ├── modules 375 | │   ├── apache -> ../../../clone/modules/apache/dev34/code 376 | │   ├── bcache -> ../../../clone/modules/apache/master/code 377 | │   └── ... 378 | └── site -> ../../clone/common/site/master/code 379 | ``` 380 | 381 | As shown, the master branch of all available repositories are used (as the 382 | _default_ dictates) however the module _apache_ has been overridden to use a 383 | different one. 384 | 385 | Environments will be written to the directory specified by the configuration 386 | key `environmentsdir`. 387 | 388 | ## What's a Jens run? 389 | 390 | It's an execution of jens-update, which is normally triggered by a cronjob or a 391 | systemd timer triggering `jens-update.service`. It will determine what's new, 392 | what branches have to be updated and what environments have to be 393 | created/modified/deleted. The following is an example of what's typically found 394 | in the log files after a run where there was not much to do (a hostgroup got 395 | new code in the QA branch and a new environment was created): 396 | 397 | ``` 398 | INFO Obtaining lock 'aijens' (attempt: 1)... 399 | INFO Refreshing metadata... 400 | INFO Refreshing repositories... 401 | INFO Fetching repositories inventory... 402 | INFO Refreshing bare repositories (modules) 403 | INFO New repositories: [] 404 | INFO Deleted repositories: [] 405 | INFO Cloning and expanding NEW bare repositories... 406 | INFO Expanding EXISTING bare repositories... 407 | INFO Purging REMOVED bare repositories... 408 | INFO Refreshing bare repositories (hostgroups) 409 | INFO New repositories: [] 410 | INFO Deleted repositories: [] 411 | INFO Cloning and expanding NEW bare repositories... 412 | INFO Expanding EXISTING bare repositories... 413 | INFO Updating ref '/mnt/puppet/aijens-3afegt67.cern.ch/clone/hostgroups/vocms/qa' 414 | INFO Purging REMOVED bare repositories... 415 | INFO Refreshing bare repositories (common) 416 | INFO New repositories: [] 417 | INFO Deleted repositories: [] 418 | INFO Cloning and expanding NEW bare repositories... 419 | INFO Expanding EXISTING bare repositories... 420 | INFO Purging REMOVED bare repositories... 421 | INFO Persisting repositories inventory... 422 | INFO Executed 'refresh_repositories' in 6287.78 ms 423 | INFO Refreshing environments... 424 | INFO New environments: ['am1286'] 425 | INFO Existing and changed environments: [] 426 | INFO Deleted environments: [] 427 | INFO Creating new environments... 428 | INFO Creating new environment 'am1286' 429 | INFO Processing modules... 430 | INFO Processing hostgroups... 431 | INFO hostgroups 'aimon' overridden to use treeish 'am1286' 432 | INFO Processing site... 433 | INFO Processing common Hiera data... 434 | INFO Purging deleted environments... 435 | INFO Recreating changed environments... 436 | INFO Refreshing not changed environments... 437 | INFO Executed 'refresh_environments' in 1395.03 ms 438 | INFO Releasing lock 'aijens'... 439 | INFO Done 440 | ``` 441 | 442 | ## Installation, configuration and deployment 443 | 444 | See [INSTALL.md](INSTALL.md) 445 | 446 | ## Contributing 447 | 448 | As mentioned before, Jens, as it is now, is very coupled to the Puppet 449 | deployment that it was designed to work with. Probably the main goal we're 450 | trying to achieve making it free software is to try to attract the interest of 451 | more Puppeteers so we can all together improve the tool and make it useful for 452 | as many Puppet installations out there as we possibly can. 453 | 454 | Hence, we'd be more than happy to get contributions of any kind! Feel free to 455 | submit bug reports and pull requests via 456 | [Github](https://github.com/cernops/jens). 457 | 458 | There's also a [DEVELOPERS.md](DEVELOPERS.md) that explains how to run the 459 | testsuite that is available for developers. 460 | 461 | ## Authors 462 | 463 | Jens has been written and it's currently being maintained by [Nacho 464 | Barrientos](https://cern.ch/nacho), however it has been designed the way it is 465 | and has evolved thanks to the feedback provided by staff of the CERN IT 466 | department. 467 | 468 | ## License 469 | 470 | See COPYING 471 | 472 | ## Etymology 473 | 474 | Jens was named after [M. Jens Vigen](https://twitter.com/jensvigen), CERN's 475 | librarian. 476 | -------------------------------------------------------------------------------- /bin/.pylintrc: -------------------------------------------------------------------------------- 1 | [MASTER] 2 | init-hook='import sys; sys.path.extend([".."])' -------------------------------------------------------------------------------- /bin/jens-config: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # Copyright (C) 2014, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | import os 10 | import sys 11 | import argparse 12 | import logging 13 | 14 | from jens.settings import Settings 15 | from jens.errors import JensError, JensConfigError 16 | 17 | def parse_cmdline_args(): 18 | """Parses command line parameters.""" 19 | parser = argparse.ArgumentParser() 20 | parser.add_argument('-c', '--config', 21 | help="Configuration file path (defaults to '/etc/jens/main.conf'", 22 | default="/etc/jens/main.conf") 23 | parser.add_argument('-s', '--section', 24 | help="Section of the configuration file (defaults to 'main')", 25 | default="main") 26 | parser.add_argument('-k', '--key', 27 | help="Key of the configuration file") 28 | opts = parser.parse_args() 29 | if opts.key is None: 30 | raise JensConfigError("--key is mandatory") 31 | return opts 32 | 33 | def get_config(section, key): 34 | settings = Settings() 35 | raw_config = settings.config 36 | if section in raw_config: 37 | if key in raw_config[section]: 38 | return raw_config[section][key] 39 | raise JensError("%s:%s not found in the configuration file" % \ 40 | (section, key)) 41 | 42 | def main(): 43 | """Application entrypoint.""" 44 | try: 45 | opts = parse_cmdline_args() 46 | except JensError as error: 47 | logging.error("Wrong command line args (%s)\n" % error) 48 | return 1 49 | 50 | settings = Settings() 51 | try: 52 | settings.parse_config(opts.config) 53 | except JensConfigError as error: 54 | logging.error(error) 55 | return 2 56 | 57 | try: 58 | value = get_config(opts.section, opts.key) 59 | except JensError as error: 60 | logging.error(str(error) + "\n") 61 | return 3 62 | 63 | print(value) 64 | return 0 65 | 66 | if __name__ == '__main__': 67 | sys.exit(main()) 68 | -------------------------------------------------------------------------------- /bin/jens-gc: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2014, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | import sys 10 | import os 11 | import logging 12 | import argparse 13 | 14 | import jens.git_wrapper as git 15 | from jens.errors import JensError, JensLockError 16 | from jens.errors import JensConfigError, JensGitError 17 | from jens.settings import Settings 18 | from jens.maintenance import validate_directories 19 | from jens.locks import JensLockFactory 20 | from jens.decorators import timed 21 | 22 | def parse_cmdline_args(): 23 | """Parses command line parameters.""" 24 | parser = argparse.ArgumentParser() 25 | parser.add_argument('-c', '--config', 26 | help="Configuration file path " 27 | "(defaults to '/etc/jens/main.conf'", 28 | default="/etc/jens/main.conf") 29 | parser.add_argument('-g', '--aggressive', 30 | action="store_true", 31 | help="Do it aggressively") 32 | parser.add_argument('-b', '--bare', 33 | action="store_true", 34 | help="Clean up bare repositories") 35 | parser.add_argument('-l', '--clones', 36 | action="store_true", 37 | help="Clean up repositories clones") 38 | parser.add_argument('-a', '--all', 39 | action="store_true", 40 | help="Clean up everything") 41 | return parser.parse_args() 42 | 43 | @timed 44 | def gc_bares(opts): 45 | settings = Settings() 46 | processed = 0 47 | for partition in ("modules", "hostgroups", "common"): 48 | base_path = settings.BAREDIR + "/%s" % partition 49 | for repository in os.listdir(base_path): 50 | try: 51 | repository_path = base_path + "/%s" % repository 52 | git.gc(repository_path, aggressive=opts.aggressive) 53 | processed = processed + 1 54 | except JensGitError as error: 55 | logging.error("Failed run git-gc on bare repo %s (%s)", 56 | repository, error) 57 | return processed 58 | 59 | @timed 60 | def gc_clones(opts): 61 | settings = Settings() 62 | processed = 0 63 | for partition in ("modules", "hostgroups", "common"): 64 | base_path = settings.CLONEDIR + "/%s" % partition 65 | for element in os.listdir(base_path): 66 | element_path = base_path + "/%s" % element 67 | for branch in os.listdir(element_path): 68 | branch_path = element_path + "/%s" % branch 69 | try: 70 | git.gc(branch_path, aggressive=opts.aggressive) 71 | processed = processed + 1 72 | except JensGitError as error: 73 | logging.error("Failed run git-gc on clone %s (%s)", 74 | branch_path, error) 75 | return processed 76 | 77 | def main(): 78 | """Application entrypoint.""" 79 | opts = parse_cmdline_args() 80 | 81 | settings = Settings("jens-gc") 82 | try: 83 | settings.parse_config(opts.config) 84 | except JensConfigError as error: 85 | logging.error(error) 86 | return 2 87 | 88 | try: 89 | validate_directories() 90 | except JensError as error: 91 | logging.error("Failed to validate directories (%s)", error) 92 | return 3 93 | 94 | try: 95 | with JensLockFactory.make_lock(tries=10, waittime=10): 96 | if opts.bare or opts.all: 97 | logging.info("GCing bare repositories...") 98 | processed_count = gc_bares(opts) 99 | logging.info("Done (%d repositories cleaned up)", 100 | processed_count) 101 | 102 | if opts.clones or opts.all: 103 | logging.info("GCing clones...") 104 | processed_count = gc_clones(opts) 105 | logging.info("Done (%d repositories cleaned up)", 106 | processed_count) 107 | except JensLockError as error: 108 | logging.error("Locking failed (%s)", error) 109 | return 50 110 | 111 | logging.info("Done") 112 | return 0 113 | 114 | if __name__ == '__main__': 115 | sys.exit(main()) 116 | -------------------------------------------------------------------------------- /bin/jens-gitlab-producer-runner: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2015, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | import sys 10 | import logging 11 | import argparse 12 | 13 | from jens.errors import JensConfigError 14 | from jens.settings import Settings 15 | from jens.webapps.gitlabproducer import app 16 | 17 | def parse_cmdline_args(): 18 | """Parses command line parameters.""" 19 | parser = argparse.ArgumentParser() 20 | parser.add_argument('-c', '--config', 21 | help="Configuration file path " 22 | "(defaults to '/etc/jens/main.conf'", 23 | default="/etc/jens/main.conf") 24 | parser.add_argument('-p', '--port', 25 | help="Port number to listen to", type=int, 26 | default=8000) 27 | return parser.parse_args() 28 | 29 | def main(): 30 | """Application entrypoint.""" 31 | opts = parse_cmdline_args() 32 | 33 | settings = Settings("jens-gitlab-producer") 34 | try: 35 | settings.parse_config(opts.config) 36 | except JensConfigError as error: 37 | logging.error(error) 38 | return 2 39 | 40 | app.config['settings'] = settings 41 | app.run(host='0.0.0.0', port=opts.port) 42 | return 0 43 | 44 | if __name__ == '__main__': 45 | sys.exit(main()) 46 | -------------------------------------------------------------------------------- /bin/jens-purge-queue: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2016, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | import sys 10 | import logging 11 | import argparse 12 | 13 | from jens.errors import JensConfigError 14 | from jens.errors import JensMessagingError 15 | from jens.settings import Settings 16 | from jens.messaging import purge_queue 17 | 18 | def parse_cmdline_args(): 19 | """Parses command line parameters.""" 20 | parser = argparse.ArgumentParser() 21 | parser.add_argument('-c', '--config', 22 | help="Configuration file path " 23 | "(defaults to '/etc/jens/main.conf'", 24 | default="/etc/jens/main.conf") 25 | return parser.parse_args() 26 | 27 | def main(): 28 | """Application entrypoint.""" 29 | opts = parse_cmdline_args() 30 | 31 | settings = Settings("jens-purge-queue") 32 | try: 33 | settings.parse_config(opts.config) 34 | except JensConfigError as error: 35 | logging.error(error) 36 | return 2 37 | 38 | try: 39 | logging.info("Purging %s...", settings.MESSAGING_QUEUEDIR) 40 | purge_queue() 41 | except JensMessagingError as error: 42 | logging.error(error) 43 | return 10 44 | 45 | logging.info("Done") 46 | return 0 47 | 48 | if __name__ == '__main__': 49 | sys.exit(main()) 50 | -------------------------------------------------------------------------------- /bin/jens-reset: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2014, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | import os 10 | import sys 11 | import argparse 12 | import shutil 13 | import logging 14 | 15 | from jens.settings import Settings 16 | from jens.errors import JensConfigError 17 | from jens.errors import JensError, JensLockError 18 | from jens.maintenance import validate_directories 19 | from jens.locks import JensLockFactory 20 | 21 | def parse_cmdline_args(): 22 | """Parses command line parameters.""" 23 | parser = argparse.ArgumentParser() 24 | parser.add_argument('-c', '--config', 25 | help="Configuration file path " 26 | "(defaults to '/etc/jens/main.conf'", 27 | default="/etc/jens/main.conf") 28 | parser.add_argument('-y', '--yes', 29 | action="store_true", 30 | help="Please do it") 31 | return parser.parse_args() 32 | 33 | def remove_bares(): 34 | settings = Settings() 35 | for partition in ("modules", "hostgroups", "common"): 36 | basepath = settings.BAREDIR + "/%s" % partition 37 | for element in os.listdir(basepath): 38 | shutil.rmtree(basepath + "/%s" % element) 39 | 40 | def remove_clones(): 41 | settings = Settings() 42 | for partition in ("modules", "hostgroups", "common"): 43 | basepath = settings.CLONEDIR + "/%s" % partition 44 | for element in os.listdir(basepath): 45 | shutil.rmtree(basepath + "/%s" % element) 46 | 47 | def remove_environments(): 48 | settings = Settings() 49 | basepath = settings.ENVIRONMENTSDIR 50 | for environment in os.listdir(basepath): 51 | shutil.rmtree(basepath + "/%s" % environment) 52 | 53 | def remove_cache(): 54 | remove_environments_cache() 55 | remove_inventory_cache() 56 | 57 | def remove_inventory_cache(): 58 | settings = Settings() 59 | path = settings.CACHEDIR + "/repositories" 60 | if os.path.exists(path): 61 | os.remove(path) 62 | 63 | def remove_environments_cache(): 64 | settings = Settings() 65 | basepath = settings.CACHEDIR + "/environments" 66 | for environment in os.listdir(basepath): 67 | os.remove(basepath + "/%s" % environment) 68 | 69 | def main(): 70 | """Application entrypoint.""" 71 | opts = parse_cmdline_args() 72 | 73 | settings = Settings() 74 | try: 75 | settings.parse_config(opts.config) 76 | except JensConfigError as error: 77 | logging.error(error) 78 | return 2 79 | 80 | try: 81 | validate_directories() 82 | except JensError as error: 83 | logging.error("Failed to validate directories (%s)", error) 84 | return 3 85 | 86 | if not opts.yes: 87 | logging.info("Are you sure about what you're doing? If so add --yes") 88 | return 10 89 | 90 | try: 91 | with JensLockFactory.make_lock(): 92 | logging.info("Cleaning everything up...") 93 | logging.info("Removing bare repositories...") 94 | remove_bares() 95 | logging.info("Removing clones of bare repositories") 96 | remove_clones() 97 | logging.info("Removing all Puppet environments...") 98 | remove_environments() 99 | logging.info("Removing all environments' cache...") 100 | remove_cache() 101 | except JensLockError as error: 102 | logging.error("Locking failed (%s)", error) 103 | return 50 104 | 105 | logging.info("Done -- Jens is sad now") 106 | return 0 107 | 108 | if __name__ == '__main__': 109 | sys.exit(main()) 110 | -------------------------------------------------------------------------------- /bin/jens-stats: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2014, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | import os 10 | import sys 11 | import argparse 12 | import logging 13 | import pprint 14 | 15 | from jens.settings import Settings 16 | from jens.errors import JensConfigError 17 | from jens.errors import JensError, JensMessagingError 18 | from jens.maintenance import validate_directories 19 | from jens.reposinventory import get_inventory 20 | from jens.messaging import count_pending_hints 21 | 22 | def parse_cmdline_args(): 23 | """Parses command line parameters.""" 24 | parser = argparse.ArgumentParser() 25 | parser.add_argument('-c', '--config', 26 | help="Configuration file path " 27 | "(defaults to '/etc/jens/main.conf'", 28 | default="/etc/jens/main.conf") 29 | parser.add_argument('-r', '--repositories', 30 | action="store_true", 31 | help="Shows stats about repositories") 32 | parser.add_argument('-e', '--environment', 33 | action="store_true", 34 | help="Shows stats about environments") 35 | parser.add_argument('-i', '--inventory', 36 | action="store_true", 37 | help="Shows inventory") 38 | parser.add_argument('-q', '--queues', 39 | action="store_true", 40 | help="Shows stats about queues") 41 | parser.add_argument('-a', '--all', 42 | action="store_true", 43 | help="Shows everything") 44 | return parser.parse_args() 45 | 46 | def get_bare_repositories(): 47 | settings = Settings() 48 | result = {} 49 | result['modules'] = [attach_information("modules", element) \ 50 | for element in sorted(os.listdir(settings.BAREDIR + "/modules"))] 51 | result['hostgroups'] = [attach_information("hostgroups", element) \ 52 | for element in sorted(os.listdir(settings.BAREDIR + "/hostgroups"))] 53 | result['common'] = [attach_information("common", element) \ 54 | for element in os.listdir(settings.BAREDIR + "/common")] 55 | return result 56 | 57 | def attach_information(partition, element): 58 | branches = get_branches(partition, element) 59 | master_size = get_size(partition, element, "master") 60 | return "%s (%s) [%s]" % (element, branches, master_size) 61 | 62 | def get_branches(partition, element): 63 | settings = Settings() 64 | path = "%s/%s/%s" % (settings.CLONEDIR, partition, element) 65 | branches = os.listdir(path) 66 | return "%s" % ",".join(sorted(branches)) 67 | 68 | def get_size(partition, element, branch): 69 | settings = Settings() 70 | path = "%s/%s/%s/%s" % \ 71 | (settings.CLONEDIR, partition, element, branch) 72 | size = 0 73 | for path, _, files in os.walk(path): 74 | for _file in files: 75 | fullpath = os.path.join(path, _file) 76 | if os.path.isfile(fullpath): 77 | size += os.path.getsize(fullpath) 78 | return fmt_bytes(size) 79 | 80 | # http://stackoverflow.com/questions/1094841/ 81 | # reusable-library-to-get-human-readable-version-of-file-size 82 | def fmt_bytes(num): 83 | for unit in ['B', 'KiB', 'MiB', 'GiB']: 84 | if num < 1024.0: 85 | return "%3.1f%s" % (num, unit) 86 | num /= 1024.0 87 | return "%3.1f%s" % (num, 'TB') 88 | 89 | def get_puppet_environments(): 90 | settings = Settings() 91 | return os.listdir(settings.ENVIRONMENTSDIR) 92 | 93 | def get_environments_in_cache(): 94 | settings = Settings() 95 | return os.listdir(settings.CACHEDIR + "/environments") 96 | 97 | def get_environments_in_metadata(): 98 | settings = Settings() 99 | return __listdir_onlyyaml(settings.ENV_METADATADIR) 100 | 101 | def __listdir_onlyyaml(directory): 102 | return [_file for _file in os.listdir(directory) 103 | if _file.endswith(".yaml")] 104 | 105 | def main(): 106 | """Application entrypoint.""" 107 | opts = parse_cmdline_args() 108 | 109 | settings = Settings() 110 | try: 111 | settings.parse_config(opts.config) 112 | except JensConfigError as error: 113 | logging.error(error) 114 | return 2 115 | 116 | try: 117 | validate_directories() 118 | except JensError as error: 119 | logging.error("Failed to validate directories (%s)", error) 120 | return 3 121 | 122 | if opts.repositories or opts.all: 123 | bares = get_bare_repositories() 124 | logging.info("There are %d modules:", len(bares['modules'])) 125 | for module in bares['modules']: 126 | logging.info("\t- %s", module) 127 | logging.info("There are %d hostgroups:", len(bares['hostgroups'])) 128 | for hostgroup in bares['hostgroups']: 129 | logging.info("\t- %s", hostgroup) 130 | 131 | if opts.environment or opts.all: 132 | environments = {} 133 | environments['synchronized'] = get_puppet_environments() 134 | environments['declared'] = get_environments_in_metadata() 135 | environments['cached'] = get_environments_in_cache() 136 | 137 | for _type, _list in environments.items(): 138 | logging.info("There are %d %s environments", len(_list), _type) 139 | for environment in sorted(_list): 140 | logging.info("\t - %s ", environment) 141 | 142 | if opts.inventory or opts.all: 143 | inventory = get_inventory() 144 | pprinter = pprint.PrettyPrinter() 145 | pprinter.pprint(inventory) 146 | 147 | if opts.queues or opts.all: 148 | try: 149 | count = count_pending_hints() 150 | logging.info("There are '%d' messages in the hints queue", count) 151 | except JensMessagingError as error: 152 | logging.error(error) 153 | 154 | return 0 155 | 156 | if __name__ == '__main__': 157 | sys.exit(main()) 158 | -------------------------------------------------------------------------------- /bin/jens-update: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2014, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | import sys 10 | import logging 11 | import argparse 12 | 13 | from jens.errors import JensError, JensLockError 14 | from jens.errors import JensConfigError, JensRepositoriesError 15 | from jens.errors import JensMessagingError 16 | from jens.errors import JensLockExistsError 17 | from jens.settings import Settings 18 | from jens.repos import refresh_repositories 19 | from jens.maintenance import refresh_metadata 20 | from jens.maintenance import validate_directories 21 | from jens.locks import JensLockFactory 22 | from jens.environments import refresh_environments 23 | from jens.messaging import fetch_update_hints 24 | 25 | def parse_cmdline_args(): 26 | """Parses command line parameters.""" 27 | parser = argparse.ArgumentParser() 28 | parser.add_argument('-c', '--config', 29 | help="Configuration file path " 30 | "(defaults to '/etc/jens/main.conf'", 31 | default="/etc/jens/main.conf") 32 | parser.add_argument('-p', '--poll', 33 | help="Force POLL mode, regardless of what's " 34 | "in the config file", 35 | action='store_true') 36 | return parser.parse_args() 37 | 38 | def main(): 39 | """Application entrypoint.""" 40 | opts = parse_cmdline_args() 41 | 42 | settings = Settings("jens-update") 43 | try: 44 | settings.parse_config(opts.config) 45 | except JensConfigError as error: 46 | logging.error(error) 47 | return 2 48 | 49 | try: 50 | validate_directories() 51 | except JensError as error: 52 | logging.error("Failed to validate directories (%s)", error) 53 | return 3 54 | 55 | if opts.poll: 56 | settings.MODE = 'POLL' 57 | 58 | try: 59 | with JensLockFactory.make_lock(): 60 | # Update metadata 61 | logging.info("Refreshing metadata...") 62 | try: 63 | refresh_metadata() 64 | except JensError as error: 65 | logging.error(error) 66 | return 20 67 | 68 | if settings.MODE == 'ONDEMAND': 69 | try: 70 | hints = fetch_update_hints() 71 | except JensMessagingError as error: 72 | logging.error(error) 73 | return 25 74 | else: 75 | hints = None 76 | 77 | # Update repositories 78 | logging.info("Refreshing repositories...") 79 | try: 80 | repositories_deltas, inventory = \ 81 | refresh_repositories(hints) 82 | except JensRepositoriesError as error: 83 | logging.error("Failed (%s)", error) 84 | return 30 85 | 86 | # Update environments 87 | logging.info("Refreshing environments...") 88 | try: 89 | refresh_environments(repositories_deltas, inventory) 90 | except JensRepositoriesError as error: 91 | logging.error("Failed (%s)", error) 92 | return 40 93 | except JensLockExistsError as error: 94 | logging.info("Locking failed (%s)", error) 95 | return 50 96 | except JensLockError as error: 97 | logging.error("Locking failed (%s)", error) 98 | return 51 99 | 100 | logging.info("Done") 101 | return 0 102 | 103 | if __name__ == '__main__': 104 | sys.exit(main()) 105 | -------------------------------------------------------------------------------- /conf/main.conf: -------------------------------------------------------------------------------- 1 | [main] 2 | baredir = /var/lib/jens/bare 3 | clonedir = /var/lib/jens/clone 4 | environmentsdir = /var/lib/jens/environments 5 | debuglevel = INFO 6 | logdir = /var/log/jens 7 | mandatorybranches = master, 8 | environmentsmetadatadir = /var/lib/jens/metadata/environments 9 | repositorymetadatadir = /var/lib/jens/metadata/repository 10 | repositorymetadata = /var/lib/jens/metadata/repository/repositories.yaml 11 | hashprefix = commit/ 12 | directory_environments = False 13 | common_hieradata_items = datacentres, environments, hardware, operatingsystems, common.yaml 14 | mode = POLL 15 | # Jens-update won't ever delete these envs, 16 | # even if they're removed from 'environmentsmetadatadir'. 17 | # This is probably only useful to protect 'golden' environments 18 | # with no overrides as 'mandatorybranches' are never deleted. 19 | protectedenvironments = production, qa 20 | 21 | [lock] 22 | type = FILE 23 | 24 | [filelock] 25 | lockdir = /run/lock/jens 26 | 27 | [messaging] 28 | queuedir = /var/spool/jens-update 29 | 30 | [git] 31 | ssh_cmd_path = /etc/jens/myssh.sh 32 | 33 | [gitlabproducer] 34 | secret_token = placeholder -------------------------------------------------------------------------------- /examples/environments/production.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | default: master 3 | notifications: admins@example.org 4 | -------------------------------------------------------------------------------- /examples/example-repositories/common-hieradata/data/common.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | foo: bar 3 | -------------------------------------------------------------------------------- /examples/example-repositories/common-hieradata/data/environments/production.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | puppet_runinterval: 1234 3 | puppet_splay_limit: 4567 4 | -------------------------------------------------------------------------------- /examples/example-repositories/common-hieradata/data/environments/qa.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | puppet_runinterval: 2600 3 | puppet_splay_limit: 900 4 | -------------------------------------------------------------------------------- /examples/example-repositories/common-hieradata/data/hardware/vendor/sinclair.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | sssd::interactiveallowusers: zx-maintainers 3 | -------------------------------------------------------------------------------- /examples/example-repositories/common-hieradata/data/operatingsystems/RedHat/5.yaml: -------------------------------------------------------------------------------- 1 | foo: bar 2 | -------------------------------------------------------------------------------- /examples/example-repositories/common-hieradata/data/operatingsystems/RedHat/6.yaml: -------------------------------------------------------------------------------- 1 | foo: bar 2 | -------------------------------------------------------------------------------- /examples/example-repositories/common-hieradata/data/operatingsystems/RedHat/7.yaml: -------------------------------------------------------------------------------- 1 | foo: bar 2 | -------------------------------------------------------------------------------- /examples/example-repositories/hostgroup-myapp/code/files/superscript.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo "Yehaaaa" 3 | -------------------------------------------------------------------------------- /examples/example-repositories/hostgroup-myapp/code/manifests/frontend.pp: -------------------------------------------------------------------------------- 1 | class hg_myapp::frontend { 2 | 3 | class { 'dummy': } 4 | 5 | } 6 | -------------------------------------------------------------------------------- /examples/example-repositories/hostgroup-myapp/code/manifests/init.pp: -------------------------------------------------------------------------------- 1 | class hg_myapp { 2 | 3 | } 4 | -------------------------------------------------------------------------------- /examples/example-repositories/hostgroup-myapp/code/templates/someotherconfig.erb: -------------------------------------------------------------------------------- 1 | key=value 2 | -------------------------------------------------------------------------------- /examples/example-repositories/hostgroup-myapp/data/fqdns/myapp-node1.example.org.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | foo:bar 3 | -------------------------------------------------------------------------------- /examples/example-repositories/hostgroup-myapp/data/hostgroup/myapp.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | location: earth 3 | -------------------------------------------------------------------------------- /examples/example-repositories/hostgroup-myapp/data/hostgroup/myapp/frontend.yaml: -------------------------------------------------------------------------------- 1 | dummy::msg: "folks" 2 | -------------------------------------------------------------------------------- /examples/example-repositories/module-dummy/code/README.md: -------------------------------------------------------------------------------- 1 | This is a dummy module 2 | -------------------------------------------------------------------------------- /examples/example-repositories/module-dummy/code/manifests/init.pp: -------------------------------------------------------------------------------- 1 | class dummy { 2 | class {'dummy::install':} 3 | } 4 | -------------------------------------------------------------------------------- /examples/example-repositories/module-dummy/code/manifests/install.pp: -------------------------------------------------------------------------------- 1 | class dummy::install ($msg = "foo") { 2 | notify{"Hello ${msg}":} 3 | } 4 | -------------------------------------------------------------------------------- /examples/example-repositories/module-dummy/code/templates/config.erb: -------------------------------------------------------------------------------- 1 | A very useful configuration file 2 | -------------------------------------------------------------------------------- /examples/example-repositories/module-dummy/data/jens.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | this: is 3 | an: example 4 | -------------------------------------------------------------------------------- /examples/example-repositories/site/code/site.pp: -------------------------------------------------------------------------------- 1 | # This is not working code, just an example on how to implement 2 | # the hostgroup autoloading :) 3 | 4 | if $hostarray[0] { 5 | $encgroup_0 = $hostarray[0] 6 | } 7 | if $hostarray[1] { 8 | $encgroup_1 = $hostarray[1] 9 | } 10 | # An so on... 11 | 12 | $hostgroup_prefix = hiera('hostgroup_prefix', 'hg_') 13 | 14 | # All your base are belong to us. 15 | class{ 'base': } 16 | 17 | $encgroup_0hg = "${hostgroup_prefix}${encgroup_0}" 18 | if is_module_path($encgroup_0hg) { 19 | include "$encgroup_0hg" 20 | } 21 | if $encgroup_1 { 22 | if is_module_path("${encgroup_0hg}::${encgroup_1}") { 23 | include "${encgroup_0hg}::${encgroup_1}" 24 | } 25 | } 26 | # And so on... 27 | -------------------------------------------------------------------------------- /examples/hiera.yaml: -------------------------------------------------------------------------------- 1 | # Installed with puppet. 2 | --- 3 | :backends: 4 | - yaml 5 | 6 | :yaml: 7 | :datadir: /etc/puppet 8 | 9 | :hierarchy: 10 | - environments/%{::foreman_env}/hieradata/fqdns/%{::encgroup_0}/%{::fqdn} 11 | - environments/%{::foreman_env}/hieradata/hostgroups/%{::encgroup_0}/%{::encgroup_0}/%{::encgroup_1}/%{::encgroup_2}/%{::encgroup_3}/%{::encgroup_4} 12 | - environments/%{::foreman_env}/hieradata/hostgroups/%{::encgroup_0}/%{::encgroup_0}/%{::encgroup_1}/%{::encgroup_2}/%{::encgroup_3} 13 | - environments/%{::foreman_env}/hieradata/hostgroups/%{::encgroup_0}/%{::encgroup_0}/%{::encgroup_1}/%{::encgroup_2} 14 | - environments/%{::foreman_env}/hieradata/hostgroups/%{::encgroup_0}/%{::encgroup_0}/%{::encgroup_1} 15 | - environments/%{::foreman_env}/hieradata/hostgroups/%{::encgroup_0}/%{::encgroup_0}/operatingsystems/%{::osfamily}/%{::operatingsystemmajorrelease} 16 | - environments/%{::foreman_env}/hieradata/hostgroups/%{::encgroup_0}/%{::encgroup_0} 17 | - environments/%{::foreman_env}/hieradata/datacentres/%{::datacentre} 18 | - environments/%{::foreman_env}/hieradata/operatingsystems/%{::osfamily}/%{::operatingsystemmajorrelease} 19 | - environments/%{::foreman_env}/hieradata/environments/%{::foreman_env} 20 | - environments/%{::foreman_env}/hieradata/module_names/%{module_name}/%{module_name} 21 | - environments/%{::foreman_env}/hieradata/hardware/vendor/%{::cern_hwvendor} 22 | 23 | - environments/%{::foreman_env}/hieradata/common 24 | - hieradata/hostgroups/%{::hostgroup} 25 | - hieradata/common 26 | 27 | -------------------------------------------------------------------------------- /examples/repositories/repositories.yaml: -------------------------------------------------------------------------------- 1 | repositories: 2 | common: 3 | hieradata: file:///tmp/example-repositories/common-hieradata 4 | site: file:///tmp/example-repositories/site 5 | hostgroups: 6 | myapp: file:///tmp/example-repositories/hostgroup-myapp 7 | modules: 8 | dummy: file:///tmp/example-repositories/module-dummy 9 | -------------------------------------------------------------------------------- /jens-tmpfiles.conf: -------------------------------------------------------------------------------- 1 | d /run/lock/jens 0700 jens jens - 2 | -------------------------------------------------------------------------------- /jens/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | -------------------------------------------------------------------------------- /jens/configfile.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | CONFIG_GRAMMAR = """ 9 | [main] 10 | baredir = string(default='/var/lib/jens/bare') 11 | clonedir = string(default='/var/lib/jens/clone') 12 | environmentsdir = string(default='/var/lib/jens/environments') 13 | debuglevel = option('INFO', 'DEBUG', 'ERROR', default='INFO') 14 | logdir = string(default='/var/log/jens') 15 | mandatorybranches = list(default=list("master", "qa")) 16 | protectedenvironments = list(default=list()) 17 | environmentsmetadatadir = string(default='/var/lib/jens/metadata/environments') 18 | repositorymetadatadir = string(default='/var/lib/jens/metadata/repository') 19 | repositorymetadata = string(default='/var/lib/jens/metadata/repository/repositories.yaml') 20 | cachedir = string(default='/var/lib/jens/cache') 21 | hashprefix = string(default='commit/') 22 | directory_environments = boolean(default=False) 23 | common_hieradata_items = list(default=list()) 24 | mode = option('POLL', 'ONDEMAND', default='POLL') 25 | [lock] 26 | type = option('DISABLED', 'FILE', default='FILE') 27 | name = string(default='jens') 28 | [filelock] 29 | lockdir = string(default='/run/lock/jens') 30 | [messaging] 31 | queuedir = string(default='/var/spool/jens-update') 32 | [git] 33 | ssh_cmd_path = string(default=None) 34 | [gitlabproducer] 35 | secret_token = string(default=None) 36 | """ 37 | -------------------------------------------------------------------------------- /jens/decorators.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import logging 10 | import os 11 | from functools import wraps 12 | from time import time 13 | 14 | import git.exc 15 | 16 | from jens.errors import JensGitError 17 | from jens.settings import Settings 18 | 19 | def timed(func): 20 | @wraps(func) 21 | def wrapper(*args, **kwargs): 22 | start = time() 23 | result = func(*args, **kwargs) 24 | elapsed = time() - start 25 | logging.info("Executed '%s' in %.2f ms", func.__name__, elapsed*1000) 26 | return result 27 | return wrapper 28 | 29 | def git_exec(func): 30 | @wraps(func) 31 | def wrapper(**w_kwargs): 32 | settings = Settings() 33 | ssh_cmd_path = settings.SSH_CMD_PATH 34 | 35 | if ssh_cmd_path: 36 | os.environ['GIT_SSH'] = ssh_cmd_path 37 | 38 | args = w_kwargs["args"] 39 | kwargs = w_kwargs["kwargs"] 40 | name = w_kwargs["name"] 41 | 42 | logging.debug("Executing git %s %s %s", name, args, kwargs) 43 | 44 | try: 45 | res = func(*args, **kwargs) 46 | except (git.exc.GitCommandError, git.exc.GitCommandNotFound) as error: 47 | raise JensGitError("Couldn't execute %s (%s)" % 48 | (error.command, error.stderr)) 49 | except git.exc.NoSuchPathError as error: 50 | raise JensGitError("No such path %s" % error) 51 | except git.exc.InvalidGitRepositoryError as error: 52 | raise JensGitError("Not a git repository: %s" % error) 53 | except AssertionError as error: 54 | raise JensGitError("Git operation failed: %s" % error) 55 | return res 56 | 57 | return wrapper 58 | -------------------------------------------------------------------------------- /jens/errors.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | class JensError(Exception): 9 | pass 10 | 11 | class JensConfigError(JensError): 12 | pass 13 | 14 | class JensMessagingError(JensError): 15 | pass 16 | 17 | class JensRepositoriesError(JensError): 18 | pass 19 | 20 | class JensEnvironmentsError(JensError): 21 | pass 22 | 23 | class JensGitError(JensError): 24 | pass 25 | 26 | class JensLockError(JensError): 27 | pass 28 | 29 | class JensLockExistsError(JensLockError): 30 | pass 31 | -------------------------------------------------------------------------------- /jens/git_wrapper.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014-2016, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import git 10 | import logging 11 | from jens.decorators import git_exec 12 | 13 | def hash_object(path): 14 | args = [path] 15 | kwargs = {} 16 | logging.debug("Hashing object %s", path) 17 | 18 | @git_exec 19 | def hash_object_exec(*args, **kwargs): 20 | return git.cmd.Git().hash_object(*args, **kwargs).strip() 21 | 22 | return hash_object_exec(name='hash-object', args=args, kwargs=kwargs) 23 | 24 | # pylint: disable=invalid-name 25 | # (too short but all match Git command names) 26 | def gc(repository_path, aggressive=False): 27 | args = [] 28 | kwargs = {"quiet": True, "aggressive": aggressive} 29 | logging.debug("Collecting garbage in %s", repository_path) 30 | 31 | @git_exec 32 | def gc_exec(*args, **kwargs): 33 | repo = git.Repo(repository_path, odbt=git.GitCmdObjectDB) 34 | repo.git.gc(*args, **kwargs) 35 | 36 | gc_exec(name='gc', args=args, kwargs=kwargs) 37 | 38 | def clone(repository_path, url, bare=False, shared=False, branch=None): 39 | args = [url, repository_path] 40 | kwargs = {"no-hardlinks": True, "shared": shared, 41 | "odbt": git.GitCmdObjectDB} 42 | logging.debug("Cloning from %s to %s", url, repository_path) 43 | if bare is True: 44 | kwargs["bare"] = True 45 | kwargs["mirror"] = True 46 | if branch is not None: 47 | kwargs["branch"] = branch 48 | 49 | @git_exec 50 | def clone_exec(*args, **kwargs): 51 | git.Repo.clone_from(*args, **kwargs) 52 | 53 | clone_exec(name='clone', args=args, kwargs=kwargs) 54 | 55 | def fetch(repository_path, prune=False): 56 | args = [] 57 | kwargs = {"no-tags": True, "prune": prune} 58 | logging.debug("Fetching new refs in %s", repository_path) 59 | 60 | @git_exec 61 | def fetch_exec(*args, **kwargs): 62 | repo = git.Repo(repository_path, odbt=git.GitCmdObjectDB) 63 | repo.remotes.origin.fetch(*args, **kwargs) 64 | 65 | fetch_exec(name='fetch', args=args, kwargs=kwargs) 66 | 67 | def reset(repository_path, treeish, hard=False): 68 | args = [treeish] 69 | kwargs = {"hard": hard} 70 | logging.debug("Resetting %s to %s", repository_path, treeish) 71 | 72 | @git_exec 73 | def reset_exec(*args, **kwargs): 74 | repo = git.Repo(repository_path, odbt=git.GitCmdObjectDB) 75 | repo.git.reset(*args, **kwargs) 76 | 77 | reset_exec(name='reset', args=args, kwargs=kwargs) 78 | 79 | def get_refs(repository_path): 80 | args = [] 81 | kwargs = {} 82 | 83 | @git_exec 84 | def get_refs_exec(*args, **kwargs): 85 | repo = git.Repo(repository_path, odbt=git.GitCmdObjectDB) 86 | return dict((h.name, h.commit.hexsha) for h in repo.heads) 87 | 88 | return get_refs_exec(name='show-ref', args=args, kwargs=kwargs) 89 | 90 | def rev_parse(repository_path, ref, short=False): 91 | args = [ref] 92 | kwargs = {"short": short} 93 | 94 | @git_exec 95 | def rev_parse_exec(*args, **kwargs): 96 | repo = git.Repo(repository_path, odbt=git.GitCmdObjectDB) 97 | return repo.git.rev_parse(*args, **kwargs) 98 | 99 | return rev_parse_exec(name='rev-parse', args=args, kwargs=kwargs) 100 | 101 | def get_head(repository_path, short=False): 102 | args = [] 103 | kwargs = {} 104 | logging.debug("Getting HEAD of %s", repository_path) 105 | 106 | @git_exec 107 | def get_head_exec(*args, **kwargs): 108 | repo = git.Repo(repository_path, odbt=git.GitCmdObjectDB) 109 | sha = repo.head.commit.hexsha 110 | if short: 111 | sha = rev_parse(repository_path, sha, short=True) 112 | return sha 113 | 114 | return get_head_exec(name='get-head', args=args, kwargs=kwargs) 115 | -------------------------------------------------------------------------------- /jens/locks.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import logging 10 | import time 11 | from jens.settings import Settings 12 | 13 | from jens.errors import JensLockError, JensLockExistsError 14 | 15 | class JensLockFactory(object): 16 | @staticmethod 17 | def make_lock(tries=1, waittime=10): 18 | settings = Settings() 19 | if settings.LOCK_TYPE == 'FILE': 20 | return JensFileLock(tries, waittime) 21 | elif settings.LOCK_TYPE == 'DISABLED': 22 | logging.warning("Danger zone: no locking has been configured!") 23 | return JensDumbLock(tries, waittime) 24 | else: # Shouldn't ever happen, config is validated 25 | raise JensLockError("Unknown lock type '%s'", settings.LOCK_TYPE) 26 | 27 | class JensLock(object): 28 | def __init__(self, tries, waittime): 29 | self.settings = Settings() 30 | self.tries = tries 31 | self.waittime = waittime 32 | 33 | def __enter__(self): 34 | for attempt in range(1, self.tries+1): 35 | logging.info("Obtaining lock '%s' (attempt: %d)...", 36 | self.settings.LOCK_NAME, attempt) 37 | try: 38 | self.obtain_lock() 39 | logging.debug("Lock acquired") 40 | return self 41 | except JensLockExistsError as error: 42 | if attempt == self.tries: 43 | raise error 44 | else: 45 | logging.debug("Couldn't lock (%s). Sleeping for %d seconds...", 46 | error, self.waittime) 47 | time.sleep(self.waittime) 48 | 49 | def __exit__(self, e_type, e_value, e_traceback): 50 | logging.info("Releasing lock '%s'...", self.settings.LOCK_NAME) 51 | self.release_lock() 52 | 53 | def renew(self, ttl=10): 54 | if ttl <= 0: 55 | logging.warning("Invalid new TTL, resetting to 1 by default") 56 | ttl = 1 57 | logging.info("Setting '%s' lock TTL to %d secs...", 58 | self.settings.LOCK_NAME, ttl) 59 | self.renew_lock(ttl) 60 | 61 | def obtain_lock(self): 62 | raise NotImplementedError('You are not meant to instantiate this class') 63 | 64 | def release_lock(self): 65 | raise NotImplementedError('You are not meant to instantiate this class') 66 | 67 | def renew_lock(self, ttl): 68 | raise NotImplementedError('You are not meant to instantiate this class') 69 | 70 | class JensFileLock(JensLock): 71 | def __init__(self, tries, waittime): 72 | super(JensFileLock, self).__init__(tries, waittime) 73 | self.lockfile = None 74 | 75 | def obtain_lock(self): 76 | lockfile_path = self.__get_lock_file_path() 77 | try: 78 | self.lockfile = open(lockfile_path, "w") 79 | except IOError as error: 80 | raise JensLockError("Can't open lock file for writing (%s)" % error) 81 | 82 | import fcntl 83 | try: 84 | fcntl.flock(self.lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB) 85 | except IOError as error: 86 | raise JensLockExistsError("Lock already taken") 87 | 88 | def release_lock(self): 89 | # Nothing to do, the OS will close the FDs after finishing. 90 | pass 91 | 92 | def renew_lock(self, ttl): 93 | # Nothing to do, local lock 94 | pass 95 | 96 | def __get_lock_file_path(self): 97 | return self.settings.FILELOCK_LOCKDIR + "/%s" % self.settings.LOCK_NAME 98 | 99 | class JensDumbLock(JensLock): 100 | def obtain_lock(self): 101 | pass 102 | 103 | def release_lock(self): 104 | pass 105 | 106 | def renew_lock(self, ttl): 107 | pass 108 | -------------------------------------------------------------------------------- /jens/maintenance.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import os 10 | import logging 11 | import fcntl 12 | 13 | from jens.settings import Settings 14 | import jens.git_wrapper as git 15 | from jens.decorators import timed 16 | from jens.errors import JensError, JensGitError 17 | 18 | @timed 19 | def refresh_metadata(): 20 | _refresh_environments() 21 | _refresh_repositories() 22 | 23 | def validate_directories(): 24 | logging.info("Validating directories...") 25 | settings = Settings() 26 | directories = [settings.BAREDIR, 27 | settings.CLONEDIR, 28 | settings.CACHEDIR, 29 | settings.CACHEDIR + "/environments", 30 | settings.ENVIRONMENTSDIR, 31 | settings.REPO_METADATADIR, 32 | settings.ENV_METADATADIR] 33 | 34 | for partition in ("modules", "hostgroups", "common"): 35 | directories.append(settings.BAREDIR + "/%s" % partition) 36 | directories.append(settings.CLONEDIR + "/%s" % partition) 37 | 38 | for directory in directories: 39 | _validate_directory(directory) 40 | 41 | if settings.LOCK_TYPE == 'FILE': 42 | _validate_directory(settings.FILELOCK_LOCKDIR) 43 | 44 | if not os.path.exists(settings.ENV_METADATADIR + "/.git"): 45 | raise JensError("%s not initialized (no Git repository found)" % 46 | settings.ENV_METADATADIR) 47 | 48 | if not os.path.exists(settings.REPO_METADATA): 49 | raise JensError("Couldn't find metadata of repositories (%s not initialized)" % 50 | settings.REPO_METADATADIR) 51 | 52 | def _validate_directory(directory): 53 | logging.debug("Validating directory '%s'...", directory) 54 | try: 55 | os.stat(directory) 56 | except OSError: 57 | raise JensError("Directory '%s' does not exist" % directory) 58 | if not os.access(directory, os.W_OK): 59 | raise JensError("Cannot read or write on directory '%s'" % directory) 60 | 61 | def _refresh_environments(): 62 | settings = Settings() 63 | logging.debug("Refreshing environment metadata...") 64 | path = settings.ENV_METADATADIR 65 | try: 66 | git.fetch(path) 67 | git.reset(path, "origin/master", hard=True) 68 | except JensGitError as error: 69 | raise JensError("Couldn't refresh environments metadata (%s)" % error) 70 | 71 | def _refresh_repositories(): 72 | settings = Settings() 73 | logging.debug("Refreshing repositories metadata...") 74 | path = settings.REPO_METADATADIR 75 | try: 76 | git.fetch(path) 77 | try: 78 | metadata = open(settings.REPO_METADATA, 'r') 79 | except IOError as error: 80 | raise JensError("Could not open '%s' to put a lock on it" % 81 | settings.REPO_METADATA) 82 | # jens-gitlab-producer collaborates with jens-update asynchronously 83 | # so have to make sure that exclusive access to the file when writing 84 | # is guaranteed. Of course, the reader will have to implement the same 85 | # protocol on the other end. 86 | try: 87 | logging.info("Trying to acquire a lock to refresh the metadata...") 88 | fcntl.flock(metadata, fcntl.LOCK_EX) 89 | logging.debug("Lock acquired") 90 | except IOError as error: 91 | metadata.close() 92 | raise JensError("Could not lock '%s'" % settings.REPO_METADATA) 93 | git.reset(path, "origin/master", hard=True) 94 | try: 95 | logging.debug("Trying to release the lock used to refresh the metadata...") 96 | fcntl.flock(metadata, fcntl.LOCK_UN) 97 | logging.debug("Lock released") 98 | except IOError as error: 99 | raise JensError("Could not unlock '%s'" % settings.REPO_METADATA) 100 | finally: 101 | metadata.close() 102 | except JensGitError as error: 103 | raise JensError("Couldn't refresh repositories metadata (%s)" % error) 104 | -------------------------------------------------------------------------------- /jens/messaging.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2015, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import logging 10 | import pickle 11 | 12 | from datetime import datetime 13 | 14 | from dirq.queue import Queue 15 | from dirq.queue import QueueLockError, QueueError 16 | 17 | from jens.errors import JensMessagingError 18 | from jens.decorators import timed 19 | from jens.settings import Settings 20 | from functools import reduce 21 | 22 | MSG_SCHEMA = {'time': 'string', 'data': 'binary'} 23 | 24 | # Ex (after unpickling 'data'): 25 | # {'time': '2015-12-10T14:06:35.339550', 'data': {'modules': ['m1']}} 26 | 27 | @timed 28 | def fetch_update_hints(): 29 | hints = {} 30 | logging.info("Getting and processing hints...") 31 | try: 32 | messages = _fetch_all_messages() 33 | except Exception as error: 34 | raise JensMessagingError("Could not retrieve messages (%s)" % error) 35 | 36 | logging.info("%d messages found", len(messages)) 37 | hints = _validate_and_merge_messages(messages) 38 | return hints 39 | 40 | def enqueue_hint(partition, name): 41 | if partition not in ("modules", "hostgroups", "common"): 42 | raise JensMessagingError("Unknown partition '%s'" % partition) 43 | hint = {'time': datetime.now().isoformat(), 44 | 'data': pickle.dumps({partition: [name]})} 45 | 46 | _queue_item(hint) 47 | logging.info("Hint '%s/%s' added to the queue", partition, name) 48 | 49 | def _queue_item(item): 50 | settings = Settings() 51 | try: 52 | queue = Queue(settings.MESSAGING_QUEUEDIR, schema=MSG_SCHEMA) 53 | except OSError as error: 54 | raise JensMessagingError("Failed to create Queue object (%s)" % error) 55 | 56 | try: 57 | queue.add(item) 58 | except QueueError as error: 59 | raise JensMessagingError("Failed to element (%s)" % error) 60 | 61 | def count_pending_hints(): 62 | settings = Settings() 63 | try: 64 | queue = Queue(settings.MESSAGING_QUEUEDIR, schema=MSG_SCHEMA) 65 | return queue.count() 66 | except OSError as error: 67 | raise JensMessagingError("Failed to create Queue object (%s)" % error) 68 | 69 | def purge_queue(): 70 | settings = Settings() 71 | try: 72 | queue = Queue(settings.MESSAGING_QUEUEDIR, schema=MSG_SCHEMA) 73 | return queue.purge() 74 | except OSError as error: 75 | raise JensMessagingError("Failed to purge Queue object (%s)" % error) 76 | 77 | def _fetch_all_messages(): 78 | settings = Settings() 79 | try: 80 | queue = Queue(settings.MESSAGING_QUEUEDIR, schema=MSG_SCHEMA) 81 | except OSError as error: 82 | raise JensMessagingError("Failed to create Queue object (%s)" % error) 83 | msgs = [] 84 | for _, name in enumerate(queue): 85 | try: 86 | item = queue.dequeue(name) 87 | except QueueLockError as error: 88 | logging.warning("Element %s was locked when dequeuing", name) 89 | continue 90 | except OSError as error: 91 | logging.error("I/O error when getting item %s", name) 92 | continue 93 | try: 94 | item['data'] = pickle.loads(item['data']) 95 | except (pickle.PickleError, EOFError) as error: 96 | logging.debug("Couldn't unpickle item %s. Will be ignored.", name) 97 | continue 98 | logging.debug("Message %s extracted and unpickled", name) 99 | msgs.append(item) 100 | 101 | return msgs 102 | 103 | def _validate_and_merge_messages(messages): 104 | hints = {'modules': set(), 'hostgroups': set(), 'common': set()} 105 | def _merger(acc, element): 106 | if 'time' not in element: 107 | logging.warning("Discarding message: No timestamp") 108 | return acc 109 | time = element['time'] 110 | if 'data' not in element or type(element['data']) != dict: 111 | logging.warning("Discarding message (%s): Bad data section", time) 112 | return acc 113 | for k, v in element['data'].items(): 114 | if k not in hints: 115 | logging.warning("Discarding message (%s): Unknown partition '%s'", time, k) 116 | continue 117 | if type(v) != list: 118 | logging.warning("Discarding message (%s): Value '%s' is not a list", time, v) 119 | continue 120 | for item in v: 121 | if type(item) == str: 122 | logging.debug("Accepted message %s:%s created at %s", 123 | k, v, element['time']) 124 | acc[k].add(item) 125 | else: 126 | logging.warning("Discarding item '%s' in (%s - %s:%s): not a str", 127 | item, time, k, v) 128 | return acc 129 | return reduce(_merger, messages, hints) 130 | -------------------------------------------------------------------------------- /jens/repos.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import yaml 10 | import os 11 | import logging 12 | import shutil 13 | import math 14 | from multiprocessing import Pool, cpu_count, Manager 15 | 16 | import jens.git_wrapper as git 17 | 18 | from jens.settings import Settings 19 | from jens.errors import JensRepositoriesError 20 | from jens.errors import JensGitError 21 | from jens.errors import JensMessagingError 22 | from jens.decorators import timed 23 | from jens.messaging import enqueue_hint 24 | from jens.reposinventory import get_inventory, persist_inventory 25 | from jens.reposinventory import get_desired_inventory 26 | from jens.tools import ref_is_commit 27 | from jens.tools import refname_to_dirname 28 | 29 | @timed 30 | def refresh_repositories(hints=None): 31 | settings = Settings() 32 | try: 33 | logging.debug("Reading metadata from %s", settings.REPO_METADATA) 34 | with open(settings.REPO_METADATA, 'r') as definition_fh: 35 | definition = yaml.safe_load(definition_fh) 36 | except Exception as error: # fixme 37 | raise JensRepositoriesError("Unable to parse %s (%s)", 38 | settings.REPO_METADATA, error) 39 | 40 | inventory = get_inventory() 41 | desired = get_desired_inventory() 42 | deltas = {} 43 | 44 | logging.debug("Initial inventory: %s", inventory) 45 | logging.debug("Needed from overrides: %s", desired) 46 | 47 | for partition in ("modules", "hostgroups", "common"): 48 | logging.info("Refreshing bare repositories (%s)", partition) 49 | logging.debug("Calculating '%s' delta...", partition) 50 | delta = _calculate_delta(definition['repositories'][partition], 51 | inventory[partition]) 52 | logging.info("New repositories: %s", delta['new']) 53 | logging.debug("Existing repositories: %s", delta['existing']) 54 | logging.info("Deleted repositories: %s", delta['deleted']) 55 | 56 | logging.info("Cloning and expanding NEW bare repositories...") 57 | delta['new'] = _create_new_repositories(delta['new'], partition, 58 | definition, inventory[partition], 59 | desired[partition]) 60 | 61 | # If hints are passed but there's nothing explicitly declared 62 | # for a given partition, we make it explicit here. 63 | if hints and partition not in hints: 64 | hints[partition] = set() 65 | 66 | logging.info("Expanding EXISTING bare repositories...") 67 | _refresh_repositories(delta['existing'], partition, inventory[partition], 68 | desired[partition], hints[partition] if hints else None) 69 | 70 | logging.info("Purging REMOVED bare repositories...") 71 | _purge_repositories(delta['deleted'], partition, 72 | inventory[partition]) 73 | 74 | deltas[partition] = delta 75 | 76 | persist_inventory(inventory) 77 | logging.debug("Final inventory: %s", inventory) 78 | 79 | return (deltas, inventory) 80 | 81 | def _create_new_repositories(new_repositories, partition, 82 | definition, inventory, desired): 83 | settings = Settings() 84 | created = [] 85 | for repository in new_repositories: 86 | logging.info("Cloning and expanding %s/%s...", partition, repository) 87 | bare_path = _compose_bare_repository_path(repository, partition) 88 | bare_url = definition['repositories'][partition][repository] 89 | try: 90 | git.clone(bare_path, bare_url, bare=True) 91 | except JensGitError as error: 92 | logging.error("Unable to clone '%s' (%s). Skipping.", 93 | repository, error) 94 | if os.path.exists(bare_path): 95 | shutil.rmtree(bare_path) 96 | continue 97 | try: 98 | refs = list(git.get_refs(bare_path).keys()) 99 | except JensGitError as error: 100 | logging.error("Unable to get refs of '%s' (%s). Skipping.", 101 | repository, error) 102 | shutil.rmtree(bare_path) 103 | logging.debug("Bare repository %s has been removed", bare_path) 104 | continue 105 | # Check if the repository has the mandatory branches 106 | if all([ref in refs for ref in settings.MANDATORY_BRANCHES]): 107 | # Expand only the mandatory and available requested branches 108 | # commits will always be attempted to be expanded 109 | new = set(settings.MANDATORY_BRANCHES) 110 | new = new.union([ref for ref in desired.get(repository, []) 111 | if ref_is_commit(ref) or ref in refs]) 112 | inventory[repository] = [] 113 | _expand_clones(partition, repository, inventory, None, new, [], []) 114 | created.append(repository) 115 | else: 116 | logging.error("Repository '%s' lacks some of the mandatory branches. Skipping.", 117 | repository) 118 | shutil.rmtree(bare_path) 119 | logging.debug("Bare repository %s has been removed", bare_path) 120 | return created 121 | 122 | # This is the most common operation Jens has to do, git-fetch 123 | # over all bare repos and the expansion of clones. 124 | def _refresh_repositories(existing_repositories, partition, 125 | inventory, desired, hints): 126 | settings = Settings() 127 | if not existing_repositories: 128 | return # Seems that passing [] to pool.map makes .join never return 129 | manager = Manager() 130 | # The inventory is the only parameter that has to be r/w 131 | # so we need a common object and a remote controller :) 132 | inventory_proxy = manager.dict(inventory) 133 | inventory_lock = manager.Lock() 134 | data = [{'settings': settings, 'partition': partition, 135 | 'repository': repository, 'inventory': inventory_proxy, 136 | 'inventory_lock': inventory_lock, 'desired': desired, 137 | 'hints': hints} for repository in existing_repositories] 138 | pool = Pool(processes=int(math.ceil(cpu_count()*1.5))) 139 | pool.map(_refresh_repository, data) 140 | pool.close() 141 | pool.join() 142 | inventory.update(inventory_proxy) 143 | 144 | def _refresh_repository(data): 145 | settings = data['settings'] 146 | repository = data['repository'] 147 | partition = data['partition'] 148 | inventory = data['inventory'] 149 | inventory_lock = data['inventory_lock'] 150 | desired = data['desired'] 151 | hints = data['hints'] 152 | logging.debug("Expanding bare and clones of %s/%s...", 153 | partition, repository) 154 | bare_path = _compose_bare_repository_path(repository, partition) 155 | 156 | try: 157 | old_refs = git.get_refs(bare_path) 158 | except JensGitError as error: 159 | logging.error("Unable to get old refs of '%s' (%s)", 160 | repository, error) 161 | return 162 | 163 | # If we know nothing or we know that we have to fetch 164 | if hints is None or repository in hints: 165 | try: 166 | if settings.MODE == "ONDEMAND": 167 | logging.info("Fetching %s/%s upon demand...", 168 | partition, repository) 169 | git.fetch(bare_path, prune=True) 170 | except JensGitError as error: 171 | logging.error("Unable to fetch '%s' from remote (%s)", 172 | repository, error) 173 | if settings.MODE == "ONDEMAND": 174 | try: 175 | enqueue_hint(partition, repository) 176 | except JensMessagingError as error: 177 | logging.error(error) 178 | return 179 | try: 180 | # TODO: Found a corner case where git fetch wiped all 181 | # all the branches in the bare repository. That led 182 | # this get_refs call to fail, and therefore in the next run 183 | # the dual get_refs to obtain old_refs failed as well. 184 | # What to do? No idea. 185 | # Executing git fetch --prune by hand in the bare repo 186 | # brought the branches back. 187 | new_refs = git.get_refs(bare_path) 188 | except JensGitError as error: 189 | logging.error("Unable to get new refs of '%s' (%s)", repository, error) 190 | return 191 | new, moved, deleted = _compare_refs(old_refs, new_refs, inventory[repository], 192 | desired.get(repository, [])) 193 | _expand_clones(partition, repository, inventory, inventory_lock, 194 | new, moved, deleted) 195 | 196 | def _purge_repositories(deleted_repositories, partition, inventory): 197 | for repository in deleted_repositories: 198 | logging.info("Deleting %s/%s...", partition, repository) 199 | bare_path = _compose_bare_repository_path(repository, partition) 200 | # Pass a copy as it will be used as interation set 201 | refs = inventory[repository][:] 202 | _expand_clones(partition, repository, inventory, None, [], [], refs) 203 | clone_path = _compose_clone_repository_path(repository, partition) 204 | shutil.rmtree(clone_path) 205 | logging.debug("Clone repository parent %s has been removed", clone_path) 206 | shutil.rmtree(bare_path) 207 | logging.debug("Bare repository %s has been removed", bare_path) 208 | inventory.pop(repository, None) 209 | 210 | # This function computes the list of refs to be expanded, refreshed or 211 | # removed based on what is available (new_refs), what was available 212 | # (old_refs), what's already present (inventory) and what's necessary 213 | # (desired) 214 | def _compare_refs(old_refs, new_refs, inventory, desired): 215 | settings = Settings() 216 | desired = set(desired).union(settings.MANDATORY_BRANCHES) 217 | # New: What we need minus what we have... 218 | new = list(desired.difference(inventory)) 219 | # ...but only refs that exist or commits 220 | new = [ref for ref in new if ref_is_commit(ref) or ref in new_refs] 221 | 222 | # Deleted: what we have that we don't need anymore 223 | deleted = list(set(inventory).difference(desired)) 224 | 225 | if new: 226 | logging.debug("New refs to be expanded: %s", new) 227 | 228 | if deleted: 229 | logging.debug("Removed refs: %s", deleted) 230 | 231 | # Candidates are those that we already have and we still need 232 | moved = [] 233 | for ref in desired.intersection(inventory): 234 | # No point in checking if a commit has moved 235 | if ref_is_commit(ref): 236 | continue 237 | # If the ref is still being used (in the inventory and desired) 238 | # but has been removed from the repo we mark it as delete. 239 | # Next run will try to get it again and skip the expansion. 240 | if ref not in new_refs: 241 | logging.info("Ref '%s' still needed but removed from repo", ref) 242 | deleted.append(ref) 243 | continue 244 | # The ref is still there and is gonna be kept, check if 245 | # it has moved. 246 | if new_refs[ref] != old_refs[ref]: 247 | logging.debug("Ref '%s' has moved and points to %s", 248 | ref, new_refs[ref]) 249 | moved.append(ref) 250 | else: 251 | logging.debug("Ref '%s' is known but didn't move", ref) 252 | 253 | return new, moved, deleted 254 | 255 | def _expand_clones(partition, name, inventory, inventory_lock, new_refs, 256 | moved_refs, deleted_refs): 257 | settings = Settings() 258 | bare_path = _compose_bare_repository_path(name, partition) 259 | if new_refs: 260 | logging.debug("Processing new refs of %s/%s (%s)...", 261 | partition, name, new_refs) 262 | for refname in new_refs: 263 | clone_path = _compose_clone_repository_path(name, partition, refname) 264 | logging.info("Populating new ref '%s'", clone_path) 265 | try: 266 | if ref_is_commit(refname): 267 | commit_id = refname.replace(settings.HASHPREFIX, '') 268 | logging.debug("Will create a clone pointing to '%s'", commit_id) 269 | git.clone(clone_path, "%s" % bare_path, shared=True) 270 | git.reset(clone_path, commit_id, hard=True) 271 | else: 272 | git.clone(clone_path, "%s" % bare_path, branch=refname) 273 | # Needs reset so the proxy notices about the change on the mutable 274 | # http://docs.python.org/2.7/library/multiprocessing.html#managers 275 | # Locking on the assignment is guarateed by the library, but 276 | # additional locking is needed as A = A + 1 is a critical section. 277 | if inventory_lock: 278 | inventory_lock.acquire() 279 | inventory[name] += [refname] 280 | if inventory_lock: 281 | inventory_lock.release() 282 | except JensGitError as error: 283 | if os.path.isdir(clone_path): 284 | shutil.rmtree(clone_path) 285 | logging.error("Unable to create clone '%s' (%s)", 286 | clone_path, error) 287 | 288 | if moved_refs: 289 | logging.debug("Processing moved refs of %s/%s (%s)...", 290 | partition, name, moved_refs) 291 | for refname in moved_refs: 292 | clone_path = _compose_clone_repository_path(name, partition, refname) 293 | logging.info("Updating ref '%s'", clone_path) 294 | try: 295 | # If this fails, the bare would have the correct HEADs 296 | # but the clone will be out of date and won't ever be 297 | # updated until a new commit arrives to the bare. 298 | # Reason: a lock file left behind because Git was killed 299 | # mid-flight. 300 | git.fetch(clone_path) 301 | git.reset(clone_path, "origin/%s" % refname, hard=True) 302 | logging.info("Updated ref '%s' (%s)", clone_path, 303 | git.get_head(clone_path, short=True)) 304 | except JensGitError as error: 305 | logging.error("Unable to refresh clone '%s' (%s)", 306 | clone_path, error) 307 | 308 | if deleted_refs: 309 | logging.debug("Processing deleted refs of %s/%s (%s)...", 310 | partition, name, deleted_refs) 311 | for refname in deleted_refs: 312 | clone_path = _compose_clone_repository_path(name, partition, refname) 313 | logging.info("Removing %s", clone_path) 314 | try: 315 | if os.path.isdir(clone_path): 316 | shutil.rmtree(clone_path) 317 | if refname in inventory[name]: 318 | if inventory_lock: 319 | inventory_lock.acquire() 320 | element = inventory[name] 321 | element.remove(refname) 322 | inventory[name] = element 323 | if inventory_lock: 324 | inventory_lock.release() 325 | logging.info("%s/%s deleted from inventory", name, refname) 326 | except OSError as error: 327 | logging.error("Couldn't delete %s/%s/%s (%s)", 328 | partition, name, refname, error) 329 | 330 | def _compose_bare_repository_path(name, partition): 331 | settings = Settings() 332 | return settings.BAREDIR + "/%s/%s" % (partition, name) 333 | 334 | def _compose_clone_repository_path(name, partition, refname=None): 335 | settings = Settings() 336 | path = settings.CLONEDIR + "/%s/%s" % (partition, name) 337 | if refname is not None: 338 | dirname = refname_to_dirname(refname) 339 | path = "%s/%s" % (path, dirname) 340 | return path 341 | 342 | def _calculate_delta(definition, current): 343 | definition = set(definition.keys()) 344 | current = set(current.keys()) 345 | 346 | return {'new': definition.difference(current), 347 | 'existing': definition.intersection(current), 348 | 'deleted': current.difference(definition)} 349 | -------------------------------------------------------------------------------- /jens/reposinventory.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import os 10 | import logging 11 | import pickle 12 | 13 | from jens.settings import Settings 14 | from jens.errors import JensRepositoriesError 15 | from jens.errors import JensEnvironmentsError 16 | from jens.environments import read_environment_definition 17 | from jens.environments import get_names_of_declared_environments 18 | from jens.tools import ref_is_commit 19 | from jens.tools import dirname_to_refname 20 | 21 | def get_inventory(): 22 | logging.info("Fetching repositories inventory...") 23 | try: 24 | return _read_inventory_from_disk() 25 | except (IOError, pickle.PickleError): 26 | logging.warning("Inventory on disk not found or corrupt, generating...") 27 | return _generate_inventory() 28 | 29 | def persist_inventory(inventory): 30 | logging.info("Persisting repositories inventory...") 31 | _write_inventory_to_disk(inventory) 32 | 33 | def get_desired_inventory(): 34 | return _read_desired_inventory() 35 | 36 | def _read_inventory_from_disk(): 37 | settings = Settings() 38 | with open(settings.CACHEDIR + "/repositories", "rb") as inventory_file: 39 | return pickle.load(inventory_file) 40 | 41 | def _write_inventory_to_disk(inventory): 42 | settings = Settings() 43 | inventory_file_path = settings.CACHEDIR + "/repositories" 44 | try: 45 | inventory_file = open(inventory_file_path, "wb") 46 | except IOError as error: 47 | raise JensRepositoriesError("Unable to write inventory to disk (%s)" % 48 | error) 49 | logging.debug("Writing inventory to %s", inventory_file_path) 50 | try: 51 | pickle.dump(inventory, inventory_file) 52 | except pickle.PickleError as error: 53 | raise JensRepositoriesError("Unable to write inventory to disk (%s)" % 54 | error) 55 | finally: 56 | inventory_file.close() 57 | 58 | def _generate_inventory(): 59 | settings = Settings() 60 | logging.info("Generating inventory of bares and clones...") 61 | inventory = {} 62 | for partition in ("modules", "hostgroups", "common"): 63 | inventory[partition] = {} 64 | baredir = settings.BAREDIR + "/%s" % partition 65 | try: 66 | names = os.listdir(baredir) 67 | except OSError as error: 68 | raise JensRepositoriesError("Unable to list %s (%s)" % 69 | (baredir, error)) 70 | for name in names: 71 | inventory[partition][name] = \ 72 | _read_list_of_clones(partition, name) 73 | return inventory 74 | 75 | def _read_list_of_clones(partition, name): 76 | settings = Settings() 77 | try: 78 | clones = os.listdir(settings.CLONEDIR + "/%s/%s" % 79 | (partition, name)) 80 | except OSError as error: 81 | raise JensRepositoriesError("Unable to list clones of %s/%s (%s)" % 82 | (partition, name, error)) 83 | return [dirname_to_refname(clone) for clone in clones] 84 | 85 | # This is basically the 'look-ahead' bit 86 | def _read_desired_inventory(): 87 | desired = {'modules': {}, 'hostgroups': {}, 'common': {}} 88 | environments = get_names_of_declared_environments() 89 | for environmentname in environments: 90 | try: 91 | environment = read_environment_definition(environmentname) 92 | if 'overrides' in environment: 93 | for partition in environment['overrides'].keys(): 94 | if partition in ("modules", "hostgroups", "common"): 95 | for name, override in \ 96 | environment['overrides'][partition].items(): 97 | # prefixhash is equivalent to PREFIXhash, contrary to 98 | # refs (branches, sic) which as case-sensitiive 99 | if ref_is_commit(override): 100 | override = override.lower() 101 | if name not in desired[partition]: 102 | desired[partition][name] = [override] 103 | else: 104 | if override not in desired[partition][name]: 105 | desired[partition][name].append(override) 106 | except JensEnvironmentsError as error: 107 | logging.error("Unable to process '%s' definition (%s). Skipping", 108 | environmentname, error) 109 | continue # Just ignore, as won't be generated later on either. 110 | return desired 111 | -------------------------------------------------------------------------------- /jens/settings.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import logging 10 | 11 | from configobj import ConfigObj, flatten_errors 12 | from configobj import ConfigObjError 13 | from validate import Validator 14 | 15 | from jens.errors import JensConfigError 16 | from jens.configfile import CONFIG_GRAMMAR 17 | 18 | class Settings(object): 19 | __shared_state = {} 20 | 21 | def __init__(self, logfile=None): 22 | self.__dict__ = self.__shared_state 23 | if 'logfile' not in self.__dict__: 24 | self.logfile = logfile 25 | 26 | def parse_config(self, config_file_path): 27 | try: 28 | config = ConfigObj(infile=config_file_path, 29 | configspec=CONFIG_GRAMMAR.split("\n")) 30 | except ConfigObjError as error: 31 | raise JensConfigError("Config file parsing failed (%s)" % error) 32 | 33 | validator = Validator() 34 | results = config.validate(validator) 35 | 36 | if results is not True: 37 | for error in flatten_errors(config, results): 38 | section_list, key, _ = error 39 | section_string = '.'.join(section_list) 40 | if key is not None: 41 | raise JensConfigError("Missing/not valid mandatory configuration key %s in section %s" 42 | % (key, section_string)) 43 | else: 44 | raise JensConfigError("Section '%s' is missing" % section_string) 45 | 46 | # Save a reference in case we need the raw values later 47 | self.config = config 48 | 49 | # [main] 50 | self.DEBUG_LEVEL = config["main"]["debuglevel"] 51 | self.LOGDIR = config["main"]["logdir"] 52 | self.BAREDIR = config["main"]["baredir"] 53 | self.CACHEDIR = config["main"]["cachedir"] 54 | self.CLONEDIR = config["main"]["clonedir"] 55 | self.ENVIRONMENTSDIR = config["main"]["environmentsdir"] 56 | self.MANDATORY_BRANCHES = config["main"]["mandatorybranches"] 57 | self.REPO_METADATA = config["main"]["repositorymetadata"] 58 | self.REPO_METADATADIR = config["main"]["repositorymetadatadir"] 59 | self.ENV_METADATADIR = config["main"]["environmentsmetadatadir"] 60 | self.HASHPREFIX = config["main"]["hashprefix"] 61 | self.DIRECTORY_ENVIRONMENTS = config["main"]["directory_environments"] 62 | self.COMMON_HIERADATA_ITEMS = config["main"]["common_hieradata_items"] 63 | self.MODE = config["main"]["mode"] 64 | self.PROTECTED_ENVIRONMENTS = config["main"]["protectedenvironments"] 65 | 66 | # [lock] 67 | self.LOCK_TYPE = config["lock"]["type"] 68 | self.LOCK_NAME = config["lock"]["name"] 69 | 70 | # [filelock] 71 | self.FILELOCK_LOCKDIR = config["filelock"]["lockdir"] 72 | 73 | # [messaging] 74 | self.MESSAGING_QUEUEDIR = config["messaging"]["queuedir"] 75 | 76 | # [git] 77 | self.SSH_CMD_PATH = config["git"]["ssh_cmd_path"] 78 | 79 | # [gitlabproducer] 80 | self.GITLAB_PRODUCER_SECRET_TOKEN = config["gitlabproducer"]["secret_token"] 81 | 82 | if self.logfile: 83 | logging.basicConfig( 84 | level=getattr(logging, self.DEBUG_LEVEL), 85 | format='%(asctime)s [%(process)d] %(levelname)s %(message)s', 86 | filename="%s/%s.log" % (self.LOGDIR, self.logfile)) 87 | else: 88 | logging.basicConfig( 89 | level=getattr(logging, self.DEBUG_LEVEL), 90 | format='%(message)s') 91 | -------------------------------------------------------------------------------- /jens/test/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | -------------------------------------------------------------------------------- /jens/test/test_git_wrapper.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014-2016, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import os 10 | import jens.git_wrapper as git_wrapper 11 | from unittest.mock import patch 12 | from jens.test.testcases import JensTestCase 13 | from jens.errors import JensGitError 14 | from jens.test.tools import * 15 | 16 | 17 | class GitWrapperTest(JensTestCase): 18 | def setUp(self): 19 | super(GitWrapperTest, self).setUp() 20 | 21 | def test_hash_object_raises_jens_git_error(self): 22 | self.assertRaises(JensGitError, git_wrapper.hash_object, 23 | 'platform-9-and-0.75') 24 | 25 | def test_gc_existing_repository(self): 26 | (bare, user) = create_fake_repository(self.sandbox_path, 27 | ['qa']) 28 | git_wrapper.gc(bare) 29 | 30 | def test_gc_non_existing_repository(self): 31 | self.assertRaises(JensGitError, git_wrapper.gc, '/tmp/37d8s8dd') 32 | 33 | def test_gc_not_repository(self): 34 | not_repo_path = create_folder_not_repository(self.sandbox_path) 35 | self.assertRaises(JensGitError, git_wrapper.gc, not_repo_path) 36 | 37 | def test_bare_clone_existing_bare_repository(self): 38 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 39 | git_wrapper.clone("%s/repo" % self.settings.CLONEDIR, bare, bare=True) 40 | 41 | def test_clone_existing_bare_repository_specific_branch(self): 42 | (bare, user) = create_fake_repository(self.sandbox_path, 43 | ['qa', 'foo']) 44 | git_wrapper.clone("%s/repo" % self.settings.CLONEDIR, bare, 45 | bare=False, branch='foo') 46 | 47 | def test_clone_existing_repository(self): 48 | (bare, user) = create_fake_repository(self.sandbox_path, 49 | ['qa']) 50 | git_wrapper.clone("%s/repo" % self.settings.CLONEDIR, user) 51 | 52 | def test_clone_non_existing_repository(self): 53 | self.assertRaises(JensGitError, git_wrapper.clone, "%s/repo" 54 | % self.settings.CLONEDIR, '/tmp/37d8s8de') 55 | 56 | def test_clone_mirrored_repository(self): 57 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 58 | clone_path = "%s/repo" % self.settings.CLONEDIR 59 | git_wrapper.clone(clone_path, bare, shared=True) 60 | self.assertTrue(os.path.isfile("%s/.git/objects/info/alternates" % 61 | clone_path)) 62 | 63 | def test_fetch_existing_repository(self): 64 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 65 | git_wrapper.fetch(user) 66 | 67 | @patch('git.remote.Remote.fetch') 68 | def test_fetch_fails_when_assertion_error_is_raised(self, stub): 69 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 70 | stub.side_effect = AssertionError() 71 | self.assertRaises(JensGitError, git_wrapper.fetch, user, prune=True) 72 | 73 | def test_fetch_existing_bare_repository_and_prune(self): 74 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa', 'f']) 75 | jens_bare = "%s/_bare" % self.settings.BAREDIR 76 | git_wrapper.clone(jens_bare, bare, bare=True) 77 | git_wrapper.fetch(jens_bare, prune=True) 78 | self.assertTrue('f' in git_wrapper.get_refs(jens_bare)) 79 | remove_branch_from_repo(user, 'f') 80 | git_wrapper.fetch(jens_bare, prune=False) 81 | self.assertTrue('f' in git_wrapper.get_refs(jens_bare)) 82 | git_wrapper.fetch(jens_bare, prune=True) 83 | self.assertFalse('f' in git_wrapper.get_refs(jens_bare)) 84 | 85 | def test_fetch_existing_bare_repository(self): 86 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 87 | new_bare_path = "%s/cloned" % self.settings.CLONEDIR 88 | git_wrapper.clone(new_bare_path, bare, bare=True) 89 | git_wrapper.fetch(new_bare_path) 90 | 91 | def test_fetch_non_existing_repository(self): 92 | self.assertRaises(JensGitError, git_wrapper.fetch, '/tmp/37d8s8df') 93 | 94 | def test_fetch_not_repository(self): 95 | not_repo_path = create_folder_not_repository(self.sandbox_path) 96 | self.assertRaises(JensGitError, git_wrapper.fetch, not_repo_path) 97 | 98 | def test_reset_to_head(self): 99 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 100 | head = get_repository_head(user) 101 | git_wrapper.reset(user, head) 102 | 103 | def test_reset_to_commit(self): 104 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 105 | head = get_repository_head(user) 106 | commit_id = add_commit_to_branch(user, "master") 107 | git_wrapper.reset(user, head) 108 | 109 | def test_reset_non_existing_repository(self): 110 | self.assertRaises(JensGitError, git_wrapper.reset, '/tmp/37d8s8e0', 111 | "37d8s8e0") 112 | 113 | def test_reset_non_existing_commit(self): 114 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 115 | self.assertRaises(JensGitError, git_wrapper.reset, user, "37d8s8e1") 116 | 117 | def test_reset_and_fetch_refs_match_after_remote_commit(self): 118 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 119 | jens_bare = "%s/_bare" % self.settings.BAREDIR 120 | git_wrapper.clone(jens_bare, bare, bare=True) 121 | jens_clone = "%s/_clone" % self.settings.CLONEDIR 122 | git_wrapper.clone(jens_clone, jens_bare, bare=False, branch='qa') 123 | fname = 'should_be_checkedout' 124 | commit_id = add_commit_to_branch(user, 'qa', fname=fname) 125 | git_wrapper.fetch(jens_bare) 126 | git_wrapper.fetch(jens_clone) 127 | git_wrapper.reset(jens_clone, 'origin/qa', hard=True) 128 | self.assertEqual(get_repository_head(jens_clone), 129 | commit_id) 130 | self.assertTrue(os.path.isfile("%s/%s" % (jens_clone, fname))) 131 | 132 | 133 | new_commit = add_commit_to_branch(user, 'qa', fname=fname, remove=True) 134 | git_wrapper.fetch(jens_bare) 135 | git_wrapper.fetch(jens_clone) 136 | git_wrapper.reset(jens_clone, 'origin/qa', hard=True) 137 | self.assertFalse(os.path.isfile("%s/%s" % 138 | (jens_clone, fname))) 139 | 140 | def test_reset_not_repository(self): 141 | not_repo_path = create_folder_not_repository(self.sandbox_path) 142 | self.assertRaises(JensGitError, git_wrapper.reset, not_repo_path, 143 | "37d8s8e2") 144 | 145 | def test_get_refs_existing_repository(self): 146 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 147 | r = git_wrapper.get_refs(user) 148 | self.assertEqual(type(r), dict) 149 | self.assertTrue('qa' in r) 150 | self.assertTrue('master' in r) 151 | 152 | def test_get_refs_non_existing_repository(self): 153 | self.assertRaises(JensGitError, git_wrapper.get_refs, '/tmp/37d8s8e3') 154 | 155 | def test_get_refs_not_repository(self): 156 | not_repo_path = create_folder_not_repository(self.sandbox_path) 157 | self.assertRaises(JensGitError, git_wrapper.get_refs, not_repo_path) 158 | 159 | def test_get_head_existing_repository(self): 160 | (bare, user) = create_fake_repository(self.sandbox_path, ['qa']) 161 | jens_clone = "%s/_clone" % self.settings.CLONEDIR 162 | git_wrapper.clone(jens_clone, bare, bare=False, branch='qa') 163 | commit_id = add_commit_to_branch(user, 'qa') 164 | git_wrapper.fetch(jens_clone) 165 | git_wrapper.reset(jens_clone, 'origin/qa', hard=True) 166 | self.assertEqual(git_wrapper.get_head(jens_clone), 167 | commit_id) 168 | self.assertEqual(git_wrapper.get_head(jens_clone, short=False), 169 | commit_id) 170 | self.assertEqual(git_wrapper.get_head(jens_clone, short=True), 171 | commit_id[0:7]) 172 | 173 | def test_get_head_non_existing_repository(self): 174 | self.assertRaises(JensGitError, git_wrapper.get_head, '/tmp/37d8s8e3') 175 | 176 | def test_get_head_not_repository(self): 177 | not_repo_path = create_folder_not_repository(self.sandbox_path) 178 | self.assertRaises(JensGitError, git_wrapper.get_head, not_repo_path) 179 | -------------------------------------------------------------------------------- /jens/test/test_maintenance.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2016, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import shutil 10 | 11 | from jens.maintenance import validate_directories 12 | from jens.git_wrapper import clone 13 | from jens.settings import Settings 14 | from jens.errors import JensError 15 | 16 | from jens.test.tools import create_fake_repository 17 | 18 | from jens.test.testcases import JensTestCase 19 | 20 | class MaintenanceTest(JensTestCase): 21 | def setUp(self): 22 | super(MaintenanceTest, self).setUp() 23 | 24 | self.settings = Settings() 25 | 26 | # validate_directories() expects both below to look 27 | # like a Git repository. 28 | 29 | (self.environments_bare, self.environments) = \ 30 | create_fake_repository(self.sandbox_path) 31 | shutil.rmtree(self.settings.ENV_METADATADIR) 32 | clone(self.settings.ENV_METADATADIR, self.environments_bare, \ 33 | branch='master') 34 | 35 | (self.repositories_bare, self.repositories) = \ 36 | create_fake_repository(self.sandbox_path) 37 | shutil.rmtree(self.settings.REPO_METADATADIR) 38 | clone(self.settings.REPO_METADATADIR, self.repositories_bare, \ 39 | branch='master') 40 | 41 | 42 | #### TESTS #### 43 | 44 | def test_all_expected_directories_are_present_and_inited(self): 45 | validate_directories() 46 | 47 | def test_no_bares_dir(self): 48 | shutil.rmtree(self.settings.BAREDIR) 49 | self.assertRaisesRegex(JensError, 50 | self.settings.BAREDIR, validate_directories) 51 | 52 | def test_no_cache_dir(self): 53 | shutil.rmtree(self.settings.CACHEDIR) 54 | self.assertRaisesRegex(JensError, 55 | self.settings.CACHEDIR, validate_directories) 56 | 57 | def test_no_clones_dir(self): 58 | shutil.rmtree(self.settings.CLONEDIR) 59 | self.assertRaisesRegex(JensError, 60 | self.settings.CLONEDIR, validate_directories) 61 | 62 | def test_no_environments_dir(self): 63 | shutil.rmtree(self.settings.ENVIRONMENTSDIR) 64 | self.assertRaisesRegex(JensError, 65 | self.settings.ENVIRONMENTSDIR, validate_directories) 66 | 67 | def test_no_repometadata_dir(self): 68 | shutil.rmtree(self.settings.REPO_METADATADIR) 69 | self.assertRaisesRegex(JensError, 70 | self.settings.REPO_METADATADIR, validate_directories) 71 | 72 | def test_no_envmetadata_dir(self): 73 | shutil.rmtree(self.settings.ENV_METADATADIR) 74 | self.assertRaisesRegex(JensError, 75 | self.settings.ENV_METADATADIR, validate_directories) 76 | -------------------------------------------------------------------------------- /jens/test/test_messaging.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2015, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import os 10 | 11 | from datetime import datetime 12 | 13 | from dirq.queue import Queue, QueueLockError 14 | from dirq.queue import QueueError 15 | 16 | from jens.messaging import _validate_and_merge_messages, _fetch_all_messages 17 | from jens.messaging import fetch_update_hints, count_pending_hints 18 | from jens.messaging import enqueue_hint, purge_queue 19 | from jens.errors import JensMessagingError 20 | from jens.test.tools import init_repositories 21 | from jens.test.tools import add_repository, del_repository 22 | from jens.test.tools import add_msg_to_queue 23 | from jens.test.tools import create_hostgroup_event, create_module_event 24 | from jens.test.tools import create_common_event 25 | 26 | from jens.test.testcases import JensTestCase 27 | 28 | from unittest.mock import Mock, patch 29 | 30 | class MessagingTest(JensTestCase): 31 | def setUp(self): 32 | super(MessagingTest, self).setUp() 33 | 34 | #### TESTS #### 35 | 36 | def test_update_hints(self): 37 | modules = ['foo', 'bar', 'baz1', 'baz2', 'm1', 'm2'] 38 | hgs = ['hg0', 'hg1', 'hg2', 'hg3', 'hg4'] 39 | for module in modules: 40 | create_module_event(module) 41 | for hg in hgs: 42 | create_hostgroup_event(hg) 43 | hints = fetch_update_hints() 44 | for m in modules: 45 | self.assertTrue(m in hints['modules']) 46 | for h in hgs: 47 | self.assertTrue(h in hints['hostgroups']) 48 | self.assertTrue('common' in hints) 49 | self.assertEqual(0, len(hints['common'])) 50 | create_module_event('m1') 51 | create_common_event('baz') 52 | hints = fetch_update_hints() 53 | self.assertTrue('hostgroups' in hints) 54 | self.assertEqual(0, len(hints['hostgroups'])) 55 | self.assertTrue('baz' in hints['common']) 56 | self.assertEqual(1, len(hints['modules'])) 57 | self.assertTrue('m1' in hints['modules']) 58 | 59 | def test_update_hints_no_dups(self): 60 | create_module_event('foo') 61 | create_module_event('foo') 62 | hints = fetch_update_hints() 63 | self.assertEqual(1, len(hints['modules'])) 64 | 65 | def test_update_hints_no_messages(self): 66 | hints = fetch_update_hints() 67 | self.assertTrue('modules' in hints) 68 | self.assertEqual(0, len(hints['hostgroups'])) 69 | self.assertTrue('hostgroups' in hints) 70 | self.assertEqual(0, len(hints['hostgroups'])) 71 | self.assertTrue('common' in hints) 72 | self.assertEqual(0, len(hints['common'])) 73 | 74 | def test_fetch_all_messages_noerrors(self): 75 | create_module_event('foo') 76 | create_hostgroup_event('bar') 77 | msgs = _fetch_all_messages() 78 | self.assertEqual(2, len(msgs)) 79 | 80 | def test_fetch_all_messages_no_queuedir_is_created(self): 81 | self.settings.MESSAGING_QUEUEDIR = "%s/notthere" % \ 82 | self.settings.MESSAGING_QUEUEDIR 83 | self.assertFalse(os.path.isdir(self.settings.MESSAGING_QUEUEDIR)) 84 | msgs = _fetch_all_messages() 85 | self.assertTrue(os.path.isdir(self.settings.MESSAGING_QUEUEDIR)) 86 | self.assertEqual(0, len(msgs)) 87 | 88 | def test_fetch_all_messages_queuedir_cannot_be_created(self): 89 | if os.getuid() == 0: 90 | return 91 | self.settings.MESSAGING_QUEUEDIR = "/oops" 92 | self.assertRaises(JensMessagingError, _fetch_all_messages) 93 | 94 | def test_purge_queue_queuedir_does_not_exist(self): 95 | if os.getuid() == 0: 96 | return 97 | self.settings.MESSAGING_QUEUEDIR = "/oops" 98 | self.assertRaises(JensMessagingError, purge_queue) 99 | 100 | def test_fetch_all_messages_ununpickable(self): 101 | create_hostgroup_event('bar') 102 | broken = {'time': datetime.now().isoformat(), 103 | 'data': '))'.encode()} 104 | add_msg_to_queue(broken) 105 | msgs = _fetch_all_messages() 106 | self.assertEqual(1, len(msgs)) 107 | 108 | @patch.object(Queue, 'dequeue', side_effect=QueueLockError) 109 | def test_fetch_all_messages_locked_item(self, mock_queue): 110 | create_module_event('foo') 111 | msgs = _fetch_all_messages() 112 | self.assertEqual(0, len(msgs)) 113 | mock_queue.assert_called_once() 114 | 115 | @patch.object(Queue, 'dequeue', side_effect=OSError) 116 | def test_fetch_all_messages_ioerror_when_dequeuing(self, mock_queue): 117 | create_module_event('foo') 118 | msgs = _fetch_all_messages() 119 | self.assertLogErrors() 120 | mock_queue.assert_called_once() 121 | self.assertEqual(0, len(msgs)) 122 | 123 | def test_count_no_messages(self): 124 | count = count_pending_hints() 125 | self.assertEqual(0, count) 126 | 127 | def test_count_some_messages(self): 128 | create_hostgroup_event('bar') 129 | create_module_event('foo') 130 | count = count_pending_hints() 131 | self.assertEqual(2, count) 132 | 133 | @patch.object(Queue, 'count', side_effect=OSError) 134 | def test_count_some_messages_queue_error(self, mock_queue): 135 | create_hostgroup_event('bar') 136 | create_module_event('foo') 137 | self.assertRaises(JensMessagingError, count_pending_hints) 138 | 139 | # TODO: Test that other messages are fetched if one is locked/broken 140 | 141 | def test_validate_and_merge_messages(self): 142 | messages = [ 143 | {}, # Bad 144 | {'data': {'modules': ['foo']}}, # Bad 145 | {'time': datetime.now().isoformat()}, # Bad 146 | {'time': datetime.now().isoformat(), 147 | 'data': ''}, # Bad 148 | {'time': datetime.now().isoformat(), 149 | 'data': {}}, # Bad 150 | {'time': datetime.now().isoformat(), 151 | 'data': {'modules': 'foo'}}, # Bad 152 | {'time': datetime.now().isoformat(), 153 | 'data': {'modules': ['fizz', []]}}, # Bad 154 | {'time': datetime.now().isoformat(), 155 | 'data': {'modules': ['foo']}}, 156 | {'time': datetime.now().isoformat(), 157 | 'data': {'modules': ['bar']}}, 158 | {'time': datetime.now().isoformat(), 159 | 'data': {'modules': ['baz1', 'baz2']}}, 160 | {'time': datetime.now().isoformat(), 161 | 'data': {'hostgroups': ['hg0']}}, 162 | {'time': datetime.now().isoformat(), 163 | 'data': {'hostgroups': ['hg1', 'hg2']}}, 164 | {'time': datetime.now().isoformat(), 165 | 'data': {'hostgroups': ['hg3', 'hg4'], 166 | 'modules': ['m1']}}, 167 | {'time': datetime.now().isoformat(), 168 | 'data': {'crap': ['hg3', 'hg4'], 169 | 'modules': ['m2']}}, 170 | {'time': datetime.now().isoformat(), 171 | 'data': {'common': ['site']}}, 172 | ] 173 | 174 | modules = ['foo', 'bar', 'baz1', 'baz2', 'm1', 'm2', 'fizz'] 175 | hgs = ['hg0', 'hg1', 'hg2', 'hg3', 'hg4'] 176 | 177 | result = _validate_and_merge_messages(messages) 178 | 179 | self.assertTrue('modules' in result) 180 | self.assertTrue('hostgroups' in result) 181 | self.assertTrue('common' in result) 182 | self.assertEqual(len(modules), len(result['modules'])) 183 | self.assertEqual(len(hgs), len(result['hostgroups'])) 184 | for m in modules: 185 | self.assertTrue(m in result['modules']) 186 | for h in hgs: 187 | self.assertTrue(h in result['hostgroups']) 188 | self.assertTrue('site' in result['common']) 189 | 190 | def test_enqueue_hint_okay(self): 191 | enqueue_hint('modules', 'foo1') 192 | enqueue_hint('modules', 'foo2') 193 | enqueue_hint('modules', 'foo3') 194 | enqueue_hint('hostgroups', 'hg1') 195 | enqueue_hint('hostgroups', 'hg2') 196 | enqueue_hint('common', 'site') 197 | # If they are malformed fetch_update_hints 198 | # will ignore them 199 | hints = fetch_update_hints() 200 | self.assertEqual(1, len(hints['common'])) 201 | self.assertEqual(2, len(hints['hostgroups'])) 202 | self.assertEqual(3, len(hints['modules'])) 203 | 204 | def test_enqueue_hint_okay(self): 205 | enqueue_hint('modules', 'foo1') 206 | enqueue_hint('modules', 'foo2') 207 | enqueue_hint('modules', 'foo3') 208 | enqueue_hint('hostgroups', 'hg1') 209 | enqueue_hint('hostgroups', 'hg2') 210 | enqueue_hint('common', 'site') 211 | hints = fetch_update_hints() 212 | self.assertEqual(1, len(hints['common'])) 213 | self.assertEqual(2, len(hints['hostgroups'])) 214 | self.assertEqual(3, len(hints['modules'])) 215 | 216 | 217 | @patch.object(Queue, 'add', side_effect=QueueError) 218 | def test_enqueue_hint_queue_error(self, mock): 219 | self.assertRaises(JensMessagingError, enqueue_hint, 'modules', 'foo') 220 | 221 | def test_enqueue_hint_bad_partition(self): 222 | self.assertRaises(JensMessagingError, enqueue_hint, 'booboo', 'foo') 223 | -------------------------------------------------------------------------------- /jens/test/test_metadata.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import shutil 10 | import fcntl 11 | import os 12 | 13 | from unittest.mock import Mock, patch 14 | 15 | from jens.maintenance import refresh_metadata 16 | from jens.git_wrapper import clone 17 | from jens.errors import JensError 18 | 19 | from jens.test.tools import create_fake_repository 20 | from jens.test.tools import add_commit_to_branch, reset_branch_to 21 | from jens.test.tools import get_repository_head 22 | from jens.settings import Settings 23 | 24 | from jens.test.testcases import JensTestCase 25 | 26 | class MetadataTest(JensTestCase): 27 | def setUp(self): 28 | super(MetadataTest, self).setUp() 29 | 30 | self.settings = Settings() 31 | 32 | (self.environments_bare, self.environments) = \ 33 | create_fake_repository(self.sandbox_path) 34 | shutil.rmtree(self.settings.ENV_METADATADIR) 35 | clone(self.settings.ENV_METADATADIR, self.environments_bare, \ 36 | branch='master') 37 | 38 | (self.repositories_bare, self.repositories) = \ 39 | create_fake_repository(self.sandbox_path) 40 | shutil.rmtree(self.settings.REPO_METADATADIR) 41 | clone(self.settings.REPO_METADATADIR, self.repositories_bare, \ 42 | branch='master') 43 | 44 | def _jens_refresh_metadata(self, errorsExpected=False, errorRegexp=None): 45 | refresh_metadata() 46 | if errorsExpected: 47 | self.assertLogErrors(errorRegexp) 48 | else: 49 | self.assertLogNoErrors() 50 | 51 | #### TESTS #### 52 | 53 | def test_basic_updates(self): 54 | self._jens_refresh_metadata() 55 | 56 | fname = 'basic_updates_1' 57 | new_commit = add_commit_to_branch(self.environments, 'master', fname=fname) 58 | self._jens_refresh_metadata() 59 | self.assertEqual(get_repository_head(self.settings.ENV_METADATADIR), new_commit) 60 | # The reset is --hard 61 | self.assertTrue(os.path.isfile("%s/%s" % 62 | (self.settings.ENV_METADATADIR, fname))) 63 | new_commit = add_commit_to_branch(self.environments, 'master', fname=fname, 64 | remove=True) 65 | self._jens_refresh_metadata() 66 | self.assertFalse(os.path.isfile("%s/%s" % 67 | (self.settings.ENV_METADATADIR, fname))) 68 | 69 | fname = 'basic_updates_2' 70 | new_commit = add_commit_to_branch(self.repositories, 'master', fname=fname) 71 | self._jens_refresh_metadata() 72 | self.assertEqual(get_repository_head(self.settings.REPO_METADATADIR), 73 | new_commit) 74 | self.assertTrue(os.path.isfile("%s/%s" % 75 | (self.settings.REPO_METADATADIR, fname))) 76 | 77 | new_commit = add_commit_to_branch(self.repositories, 'master', fname=fname, 78 | remove=True) 79 | self._jens_refresh_metadata() 80 | self.assertEqual(get_repository_head(self.settings.REPO_METADATADIR), new_commit) 81 | self.assertFalse(os.path.isfile("%s/%s" % 82 | (self.settings.REPO_METADATADIR, fname))) 83 | 84 | def test_metadata_updates_if_ondemand_mode_is_enabled(self): 85 | self.settings.MODE = "ONDEMAND" 86 | self._jens_refresh_metadata() 87 | 88 | new_commit = add_commit_to_branch(self.environments, 'master') 89 | self._jens_refresh_metadata() 90 | self.assertEqual(get_repository_head(self.settings.ENV_METADATADIR), new_commit) 91 | 92 | new_commit = add_commit_to_branch(self.repositories, 'master') 93 | self._jens_refresh_metadata() 94 | self.assertEqual(get_repository_head(self.settings.REPO_METADATADIR), new_commit) 95 | 96 | def test_fails_if_remote_repositories_unavailable(self): 97 | initial = get_repository_head(self.repositories) 98 | self.assertEqual(get_repository_head(self.settings.REPO_METADATADIR), initial) 99 | self._jens_refresh_metadata() 100 | self.assertEqual(get_repository_head(self.settings.REPO_METADATADIR), initial) 101 | 102 | # -- "not available" -- 103 | 104 | temporary_path = "%s-temp" % self.repositories_bare 105 | shutil.move(self.repositories_bare, temporary_path) 106 | self.assertRaises(JensError, self._jens_refresh_metadata) 107 | 108 | # -- "available again" -- 109 | 110 | shutil.move(temporary_path, self.repositories_bare) 111 | self._jens_refresh_metadata() 112 | 113 | def test_fails_if_remote_environments_unavailable(self): 114 | initial = get_repository_head(self.environments) 115 | self.assertEqual(get_repository_head(self.settings.ENV_METADATADIR), initial) 116 | self._jens_refresh_metadata() 117 | self.assertEqual(get_repository_head(self.settings.ENV_METADATADIR), initial) 118 | 119 | # -- "not available" -- 120 | 121 | temporary_path = "%s-temp" % self.environments_bare 122 | shutil.move(self.environments_bare, temporary_path) 123 | self.assertRaises(JensError, self._jens_refresh_metadata) 124 | 125 | # -- "available again" -- 126 | 127 | shutil.move(temporary_path, self.environments_bare) 128 | self._jens_refresh_metadata() 129 | 130 | def test_repositories_is_history_is_mangled(self): 131 | self._jens_refresh_metadata() 132 | 133 | bombs = [] 134 | for x in range(0,4): 135 | bombs.append(add_commit_to_branch(self.repositories, 'master')) 136 | 137 | self._jens_refresh_metadata() 138 | 139 | self.assertEqual(get_repository_head(self.repositories), bombs[-1]) 140 | 141 | reset_branch_to(self.repositories, "master", bombs[0]) 142 | new_commit = add_commit_to_branch(self.repositories, \ 143 | 'master', force=True) 144 | 145 | self._jens_refresh_metadata() 146 | 147 | # Should be the same if it did a reset 148 | self.assertEqual(get_repository_head(self.settings.REPO_METADATADIR), new_commit) 149 | 150 | def test_environments_is_history_is_mangled(self): 151 | self._jens_refresh_metadata() 152 | 153 | bombs = [] 154 | for x in range(0,4): 155 | bombs.append(add_commit_to_branch(self.environments, 'master')) 156 | 157 | self._jens_refresh_metadata() 158 | 159 | self.assertEqual(get_repository_head(self.environments), bombs[-1]) 160 | 161 | reset_branch_to(self.environments, "master", bombs[0]) 162 | new_commit = add_commit_to_branch(self.environments, \ 163 | 'master', force=True) 164 | 165 | self._jens_refresh_metadata() 166 | 167 | # Should be the same if it did a reset 168 | self.assertEqual(get_repository_head(self.settings.ENV_METADATADIR), new_commit) 169 | 170 | @patch.object(fcntl, 'flock', side_effect=IOError) 171 | def test_fails_if_lock_cannot_be_acquired(self, mock): 172 | self.assertRaises(JensError, self._jens_refresh_metadata) 173 | -------------------------------------------------------------------------------- /jens/test/testcases.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | from __future__ import print_function 10 | import os 11 | import unittest 12 | import logging 13 | import re 14 | import tempfile 15 | import time 16 | 17 | from string import Template 18 | from configobj import ConfigObj 19 | 20 | from jens.locks import JensLockFactory 21 | from jens.settings import Settings 22 | from jens.reposinventory import get_inventory 23 | from jens.test.tools import init_sandbox, destroy_sandbox 24 | from jens.tools import refname_to_dirname 25 | from jens.tools import dirname_to_refname 26 | from jens.test.tools import get_repository_head 27 | 28 | BASE_CONFIG = Template(""" 29 | [main] 30 | baredir = $sandbox/lib/bare 31 | clonedir = $sandbox/lib/clone 32 | cachedir = $sandbox/lib/cache 33 | environmentsdir = $sandbox/lib/environments 34 | debuglevel = $debuglevel 35 | logdir = $sandbox/log 36 | mandatorybranches = $mandatory_branches 37 | environmentsmetadatadir = $sandbox/lib/metadata/environments 38 | repositorymetadatadir = $sandbox/lib/metadata/repositories 39 | repositorymetadata = $sandbox/lib/metadata/repositories/repositories.yaml 40 | hashprefix = $hashprefix 41 | common_hieradata_items = environments, hardware, operatingsystems, datacentres, common.yaml 42 | 43 | [lock] 44 | type = DISABLED 45 | 46 | [messaging] 47 | queuedir = $sandbox/spool 48 | """) 49 | 50 | class JensTestCase(unittest.TestCase): 51 | COMMIT_PREFIX = 'commit/' 52 | DEFAULT_DEBUG_LEVEL = 'DEBUG' 53 | MANDATORY_BRANCHES = ['master', 'qa'] 54 | 55 | def setUp(self): 56 | self.sandbox_path = tempfile.mkdtemp( 57 | prefix="jens_sandbox_%s-" % self._testMethodName, 58 | suffix='-' + str(time.time())) 59 | init_sandbox(self.sandbox_path) 60 | self.keep_sandbox = bool(os.getenv('JENS_TEST_KEEP_SANDBOX', False)) 61 | self.debug_level = JensTestCase.DEFAULT_DEBUG_LEVEL 62 | self.config_file_path = "%s/etc/main.conf" % self.sandbox_path 63 | config_file = open(self.config_file_path, 'w+') 64 | config_file.write(BASE_CONFIG.substitute( 65 | sandbox=self.sandbox_path, 66 | hashprefix=JensTestCase.COMMIT_PREFIX, 67 | debuglevel=self.debug_level, 68 | mandatory_branches=','.join(JensTestCase.MANDATORY_BRANCHES))) 69 | config_file.close() 70 | 71 | self.settings = Settings("jens-test") 72 | self.settings.parse_config(self.config_file_path) 73 | 74 | self.lock = JensLockFactory.make_lock(self.settings) 75 | 76 | def tearDown(self): 77 | logging.shutdown() 78 | 79 | # Remove the logger that writes to the log file of this test 80 | # so when the next test setUp()s a new handler with a new log 81 | # file is configured. 82 | root = logging.getLogger() 83 | for handler in root.handlers[:]: 84 | root.removeHandler(handler) 85 | for _filter in root.filters[:]: 86 | root.removeFilter(_filter) 87 | 88 | if hasattr(self, 'log'): 89 | self.log.close() 90 | 91 | if self.keep_sandbox: 92 | print("Sandbox kept in", self.sandbox_path) 93 | else: 94 | destroy_sandbox(self.sandbox_path) 95 | 96 | def assertLogNoErrors(self): 97 | if not hasattr(self, 'log'): 98 | self.log = open("%s/jens-test.log" % self.settings.LOGDIR) 99 | for line in self.log.readlines(): 100 | if re.match(r'.+ERROR.+', line): 101 | raise AssertionError(line) 102 | 103 | def assertLogErrors(self, errorRegexp=None): 104 | if not hasattr(self, 'log'): 105 | self.log = open("%s/jens-test.log" % self.settings.LOGDIR) 106 | found = False 107 | regexp = r'.+ERROR.+' 108 | if errorRegexp is not None: 109 | regexp = r'.+ERROR.+%s.+' % errorRegexp 110 | for line in self.log.readlines(): 111 | if re.match(regexp, line): 112 | found = True 113 | if not found: 114 | raise AssertionError("There should be errors") 115 | 116 | def assertClone(self, identifier, pointsto=None): 117 | partition, element, dirname = identifier.split('/') 118 | path = "%s/%s/%s/%s" % (self.settings.CLONEDIR, \ 119 | partition, element, dirname) 120 | if not os.path.isdir(path): 121 | raise AssertionError("Clone '%s' not found" % path) 122 | if not os.path.isdir("%s/code" % path): 123 | raise AssertionError("Clone '%s' does not have code dir" % path) 124 | if not os.path.isdir("%s/data" % path): 125 | raise AssertionError("Clone '%s' does not have data dir" % path) 126 | inventory = get_inventory() 127 | self.assertTrue(partition in inventory) 128 | self.assertTrue(element in inventory[partition]) 129 | refname = dirname_to_refname(dirname) 130 | self.assertTrue(refname in inventory[partition][element]) 131 | if pointsto is not None: 132 | self.assertEqual(get_repository_head(path), 133 | pointsto) 134 | 135 | def assertCloneFileExists(self, identifier, fname): 136 | partition, element, dirname = identifier.split('/') 137 | path = "%s/%s/%s/%s/%s" % (self.settings.CLONEDIR, \ 138 | partition, element, dirname, fname) 139 | if not os.path.isfile(path): 140 | raise AssertionError("File '%s' not found" % path) 141 | 142 | def assertNotClone(self, identifier): 143 | try: 144 | self.assertClone(identifier) 145 | except AssertionError: 146 | return 147 | raise AssertionError("Clone '%s' seems present" % identifier) 148 | 149 | def assertBare(self, identifier): 150 | partition, element = identifier.split('/') 151 | path = "%s/%s/%s" % (self.settings.BAREDIR, \ 152 | partition, element) 153 | if not os.path.isdir(path): 154 | raise AssertionError("Bare '%s' not found" % path) 155 | if not os.path.isfile("%s/HEAD" % path): 156 | raise AssertionError("Bare '%s' does not have HEAD" % path) 157 | 158 | def assertNotBare(self, identifier): 159 | try: 160 | self.assertBare(identifier) 161 | except AssertionError: 162 | return 163 | raise AssertionError("Bare '%s' seems present" % identifier) 164 | 165 | def assertEnvironmentLinks(self, environment): 166 | base_path = "%s/%s" % (self.settings.ENVIRONMENTSDIR, environment) 167 | common_hieradata_items = set(self.settings.COMMON_HIERADATA_ITEMS) 168 | if not os.path.isdir(base_path): 169 | raise AssertionError("Environment '%s' not present" % environment) 170 | for path, dirs, files in os.walk(base_path): 171 | if path.endswith('hieradata'): 172 | self.assertTrue(common_hieradata_items.issubset(set(dirs + files))) 173 | for file in files + dirs: 174 | file_apath = "%s/%s" % (path, file) 175 | if os.path.islink(file_apath): 176 | if not self._verify_link(path, file_apath): 177 | raise AssertionError("Environment '%s' -- '%s' link broken" % \ 178 | (environment, file_apath)) 179 | 180 | def assertEnvironmentHasAConfigFile(self, environment): 181 | # https://docs.puppetlabs.com/puppet/latest/reference/config_file_environment.html 182 | conf_file_path = "%s/%s/environment.conf" % \ 183 | (self.settings.ENVIRONMENTSDIR, environment) 184 | if not os.path.isfile(conf_file_path): 185 | raise AssertionError("Environment '%s' doesn't have a config file" % environment) 186 | if not os.stat(conf_file_path).st_size > 0: 187 | raise AssertionError("Environment '%s''s config file seems empty" % environment) 188 | 189 | def assertEnvironmentHasAConfigFileAndParserSet(self, environment, parser): 190 | conf_file_path = "%s/%s/environment.conf" % \ 191 | (self.settings.ENVIRONMENTSDIR, environment) 192 | 193 | config = ConfigObj(conf_file_path) 194 | if config.get('parser', None) != parser: 195 | raise AssertionError("Environment '%s''s parser:'s value is not %s" % 196 | (environment, parser)) 197 | 198 | def assertEnvironmentDoesNotHaveAConfigFile(self, environment): 199 | try: 200 | self.assertEnvironmentHasAConfigFile(environment) 201 | except AssertionError: 202 | return 203 | raise AssertionError("Environment '%s' seems to have a config file" % environment) 204 | 205 | def assertEnvironmentBrokenLinks(self, environment): 206 | try: 207 | self.assertEnvironmentLinks(environment) 208 | except AssertionError: 209 | return 210 | raise AssertionError("Environment '%s' seems fine" % environment) 211 | 212 | def assertEnvironmentDoesntExist(self, environment): 213 | base_path = "%s/%s" % (self.settings.ENVIRONMENTSDIR, environment) 214 | if os.path.isdir(base_path): 215 | raise AssertionError("Environment '%s' present" % environment) 216 | 217 | def assertEnvironmentOverride(self, environment, identifier, desired): 218 | base_path = "%s/%s" % (self.settings.ENVIRONMENTSDIR, environment) 219 | partition, element = identifier.split('/') 220 | links = [] 221 | links.append("%s/%s/%s" % (base_path, partition, element)) 222 | if partition == 'modules': 223 | links.append("%s/hieradata/module_names/%s" % (base_path, element)) 224 | if partition == 'hostgroups': 225 | canonical_name = element.replace('hg_', '') 226 | links.append("%s/hieradata/hostgroups/%s" % \ 227 | (base_path, canonical_name)) 228 | links.append("%s/hieradata/fqdns/%s" % \ 229 | (base_path, canonical_name)) 230 | for link in links: 231 | # Override created 232 | if not os.path.lexists(link): 233 | raise AssertionError("Env '%s' -- '%s' -- '%s' does not exist" % \ 234 | (environment, identifier, link)) 235 | target = os.readlink(link) 236 | dirname = refname_to_dirname(desired) 237 | # Link not broken 238 | link_parent = os.path.abspath(os.path.join(link, os.pardir)) 239 | if not self._verify_link(link_parent, link): 240 | raise AssertionError("Env '%s' -- '%s' -- '%s' is broken" % \ 241 | (environment, identifier, link)) 242 | # And points to the correct clone 243 | link_type = 'data' if re.match(r'^%s/hieradata/.+' % \ 244 | base_path, link) else 'code' 245 | if not re.match(r".+/%s/%s.*" % (dirname, link_type), target): 246 | raise AssertionError("Env '%s' '%s' // '%s' -> '%s' not to '%s' (%s)" % \ 247 | (environment, identifier, link, target, dirname, desired)) 248 | 249 | def assertEnvironmentOverrideDoesntExist(self, environment, identifier): 250 | base_path = "%s/%s" % (self.settings.ENVIRONMENTSDIR, environment) 251 | partition, element = identifier.split('/') 252 | links = [] 253 | links.append("%s/%s/%s" % (base_path, partition, element)) 254 | if partition == 'modules': 255 | links.append("%s/hieradata/module_names/%s" % (base_path, element)) 256 | if partition == 'hostgroups': 257 | canonical_name = element.replace('hg_', '') 258 | links.append("%s/hieradata/hostgroups/%s" % \ 259 | (base_path, canonical_name)) 260 | links.append("%s/hieradata/fqdns/%s" % \ 261 | (base_path, canonical_name)) 262 | for link in links: 263 | if os.path.lexists(link): 264 | raise AssertionError("Env '%s' -- '%s' -- '%s' exists" % \ 265 | (environment, identifier, link)) 266 | 267 | def assertEnvironmentOverrideExistsButBroken(self, environment, identifier, desired): 268 | base_path = "%s/%s" % (self.settings.ENVIRONMENTSDIR, environment) 269 | partition, element = identifier.split('/') 270 | links = [] 271 | links.append("%s/%s/%s" % (base_path, partition, element)) 272 | if partition == 'modules': 273 | links.append("%s/hieradata/module_names/%s" % (base_path, element)) 274 | if partition == 'hostgroups': 275 | canonical_name = element.replace('hg_', '') 276 | links.append("%s/hieradata/hostgroups/%s" % \ 277 | (base_path, canonical_name)) 278 | links.append("%s/hieradata/fqdns/%s" % \ 279 | (base_path, canonical_name)) 280 | for link in links: 281 | # Override created 282 | if not os.path.lexists(link): 283 | raise AssertionError("Env '%s' -- '%s' -- '%s' does not exist" % \ 284 | (environment, identifier, link)) 285 | target = os.readlink(link) 286 | dirname = refname_to_dirname(desired) 287 | link_parent = os.path.abspath(os.path.join(link, os.pardir)) 288 | # Link broken 289 | if self._verify_link(link_parent, link): 290 | raise AssertionError("Env '%s' -- '%s' -- '%s' is not broken" % \ 291 | (environment, identifier, link)) 292 | 293 | def assertEnvironmentNumberOf(self, environment, partition, count): 294 | base_path = "%s/%s/%s" % \ 295 | (self.settings.ENVIRONMENTSDIR, environment, partition) 296 | actual = len(os.listdir(base_path)) 297 | if actual != count: 298 | raise AssertionError("'%s' has %d %s (expected: %d)" % \ 299 | (environment, actual, partition, count)) 300 | 301 | def _verify_link(self, base, path): 302 | cwd = os.getcwd() 303 | os.chdir(base) 304 | try: 305 | os.stat(os.readlink(path)) 306 | except OSError: 307 | return False 308 | finally: 309 | os.chdir(cwd) 310 | return True 311 | -------------------------------------------------------------------------------- /jens/test/tools.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import os 10 | import yaml 11 | import tempfile 12 | import shutil 13 | import time 14 | import pickle 15 | import logging 16 | from subprocess import Popen, PIPE 17 | from datetime import datetime 18 | from dirq.queue import Queue, QueueError, QueueLockError 19 | from jens.settings import Settings 20 | 21 | from jens.errors import JensGitError 22 | from jens.messaging import MSG_SCHEMA 23 | 24 | GITBINPATH = "git" 25 | 26 | def _git(args, gitdir=None, gitworkingtree=None): 27 | env = os.environ.copy() 28 | if gitdir is not None: 29 | logging.debug("Setting GIT_DIR to %s" % gitdir) 30 | env['GIT_DIR'] = gitdir 31 | if gitworkingtree is not None: 32 | logging.debug("Setting GIT_WORK_TREE to %s" % gitworkingtree) 33 | env['GIT_WORK_TREE'] = gitworkingtree 34 | args = [GITBINPATH] + args 35 | logging.debug("Executing git %s" % args) 36 | (returncode, stdout, stderr) = _exec(args, env) 37 | if returncode != 0: 38 | raise JensGitError("Couldn't execute %s (%s)" % \ 39 | (args, stderr.strip())) 40 | return (stdout, returncode) 41 | 42 | def _exec(args, environment): 43 | git = Popen(args, stdout = PIPE, stderr=PIPE, env=environment) 44 | (stdout, stderr) = git.communicate() 45 | return (git.returncode, stdout, stderr) 46 | 47 | def init_sandbox(path): 48 | dirs = [ 49 | "%s/lib/bare/common" % path, 50 | "%s/lib/bare/modules" % path, 51 | "%s/lib/bare/hostgroups" % path, 52 | "%s/lib/clone/common" % path, 53 | "%s/lib/clone/modules" % path, 54 | "%s/lib/clone/hostgroups" % path, 55 | "%s/lib/cache/environments" % path, 56 | "%s/lib/environments" % path, 57 | "%s/lib/metadata/environments" % path, 58 | "%s/lib/metadata/repositories" % path, 59 | "%s/log" % path, 60 | "%s/etc" % path, 61 | "%s/spool" % path, 62 | "%s/repos/user" % path, 63 | "%s/repos/bare" % path] 64 | for _dir in dirs: 65 | os.makedirs(_dir) 66 | 67 | def destroy_sandbox(path): 68 | shutil.rmtree(path) 69 | 70 | def ensure_environment(envname, default, 71 | modules=[], hostgroups=[], common=[], parser=None): 72 | environment = {'notifications': 'higgs@example.org'} 73 | if default is not None: 74 | environment['default'] = default 75 | 76 | if len(modules) + len(hostgroups) + len(common) > 0: 77 | environment['overrides'] = {} 78 | 79 | if len(modules) >= 1: 80 | environment['overrides']['modules'] = {} 81 | for module in modules: 82 | name, override = module.split(':') 83 | environment['overrides']['modules'][name] = override 84 | 85 | if len(hostgroups) >= 1: 86 | environment['overrides']['hostgroups'] = {} 87 | for hostgroup in hostgroups: 88 | name, override = hostgroup.split(':') 89 | environment['overrides']['hostgroups'][name] = override 90 | 91 | if parser: 92 | environment['parser'] = parser 93 | 94 | settings = Settings() 95 | environment_file = open("%s/%s.yaml" % (settings.ENV_METADATADIR, envname), 'w+') 96 | yaml.dump(environment, environment_file, default_flow_style=False) 97 | environment_file.close() 98 | 99 | def destroy_environment(envname): 100 | settings = Settings() 101 | os.remove("%s/%s.yaml" % (settings.ENV_METADATADIR, envname)) 102 | 103 | def init_repositories(): 104 | settings = Settings() 105 | data = {'repositories': {'modules': {}, 106 | 'hostgroups': {}, 107 | 'common': {}}} 108 | repositories_file = open(settings.REPO_METADATA, 'w+') 109 | yaml.dump(data, repositories_file, default_flow_style=False) 110 | repositories_file.close() 111 | 112 | def add_repository(partition, name, url): 113 | settings = Settings() 114 | repositories_file = open(settings.REPO_METADATA, 'r') 115 | data = yaml.safe_load(repositories_file) 116 | repositories_file.close() 117 | data['repositories'][partition][name] = 'file://' + url 118 | repositories_file = open(settings.REPO_METADATA, 'w+') 119 | yaml.dump(data, repositories_file, default_flow_style=False) 120 | repositories_file.close() 121 | 122 | def del_repository(partition, name): 123 | settings = Settings() 124 | repositories_file = open(settings.REPO_METADATA, 'r') 125 | data = yaml.safe_load(repositories_file) 126 | repositories_file.close() 127 | del data['repositories'][partition][name] 128 | repositories_file = open(settings.REPO_METADATA, 'w+') 129 | yaml.dump(data, repositories_file, default_flow_style=False) 130 | repositories_file.close() 131 | 132 | def create_folder_not_repository(base): 133 | not_repo_path = tempfile.mkdtemp(dir="%s/repos/user" % base) 134 | return not_repo_path 135 | 136 | def create_fake_repository(base, branches=[]): 137 | bare_repo_path = tempfile.mkdtemp(dir="%s/repos/bare" % base) 138 | gitdir = "%s" % bare_repo_path 139 | args = ["init", "--bare"] 140 | _git(args, gitdir=gitdir) 141 | repo_path = bare_repo_path.replace('/repos/bare/', '/repos/user/') 142 | gitdir = "%s/.git" % repo_path 143 | args = ["clone", bare_repo_path, repo_path] 144 | _git(args) 145 | fake_file = open("%s/dummy" % repo_path, 'w+') 146 | fake_file.write("foo") 147 | fake_file.close() 148 | os.mkdir("%s/code" % repo_path) 149 | os.mkdir("%s/data" % repo_path) 150 | os.mkdir("%s/data/fqdns" % repo_path) 151 | os.mkdir("%s/data/hostgroup" % repo_path) 152 | os.mkdir("%s/data/operatingsystems" % repo_path) 153 | os.mkdir("%s/data/datacentres" % repo_path) 154 | os.mkdir("%s/data/hardware" % repo_path) 155 | os.mkdir("%s/data/environments" % repo_path) 156 | shutil.copy("%s/dummy" % repo_path, "%s/code" % repo_path) 157 | shutil.copy("%s/dummy" % repo_path, "%s/data/hostgroup" % repo_path) 158 | shutil.copy("%s/dummy" % repo_path, "%s/data/fqdns" % repo_path) 159 | shutil.copy("%s/dummy" % repo_path, "%s/data/common.yaml" % repo_path) 160 | shutil.copy("%s/dummy" % repo_path, "%s/data/operatingsystems" % repo_path) 161 | shutil.copy("%s/dummy" % repo_path, "%s/data/datacentres" % repo_path) 162 | shutil.copy("%s/dummy" % repo_path, "%s/data/hardware" % repo_path) 163 | shutil.copy("%s/dummy" % repo_path, "%s/data/environments" % repo_path) 164 | shutil.copy("%s/dummy" % repo_path, "%s/repositories.yaml" % repo_path) 165 | gitdir = "%s/.git" % repo_path 166 | args = ["init"] 167 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 168 | args = ["add", "dummy", "code", "data", "repositories.yaml"] 169 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 170 | args = ["commit", "-m", "init"] 171 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 172 | for branch in branches: 173 | args = ["checkout", "-b", branch] 174 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 175 | args = ["push", "origin", branch] 176 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 177 | args = ["checkout", "master"] 178 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 179 | fake_file = open("%s/dummy" % repo_path, 'w+') 180 | fake_file.write("bar") 181 | fake_file.close() 182 | args = ["commit", "-a", "-m", "init"] 183 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 184 | fake_file = open("%s/dummy" % repo_path, 'w+') 185 | fake_file.write("baz") 186 | fake_file.close() 187 | args = ["commit", "-a", "-m", "init"] 188 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 189 | args = ["push", "origin", "master"] 190 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 191 | # To simulate client activity, operations like committing, 192 | # removing a branch etc should be done thru repo_path, 193 | # bare_repo_path is returned to be added to the lib only. 194 | return (bare_repo_path, repo_path) 195 | 196 | def get_repository_head(repo_path): 197 | args = ["rev-parse", "HEAD"] 198 | gitdir = "%s/.git" % repo_path 199 | (out, code) =_git(args, gitdir=gitdir, gitworkingtree=repo_path) 200 | return out.strip().decode() 201 | 202 | def add_branch_to_repo(repo_path, branch): 203 | gitdir = "%s/.git" % repo_path 204 | args = ["checkout", "-b", branch] 205 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 206 | args = ["push", "origin", branch] 207 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 208 | 209 | def add_commit_to_branch(repo_path, branch, 210 | force=False, fname=None, remove=False): 211 | if fname is None: 212 | fname = time.time() 213 | gitdir = "%s/.git" % repo_path 214 | args = ["checkout", branch] 215 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 216 | fake_file_path = "%s/%s" % (repo_path, fname) 217 | if not remove: 218 | fake_file = open(fake_file_path, 'w+') 219 | fake_file.write("foo") 220 | fake_file.close() 221 | args = ["add", fake_file_path] 222 | else: 223 | args = ["rm", fake_file_path] 224 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 225 | args = ["commit", "-a", "-m", "update"] 226 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 227 | args = ["push", "origin", branch] 228 | if force: 229 | args.append("--force") 230 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 231 | return get_repository_head(repo_path) 232 | 233 | def remove_branch_from_repo(repo_path, branch): 234 | gitdir = "%s/.git" % repo_path 235 | args = ["checkout", "master"] 236 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 237 | args = ["branch", "-D", branch] 238 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 239 | args = ["push", "origin", ":%s" % branch] 240 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 241 | 242 | def reset_branch_to(repo_path, branch, commit_id): 243 | gitdir = "%s/.git" % repo_path 244 | args = ["checkout", branch] 245 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 246 | args = ["reset", "--hard", commit_id] 247 | _git(args, gitdir=gitdir, gitworkingtree=repo_path) 248 | 249 | ## Messaging 250 | 251 | def add_msg_to_queue(msg): 252 | settings = Settings() 253 | queue = Queue(settings.MESSAGING_QUEUEDIR, schema=MSG_SCHEMA) 254 | queue.enqueue(msg) 255 | 256 | def create_module_event(module): 257 | msg = {'time': datetime.now().isoformat(), 258 | 'data': pickle.dumps({'modules': [module]})} 259 | add_msg_to_queue(msg) 260 | 261 | def create_hostgroup_event(hostgroup): 262 | msg = {'time': datetime.now().isoformat(), 263 | 'data': pickle.dumps({'hostgroups': [hostgroup]})} 264 | add_msg_to_queue(msg) 265 | 266 | def create_common_event(element): 267 | msg = {'time': datetime.now().isoformat(), 268 | 'data': pickle.dumps({'common': [element]})} 269 | add_msg_to_queue(msg) 270 | -------------------------------------------------------------------------------- /jens/tools.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | from __future__ import absolute_import 9 | import re 10 | from jens.settings import Settings 11 | 12 | def refname_to_dirname(refname): 13 | match = ref_is_commit(refname) 14 | if match: 15 | return ".%s" % match.group(1) 16 | return refname 17 | 18 | def dirname_to_refname(dirname): 19 | settings = Settings() 20 | match = re.match(r'^\.([^\.]+)', dirname) 21 | if match: 22 | return "%s%s" % (settings.HASHPREFIX, match.group(1)) 23 | return dirname 24 | 25 | def ref_is_commit(refname): 26 | settings = Settings() 27 | return re.match(r'^%s([0-9A-Fa-f]+)' % settings.HASHPREFIX, 28 | refname, re.IGNORECASE) 29 | -------------------------------------------------------------------------------- /jens/webapps/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2014, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | -------------------------------------------------------------------------------- /jens/webapps/gitlabproducer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2015, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | from __future__ import absolute_import 10 | import fcntl 11 | import json 12 | import logging 13 | import yaml 14 | 15 | from flask import Flask, request, current_app 16 | 17 | from jens.errors import JensError, JensMessagingError 18 | from jens.messaging import enqueue_hint 19 | 20 | app = Flask(__name__) 21 | 22 | @app.route('/gitlab', methods=['POST']) 23 | def hello_gitlab(): 24 | try: 25 | settings = current_app.config['settings'] 26 | 27 | payload = request.get_json(silent=True) or {} 28 | if payload: 29 | logging.debug('Incoming request with payload: %s' % str(payload)) 30 | 31 | if settings.GITLAB_PRODUCER_SECRET_TOKEN: 32 | server_token = request.headers.get('X-Gitlab-Token') 33 | if server_token != settings.GITLAB_PRODUCER_SECRET_TOKEN: 34 | logging.error("Bad Gitlab Token (%s)", server_token) 35 | return 'Unauthorised', 401 36 | 37 | try: 38 | url = payload['repository']['git_ssh_url'] 39 | except (KeyError, TypeError) as error: 40 | logging.error("Malformed payload (%s)" % json.dumps(payload)) 41 | return 'Malformed request', 400 42 | 43 | try: 44 | with open(settings.REPO_METADATA, 'r') as metadata: 45 | fcntl.flock(metadata, fcntl.LOCK_SH) 46 | repositories = yaml.safe_load(metadata)['repositories'] 47 | fcntl.flock(metadata, fcntl.LOCK_UN) 48 | except Exception as error: 49 | raise JensError("Could not read '%s' ('%s')" 50 | "" % (settings.REPO_METADATA, error)) 51 | 52 | for _partition, _mapping in repositories.items(): 53 | for _name, _url in _mapping.items(): 54 | if _url == url: 55 | partition, name = _partition, _name 56 | 57 | enqueue_hint(partition, name) 58 | return 'OK', 201 59 | except JensMessagingError as error: 60 | logging.error("%s/%s couldn't be added to the queue (%s)" % 61 | (partition, name, str(error))) 62 | return 'Queue not accessible', 500 63 | except NameError as error: 64 | logging.error("'%s' couldn't be found in repositories" % (url)) 65 | return 'Repository not found', 200 66 | except Exception as error: 67 | logging.error("Unexpected error (%s)" % repr(error)) 68 | return 'Internal Server Error!', 500 69 | 70 | -------------------------------------------------------------------------------- /jens/webapps/test/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) 2022, CERN 2 | # This software is distributed under the terms of the GNU General Public 3 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 4 | # In applying this license, CERN does not waive the privileges and immunities 5 | # granted to it by virtue of its status as Intergovernmental Organization 6 | # or submit itself to any jurisdiction. 7 | 8 | -------------------------------------------------------------------------------- /jens/webapps/test/test_gitlabproducer.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import 2 | import json 3 | from unittest.mock import patch 4 | 5 | from jens.errors import JensMessagingError 6 | from jens.test.tools import init_repositories 7 | from jens.test.tools import add_repository 8 | from jens.test.tools import create_fake_repository 9 | from jens.test.testcases import JensTestCase 10 | from jens.webapps.gitlabproducer import app as gitlabproducer 11 | from jens.settings import Settings 12 | 13 | class GitlabProducerTestCase(JensTestCase): 14 | 15 | def setUp(self): 16 | super().setUp() 17 | init_repositories() 18 | (bare, _) = create_fake_repository(self.sandbox_path, ['qa']) 19 | add_repository('common', 'site', bare) 20 | self.site_bare = bare 21 | gitlabproducer.config['settings'] = Settings() 22 | self.app = gitlabproducer.test_client() 23 | 24 | def test_get(self): 25 | self.assertEqual(self.app.get('/gitlab').status_code, 405) 26 | 27 | def test_no_payload(self): 28 | reply = self.app.post('/gitlab') 29 | self.assertEqual(reply.data.decode(), 'Malformed request') 30 | self.assertEqual(reply.status_code, 400) 31 | 32 | def test_wrong_payload(self): 33 | reply = self.app.post('/gitlab', data={'iam':'wrong'}, content_type='application/json') 34 | self.assertEqual(reply.data.decode(), 'Malformed request') 35 | self.assertEqual(reply.status_code, 400) 36 | 37 | def test_wrong_payload2(self): 38 | reply = self.app.post('/gitlab', data=json.dumps({'repository':'wrong'}), content_type='application/json') 39 | self.assertEqual(reply.data.decode(), 'Malformed request') 40 | self.assertEqual(reply.status_code, 400) 41 | 42 | def test_no_content_type(self): 43 | reply = self.app.post('/gitlab', 44 | data=json.dumps({'repository': 45 | { 46 | 'name': 'it-puppet-hostgroup-playground', 47 | 'git_ssh_url': 'http://git.cern.ch/cernpub/it-puppet-hostgroup-playground' 48 | } 49 | })) 50 | self.assertEqual(reply.data.decode(), 'Malformed request') 51 | self.assertEqual(reply.status_code, 400) 52 | 53 | @patch('jens.webapps.gitlabproducer.enqueue_hint') 54 | def test_known_repository(self, mock_eq): 55 | reply = self.app.post('/gitlab', content_type='application/json', 56 | data=json.dumps({'repository': 57 | { 58 | 'name': 'it-puppet-site', 59 | 'git_ssh_url': f"file://{self.site_bare}" 60 | } 61 | })) 62 | mock_eq.assert_called_once_with('common', 'site') 63 | self.assertEqual(reply.status_code, 201) 64 | 65 | @patch('jens.webapps.gitlabproducer.enqueue_hint', side_effect=JensMessagingError) 66 | def test_queue_error(self, mock_eq): 67 | reply = self.app.post('/gitlab', content_type='application/json', 68 | data=json.dumps({'repository': 69 | { 70 | 'name': 'it-puppet-site', 71 | 'git_ssh_url': f"file://{self.site_bare}" 72 | } 73 | })) 74 | mock_eq.assert_called_once() 75 | self.assertEqual(reply.data.decode(), 'Queue not accessible') 76 | self.assertEqual(reply.status_code, 500) 77 | 78 | def test_repository_not_found(self): 79 | reply = self.app.post('/gitlab', content_type='application/json', 80 | data=json.dumps({'repository': 81 | { 82 | 'name': 'it-puppet-site', 83 | 'git_ssh_url': "file://foo" 84 | } 85 | })) 86 | self.assertEqual(reply.status_code, 200) 87 | 88 | @patch('jens.webapps.gitlabproducer.enqueue_hint') 89 | def test_secret_token_ignored_if_not_configured(self, mock_eq): 90 | self.settings.GITLAB_PRODUCER_SECRET_TOKEN = None 91 | _payload = { 92 | 'repository': { 93 | 'name': 'it-puppet-site', 94 | 'git_ssh_url': "file://%s" % self.site_bare 95 | } 96 | } 97 | reply = self.app.post('/gitlab', 98 | content_type='application/json', 99 | headers={'X-Gitlab-Token': 'tokenvalue'}, 100 | data=json.dumps(_payload)) 101 | mock_eq.assert_called_once_with('common', 'site') 102 | self.assertEqual(reply.status_code, 201) 103 | 104 | def test_wrong_secret_token(self): 105 | self.settings.GITLAB_PRODUCER_SECRET_TOKEN = 'expected' 106 | _payload = { 107 | 'repository': { 108 | 'name': 'it-puppet-site', 109 | 'git_ssh_url': "file://%s" % self.site_bare 110 | } 111 | } 112 | reply = self.app.post('/gitlab', 113 | content_type='application/json', 114 | data=json.dumps(_payload)) 115 | self.assertEqual(reply.status_code, 401) 116 | 117 | def test_secret_token_configured_wrong_token(self): 118 | self.settings.GITLAB_PRODUCER_SECRET_TOKEN = 'expected' 119 | _payload = { 120 | 'repository': { 121 | 'name': 'it-puppet-site', 122 | 'git_ssh_url': "file://%s" % self.site_bare 123 | } 124 | } 125 | reply = self.app.post('/gitlab', 126 | content_type='application/json', 127 | headers={'X-Gitlab-Token': 'tokenvalue'}, 128 | data=json.dumps(_payload)) 129 | self.assertEqual(reply.status_code, 401) 130 | 131 | @patch('jens.webapps.gitlabproducer.enqueue_hint') 132 | def test_secret_token_configured_good_token(self, mock_eq): 133 | self.settings.GITLAB_PRODUCER_SECRET_TOKEN = 'expected' 134 | _payload = { 135 | 'repository': { 136 | 'name': 'it-puppet-site', 137 | 'git_ssh_url': "file://%s" % self.site_bare 138 | } 139 | } 140 | reply = self.app.post('/gitlab', 141 | content_type='application/json', 142 | headers={'X-Gitlab-Token': 'expected'}, 143 | data=json.dumps(_payload)) 144 | mock_eq.assert_called_once_with('common', 'site') 145 | self.assertEqual(reply.status_code, 201) 146 | -------------------------------------------------------------------------------- /man/jens-gc.1: -------------------------------------------------------------------------------- 1 | .TH JENS-GC "1" "July 2013" "PUPPET-JENS" "User Commands" 2 | .SH NAME 3 | jens-gc \- does garbage collection operations on local Git repositories 4 | .SH SYNOPSIS 5 | .B jens-gc 6 | [\fIOPTION\fR]... 7 | .SH DESCRIPTION 8 | .PP 9 | This is a maintenance tool to basically run git-gc on the bare clones 10 | and on all the branch clones. If there's a lock file present because 11 | jens-update is running, instead of dying jens-gc will try a limited 12 | amount of times to gain to lock and do the clean up. 13 | .PP 14 | It's probably a good idea to run it once a day out of office hours. 15 | .TP 16 | \fB\-c\fR, \fB\-\-config\fR 17 | Path to Jens' configuration file (defaults to /etc/jens/main.conf) 18 | .TP 19 | \fB\-a\fR, \fB\-\-all\fR 20 | Cleans everything 21 | .TP 22 | \fB\-g\fR, \fB\-\-aggressive\fR 23 | Runs git-gc with --aggressive 24 | .TP 25 | \fB\-b\fR, \fB\-\-bare\fR 26 | Cleans bare repositories 27 | .TP 28 | \fB\-l\fR, \fB\-\-clones\fR 29 | Cleans all the clones where the branches are checked out. 30 | .TP 31 | \fB\-\-help\fR 32 | display this help and exit 33 | .SS "Exit status:" 34 | .TP 35 | 0 36 | if OK, 37 | .TP 38 | 2 39 | if there's any problem with the configuration file, 40 | .TP 41 | 3 42 | if default directories validation fails, 43 | .TP 44 | 50 45 | if it was not possible to gain the lock after all the attempts. 46 | .SH EXAMPLES 47 | .TP 48 | jens-gc -g --all 49 | .TP 50 | jens-gc -b 51 | .SH AUTHOR 52 | Written by Nacho Barrientos 53 | .SH "REPORTING BUGS" 54 | Report bugs directly on Github. 55 | .SH COPYRIGHT 56 | Copyright \(co 2013 CERN. 57 | License GPLv3+: GNU GPL version 3 or later . 58 | .br 59 | This is free software: you are free to change and redistribute it. 60 | There is NO WARRANTY, to the extent permitted by law. 61 | .SH "SEE ALSO" 62 | jens-update (1), jens-reset (1) 63 | -------------------------------------------------------------------------------- /man/jens-purge-queue.1: -------------------------------------------------------------------------------- 1 | .TH JENS-PURGE-QUEUE "1" "November 2016" "PUPPET-JENS" "User Commands" 2 | .SH NAME 3 | jens-purge-queue \- cleans up the message queues 4 | .SH SYNOPSIS 5 | .B jens-purge-queue 6 | [\fIOPTION\fR]... 7 | .SH DESCRIPTION 8 | .PP 9 | This program calls purge() on the queues used to share information between 10 | jens-gitlab-producer and jens-update, removing obsolete (and usually empty) 11 | directories in the queue directory. This CLI should be called regularly. 12 | .TP 13 | \fB\-c\fR, \fB\-\-config\fR 14 | Path to Jens' configuration file (defaults to /etc/jens/main.conf) 15 | .TP 16 | \fB\-\-help\fR 17 | display this help and exit 18 | .SS "Exit status:" 19 | .TP 20 | 0 21 | if OK, 22 | .TP 23 | 2 24 | if there's any problem with the configuration file, 25 | .TP 26 | 10 27 | if there are errors purging the queue, 28 | .SH AUTHOR 29 | Written by Nacho Barrientos 30 | .SH "REPORTING BUGS" 31 | Report bugs directly on Github. 32 | .SH COPYRIGHT 33 | Copyright \(co 2016 CERN. 34 | License GPLv3+: GNU GPL version 3 or later . 35 | .br 36 | This is free software: you are free to change and redistribute it. 37 | There is NO WARRANTY, to the extent permitted by law. 38 | .SH "SEE ALSO" 39 | jens-stats (1), jens-update (1), jens-reset (1) 40 | -------------------------------------------------------------------------------- /man/jens-reset.1: -------------------------------------------------------------------------------- 1 | .TH JENS-RESET "1" "July 2013" "PUPPET-JENS" "User Commands" 2 | .SH NAME 3 | jens-reset \- deletes all Jens-generated data 4 | .SH SYNOPSIS 5 | .B jens-reset 6 | [\fIOPTION\fR]... 7 | .SH DESCRIPTION 8 | .PP 9 | This program removes: bare repositories, all clones branches 10 | of bare repositories and all generated environments (data and 11 | caches) 12 | .PP 13 | It might be useful to start a generation process from the beginning 14 | in case of emergency or data corruption. \fBUse carefully\fR. 15 | .TP 16 | \fB\-c\fR, \fB\-\-config\fR 17 | Path to Jens' configuration file (defaults to /etc/jens/main.conf) 18 | .TP 19 | \fB\-y\fR, \fB\-\-yes\fR 20 | Flag to confirm the operation. Won't do anything if not specified. 21 | .TP 22 | \fB\-\-help\fR 23 | display this help and exit 24 | .SS "Exit status:" 25 | .TP 26 | 0 27 | if OK, 28 | .TP 29 | 2 30 | if there's any problem with the configuration file, 31 | .TP 32 | 3 33 | if default directories validation fails, 34 | .TP 35 | 10 36 | if -y/--yes is not passed, 37 | .TP 38 | 50 39 | if jens-update or jens-gc is running (lock file exists). 40 | .SH AUTHOR 41 | Written by Nacho Barrientos 42 | .SH "REPORTING BUGS" 43 | Report bugs directly on Github. 44 | .SH COPYRIGHT 45 | Copyright \(co 2013 CERN. 46 | License GPLv3+: GNU GPL version 3 or later . 47 | .br 48 | This is free software: you are free to change and redistribute it. 49 | There is NO WARRANTY, to the extent permitted by law. 50 | .SH "SEE ALSO" 51 | jens-stats (1), jens-update (1), jens-gc (1) 52 | -------------------------------------------------------------------------------- /man/jens-stats.1: -------------------------------------------------------------------------------- 1 | .TH JENS-STATS "1" "July 2013" "PUPPET-JENS" "User Commands" 2 | .SH NAME 3 | jens-stats \- shows statistics about the local Jens instance 4 | .SH SYNOPSIS 5 | .B jens-stats 6 | [\fIOPTION\fR]... 7 | .SH DESCRIPTION 8 | .\" Add any additional description here 9 | .PP 10 | Prints to standard output several statistics about the repositories 11 | that are registered (along with branches and sizes) and the declared 12 | and synced environments. 13 | .TP 14 | \fB\-c\fR, \fB\-\-config\fR 15 | Path to Jens' configuration file (defaults to /etc/jens/main.conf) 16 | .TP 17 | \fB\-a\fR, \fB\-\-all\fR 18 | Show all available statistics 19 | .TP 20 | \fB\-r\fR, \fB\-\-repositories\fR 21 | Show information about bare repositories and their clones 22 | .TP 23 | \fB\-e\fR, \fB\-\-environments\fR 24 | Show information about environments 25 | .TP 26 | \fB\-i\fR, \fB\-\-inventory\fR 27 | Dump the current inventory 28 | .TP 29 | \fB\-\-help\fR 30 | display this help and exit 31 | .SS "Exit status:" 32 | .TP 33 | 0 34 | if OK, 35 | .TP 36 | 2 37 | if there's any problem with the configuration file, 38 | .TP 39 | 3 40 | if default directories validation fails. 41 | .SH EXAMPLES 42 | .TP 43 | jens-stats --all 44 | .TP 45 | jens-stats -b -r 46 | .SH AUTHOR 47 | Written by Nacho Barrientos 48 | .SH "REPORTING BUGS" 49 | Report bugs directly on Github. 50 | .SH COPYRIGHT 51 | Copyright \(co 2013 CERN. 52 | License GPLv3+: GNU GPL version 3 or later . 53 | .br 54 | This is free software: you are free to change and redistribute it. 55 | There is NO WARRANTY, to the extent permitted by law. 56 | .SH "SEE ALSO" 57 | jens-update (1), jens-reset (1), jens-gc (1) 58 | -------------------------------------------------------------------------------- /man/jens-update.1: -------------------------------------------------------------------------------- 1 | .TH JENS-UPDATE "1" "July 2013" "PUPPET-JENS" "User Commands" 2 | .SH NAME 3 | jens-update \- updates local environments and repositories 4 | .SH SYNOPSIS 5 | .B jens-update 6 | [\fIOPTION\fR]... 7 | .SH DESCRIPTION 8 | .\" Add any additional description here 9 | .PP 10 | This is the core tool of Jens. It basically refreshes the metadata 11 | about available modules/hostgroups, the bare clones, the clones 12 | of all the necessary branches and the generated environments. 13 | .PP 14 | This tool logs into /var/log/jens/jens-update.log by default. 15 | .TP 16 | \fB\-c\fR, \fB\-\-config\fR 17 | Path to Jens' configuration file (defaults to /etc/jens/main.conf) 18 | .TP 19 | \fB\-p\fR, \fB\-\-poll\fR 20 | Force POLL mode, regardless of what's in the config file 21 | .TP 22 | \fB\-\-help\fR 23 | display this help and exit 24 | .SS "Exit status:" 25 | .TP 26 | 0 27 | if OK, 28 | .TP 29 | 2 30 | if there's any problem with the configuration file, 31 | .TP 32 | 3 33 | if default directories validation fails, 34 | .TP 35 | 20 36 | if the metadata (repositories or environments) couldn't be refreshed, 37 | .TP 38 | 30 39 | if the bare and clone repositories couldn't be refreshed, 40 | .TP 41 | 40 42 | if the environments couldn't be refreshed, 43 | .TP 44 | 50 45 | if there's another instance of jens-update running. 46 | .TP 47 | 51 48 | if the locking process utterly failed. 49 | .SH AUTHOR 50 | Written by Nacho Barrientos 51 | .SH "REPORTING BUGS" 52 | Report bugs directly on Github. 53 | .SH COPYRIGHT 54 | Copyright \(co 2013 CERN. 55 | License GPLv3+: GNU GPL version 3 or later . 56 | .br 57 | This is free software: you are free to change and redistribute it. 58 | There is NO WARRANTY, to the extent permitted by law. 59 | .SH "SEE ALSO" 60 | jens-stats (1), jens-reset (1), jens-gc (1) 61 | -------------------------------------------------------------------------------- /puppet-jens.spec: -------------------------------------------------------------------------------- 1 | Summary: Jens is a Puppet modules/hostgroups librarian 2 | Name: puppet-jens 3 | Version: 1.4.1 4 | Release: 2%{?dist} 5 | 6 | License: GPLv3 7 | Group: Applications/System 8 | URL: https://github.com/cernops/jens 9 | Source: %{name}-%{version}.tar.gz 10 | BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root 11 | BuildArch: noarch 12 | 13 | BuildRequires: systemd-rpm-macros, python3-devel, epel-rpm-macros 14 | # The following requires are for the %check 15 | BuildRequires: python3-pyyaml, python3-urllib3, python3-configobj 16 | BuildRequires: python3-dirq, python3-flask, python3-GitPython, git 17 | 18 | Requires: git 19 | Requires(pre): shadow-utils 20 | 21 | %description 22 | Python toolkit to generate Puppet environments dynamically 23 | based on files containing metadata. 24 | 25 | %prep 26 | %setup -q 27 | 28 | %build 29 | %py3_build 30 | 31 | %install 32 | %{__rm} -rf %{buildroot} 33 | %py3_install 34 | %{__install} -D -p -m 644 conf/main.conf %{buildroot}/%{_sysconfdir}/jens/main.conf 35 | mkdir -m 755 -p %{buildroot}/%{_mandir}/man1 36 | %{__install} -D -p -m 644 man/* %{buildroot}/%{_mandir}/man1 37 | mkdir -m 750 -p %{buildroot}/var/lib/jens/bare/modules 38 | mkdir -m 750 -p %{buildroot}/var/lib/jens/bare/hostgroups 39 | mkdir -m 750 -p %{buildroot}/var/lib/jens/bare/common 40 | mkdir -m 750 -p %{buildroot}/var/lib/jens/clone/modules 41 | mkdir -m 750 -p %{buildroot}/var/lib/jens/clone/hostgroups 42 | mkdir -m 750 -p %{buildroot}/var/lib/jens/clone/common 43 | mkdir -m 750 -p %{buildroot}/var/lib/jens/cache/environments 44 | mkdir -m 750 -p %{buildroot}/var/lib/jens/environments 45 | mkdir -m 750 -p %{buildroot}/var/lib/jens/metadata 46 | mkdir -m 750 -p %{buildroot}/var/log/jens/ 47 | mkdir -m 750 -p %{buildroot}/var/spool/jens-update/ 48 | mkdir -m 750 -p %{buildroot}/var/www/jens 49 | %{__install} -D -p -m 755 wsgi/* %{buildroot}/var/www/jens 50 | mkdir -p %{buildroot}%{_tmpfilesdir} 51 | install -m 0644 jens-tmpfiles.conf %{buildroot}%{_tmpfilesdir}/%{name}.conf 52 | mkdir -p %{buildroot}%{_unitdir} 53 | install -p -m 644 systemd/jens-update.service %{buildroot}%{_unitdir}/jens-update.service 54 | install -p -m 644 systemd/jens-purge-queue.service %{buildroot}%{_unitdir}/jens-purge-queue.service 55 | 56 | %check 57 | export EMAIL="noreply@cern.ch" 58 | export GIT_AUTHOR_NAME="RPM build" 59 | export GIT_COMMITTER_NAME="RPM build" 60 | %{__python3} -m unittest 61 | 62 | %clean 63 | %{__rm} -rf %{buildroot} 64 | 65 | %pre 66 | /usr/bin/getent group jens || /usr/sbin/groupadd -r jens 67 | /usr/bin/getent passwd jens || /usr/sbin/useradd -r -g jens -d /var/lib/jens -s /sbin/nologin jens 68 | 69 | %files 70 | %defattr(-,root,root,-) 71 | %doc README.md ENVIRONMENTS.md examples 72 | %{_mandir}/man1/* 73 | /var/www/jens/* 74 | %{python3_sitelib}/* 75 | %{_bindir}/jens-* 76 | %attr(750, jens, jens) /var/lib/jens/* 77 | %attr(750, jens, jens) /var/log/jens 78 | %attr(750, jens, jens) /var/spool/jens-update 79 | %config(noreplace) %{_sysconfdir}/jens/main.conf 80 | %{_tmpfilesdir}/%{name}.conf 81 | %{_unitdir}/jens-update.service 82 | %{_unitdir}/jens-purge-queue.service 83 | 84 | %changelog 85 | * Wed Jun 28 2023 Nacho Barrientos - 1.4.1-2 86 | - Rebuild for RHEL9. 87 | 88 | * Fri Mar 24 2023 Nacho Barrientos - 1.4.1-1 89 | - Switch to Setuptools. 90 | - Adapt SPEC file so the software builds in EL9. 91 | - Run unit tests too when the RPM is built. 92 | - Use built-in unittest.mock instead of mock. 93 | 94 | * Mon Mar 13 2023 Nacho Barrientos - 1.4.0-1 95 | - Gitlab producer: Return 201 if the hint can be enqueued. 96 | - Gitlab producer: Return 200 if the hinted repository is not part of the library. 97 | 98 | * Tue Jan 17 2023 Nacho Barrientos - 1.3.0-1 99 | - Switch to semanting versioning. 100 | - Add support for Gitlab webhook tokens. 101 | - Use Python's unittest runner. 102 | - Close file descriptions explicitly. 103 | - Some Python-3 related code changes: f-strings, super(), etc. 104 | 105 | * Tue May 11 2021 Nacho Barrientos - 1.2-2 106 | - Rebuild for CentOS Stream 8 107 | 108 | * Wed Oct 28 2020 Nacho Barrientos - 1.2-1 109 | - Ship systemd service unit files for jens-update and jens-purge-queue. 110 | 111 | * Tue Jul 07 2020 Nacho Barrientos - 1.1-1 112 | - Python3-only compatibility. 113 | - Use tmpfiles.d to create the lock directory. 114 | 115 | * Mon Jun 29 2020 Nacho Barrientos - 0.25-1 116 | - Minor fixes to the Spec file. 117 | 118 | * Mon Jun 11 2018 Nacho Barrientos - 0.24-1 119 | - Migrate from optparse to argparse. 120 | 121 | * Tue Apr 03 2018 Nacho Barrientos - 0.23-1 122 | - Python 3.x compatibility. 123 | - Spelling and broken links in the documentation. 124 | 125 | * Mon Mar 20 2017 Nacho Barrientos - 0.22-1 126 | - A big bunch of Lint fixes, no new functionality nor bugfixes. 127 | 128 | * Wed Jan 11 2017 Nacho Barrientos - 0.21-1 129 | - Make sure that settings.ENVIRONMENTSDIR exists. 130 | - Add the process ID to the log messages. 131 | - Show the new HEAD when a clone has been updated. 132 | 133 | * Wed Nov 09 2016 Nacho Barrientos - 0.20-1 134 | - Handle AssertionError when doing Git ops 135 | 136 | * Tue Nov 08 2016 Nacho Barrientos - 0.19-1 137 | - Add an option to protect environments. 138 | 139 | * Tue Nov 01 2016 Nacho Barrientos - 0.18-1 140 | - Add jens-purge-queue. 141 | 142 | * Wed Jul 27 2016 Nacho Barrientos - 0.17-1 143 | - Transform the Settings class into a Borg. 144 | - Add an option to set GIT_SSH (fixes AI-4385). 145 | 146 | * Tue Jul 19 2016 Nacho Barrientos - 0.16-1 147 | - Fix git_wrapper so reset(hard=True) actually works. 148 | 149 | * Wed Jul 13 2016 Nacho Barrientos - 0.15-1 150 | - Configure GitPython differently to avoid leaking file descriptors as per 151 | upstream's recommendation. 152 | 153 | * Tue Jul 12 2016 Nacho Barrientos - 0.14-1 154 | - Remove support for etcd. 155 | - Use GitPython for Git operations instead of subprocessing directly. 156 | 157 | * Tue Mar 29 2016 Nacho Barrientos - 0.13-1 158 | - Allow setting 'parser' in environment.conf 159 | 160 | * Mon Jan 11 2016 Nacho Barrientos - 0.12-1 161 | - Add on-demand mode to jens-update. 162 | - Add webapps/gitlabproducer. 163 | 164 | * Thu Sep 03 2015 Nacho Barrientos - 0.11-2 165 | - Prevent RPM from replacing the configuration file. 166 | 167 | * Thu Sep 03 2015 Nacho Barrientos - 0.11-1 168 | - Support variable number of elements in common hieradata. 169 | 170 | * Wed Nov 19 2014 Nacho Barrientos - 0.10-1 171 | - Add support for directory environments 172 | - Add tons of user documentation (README, ENVIRONMENTS, INSTALL) 173 | 174 | * Wed Aug 13 2014 Nacho Barrientos - 0.9-1 175 | - Reset instead of merge when refreshing metadata. 176 | - Add a new testsuite for the metadata refreshing step. 177 | - Add more assertion to some tests. 178 | 179 | * Tue May 13 2014 Nacho Barrientos - 0.8-1 180 | - Ignore malformed keys when reading the desided inventory. 181 | 182 | * Mon Mar 31 2014 Nacho Barrientos - 0.7-1 183 | - Set GIT_HTTP_LOW_SPEED_LIMIT 184 | - Relative path for Git bin 185 | 186 | * Tue Mar 18 2014 Nacho Barrientos - 0.6-1 187 | - No hard timeouts. 188 | - Shared objects for static clones. 189 | 190 | * Mon Mar 10 2014 Nacho Barrientos - 0.5-1 191 | - Add support for etcd locks. 192 | - Add support for commits as overrides. 193 | - Only expand required branches/commits. 194 | - Add soft and hard timeouts for Git calls. 195 | 196 | * Tue Oct 01 2013 Nacho Barrientos - 0.4-1 197 | - Be more chatty about new/changed/deleted branches 198 | - Stop checking for broken links 199 | - Use relative links when binding environments to clones 200 | - Use fcntl to gain the lock 201 | 202 | * Wed Sep 11 2013 Nacho Barrientos - 0.3-1 203 | - Only inform about broken overrides. 204 | 205 | * Wed Sep 04 2013 Ben Jones - 0.2-1 206 | - git clones now reset rather than merged 207 | 208 | * Mon Jul 09 2012 Nacho Barrientos - 0.1-1 209 | - Initial release 210 | -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | # If the logging module is enabled pytest captures all calls to 3 | # logging.*() so the tests that check if errors are written (or not) 4 | # to the log file will fail as there's no log file being created (all 5 | # the logging will be captured by pytest and dumped to stdout). 6 | addopts = -p no:logging -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pyyaml 2 | urllib3 3 | configobj 4 | dirq 5 | flask 6 | gitpython 7 | -------------------------------------------------------------------------------- /scripts/compare_instances.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Copyright (C) 2023, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | # 9 | # This script can compare two Jens instances by looking at 10 | # random environments, random modules and hostgroups. It makes sure 11 | # that the HEAD is the same in both instances for each tuple. 12 | # 13 | # Usage: script.sh MOUNTPOINT_INSTANCE_A MOUNTPOINT_INSTANCE_B ITERATIONS 14 | # 15 | # Example: script.sh /somewhere/aijens-dev08.cern.ch /somewhere/aijens802.cern.ch 100 qa 16 | # 17 | # If the environment is omitted, the environment choices will be 18 | # random too. 19 | # 20 | # It's a good idea to temporarily suspend updates in both instances to 21 | # compare apples to apples. 22 | # 23 | # The script will stop when a test fails. 24 | 25 | if [ $# -lt 3 ]; then 26 | exit 1; 27 | fi 28 | 29 | BASE_A=$1 30 | BASE_B=$2 31 | CHECKS_MAX=$3 32 | FIXED_ENV=$4 33 | 34 | if ! [[ -d $BASE_A ]] || ! [[ -d $BASE_B ]]; then 35 | "The provided mountpoints are not readable directories" 36 | exit 1 37 | fi 38 | 39 | function progress_info { 40 | local CHECK_COUNTER="[$CUR_CHECK/$CHECKS_MAX]" 41 | echo "$CHECK_COUNTER $1" 42 | } 43 | 44 | function compare_paths { 45 | local _PATH=$1 46 | local DIR_A="$BASE_A/$_PATH" 47 | local DIR_B="$BASE_B/$_PATH" 48 | 49 | if ! [[ -h $DIR_B ]]; then 50 | progress_info "FAIL $DIR_B does not exist in $BASE_B" 51 | exit 3 52 | fi 53 | if [[ -d $DIR_A ]] && ! [[ -d $DIR_B ]]; then 54 | progress_info "FAIL $DIR_A is not a broken link (it's broken in B)" 55 | exit 3 56 | elif ! [[ -d $DIR_A ]] && [[ -d $DIR_B ]]; then 57 | progress_info "FAIL $DIR_B is not a broken link (it's broken in A)" 58 | exit 3 59 | elif ! [[ -d $DIR_A ]] && ! [[ -d $DIR_B ]]; then 60 | progress_info "PASS $_PATH (broken links in both trees)" 61 | return 0 62 | fi 63 | 64 | pushd $DIR_A > /dev/null 65 | REV_A=$(git rev-parse HEAD) 66 | popd > /dev/null 67 | pushd $DIR_B > /dev/null 68 | REV_B=$(git rev-parse HEAD) 69 | popd > /dev/null 70 | 71 | if [ "x$REV_A" == "x$REV_B" ]; then 72 | progress_info "PASS $_PATH" 73 | echo -e "\t(A: $REV_A, B: $REV_B)" 74 | else 75 | progress_info "FAILURE $_PATH (A: $REV_A, B: $REV_B)" 76 | date 77 | pushd $DIR_A > /dev/null 78 | git show 79 | popd > /dev/null 80 | pushd $DIR_B > /dev/null 81 | git show 82 | popd > /dev/null 83 | exit 2 84 | fi 85 | 86 | return 0 87 | } 88 | 89 | echo "Comparing $BASE_A to $BASE_B..." 90 | 91 | CUR_CHECK=1 92 | while [ $CUR_CHECK -ne $((CHECKS_MAX+1)) ]; do 93 | 94 | if [[ -z $FIXED_ENV ]]; then 95 | ENV=$(ls $BASE_A/environments/ | shuf -n 1) 96 | else 97 | ENV=$4 98 | fi 99 | MODULE=$(ls $BASE_A/environments/$ENV/modules | shuf -n 1) 100 | HOSTGROUP=$(ls $BASE_A/environments/$ENV/hostgroups | shuf -n 1) 101 | 102 | compare_paths "environments/$ENV/modules/$MODULE" 103 | compare_paths "environments/$ENV/hostgroups/$HOSTGROUP" 104 | compare_paths "environments/$ENV/site" 105 | compare_paths "environments/$ENV/hieradata/operatingsystems" 106 | 107 | CUR_CHECK=$((CUR_CHECK+1)) 108 | echo 109 | done 110 | 111 | echo "Done, $CHECKS_MAX tests executed" 112 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | from __future__ import absolute_import 4 | from setuptools import setup 5 | 6 | try: 7 | with open("requirements.txt", encoding='utf-8') as requirements: 8 | INSTALL_REQUIRES = [req.strip() for req in requirements.readlines()] 9 | except OSError: 10 | INSTALL_REQUIRES = None 11 | 12 | setup(name='jens', 13 | version='1.4.1', 14 | description='Jens is a Puppet modules/hostgroups librarian', 15 | classifiers=[ 16 | 'Programming Language :: Python :: 3', 17 | 'Programming Language :: Python :: 3 :: Only', 18 | 'License :: OSI Approved :: GNU General Public License v3 (GPLv3)', 19 | ], 20 | author='Nacho Barrientos', 21 | author_email='nacho.barrientos@cern.ch', 22 | url='https://github.com/cernops/jens', 23 | install_requires=INSTALL_REQUIRES, 24 | packages=[ 25 | 'jens', 'jens.webapps' 26 | ], 27 | scripts=[ 28 | 'bin/jens-update', 'bin/jens-stats', 29 | 'bin/jens-gitlab-producer-runner', 'bin/jens-purge-queue', 30 | 'bin/jens-reset', 'bin/jens-gc', 'bin/jens-config' 31 | ], 32 | ) 33 | -------------------------------------------------------------------------------- /systemd/jens-purge-queue.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=jens-purge-queue cycle to clean dead messages in the hints queue 3 | 4 | [Service] 5 | ExecStart=/usr/bin/jens-purge-queue 6 | WorkingDirectory=/var/lib/jens 7 | Restart=no 8 | SyslogIdentifier=jens-purge-queue 9 | User=jens 10 | Group=jens 11 | Type=oneshot 12 | 13 | [Install] 14 | WantedBy=multi-user.target 15 | -------------------------------------------------------------------------------- /systemd/jens-update.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=jens-update cycle to update modules, hostgroups and environments 3 | 4 | [Service] 5 | ExecStart=/usr/bin/jens-update 6 | WorkingDirectory=/var/lib/jens 7 | Restart=no 8 | SyslogIdentifier=jens-update 9 | User=jens 10 | Group=jens 11 | Type=oneshot 12 | 13 | [Install] 14 | WantedBy=multi-user.target 15 | -------------------------------------------------------------------------------- /wsgi/gitlab-producer.wsgi: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # Copyright (C) 2015, CERN 3 | # This software is distributed under the terms of the GNU General Public 4 | # Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING". 5 | # In applying this license, CERN does not waive the privileges and immunities 6 | # granted to it by virtue of its status as Intergovernmental Organization 7 | # or submit itself to any jurisdiction. 8 | 9 | from jens.settings import Settings 10 | from jens.webapps.gitlabproducer import app as application 11 | 12 | settings = Settings('jens-gitlab-producer') 13 | settings.parse_config('/etc/jens/main.conf') 14 | application.config['settings'] = settings 15 | --------------------------------------------------------------------------------