├── CI ├── 00-README.md ├── 01-Gitlab+Gerrit+Jenkins │ ├── 01-基础知识介绍、环境设置、规划.md │ ├── 02-Gitlab安装配置.md │ ├── 03-Gerrit安装配置.md │ ├── 04-Jenkins安装配置.md │ └── 05-整合Gitlab,Gerrit,Jenkins.md ├── 02-Jenkins_Job_Builder │ └── 01-Jenkins job builder.md └── 03-fabric │ ├── 01-autossh+haproxy打通网络.md │ └── 02-fabric远程部署.md ├── Ceilometer └── 01-Ceilometer简介.md ├── Horizon └── __init__.py ├── Keystone └── __init__.py ├── Neutron └── __init__.py └── README.md /CI/00-README.md: -------------------------------------------------------------------------------- 1 | # 00-README 2 | 3 | 4 | 5 | 7 | 8 | ## Jenkins 学习系列 9 | 10 | - [Jenkins 入门进阶系列](http://www.cnblogs.com/zz0412/tag/jenkins/) 11 | 12 | - [Jenkins 入门教程](http://www.cnblogs.com/itech/archive/2011/11/23/2260009.html) 13 | 14 | ## Jenkins Job Builder 15 | - [jenkins\_job\_builder官方文档](http://docs.openstack.org/infra/jenkins-job-builder/) 16 | 17 | - [manage-jenkins-jobs-with-yaml](http://techs.enovance.com/6006/manage-jenkins-jobs-with-yaml) 18 | - [pyYAML官方文档](http://pyyaml.org/wiki/PyYAMLDocumentation) 19 | 20 | ## Fabric 21 | 22 | - [fabric 实现远程操作和部署](http://wklken.me/posts/2013/03/25/python-tool-fabric.html) 23 | 24 | - [fabric官网](http://docs.fabfile.org/en/latest/tutorial.html) 25 | 26 | [python-jenkins包](https://python-jenkins.readthedocs.org/en/latest/) 27 | 28 | [windows安装git-review及常见报错Troubleshoot](https://www.mediawiki.org/wiki/Gerrit/git-review#Windows) -------------------------------------------------------------------------------- /CI/01-Gitlab+Gerrit+Jenkins/01-基础知识介绍、环境设置、规划.md: -------------------------------------------------------------------------------- 1 | # 01-基础知识介绍、环境设置、规划 2 | 3 | 4 | 5 | 7 | 8 | 基础概念 9 | 10 | - git 11 | - git push 12 | - git review 13 | - gitlab 14 | - gerrit 15 | - git clone --bare 16 | - replication 17 | - jenkins 18 | - gerrit trigger 19 | - fabric 20 | - jenkins job builder(yaml) 21 | - Testing 22 | - .testr.conf 23 | - tox.ini 24 | 25 | 工作流 26 | 27 | ``` 28 | gitlab: 29 | 创建 project 30 | gerrit: 31 | 克隆该仓库并配置 replication 32 | 配置与 gitlab、gerrit 互信, 允许 jenkins 匿名访问 33 | 增加 verified 选项 34 | jenkins: 35 | 配置 gerrit trigger 监听 gerrit event stream 36 | 配置新的任务(jenkins_job_builder、fabric) 37 | ``` 38 | OpenStack Workflow 39 | ![](/Users/frank/Documents/marboo/media/CI/showcase/openstack-workflow.png) 40 | 41 | 软件版本 42 | 43 | - gitlab `6.4.3 3173626` 44 | - gerrit `2.8.1` 45 | - jenkins `ver. 1.565.3` 46 | 47 | 系统版本 48 | 49 | - ubuntu `12.04.4-server-amd64` 50 | 51 | 设置主机名 52 | 53 | ```bash 54 | hostname www.pystack.com 55 | sysctl -w kernel.hostname=www.pystack.com 56 | echo www.pystack.com > /etc/hostname 57 | echo '192.168.8.224 www.pystack.com www' >> /etc/hosts
 58 | ``` 59 | 修改 ssh 时候不需要输入 yes，如果不设置后面 Jenkins 连接 Gerrit 时候会报错 60 | 61 | ```bash 62 | echo 'StrictHostKeyChecking no' >> /etc/ssh/ssh_config 63 | /etc/init.d/ssh restart 64 | ``` 65 | 66 | 三套系统默认都监听8080端口，因此需要规划三个域名和端口 67 | 68 | ```bash 69 | gitlab.pystack.com:8081 70 | review.pystack.com:8082 71 | jenkins.pystack.com:8083 72 | ``` 73 | 74 | 注册四个邮箱用于系统账户，一个普通邮箱用于测试用户，其中 admin@pystack.com 用来管理三套系统 75 | 76 | ```bash 77 | admin@pystack.com 78 | gitlab@pystack.com 79 | gerrit@pystack.com 80 | jenkins@pystack.com 81 | ``` 82 | 83 | 设置解析 84 | 85 | ```bash 86 | vim /etc/hosts 87 | 192.168.8.224 gitlab.pystack.com gitlab 88 | 192.168.8.224 review.pystack.com review 89 | 192.168.8.224 jenkins.pystack.com jenkins 90 | ``` 91 | > [CI系统搭建参考文档系列](http://longgeek.com/category/ci/) 92 | > 93 | > [OpenStack开发工作流](http://docs.openstack.org/infra/manual/developers.html) -------------------------------------------------------------------------------- /CI/01-Gitlab+Gerrit+Jenkins/02-Gitlab安装配置.md: -------------------------------------------------------------------------------- 1 | # 02-Gitlab的安装配置 2 | 3 | 4 | 5 | 7 | [toc] 8 | ## 设置源 9 | 10 | 设置国内163的源 11 | 12 | ```bash 13 | # vim /etc/apt/sources.list 14 | deb http://mirrors.163.com/ubuntu/ precise main universe restricted multiverse 15 | deb http://mirrors.163.com/ubuntu/ precise-security universe main multiverse restricted 16 | deb http://mirrors.163.com/ubuntu/ precise-updates universe main multiverse restricted 17 | deb http://mirrors.163.com/ubuntu/ precise-proposed universe main multiverse restricted 18 | deb http://mirrors.163.com/ubuntu/ precise-backports universe main multiverse restricted 19 | 20 | deb-src http://mirrors.163.com/ubuntu/ precise main universe restricted multiverse 21 | deb-src http://mirrors.163.com/ubuntu/ precise-security universe main multiverse restricted 22 | deb-src http://mirrors.163.com/ubuntu/ precise-proposed universe main multiverse restricted 23 | deb-src http://mirrors.163.com/ubuntu/ precise-backports universe main multiverse restricted 24 | deb-src http://mirrors.163.com/ubuntu/ precise-updates universe main multiverse restricted 25 | # apt-get update 26 | ``` 27 | 28 | ## 安装依赖包 29 | 30 | Gitlab 依赖包、库 31 | 32 | ```bash 33 | sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev \ 34 | libncurses5-dev libffi-dev curl openssh-server redis-server checkinstall \ 35 | libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate 36 | ``` 37 | 38 | 安装 markdown 文档风格依赖包 39 | 40 | ```bash 41 | sudo apt-get install -y python-docutils 42 | ``` 43 | 44 | 安装 git，Gerrit 依赖 gitweb，同时 GitLab 依赖 git 版本 >= 1.7.10，Ubuntu apt-get 默认安装的是 1.7.9.5 45 | 46 | **git-review用于发送代码审查请求** 47 | 48 | ```bash 49 | sudo apt-get install -y git gitweb git-review 50 | sudo apt-get install -y libcurl4-openssl-dev libexpat1-dev gettext libz-dev libssl-dev build-essential 51 | cd /tmp 52 | curl --progress https://git-core.googlecode.com/files/git-1.8.4.1.tar.gz | tar xz 53 | cd git-1.8.4.1/ 54 | cd git-1.8.4.1/ 55 | make prefix=/usr/local all 56 | sudo make prefix=/usr/local install 57 | ``` 58 | 59 | Gitlab 需要收发邮件，安装邮件服务器 60 | 61 | ```bash 62 | sudo apt-get install -y postfix 63 | ``` 64 | 65 | 如果安装了 ruby1.8，卸载掉，Gitlab 依赖 2.0 以上 66 | 67 | ```bash 68 | sudo apt-get remove ruby1.8 69 | ``` 70 | 71 | 下载编译 ruby2.0 72 | 73 | ```bash 74 | mkdir /tmp/ruby && cd /tmp/ruby 75 | wget ftp://ftp.ruby-lang.org/pub/ruby/2.0/ruby-2.0.0-p353.tar.gz | tar xz 76 | cd ruby-2.0.0-p353 77 | ./configure --disable-install-rdoc 78 | make 79 | sudo make install 80 | ``` 81 | 82 | 修改 gem 源指向 taobao 83 | 84 | ```bash 85 | gem source -r https://rubygems.org/ 86 | gem source -a http://ruby.taobao.org/ 87 | ``` 88 | 89 | 安装 Bundel 命令 90 | 91 | ```bash 92 | sudo gem install bundler --no-ri --no-rdoc 93 | ``` 94 | 95 | ## 创建用户 96 | 97 | 给 Gitlab 创建一个 git 用户 98 | 99 | ```bash 100 | sudo adduser --disabled-login --gecos 'GitLab' git 101 | ``` 102 | 103 | ## Gitlab Shell 104 | 105 | 下载 Gitlab Shell，用来 ssh 访问仓库的管理软件 106 | 107 | ==**注意需要切换分支,高版本的gitlab-shell API有改动**== 108 | 109 | ```bash 110 | cd /home/git 111 | sudo -u git -H git clone https://github.com/gitlabhq/gitlab-shell.git 112 | cd gitlab-shell 113 | git checkout -b v1.8.0 v1.8.0 114 | sudo -u git -H cp config.yml.example config.yml 115 | ``` 116 | 117 | 修改 gitlab-shell/config.yml 118 | 119 | ```bash 120 | sudo -u git -H vim /home/git/gitlab-shell/config.yml 121 | gitlab_url: "http://gitlab.pystack.com/" 122 | ``` 123 | 124 | 安装 GitLab Shell 125 | 126 | ```bash 127 | cd /home/git/gitlab-shell 128 | sudo -u git -H ./bin/install 129 | ``` 130 | 131 | ## Mysql 132 | 133 | 安装 Mysql 包 134 | 135 | ```bash 136 | sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev 137 | ``` 138 | 139 | 给 Gitlab 创建 Mysql 数据库并授权用户访问 140 | 141 | ```bash 142 | sudo mysql -uroot -p 143 | > create database gitlabdb; 144 | > grant all on gitlabdb.* to 'gitlabuser'@'localhost' identified by 'gitlabpass'; 145 | ``` 146 | 147 | ## Gitlab 148 | 149 | 下载 GitLab 源代码，并切换到对应的分支上 150 | 151 | ```bash 152 | cd /home/git 153 | sudo -u git -H git clone https://github.com/gitlabhq/gitlabhq.git gitlab 154 | cd gitlab 155 | sudo -u git -H git checkout 6-4-stable 156 | ``` 157 | 158 | 配置 GitLab，修改 gitlab.yml，其中 host： 项和 gitlab-shell 中 gitlab_url 的主机一致 159 | 160 | ```bash 161 | cd /home/git/gitlab 162 | sudo -u git -H cp config/gitlab.yml.example config/gitlab.yml 163 | sudo -u git -H vim config/gitlab.yml 164 | host: gitlab.pystack.com 165 | email_from: gitlab@pystack.com 166 | support_mail: gitlab@pystack.com 167 | signup_enabled: true # 开启用户注册 168 | bin_path: /usr/local/bin/git # git可执行文件路径 169 | ``` 170 | 171 | 创建相关目录 172 | 173 | ```bash 174 | cd /home/git/gitlab 175 | sudo -u git -H mkdir tmp/pids/ tmp/sockets/ public/uploads /home/git/repositories /home/git/gitlab-satellites 176 | ``` 177 | 178 | 修改相关目录权限 179 | 180 | ```bash 181 | sudo chown -R git:git log/ tmp/ 182 | sudo chmod -R u+rwX log/ tmp/ public/uploads 183 | ``` 184 | 185 | 修改 unicorn.rb 监听端口为：8081 186 | 187 | ```bash 188 | cd /home/git/gitlab/ 189 | sudo -u git -H cp config/initializers/rack_attack.rb.example config/initializers/rack_attack.rb 190 | sudo -u git -H cp config/unicorn.rb.example config/unicorn.rb 191 | sudo -u git -H vim config/unicorn.rb 192 | listen "gitlab.pystack.com:8081", :tcp_nopush => true 193 | ``` 194 | 195 | 配置 GitLab 访问 mysql 数据库设置 196 | 197 | ```bash 198 | cd /home/git/gitlab/ 199 | sudo -u git cp config/database.yml.mysql config/database.yml 200 | sudo -u git -H vim config/database.yml 201 | *修改 Production 部分:* 202 | production: 203 | adapter: mysql2 204 | encoding: utf8 205 | reconnect: false 206 | database: gitlabdb 207 | pool: 10 208 | username: gitlabuser 209 | password: "gitlabpass" 210 | host: localhost 211 | socket: /var/run/mysqld/mysqld.sock 212 | ``` 213 | 214 | 根据所选用的邮件服务厂商 215 | 216 | 设置 GitLab 使用指定邮箱发送邮件，注意 production.rb 的文件格式，开头空两格 217 | 218 | ```bash 219 | cd /home/git/gitlab/ 220 | sudo -u git -H vim config/environments/production.rb 221 | #修改 :sendmail 为 :smtp 222 | config.action_mailer.delivery_method = :smtp 223 | config.action_mailer.smtp_settings = { 224 | :address => "smtp.exmail.qq.com", 225 | :port => 25, 226 | :domain => 'plcloud.com', 227 | :user_name => 'git@plcloud.com', 228 | :password => 'ljm140920gumi', 229 | :authentication => :plain, 230 | :enable_starttls_auto => true 231 | } 232 | end # 上面内容加入到 end 里面 233 | ``` 234 | 235 | 安装 gem, 修改 Gemfile 文件中源指向为 taobao 236 | 237 | ```bash 238 | cd /home/git/gitlab/ 239 | sudo -u git -H vim Gemfile 240 | source "http://ruby.taobao.org/" 241 | ``` 242 | 243 | ```bash 244 | cd /home/git/gitlab/ 245 | sudo -u git -H bundle install --deployment --without development test postgres aws 246 | ``` 247 | 如果报 modernizer-2.6.2 无法找到的错误,是因为modernizer-2.6.2包已经下架 248 | 249 | 参考[modernizr2.6.2依赖修复](http://stackoverflow.com/questions/22825497/installing-gitlab-missing-modernizer)进行修复, 修改相应文件后重新执行上述命令 250 | 251 | 修改 Gemfile `"modernizr", "2.6.2"` 到 `"modernizr-rails", "2.7.1"` 252 | 253 | 修改 Gemfile.lock `modernizr (2.6.2)` 到 `modernizr-rails (2.7.1)` 254 | 255 | 修改 Gemfile.lock `modernizr (= 2.6.2)` 到 `modernizr-rails (= 2.7.1)` 256 | 257 | 初始化数据库并激活高级功能 258 | 259 | ```bash 260 | cd /home/git/gitlab/ 261 | sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production 262 | ``` 263 | 264 | 输入 yes 来初始化数据库、创建相关表，最后会输出 GitLab Web 管理员用来登录的账号和密码 265 | 266 | ```bash 267 | Do you want to continue (yes/no)? yes 268 | ... 269 | Administrator account created: 270 | login.........admin@local.host 271 | password......5iveL!fe 272 | ``` 273 | 274 | 设置 GitLab 启动服务 275 | 276 | ```bash 277 | cd /home/git/gitlab/ 278 | sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab 279 | sudo update-rc.d gitlab defaults 21 280 | ``` 281 | 282 | 设置 GitLab 使用 Logrotate 备份 Log 283 | 284 | ```bash 285 | cd /home/git/gitlab/ 286 | sudo cp lib/support/logrotate/gitlab /etc/logrotate.d/gitlab 287 | ``` 288 | 289 | 检查GitLab及其环境的配置是否正确： 290 | 291 | ```bash 292 | sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production 293 | 294 | System information 295 | System: Ubuntu 12.04 296 | Current User: git 297 | Using RVM: no 298 | Ruby Version: 2.0.0p353 299 | Gem Version: 2.0.14 300 | Bundler Version:1.10.6 301 | Rake Version: 10.1.0 302 | 303 | GitLab information 304 | Version: 6.4.3 305 | Revision: 3173626 306 | Directory: /home/git/gitlab 307 | DB Adapter: mysql2 308 | URL: http://gitlab.pystack.com 309 | HTTP Clone URL: http://gitlab.pystack.com/some-project.git 310 | SSH Clone URL: git@gitlab.pystack.com:some-project.git 311 | Using LDAP: no 312 | Using Omniauth: no 313 | 314 | GitLab Shell 315 | Version: 1.8.0 316 | Repositories: /home/git/repositories/ 317 | Hooks: /home/git/gitlab-shell/hooks/ 318 | Git: /usr/local/bin/git 319 | ``` 320 | 321 | 启动 GitLab 服务 322 | 323 | ```bash 324 | /etc/init.d/gitlab restart 325 | Shutting down both Unicorn and Sidekiq. 326 | GitLab is not running. 327 | Starting both the GitLab Unicorn and Sidekiq.. 328 | The GitLab Unicorn web server with pid 17771 is running. 329 | The GitLab Sidekiq job dispatcher with pid 17778 is running. 330 | GitLab and all its components are up and running 331 | ``` 332 | 333 | **选做**:如果启动服务报错,需要执行检查命令并按照提示修复报错信息 334 | 335 | 也可以参考[官网排错指南](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Trouble-Shooting-Guide) 336 | 337 | ```bash 338 | # 检查一 339 | sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production 340 | 341 | # 检查二 342 | cd /home/git/gitlab-shell 343 | ./bin/check 344 | ``` 345 | 346 | 最后编译一下 347 | 348 | ```bash 349 | sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production 350 | ``` 351 | 352 | ## Nginx 353 | 354 | 安装 Nginx 包 355 | 356 | ```bash 357 | apt-get install -y nginx 358 | ``` 359 | 360 | 配置 Nginx 361 | 362 | ```bash 363 | cd /home/git/gitlab 364 | sudo cp lib/support/nginx/gitlab /etc/nginx/sites-available/gitlab 365 | sudo ln -s /etc/nginx/sites-available/gitlab /etc/nginx/sites-enabled/gitlab 366 | sudo vim /etc/nginx/sites-available/gitlab 367 | listen *:80 default_server; 368 | server_name gitlab.pystack.com; 369 | proxy_pass http://gitlab.pystack.com:8081; 370 | ``` 371 | 372 | 启动 Nginx 373 | 374 | ```bash 375 | /etc/init.d/apache2 stop 376 | /etc/init.d/nginx restart 377 | ``` 378 | 379 | 访问 380 | 381 | ```bash 382 | 用浏览器访问: http://gitlab.pystack.com 383 | 用户名：admin@local.host 384 | 密码：5iveL!fe 385 | ``` 386 | 387 | ## 界面使用 388 | 389 | - 使用默认账户 admin@local.host 登陆后重置 admin 密码,然后重新登陆 390 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/00-gitlab-login.png) 391 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/01-admin-reset-passwd.png) 392 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/02-welcome.png) 393 | - 进入欢迎页面点右上角用户个人设置,重置绑定邮箱,并去邮箱验证 394 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/03-reset-email.png) 395 | - 创建用户组 devgroup,创建新项目 openstack 396 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/04-new-group.png) 397 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/05-create-project.png) 398 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/06-new-project.png) 399 | - 添加 ssh key, 生成 ssh key, 填入文本框 400 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/07-add-ssh.png) 401 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/08-admin-gen-ssh.png) 402 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/09-admin-add-ssh.png) 403 | - 创建测试用户 frank ,生成 ssh key,填入文本框 404 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/10-signup-new-user-frank.png) 405 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/11-adduser-frank.png) 406 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/12-frank-gen-ssh.png) 407 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/13-frank-add-ssh.png) 408 | - admin 用户重新登陆后添加 frank 至 devgroup ,分配其在 openstack 项目权限为 reporter 409 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/14-admin-area.png) 410 | ![](/Users/frank/Documents/marboo/media/CI/gitlab/15-admin-add-frank.png) 411 | - 此时用 frank 重新登陆后可以看见 openstack 项目, 412 | 413 | 配置`git config user.name 'frank'`, 414 | 415 | 配置`git config user.email 'rainyday_hubei@163.com` 416 | 417 | 进行修改、提交push请求,因为是 reporter 身份, 所以只能 clone 项目,不能 push 418 | 419 | > [github安装指导](https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md) 420 | > 421 | > [gitlab-shell安装文档](https://github.com/gitlabhq/gitlab-shell) -------------------------------------------------------------------------------- /CI/01-Gitlab+Gerrit+Jenkins/03-Gerrit安装配置.md: -------------------------------------------------------------------------------- 1 | # 03-gerrit安装配置 2 | 3 | 4 | 5 | 7 | 8 | ## 下载 Gerrit 安装包 9 | 10 | ```bash 11 | wget http://gerrit-releases.storage.googleapis.com/gerrit-2.8.war 12 | ``` 13 | 14 | ## 安装依赖包 15 | 16 | Gerrit 的安装包是 java 格式,需要安装 jre 17 | 18 | ```bash 19 | apt-get install default-jre daemon 20 | ``` 21 | 22 | ## 创建 Gerrit 数据库 23 | 24 | ```bash 25 | sudo mysql -uroot -p 26 | > create database gerritdb; 27 | > grant all on gerritdb.* to 'gerrituser'@'localhost' identified by 'gerritpass'; 28 | ``` 29 | 30 | ## 开始安装 31 | 32 | 把 gerrit 安装再 /etc/gerrit/ 下 33 | 34 | ```bash 35 | # java -jar gerrit-2.8.war init -d /etc/gerrit/ 36 | 37 | *** Gerrit Code Review 2.8 38 | *** 39 | 40 | Create '/etc/gerrit' [Y/n]? y 41 | 42 | *** Git Repositories 43 | *** 44 | 45 | Location of Git repositories [git]: 46 | 47 | *** SQL Database 48 | *** 49 | 50 | Database server type [h2]: mysql 51 | 52 | Gerrit Code Review is not shipped with MySQL Connector/J 5.1.21 53 | ** This library is required for your configuration. ** 54 | Download and install it now [Y/n]? y 55 | Downloading http://repo2.maven.org/maven2/mysql/mysql-connector-java/5.1.21/mysql-connector-java-5.1.21.jar ... OK 56 | Checksum mysql-connector-java-5.1.21.jar OK 57 | Server hostname [localhost]: 58 | Server port [(mysql default)]: 59 | Database name [reviewdb]: gerritdb 60 | Database username [root]: gerrituser 61 | gerrituser's password : 62 | confirm password : 63 | 64 | *** User Authentication 65 | *** 66 | 67 | Authentication method [OPENID/?]: http 68 | Get username from custom HTTP header [y/N]? n 69 | SSO logout URL : 70 | 71 | *** Email Delivery 72 | *** 73 | 74 | SMTP server hostname [localhost]: smtp.exmail.qq.com 75 | SMTP server port [(default)]: 25 76 | SMTP encryption [NONE/?]: 77 | SMTP username [root]: git@plcloud.com 78 | review@ci-example.com's password : 79 | confirm password : 80 | 81 | *** Container Process 82 | *** 83 | 84 | Run as [root]: 85 | Java runtime [/usr/lib/jvm/java-6-openjdk-amd64/jre]: 86 | Copy gerrit-2.8.war to /etc/gerrit/bin/gerrit.war [Y/n]? y 87 | Copying gerrit-2.8.war to /etc/gerrit/bin/gerrit.war 88 | 89 | *** SSH Daemon 90 | *** 91 | 92 | Listen on address [*]: 93 | Listen on port [29418]: 94 | 95 | Gerrit Code Review is not shipped with Bouncy Castle Crypto v144 96 | If available, Gerrit can take advantage of features 97 | in the library, but will also function without it. 98 | Download and install it now [Y/n]? y 99 | Downloading http://www.bouncycastle.org/download/bcprov-jdk16-144.jar ... OK 100 | Checksum bcprov-jdk16-144.jar OK 101 | Generating SSH host key ... rsa... dsa... done 102 | 103 | *** HTTP Daemon 104 | *** 105 | 106 | Behind reverse proxy [y/N]? y 107 | Proxy uses SSL (https://) [y/N]? n 108 | Subdirectory on proxy server [/]: 109 | Listen on address [*]: 110 | Listen on port [8081]: 8082 111 | Canonical URL [http://www.ci-example.com/]: http://review.ci-example.com/ 112 | 113 | *** Plugins 114 | *** 115 | 116 | Install plugin reviewnotes version v2.8 [y/N]? y 117 | Install plugin download-commands version v2.8 [y/N]? y 118 | Install plugin replication version v2.8 [y/N]? y 119 | Install plugin commit-message-length-validator version v2.8 [y/N]? y 120 | 121 | Initialized /etc/gerrit 122 | Executing /etc/gerrit/bin/gerrit.sh start 123 | Starting Gerrit Code Review: OK 124 | Waiting for server on review.ci-example.com:80 ... OK 125 | Opening http://review.ci-example.com/#/admin/projects/ ...FAILED 126 | Open Gerrit with a JavaScript capable browser: 127 | 128 | http://review.ci-example.com/#/admin/projects/ 129 | ``` 130 | 131 | 在安装完后 Gerrit 默认会打开浏览器，由于我的系统是 Server 版没有桌面和浏览器，所以才会出现上面倒数第三行的 …FAILED 同时从上面的信息看出我使用了 http 方式验证、启用代理服务器，指定了 Gerrit 端口为 8082，URL 为 review.ci-example.com。需要在 nginx 里设置下端口转发。 132 | 133 | gerrit邮件配置sendmail 134 | 135 | ```bash 136 | vim /etc/gerrit/etc/gerrit.config 137 | [sendmail] 下增加 from = git@plcloud.com 否则邮件会被拦截 138 | ``` 139 | 140 | Gerrit 启动脚本 141 | 142 | ```bash 143 | cp /etc/gerrit/bin/gerrit.sh /etc/init.d/gerrit 144 | vim /etc/init.d/gerrit 145 | GERRIT_SITE=/etc/gerrit/ # 在代码 47 行增加 146 | sudo update-rc.d gerrit defaults 21 147 | service gerrit restart 148 | ``` 149 | 150 | ## Nginx 151 | 152 | 修改 Nginx 配置文件，给 Gerrit 做端口转发和访问控制，把下面内容写入到文件最后 153 | 154 | ```bash 155 | # vim /etc/nginx/sites-available/gitlab 156 | server { 157 | listen *:80; 158 | server_name review.ci-example.com; 159 | allow all; 160 | deny all; 161 | auth_basic "Review System Login"; 162 | auth_basic_user_file /etc/gerrit/etc/htpasswd.conf; 163 | 164 | location / { 165 | proxy_pass http://review.ci-example.com:8082; 166 | } 167 | } 168 | ``` 169 | 170 | 创建 htpasswd.conf 文件，并添加 admin 用户、密码到文件中 171 | 172 | ```bash 173 | # touch /etc/gerrit/etc/htpasswd.conf 174 | # htpasswd /etc/gerrit/etc/htpasswd.conf admin 175 | New password: 176 | Re-type new password: 177 | Adding password for user admin 178 | ``` 179 | 180 | 默认第一个登录 Gerrit 的用户是 Admin。 181 | 182 | ## 访问 183 | 184 | 在浏览器 url 输入：http://review.ci-example.com/，记得添加 hosts 解析。由于做了访问控制，会出现如下验证框，输入刚才创建的 admin 和密码 185 | 186 | ![](/Users/frank/Documents/marboo/media/CI/gerrit/00-gerrit-admin-login.png) 187 | 188 | 注册 admin 的邮箱，去邮箱验证，然后添加 git@plcloud.com 密钥 189 | 190 | ![](/Users/frank/Documents/marboo/media/CI/gerrit/01-gerrit-bind-email.png) 191 | ![](/Users/frank/Documents/marboo/media/CI/gerrit/02-gerrit-add-ssh-key.png) 192 | 193 | 使用 htpasswd 创建 frank 用户和密码 194 | 195 | ```bash 196 | htpasswd /etc/gerrit/etc/htpasswd.conf frank 197 | ``` 198 | 199 | 换个浏览器或清除下浏览器保存的 admin 的密码 访问 http://review.ci-example.com/ 输入 frank 和密码： 200 | 201 | 注册邮箱和添加 rainyday_hubei@163.com 的密钥 202 | 203 | ## 最后 204 | 205 | 如果想 Gitlab 上创建的项目使用 Gerrit 的 Code Review 功能，两个系统的用户必须统一，也就是说不管哪个用户使用 Gerrit，前提是这个用户在 Gitlab 和 Gerrit 上都已注册，邮箱一致、sshkey 一致。当然 Nginx 访问控制用户的密码那就随意了。至于 Gitlab 上创建的项目如何同步到 Gerrit 上、Gitlab 如何使用 Gerrit 的 Code Review 功能等等都在 Jenkins 安装完之后会一起整合在一起。请关注后面的文章。 -------------------------------------------------------------------------------- /CI/01-Gitlab+Gerrit+Jenkins/04-Jenkins安装配置.md: -------------------------------------------------------------------------------- 1 | # 04-Jenkins安装配置 2 | 3 | 4 | 5 | 7 | 8 | ## 下载包 9 | 10 | ```bash 11 | wget http://pkg.jenkins-ci.org/debian/binary/jenkins_1.544_all.deb 12 | ``` 13 | 14 | ## 安装 15 | 16 | 安装依赖 17 | 18 | ```bash 19 | apt-get install daemon 20 | ``` 21 | 22 | 安装 Jenkins 23 | 24 | ```bash 25 | dpkg -i jenkins_1.544_all.deb 26 | ``` 27 | 28 | jenkins 默认监听了 8080 端口，修改为 8083 29 | 30 | ```bash 31 | vim /etc/default/jenkins 32 | HTTP_PORT=8083 33 | ``` 34 | 35 | 重新 jenkins 服务 36 | 37 | ```bash 38 | /etc/init.d/jenkins restart 39 | ``` 40 | 41 | ## Nginx 42 | 43 | 配置 Nginx 端口转发，在文件末尾加入下面配置 44 | 45 | ```bash 46 | # vim /etc/nginx/sites-available/gitlab 47 | server { 48 | listen *:80; 49 | server_name jenkins.ci-example.com; 50 | 51 | location / { 52 | proxy_pass http://jenkins.ci-example.com:8083; 53 | } 54 | } 55 | ``` 56 | 57 | 重启 Nginx，就可以用 jenkins.thstack.com 访问 Jenkins 了 58 | 59 | ```bash 60 | /etc/init.d/nginx restart 61 | ``` 62 | 63 | ## 访问 64 | 65 | http://jenkins.ci-example.com/ 66 | 67 | 开启用户注册功能，点击 -> 系统管理 -> Configure Global Security -> 勾上启用安全，就可以看到下图 68 | 69 | 保存后，会自动跳转到登录页面，点击右上角注册按钮 70 | 71 | 输入管理员信息 72 | 73 | 为了安全，设置 Jenkins 不对普通用户开放登录权限，只有管理员可以设置、构建任务，普通用户可以查看任务状态 点击 系统管理 -> Configure Global Security -> 去掉开放用户注册勾 74 | 75 | 接下来要安装 Jenkins 的插件，来支持 Gerrit 点击 系统管理 -> 管理插件，会看到下图显示 76 | 77 | 如果上图种 可更新、可选插件、已安装 三个菜单点开为空白的话，需要获取更新下 Jenkins 的信息，之后就可以看到插件信息了 点击 系统管理 -> 管理插件 -> 高级 78 | 79 | 安装所需插件,由于国外网络连接很差,需要手动下载插件安装 80 | 81 | ```bash 82 | su - jenkins # jenkins安装后会创建jenkins用户,家目录在/var/lin/jenkins 83 | cd plugins/ 84 | wget http://mirror.xmission.com/jenkins/plugins/gerrit-trigger/2.10.1/gerrit-trigger.hpi 85 | wget http://mirror.xmission.com/jenkins/plugins/scm-api/0.2/scm-api.hpi 86 | wget http://mirror.xmission.com/jenkins/plugins/git/2.4.0/git.hpi 87 | wget http://mirror.xmission.com/jenkins/plugins/git-client/1.9.0/git-client.hpi 88 | service jenkins restart # 重启 jenkins 会自动安装下载的插件 89 | ``` 90 | 91 | 在系统中给 Jenkins 用户生成 ssh 密钥，Jenkins 用户在安装包的时候自动创建了，家目录在 /var/lib/jenkins 92 | 93 | ```bash 94 | su - jenkins 95 | ssh-keygen -C git@plcloud.com 96 | cat ~/.ssh/id_rsa.pub # 把公钥内容复制一下，后面需要添加到 Gerrit 中 97 | ``` 98 | 99 | 用 htpasswd 创建 jenkins 用户访问控制密码 100 | 101 | ```bash 102 | htpasswd /etc/gerrit/etc/htpasswd.conf jenkins 103 | ``` 104 | 105 | 继续换个浏览器或者清除浏览器记录，用 jenkins 用户访问 Gerrit 106 | 107 | http://jenkins.ci-example.com 108 | 109 | 设置 jenkins 用户邮箱，顺便去邮箱里确认 110 | 111 | 添加刚才对 git@plcloud.com 邮箱生成的 ssh 密钥 112 | -------------------------------------------------------------------------------- /CI/01-Gitlab+Gerrit+Jenkins/05-整合Gitlab,Gerrit,Jenkins.md: -------------------------------------------------------------------------------- 1 | # 05-整合Gitlab,Gerrit,Jenkins 2 | 3 | 4 | 5 | 7 | 8 | > [GerritTrigger Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger) 9 | > 10 | > [InstallingJenkinsPlugin](https://wiki.jenkins-ci.org/display/JENKINS/Plugins#Plugins-Installingthenewestversion) 11 | > 12 | > [OpenStack Testing](https://wiki.openstack.org/wiki/Testing) 13 | > 14 | > [Neutron Setting Up a Development Environment](http://docs.openstack.org/developer/neutron/devref/development.environment.html) 15 | > 16 | > [IBM test Guide](https://developer.ibm.com/opentech/2015/02/06/guide-testing-openstack/) -------------------------------------------------------------------------------- /CI/02-Jenkins_Job_Builder/01-Jenkins job builder.md: -------------------------------------------------------------------------------- 1 | # 01-Jenkins job builder 2 | 3 | 4 | 5 | 7 | 8 | 默认的 Jenkins jobs 使用图形界面创建, 然后存储为 xml 格式,阅读性差、不便于版本管理 9 | 10 | jenkins job builder 是一个 Python 工具包, 使用 yaml 格式配置, 然后调用 jenkins 接口写入到 jenkins中,更加方便阅读和进行版本管控 11 | 12 | ## 安装 13 | 14 | ```bash 15 | git clone https://github.com/openstack-infra/jenkins-job-builder.git 16 | cd jenkins-job-builder 17 | sudo python setup.py install 18 | ``` 19 | 20 | ## 配置 21 | 22 | - 配置 jenkins 连接 23 | 24 | ```bash 25 | vim /etc/jenkins_jobs/jenkins_jobs.ini 26 | [jenkins] 27 | user=USERNAME 28 | password=PASSWORD 29 | url=JENKINS_URL 30 | ``` 31 | 32 | another example config 33 | 34 | ```bash 35 | [job_builder] 36 | ignore_cache=True 37 | keep_descriptions=False 38 | include_path=.:scripts:~/git/ 39 | recursive=False 40 | allow_duplicates=False 41 | 42 | [jenkins] 43 | user=admin 44 | password=xxxxx 45 | url=https://jenkins.plcloud.com 46 | 47 | [hipchat] 48 | authtoken=dummy 49 | ``` 50 | 51 | ## 编写任务 52 | 53 | sample 54 | 55 | ```bash 56 | - job: 57 | name: eDeploy-UnitTests-YAML 58 | description: 'Do not edit this job through the web!' 59 | project-type: freestyle 60 | block-downstream: false 61 | scm: 62 | - git: 63 | skip-tag: false 64 | url: git@github.com:enovance/edeploy.git 65 | triggers: 66 | - pollscm: '@hourly' 67 | builders: 68 | - shell: | 69 | git clean -dxf 70 | sloccount --duplicates --wide --details . | fgrep -v .svn > sloccount.sc || : 71 | find . -name test\*.py|xargs nosetests --with-xunit --verbose || : 72 | find . -name \*.py|egrep -v '^./tests/'|xargs pyflakes > pyflakes.log || : 73 | rm -f pylint.log 74 | for f in `find . -name \*.py|egrep -v '^./tests/'`; do 75 | pylint --output-format=parseable --reports=y $f >> pylint.log 76 | done || : 77 | python /usr/local/lib/python2.7/dist-packages/clonedigger/clonedigger.py --cpd-output . || : 78 | publishers: 79 | - warnings: 80 | workspace-file-scanners: 81 | - file-pattern: pyflakes.log 82 | scanner: PyFlakes 83 | - junit: 84 | results: nosetests.xml 85 | - sloccount: 86 | pattern: sloccount.sc 87 | - violations: 88 | cpd: 89 | pattern: output.xml 90 | pylint: 91 | pattern: pylint.log 92 | - email: 93 | recipients: devops@mycompany.com 94 | ``` 95 | 96 | ## 测试 97 | 98 | 正式写入 jenkins 之前最好使用如下命令生成 xml 进行检查后再写入 99 | 100 | ```bash 101 | jenkins-jobs test my-job.yaml -o . 102 | ``` 103 | 104 | ## 写入 Jenkins 105 | 106 | 写入 jenkins, 可以不指定配置文件, 则使用默认路径配置文件 107 | 108 | ```bash 109 | jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini update my-job.yaml 110 | ``` 111 | 112 | ## 删除任务 113 | 114 | ```bash 115 | jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini delete my-job 116 | ``` 117 | 118 | > [jenkins job builder官网](http://docs.openstack.org/infra/jenkins-job-builder/) 119 | > 120 | > [官方范例](http://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/tree/tests) 121 | > 122 | > [PyYAML官网](http://pyyaml.org/wiki/PyYAMLDocumentation) 123 | > 124 | > [enovance样例](http://techs.enovance.com/6006/manage-jenkins-jobs-with-yaml) -------------------------------------------------------------------------------- /CI/03-fabric/01-autossh+haproxy打通网络.md: -------------------------------------------------------------------------------- 1 | # 01-autossh+haproxy打通网络 2 | 3 | 4 | 5 | 7 | 8 | ## ssh 反向隧道 9 | 10 | 原理图 11 | 12 | ![](http://ww3.sinaimg.cn/mw690/663a9daagw1eveljexnluj20qo0k0abr.jpg) 13 | 14 | 实现 15 | 16 | 内网主机 A 执行如下命令 17 | 18 | ```bash 19 | ssh -NCfR remout_port:localhost:local_port remount_user@remout_host 20 | ``` 21 | 22 | 命令解释: 23 | 24 | - -N 不执行远程命令 25 | - -C 压缩数据 26 | - -f 在认证完成后自动放到后台执行 27 | - -R 创建一个反向代理 28 | - 任何人访问remote_host:remote_port相当于访问localhost:localport 29 | 30 | 假设 local_port 为 提供 ssh 服务的 22 端口,则执行上述命令后,连接 `remote_host:remote_port` 即可连接上内网主机 ssh 31 | 32 | ## ssh 免密码 33 | 34 | 生成 key 执行`ssh-keygen`, 将生成的 `id_rsa.pub` 复制到远程主机, 并追加到 `~/.ssh/authorized_keys` 中 35 | 36 | ```bash 37 | cat id_rsa.pub >> ~/.ssh/authorized_keys 38 | # 或者 ssh-copy-id user1@123.123.123.123 39 | ``` 40 | 41 | ## autossh 42 | 43 | 作用为监控 ssh 执行状态, 断线自动重连 44 | 45 | 用法 46 | 47 | ```bash 48 | autossh -M 5678 -NCR 1234:localhost:2223 user1@123.123.123.123 -p2288 49 | ``` 50 | 51 | 要点解释 52 | 53 | - -M 5678为监控端口,负责通过该端口监控运行状态 54 | - 去掉了`-f`参数,因为 autossh 本身会在后台执行 55 | - autossh还有很多参数,用来设置重连间隔等等,更复杂的用法可另行查阅或参见底部参考资料 56 | 57 | P.S. 58 | 59 | ```bash 60 | 1.家里是ADSL的话，用DDNS，解决ip问题 61 | 62 | 2.外网有路由的可设下端口映射 63 | 64 | 3.虽然有密钥和密码保护，但还请小心使用 65 | ``` 66 | 67 | ## 设置为 upstart 服务 68 | 69 | 为防止重启后掉线,加入开机启动服务,此处使用 `upstart`,`SysV`、`Systemd`另行查阅 70 | 71 | ```bash 72 | # 讲以下内容写入 /etc/init/autossh_1.conf 73 | description "autossh" 74 | author "Franko " 75 | 76 | start on runlevel [2345] 77 | stop on runlevel [06] 78 | 79 | respawn 80 | respawn limit 2 5 81 | 82 | exec autossh -M 9030 -CNR 9080:localhost:8080 root@114.119.4.36 -p 2288 83 | ``` 84 | 85 | ## haproxy 反向代理,转发至内网机器 86 | 87 | 如果需要打通内网主机与外网的局域网主机连接,则还需要在远端进行反向代理,可使用`Nginx`或`HAProxy`等工具实现,此处使用`HAProxy` 88 | 89 | HAProxy 参考配置文件 90 | 91 | ```bash 92 | #--------------------------------------------------------------------- 93 | # Example configuration for a possible web application. See the 94 | # full configuration options online. 95 | # 96 | # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt 97 | # 98 | #--------------------------------------------------------------------- 99 | 100 | #--------------------------------------------------------------------- 101 | # Global settings 102 | #--------------------------------------------------------------------- 103 | global 104 | # to have these messages end up in /var/log/haproxy.log you will 105 | # need to: 106 | # 107 | # 1) configure syslog to accept network log events. This is done 108 | # by adding the '-r' option to the SYSLOGD_OPTIONS in 109 | # /etc/sysconfig/syslog 110 | # 111 | # 2) configure local2 events to go to the /var/log/haproxy.log 112 | # file. A line like the following can be added to 113 | # /etc/sysconfig/syslog 114 | # 115 | # local2.* /var/log/haproxy.log 116 | # 117 | log 127.0.0.1 local2 118 | 119 | chroot /var/lib/haproxy 120 | pidfile /var/run/haproxy.pid 121 | maxconn 4000 122 | user haproxy 123 | group haproxy 124 | daemon 125 | 126 | # turn on stats unix socket 127 | stats socket /var/lib/haproxy/stats 128 | 129 | #--------------------------------------------------------------------- 130 | # common defaults that all the 'listen' and 'backend' sections will 131 | # use if not designated in their block 132 | #--------------------------------------------------------------------- 133 | defaults 134 | mode http 135 | log global 136 | option httplog 137 | option dontlognull 138 | option http-server-close 139 | option forwardfor except 127.0.0.0/8 140 | option redispatch 141 | retries 3 142 | timeout http-request 10s 143 | timeout queue 1m 144 | timeout connect 10s 145 | timeout client 1m 146 | timeout server 1m 147 | timeout http-keep-alive 10s 148 | timeout check 10s 149 | maxconn 3000 150 | 151 | #--------------------------------------------------------------------- 152 | # main frontend which proxys to the backends 153 | #--------------------------------------------------------------------- 154 | frontend main *:5000 155 | acl url_static path_beg -i /static /images /javascript /stylesheets 156 | acl url_static path_end -i .jpg .gif .png .css .js 157 | 158 | use_backend static if url_static 159 | default_backend app 160 | 161 | #--------------------------------------------------------------------- 162 | # static backend for serving up images, stylesheets and such 163 | #--------------------------------------------------------------------- 164 | backend static 165 | balance roundrobin 166 | server static 127.0.0.1:4331 check 167 | 168 | #--------------------------------------------------------------------- 169 | # round robin balancing between the various backends 170 | #--------------------------------------------------------------------- 171 | backend app 172 | balance roundrobin 173 | server app1 127.0.0.1:5001 check 174 | server app2 127.0.0.1:5002 check 175 | server app3 127.0.0.1:5003 check 176 | server app4 127.0.0.1:5004 check 177 | 178 | listen git-proxy 179 | bind 172.20.0.12:9080 180 | mode http 181 | balance roundrobin 182 | server git-1 127.0.0.1:9080 183 | ``` 184 | 185 | 配置完成后,远端主机同一网段访问 `172.20.0.12:9080` 即相当于访问 local_host:local_port 186 | 187 | 注意配置文件最后几行的 `mode http` 表示访问的模式为 HTTP ,此处的本地端口为 `gitlab` 服务,访问即可实现 `git pull http://...` 操作 188 | 189 | 190 | > [1] [ssh反向连接及autossh](http://www.cnblogs.com/eshizhan/archive/2012/07/16/2592902.html) 191 | > 192 | > [2] [ssh反向代理小实验](http://blog.chinaunix.net/uid-29143273-id-4554257.html) -------------------------------------------------------------------------------- /CI/03-fabric/02-fabric远程部署.md: -------------------------------------------------------------------------------- 1 | # 02-fabric远程部署 2 | 3 | 4 | 5 | 7 | 8 | 9 | 样例代码 10 | 11 | ```python 12 | from fabric.api import * 13 | import time 14 | import re 15 | import sys 16 | import os 17 | import json 18 | 19 | env.passwords = { 20 | 'root@vip.plcloud.com:22': 'xxxxxxxxxx', 21 | 'root@192.168.215.101:22': '111111', 22 | 'root@58.67.194.90:2288': 'xxxxxxxxxx', 23 | 'root@10.0.0.14:22': 'xxxxxxxxxx', 24 | 'root@10.0.0.15:22': 'xxxxxxxxxx', 25 | 'root@10.0.0.18:22': 'xxxxxxxxxx', 26 | 'root@10.0.0.23:22': 'xxxxxxxxxx', 27 | } 28 | 29 | gitpwd = 'gitdeploy' 30 | 31 | @hosts('vip.plcloud.com') 32 | def plcloud_front_master(): 33 | with cd('/var/plcloud/plcloud-front'): 34 | run('git pull http://git:%s@127.0.0.1:9080/powerleader/plcloud-front.git' % gitpwd) 35 | run('python manage.py collectstatic -c --noinput') 36 | run('touch /etc/uwsgi/vassals/front_old.xml') 37 | 38 | 39 | @hosts('vip.plcloud.com') 40 | def plcloud_front_v2_0(): 41 | with cd('/var/plcloud/v2_0/plcloud-front'): 42 | run('git pull http://git:%s@127.0.0.1:9080/powerleader/plcloud-front.git v2.0' % gitpwd) 43 | run('python manage.py collectstatic -c --noinput') 44 | run('touch /etc/uwsgi/vassals/front_uwsgi_2.xml') 45 | 46 | 47 | @hosts('vip.plcloud.com') 48 | def plcloud_admin_master(): 49 | with cd('/var/plcloud/plcloud-admin'): 50 | run('git pull http://git:%s@127.0.0.1:9080/powerleader/plcloud-admin.git master' % gitpwd) 51 | run('python manage.py collectstatic -c --noinput') 52 | run('touch /etc/uwsgi/vassals/admin_uwsgi.xml') 53 | 54 | 55 | @hosts('vip.plcloud.com') 56 | def plcloud_vip_master(): 57 | with cd('/var/plcloud/plcloud-vip'): 58 | run('git pull http://git:%s@127.0.0.1:9080/powerleader/plcloud-vip.git master' % gitpwd) 59 | run('python manage.py collectstatic -c --noinput') 60 | run('touch /etc/uwsgi/vassals/vip_uwsgi.xml') 61 | 62 | 63 | @hosts('192.168.215.101') 64 | def horizon_master(): 65 | with cd('/opt/horizon'): 66 | run('git pull origin master') 67 | run('rm -rf static') 68 | run('python manage.py compress -f') 69 | run('python manage.py collectstatic -c --noinput') 70 | run('touch /etc/uwsgi/vassals/openstack.ini') 71 | 72 | def trigger_ci_master(): 73 | with lcd('/var/plcloud/trigger-ci'): 74 | local('git checkout -f master') 75 | local('git clean -f -d') 76 | local('git pull origin master') 77 | local('touch /etc/uwsgi/vassals/trigger-ci.ini') 78 | 79 | @hosts('192.168.215.101') 80 | def plcloud_operation_master(): 81 | with cd('/var/plcloud/plcloud-operation'): 82 | run('git pull origin master') 83 | #run('rm -rf static') 84 | #run('python manage.py collectstatic -c --noinput') 85 | run('touch /etc/uwsgi/vassals/operation.ini') 86 | 87 | def normalize(value): 88 | return re.sub('[-\s\.]+', '_', value.lower()) 89 | 90 | def test(): 91 | local('uname -a') 92 | 93 | def main(*argc): 94 | #repo = os.environ.get('repo') 95 | #branch = os.environ.get('branch') 96 | if len(argc) == 0: 97 | gitlab_ci() 98 | return 99 | repo = normalize(argc[0]) 100 | branch = normalize(os.environ.get('gitlabBranch')) 101 | print 'event: (%s, %s)' % (repo, branch) 102 | func_name = '%s_%s' % (repo, branch) 103 | task = globals().get(func_name, test) 104 | print 'task: %s' % task 105 | out = execute(task) 106 | print 'out: %s' % out 107 | 108 | def gitlab_ci(): 109 | context = json.loads(os.environ.get('CONTEXT')) 110 | repo = normalize(context['repository']['name']) 111 | branch = normalize(context['ref'].split('/')[-1]) 112 | print 'event: (%s, %s)' % (repo, branch) 113 | func_name = '%s_%s' % (repo, branch) 114 | task = globals().get(func_name, test) 115 | print 'task: %s' % task 116 | out = execute(task) 117 | print 'out: %s' % out 118 | 119 | def run_git_pull(url, branch='master', gitpwd=None, gituser='git', is_local=False): 120 | schema, path = url.split('//') 121 | fastprint('git pull %s//%s:%s@%s %s' % (schema, gituser, '***', path, branch), show_prefix=True, end='\r\n') 122 | with hide('running'): 123 | if is_local: 124 | local('git pull %s//%s:%s@%s %s' % (schema, gituser, gitpwd, path, branch)) 125 | else: 126 | run('git pull %s//%s:%s@%s %s' % (schema, gituser, gitpwd, path, branch)) 127 | 128 | @hosts('10.0.0.14', '10.0.0.15') 129 | @with_settings(gateway='root@58.67.194.90:2288') 130 | def horizon(): 131 | with cd('/var/plcloud/horizon'): 132 | run_git_pull('http://10.0.0.10:9080/powerleader/horizon.git', 'master', gitpwd) 133 | run('rm -rf static') 134 | run('python manage.py compress -f') 135 | run('python manage.py collectstatic -c --noinput') 136 | run('touch /etc/uwsgi/vassals/horizon.ini') 137 | 138 | @hosts('10.0.0.18', '10.0.0.23') 139 | @with_settings(gateway='root@58.67.194.90:2288') 140 | def ceilometer(): 141 | with cd('/var/plcloud/ceilometer'): 142 | run_git_pull('http://10.0.0.10:9080/icehouse/ceilometer.git', 'master', gitpwd) 143 | run('killall -e ceilometer-agent-notification') 144 | run('/etc/init.d/openstack-ceilometer-notification start') 145 | time.sleep(2) 146 | run('tail -n 10 /var/log/ceilometer/agent-notification.log') 147 | 148 | @hosts('10.0.0.14', '10.0.0.15') 149 | @with_settings(gateway='root@58.67.194.90:2288') 150 | def keystone(): 151 | with cd('/var/plcloud/keystone'): 152 | run_git_pull('http://10.0.0.10:9080/icehouse/keystone.git', 'master', gitpwd) 153 | run('service openstack-keystone restart') 154 | run('tail -n 10 /var/log/keystone/keystone.log') 155 | 156 | @hosts('10.0.0.14', '10.0.0.15', '10.0.0.18', '10.0.0.23') 157 | @with_settings(gateway='root@58.67.194.90:2288') 158 | def keystoneclient(): 159 | with cd('/var/plcloud/python-keystoneclient'): 160 | run_git_pull('http://10.0.0.10:9080/icehouse/python-keystoneclient.git', 'master', gitpwd) 161 | 162 | if __name__ == '__main__': 163 | main(*sys.argv[1:]) 164 | 165 | ``` 166 | 167 | 更多用法参考以下文档 168 | 169 | > [1] [fabric官方文档](http://docs.fabfile.org/en/latest/tutorial.html) 170 | > 171 | > [2] [fabric入门参考](http://wklken.me/posts/2013/03/25/python-tool-fabric.html) -------------------------------------------------------------------------------- /Ceilometer/01-Ceilometer简介.md: -------------------------------------------------------------------------------- 1 | [toc] 2 | 3 | # ceilometer概略 4 | 5 | ## ceilometer主要概念 6 | - **resource:**可监控资源,云主机、硬盘、镜像、路由器、交换机... 7 | - **meter:**监控项,cpu_util,memory.usage,network.incoming.bytes.rate... 8 | - **sample:**样本数据点 9 | - **statistics:**sample处理过的数据,包含max,min,avg,sum,count等 10 | - **alarm:**告警,定义监控对象、监控周期、触发条件等参数 11 | - **evaluator:**评估是否满足告警条件,满足条件发消息到notifier 12 | - **notifier:**通知器,定义告警条件满足后的动作,邮件、短信、日志 13 | - **collector:**数据收集器,将收集到的数据存入后端数据库 14 | - **pushing agents:** 15 | - **polling agents:** 16 | - [完整术语表参见](http://docs.openstack.org/developer/ceilometer/glossary.html#term-metering) 17 | 18 | ## 获取测量值数据方式 19 | - 第三方的数据发送者把数据以通知消息(Notification Message)的方式发送到消息总线(Notification Bus)上,Ceilometer 中的 Notification Agent 会获取这些通知事件,从中提取测量数据 20 | - Ceilometer 中的 Polling Agent 会根据配置,定期轮询,主动通过各种 API 或者其他通信协议去远端或者本地的不同服务实体中获取所需要的测量数据 21 | - 用户通过调用 Ceilometer RESTful API 直接把利用其他方式获取的任意测量数据送达给 Ceilometer 22 | - [**ceilometer路由方式**](https://pecan.readthedocs.org/en/latest/routing.html) 23 | 24 | 25 | ## ceilometer服务 26 | 由 setup.cfg [entry_points]配置块下配置及 ceilometer/cli.py 下对应函数名,可见有以下几个主要服务 27 | 28 | 29 | [参考RDO文档](https://www.rdoproject.org/CeilometerQuickStart) 30 | 31 | - **compute agent**: polls the local libvirt daemon to acquire performance data for the local instances, messages and emits these data as AMQP notifications 32 | - **central agent**: polls the public RESTful APIs of other openstack services such as nova and glance, in order to keep tabs on resource existence 33 | - **collector service**: consumes AMQP notifications from the agents and other openstack services, then dispatch these data to the metering store 34 | - **API service**: presents aggregated metering data to consumers (such as billing engines, analytics tools etc.) 35 | - **alarm-evaluator service**: determines when alarms fire due to the associated statistic trend crossing a threshold over a sliding time window 36 | - **alarm-notifier service**: initiates alarm actions, for example calling out to a webhook with a description of the alarm state transition 37 | 38 | ```python 39 | console_scripts = 40 | # 提供 RestfulAPI 调用接口 41 | ceilometer-api = ceilometer.cli:api 42 | 43 | # 44 | ceilometer-agent-central = ceilometer.cli:agent_central 45 | 46 | # 运行在计算节点上,查询实例的性能信息,相关信息,通过 AMQP 消息队列发送数据给 collector 47 | ceilometer-agent-compute = ceilometer.cli:agent_compute 48 | 49 | # 通过 AMQP 消息队列货期其他 OpenStack 服务的事件通知 50 | ceilometer-agent-notification = ceilometer.cli:agent_notification 51 | 52 | # 53 | ceilometer-send-sample = ceilometer.cli:send_sample 54 | 55 | # 56 | ceilometer-dbsync = ceilometer.cli:storage_dbsync 57 | 58 | # 59 | ceilometer-expirer = ceilometer.cli:storage_expirer 60 | 61 | # 62 | ceilometer-collector = ceilometer.cli:collector_service 63 | 64 | # 告警触发评估 65 | ceilometer-alarm-evaluator = ceilometer.cli:alarm_evaluator 66 | 67 | # 告警通知器,执行触发后的动作 sms,log,email... 68 | ceilometer-alarm-notifier = ceilometer.cli:alarm_notifier 69 | ``` 70 | 71 | 72 | ## 系统架构 73 | 74 | **总体架构** 75 | 76 | ![ceilometer架构](http://ww2.sinaimg.cn/mw690/663a9daagw1et65y1o6usj20o40hnmyr.jpg) 77 | 78 | **0.1版本概略图** 79 | 80 | ![0.1版本概略图](http://ww1.sinaimg.cn/large/663a9daagw1etuo5bi3y6j20kc0giaas.jpg) 81 | 82 | **收集数据** 83 | 84 | ![收集数据](http://ww1.sinaimg.cn/large/663a9daagw1etuo5do7a1j20w80gp75d.jpg) 85 | 86 | **pipeline 处理流程** 87 | 88 | ![pipeline 处理流程](http://ww2.sinaimg.cn/large/663a9daagw1etup94p7lmj20tn0dsmxz.jpg) 89 | 90 | **转换数据** 91 | 92 | ![转换数据](http://ww1.sinaimg.cn/large/663a9daagw1etup95yoaxj20l909u755.jpg) 93 | 94 | **发布数据** 95 | 96 | ![发布数据](http://ww4.sinaimg.cn/large/663a9daagw1etup97ftw4j20hk08pwey.jpg) 97 | 98 | - 如果发送到 collector,则通过数据抽象层存储至对应数据库(mongo/mysql/hbase...) 99 | 100 | **访问数据** 101 | 102 | ![访问数据](http://ww4.sinaimg.cn/large/663a9daagw1etup93lj34j20tv0avwfe.jpg) 103 | 104 | - 为后续评估数据,触发事件提供待处理数据 105 | 106 | >[1] [RDO Ceilometer介绍](https://www.rdoproject.org/CeilometerQuickStart) 107 | > 108 | >[2] [Ceilometer模块学习](http://www.aboutyun.com/forum-196-1.html) 109 | > 110 | >[3] [Ceilometer 高层架构图](http://docs.openstack.org/developer/ceilometer/architecture.html#storing-the-data) 111 | > 112 | >[4] [Ceilometer0.1规划图](http://ceilometer.readthedocs.org/en/latest/architecture.html#high-level-description) 113 | -------------------------------------------------------------------------------- /Horizon/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cooladam201212/OpenStack_Learning/461baf4628f4da6f5b4d6f42a11d3bfc5ddd123d/Horizon/__init__.py -------------------------------------------------------------------------------- /Keystone/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cooladam201212/OpenStack_Learning/461baf4628f4da6f5b4d6f42a11d3bfc5ddd123d/Keystone/__init__.py -------------------------------------------------------------------------------- /Neutron/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cooladam201212/OpenStack_Learning/461baf4628f4da6f5b4d6f42a11d3bfc5ddd123d/Neutron/__init__.py -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # OpenStack学习资源汇总 2 | 3 | 本文档为学习OpenStack过程中搜集的一些学习资料链接汇总，学习笔记请放至对应文件夹 4 | 5 | ## **目录** 6 | 7 | - [OpenStack基础](#OpenStack基础) 8 | 9 | 10 | ## OpenStack基础 11 | 12 | - 消息队列 13 | - [zmq中文文档(官网翻译,版本不是最新)](https://github.com/anjuke/zguide-cn 14 | ) 15 | - [zmq官网文档](http://zguide.zeromq.org/page:all) 16 | - [zeromq深度探索系列](http://blog.csdn.net/shagoo/article/details/8964009) 17 | - [pyzmq手册](https://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/basics.html) 18 | - [zmq用来怎么玩(zhihu)](http://www.zhihu.com/question/28648575) 19 | - zeromq安装 20 | - [官网安装指南](http://zeromq.org/docs:source-git) 21 | - [依赖包libsodium](https://wiki.tox.im/Installing#Libsodium) 22 | - oslo 23 | - [rabbitmq-oslo-messaging](http://lingxiankong.github.io/blog/2015/04/01/rabbitmq-oslo-messaging/) 24 | - WSGI & Restful 25 | - [pecan](https://pecan.readthedocs.org/en/latest/) 26 | - [Routes](http://routes.readthedocs.org/en/latest/) 27 | - [eventlet](http://eventlet.net/doc/examples.html#wsgi-server) 28 | - [Paste](http://pythonpaste.org/do-it-yourself-framework.html) 29 | - sqlalchemy 30 | - [官网文档](http://docs.sqlalchemy.org/en/rel_1_0/) 31 | - [sqlalchemy笔记](http://wangye.org/blog/archives/718/) 32 | - [sqlalchemy指南](http://gashero.yeax.com/?p=6%23id18) 33 | - redis 34 | - [redis学习手册](http://www.cnblogs.com/stephen-liu74/archive/2012/04/16/2370212.html) 35 | - [redis命令手册](http://www.redis.cn/commands.html#generic) 36 | - [NoSqlFan](http://blog.nosqlfan.com/html/3537.html) 37 | - 测试框架 38 | - [单元测试](http://www.ibm.com/developerworks/cn/linux/l-pyunit/index.html) 39 | - [testtools](https://pypi.python.org/pypi/testtools) 40 | - [OpenStack自动测试框架Tempest](http://www.ibm.com/developerworks/cn/cloud/library/1403_liuyu_openstacktempest/index.html) 41 | - [stevedore](http://docs.openstack.org/developer/stevedore/) 42 | - 自动化运维 43 | - [ansible](http://www.178linux.com/doc/ansible/) 44 | 45 | ## 博客 46 | 47 | - OpenStack博客资源 48 | - [sammyliu](http://www.cnblogs.com/sammyliu) 49 | - [Ustack官方博客](https://www.ustack.com/blog/ ) 50 | - [OpenStack Hack养成指南](https://www.ustack.com/blog/openstack_hacker/)(**from Ustack**) 51 | - [bingotree](http://bingotree.cn/) 52 | - [孔令贤](http://lingxiankong.github.io/) 53 | - [虚拟机创建的50个步骤和100个知识点](http://www.cnblogs.com/popsuper1982/p/3927390.html) 54 | - [中科院-刘冬](http://blog.csdn.net/gaoxingnengjisuan) 55 | - [Neutron、Ceilometer运行机制](http://panpei.net.cn/tags/OpenStack/) 56 | - [臭蛋](http://www.choudan.net/tags.html?#anchorOpenStack) 57 | - [菩提没有树](http://1.chaoxu.sinaapp.com/archives/category/%E4%BA%91%E8%AE%A1%E7%AE%97/openstack) 58 | - Python博客资源 59 | - [吴坤亮](http://wklken.me/archives.html)(网站:[甜品礼物](http://www.itianpin.com/)) 60 | - [小明明](http://www.dongwm.com/blog/archives/) 61 | - [the51fire](http://www.the5fire.com/) 62 | - [@Coder_猪之哀伤](http://www.zlovezl.cn/) 63 | - [BeginMan](beginman.github.io) 64 | 65 | - 趣味Python 66 | - [CodeWars](http://www.codewars.com/) 67 | - [Leetcode](https://leetcode.com/) 68 | - [Python片段](http://code.activestate.com/recipes/langs/#python) 69 | - [PythonModuleOfTheWeek](http://pymotw.com/2/) 70 | 71 | ## 常用工具 72 | 73 | ### Markdown 74 | ### Git 75 | - [Git专题](http://www.jianshu.com/collection/b0dbaed3c1d3) 76 | 77 | ## 聚合&阅读 78 | 79 | - [FreeProgrammingBooks](https://github.com/vhf/free-programming-books/blob/master/free-programming-books-zh.md#markdown) 80 | - [码农周刊分类](https://github.com/nemoTyrant/manong) 81 | - ... 82 | --------------------------------------------------------------------------------