├── LICENSE.md ├── _config.yml ├── articles └── ci_pipeline_and_custom_tools_of_android_projects │ ├── README.md │ └── images │ ├── build_merge_result_and_commit.png │ └── ci_pipeline_android.png └── specs └── rxjava-2-migration.md /LICENSE.md: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright 2017 Juno Inc. 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-cayman -------------------------------------------------------------------------------- /articles/ci_pipeline_and_custom_tools_of_android_projects/README.md: -------------------------------------------------------------------------------- 1 | # CI Pipeline and Custom Tools of Android projects at Juno 2 | 3 | Article describes almost all components of CI Pipeline we use for our Rider Android app at Juno. Driver team has similar, but slightly different configuration. 4 | 5 | Disclaimer: 6 | 7 | >Article mostly based on personal opinions of [Artem Zinnatullin][artem_zin], some Juno engineers may not agree with some of them, and it's fine! 8 | 9 | #### Table of Contents 10 | 11 | - [Overview](#overview) 12 | - [General Pipeline](#general-pipeline) 13 | - [Release Candidate and Release Jobs](#release-candidate-and-release-jobs) 14 | - [What Actually Happens on CI](#what-actually-happens-on-ci) 15 | - [Docker](#docker) 16 | - [Docker TL;TR](#docker-tltr) 17 | - [Why Docker](#why-docker) 18 | - [Docker issues](#docker-issues) 19 | - [Build Scripts Under Version Control](#build-scripts-under-version-control) 20 | - [Why store build scripts under version control](#why-store-build-scripts-under-version-control) 21 | - [How do we organize Build Scripts](#how-we-organize-build-scripts) 22 | - [Why Bash and not Kotlin Script, etc?](#why-bash-and-not-kotlin-script-etc) 23 | - [Back to "What actually happens on CI"](#back-to-what-actually-happens-on-ci) 24 | - [Swarmer — our tool to create and start Android Emulators in Parallel](#swarmer) 25 | - [Composer — our test runner that runs Instrumentation Tests in Parallel and generates HTML Report](#composer) 26 | - [Jenkinsfile](#jenkinsfile) 27 | - [Bonus Material](#bonus-material) 28 | - [Why Jenkins](#why-jenkins) 29 | - [What alternatives did we look into](#what-alternatives-did-we-look-into) 30 | - [TeamCity](#teamcity) 31 | - [Concourse](#concourse) 32 | - [Why Bitbucket Server](#why-bitbucket-server) 33 | - [Mainframer — How we build apk with source code changes in under 10 seconds during local development](#mainframer) 34 | 35 | ## Overview 36 | 37 | ![Overall Picture](images/ci_pipeline_android.png) 38 | 39 | ### General Pipeline 40 | 41 | General Pipeline builds: 42 | 43 | * Every branch in the repo. 44 | * `pr/*/merge` refs provided by [Bitbucket Server][bitbucket server] (GitHub also have similar feature JFYI). 45 | 46 | `pr/*/merge` ref updates each time target branch or PR itself changes. 47 | 48 | This gives us **automatic rebuilds** for Pull Request if branch it is targeting was changed — shortest CI feedback loop possible for a given PR. 49 | 50 | Two Build Statuses 51 | 52 | Building `pr/*/merge` refs is **essential if you want to have stable dev/master branch**. There are cases (happened 1-2 times a week even in our small team) when merge of a PR will break its target branch because there was logical conflict. 53 | 54 | Real life example: you change some class name in PR1, PR2 used old class name, for Git there might be no conflict at all, however result of merge of both PRs will not even be compilable. 55 | 56 | ### Release Candidate and Release Jobs 57 | 58 | These are separate jobs from the General Pipeline. 59 | 60 | The reason we're keeping them separate is because we didn't find a good way yet to have different "Discard old builds" settings for different "branches" in a Jenkins Pipeline. In our case we generally want to keep builds only for about a week, but release and release candidate builds should be kept forever (just in case, we also have separate git repo for released apks which is very useful for QAs and Developers). 61 | 62 | We won't be discussing these jobs in the article, but for the sake of details — they use same `build.sh` infrastructure as General Pipeline (see below) which is nice. 63 | 64 | ## What actually happens on CI 65 | 66 | First of all let's define few major building blocks that allow us to have consistent and reproducible CI builds and then actually combine them into what we really do. 67 | 68 | ### Docker 69 | 70 | We've wrapped all our CI jobs with [Docker][docker] in August 2016 and since then it helped us a lot. There are issues though, but it's worth it. 71 | 72 | #### Docker TL;TR 73 | 74 | If we abstract away many many details, Docker gives you ability to build Image(s) (think of it as about VM image) and then run instances of Image(s) — Container(s). 75 | 76 | On top of that Docker has DSL to describe how you want to build the Image and tooling to work with them at scale, i.e — have own registry of images, orchestrate Containers and so on and so on. 77 | 78 | #### Why Docker 79 | 80 | >If you want reproducible builds you need reproducible environment. 81 | 82 | Unfortunately, Android SDK and its components were not designed with user-friendly versioning in mind. Many Android SDK components simply don't have a version: `tools`, `platform-tools`, `android-` apis, `extra-android-m2repository` and others, so whenever you update SDK — you get latest versions which may or may not break your build. 83 | 84 | Oh, and if you have a build running and another build on the same machine decided to update Android SDK shared between them — chances are first build will fail because Android SDK may remove files of a component it is updating and then unpack new ones so there is a time window when first build may try to access non-existing Android SDK files. 85 | 86 | >New CI machines should be easy to add, old machines should be easy to maintain. 87 | 88 | Android SDK update process is not easy and what's even worse is that when you have lots of CI machines it might get out of hand real quick. 89 | 90 | With Docker we only need, hm, um, Docker on any given machine! And Java to run Jenkins ~~slave~~ node `jar`. No more nightly Android SDK Update Jobs or searching for those `lib32stdc++6` and `lib32z1` that differs between Linux distros! 91 | 92 | Also, Docker is very lightweight in terms of resources consumption in compare to Virtual Machines. 93 | 94 | #### Docker issues 95 | 96 | However, it'll be unfair to say that Docker did not create any challenges for us: 97 | 98 | - Docker image for our project weights about 5 GB (used to be 8). It takes minutes to download and consumes space on disk. 99 | - If your project uses lots of Gradle/Maven dependencies you might want to cache them in the Docker Image to save few minutes for each build. Then you'll need a strategy on updating Image because of dependencies cache miss. 100 | - In certain cases Docker Container may not stop properly and you may see zombie Containers (very rare, but still possible). 101 | - In certain cases Docker Image may not build properly and you may see untagged images that weight gigabytes (pretty rare, definitely possible). 102 | - "No space left on device" will give you few bad evenings. 103 | - Passing hardware devices and GPU + XServer access is complicated, however it worth all 100% for UI tests. 104 | - Running [Docker in Docker][docker in docker 2015] is [still painful in 2017][docker in docker 2017] and you might want to avoid that if possible. 105 | 106 | We've solved most of these problems and Dockerization of our build probably deserves whole separate article (ask us in comments/twitter if you want to read about it). 107 | 108 | ### Build Scripts Under Version Control 109 | 110 | Even before Dockerization, we had all our build processes described as [Bash][bash] scripts and stored under version control. 111 | 112 | #### Why store build scripts under version control 113 | 114 | - Updates to Build Scripts **can and should be reviewed** by teammates as regular PRs. 115 | - Updates to Build Scripts **can and should be tested** in isolation by CI (meta!). You don't want to break all builds, right? 116 | - We can migrate to other CI Server without much troubles since CI Server simply invokes Build Scripts (and we did twice). 117 | - We can change main build tool (Gradle) and involve other tools into build process without updating CI configs. 118 | - This allowed us Dockerize build without changes on CI Server side! 119 | - Everybody can reproduce CI build locally by running script that CI runs. 120 | - Initially we had 6 different build CI Jobs, obviously 99% of build process was the same, storing it in the VCS allowed us to avoid copy-pasting. 121 | 122 | #### How we organize build scripts 123 | 124 | Build scripts are code, we apply same practices by splitting functionality into files and folder as we do for regular code: 125 | 126 | * `ci/build.sh` — entrypoint for our build, it invokes other scripts to prepare Docker image and then starts Docker Container. 127 | * `ci/internal/build.sh` — actual build script that runs inside Docker Container, invokes `./gradlew` and so on. 128 | * `ci/internal/run_ui_tests.sh` — build script that runs our UI Tests inside Docker Container. 129 | * `ci/internal/*.sh` — other scripts related to the build, usually used by `ci/internal/build.sh`. 130 | 131 | What also important is that Build Scripts are organized in a way similar to **feature flags** in the app: by setting values of different environment variables you can change behavior of the build. So we can use same `build.sh` entrypoint for different types of builds. 132 | 133 | In general: we try to shift execution into Docker Container as soon as possible because there we have more control over environment and can have more tools to work with that we've installed into the Docker Image. 134 | 135 | #### Why Bash and not Kotlin Script, etc? 136 | 137 | It's fair to say that [Bash][bash] is far, [very far][bash tutorial] from a perfect scripting language. It has lots of caveats and strangies. 138 | 139 | However, there are several reasons why we chose it and continued to maintain CI scripts in Bash: 140 | 141 | - Availability: We can't launch Docker container directly because we have some pre-setup, so we need a language that runs on most Linux distros out of the box. 142 | - Compatibility: it's pretty easy to write Bash scripts that will work on even relatively old versions of Bash interpreter. 143 | - Usability: Mostly our build scripts check environment variables, concatenate strings and run other processes. This is what Bash was designed for! 144 | - Complexity: If we need to do more complex stuff, like [creating and running Android Emulators in parallel][swarmer], we write it in Kotlin (see [Swarmer][swarmer]) and then invoke it from Bash. 145 | 146 | We've tried to migrate some of the scripts to [Kotlin Script][kotlin script], but immediately faced issues: 147 | 148 | - Java APIs for launching external processes ~~suck~~ are not very great, even with Kotlin it's painful. 149 | - Btw we have a library for that: [Commander][commander] 150 | - To run Kotlin Script you need JVM and Kotlin Compiler on a machine 151 | - Docker may help here, but what if you need a script to start Docker Container in the first place? 152 | - Kotlin Script still [does not have official solution for dependency management][kotlin script depmngmt], even in small scripts you might want to use some third-party library and it's hard to do. 153 | - Bash tends to be faster than Kotlin Script if script is relatively small and not complex: 154 | - To run `.kts` you need to compile it first, Kotlin Compiler is pretty heavy so it may take a few seconds to just start it. 155 | - To *actually* run `.kts` you need to launch JVM, which is heavier than Bash interpreter. 156 | - However if you're doing complex stuff and script runs for a long time, JVM will most likely beat Bash. 157 | - However however if you're planning to do complex stuff in a script, you might probably want to extract it in a separate tool and invoke from Bash. 158 | 159 | ## Back to "What actually happens on CI" 160 | 161 | So now that we know about Docker and Build Scripts we can describe how we build our project: 162 | 163 | 1. CI Pipeline detects change in Source Code Repository 164 | 1. CI notifies Source Code Repository about started build 165 | 1. CI runs `bash ci/build.sh` 166 | 1. `ci/build.sh` prepares Docker Image 167 | - Either it is already on machine 168 | - Or it can be pulled from the Docker registry 169 | - Or it should be built and pushed to the registry 170 | 1. `ci/build.sh` starts Docker Container with workspace mounted as a volume 171 | - If it's "Build" stage of Pipeline: 172 | 1. Docker Container runs `bash ci/internal/build.sh` as a target command 173 | 1. `ci/internal/build.sh` constructs `./gradlew` invocation with required parameters depending on build feature flags 174 | 1. Gradle builds the app, runs Unit tests (we have lots of them), runs Lint and produces apks 175 | 1. CI stashes resulting `rider-android.apk` and `rider-androidTest.apk` apks so they could be used by "UI Tests" stage of Pipeline 176 | - If it's "UI Tests" stage of Pipeline: 177 | 1. CI unstashes apks built by "Build" stage of Pipeline 178 | 1. 1. Docker Container also mounts GPU device and XServer and runs `bash ci/internal/run_ui_tests.sh` as a target command 179 | 1. `ci/internal/run_ui_tests.sh` invokes [Swarmer][swarmer] to start Android Emulators in parallel 180 | 1. `ci/internal/run_ui_tests.sh` invokes [Composer][composer] that runs Instrumentation Tests in parallel 181 | 1. `ci/internal/collect_artifacts.sh` collects artifacts from different folders to `artifacts` folder 182 | 1. CI saves content of `artifacts` folder 183 | 1. CI notifies Source Code Repository about finished build 184 | 185 | ### Swarmer 186 | 187 | [Swarmer][swarmer] is our open source command line tool that creates and starts Android Emulators in parallel. 188 | 189 | It is written in Kotlin with RxJava and uses new `avdmanager` tool from SDK, contributions are welcome! 190 | 191 | How we use it: 192 | 193 | - Config of an emulator is stored under version control, so we apply regular code review process to it. 194 | - Number of emulators to start is provided to `run_ui_tests.sh` as an Environment Variable through Jenkins Node configuration because different machines can run different amount of emulators at a time 195 | - We run Android Emulators in Docker Container. For now all of them launched in the same Container, but we're thinking about launching each one in a separate Container to reduce some problems introduced by Android SDK 196 | 197 | [Swarmer][swarmer] is CI independent, so you can use it on Travis, TeamCity, Concourse, CircleCI and other CI systems. 198 | 199 | ### Composer 200 | 201 | [Composer][composer] is our open source command line tool to run Instrumentation Tests in parallel. 202 | 203 | And it's our replacement for [Spoon from Square][spoon] (thank you guys for it ❤️, but we overgrew it). 204 | 205 | - It installs target and test apks on available emulators/devices in parallel 206 | - It runs Instrumentation Tests in parallel using static sharding (and we're thinking about [dynamic sharding][composer dynamic sharding]) 207 | - It does most of the things reactively — ie pull files and collect logs asap, thanks to RxJava 208 | - It saves logcat and instrumentation output of each device 209 | - It saves logcat and instrumentation output of each test 210 | - It pulls screenshots (compatible with [Spoon][spoon] protocol) 211 | - It concisely logs execution process to stout 212 | - It generates JUnit4 report so CI could analyze it 213 | - It generates HTML Report (thanks to Juno Frontend Team) so we could view the results 214 | - Its HTML Report was developed with performance and usabilty in mind (still work in progress though) 215 | 216 | ### Jenkinsfile 217 | 218 | [Jenkinsfile][jenkinsfile] is a [Groovy 😿][groovy] based DSL for describing Pipelines in [Jenkins][jenkins]. 219 | 220 | We've finally migrated to Jenkins Pipeline in the beginning of 2017 and mostly it worked great and better than 6 separate Jobs we had before. Especially with [`stash & unstash`][jenkinsfile stash unstash] feature that allowed us eliminate overhead we had with separate Jobs for UI tests that basically had to build apks from scratch each time and partially repeat what regular build Jobs did. 221 | 222 | ```groovy 223 | pipeline { 224 | 225 | // Since we don't use any Bitbucket plugin on Jenkins (we've tried, they're all you know, uh…) 226 | // We need to manually notify Bitbucket about build status. 227 | // Due to specifics on how Jenkins evaluates the Pipeline we decided to use `master` node for initial notification 228 | // So it could do it even if all worker nodes are busy and we can see pending build status on Bitbucket asap. 229 | agent { 230 | label 'master' 231 | } 232 | 233 | stages { 234 | stage('Notify Bitbucket') { 235 | steps { 236 | ansiColor('xterm') { 237 | sh 'bash ci/internal/notify_bitbucket.sh INPROGRESS' 238 | } 239 | } 240 | } 241 | 242 | stage('Build') { 243 | agent { 244 | label 'docker && no-gpu' 245 | } 246 | 247 | environment { 248 | SIGN_RELEASE_AS_DEBUG = 'true' 249 | DELETE_BUILD_OUTPUT = 'true' 250 | } 251 | 252 | steps { 253 | ansiColor('xterm') { 254 | sh 'bash ci/build.sh' 255 | } 256 | stash includes: 'artifacts/', name: 'build' 257 | } 258 | 259 | post { 260 | always { 261 | archiveArtifacts artifacts: 'artifacts/' 262 | 263 | publishHTML(target: [ 264 | reportName: "Lint", 265 | reportDir: "app/build/reports", 266 | reportFiles: "lint-results.html", 267 | alwaysLinkToLastBuild: false, 268 | keepAll: true, 269 | allowMissing: false 270 | ]) 271 | 272 | junit '**/test-results/**/*.xml' 273 | 274 | // Too many small files consume inodes on filesystem. 275 | sh 'bash ci/internal/delete_build_output.sh' 276 | } 277 | } 278 | } 279 | 280 | stage('UI Tests') { 281 | agent { 282 | label 'docker && gpu' 283 | } 284 | 285 | environment { 286 | RUN_UI_TESTS_ONLY = 'true' 287 | } 288 | 289 | steps { 290 | unstash 'build' 291 | ansiColor('xterm') { 292 | sh 'bash ci/build.sh' 293 | } 294 | } 295 | 296 | post { 297 | always { 298 | archiveArtifacts artifacts: 'artifacts/' 299 | 300 | junit 'artifacts/**/junit4-reports/*.xml' 301 | 302 | publishHTML(target: [ 303 | reportName: "Composer", 304 | reportDir: "artifacts/composer-output", // Set `composer-output` as root dir to access logs/screenshots/etc from HTML report. 305 | reportFiles: "html-report/suites/0.html", 306 | alwaysLinkToLastBuild: false, 307 | keepAll: true, 308 | allowMissing: false 309 | ]) 310 | } 311 | } 312 | } 313 | } 314 | 315 | post { 316 | failure { 317 | ansiColor('xterm') { 318 | sh 'bash ci/internal/notify_bitbucket.sh FAILED' 319 | } 320 | } 321 | 322 | success { 323 | ansiColor('xterm') { 324 | sh 'bash ci/internal/notify_bitbucket.sh SUCCESSFUL' 325 | } 326 | } 327 | } 328 | } 329 | ``` 330 | 331 | ## Bonus Material 332 | 333 | ### Why Jenkins 334 | 335 | So… [Jenkins][jenkins] is not perfect by any means, it has weird bugs, we're constantly hitting different limits of its design and we've tryed alternatives several times. 336 | 337 | For instance just recently it corrupted configuration files of many projects in our cluster [JENKINS-44809][jenkins 44809] which resulted in downtime of CI for few hours. 338 | 339 | Also, Jenkins' integration with [Bitbucket Server][bitbucket server] is really bad. As you might notice from our [Jenkinsfile](#jenkinsfile), we're actually not using any Jenkins Bitbucket plugin because none of them were able to fit our requirements on building `pr/*/merge` refs, notifying Bitbucket about the build correctly and having good integration with Jenkins Pipeline. We've tried, cried, patched them and threw them away. 340 | 341 | Another problem with Jenkins is its Plugins Ecosystem. Amazing how plugins can be good and bad at the same time. We used to have lots of them installed and once a month we had some compatibility issue between them and Jenkins API changes. Since then we've moved almost everything into our Build Scripts and custom tools so our Jenkins instance is on Plugin Diet for a long time now, feels good. 342 | 343 | Pipelines themselves are pretty stable now, however [Blue Ocean][jenkins blueocean] UI is still not very reliable, slow and lacks lots of features in compare to old one. Gladly you can use both old and new UIs. 344 | 345 | **But**, Jenkins is free (there is [Enterprise version][jenkins enterprise] though) and mostly solves our tasks. 346 | 347 | Ideally (in Artem's dreams) company can own one autoscaling CI cluster with Docker on each machine, some Macs for iOS guys and some machines with GPU for our UI Tests and that should be fine for Backend, Frontend, Data Science and Mobile projects of the company. With autoscaling this should be very efficient in both cost and performance. 348 | 349 | Fortunately or unfortunately each department has own CI setups and we even have different Jenkins clusters for Android and iOS projects (we could build Android apps on Macs, just saying 😬) so we have isolation which helps in case of CI Server issues (which happens with Jenkins from time to time) but at the same time duplication of work and maintenance, and probably ineficient use of computing resources. 350 | 351 | #### What alternatives did we look into 352 | 353 | ##### [TeamCity][teamcity] 354 | 355 | Pros: 356 | 357 | - It's mature and stable product made by company we trust 358 | - It has powerful [Pipeline DSL][teamcity pipeline] as well 359 | - Pipeline DSL built on top of Kotlin rather than Groovy 360 | - Supports LDAP 361 | - Reliable real-time web ui 362 | 363 | Cons: 364 | 365 | - Pricing. 366 | 367 | In our case we would have to pay $3700 for Android CI Cluster + cost of machines. Plus another $3700 for two iOS teams and a shitton more for Backend, Frontend and Data Science teams. 368 | 369 | However big autoscaling TeamCity cluster may worth it since it has fine-grained permissions control and allows to use scalable database solutions like PostgreSQL and MySQL resulting in handling hundreds and probably even thousands of projects. 370 | 371 | ##### [Concourse][concourse] 372 | 373 | Pros: 374 | 375 | - Unix-way designed CI system 376 | - Just three concepts: Tasks, Resources and Jobs that cover all the things you need from CI 377 | - Designed for Pipelines 378 | - Made with Containerization in mind 379 | - To add a worker you just need to configure pair of SSH-keys, no JNLP or other pain 380 | - Good HTTP API 381 | - Great command line tool `fly` to control CI without browser 382 | - YML-based Pipeline DSL, great for simple pipelines 383 | - The only way to set configuration is CLI tool or API 384 | - It's free and open source 385 | 386 | Cons: 387 | 388 | - Web UI made by aliens, you'll need a week or so to get used to it 389 | - Log output [eats all the CPU and can crash Chrome Tabs][concourse log output] while Jenkins renders same log just fine 390 | - Containerization may complicate things: 391 | - Concourse is designed to run builds in containers like Docker 392 | - If you have complicated setup you may end up running [Docker in Docker][docker in docker 2015] which never works well at scale 😿 393 | - YML-based Pipeline DSL, bad for sophisticated pipelines 394 | - OAuth-based user management makes it hard to work with existing LDAP infrastructure 395 | - No built-in artifact management: "just upload them to S3 and print url to the build log" they said (It's a great idea but complicates things) 396 | 397 | >There are few other options, however for different reasons they were not fitting our needs and not listed here. 398 | 399 | ### Why [Bitbucket Server][bitbucket server] 400 | 401 | Historical reasons. Some of us really love GitHub and will be happy to migrate as soon as possible. There are few things though that Bitbucket is able to do out of the box which GitHub isn't: 402 | 403 | - [Automatic branch merging][bitbucket automatic branch merging]. We follow [Git FTFY][git ftfy] branching model and this feature allows us automatically merge changes from say `dev/1.27.0` branch to `dev/1.28.0`. 404 | - [Branch permissions][bitbucket branch permissions]. Gives more fine-grained control over branch and tags permissions than GitHub with support for regexp. Prevents deletion/modification of tags, changes without pull requests for particular branches and so on. 405 | 406 | But GitHub has much more small, but important thingies here and there which overall makes it more pleasant to use. 407 | 408 | ### [Mainframer][mainframer] — How we build apk with source code changes in under 10 seconds during local development 409 | 410 | In 2017 building Android project with Gradle takes minute(s) on top 2017 15'' MacBook Pro, makes it hot and significantly drains battery. Also, laptop is basically unusable during the build time. This is not great. 411 | 412 | So we've made a tool that allows you build projects remotely on a much more performant hardware. 413 | 414 | TL;TR 3 simple steps: 415 | 416 | 1. Sync project from local to remote machine (rsync) 417 | 1. Execute command on the remote machine (ssh) 418 | 1. Sync project from remote to local machine (rsync) 419 | 420 | In such a scheme speed of your build depends not on the hardware of your local machine, but on the hardware on your remote machine. 421 | 422 | We're using machine with Intel Core i7 6770k, 64 GB of memory and fast SSD that serves both Rider and Driver Android Teams, costs less than a half of one MacBook Pro and saves hours of development time every day! 423 | 424 | Guys from [EL Passion][el passion] even made an [IntelliJ Plugin][mainframer intellij plugin] that smoothly integrates Mainframer with IntelliJ's Run Configurations and we use it to build and run apks in Android Studio, as well as compiling and running Unit and Instrumentation tests. 425 | 426 | With Kotlin inremental compilation and recent update that made kapt incremental (in many but not all cases) we're able to build apk with source code & resources changes in under 10 seconds including sync time! 427 | 428 | So we encourage teams who suffer from slow builds to try [Mainframer][mainframer] and [share results][mainframer share results] with community! 429 | 430 | --- 431 | 432 | ## That's it, thank you for reading! 433 | 434 | Author: [Artem Zinnatullin][artem_zin] 435 | 436 | We use GitHub issues for discussions and comments: https://github.com/gojuno/engineering/issues/3 437 | 438 | You can follow this repo to receive updates about new articles from Juno Engineering Team and new comments in discussions. 439 | 440 | [bitbucket server]: https://www.atlassian.com/software/bitbucket/server 441 | [docker]: https://www.docker.com/ 442 | [docker in docker 2015]: https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ 443 | [docker in docker 2017]: https://github.com/jpetazzo/dind 444 | [bash]: https://en.wikipedia.org/wiki/Bash_%28Unix_shell%29 445 | [bash tutorial]: https://www.ibm.com/developerworks/library/l-bash2/index.html 446 | [kotlin script]: https://kotlinlang.org/docs/tutorials/command-line.html#using-the-command-line-to-run-scripts 447 | [kotlin script depmngmt]: https://youtrack.jetbrains.com/issue/KT-15266 448 | [commander]: https://github.com/gojuno/commander 449 | [swarmer]: https://github.com/gojuno/swarmer 450 | [composer]: https://github.com/gojuno/composer 451 | [spoon]: https://github.com/square/spoon 452 | [composer dynamic sharding]: https://github.com/gojuno/composer/issues/66 453 | [jenkinsfile]: https://jenkins.io/doc/book/pipeline/jenkinsfile/ 454 | [groovy]: http://www.groovy-lang.org/ 455 | [jenkins]: https://jenkins.io/ 456 | [jenkinsfile stash unstash]: https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#stash-stash-some-files-to-be-used-later-in-the-build 457 | [jenkins 44809]: https://issues.jenkins-ci.org/browse/JENKINS-44809 458 | [jenkins blueocean]: https://jenkins.io/projects/blueocean/ 459 | [jenkins enterprise]: https://www.cloudbees.com/products/cloudbees-jenkins-enterprise 460 | [teamcity]: https://www.jetbrains.com/teamcity/ 461 | [teamcity pipeline]: https://blog.jetbrains.com/teamcity/2017/02/kotlin-configuration-scripts-extending-the-teamcity-dsl/ 462 | [concourse]: https://concourse.ci/introduction.html 463 | [concourse log output]: https://github.com/concourse/concourse/issues/1009 464 | [bitbucket automatic branch merging]: https://confluence.atlassian.com/bitbucketserver/automatic-branch-merging-776639993.html 465 | [bitbucket branch permissions]: https://confluence.atlassian.com/bitbucketserver/using-branch-permissions-776639807.html 466 | [git ftfy]: https://artemzin.com/blog/git-ftfy-branching-model-continuation-of-git-flow-considered-harmful/ 467 | [mainframer]: https://github.com/gojuno/mainframer 468 | [el passion]: https://github.com/elpassion 469 | [mainframer intellij plugin]: https://github.com/elpassion/mainframer-intellij-plugin 470 | [mainframer share results]: https://github.com/gojuno/mainframer/issues/17 471 | [artem_zin]: https://twitter.com/artem_zin 472 | -------------------------------------------------------------------------------- /articles/ci_pipeline_and_custom_tools_of_android_projects/images/build_merge_result_and_commit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gojuno/engineering/5dc5191df875dedc2ae2f9a8e6889ef7e42d61a6/articles/ci_pipeline_and_custom_tools_of_android_projects/images/build_merge_result_and_commit.png -------------------------------------------------------------------------------- /articles/ci_pipeline_and_custom_tools_of_android_projects/images/ci_pipeline_android.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gojuno/engineering/5dc5191df875dedc2ae2f9a8e6889ef7e42d61a6/articles/ci_pipeline_and_custom_tools_of_android_projects/images/ci_pipeline_android.png -------------------------------------------------------------------------------- /specs/rxjava-2-migration.md: -------------------------------------------------------------------------------- 1 | # RxJava 2 Migration 2 | 3 | ## Proposed Plan 4 | 5 | * Migrate independent semi-standalone services. 6 | * Migrate UI components where possible using interop to communicate with required services. 7 | * Migrate remaining services since consumers (UI components) will be ready for such changes at this point. 8 | * Remove interop and RxJava 1-related dependencies. 9 | 10 | ## Tips and Tricks 11 | 12 | * Read [the official documentation](https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0) 13 | carefully, it contains answers to most of your questions. 14 | * At this point RxJava 1 and RxJava 2 can be used side-by-side without any consequences. 15 | All fundamental thingies, such as Retrofit adapter and custom operators are already in place, 16 | but you may need to write your own. 17 | * Refer to a package migrated first when in doubt. 18 | You may find it useful as a starting searching point. 19 | * Do not forget to change (and run) unit tests, this can and will presume the correct behavior. 20 | If there is no test, prefer to write it first if possible and reasonable. 21 | * RxJava 2 crashes on `null` emissions. Make sure you are not consuming such streams. 22 | The final goal is to replace `Observable` with either `Observable>` 23 | or `Optional` everywhere, no exceptions. 24 | * Migration changes should be as isolated as possible. Change god-like services 25 | and friends, last. It is always better to migrate a single atomic thing and test it properly 26 | instead of changing basically everything. 27 | * Create a tech task for each migration with description what to recheck. Your testing team is really good, 28 | but help them out to catch possible issues. 29 | 30 | --------------------------------------------------------------------------------