├── LICENSE ├── README.md ├── bitrise.yml └── start_emulator.sh /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 ychescale9 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Bitrise UI Tests Workflow 2 | 3 | Custom [bitrise.io](https://app.bitrise.io) workflow for running instrumented tests on emulators. 4 | 5 | :warning: **Update: 2020-04-26** - there have been a few improvements in Android Testing on CI for the last few months so some of the information presented below no longer be relevant. Specifically: 6 | 7 | - many projects are migrating to **GitHub Actions** for Android CI and I created [Android Emulator Runner](https://github.com/ReactiveCircus/android-emulator-runner), a custom GitHub Action that installs, configures and runs **hardware-accelerated** Android Emulators on macOS virtual machines. 8 | - GitHub Actions' `macos` VMs are free for open-source projects but they are significantly more expensive than the Linux VMs for private repos. While **Android Emulator Runner** works with `ubuntu` VMs without hardware acceleration, I [don't recommand it](https://github.com/ReactiveCircus/android-emulator-runner/issues/46). Intead I'd encourage you to take a look at [Cirrus CI](https://cirrus-ci.org/) which provides **KVM-enabled** Linux VMs and pricing seems pretty reasonable (100% free for opensource projects). 9 | 10 | If you'd like to learn more about running Android instrumented tests on CI in general, I wrote a couple of articles: 11 | 12 | - [Running Android Emulators on CI - from bitrise.io to GitHub Actions](https://dev.to/ychescale9/running-android-emulators-on-ci-from-bitrise-io-to-github-actions-3j76) - I explored my journey on finding the best solution for runing Android instrumented tests on CI, expecially opensource projects. I also covered how I migrated one of my opensource libraries from **bitrise.io** (using the workflow in this repo) to the custom **Android Emulator Runner** GitHub Action. 13 | - [Exploring Cirrus CI for Android](https://dev.to/ychescale9/exploring-cirrus-ci-for-android-434) - A deep dive into **Cirrus CI** and all the features relevant to Android and how you can optimize your pipeline etc. I also created some [templates](https://github.com/ReactiveCircus/cirrusci-android-templates) to help you get started. 14 | 15 | ## Why do I need this? 16 | 17 | Running proper automated UI tests on CI remains a challenge especially when operating with free plans for side projects. AFAIK **Bitrise** is the only service that supports running x86 emulators natively with `-gpu swiftshader`. They offer 1 free container for open-source projects. Other solutions I've looked at: 18 | 19 | * CircleCI (and most other hosted CI providers) does not yet support running x86 emulators on their host machines which requires KVM support. 20 | * Firebase Test Lab only allows 10 tests/day on virtual device and 5 tests/day on physical device with the Spark (free) Plan. 21 | * Robolectric 4.x introduced support for running Espresso tests. While sharing test source between JVM and on-device tests mostly works, robolectric's shadowed environment is still too unstable / immature to be considered useful as they often have very different behaviors than when running on an emulator or real device. 22 | * [Emulator 28.1.8 Canary](https://androidstudio.googleblog.com/2019/02/emulator-2818-canary.html) introduced a `emulator-headless` mode (replaced by `emulator -no-window` in [Emulator 29.2.7 Canary](https://androidstudio.googleblog.com/2019/11/emulator-2927-canary.html)) but with `-no-accel` on starting an emulator and installing APKs are still order of magnitude slower than running with `host` or `swiftshader` GPU acceleration. 23 | 24 | Bitrise provides the [Android Build for UI Testing](https://blog.bitrise.io/new-step-android-build-for-ui-testing) and [Virtual Device Testing for Android](https://github.com/bitrise-steplib/steps-virtual-device-testing-for-android) steps which use **Firebase Test Lab** to run tests for the chosen module / build variant with no limit on device hours / no. of test executions. But it has a couple of limitation: 25 | 26 | * Only 1 build variant from 1 module can be run for each step. This means if you have multiple library modules with instrumented tests, you'll have to manually configure a **Android Build for UI Testing** or **Virtual Device Testing for Android** step for each module (or setup parallel workflows for running these in parallel if you wish to pay for additional containers). 27 | * Running tests in a library module doesn't work unless an app APK is also present. The [workaround](https://discuss.bitrise.io/t/vdt-not-able-to-run-instrumentation-tests-on-android-library-project/3197/7) is to also run `app:assembleDebug` for your library module tests. 28 | 29 | You could also use the [AVD Manager](https://github.com/bitrise-steplib/steps-avd-manager) and [Wait for Android emulator](https://github.com/bitrise-steplib/steps-wait-for-android-emulator) steps to spin up an Emulator on Bitrise locally and then run all your tests with gradle, but you don't have full control over the Emulator configs. 30 | 31 | With a custom Workflow, we are able to control how we want to configure and run an Emulator on Bitrise with a custom shell script, and run all (or any) tests by running a single gradle command. 32 | 33 | ## Steps 34 | 1. Copy [start_emulator.sh](start_emulator.sh) into your `${projectRoot}/.bitrise/` (you could also just copy and paste it into a **Script** step in your Bitrise workflow directly). 35 | 2. Specify `API_LEVEL` and `BUILD_TOOLS_VERSION` in the script. You could also use environment variables for these. 36 | 3. Add a **Script Runner** step to your Bitrise workflow and specify `.bitrise/start_emulator.sh` for **Script location** 37 | 4. Add a **Gradle Runner** step to your Bitrise workflow and specify `connectedDebugAndroidTest -Dorg.gradle.daemon=false -Dkotlin.incremental=false` for **Gradle task to run**. You could also use more fine-grained commands to only run tests for certain modules / build variants. 38 | 5. Push your change to trigger the build. 39 | 40 | A sample `bitrise.yml` can be found [here](bitrise.yml). 41 | -------------------------------------------------------------------------------- /bitrise.yml: -------------------------------------------------------------------------------- 1 | --- 2 | format_version: '8' 3 | default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git 4 | project_type: android 5 | trigger_map: 6 | - push_branch: master 7 | workflow: run-instrumented-tests 8 | - pull_request_source_branch: "*" 9 | workflow: run-instrumented-tests 10 | workflows: 11 | run-instrumented-tests: 12 | steps: 13 | - activate-ssh-key: 14 | run_if: '{{getenv "SSH_RSA_PRIVATE_KEY" | ne ""}}' 15 | - git-clone: {} 16 | - cache-pull: {} 17 | - script-runner: 18 | inputs: 19 | - file_path: ".bitrise/run_emulator.sh" 20 | title: Start Emulator 21 | - gradle-runner: 22 | inputs: 23 | - gradlew_path: "./gradlew" 24 | - app_file_include_filter: "*.apk" 25 | - app_file_exclude_filter: '' 26 | - test_apk_file_include_filter: '' 27 | - mapping_file_include_filter: '' 28 | - gradle_task: connectedDebugAndroidTest -Dorg.gradle.daemon=false -Dkotlin.incremental=false --no-parallel 29 | - cache_level: all 30 | title: Run Tests 31 | - cache-push: {} 32 | app: 33 | envs: 34 | - opts: 35 | is_expand: false 36 | CI: 'true' 37 | - opts: 38 | is_expand: false 39 | PROJECT_LOCATION: "." 40 | -------------------------------------------------------------------------------- /start_emulator.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # fail if any commands fails 3 | set -e 4 | # debug log 5 | set -x 6 | 7 | API_LEVEL=23 8 | BUILD_TOOLS_VERSION=29.0.3 9 | 10 | sdkmanager "system-images;android-${API_LEVEL};google_apis;x86" 11 | sdkmanager "build-tools;${BUILD_TOOLS_VERSION}" 12 | echo "y" | sdkmanager --update 13 | 14 | echo no | avdmanager create avd --force --name "api-${API_LEVEL}" --abi "google_apis/x86" --package "system-images;android-${API_LEVEL};google_apis;x86" --device "Nexus 5X" 15 | 16 | "$ANDROID_HOME"/emulator/emulator -avd "api-${API_LEVEL}" -gpu swiftshader_indirect -no-window -no-snapshot -noaudio -no-boot-anim -camera-back none > /dev/null 2>&1 & 17 | 18 | adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 5; done; input keyevent 82' 19 | 20 | "$ANDROID_HOME"/platform-tools/adb shell settings put global window_animation_scale 0.0 21 | "$ANDROID_HOME"/platform-tools/adb shell settings put global transition_animation_scale 0.0 22 | "$ANDROID_HOME"/platform-tools/adb shell settings put global animator_duration_scale 0.0 23 | --------------------------------------------------------------------------------