├── .gitignore ├── Android ├── Emulator │ ├── Calabash Interaction with Emulator and App.md │ └── V2 - Data Storage and Privacy │ │ ├── Testing For Sensitive Data in Logs │ │ ├── README.md │ │ └── features │ │ │ ├── logging.feature │ │ │ └── step_definitions │ │ │ └── calabash_steps.rb │ │ ├── Testing for Sensitive Data in Databases │ │ ├── README.md │ │ ├── features.feature │ │ └── steps │ │ │ └── steps.rb │ │ ├── Testing for Sensitive Data in the Clipboard │ │ ├── README.md │ │ ├── clipboard_cache.feature │ │ └── steps │ │ │ └── clipboard_steps.rb │ │ ├── Testing for Sensitive Data in the Keyboard Cache │ │ ├── README.md │ │ ├── features.feature │ │ └── steps │ │ │ └── steps.rb │ │ └── Testing for Sensitive Data in the Shared Preferences │ │ ├── README.md │ │ ├── features.feature │ │ └── steps │ │ └── steps.rb └── README.md ├── FirstTest ├── README.md ├── my_first_test.feature └── step_definitions │ └── calabash_steps.rb ├── LICENSE ├── README.md ├── Setup.md ├── app ├── README.md ├── app-arm-debug-Android5.apk └── app-x86-debug.apk ├── docker ├── Calabash Syntax.md ├── Dockerfile └── README.md └── iOS ├── Emulator └── V2 - Data Storage and Privacy │ ├── Testing For Sensitive Data in Logs │ ├── README.md │ ├── features.feature │ └── steps │ │ └── steps.rb │ ├── Testing for Sensitive Data in the Clipboard │ ├── README.md │ ├── clipboard_cache.feature │ └── steps │ │ └── clipboard_steps.rb │ └── Testing for Sensitive Data in the Keyboard Cache │ ├── README.md │ ├── features.feature │ └── steps │ └── steps.rb └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /Android/Emulator/Calabash Interaction with Emulator and App.md: -------------------------------------------------------------------------------- 1 | ## How to interact with the MSTG-Hacking-Playground UI 2 | 3 | ### Buttons on the screen 4 | 5 | Every button is an android.support.v7.widget.AppCompatButton, and it is possible to go through all of them just incrementing the `index` value of the command 6 | 7 | `query("android.support.v7.widget.AppCompatButton index:0")` 8 | 9 | In example if we want to query the first element: __OMTG DATAST 001 BADENCRYPTION__, from the calabash-android console we will have: 10 | 11 | ``` 12 | irb(main):017:0> query("android.support.v7.widget.AppCompatButton index:0") 13 | [ 14 | [0] { 15 | "class" => "android.support.v7.widget.AppCompatButton", 16 | "description" => "android.support.v7.widget.AppCompatButton{f795b5f VFED..C.. ........ 0,158-1080,284}", 17 | "tag" => nil, 18 | "id" => nil, 19 | "text" => "OMTG-DATAST-001-BadEnryption", 20 | "visible" => true, 21 | "rect" => { 22 | "height" => 126, 23 | "width" => 1080, 24 | "y" => 232, 25 | "x" => 0, 26 | "center_x" => 540, 27 | "center_y" => 295 28 | }, 29 | "enabled" => true, 30 | "contentDescription" => nil 31 | } 32 | ] 33 | ``` 34 | 35 | ### ADB and interaction with the emulator/device 36 | 37 | Every feature that is developed is a standard Ruby function and, therefore, we can use every Ruby syntax to achieve our goal. Because we want to interact with the OS using the Android tools, we need to execute commands through the docker container and get the results back. 38 | 39 | Ruby has a built in function called `%x(cmd)` that allows to execute commands on the hos machine (in our case the docker container) 40 | 41 | So in example: 42 | 43 | ``` 44 | %x(adb logcat -b #{log} -c) 45 | 46 | ``` 47 | 48 | will execute the adb command through the docker container, and our function will look like: 49 | 50 | ``` 51 | Given /^I clean "(.*)" the application log$/ do |log| 52 | %x(adb logcat -b #{log} -c) 53 | end 54 | 55 | ``` 56 | 57 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing For Sensitive Data in Logs/README.md: -------------------------------------------------------------------------------- 1 | # Testing For Sensitive Data in Logs 2 | 3 | The following features read the application logs while the app is running and search for sensitive information in the "ERROR","DEBUG", "WARN" and "INFO" logs. The feature runs logcat on the device and detect the regex specified by the tester. The keyword must be a sensitive information that should not be logged. 4 | 5 | 6 | ## Clean all the logs 7 | 8 | To clean the logs on the device use the following feature 9 | 10 | Given I clean "all" the application log 11 | 12 | where "all" is the keyword used on the adb command 13 | 14 | adb logcat -b #{log} -c 15 | 16 | ## Log parsing: detect sensitive information 17 | 18 | The following step 19 | 20 | I should not see text with "SENSITIVE_INFORMATION" in my "LOG_LEVEL" log 21 | 22 | searches for the specified **SENSITIVE_INFORMATION** in the **LOG_LEVEL** Android logs. 23 | 24 | LOG_LEVEL can only have the following values: 25 | 26 | * Verbose (lowest priority) 27 | * Debug 28 | * Info 29 | * Warning 30 | * Error 31 | * Fatal (highest priority) 32 | 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing For Sensitive Data in Logs/features/logging.feature: -------------------------------------------------------------------------------- 1 | Feature: Check that no sensitive info are written in the log files 2 | 3 | Scenario: Open the app and press OMTG_DATAST_002_Logging 4 | Given I clean "all" the application log 5 | When I press "OMTG_DATAST_002_Logging" 6 | Then I enter "Joe Hacker" in "loggingUsername" 7 | Then I enter "secret" in "loggingPassword" 8 | Then I press "Login" 9 | And I should not see text with "secret" in my "Debug" log 10 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing For Sensitive Data in Logs/features/step_definitions/calabash_steps.rb: -------------------------------------------------------------------------------- 1 | require 'calabash-android/calabash_steps' 2 | 3 | 4 | Given(/^I clean "([^"]*)" the application log$/) do |log| 5 | %x(adb logcat -b #{log} -c) 6 | end 7 | 8 | 9 | Then(/^I enter "([^"]*)" in "([^"]*)"$/) do |input, id| 10 | enter_text("android.widget.EditText id:'#{id}'", input) 11 | end 12 | 13 | 14 | Then(/^I should not see text with "([^"]*)" in my "([^"]*)" log$/) do |text, type| 15 | loglevel = case type 16 | when "Debug" 17 | loglevel = "D" 18 | when "Info" 19 | loglevel = "I" 20 | when "Warning" 21 | loglevel = "W" 22 | when "Error" 23 | loglevel = "E" 24 | when "Fatal" 25 | loglevel = "F" 26 | else 27 | loglevel = "S" 28 | end 29 | 30 | counter = %x(adb logcat -d *:#{loglevel} | grep #{text} | wc -l) 31 | 32 | clean_counter = counter.to_i 33 | 34 | 35 | if clean_counter > 0 36 | fail(msg="MSTG V2.1: Sensitive information #{text} found #{clean_counter} times in log file") 37 | end 38 | end 39 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in Databases/README.md: -------------------------------------------------------------------------------- 1 | # Testing for sensitive information in Databases 2 | 3 | The following feature checks that sensitive information are not stored in plaintext in a specific database 4 | 5 | 6 | ## Check a specific database to find sensitive information 7 | 8 | The feature 9 | 10 | When the database "privateNotSoSecure" is created 11 | 12 | checks that the database we want to investigate, has been created by the application. If the database does not exist, the test fails 13 | 14 | 15 | When the database has been created the feature 16 | 17 | Then the tables should not contain "Admin" password in cleartext 18 | 19 | will inspect the sqlite database to see if a specific plaintext secret has been stored. 20 | 21 | 22 | ### Command 23 | 24 | To read info from the database, we dump all the tables and look for the specific secret 25 | 26 | 27 | adb shell sqlite3 " + $path + $database + " .dump | grep #{plaintext} | wc -l 28 | 29 | 30 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in Databases/features.feature: -------------------------------------------------------------------------------- 1 | Feature: Databases must not 2 | contain sensitive information 3 | 4 | Scenario: As mobile application 5 | I do not want to store 6 | sensitive information in 7 | databases 8 | 9 | 10 | 11 | When I press "OMTG_DATAST_001_SQLite" 12 | And I wait for 2 seconds 13 | When the database "privateNotSoSecure" is created 14 | And I wait for 2 seconds 15 | Then the tables should not contain "Admin" password in cleartext 16 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in Databases/steps/steps.rb: -------------------------------------------------------------------------------- 1 | 2 | 3 | $gvar = "sg.vp.owasp_mobile.omtg_android" 4 | $database = "" 5 | $path = "" 6 | 7 | 8 | And /^the database "(.*)" is created$/ do |database| 9 | 10 | #The following command dumps the whole database 11 | $path = "/data/data/" + $gvar + "/databases/" 12 | 13 | $database = "#{database}" 14 | 15 | 16 | command = "adb shell ls " + $path + $database 17 | 18 | $output = %x(#{command}) 19 | 20 | if $output != '' 21 | puts "Database #{database} has been found!" 22 | else 23 | fail("Database #{database} has NOT been found! Please make sure the database exists") 24 | end 25 | 26 | end 27 | 28 | 29 | 30 | 31 | Then /^the tables should not contain "(.*)" password in cleartext$/ do |plaintext| 32 | 33 | 34 | puts "Reading the database " + $database 35 | 36 | #Dump the database and look for plaintext secrets 37 | command = "adb shell sqlite3 " + $path + $database + " .dump | grep #{plaintext} | wc -l" 38 | 39 | #Executes the command 40 | output = %x(#{command}) 41 | 42 | 43 | count = "#{output}".to_i 44 | 45 | 46 | if count > 0 47 | fail(msg="MSTG V2: #{plaintext} found in Database " + $database) 48 | end 49 | end 50 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Clipboard/README.md: -------------------------------------------------------------------------------- 1 | # Testing For Sensitive Data in Clipboard 2 | 3 | Input fields and labels that shows sensitive information should not have the possibility to copy the text in the clipboard enables, exposing sensitive information outside of the app context. 4 | 5 | 6 | ## Check the clipboard 7 | 8 | This feature tries to copy information put in input fields. The information that is copied should be a sensitive information that should not be copied by the user. 9 | 10 | Then I should not be able to "Copy" in my clipboard 11 | 12 | 13 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Clipboard/clipboard_cache.feature: -------------------------------------------------------------------------------- 1 | Feature: Clipboard cache should not 2 | contain sensitive information 3 | 4 | Scenario: As a user I try to copy a 5 | sensitive information in my 6 | clipboard 7 | 8 | When I enter "THISISASENSITIVEINFORMATION" in textField1 9 | And I "Select All" the text in "usernameField" 10 | Then I should not be able to "Copy" in my clipboard -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Clipboard/steps/clipboard_steps.rb: -------------------------------------------------------------------------------- 1 | And /^I "(.*)" the text in "(.*)"$/ do |menu,input_field| 2 | #tap will just highlight a word 3 | touch_hold(query("* marked:'#{input_field}'"), options = {}) 4 | sleep(0.4) 5 | touch("button marked:'#{menu}'") 6 | sleep(0.4) 7 | 8 | end 9 | 10 | Then /^I should not be able to "(.*)" in my clipboard$/ do |menu| 11 | if query("button marked:'#{menu}'").count > 0 12 | fail(msg="Sensitive information found in Clipboard Cache") 13 | end 14 | end 15 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Keyboard Cache/README.md: -------------------------------------------------------------------------------- 1 | # Testing whether the keyboard cache is disabled for text input fields 2 | 3 | 4 | The following feature checks that a sensitive information inserted in the input text is not stored in the keyboard cache 5 | 6 | 7 | ## Check the keyboard cache 8 | 9 | The following feature 10 | 11 | Then I should not see text with "THISISASENSITIVEINFORMATION" in my keyboad_cache 12 | 13 | checks for sensitive data in the keyboard cache. The information must first be added to the cache explicitely when is passed as a input value 14 | 15 | ### More details 16 | 17 | [Sensitive data in the keyboard cache](https://github.com/OWASP/owasp-mstg/blob/master/Document/0x06d-Testing-Data-Storage.md#testing-for-sensitive-data-in-the-keyboard-cache) 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Keyboard Cache/features.feature: -------------------------------------------------------------------------------- 1 | When I enter "THISISASENSITIVEINFORMATION" in textField1 2 | Then I should not see text with "THISISASENSITIVEINFORMATION" in my keyboad_cache -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Keyboard Cache/steps/steps.rb: -------------------------------------------------------------------------------- 1 | Then /^I (?:should not)? see text with "(.*)" in my keyboard_cache$/ do |text| 2 | if %x( cat /YOUR_PATH_TO_THE_KEYBARD_CACHE_FILE | grep #{text} | wc -l ).delete!("\n").delete!(" ").to_i > 0 3 | fail(msg="#{text} found in keyboard cache") 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Shared Preferences/README.md: -------------------------------------------------------------------------------- 1 | # Testing for sensitive information in Shared Preferences 2 | 3 | 4 | The following feature checks that a sensitive information inserted in the input text is not stored in the keyboard cache 5 | 6 | 7 | ## Check the .xml files to find sensitive information 8 | 9 | The following feature 10 | 11 | Then I should not see "THISISASENSITIVEINFORMATION" in the shared preferences 12 | 13 | checks for sensitive data in the file stored in shared preferences 14 | 15 | 16 | ###Command 17 | 18 | 19 | 20 | cat /data/data/ANDROID_PACKAGE/shared_prefs/* 21 | 22 | 23 | -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Shared Preferences/features.feature: -------------------------------------------------------------------------------- 1 | Feature: Shared Preferences must not 2 | contain sensitive information 3 | 4 | Scenario: As mobile application 5 | I do not want to store 6 | sensitive information in the 7 | shared preferences 8 | 9 | 10 | 11 | When I press "OMTG-DATAST-001-Shared Preferences" 12 | Then I wait for 3 seconds 13 | Then I should not see "secret" in the shared preferences -------------------------------------------------------------------------------- /Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Shared Preferences/steps/steps.rb: -------------------------------------------------------------------------------- 1 | 2 | 3 | $gvar = "sg.vp.owasp_mobile.omtg_android" 4 | 5 | 6 | 7 | Then /^I (?:should not)? see "(.*)" in the shared preferences$/ do |text| 8 | 9 | string = "adb shell cat /data/data/" + $gvar + "/shared_prefs/*.xml | grep #{text} | wc -l" 10 | 11 | $var = %x(#{string}).to_i 12 | 13 | if $var > 0 14 | fail(msg="MSTG V2.1: #{text} found in Shared Preferences") 15 | end 16 | end 17 | -------------------------------------------------------------------------------- /Android/README.md: -------------------------------------------------------------------------------- 1 | # BDD security tests for Android application 2 | 3 | The test for iOS are written using calabash-ios as engine and the Android tools 4 | 5 | ## Requirements 6 | 7 | Check the main README to see the requirements 8 | 9 | 10 | 11 | -------------------------------------------------------------------------------- /FirstTest/README.md: -------------------------------------------------------------------------------- 1 | # First Test 2 | 3 | This is the first test that the team should run to check if everything works after Calabash has been setup. 4 | 5 | 6 | ## Run the application 7 | 8 | The feature file contains basic steps like run the app and wait for a timeout. If both tests succeed Calabash is configured correctly and you can proceed with the other tests 9 | 10 | Given my app is running 11 | Then I wait 5 seconds 12 | 13 | 14 | 15 | 16 | -------------------------------------------------------------------------------- /FirstTest/my_first_test.feature: -------------------------------------------------------------------------------- 1 | Feature: Test feature 2 | Scenario: As a user I want to see if the app starts properly 3 | Given my app is running 4 | Then I wait 5 seconds 5 | -------------------------------------------------------------------------------- /FirstTest/step_definitions/calabash_steps.rb: -------------------------------------------------------------------------------- 1 | require 'calabash-android/calabash_steps' 2 | 3 | Given(/^my app is running$/) do 4 | sleep(1) 5 | end 6 | 7 | Then(/^I wait (\d+) seconds$/) do |arg1| 8 | sleep(2) 9 | end 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 ING Bank 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Automating OWASP MSTG through BDD mobile security testings 2 | 3 | ING provides a set of BDD security tests with Calabash,Cucumber and Ruby, following the OWASP Mobile Security Testing Guide and the OWASP Mobile Top 10 2016, that can be easily customized and implemented through the entire CI/CD pipeline. 4 | 5 | The tests use the power of [calabash.io](https://calabash.io) to automate the UI, [Cucumber](https://cucumber.io/) and Gherkin for its simple language and the Android tools to check for security issues. 6 | 7 | ### Prerequisites for Android 8 | 9 | * Docker 10 | * Genymotion or real devices 11 | 12 | ### Prerequisites for iOS 13 | 14 | * macOS 15 | * Xcode CLI tools 16 | 17 | ## Docker container for Android testing 18 | 19 | The docker container can be used to spin up Calabash and the Android tools. Follow the README in the docker folder. 20 | 21 | __At the moment the docker container can only be used to test Android applications. If you want to run iOS tests you need to install Calabash native on macOS__ 22 | 23 | 24 | ## First Test 25 | 26 | The repo contains a simple test that can be used as a sample to check that everything works. 27 | 28 | 29 | ## OWASP MASVS and MSTG 30 | The repo's structure is based on the [OWASP Mobile Application Security Verification Standard](https://github.com/OWASP/owasp-masvs) and the corresponding tests from the [OWASP Mobile Security Testing Guide](https://github.com/OWASP/owasp-mstg). For each test many features are defined with the corresponding steps. 31 | 32 | 33 | ## How to contribute 34 | 35 | In order to contribute in creating new tests, the best way is to identify first which command will be usefull to automate specific controls. Follow the following steps: 36 | 37 | - Go to the "Issues" tab and select a test that you would like to write 38 | - Create a folder in the repo with the following structure OS/Emulator or Device/MASVS chapter/ MSTG Test/ 39 | - so in example Android/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Information in log files/ 40 | - Create a README.md with the description of the commands to use and how to use it 41 | - Create a .feature file 42 | - Create a steps.rb file 43 | 44 | Follow the Wiki to check how to create pull requests. 45 | 46 | 47 | ## Tested on 48 | 49 | The following BDD tests have been tested using the following OWASP vulnerable apps as test cases: 50 | 51 | * Android: [MSTG Hacking Playground](https://github.com/OWASP/MSTG-Hacking-Playground) 52 | * iOS: [iGoat](https://github.com/OWASP/igoat) 53 | 54 | 55 | 56 | 57 | 58 | 59 | -------------------------------------------------------------------------------- /Setup.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Clone ING GitHub repo: 4 | 5 | `git clone https://github.com/ing-bank/bdd-mobile-security-automation-framework` 6 | 7 | 8 | ## Install Docker (if you don't have it already) 9 | `https://docker.com/` 10 | 11 | ## Install VirtualBox 12 | `https://www.virtualbox.org/wiki/Downloads` 13 | 14 | ## Install Genymotion for personal use and install emulator 15 | `https://www.genymotion.com/fun-zone/` 16 | 17 | ## docker build (from the docker/ folder of the repo) 18 | `docker build -t "mobile_testing_automation:calabash_android" .` 19 | 20 | ## Locate apk folders in the Hacking Playground (i.e) 21 | `REPO/app` 22 | 23 | 24 | ## Execute docker run with the mounted folder 25 | 26 | `docker run -v /REPO/app/:/masvs_automation --privileged -i -t mobile_testing_automation:calabash_android /bin/bash` 27 | 28 | ## Connect the device to the emulator 29 | 30 | `adb connect IP_OF_THE_EMULATOR_OR_DEVICE` 31 | 32 | 33 | 34 | -------------------------------------------------------------------------------- /app/README.md: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /app/app-arm-debug-Android5.apk: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ing-bank/bdd-mobile-security-automation-framework/dc39de3b70682175d3ec28de1e7a57f24baca4b8/app/app-arm-debug-Android5.apk -------------------------------------------------------------------------------- /app/app-x86-debug.apk: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ing-bank/bdd-mobile-security-automation-framework/dc39de3b70682175d3ec28de1e7a57f24baca4b8/app/app-x86-debug.apk -------------------------------------------------------------------------------- /docker/Calabash Syntax.md: -------------------------------------------------------------------------------- 1 | Ruby API 2 | ======== 3 | 4 | When writing custom steps, you'll need to use the Ruby API to interact with your application. This document describes the API at a high level. If you want to see details you can look at the source code in the files at [ruby-gem/lib/calabash-android](../ruby-gem/lib/calabash-android/). There are functions in the source code which aren't documented here. Those are way more likely to change (so be warned if you rely on those). 5 | 6 | Calabash Android has a client-server architecture. The Calabash Ruby API is the client side which speaks HTTP with the test server that running on the device along with your app. To get an architectural overview of Calabash Android please read the blog posts: 7 | 8 | [AN OVERVIEW OF CALABASH ANDROID](http://blog.lesspainful.com/2012/03/07/Calabash-Android/) 9 | 10 | # General 11 | ### `start_test_server_in_background` 12 | Starts the test server and the app under test (AUT). If the app is already running it will be restarted. 13 | 14 | ### `reinstall_apps` 15 | Will reinstall both the test server and the AUT to be sure the newest versions are installed. 16 | 17 | 18 | # Query 19 | ### `query(uiquery, *args)` 20 | Query returns an [array](http://www.ruby-doc.org/core-1.9.3/Array.html) of its results. The query function gives powerful query capability from your test code. You can find views and other application objects, and make assertions about them or extract data from them. 21 | 22 | Calabash Android tries to return results that carry useable information by default. For view objects this includes coordinates, class and contentdescription: 23 | 24 | irb(main):002:0> query("button index:1") 25 | => [{"id"=>"save", "enabled"=>true, "contentDescription"=>nil, "class"=>"android.widget.Button", "text"=>"Save", "rect"=>{"center_y"=>724.0, "center_x"=>645.5, "height"=>64, "y"=>692, "width"=>71, "x"=>610}, "description"=>"android.widget.Button{4267b4a0 VFED..C. ........ 497,243-568,307 #7f070023 app:id/save}"}] 26 | 27 | A view is represented as a ruby [Hash](http://www.ruby-doc.org/core-1.9.3/Hash.html) (hash map) so you can look into the result 28 | 29 | irb(main):003:0> query("button index:1").first.keys 30 | => ["id", "enabled", "contentDescription", "class", "text", "rect", "description"] 31 | 32 | The `*args` parameter lets you perform methods on the query result *before* it is returned to your Ruby script code (remember that the query is evaluated as Java code inside the app and the result is sent back to the Ruby code). The form `*args` is Ruby-speak for a variable number of args. For example, if you have a button you can do 33 | 34 | irb(main):005:0> query("button", "text") 35 | => ["Optional Settings", "Save", "Cancel", "Get a free blog at WordPress.com"] 36 | 37 | This calls a 'getter' method "text" (that is text(), getText() or isText()) on each of the buttons in the view (it always returns an array). You can perform a sequence of methods: 38 | 39 | irb(main):007:0> query("button", "text", "length") 40 | => [17, 4, 6, 32] 41 | 42 | irb(main):008:0> query("button", "text", "toLowerCase") 43 | => ["optional settings", "save", "cancel", "get a free blog at wordpress.com"] 44 | 45 | 46 | For methods with arguments you can use hashes. In Ruby 1.9 this has quite nice syntax: 47 | 48 | irb(main):033:0> query("edittext index:1", setText:"1234") 49 | => [""] 50 | 51 | On Ruby 1.8 you can't use key:val as literal Hash syntax so you must do: 52 | 53 | irb(main):034:0> query("edittext index:1", :setText => "1234") 54 | => [""] 55 | 56 | Behind the scenes the Java method `setText` will be execute with the argument `"12345"` on all view elements that were matched by the query. 57 | 58 | Notice that the string `` is Calabash's way of returning from a Java method with return type `void`. 59 | 60 | For more complex methods you use Arrays of Hashes. Here is a complex Ruby 1.9 example: 61 | 62 | TODO: Example 63 | 64 | 65 | ### `element_does_not_exist(uiquery)` 66 | ### `element_exists(uiquery)` 67 | ### `view_with_mark_exists(expected_mark)` 68 | The `element_exists` function returns true if an element exists matching query `uiquery`. 69 | The `element_does_not_exist` function returns true if an element matching query `uiquery` does not exist. 70 | 71 | The function `view_with_mark_exists(expected_mark)` is shorthand for 72 | 73 | element_exists("* marked:'#{expected_mark}'") 74 | 75 | # Waiting 76 | ### `wait_for(options, &block)` 77 | 78 | Waits for a condition to occur. Takes a hash of options and a block to be called repeatedly. The options (which are described below) have the following defaults: 79 | 80 | 81 | { 82 | :timeout => 10, #maximum number of seconds to wait 83 | :retry_frequency => 0.2, #wait this long before retrying the block 84 | :post_timeout => 0.1, #wait this long after the block returns true 85 | :timeout_message => "Timed out waiting...", #error message in case options[:timeout] is exceeded 86 | :screenshot_on_error => true # take a screenshot in case of error 87 | } 88 | 89 | 90 | The `timeout` argument should be a number indicating the maximal number of seconds you are willing to wait (after that amount of time the step will cause your test to fail). The `:post_timeout` (0.1 by default) is an number of seconds to wait *after the condition becomes true*. 91 | 92 | The `&block` parameter is Ruby syntax for saying that this method takes a block of code. This block specifies the condition to wait for. The block should return `true` when the the condition occurs. 93 | 94 | The `:retry_frequency` is a small sleep that is made between each call to the specified block. This describes how often Calabash should poll for the condition to be true. 95 | 96 | 97 | Here is a simple example: 98 | 99 | irb(main):030:0> wait_for(:timeout => 5) { query("button marked:'Save'").size > 0 } 100 | 101 | This will check for the existence of a view matching: "button marked:'Save'". It will wait *at most* 5 seconds (failing if more than 5 seconds pass). It will issue the query repeatedly until it is found or 5 seconds pass. 102 | 103 | A typical form uses `element_exists`. 104 | 105 | irb(main):031:0> wait_for(:timeout => 5) { element_exists("button marked:'Save'") } 106 | 107 | In Ruby short blocks are written with braces (like: `{ element_exists("button marked:'Save'") }`), and more complicated blocks are written using `do`-`end`. For example: 108 | 109 | wait_for(:timeout => 30) do 110 | res = query("checkbox marked:'Geotag Posts'", 'checked') 111 | res.first == true 112 | end 113 | 114 | A Ruby block always returns the value of its last expression (`res.first == true` in this case). 115 | 116 | *Notes:* Waiting for a condition to occur is superior to using the `sleep` function. With `sleep` you end up either specifying too long waits which slows the test down or you become sensitive to timing issues. Sometimes you do need sleep (to wait for animations to complete), but try to use waiting as much as possible. 117 | 118 | ### wait_for_element_exists(uiquery, options={}) 119 | 120 | A high-level waiting function. This captures the common practice of waiting for UI elements, i.e., combining `wait_for` and `element_exists`. 121 | 122 | Takes a query and waits for it to return a results. Calls `wait_for` supplying `options`. 123 | 124 | 125 | irb(main):009:0> wait_for_elements_exist( "* marked:'Please sign in'", :timeout => 10) 126 | 127 | 128 | ### wait_for_elements_exist(elements_arr, options={}) 129 | 130 | Like `wait_for_element_exists` but takes an *array* of queries and waits for all of those queries to return results. Calls `wait_for` supplying `options`. 131 | 132 | 133 | irb(main):008:0> wait_for_elements_exist( ["button marked:'Save'", "* marked:'Please sign in'"], :timeout => 2) 134 | 135 | ### wait_for_element_does_not_exist(uiquery, options={}) 136 | 137 | Similar to `wait_for_element_exists`, but waits for an element to not exist. 138 | 139 | 140 | ### wait_for_elements_do_not_exist(elements_arr, options={}) 141 | 142 | Similar to `wait_for_elements_exist`, but waits for all of the elements to not exist. 143 | 144 | 145 | # Assertions 146 | #### `fail(msg="Error. Check log for details.")` 147 | Will fail the test with message `msg`. Takes a screenshot. 148 | 149 | ### `check_element_exists(query)` 150 | ### `check_element_does_not_exist(query)` 151 | ### `check_view_with_mark_exists(expected_mark)` 152 | Asserts that an element exists using the query function on the parameter `query`. 153 | 154 | The function `check_view_with_mark_exists(expected_mark)` is shorthand for 155 | 156 | check_element_exists("view marked:'#{expected_mark}'") 157 | 158 | # Touch 159 | ### `touch(uiquery, options={})` 160 | 161 | Touches a view found by performing the query `uiquery`. It is recommended that `uiquery` only produce one match, but the default is to just touch the first of the results if there are several. 162 | 163 | The `touch` method is one of the most used in Calabash. It is mostly used in its simplest form: 164 | 165 | irb(main):037:0> touch("* marked:'Save'") 166 | 167 | Which uses content descriptions, ids or texts. This form is so common that there is a short-hand for it: `tap`: 168 | 169 | irb(main):038:0> tap 'Save' 170 | 171 | For flexibility you can also pass in a hash representation of a view and the the touch event will be calculated based on those values and no query will be executed. `touch` will also accept a list of hashes in which case Calabash will touch the first one view in the list. 172 | 173 | The following are all equivalent 174 | 175 | touch("button index:0") 176 | touch("button") 177 | touch(query("button index:0")) 178 | touch(query("button").first) 179 | touch(query("button")) 180 | 181 | # Entering text 182 | ### `keyboard_enter_text(text, options={})` 183 | 184 | Enters **text** into the currently focused view. 185 | 186 | ### `enter_text(uiquery, text, options={})` 187 | 188 | Taps the first element returned by **uiquery**, then enters **text** into the view. 189 | 190 | # Screenshot 191 | ### `screenshot(options={:prefix=>nil, :name=>nil})` 192 | Takes a screenshot of the app. 193 | 194 | screenshot({:prefix => "/tmp", :name=>"my.png"}) 195 | 196 | If prefix and name are nil it will use default values (which is currently the line in the current feature). 197 | 198 | ### `screenshot_embed(options={:prefix=>nil, :name=>nil, :label => nil})` 199 | Takes a screenshot of the app and embeds to cucumber reporters (e.g. html reports). 200 | 201 | screenshot_embed({:prefix => "/tmp", :name=>"my.png", :label => "Mine"}) 202 | 203 | If prefix and name are nil it will use default values (which is currently the line in the current feature). 204 | 205 | Label is the label used in the cucumber report output (equals to name if not specified). 206 | 207 | # Pull and push files and folders from and to the device 208 | ### `pull(remote, local)` 209 | Pulls a file from the device to local computer: 210 | 211 | pull("/sdcard/file.jpg", "file.jpg") 212 | 213 | ### `push(local, remote)` 214 | Pushes a file from the local computer to the device: 215 | 216 | push("file.jpg", "/sdcard/file.jpg") 217 | 218 | Uses [adb](http://developer.android.com/tools/help/adb.html) so same rules apply: 219 | 220 | * Won't be able to pull or push from restricted folders such as /data/data 221 | * If destination path already exists, it's overwritten without warning 222 | * For files, full destination path must be provided, ie: 223 | 224 | Won't work: 225 | ``` 226 | push("file.jpg", "/sdcard/folder") 227 | ``` 228 | 229 | Will work: 230 | ``` 231 | push("file.jpg", "/sdcard/folder/file.jpg") 232 | ``` 233 | 234 | # Read, write and clear SharedPreferences 235 | Simple API over [SharedPreferences](http://developer.android.com/guide/topics/data/data-storage.html#pref), all 236 | methods require the name of the SharedPreferences file as the first argument. Supports ints, floats, booleans and strings. 237 | 238 | It is important to notice that depending on your application you might need to poke around with SharedPreferences 239 | before or after your application or activity starts. In that case you will need to call these methods either 240 | before or after your scenario. 241 | 242 | To do so, you can tag a particular scenario and edit your application lifecycle hooks as explained [here](https://groups.google.com/forum/?fromgroups=#!topic/calabash-android/Ql3iluRMijg). 243 | 244 | ### `get_preferences(name)` 245 | Returns a hash with the preferences available for the given name: 246 | 247 | preferences = get_preferences("my_preferences") 248 | 249 | ### `set_preferences(name, hash)` 250 | Sets the given hash as preferences for the given name: 251 | 252 | set_preferences("my_preferences", {:name => "wadus", :email => "wadus@wadus.com", :id => 8, :active => true}) 253 | 254 | ### `clear_preferences(name)` 255 | Clears the preferences for the given name: 256 | 257 | clear_preferences("my_preferences") -------------------------------------------------------------------------------- /docker/Dockerfile: -------------------------------------------------------------------------------- 1 | ############################################################ 2 | # Dockerfile to run Calabash-android acceptance testing . 3 | ############################################################ 4 | 5 | #Base OS 6 | 7 | FROM ubuntu:14.04 8 | 9 | #Ruby installation 10 | 11 | FROM ruby:2.3.0 12 | 13 | # Install JDK 14 | 15 | RUN apt-get update 16 | RUN apt-get install -y openjdk-7-jdk && \ 17 | rm -rf /var/lib/apt/lists/* 18 | 19 | # Install android sdk 20 | 21 | RUN wget http://dl.google.com/android/android-sdk_r24.4.1-linux.tgz 22 | RUN tar -xvzf android-sdk_r24.4.1-linux.tgz 23 | RUN mv android-sdk-linux /usr/local/android-sdk 24 | RUN rm android-sdk_r24.4.1-linux.tgz 25 | 26 | # Install Android tools 27 | RUN echo y | /usr/local/android-sdk/tools/android update sdk --filter tools,platform-tools,build-tools-21.1.1,build-tools-21.1.0,android-21,extra-google-google_play_services,extra-android-support,extra-android-m2repository,extra-google-analytics_sdk_v2 --no-ui --force -a 28 | 29 | #install android dependencies 30 | 31 | RUN dpkg --add-architecture i386 32 | RUN apt-get update 33 | RUN apt-get -y install lib32stdc++6 lib32z1 lib32z1-dev 34 | 35 | #RUN apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 zlib1g:i386 36 | #install calabash-android 37 | 38 | RUN gem install calabash-android --version 0.9.0 39 | 40 | ENV ANDROID_HOME /usr/local/android-sdk 41 | ENV GRADLE_HOME /usr/local/gradle 42 | ENV ANDROID_SDK_HOME $ANDROID_HOME 43 | ENV PATH $PATH:$ANDROID_SDK_HOME/tools 44 | ENV PATH $PATH:$ANDROID_SDK_HOME/platform-tools 45 | ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk 46 | -------------------------------------------------------------------------------- /docker/README.md: -------------------------------------------------------------------------------- 1 | # Docker and emulator/device setup 2 | 3 | 4 | This document provides a steb by step guide on how to build the docker container, use Genymotion to emulate Android devices and use the container to run the test. 5 | 6 | ## On the host machine 7 | 8 | 1. Install [Genymotion "personal use"](https://www.genymotion.com/fun-zone/) and run an instance of Android API 22 9 | 2. Create a folder with the .apk 10 | 3. Build the image using the Dockerfile (see below) 11 | 4. Run the image and mount the folder with APK and tests (see below) 12 | 13 | 14 | ### Build the image using the Dockerfile 15 | 16 | From the docker directory run 17 | 18 | ``` 19 | docker build -t "mobile_testing_automation:calabash_android" . 20 | ``` 21 | 22 | ### Run the image and mount the folder with APK and tests 23 | 24 | 25 | ``` 26 | docker run -v /PATH/TO/YOUR/FOLDER/WITH/APK/AND/TESTS:/masvs_automation --privileged -i -t mobile_testing_automation:calabash_android /bin/bash 27 | ``` 28 | 29 | 30 | 31 | ## From the container 32 | 33 | ### Connect the emulator/device 34 | 35 | `adb connect IP_OF_THE_EMULATOR_OR_DEVICE` 36 | 37 | To get the IP of the __emulator__ run `adb devices -l` from the host machine. 38 | 39 | To get the IP of the __device__ run `adb shell ifconfig` and take the address of eth0 40 | 41 | ### Create the feature folder 42 | 43 | Run `calabash-android gen` to generate the feature folder 44 | 45 | It will create a Cucumber skeleton in the current folder like this: 46 | 47 | 48 | features 49 | |_support 50 | | |_app_installation_hooks.rb 51 | | |_app_life_cycle_hooks.rb 52 | | |_env.rb 53 | | |_hooks.rb 54 | |_step_definitions 55 | | |_calabash_steps.rb 56 | |_my_first.feature 57 | 58 | where `my_first.feature` will contain the gherkin syntax and `calabash_steps.rb` will contain the Ruby code 59 | 60 | ### Fix keystore related issues after set up 61 | 62 | After setting up the container the keystore related issues may raise. The following steps will help to fix the issue 63 | 64 | - Navigate to `~/.android/` directory . 65 | - Remove the existing `debug.keystore` file in case one exist 66 | - Apply the following command 67 | 68 | ``` 69 | $keytool -genkey -v -keystore debug.keystore -alias androiddebugkey -storepass android -keypass android -keyalg RSA -keysize 2048 -validity 10000 70 | ``` 71 | - Following that navigate to the directory where the feature files exist and setup the calabash android by using the command 72 | 73 | ``` 74 | $ calabash-android setup 75 | 76 | ``` 77 | Insert the complete path `~/.android/debug.keystore` 78 | 79 | - The resign the new debug.keystore file using the command 80 | 81 | ``` 82 | $ calabash-android resign 83 | ``` 84 | - And finally run the test by using the command 85 | 86 | ``` 87 | $ calabash-android run 88 | ``` 89 | 90 | 91 | ### Run the test with Calabash 92 | 93 | `calabash-android run APP.apk` 94 | 95 | 96 | # Troubleshooting: 97 | 98 | ### TLS/ problem: 99 | 100 | On the host machine run 101 | 102 | ``` 103 | sysctl -w net.ipv4.tcp_mtu_probing=1net.ipv4.tcp_mtu_probing = 1 104 | ``` 105 | 106 | -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing For Sensitive Data in Logs/README.md: -------------------------------------------------------------------------------- 1 | # Testing For Sensitive Data in Logs 2 | 3 | The following features read the application logs while the app is running and search for sensitive information in the "ERROR" log. The keyword "ERROR" has to be mapped with your log grammar. 4 | -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing For Sensitive Data in Logs/features.feature: -------------------------------------------------------------------------------- 1 | Then I should not see text with "mysensitivetext" in my "ERROR" log -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing For Sensitive Data in Logs/steps/steps.rb: -------------------------------------------------------------------------------- 1 | Then /^I (?:should not)? see text with "(.*)" in my "([^"]*)" log$/ do |text,type| 2 | if %x( xcrun simctl spawn booted syslog | grep #{text} |grep #{type} | wc -l ).delete!("\n").delete!(" ").to_i > 0 3 | fail(msg="#{text} found in #{type} logs") 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Clipboard/README.md: -------------------------------------------------------------------------------- 1 | # Testing For Sensitive Data in Clipboard 2 | 3 | Input fields and labels that shows sensitive information should not have the possibility to copy the text in the clipboard enables, exposing sensitive information outside of the app context. 4 | 5 | 6 | ## Check the clipboard 7 | 8 | This feature tries to copy information put in input fields. The information that is copied should be a sensitive information that should not be copied by the user. 9 | 10 | Then I should not be able to "Copy" in my clipboard 11 | 12 | 13 | -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Clipboard/clipboard_cache.feature: -------------------------------------------------------------------------------- 1 | Feature: Clipboard cache should not 2 | contain sensitive information 3 | 4 | Scenario: As a user I try to copy a 5 | sensitive information in my 6 | clipboard 7 | 8 | When I use the native keyboard to enter "THISISASENSITIVEINFORMATION" into the "username" input field 9 | And I "Select All" the text in "usernameField" 10 | Then I should not be able to "Copy" in my clipboard -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Clipboard/steps/clipboard_steps.rb: -------------------------------------------------------------------------------- 1 | And /^I "(.*)" the text in "(.*)"$/ do |menu,input_field| 2 | #tap will just highlight a word 3 | touch_hold(query("* marked:'#{input_field}'"), options = {}) 4 | sleep(0.4) 5 | touch("button marked:'#{menu}'") 6 | sleep(0.4) 7 | 8 | end 9 | 10 | Then /^I should not be able to "(.*)" in my clipboard$/ do |menu| 11 | if query("button marked:'#{menu}'").count > 0 12 | fail(msg="Sensitive information found in Clipboard Cache") 13 | end 14 | end 15 | -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Keyboard Cache/README.md: -------------------------------------------------------------------------------- 1 | # Testing whether the keyboard cache is disabled for text input fields 2 | 3 | 4 | The following feature checks that a sensitive information inserted in the input text is not stored in the keyboard cache 5 | 6 | 7 | ## Check the keyboard cache 8 | 9 | The following feature 10 | 11 | Then I should not see text with "THISISASENSITIVEINFORMATION" in my keyboad_cache 12 | 13 | checks for sensitive data in the keyboard cache. The information must first be added to the cache explicitely when is passed as a input value 14 | 15 | ### More details 16 | 17 | [Sensitive data in the keyboard cache](https://github.com/OWASP/owasp-mstg/blob/master/Document/0x06d-Testing-Data-Storage.md#testing-for-sensitive-data-in-the-keyboard-cache) 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Keyboard Cache/features.feature: -------------------------------------------------------------------------------- 1 | When I enter "THISISASENSITIVEINFORMATION" in textField1 2 | Then I should not see text with "THISISASENSITIVEINFORMATION" in my keyboad_cache -------------------------------------------------------------------------------- /iOS/Emulator/V2 - Data Storage and Privacy/Testing for Sensitive Data in the Keyboard Cache/steps/steps.rb: -------------------------------------------------------------------------------- 1 | Then /^I (?:should not)? see text with "(.*)" in my keyboard_cache$/ do |text| 2 | if %x( cat /YOUR_PATH_TO_THE_KEYBARD_CACHE_FILE | grep #{text} | wc -l ).delete!("\n").delete!(" ").to_i > 0 3 | fail(msg="#{text} found in keyboard cache") 4 | end 5 | end 6 | -------------------------------------------------------------------------------- /iOS/README.md: -------------------------------------------------------------------------------- 1 | # BDD security tests for iOS application 2 | 3 | The test for iOS are written using calabash-ios as engine and the Xcode cli tools 4 | 5 | ## Requirements 6 | 7 | - macOS 8 | - Xcode CLI 9 | 10 | 11 | 12 | 13 | --------------------------------------------------------------------------------