├── .about.yml ├── .gitignore ├── CONTRIBUTING.md ├── Gemfile ├── Gemfile.lock ├── LICENSE.md ├── README.md ├── _config.yml ├── _layouts ├── page.html └── post.html ├── assets └── css │ └── styles.scss ├── go ├── images ├── crossing-the-chasm-rainbow-of-death.jpg ├── rainbow-of-death.jpg ├── sml.jpg ├── testing-grouplet-rainbow-of-death.jpg └── the-chasm.jpg ├── index.html └── pages ├── browser ├── README.md ├── VirtualBox.md ├── automation.md ├── selenium+sauce.md └── services.md ├── go └── README.md ├── javascript ├── README.md └── examples │ ├── README.md │ └── node-mocha │ ├── .gitignore │ ├── README.md │ ├── gadget.js │ ├── package.json │ └── test │ └── index.js ├── load.md ├── python ├── README.md └── nose.md └── ruby └── README.md /.about.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # .about.yml project metadata 3 | # 4 | # Short name that acts as the project identifier (required) 5 | name: testing-cookbook 6 | 7 | # Full proper name of the project (required) 8 | full_name: Testing Cookbook 9 | 10 | # The type of content in the repo 11 | # values: app, docs, policy 12 | type: docs 13 | 14 | # Describes whether a project team, working group/guild, etc. owns the repo (required) 15 | # values: guild, working-group, project 16 | owner_type: working-group 17 | 18 | # Name of the main project repo if this is a sub-repo; name of the working group/guild repo if this is a working group/guild subproject 19 | parent: wg-testing 20 | 21 | # Maturity stage of the project (required) 22 | # values: discovery, alpha, beta, live, sunset, transfer, end 23 | stage: beta 24 | 25 | # Description of the project 26 | description: The 18F testing cookbook contains recipes and best practices for automated and manual testing in lots of different environments, languages, stacks and platforms. 27 | 28 | # Tags that describe the project or aspects of the project 29 | tags: python,ruby,node,testing,unittests,automation 30 | - 31 | 32 | # Should be 'true' if the project has a continuous build (required) 33 | # values: true, false 34 | testable: true 35 | 36 | # Team members contributing to the project (required) 37 | # Items: 38 | # - github: GitHub user name 39 | # id: Internal team identifier/user name 40 | # role: Team member's role; leads should be designated as 'lead' 41 | team: 42 | - github: mbland 43 | id: mbland 44 | role: lead 45 | - github: arowla 46 | id: alison 47 | role: lead 48 | - github: jmcarp 49 | id: joshcarp 50 | role: contributor 51 | - github: shawnbot 52 | id: shawn 53 | role: contributor 54 | - github: jscottiii 55 | id: jamesscott 56 | role: contributor 57 | 58 | # Technologies used to build the project 59 | stack: 60 | - jekyll 61 | - ruby 62 | - markdown 63 | 64 | # Brief description of the project's outcomes 65 | impact: To disseminate knowledge of testing best practices, and help teams get started testing more quickly. 66 | 67 | # Services used to supply project status information 68 | # Items: 69 | # - name: Name of the service 70 | # category: Type of the service 71 | # url: URL for detailed information 72 | # badge: URL for the status badge 73 | services: 74 | - name: travis 75 | category: ci 76 | url: https://travis-ci.org/18F/testing-cookbook/ 77 | badge: https://travis-ci.org/18F/testing-cookbook.svg 78 | 79 | # Licenses that apply to the project and/or its components (required) 80 | # Items by property name pattern: 81 | # .*: 82 | # name: Name of the license from the Software Package Data Exchange (SPDX): https://spdx.org/licenses/ 83 | # url: URL for the text of the license 84 | licenses: 85 | testing-cookbook: 86 | name: CC0 87 | url: https://github.com/18F/testing-cookbook/blob/18f-pages/LICENSE.md 88 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | _site/ 2 | .sass-cache/ 3 | .DS_store 4 | .*.swp 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | ## Welcome! 2 | 3 | We're so glad you're thinking about contributing to an 18F open source project! If you're unsure or afraid of anything, just ask or submit the issue or pull request anyways. The worst that can happen is that you'll be politely asked to change something. We appreciate any sort of contribution, and don't want a wall of rules to get in the way of that. 4 | 5 | Before contributing, we encourage you to read our CONTRIBUTING policy (you are here), our LICENSE, and our README, all of which should be in this repository. If you have any questions, or want to read more about our underlying policies, you can consult the 18F Open Source Policy GitHub repository at https://github.com/18f/open-source-policy, or just shoot us an email/official government letterhead note to [18f@gsa.gov](mailto:18f@gsa.gov). 6 | 7 | ## Public domain 8 | 9 | This project is in the public domain within the United States, and 10 | copyright and related rights in the work worldwide are waived through 11 | the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/). 12 | 13 | All contributions to this project will be released under the CC0 14 | dedication. By submitting a pull request, you are agreeing to comply 15 | with this waiver of copyright interest. 16 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | gem 'jekyll', '~> 3.1.0' 4 | 5 | group :jekyll_plugins do 6 | gem 'guides_style_18f' 7 | end 8 | -------------------------------------------------------------------------------- /Gemfile.lock: -------------------------------------------------------------------------------- 1 | GEM 2 | remote: https://rubygems.org/ 3 | specs: 4 | colorator (0.1) 5 | ffi (1.9.18) 6 | guides_style_18f (1.0.0) 7 | jekyll 8 | jekyll_pages_api 9 | jekyll_pages_api_search 10 | rouge 11 | sass 12 | htmlentities (4.3.4) 13 | jekyll (3.1.6) 14 | colorator (~> 0.1) 15 | jekyll-sass-converter (~> 1.0) 16 | jekyll-watch (~> 1.1) 17 | kramdown (~> 1.3) 18 | liquid (~> 3.0) 19 | mercenary (~> 0.3.3) 20 | rouge (~> 1.7) 21 | safe_yaml (~> 1.0) 22 | jekyll-sass-converter (1.5.0) 23 | sass (~> 3.4) 24 | jekyll-watch (1.5.0) 25 | listen (~> 3.0, < 3.1) 26 | jekyll_pages_api (0.1.6) 27 | htmlentities (~> 4.3) 28 | jekyll (>= 2.0, < 4.0) 29 | jekyll_pages_api_search (0.4.4) 30 | jekyll_pages_api (~> 0.1.4) 31 | sass (~> 3.4) 32 | kramdown (1.13.2) 33 | liquid (3.0.6) 34 | listen (3.0.8) 35 | rb-fsevent (~> 0.9, >= 0.9.4) 36 | rb-inotify (~> 0.9, >= 0.9.7) 37 | mercenary (0.3.6) 38 | rb-fsevent (0.9.8) 39 | rb-inotify (0.9.8) 40 | ffi (>= 0.5.0) 41 | rouge (1.11.1) 42 | safe_yaml (1.0.4) 43 | sass (3.4.23) 44 | 45 | PLATFORMS 46 | ruby 47 | 48 | DEPENDENCIES 49 | guides_style_18f 50 | jekyll (~> 3.1.0) 51 | 52 | BUNDLED WITH 53 | 1.13.6 54 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | As a work of the United States Government, this project is in the 2 | public domain within the United States. 3 | 4 | Additionally, we waive copyright and related rights in the work 5 | worldwide through the CC0 1.0 Universal public domain dedication. 6 | 7 | ## CC0 1.0 Universal Summary 8 | 9 | This is a human-readable summary of the [Legal Code (read the full text)](https://creativecommons.org/publicdomain/zero/1.0/legalcode). 10 | 11 | ### No Copyright 12 | 13 | The person who associated a work with this deed has dedicated the work to 14 | the public domain by waiving all of his or her rights to the work worldwide 15 | under copyright law, including all related and neighboring rights, to the 16 | extent allowed by law. 17 | 18 | You can copy, modify, distribute and perform the work, even for commercial 19 | purposes, all without asking permission. 20 | 21 | ### Other Information 22 | 23 | In no way are the patent or trademark rights of any person affected by CC0, 24 | nor are the rights that other persons may have in the work or in how the 25 | work is used, such as publicity or privacy rights. 26 | 27 | Unless expressly stated otherwise, the person who associated a work with 28 | this deed makes no warranties about the work, and disclaims liability for 29 | all uses of the work, to the fullest extent permitted by applicable law. 30 | When using or citing the work, you should not imply endorsement by the 31 | author or the affirmer. 32 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | permalink: / 4 | --- 5 | # Testing Cookbook 6 | 7 | The 18F testing cookbook contains recipes and best practices for automated and manual testing in lots of different environments, languages, stacks and platforms. Check 'em out: 8 | 9 | * [JavaScript](javascript/) testing in Node and the browser 10 | * [Python](python/) testing in pure Python, Django and Flask, etc. 11 | * [Ruby](ruby/) testing with RSpec, minitest, etc. 12 | * [Browser testing](browser/) and [VirtualBox](browser/virtualbox/) 13 | * [Load testing](load.md) with Locust and wrk 14 | 15 | 16 | ### Public domain 17 | 18 | This project is in the worldwide [public domain](LICENSE.md). As stated in [CONTRIBUTING](CONTRIBUTING.md): 19 | 20 | > This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/). 21 | > 22 | > All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest. 23 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | # Site settings 2 | title: 18F Testing Cookbook 3 | email: shawn.allen@gsa.gov 4 | description: > 5 | The 18F Testing Cookbook contains recipes and best practices for automated 6 | and manual testing in lots of different environments, languages, stacks and 7 | platforms. 8 | url: "http://pages.18f.gov/testing-cookbook/" 9 | github_username: 18F 10 | 11 | # Build settings 12 | exclude: 13 | - Gemfile 14 | - Gemfile.lock 15 | - go 16 | 17 | permalink: pretty 18 | highlighter: rouge 19 | 20 | sass: 21 | style: :compressed 22 | 23 | # DOCter bits after this 24 | 25 | name: Testing Cookbook 26 | subtitle: 27 | 28 | # Author/Organization info to be displayed in the templates 29 | author: 30 | name: 18F 31 | url: https://18f.gsa.gov/ 32 | 33 | # Point the logo URL at a file in your repo or hosted elsewhere by your organization 34 | logourl: /assets/img/18f-logo.png 35 | logoalt: 18F logo 36 | 37 | # Navigation 38 | # List links that should appear in the site sidebar here 39 | navigation: 40 | - text: Introduction 41 | url: index.html 42 | internal: true 43 | - text: JavaScript testing 44 | url: javascript/ 45 | internal: true 46 | children: 47 | - text: Mocha in Node 48 | url: node-mocha/ 49 | internal: true 50 | - text: JavaScript testing examples 51 | url: examples/ 52 | internal: true 53 | - text: Python testing 54 | url: python/ 55 | internal: true 56 | children: 57 | - text: Python testing with nose 58 | url: nose/ 59 | internal: true 60 | - text: Ruby testing 61 | url: ruby/ 62 | internal: true 63 | - text: Go testing 64 | url: go/ 65 | internal: true 66 | - text: Browser testing 67 | url: browser/ 68 | internal: true 69 | children: 70 | - text: Automation 71 | url: automation/ 72 | internal: true 73 | - text: Selenium and Sauce Labs 74 | url: selenium-sauce/ 75 | internal: true 76 | - text: Browser testing services 77 | url: services/ 78 | internal: true 79 | - text: VirtualBox 80 | url: virtualbox/ 81 | internal: true 82 | - text: Load testing 83 | url: load/ 84 | internal: true 85 | 86 | repos: 87 | - name: 18F Testing Cookbook 88 | description: Main repository 89 | url: https://github.com/18F/testing-cookbook 90 | 91 | google_analytics_ua: UA-48605964-19 92 | 93 | back_link: 94 | url: "https://pages.18f.gov/guides/" 95 | text: Read more 18F Guides 96 | 97 | defaults: 98 | - 99 | scope: 100 | path: "" 101 | values: 102 | layout: "guides_style_18f_default" 103 | -------------------------------------------------------------------------------- /_layouts/page.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | --- 4 |

{{ page.title }}

5 | 6 | {{ content }} 7 | -------------------------------------------------------------------------------- /_layouts/post.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | --- 4 |
5 |

{{ page.title }}

6 |

{{ page.date | date_to_string }}

7 | 8 |
9 | {{ content }} 10 |
11 |
-------------------------------------------------------------------------------- /assets/css/styles.scss: -------------------------------------------------------------------------------- 1 | --- 2 | --- 3 | 4 | @import "guides_style_18f"; 5 | -------------------------------------------------------------------------------- /go: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env ruby 2 | 3 | require 'English' 4 | 5 | Dir.chdir File.dirname(__FILE__) 6 | 7 | def try_command_and_restart(command) 8 | exit $CHILD_STATUS.exitstatus unless system command 9 | exec RbConfig.ruby, *[$PROGRAM_NAME].concat(ARGV) 10 | end 11 | 12 | begin 13 | require 'bundler/setup' if File.exist? 'Gemfile' 14 | rescue LoadError 15 | try_command_and_restart 'gem install bundler' 16 | rescue SystemExit 17 | try_command_and_restart 'bundle install' 18 | end 19 | 20 | begin 21 | require 'go_script' 22 | rescue LoadError 23 | try_command_and_restart 'gem install go_script' unless File.exist? 'Gemfile' 24 | abort "Please add \"gem 'go_script'\" to your Gemfile" 25 | end 26 | 27 | require 'guides_style_18f' 28 | 29 | extend GoScript 30 | check_ruby_version '2.1.5' 31 | 32 | command_group :dev, 'Development commands' 33 | 34 | def_command :update_nav, 'Update the \'navigation:\' data in _config.yml' do 35 | GuidesStyle18F.update_navigation_configuration Dir.pwd 36 | end 37 | 38 | def_command :update_theme, 'Update the guides_style_18f gem' do 39 | exec_cmd 'bundle update --source guides_style_18f' 40 | end 41 | 42 | def_command :update_gems, 'Update Ruby gems' do |gems| 43 | update_gems gems 44 | end 45 | 46 | def_command :serve, 'Serve the site at localhost:4000' do 47 | serve_jekyll 48 | end 49 | 50 | def_command :build, 'Build the site' do 51 | build_jekyll 52 | end 53 | 54 | execute_command ARGV 55 | -------------------------------------------------------------------------------- /images/crossing-the-chasm-rainbow-of-death.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/18F/testing-cookbook/0bf554abf2d2015f3fe06062493f620ed7aab797/images/crossing-the-chasm-rainbow-of-death.jpg -------------------------------------------------------------------------------- /images/rainbow-of-death.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/18F/testing-cookbook/0bf554abf2d2015f3fe06062493f620ed7aab797/images/rainbow-of-death.jpg -------------------------------------------------------------------------------- /images/sml.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/18F/testing-cookbook/0bf554abf2d2015f3fe06062493f620ed7aab797/images/sml.jpg -------------------------------------------------------------------------------- /images/testing-grouplet-rainbow-of-death.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/18F/testing-cookbook/0bf554abf2d2015f3fe06062493f620ed7aab797/images/testing-grouplet-rainbow-of-death.jpg -------------------------------------------------------------------------------- /images/the-chasm.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/18F/testing-cookbook/0bf554abf2d2015f3fe06062493f620ed7aab797/images/the-chasm.jpg -------------------------------------------------------------------------------- /index.html: -------------------------------------------------------------------------------- 1 | --- 2 | title: Introduction 3 | permalink: / 4 | --- 5 |
6 |

The 18F Testing Cookbook contains recipes and best practices for automated and manual testing in lots of different environments, languages, stacks and platforms.

7 |
8 | -------------------------------------------------------------------------------- /pages/browser/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Browser testing 3 | permalink: /browser/ 4 | --- 5 | Browser testing is [integration testing] for client-side functionality—that is, anything that happens in a web browser. Included in this directory: 6 | 7 | * The [automated browser testing guide](automation/) lists tools and techniques for browser automation. 8 | * For a more in-depth look at automated cross-browser testing, check out the [Selenium & Sauce guide](selenium-sauce/). 9 | * The [VirtualBox testing guide](virtualbox/) gets you up and running with [VirtualBox] for manual testing 10 | * The [hosted service guide](services/) lists resources for manual testing and screenshot-taking 11 | 12 | ## Manual Testing 13 | Manual browser testing involves simply opening up the URL of the site you're testing in one or more browsers and interacting with it. You can install as many browsers as you'd like on your computer, but at some point you will likely need to test other browers, platforms, and devices. See the [VirtualBox guide](virtualbox/) for setting up a cross-platform testing environment on your own computer, or peruse the [service guide](services/) for services that can help you interactively test and take screenshots of sites with lots of different browsers. 14 | 15 | ## Automated Testing 16 | Automated cross-browser testing involves writing tests and using tools to script the launching of one or more browsers and the execution of your tests in each of them. The automated browser testing stack typically consists of three distinct components: 17 | 18 | 1. A development server, e.g., a Django or Rails application running on your personal computer. 19 | 2. One or more web browsers (running on your computer natively, in a virtual environment like [VirtualBox], or remotely via a service such as [Sauce Labs]) that can be programmatically controlled. 20 | 3. One or more test suites that either: 21 | * run in the browser directly (like [QUnit]), or 22 | * control the browser remotely and test the results of scripted interactions, as in [Selenium]. 23 | 24 | Many web-based virtualization environments (namely [Sauce Labs]) have APIs of their own that allow us to script both the creation of virtual browsers to test our apps *and* determine the outcome of automated tests in each of them. Take a look at the [automated browser testing guide](automation/) for more info. 25 | 26 | [Selenium]: http://docs.seleniumhq.org/ 27 | [Sauce Labs]: https://saucelabs.com/ 28 | [PhantomJS]: http://phantomjs.org/ 29 | [Browserling]: https://browserling.com/ 30 | [integration testing]: http://en.wikipedia.org/wiki/Integration_testing 31 | [VirtualBox]: http://virtualbox.org/ 32 | [QUnit]: http://qunitjs.com/ 33 | -------------------------------------------------------------------------------- /pages/browser/VirtualBox.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: VirtualBox 3 | permalink: /browser/virtualbox/ 4 | parent: Browser testing 5 | --- 6 | ## Browser Testing with VirtualBox 7 | [VirtualBox] is a virtualization tool that allows you to run "guest" operating systems *within* your computer's native (or "host") OS. Some 18F projects use VirtualBox with [Vagrant] to create isolated development environments (namely, Linux) that more closely match their production environments. Thanks to Microsoft, we can also download canned Windows OS virtual machines and run them on our Macs to interactively test sites in Internet Explorer. 8 | 9 | > Installing VirtualBox and downloading the Windows VMs can be time consuming. If you don't have the time or you'd rather not do this, check out some [other testing services](../services/). 10 | 11 | ### 1. Download VirtualBox 12 | Visit [virtualbox.org](https://www.virtualbox.org/wiki/Downloads), find the **VirtualBox 4.X.XX for OS X hosts** item under **VirtualBox platform packages**, and click the **x86/amd64** link next to it: 13 | 14 | ![image](https://cloud.githubusercontent.com/assets/113896/6258476/9f8bddcc-b77d-11e4-80f8-2553f07482f4.png) 15 | 16 | 17 | ### 2. Download the Internet Explorer Virtual Machines 18 | There are two ways to do this: 19 | 20 | 1. Visit [modern.ie](https://www.modern.ie/en-us/virtualization-tools#downloads) and download each VM that you wish to test individually. After downloading a VM file, just double-click it in the Finder and VirtualBox should add it for you. 21 | 2. Use [ievms](https://github.com/xdissent/ievms) to download and install all of the IE VMs from the terminal in one fell swoop: 22 | 23 | ```sh 24 | curl -s https://raw.githubusercontent.com/xdissent/ievms/master/ievms.sh | bash 25 | ``` 26 | 27 | See the [ievms installation guide](https://github.com/xdissent/ievms#installation) for more information. 28 | 29 | In either case, you might want to download the VMs overnight because they're *big*, and even on fast connections they will likely take several hours to download. 30 | 31 | ### 3. Launch VirtualBox 32 | Use OS X's Spotlight CMD-Space or navigate to the Applications directory in the Finder by hitting CMD-SHIFT-G and entering `/Applications`, then open VirtualBox. You should see something like this: 33 | 34 | ![image](https://cloud.githubusercontent.com/assets/113896/6258517/0322672a-b77e-11e4-84c0-03e76340b2d1.png) 35 | 36 | The panel on the left-hand side lists your installed virtual machines. If you don't see anything there, you may not have installed your VMs correctly. Go back to [step 2](#download-vms) and try again. 37 | 38 | If you *do* see virtual machines listed, just double-click one to launch it. VirtualBox should prompt you to install extensions in the guest OS (the virtual Windows) the first time that it launches a new VM. Do this; it'll make working with the VMs easier. 39 | 40 | ### 4. Find Internet Explorer 41 | If all goes well, you should see the Windows boot screen in a separate window and, eventually, a mostly blank Windows desktop. Depending on the version of Windows that you're running, you should see a Start button in the lower left-hand corner. Click it, and you should see something like one of these (Windows XP on the left, Windows 7 on the right): 42 | 43 | ![image](https://cloud.githubusercontent.com/assets/113896/6258693/9d5d04e8-b77f-11e4-983f-1d118c912f43.png) ![image](https://cloud.githubusercontent.com/assets/113896/6258737/f70316fe-b77f-11e4-94ff-e2792aea0c0c.png) 44 | 45 | In XP (on the left) you can launch IE by clicking on the **Internet** link at the top; in Windows 7 you can either just type `internet` and hit RETURN in the *Search programs and files* input at the bottom, or navigate to **All Programs** > **Internet Explorer**. 46 | 47 | ### 5. Test Your Stuff 48 | If you're looking to test a public site, you should be able to just enter the URL into IE and test it interactively. To test sites running on your *host* OS, replace `localhost` or `127.0.0.1` in your local development URL with `10.0.2.2`. For instance, if you're testing the [18F site](https://github.com/18f/18f.gsa.gov) (or any Jekyll site serving on the default port 4000), you would use the URL `http://10.0.2.2:4000`. **Note**: If you leave off the `http://` prefix in Internet Explorer, it may send you to a Bing search. 49 | 50 | [VirtualBox]: http://virtualbox.org/ 51 | [Vagrant]: https://www.vagrantup.com/ 52 | -------------------------------------------------------------------------------- /pages/browser/automation.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Automation 3 | permalink: /browser/automation/ 4 | parent: Browser testing 5 | --- 6 | This is an overview of a selection of automated testing tools for frontend [integration tests]. See the [README](..) for background. 7 | 8 | ## Selenium 9 | [Selenium] is a suite of tools for launching, controlling browsers and testing a variety of browsers. The tools are: 10 | 11 | 1. [WebDriver] is the API by which you launch, control and test various web browsers. It has hooks for Android, Chrome, Firefox, Internet Explorer, [PhantomJS], Safari and remote servers (such as [Sauce Labs]). Implementations are available for JavaScript, Ruby, Python, Java and .Net. 12 | 2. [Selenium IDE] is a Firefox extension for recording, editing and debugging your tests. You can also write your tests yourself, of course. 13 | 14 | ### Selenium-based Tools 15 | * [Capybara] is a Ruby acceptance testing framework that uses Selenium under the hood. 16 | * [Nightwatch] is a Node.js "end-to-end" testing framework with a declarative API. 17 | * [Protractor] is an end-to-end test framework from the AngularJS team. It can be used to test both Angular and non-Angular applications. Protractor runs tests against your application running in a real browser, interacting with it as a user would. It is really quick and simple to get started. 18 | 19 | ## Sauce Labs 20 | [Sauce Labs] provides both [interactive](https://saucelabs.com/features/#features-manual-testing) (manual) and [automated](https://saucelabs.com/features/#features-cross-browser) remote cross-browser testing capabilities. 21 | 22 | ## Testling CI 23 | [Testling] is a tool from [Browserling] that allows you to run cross-browser tests whenever you push code to GitHub. You can [configure](https://saucelabs.com/features/#features-cross-browser) applications to be tested in any number of browser/version combinations, plus Safari on iPhone and iPad and the Android browser. 24 | 25 | ## Zuul 26 | [zuul] is similar to Testling, but uses [Sauce Labs] as the vehicle for tests and looks for its configuration in a [.zuul.yml file](https://github.com/defunctzombie/zuul/wiki/Zuul.yml) in your GitHub repository that specifies the tests, test UI (qunit, mocha-tdd, mocha-qunit, mocha-bdd, or tape), a list of browser/version combinations, and even the server to provide additional support (for instance, to run a Django or Rails app that exposes a REST API against which your JavaScript might test). For instance, [aight](https://github.com/shawnbot/aight/) uses zuul to run [tests](https://github.com/shawnbot/aight/blob/master/test/tests.js) in IE8 and IE9 like so: 27 | 28 | ```yaml 29 | browsers: 30 | - name: ie 31 | version: 8..9 32 | ``` 33 | 34 | ## PhantomJS 35 | [PhantomJS] is a standalone, headless web browser that combines the [WebKit] engine and the [Rhino] JavaScript interpreter. You can run your tests directly in PhantomJS, or you can use the [Selenium WebDriver] to script interactions with it, the benefit of which is that your tests will run completely from the command line and without any "live" browser running alongside your terminal. 36 | 37 | ### PhantomJS-based tools 38 | * [CasperJS] is a tool for navigating and [testing](http://docs.casperjs.org/en/latest/modules/tester.html) sites with [PhantomJS] and [SlimerJS], Phantom's Firefox equivalent. 39 | 40 | ## SlimerJS 41 | [SlimerJS] is PhantomJS but with Firefox's Gecko rendering engine under the hood, except that it doesn't run headlessly by default. 42 | 43 | ## jsdom 44 | [jsdom] is a *browser-like* DOM (Document Object Model) implementation for Node.js. You can use it to script pure-DOM interactions against JavaScript of your choosing if you don't need anything fancy like images, CSS, or AJAX. For example, to test a DOM JavaScript library with [mocha]: 45 | 46 | ```js 47 | var jsdom = require('jsdom'), 48 | assert = require('assert'); 49 | 50 | describe('library', function() { 51 | // before this suite, create a jsdom environment and stash 52 | // references to the window and document objects 53 | var window, 54 | document; 55 | 56 | before(function(done) { 57 | jsdom.env('path/to/test.html', ['path/to/library.js'], 58 | function(errors, win) { 59 | if (errors) throw errors; 60 | window = win; 61 | document = win.document; 62 | done(); 63 | }); 64 | }); 65 | 66 | it('doStuff()', function() { 67 | // global variables are accessible via the window object: 68 | assert.ok(window.library.doStuff(), 'it does not do stuff'); 69 | }); 70 | 71 | after(function() { 72 | // clean up memory 73 | window.close(); 74 | }); 75 | }); 76 | ``` 77 | 78 | 79 | [Selenium]: http://docs.seleniumhq.org/ 80 | [WebDriver]: http://docs.seleniumhq.org/projects/webdriver/ 81 | [Selenium WebDriver]: http://docs.seleniumhq.org/projects/webdriver/ 82 | [Selenium IDE]: http://docs.seleniumhq.org/projects/ide/ 83 | [Sauce Labs]: https://saucelabs.com/ 84 | [zuul]: https://github.com/defunctzombie/zuul 85 | [PhantomJS]: http://phantomjs.org/ 86 | [Browserling]: https://browserling.com/ 87 | [Testling]: https://ci.testling.com/ 88 | [integration tests]: http://en.wikipedia.org/wiki/Integration_testing 89 | [jsdom]: https://github.com/tmpvar/jsdom 90 | [mocha]: http://mochajs.org/ 91 | [Capybara]: https://github.com/jnicklas/capybara 92 | [CasperJS]: http://casperjs.org/ 93 | [Nightwatch]: http://nightwatchjs.org/ 94 | [SlimerJS]: http://slimerjs.org/ 95 | [VirtualBox]: http://virtualbox.org/ 96 | [WebKit]: http://en.wikipedia.org/wiki/WebKit 97 | [Rhino]: https://github.com/mozilla/rhino 98 | [Protractor]:(https://angular.github.io/protractor/#/) 99 | -------------------------------------------------------------------------------- /pages/browser/selenium+sauce.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Selenium and Sauce Labs 3 | permalink: /browser/selenium-sauce/ 4 | parent: Browser testing 5 | --- 6 | 7 | ## Testing with Selenium and Sauce Labs 8 | As we learned [earlier](../), the [Selenium WebDriver] API can be used to 9 | communicate with both *local* browsers (such as [PhantomJS], Chrome and Firefox) 10 | and *remote* browsers using [Selenium Remote Control]. [Sauce Labs] is a 11 | hosted service that runs "over 450 browser/OS combinations" in the cloud, all 12 | programmatically accessible via Selenium Remote Control. 13 | 14 | **Note**: The WebDriver API is implemented in pretty much every language that 15 | you might use for web development. It's perfectly acceptable to write your 16 | Selenium tests in a different language entirely, so long as the environments 17 | that you're going to run them in have access to both languages (for instance, 18 | if you're using [Travis CI], you can safely write your Selenium tests in Node 19 | because it's always available in that environment. 20 | 21 | With Selenium WebDriver you can run automated tests against a variety of 22 | configurations: 23 | 24 | ### Local/Local Testing 25 | This refers to tests run exclusively on your computer with Selenium and any 26 | WebDriver browser available on your computer. If you're on a Mac, then your 27 | options are: 28 | 29 | * Chrome with the [Selenium ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver) 30 | * Firefox with the [Selenium FirefoxDriver](https://code.google.com/p/selenium/wiki/FirefoxDriver) 31 | * Safari with the [Selenium SafariDriver](https://code.google.com/p/selenium/wiki/SafariDriver) 32 | * [PhantomJS], a headless WebKit browser (like Safari and, formerly, Chrome) with [ghostdriver](https://github.com/detro/ghostdriver) 33 | 34 | In this configuration, you point your tests at your web app running on 35 | `localhost`, and Selenium sends all of its commands directly to a browser 36 | on your computer. In the case of the Chrome, Firefox and Safari drivers, a 37 | full GUI is launched and you can watch the tests being run in realtime. 38 | PhantomJS is a "headless" browser, so you won't see anything on your screen, 39 | but you can take screenshots (using the Selenium API) of your app for review. 40 | 41 | ### Local/Remote Testing 42 | This configuration refers to tests run on Sauce (or [BrowserStack], or any 43 | other service that runs the [Selenium Remote Control] protocol). This is the 44 | configuration that you want if you need to run automated tests on your 45 | locally running web app against a browser you can't run locally, such as any 46 | version of Internet Explorer on Windows (assuming you're on a Mac). 47 | 48 | The technique that makes this possible is called *tunneling*, a process that 49 | creates a secure, direct connection between your computer and another service 50 | so that your web app can be accessed from the outside world. You have the 51 | following options: 52 | 53 | #### Sauce Connect 54 | [Sauce Connect] creates a secure connection between your computer and Sauce 55 | Labs, effectively giving any browser started with your Sauce Labs credentials 56 | access to `localhost` as though it were on your computer. This includes manual 57 | testing sessions, which makes it a good option if you don't have the time or 58 | patience to set up [VirtualBox](../VirtualBox/). 59 | 60 | One nice thing about Sauce Connect is that [Travis CI] has 61 | [support built in](http://docs.travis-ci.com/user/sauce-connect/), which (in 62 | theory) should make running remote tests as part of your continuous integration 63 | relatively straightforward. The `sc` utility also does a good job of reopening 64 | connections when they close, for instance if your network connection hiccups. 65 | 66 | **Note**: Sauce Connect does have its issues. For instance, their 67 | [Python library](https://github.com/saucelabs/python-sauceconnect) is 68 | basically abandoned, so you'll need to [download](https://docs.saucelabs.com/reference/sauce-connect/) 69 | the appropriate binary for your system and run it manually within your 70 | development environment. 71 | 72 | #### Use localtunnel 73 | [localtunnel] is a command-line utility that creates a unique, public URL 74 | for a specific port on your computer. For instance, if your web app is 75 | running at `localhost:8000` (on port 8000), you can just run `lt`: 76 | 77 | ```sh 78 | lt --port 8000 79 | ``` 80 | 81 | and it'll print a unique URL that you can copy and paste into your browser, 82 | send to collaborators in Slack, or use in a manual browser testing session on 83 | Sauce Labs. One really nice fringe benefit of localtunnel is that **your URL 84 | is accessible over HTTPS**, which means that you get to test your web app 85 | under conditions more similar to their production environment. 86 | 87 | #### Use localtunnel-runner 88 | Both Sauce Connect and localtunnel require you to run a command in a separate 89 | shell, separately from the process that starts your web app. This means that 90 | you have to keep two shells open, for instance, in one shell: 91 | 92 | ```sh 93 | path/to/sc -u username -k access-key 94 | # or 95 | lt --port 1337 96 | ``` 97 | 98 | and in the second: 99 | 100 | ```sh 101 | # start a Node app 102 | npm start 103 | # or Django 104 | ./manage.py runserver 105 | ``` 106 | 107 | [localtunnel-runner](https://github.com/shawnbot/localtunnel-runner) solves 108 | this particular problem with a command line tool that does both in one go: 109 | 110 | ```sh 111 | # e.g., running a Django server 112 | lt-run --port 8000 -- ./manage.py runserver 113 | ``` 114 | 115 | The thing to note here is that your unique localtunnel.me URL is available 116 | to the web server as an environment variable, so your Selenium tests can 117 | detect the presence of `LOCAL_TUNNEL_URL` and decide to run them against 118 | this URL, either locally or over Selenium Remote Control. 119 | 120 | 121 | ### Remote/Remote Testing 122 | This is really just a variation on "local/remote" testing in which the web 123 | app is running somewhere else: for instance, in the staging or production 124 | environments, or on a CI service. This boils down to changing the URLs that 125 | are loaded by [Selenium WebDriver] in your test code. 126 | 127 | 128 | ### Cross-Browser Testing 129 | If you're running JavaScript *unit tests* in the browser, then there are 130 | some nice tools at your disposal: 131 | 132 | * [Karma] runs your unit tests whenever your code changes (which can be good or bad), 133 | and has a [Sauce Launcher](https://github.com/karma-runner/karma-sauce-launcher). 134 | * [zuul] gives you Sauce tests on multiple browsers from 135 | [a single configuration file](https://github.com/defunctzombie/zuul/wiki/cloud-testing). 136 | 137 | If you're running *functional* tests, though, then you've got some work to do. 138 | Your plan should look like this: 139 | 140 | 1. Get your Selenium tests working on one browser, locally. Start with Chrome or Firefox, 141 | then test PhantomJS. (PhantomJS tests will generally run more quickly since they don't 142 | need a GUI.) 143 | 2. Set up your [Sauce Labs] account if you haven't already. 144 | 3. Use [Selenium Remote Control] in the WebDriver API of your choice to run the tests 145 | on Sauce's infrastructure using one of the "local/remote" configurations above. 146 | *Start with a browser you know* to avoid browser-specific failures, just so you can 147 | be sure that things are working as you expect them to. 148 | 4. Next, update your WebDriver's [desired capabilities](https://docs.saucelabs.com/reference/platforms-configurator/) 149 | so that Sauce runs a less predictable browser, such as IE9. 150 | 5. Fix some bugs. If your tests pass the first time, then you're either *amazing* or 151 | your tests aren't covering enough of your application. 152 | 6. Refactor your tests to run on a list of hard-coded WebDriver instances. For instance: 153 | * Chrome on Mac OS 154 | * Firefox on Windows 155 | * Internet Explorer 9 (Windows 7) 156 | * Mobile Safari on iOS 8 157 | * Android 158 | 7. Move your WebDriver list specs to a configuration file, so you can more easily 159 | add or remove testing configurations. 160 | 8. Run your tests some more. 161 | 9. :tada: 162 | 163 | [localtunnel]: https://localtunnel.me 164 | [BrowserStack]: https://www.browserstack.com/ 165 | [PhantomJS]: http://phantomjs.org 166 | [Sauce Labs]: https://saucelabs.com 167 | [Sauce Connect]: https://docs.saucelabs.com/reference/sauce-connect/ 168 | [Selenium WebDriver]: http://www.seleniumhq.org/projects/webdriver/ 169 | [Selenium Remote Control]: http://www.seleniumhq.org/projects/remote-control/ 170 | [Travis CI]: https://travis-ci.org/ 171 | [zuul]: https://github.com/defunctzombie/zuul/ 172 | [Karma]: http://karma-runner.github.io/0.12/index.html 173 | -------------------------------------------------------------------------------- /pages/browser/services.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Browser testing services 3 | permalink: /browser/services/ 4 | parent: Browser testing 5 | --- 6 | These services allow you to run virtual browsers right in your browser: 7 | 8 | * [Browserling] is the original live cross-browser testing service. The [free plan](https://browserling.com/#pricing) gives you access to IE9 in 3-minute sessions at 1024x768. All other browsers are free. 9 | * [BrowserStack] features an [exhaustive platform list](http://www.browserstack.com/list-of-browsers-and-platforms?product=live) but [lacks a free plan](https://www.browserstack.com/pricing). 10 | * [Cross-Browser Testing] supports [tons of different platforms](https://crossbrowsertesting.com/browsers), but [lacks a free plan](https://crossbrowsertesting.com/pricing). 11 | * [Sauce Labs] also [requires an account](https://saucelabs.com/pricing), but gives you [tons of platforms](https://saucelabs.com/platforms/) and provides a native application, which tends to be marginally faster than the in-browser environment. It's also [free for open source projects](https://saucelabs.com/opensauce/). 12 | * [TestingBot] has a [free trial plan](http://testingbot.com/pricing) and supports [lots of different platforms](http://testingbot.com/support/getting-started/browsers.html). 13 | 14 | ## Screenshot Services 15 | These services can be used to take screenshots of public sites: 16 | 17 | * [Browsera] visually analyzes the layout differences between browsers. Its [free plan is very limited](https://www.browsera.com/plans). 18 | * [Browser Shots] takes screenshots with lots of different browsers at once and presents them to you on one page for comparison. **No IE support, unfortunately.** 19 | * [IE NetRenderer] takes screenshots with different versions of IE (5.5, 6, 7, 8, 9, 10, and 11 as of this writing). 20 | 21 | ## Other Testing Tools 22 | 23 | * [Ghostlab] *looks* very cool (especially for mobile testing), but it does not appear to run virtual machines. 24 | * [Spoon's](https://spoon.net) [Browser Sandbox] has [a free plan for open source](https://spoon.net/pricing) (for automated testing only), but it doesn't include legacy OS support. It's also Windows-only. 25 | 26 | [Browsera]: https://www.browsera.com/ 27 | [Browserling]: https://browserling.com/ 28 | [BrowserStack]: http://www.browserstack.com/ 29 | [Browser Sandbox]: https://spoon.net/browsers/ 30 | [Browser Shots]: http://browsershots.org/ 31 | [Cross-Browser Testing]: https://crossbrowsertesting.com/browsers 32 | [Ghostlab]: http://vanamco.com/ghostlab/ 33 | [IE NetRenderer]: http://netrenderer.com/ 34 | [Sauce Labs]: https://saucelabs.com/ 35 | [TestingBot]: http://testingbot.com/ 36 | -------------------------------------------------------------------------------- /pages/go/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Go testing 3 | permalink: /go/ 4 | --- 5 | 6 | ## The `testing` package 7 | 8 | ### What Is It? 9 | Go's [testing](https://golang.org/pkg/testing/) package provides automated testing out-of-the-box. 10 | 11 | ### When to Use It? 12 | Great for: 13 | 14 | 1. unit testing 15 | 2. benchmarks (i.e. testing how long a certain section of code runs) 16 | 17 | ### How to Use It? 18 | 19 | Here's an example of writing a unit test 20 | 21 | #### Sample source file 22 | Given a source code file `hello.go` which just prints out "Hello" twice via a function `SayTwice` upon execution: 23 | 24 | ```go 25 | package main 26 | 27 | import ( 28 | "fmt" 29 | ) 30 | 31 | func SayTwice(text string) string { 32 | return text + text 33 | } 34 | 35 | func main() { 36 | fmt.Println(SayTwice("Hello")) 37 | } 38 | ``` 39 | 40 | #### Creating unit tests. 41 | 1. Create a file called `hello_test.go` (all Go test files must have the suffix `_test.go`) 42 | 2. Import `testing` 43 | 3. Create a function in which the name begins with `Test` and takes in `*testing.T` (used by the test runner) 44 | 1. Tabular tests are **only** good for pure functions like this one. (**Refer to this [link](https://pages.18f.gov/automated-testing-playbook/principles-practices-idioms/#avoid-data-driven-tests) as to why you shouldn't use the tabular testing pattern otherwise**) 45 | 2. For our tabular test, create a simple struct that will hold simple string input and output variables. 46 | 3. Create the various test cases to be tested. 47 | 4. Loop through the tests. 48 | 4. Use [Error](https://golang.org/pkg/testing/#T.Error), [Fail](https://golang.org/pkg/testing/#T.Fail) or [Fatal](https://golang.org/pkg/testing/#T.Fatal) to indicate that a test has failed. 49 | 50 | (_Refer to the numbering above to the comments in the code below_) 51 | 52 | ```go 53 | package main 54 | 55 | import ( 56 | "testing" // 2 57 | ) 58 | 59 | // 3.2 60 | type sayTwiceTest struct { 61 | input string 62 | expectedOutput string 63 | } 64 | 65 | // 3.3 66 | var sayTwiceTests = []sayTwiceTest{ 67 | {"Hello", "HelloHello"}, 68 | {"Bye", "ByeBye"}, 69 | } 70 | 71 | func TestSayTwice(t *testing.T) { 72 | for _, test := range sayTwiceTests { // 3.4 73 | if output := SayTwice(test.input); output != test.expectedOutput { 74 | // 4 75 | t.Errorf("Expected SayTwice to return (%s), Found (%s)\n", test.expectedOutput, output) 76 | } 77 | } 78 | } 79 | ``` 80 | 81 | ## `agouti` + `ginkgo` + `gomega` stack 82 | 83 | 84 | ### What Is It? 85 | * [`Ginkgo`](http://onsi.github.io/ginkgo/) is a BDD testing framework 86 | * [`Gomega`](http://onsi.github.io/gomega/) is the preferred matcher library for Ginkgo (although you can use any matcher library with Ginkgo) 87 | * [`Agouti`](http://agouti.org/) is a webdriver client (Selenium, PhantomJS, SauceLabs, etc) 88 | 89 | When `Agouti`, `Ginkgo`, and `Gomega` are combined, you have an acceptance testing framework for web apps. 90 | 91 | ### When to Use It? 92 | Great for writing expressive, human readable (for other technical people) acceptance / integration tests for a web app. 93 | 94 | ### How to Use It? 95 | `ginkgo` can generate tests for agouti. 96 | 97 | An example from the `agouti` website shows how one might test user login. 98 | 99 | ```go 100 | 101 | package potato_test 102 | 103 | import ( 104 | . "path/to/potato" 105 | . "github.com/onsi/ginkgo" 106 | . "github.com/onsi/gomega" 107 | . "github.com/sclevine/agouti/matchers" 108 | "github.com/sclevine/agouti" 109 | ) 110 | 111 | var _ = Describe("UserLogin", func() { 112 | var page *agouti.Page 113 | 114 | BeforeEach(func() { 115 | StartMyApp(3000) 116 | 117 | var err error 118 | page, err = agoutiDriver.NewPage(agouti.Browser("firefox")) 119 | Expect(err).NotTo(HaveOccurred()) 120 | }) 121 | 122 | AfterEach(func() { 123 | Expect(page.Destroy()).To(Succeed()) 124 | }) 125 | 126 | It("should manage user authentication", func() { 127 | By("redirecting the user to the login form from the home page", func() { 128 | Expect(page.Navigate("http://localhost:3000")).To(Succeed()) 129 | Expect(page).To(HaveURL("http://localhost:3000/login")) 130 | }) 131 | 132 | By("allowing the user to fill out the login form and submit it", func() { 133 | Eventually(page.FindByLabel("E-mail")).Should(BeFound()) 134 | Expect(page.FindByLabel("E-mail").Fill("spud@example.com")).To(Succeed()) 135 | Expect(page.FindByLabel("Password").Fill("secret-password")).To(Succeed()) 136 | Expect(page.Find("#remember_me").Check()).To(Succeed()) 137 | Expect(page.Find("#login_form").Submit()).To(Succeed()) 138 | }) 139 | 140 | By("allowing the user to view their profile", func() { 141 | Eventually(page.FindByLink("Profile Page")).Should(BeFound()) 142 | Expect(page.FindByLink("Profile Page").Click()).To(Succeed()) 143 | profile := page.Find("section.profile") 144 | Eventually(profile.Find(".greeting")).Should(HaveText("Hello Spud!")) 145 | Expect(profile.Find("img#profile_pic")).To(BeVisible()) 146 | }) 147 | 148 | By("allowing the user to log out", func() { 149 | Expect(page.Find("#logout").Click()).To(Succeed()) 150 | Expect(page).To(HavePopupText("Are you sure?")) 151 | Expect(page.ConfirmPopup()).To(Succeed()) 152 | Eventually(page).Should(HaveTitle("Login")) 153 | }) 154 | }) 155 | }) 156 | ``` 157 | -------------------------------------------------------------------------------- /pages/javascript/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: JavaScript testing 3 | permalink: /javascript/ 4 | --- 5 | ### Mocha 6 | [Mocha] is a slick, simple testing framework for Node and browser JavaScript with asynchronous support and a [slew of reporters](http://mochajs.org/#reporters), including [TAP] output. See the [node-mocha example](examples/node-mocha) for usage. 7 | 8 | #### Resources 9 | * Check out [Testling's mocha guide](https://ci.testling.com/guide/mocha) for running your [cross-browser tests](../browser/) with mocha. 10 | 11 | ### Jasmine 12 | [Jasmine] shares basic semantics with [Mocha], but also includes lots of nice assertion helpers, object [spies](http://jasmine.github.io/2.2/introduction.html#section-Spies), and async support. 13 | 14 | #### Resources 15 | * See [the docs](http://jasmine.github.io/2.2/node.html) for Node usage. 16 | * There's also a [Python package](http://jasmine.github.io/2.2/python_egg.html) and a [Ruby gem](http://jasmine.github.io/2.2/ruby_gem.html). 17 | 18 | ### Tape 19 | [Tape] is a [TAP]-producing test harness for Node with great asynchronous support. 20 | 21 | #### Resources 22 | * Check out the [Testling tape guide](https://ci.testling.com/guide/tape) for running your [cross-browser tests](../browser/) with tape. 23 | 24 | ### QUnit 25 | [QUnit] is a unit testing framework for Node and the browser developed by and for the jQuery team. It has a nice-looking browser test runner and lots of [plugins](http://qunitjs.com/plugins/) for more specific needs. 26 | 27 | [QUnit]: http://qunitjs.com/ 28 | [Mocha]: http://mochajs.org 29 | [TAP]: http://en.wikipedia.org/wiki/Test_Anything_Protocol 30 | [Tape]: https://www.npmjs.com/package/tape 31 | [Jasmine]: https://github.com/jasmine/jasmine 32 | -------------------------------------------------------------------------------- /pages/javascript/examples/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: JavaScript testing examples 3 | permalink: /javascript/examples/ 4 | parent: JavaScript testing 5 | --- 6 | Each of these directories is a self-contained example of automated JavaScript tests using a different tool or technique. See [the parent directory](../) for more on JavaScript testing. 7 | -------------------------------------------------------------------------------- /pages/javascript/examples/node-mocha/.gitignore: -------------------------------------------------------------------------------- 1 | # ignore npm dependencies 2 | node_modules 3 | -------------------------------------------------------------------------------- /pages/javascript/examples/node-mocha/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Mocha in Node 3 | permalink: /javascript/node-mocha/ 4 | parent: JavaScript testing 5 | --- 6 | This directory provides a simple example of writing unit tests for [Mocha](http://mochajs.com) in [Node](https://nodejs.org/). 7 | 8 | Here's what we did: 9 | 10 | 1. The API that we want to test, a simple JavaScript class called `Gadget`, is defined in `gadget.js`. 11 | 2. Mocha looks for tests in `test/*.js`, so we wrote our unit tests in `test/index.js`. 12 | 3. We ran `npm init` to create a `package.json` with the default values. 13 | 4. We added `mocha` to the development dependencies with `npm install --save-dev mocha`. 14 | 5. We update `package.json` and change the `scripts.test` value to simply `mocha`. 15 | 16 | To execute the tests, you'll need to have a clone of the repo, then `cd` to this directory and run: 17 | 18 | ```sh 19 | npm install --dev 20 | ``` 21 | 22 | This will install the development dependencies (just `mocha` in this case) into the `node_modules` directory locally (which, thanks to our `.gitignore`, will be ignored by git). Then you can run the tests with: 23 | 24 | ```sh 25 | npm test 26 | ``` 27 | 28 | Behind the scenes, npm checks `package.json` for `scripts.test` and runs that command. Thankfully, npm updates its `$PATH` to include `./node_modules/.bin`, which is where each installed dependency's `bin` scripts are aliased. 29 | -------------------------------------------------------------------------------- /pages/javascript/examples/node-mocha/gadget.js: -------------------------------------------------------------------------------- 1 | var Gadget = function(speed) { 2 | this.speed = speed; 3 | }; 4 | 5 | Gadget.prototype.slice = function() { 6 | return true; 7 | }; 8 | 9 | Gadget.prototype.dice = function() { 10 | return true; 11 | }; 12 | 13 | Gadget.prototype.blend = function(what) { 14 | if (what === 'iPhone') return this.speed > 100; 15 | return true; 16 | }; 17 | 18 | module.exports = Gadget; 19 | -------------------------------------------------------------------------------- /pages/javascript/examples/node-mocha/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "node-mocha", 3 | "version": "1.0.0", 4 | "description": "This directory provides a simple example of writing unit tests for [Mocha](http://mochajs.com) in [Node](https://nodejs.org/).", 5 | "main": "gadget.js", 6 | "directories": { 7 | "test": "test" 8 | }, 9 | "scripts": { 10 | "test": "mocha" 11 | }, 12 | "author": "", 13 | "license": "ISC", 14 | "devDependencies": { 15 | "mocha": "^2.2.1" 16 | } 17 | } 18 | -------------------------------------------------------------------------------- /pages/javascript/examples/node-mocha/test/index.js: -------------------------------------------------------------------------------- 1 | // require the parent module, in this case a class constructor 2 | var Gadget = require('../gadget'); 3 | // Node's assert module is a built-in 4 | var assert = require('assert'); 5 | 6 | // this describes the context of your unit tests 7 | describe('Gadget', function() { 8 | 9 | it('it slices', function() { 10 | var gadget = new Gadget(); 11 | assert.ok(gadget.slice(), 'it does not slice'); 12 | }); 13 | 14 | it('it dices', function() { 15 | var gadget = new Gadget(); 16 | assert.ok(gadget.dice(), 'it does not dice'); 17 | }); 18 | 19 | it('but does it blend?', function() { 20 | var gadget = new Gadget(200); 21 | assert.ok(gadget.blend('iPhone')); 22 | }); 23 | 24 | }); 25 | -------------------------------------------------------------------------------- /pages/load.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Load testing 3 | permalink: /load/ 4 | --- 5 | Load testing simulates the performance of an application under (typically concurrent) use. Load testing can be useful to estimate resources required for an application to perform well under varying levels of activity. Simple load tests may send requests to a set of URLs at fixed intervals; more realistic tests may simulate sequences of user behavior (load homepage, log in, upload a file, log out). 6 | 7 | ## Locust.io 8 | Locust is an open-source load testing tool written in Python 2 (Note: because Locust depends on the gevent library, it doesn't work under Python 3). Locust provides both a interactive graphical interface through the browser and a command-line interface, and can be run with multiple slave nodes for highly concurrent testing. Locust configuration files are Python programs that describe the behavior of simulated users. From the [docs](locust.io): 9 | 10 | from locust import HttpLocust, TaskSet, task 11 | 12 | class WebsiteTasks(TaskSet): 13 | def on_start(self): 14 | self.client.post("/login", { 15 | "username": "test_user", 16 | "password": "" 17 | }) 18 | 19 | @task 20 | def index(self): 21 | self.client.get("/") 22 | 23 | @task 24 | def about(self): 25 | self.client.get("/about/") 26 | 27 | class WebsiteUser(HttpLocust): 28 | task_set = WebsiteTasks 29 | min_wait = 5000 30 | max_wait = 15000 31 | 32 | Locust is currently used by the OpenFEC team; for an example configuration file, see https://github.com/18F/openFEC/blob/develop/locustfile.py. 33 | 34 | ## wrk 35 | 36 | [wrk](https://github.com/wg/wrk) is a lightweight benchmarking tool capable of high concurrency on a single core. From the docs: 37 | 38 | Basic Usage 39 | 40 | wrk -t12 -c400 -d30s http://127.0.0.1:8080/index.html 41 | 42 | This runs a benchmark for 30 seconds, using 12 threads, and keeping 43 | 400 HTTP connections open. 44 | 45 | Output: 46 | 47 | Running 30s test @ http://127.0.0.1:8080/index.html 48 | 12 threads and 400 connections 49 | Thread Stats Avg Stdev Max +/- Stdev 50 | Latency 635.91us 0.89ms 12.92ms 93.69% 51 | Req/Sec 56.20k 8.07k 62.00k 86.54% 52 | 22464657 requests in 30.00s, 17.76GB read 53 | Requests/sec: 748868.53 54 | Transfer/sec: 606.33MB 55 | 56 | ## More options 57 | * Commercial solutions 58 | * [BlazeMeter](https://blazemeter.com) 59 | * [Blitz](https://blitz.io) 60 | * [Loader](https://loader.io) 61 | * [List on Wikipedia](https://en.wikipedia.org/wiki/Web_server_benchmarking#Tools_for_benchmarking) 62 | -------------------------------------------------------------------------------- /pages/python/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Python testing 3 | permalink: /python/ 4 | --- 5 | 6 | # Test runners 7 | 8 | ## pytest 9 | [pytest] is a mature testing framework that emphasizes flexible, no-boilerplate usage. Unlike other test runners, pytest doesn't require special syntax for assertions: `assert 'pytest' in runners` instead of `self.assertIn('pytest', runners)`. Tests can be written as xUnit-style classes or as functions. Pytest can run tests written using nose, unittest, and doctest. 10 | 11 | ## nose 12 | [nose] is a test runner that extends the built-in unittest library by adding more powerful test discovery and execution. Nose has an extensive set of build-in and third-party plugins, including support for test coverage, parallel test execution, and test debugging. 13 | 14 | Read on for [tips and examples](nose/). 15 | 16 | ## unittest 17 | [unittest] is an [xUnit]-style test framework included in the python standard library. Tests are structured into test suites (represented by python classes) containing test cases (represented by python methods); assertions are provided as methods. Tests written with unittest are typically run with a more powerful test runner, such as pytest or nose. 18 | 19 | ## doctest 20 | [doctest] identifies text that looks like python code in docstrings and verifies that these code examples run as expected. Doctest can be useful for prevent regressions in inline documentation, but isn't a full-fledged testing framework and shouldn't be used as such. Doctests can be run using the pytest and nose test runners. 21 | 22 | # Testing tools 23 | 24 | * [tox](https://tox.readthedocs.org/en/latest/): Run tests using different python versions 25 | * [coverage.py](https://coverage.readthedocs.org/en/latest/): Measure test coverage; typically called by test runner plugins, not invoked directly 26 | * [mock](https://docs.python.org/3/library/unittest.mock.html): Replace code with mock objects and make assertions about how they have been used 27 | * [HTTPretty](http://falcao.it/HTTPretty/): Mock HTTP interactions; useful for testing code that interacts with remote APIs 28 | * [factory_boy](https://factoryboy.readthedocs.org/en/latest/): Create database fixtures for SQLAlchemy or Django models 29 | 30 | # Testing resources 31 | 32 | * http://docs.python-guide.org/en/latest/writing/tests/ 33 | * http://pythontesting.net/test-podcast/ 34 | 35 | [pytest]: https://pytest.org/latest/ 36 | [nose]: https://nose.readthedocs.org/en/latest/ 37 | [doctest]: https://docs.python.org/3.4/library/doctest.html 38 | [unittest]: https://docs.python.org/3.4/library/unittest.html 39 | [xUnit]: https://en.wikipedia.org/wiki/XUnit 40 | -------------------------------------------------------------------------------- /pages/python/nose.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Python testing with nose 3 | permalink: /python/nose/ 4 | --- 5 | 6 | # nose 7 | [nose] is a Python test runner which has a whole host of features, including plugins, which make testing in Python go a whole lot smoother, regardless 8 | of what module(s) you've used to write your tests. 9 | 10 | ## What It Is 11 | *nose* is a test runner, which means that it does all of the following categories of things: 12 | 13 | * discovers tests amongst your code; 14 | * lets you define and choose subsets of tests to run; 15 | * runs tests; 16 | * runs coverage; 17 | * and controls test output. 18 | 19 | ## When to Use It 20 | It's a great idea to use nose from the very start, and it's not hard to start using. However, there are some specific cases where nose can really help out with existing projects. 21 | 22 | ### Assertions 23 | Nose includes assertion methods with PEP8-compliant names (`assert_equal`, not `assertEqual`): 24 | 25 | ```python 26 | from nose.tools import * 27 | 28 | def my_test(): 29 | assert_equal(42, 42) 30 | assert_not_in('camelCase', inflections) 31 | with assert_raises(AttributeError): 32 | object.foo 33 | ``` 34 | 35 | ### Fail Fast 36 | `nosetests -x` 37 | 38 | Use nose's "fail fast" feature when you've got a lot of test failures, but are just trying to see and fix the first failed test. Run that test quickly, fix, and repeat, moving on to the next. 39 | 40 | ### Capture/Suppress STDOUT/Logs 41 | By default, Nose captures STDOUT and logging output. This means that when you have test failures, it will capture STDOUT and logging output for each failed test and print it 42 | along with the failure message and stack trace, rather than letting STDOUT go to the console at runtime and get mixed up with everything else that might be going to STDOUT. 43 | Outputs become much more useful when they are split up by their test of origin. 44 | 45 | Note: Capturing STDOUT breaks debugging tools like pdb, ipdb, etc., and can be disabled using the `--nocapture` or `-s` flag. 46 | 47 | ## How to Use It 48 | In projects without their own custom test runner, it's just a matter of running `pip install nose` and then invoking `nosetests` from the command line. 49 | 50 | In the case of Django, which has its own custom test runner, you'll need to do a little more work (but just a little). TODO: link to django-nose 51 | 52 | [nose]: https://nose.readthedocs.org/en/latest/ 53 | -------------------------------------------------------------------------------- /pages/ruby/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Ruby testing 3 | permalink: /ruby/ 4 | --- 5 | ## RSpec 6 | RSpec is a behavior-driven development (BDD) framework for the Ruby programming language. 7 | - [Official Page](http://rspec.info/) 8 | - [RelishApp Docs](https://relishapp.com/rspec) 9 | - [Github](https://github.com/rspec/rspec) 10 | 11 | ## Capybara 12 | A library that makes it easy to simulate how a user interacts with your application. 13 | - [Github Pages Page](https://jnicklas.github.io/capybara/) 14 | 15 | ## Cucumber 16 | A "story runner" with a plaintext DSL. 17 | - [Official Page](https://cukes.info/) 18 | - [Github Wiki](https://github.com/cucumber/cucumber/wiki) 19 | 20 | ## Nyan Cat Formatter 21 | An RSpec output formatter featuring the Nyan Cat. 22 | - [Github](https://github.com/mattsears/nyan-cat-formatter) 23 | - [The Original](https://www.youtube.com/watch?v=QH2-TGUlwu4) 24 | 25 | ## minitest 26 | Something about [minitest]. 27 | 28 | 29 | [RSpec]: http://rspec.info/ 30 | [minitest]: https://github.com/seattlerb/minitest 31 | --------------------------------------------------------------------------------