├── .gitignore ├── Makefile ├── README.md ├── client └── card │ ├── card.jsx │ └── visualtests │ ├── ltr-1.jsx │ ├── ltr-2.jsx │ ├── rtl-1.jsx │ └── rtl-2.jsx ├── css-visual-test ├── Makefile ├── SupportedSeleniumEnvironments.js ├── component-library-generator.js ├── component-library-server.js ├── lib │ ├── ComponentLibraryGenerator.js │ ├── Config.js │ ├── ManifestBuilder.js │ ├── ParallelTestPartitioner.js │ └── ParallelTestRunner.js ├── run-visual-tests.js └── test │ ├── ParallelTestPartitionerTests.js │ └── data │ ├── FifteenBucketsWithOneTestInEachBucket.js │ ├── FiveTestsAcrossTwoDirectoriesInputData.js │ ├── FourTestsFromOneDirectoryInputData.js │ ├── SevenBucketsWithAVaryingNumberOfTestsInEachBucket.js │ ├── ThreeBrowsersSupportedSeleniumEnvironmentsStub.js │ └── ThreeBucketsWithFourTestsInEachBucket.js ├── package.json ├── public ├── style-rtl.css └── style.css └── server └── pages └── index.jade /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | .DS_Store 3 | .idea 4 | node_modules 5 | /npm-debug.log 6 | 7 | css-visual-test/component-library/ 8 | css-visual-test/ApplitoolsEyesParallelDebug.log 9 | css-visual-test/lib/PrivateConfig.js 10 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # applications 2 | NODE ?= node 3 | NPM ?= $(NODE) $(shell which npm) 4 | 5 | # The `install` task is the default rule in the Makefile. 6 | install: node_modules 7 | 8 | # ensures that the `node_modules` directory is installed and up-to-date with 9 | # the dependencies listed in the "package.json" file. 10 | node_modules: package.json 11 | @$(NPM) prune 12 | @$(NPM) install 13 | @touch node_modules 14 | 15 | project-unit-test: install 16 | node_modules/.bin/mocha css-visual-test/test 17 | 18 | # run the component library server, building the public app first 19 | componentlibrary: install 20 | @$(NODE) css-visual-test/component-library-generator.js 21 | @$(NODE) css-visual-test/component-library-server.js 22 | 23 | # run the css visual test tool, building the public app first 24 | visualtest: install 25 | @$(NODE) css-visual-test/component-library-server.js & 26 | @$(NODE) css-visual-test/run-visual-tests.js 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Css Visual Test 2 | =============== 3 | 4 | Overall Goal 5 | ------------ 6 | 7 | How do we know that if css changes for all the components comprising an application, that the application looks as expected for every state mutation the application supports? 8 | 9 | Usage 10 | ----- 11 | 12 | `make componentlibrary` 13 | 14 | This will run a build, then generate the component library and a server to view it on port 4000 to be used for visual testing, manual inspection to understand the codebase, and/or debugging creation of visual tests. Running node css-visual-test/component-library-server.js will regenerate the component library if needed as well if you change something. 15 | 16 | `make visualtest` 17 | 18 | This will run all the visual tests using Applitools Eyes for image comparison, and Sauce Labs for browser automation. If differences are noted between runs or new visual tests are added, a failed test is reported. When tests fail, make returns an Error 1 code, which will fail the build. This will run the componentlibrary task along with the component library server if it's not already running before it runs. 19 | 20 | You'll need to create an account at both Sauce Labs and Applitools to use this tool. They both have free trials. 21 | 22 | [Sauce Labs Signup Page](http://saucelabs.com/signup) 23 | 24 | [Applitools Signup Page](https://applitools.com/sign-up) 25 | 26 | From the Applitools UI you can approve a new test as a baseline, or see any screenshots marked as failures. You can also change the algorithm used for image comparison, mark areas of the viewport to ignore from the screenshot, and apply changes to the configuration in batches. 27 | 28 | Open Files Limit 29 | ---------------- 30 | 31 | It is recommended to increase your open files limit before running massively parallelized tests. Do this with this command: `ulimit -n 8192` 32 | 33 | Creating New Visual Tests 34 | ------------------------- 35 | 36 | To create new visual tests, create a `visualtests` directory as a subdirectory under any directory that is a child of the `client` directory. Create any number of files you want with the format `-.jsx` where `direction` is either `ltr` for right to left languages, `rtl` for right to left languages, and `testDataNumber` is any number from 1 to n which represents a variation of the component's data that you would like to test. 37 | 38 | For example, if you look at the client/card directory, you can see a subdirectory called visualtests there. You can see four files: 39 | 40 | - ltr-1.jsx is an instantiation of the card component with a single word "test" in left to right direction. 41 | - ltr-2.jsx is an instantiation of the card component with a roughly fifty repetitions of the word "test" in left to right 42 | direction. 43 | - rtl-1.jsx is an instantiation of the card component with a single word "test" in right to left direction. 44 | - rtl-2.jsx is an instantiation of the card component with a roughly fifty repetitions of the word "test" in right to left 45 | direction. 46 | 47 | Here is an image which shows this: 48 | 49 | ![Visual Tests Directory Structure](https://cloud.githubusercontent.com/assets/381633/7289360/c9d8ac5e-e923-11e4-9101-e8fea1e5c45c.png) 50 | 51 | The test runner will automatically pick up files if you follow this format. There is no need to create a separate manifest file listing tests. Also, you can write the component bootstrap files ( ie: ltr-1.jsx ) just like you would in the real application. 52 | 53 | The jsx will be transpiled, the requires will be resolved. The css is from the output of the scss being compiled via make build. The template to host the components in the component library uses a jade template. 54 | 55 | Configuring Secret Keys 56 | ----------------------- 57 | 58 | Add a file to `css-visual-test/lib` called `PrivateConfig.js`, as in `css-visual-test/lib/PrivateConfig.js`. 59 | 60 | Copy and paste the following and fill in your Sauce Labs username and access key, and your Applitools access key: 61 | 62 | module.exports = { 63 | sauceLabsUsername: '', 64 | sauceLabsAccessKey: '', 65 | applitoolsEyesAccessKey: '' 66 | }; 67 | 68 | 69 | Running Visual Tests That Pass 70 | ------------------------------ 71 | 72 | `make visualtest` output: 73 | 74 | ![Make visualtest output](https://cloud.githubusercontent.com/assets/381633/7289373/1489c706-e924-11e4-89d2-ae825a4a79eb.png) 75 | 76 | Sauce Labs selenium grid runner status UI output: 77 | 78 | ![Sauce Labs selenium grid runner status UI output](https://cloud.githubusercontent.com/assets/381633/7289394/46e2b168-e924-11e4-9b17-5b4bd31bbcc9.png) 79 | 80 | A single passing test for one environment in the Applitools Eyes test runner status UI: 81 | 82 | ![A single passing test for one environment in the Applitools Eyes test runner status UI](https://cloud.githubusercontent.com/assets/381633/7289693/739ff798-e928-11e4-9a17-497cf4643c95.png) 83 | 84 | Running visual tests that fail 85 | ------------------------------------- 86 | 87 | After changing the `public/style.css` from: 88 | 89 | @media only screen and (min-width: 480px) { 90 | .card { 91 | margin-bottom: 16px; 92 | padding: 24px; } } 93 | 94 | to: 95 | 96 | @media only screen and (min-width: 480px) { 97 | .card { 98 | margin-bottom: 16px; 99 | padding: 240px; } } 100 | 101 | `make visualtest` output: 102 | 103 | ![screen shot 2015-04-08 at 6 10 47 pm](https://cloud.githubusercontent.com/assets/381633/7289786/ae96aa80-e929-11e4-80b1-3bca406c47a5.png) 104 | 105 | Applitools Eyes test runner status UI shows the failure: 106 | 107 | ![screen shot 2015-04-08 at 6 12 00 pm](https://cloud.githubusercontent.com/assets/381633/7289805/ede6befa-e929-11e4-85c3-f52d5f27b35a.png) 108 | 109 | A single failing test for one environment in the Applitools Eyes test runner status UI: 110 | 111 | ![screen shot 2015-04-08 at 6 13 00 pm](https://cloud.githubusercontent.com/assets/381633/7289815/1ca37c74-e92a-11e4-9e62-ab5ddb365ff8.png) 112 | 113 | Generating Your CSS 114 | ------------------- 115 | 116 | It's helpful sometimes to use a tool like less or sass to aid css development. This tool allows you to plug that in 117 | however you see fit, but doesn't require one. Just output your left to right stylesheet into the `public/style.css` 118 | file and your right to left stylesheet into the `public/style-rtl.css` using whatever tool you choose to use. 119 | 120 | Running The Project's Unit Tests 121 | -------------------------------- 122 | 123 | `make project-unit-test` 124 | 125 | Example Project 126 | --------------- 127 | 128 | The included example project uses a few tools you might want to swap out with your own preferred tools: 129 | 130 | - commonjs 131 | - browserify 132 | - react 133 | - make 134 | 135 | Cost Considerations For Various Configurations 136 | ---------------------------------------------- 137 | 138 | The ideal way to use the tool is to purchase as many parallel vms as you can from Sauce Labs, and as many licenses as you 139 | can from Applitools. If this is too expensive, you have various options to reduce costs: 140 | 141 | - Use less parallel vms. This will make your test suite run slower. 142 | - Only run the Applitools tests on merges to master. This will give you slower feedback cycles. 143 | 144 | Future Goals ( Pull Requests Appreciated! ) :-) 145 | ------------------------------------------------ 146 | 147 | - A mode to only run by comparing master to current branch for local usage, and non merge to master ci usage. 148 | - Simplifying the jsx naming scheme. 149 | - Add plugin hooks for non commonjs dependency management systems. 150 | - Add plugin hooks for non react based projects. 151 | - Add an example project for angular. 152 | - Add a file system watcher to regenerate the component library when css, js, and templates are changed. 153 | - Add an integration test using the local only run feature. 154 | - Add a functional test using Sauce Labs And Applitools. 155 | - Group all components into a single page which is used for batch diffing the components with links to each individual page still. Unfortunately, it doesn't look this this is a viable approach from Chrome ( https://code.google.com/p/chromedriver/issues/detail?id=294 ). According to ( https://bugsnag.com/blog/implementing-a-visual-css-testing-framework ) it also doesn't work in IE, but it does work in Firefox. Here's a nice breakdown of full page screenshot support across all four major browsers as of June 2014 ( https://support.saucelabs.com/entries/42638820-My-screenshots-aren-t-full-page-or-have-black-bars-in-IE-Chrome-Safari ). There is a workaround which would be to scroll the page and stitch sections of screenshots together. A great trick for implementing this is to focus an element at the bottom of the host page ( http://sauceio.com/index.php/2009/12/selenium-totw-capturing-screenshots-vs-scrollbars/ ). Applitools Eyes has a full page scanning feature ( https://applitools.com/web-app-testing/#Full_Page_Scanning ). I asked them if it supports all the major browsers / os / devices in the full page screenshot sense. They have it for some language sdks and are adding it to the javascript sdk within the next month or so. 156 | - Create a shared layout wrapper component for each component on the index grouped page, host this as it’s own npm project to allow selective upgrades. 157 | - For the react / commonjs example, load react from the host pages while still making them work from the require calls, to speed generation time of the component library. 158 | - Add a feature where you can have less vms than selenium environments in case you want to run many os / browser / device combos but not pay for a ton of parallel vms. 159 | - Change the time estimation calculation to account for vm startup time, which is what takes the longest. 160 | - Add a debug or local mode where if a sauce connect tunnel is already open it doesn’t try to reopen it 161 | - Add instructions in the readme for how to run on ci. 162 | - Describe the architecture in a wiki article linked to in the readme. 163 | -------------------------------------------------------------------------------- /client/card/card.jsx: -------------------------------------------------------------------------------- 1 | var React = require( 'react/addons' ); 2 | 3 | module.exports = React.createClass( { 4 | displayName: 'Card', 5 | 6 | render: function() { 7 | return React.createElement( 'div', { className: 'card' }, this.props.children ); 8 | } 9 | } ); 10 | -------------------------------------------------------------------------------- /client/card/visualtests/ltr-1.jsx: -------------------------------------------------------------------------------- 1 | var React = require( 'react' ), 2 | Card = require( 'card/card' ); 3 | 4 | React.render( test, document.getElementById( 'root' ) ); 5 | -------------------------------------------------------------------------------- /client/card/visualtests/ltr-2.jsx: -------------------------------------------------------------------------------- 1 | var React = require( 'react' ), 2 | Card = require( 'card/card' ); 3 | 4 | React.render( test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test , document.getElementById( 'root' ) ); 5 | -------------------------------------------------------------------------------- /client/card/visualtests/rtl-1.jsx: -------------------------------------------------------------------------------- 1 | var React = require( 'react' ), 2 | Card = require( 'card/card' ); 3 | 4 | React.render( test, document.getElementById( 'root' ) ); 5 | -------------------------------------------------------------------------------- /client/card/visualtests/rtl-2.jsx: -------------------------------------------------------------------------------- 1 | var React = require( 'react' ), 2 | Card = require( 'card/card' ); 3 | 4 | React.render( test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test test , document.getElementById( 'root' ) ); 5 | -------------------------------------------------------------------------------- /css-visual-test/Makefile: -------------------------------------------------------------------------------- 1 | REPORTER ?= spec 2 | MOCHA ?= ../node_modules/.bin/mocha 3 | 4 | # In order to simply stub modules, add test to the NODE_PATH 5 | test: 6 | @NODE_ENV=test $(MOCHA) --reporter $(REPORTER) 7 | 8 | .PHONY: test 9 | -------------------------------------------------------------------------------- /css-visual-test/SupportedSeleniumEnvironments.js: -------------------------------------------------------------------------------- 1 | // The highest value here for the lowest cost at the highest speed is to use the most popular operations systems, 2 | // and the most popular rendering engines across six parallel virtual machines on sauce labs: 3 | 4 | // Windows 7 Chrome 41/Latest, Blink rendering engine, Windows 7 operating system 5 | // Windows 7 Firefox 37/Latest, Gecko rendering engine 6 | // Windows 7 IE 11/Latest, Trident Rendering Engine 7 | // IOS Safari 8.2/Latest, Webkit Rendering Engine, IOS operating system 8 | // Android 4.4, Android operating system 9 | // OSX Chrome 41/Latest, OSX Operating System 10 | module.exports = { 11 | "environments": [ 12 | { 13 | "browserName": "chrome", 14 | "platform": "Windows 7", 15 | "version": "41.0", 16 | "commandTimeout": 600, 17 | "idleTimeout": 600 18 | } 19 | // Applitools free plan allows for one concurrent vm and sauce labs free plan allows for two concurrent vms. 20 | // Therefore, leave the most useful variations commented out in this file. You can add as many as you want 21 | // using the Sauce Labs Platform Configurator Tool ( select nodejs ): https://docs.saucelabs.com/reference/platforms-configurator/#/ 22 | /*, 23 | { 24 | "browserName": "firefox", 25 | "platform": "Windows 7", 26 | "version": "37.0", 27 | "commandTimeout": 600, 28 | "idleTimeout": 600 29 | }, 30 | { 31 | "browserName": "internet explorer", 32 | "platform": "Windows 7", 33 | "version": "11.0", 34 | "commandTimeout": 600, 35 | "idleTimeout": 600 36 | }, 37 | { 38 | "browserName": "iphone", 39 | "platform": "OS X 10.10", 40 | "version": "8.2", 41 | "deviceName": "iPhone Simulator", 42 | "device-orientation": "portrait", 43 | "commandTimeout": 600, 44 | "idleTimeout": 600 45 | }, 46 | { 47 | "browserName": "android", 48 | "platform": "Linux", 49 | "version": "4.4", 50 | "deviceName": "Android Emulator", 51 | "device-orientation": "portrait", 52 | "commandTimeout": 600, 53 | "idleTimeout": 600 54 | }, 55 | { 56 | "browserName": "chrome", 57 | "platform": "OS X 10.10", 58 | "version": "41.0", 59 | "commandTimeout": 600, 60 | "idleTimeout": 600 61 | }*/ 62 | ] 63 | }; 64 | -------------------------------------------------------------------------------- /css-visual-test/component-library-generator.js: -------------------------------------------------------------------------------- 1 | var componentLibraryGenerator = require( __dirname + '/lib/ComponentLibraryGenerator' ), 2 | manifestBuilder = require( __dirname + '/lib/ManifestBuilder' ); 3 | 4 | manifestBuilder.buildTestManifest( 'Calypso' ) 5 | .then( componentLibraryGenerator.generateComponentLibrary ) 6 | .then( function() { 7 | process.exit( 0 ); 8 | } ) 9 | .catch( function( error ) { 10 | console.log( error.stack ); 11 | process.exit( 1 ); 12 | } ); 13 | -------------------------------------------------------------------------------- /css-visual-test/component-library-server.js: -------------------------------------------------------------------------------- 1 | var path = require( 'path' ), 2 | connect = require( path.resolve( 'node_modules' ) + '/connect' ), 3 | winston = require( path.resolve( 'node_modules' ) + '/winston' ), 4 | portScanner = require( path.resolve( 'node_modules' ) + '/portscanner' ), 5 | psAux = require( path.resolve( 'node_modules' ) + '/ps-aux' ), 6 | config = require( __dirname + '/lib/Config' ), 7 | componentLibraryServerRunning = false, 8 | port9200Open = false, 9 | componentLibraryServerPort = config.componentLibraryPort, 10 | printToConsoleAndExit = function( message, code ) { 11 | winston.error( message ); 12 | process.exit( code ); 13 | }; 14 | 15 | psAux.singleton.parsed(function ( error, processList ) { 16 | var serverProcessesRunning = 0; 17 | 18 | if( error ) { 19 | printToConsoleAndExit( 'Error calling ps aux: ' + error.message ); 20 | } 21 | 22 | processList.forEach( function( process ) { 23 | if( process.command.indexOf( 'css-visual-test/component-library-server.js' ) > 0 ) { 24 | serverProcessesRunning++; 25 | } 26 | } ); 27 | 28 | // exclude the process running the check 29 | if( serverProcessesRunning > 1 ) { 30 | componentLibraryServerRunning = true; 31 | } 32 | 33 | portScanner.checkPortStatus( componentLibraryServerPort, 'localhost', function( error, status ) { 34 | if( status === 'open' ) { 35 | port9200Open = true; 36 | } 37 | 38 | // I'm intentionally not including the case where the server is running on a port that is not 9200, 39 | // because the tests depend on that port. 40 | if( port9200Open && 41 | ! componentLibraryServerRunning ) { 42 | // the port being in use is a fatal error and should stop excecution of the rest of the steps in the test 43 | printToConsoleAndExit( 'Port ' + componentLibraryServerPort + ' is in use by something other than the Component Library ' + 44 | 'Server. Please shut down the program using this port and try again.', 1 ); 45 | } else if( ! port9200Open && 46 | ! componentLibraryServerRunning ) { 47 | connect( 48 | connect.static( path.join( __dirname, 'component-library' ) ) // Serve the component library 49 | ).listen( componentLibraryServerPort ); 50 | 51 | winston.info( 'Component Library Server started on port ' + componentLibraryServerPort + '.' ); 52 | } else if( port9200Open && 53 | componentLibraryServerRunning ) { 54 | printToConsoleAndExit( 'Component Library Server is already running on port ' + componentLibraryServerPort + 55 | '. Server not started.', 0 ); 56 | } 57 | }); 58 | } ); 59 | 60 | -------------------------------------------------------------------------------- /css-visual-test/lib/ComponentLibraryGenerator.js: -------------------------------------------------------------------------------- 1 | var path = require( 'path' ), 2 | fs = require( 'fs'), 3 | wrench = require( path.resolve( 'node_modules' ) + '/wrench' ), 4 | jade = require( path.resolve( 'node_modules' ) + '/jade/jade' ), 5 | browserify = require ( path.resolve( 'node_modules' + '/browserify' ) ), 6 | fsExtra = require( path.resolve( 'node_modules' ) + '/fs-extra' ), 7 | reactTools = require( path.resolve( 'node_modules' ) + '/react-tools' ), 8 | lodashFindIndex = require( path.resolve( 'node_modules' ) + '/lodash/array/findIndex' ), 9 | Q = require( path.resolve( 'node_modules' ) + '/q' ), 10 | winston = require( 'winston' ), 11 | supportedSeleniumEnvironments = require( path.resolve( __dirname, '../' ) + '/SupportedSeleniumEnvironments' ), 12 | componentLibraryFullPath = path.resolve( 'css-visual-test/component-library' ), 13 | components = [], 14 | ComponentLibraryGenerator = null, 15 | browserifyConfig = { 16 | transforms: [ 'reactify' ], 17 | paths: [ 18 | path.resolve( __dirname, '../../client' ) 19 | ], 20 | debug: true, 21 | extensions: [ '.jsx' ], 22 | root: path.resolve( __dirname, '../..' ) 23 | }; 24 | 25 | 26 | 27 | // Because this algorithm runs in a single process before any multiple processes are spawned for test run speed, 28 | // synchronous calls are better due to being simpler. 29 | ComponentLibraryGenerator = { 30 | initializeGlobalTestEnvironment: function() { 31 | if( fs.existsSync( componentLibraryFullPath ) ) { 32 | wrench.rmdirSyncRecursive( componentLibraryFullPath, false ); // don't fail silently 33 | } 34 | 35 | wrench.copyDirSyncRecursive( 'public', componentLibraryFullPath ); 36 | 37 | // copy the phantomjs react shim 38 | fsExtra.copySync( path.resolve( 'node_modules') + '/react-tools/src/test/phantomjs-shims.js', componentLibraryFullPath + '/phantomjs-shims.js' ); 39 | }, 40 | createComponentTestDirectory: function( componentName ) { 41 | var componentIndex = ComponentLibraryGenerator.findComponentIndex( components, componentName ); 42 | 43 | if( componentIndex < 0 ) { 44 | components.push( { 45 | componentName: componentName, 46 | componentHostPages: [] 47 | } ); 48 | } 49 | 50 | if( ! fs.existsSync( componentLibraryFullPath + '/' + componentName ) ) { 51 | fs.mkdirSync( componentLibraryFullPath + '/' + componentName ); 52 | } 53 | }, 54 | createComponentEntryPoint: function( componentName, testDataNumber, bootstrapJsxFileFullPath ) { 55 | // create an entry point file to render the component 56 | var componentBootstrap = fs.readFileSync( bootstrapJsxFileFullPath, 'utf8' ); 57 | var renderComponentBootstrap = reactTools.transform( componentBootstrap ); 58 | 59 | // save the bundle for this test variation to disk 60 | fs.writeFileSync( componentLibraryFullPath + '/' + componentName + '/' + componentName + '-entry-point-test-data-' + testDataNumber.toString() + '.js', renderComponentBootstrap ); 61 | }, 62 | createComponentBrowserifyBundleAsync: function( componentName, testDataNumber ) { 63 | // browserify bundle the entry point file containing the render function 64 | // consider making bundler.js export and not call createBundle and use that code directly for reuse purposes 65 | var writeStream = fs.createWriteStream( componentLibraryFullPath + '/' + componentName + '/' + componentName +'-bundle-test-data-' + testDataNumber.toString() + '.js' ), 66 | deferred = Q.defer(); 67 | 68 | writeStream.on( 'finish', function() { 69 | deferred.resolve( testDataNumber ); 70 | } ).on( 'error', function( exception ) { 71 | deferred.reject( exception ); 72 | } ); 73 | 74 | // bundle is asynchronous 75 | // TODO: don't bundle react and/or lodash. load them externally from the host pages instead to increase test run speed. 76 | browserify( browserifyConfig ) 77 | .add( componentLibraryFullPath + '/' + componentName + '/' + componentName + '-entry-point-test-data-' + testDataNumber.toString() + '.js' ) 78 | .transform( 'reactify' ) 79 | .bundle() 80 | .pipe( writeStream ); 81 | 82 | return deferred.promise; 83 | }, 84 | createComponentHostPage: function ( componentName, direction, testDataNumber ) { 85 | var isRTL = false, 86 | hostPageFilename = '', 87 | indexJadeFileContents = '', 88 | compiledTemplate = '', 89 | componentIndex = 0; 90 | 91 | if( direction === 'RTL' ) { 92 | isRTL = true; 93 | hostPageFilename = 'index-rtl-test-data-' + testDataNumber.toString() + '.html'; 94 | } else { 95 | hostPageFilename = 'index-ltr-test-data-' + testDataNumber.toString() + '.html'; 96 | } 97 | 98 | // compile the jade template for ltr languages for each test 99 | indexJadeFileContents = fs.readFileSync( path.resolve( 'server' ) + '/pages/index.jade', 'utf8' ); 100 | 101 | compiledTemplate = jade.compile( indexJadeFileContents )( { 102 | compileDebug: false, 103 | urls: { 104 | 'style.css': '/style.css', 105 | 'style-rtl.css': '/style-rtl.css', 106 | 'build.js': '/' + componentName + '/' + componentName + '-bundle-test-data-' + testDataNumber.toString() + '.js' 107 | }, 108 | isRTL: isRTL, 109 | lang: 'en' 110 | } ); 111 | 112 | // react does not work by default with phantomjs due to phantomjs not supporting .bind. apply shim to fix. 113 | // leave this in even when removing phantonflow since we may want to use phantom/ghost as a selenium driver 114 | // for quick smoke tests before long test runs 115 | compiledTemplate = compiledTemplate.replace('', ''); 116 | compiledTemplate = compiledTemplate.replace('', ''); 117 | 118 | fs.writeFileSync( componentLibraryFullPath + '/' + componentName + '/' + hostPageFilename, compiledTemplate ); 119 | 120 | componentIndex = ComponentLibraryGenerator.findComponentIndex( components, componentName ); 121 | 122 | components[componentIndex].componentHostPages.push( { 123 | componentHostPageName: direction + ' Test Data ' + testDataNumber.toString(), 124 | componentHostPageUrl: '/' + componentName + '/' + hostPageFilename 125 | } ); 126 | }, 127 | findComponentIndex: function( components, componentName ) { 128 | return lodashFindIndex( components, function( component ) { 129 | return component.componentName === componentName; 130 | } ); 131 | }, 132 | generateComponentLibraryIndex: function() { 133 | // consider using jade for this 134 | var indexPage = '

Component Library

'; 135 | 136 | // TODO: sort the components and host pages alphabetically. they can become out of order since they are async. 137 | components.forEach( function( component ) { 138 | indexPage += '

' + component.componentName + '

'; 145 | } ); 146 | 147 | indexPage += ''; 148 | 149 | fs.writeFileSync( componentLibraryFullPath + '/index.html', indexPage ); 150 | }, 151 | generateComponentLibrary: function( componentManifest ) { 152 | var deferred = Q.defer(), 153 | browserifyBundlePromises = []; 154 | 155 | winston.info( 'Component Library Generator started.' ); 156 | 157 | ComponentLibraryGenerator.initializeGlobalTestEnvironment(); 158 | 159 | componentManifest.variations.forEach( function( componentVariation ) { 160 | var browserifyBundlePromise = null; 161 | 162 | ComponentLibraryGenerator.createComponentTestDirectory( componentVariation.componentName ); 163 | 164 | var componentTestDirectionUpperCase = componentVariation.direction.toUpperCase(); 165 | 166 | ComponentLibraryGenerator.createComponentEntryPoint( componentVariation.componentName, componentVariation.testDataNumber, componentVariation.bootstrapJsxFileFullPath ); 167 | 168 | browserifyBundlePromise = ComponentLibraryGenerator.createComponentBrowserifyBundleAsync( componentVariation.componentName, componentVariation.testDataNumber ); 169 | 170 | browserifyBundlePromises.push( browserifyBundlePromise ); 171 | 172 | browserifyBundlePromise.then( function( currentIndex ) { 173 | ComponentLibraryGenerator.createComponentHostPage( componentVariation.componentName, componentTestDirectionUpperCase, currentIndex ); 174 | }, function( error ) { 175 | deferred.reject( error ); 176 | } ); 177 | } ); 178 | 179 | Q.allSettled( browserifyBundlePromises ).then( function() { 180 | ComponentLibraryGenerator.generateComponentLibraryIndex(); 181 | 182 | componentManifest.supportedSeleniumEnvironments = supportedSeleniumEnvironments; 183 | 184 | winston.info( 'Component Library Generator completed.' ); 185 | 186 | deferred.resolve( componentManifest ); 187 | } ); 188 | 189 | return deferred.promise; 190 | } 191 | }; 192 | 193 | module.exports = ComponentLibraryGenerator; 194 | -------------------------------------------------------------------------------- /css-visual-test/lib/Config.js: -------------------------------------------------------------------------------- 1 | var privateConfig = require( __dirname + '/PrivateConfig' ), 2 | path = require( 'path' ), 3 | lodashExtend = require( path.resolve( 'node_modules' ) + '/lodash/object/extend' ), 4 | componentLibraryPort = 4000, 5 | Config = { 6 | // IOS Tests vms are crazy slow to start 7 | singleMobileTestRunWorstCaseRunningTimeSeconds: 50, 8 | // there must always be at least as many parallel virtual machines as there is environments ( browser/os/device combinations ) 9 | parallelVirtualMachinesCount: 1, 10 | projectName: 'CSS Visual Test', 11 | seleniumGridHubUrl: 'http://ondemand.saucelabs.com:80/wd/hub', 12 | componentLibraryPort: componentLibraryPort, 13 | componentLibraryUrl: 'http://localhost:' + componentLibraryPort + '/', 14 | logLevel: 'info' // use debug for debugging, info otherwise 15 | }; 16 | 17 | lodashExtend( Config, privateConfig ); 18 | 19 | module.exports = Config; 20 | -------------------------------------------------------------------------------- /css-visual-test/lib/ManifestBuilder.js: -------------------------------------------------------------------------------- 1 | var readDirectory = require( 'readdirp' ), 2 | Q = require( 'q' ), 3 | path = require( 'path' ), 4 | winston = require( 'winston' ); 5 | 6 | var ManifestBuilder = { 7 | buildTestManifest: function( config ) { 8 | ManifestBuilder.projectName = config.projectName; 9 | ManifestBuilder.config = config; 10 | 11 | winston.info( 'Started building test manifest by reading the file system.' ) 12 | 13 | return ManifestBuilder.findVisualTestDirectories() 14 | .then( ManifestBuilder.attachJsxFilesToDirectories ) 15 | .then( ManifestBuilder.assembleTests ) 16 | .then( ManifestBuilder.outputSuccessMessage ) 17 | .catch( function( error ) { 18 | winston.error( error.stack ); 19 | process.exit( 1 ); 20 | } ); 21 | }, 22 | findVisualTestDirectories: function() { 23 | var deferred = Q.defer(); 24 | 25 | readDirectory( { 26 | root: path.resolve( 'client' ), 27 | entryType: 'directories' 28 | }, function( errors, directoryEntries ) { 29 | var visualTestDirectories = []; 30 | 31 | ManifestBuilder.handleReadDirectoryError( deferred, 'One or more errors occurred finding visual test directories: ', errors ); 32 | 33 | visualTestDirectories = directoryEntries.directories.filter( function( element ) { 34 | return element.fullPath.indexOf( 'visualtests' ) > -1; 35 | } ); 36 | 37 | visualTestDirectories.sort( function (a, b) { 38 | if (a.parentDir > b.parentDir) { 39 | return 1; 40 | } 41 | if (a.parentDir < b.parentDir) { 42 | return -1; 43 | } 44 | 45 | return 0; 46 | } ); 47 | 48 | deferred.resolve( visualTestDirectories ); 49 | } ); 50 | 51 | return deferred.promise; 52 | }, 53 | attachJsxFilesToDirectories: function( visualTestDirectories ) { 54 | var deferred = Q.defer(), 55 | jsxDirectoriesAttached = 0; 56 | 57 | visualTestDirectories.forEach( function( directoryEntry, index ) { 58 | readDirectory( { 59 | root: directoryEntry.fullPath, 60 | depth: 0, 61 | fileFilter: '*.jsx', 62 | entryType: 'files' 63 | }, function( errors, jsxFileEntries ) { 64 | ManifestBuilder.handleReadDirectoryError( deferred, 'One or more errors occured finding jsx files: ', errors ); 65 | 66 | jsxFileEntries.files.sort( function (a, b) { 67 | if (a.name > b.name) { 68 | return 1; 69 | } 70 | if (a.name < b.name) { 71 | return -1; 72 | } 73 | 74 | return 0; 75 | } ); 76 | 77 | visualTestDirectories[ index ].jsxFiles = jsxFileEntries.files; 78 | 79 | jsxDirectoriesAttached++; 80 | 81 | if( jsxDirectoriesAttached === visualTestDirectories.length ) { 82 | deferred.resolve( visualTestDirectories ); 83 | } 84 | } ); 85 | } ); 86 | 87 | return deferred.promise; 88 | }, 89 | assembleTests: function( visualTestDirectories ) { 90 | var deferred = Q.defer(), 91 | componentManifest = { 92 | projectName: ManifestBuilder.projectName 93 | }, 94 | variations = []; 95 | 96 | visualTestDirectories.forEach( function( directoryInfo ) { 97 | directoryInfo.jsxFiles.forEach( function( jsxFileEntry ) { 98 | var jsxFilename = jsxFileEntry.name, 99 | jsxFilenameAndDirection = jsxFilename.replace( '.jsx', '' ).split( '-' ), 100 | jsxFilenameDirection = jsxFilenameAndDirection[ 0 ], 101 | jsxFilenameTestDataNumber = parseInt( jsxFilenameAndDirection[ 1 ], 10 ), 102 | variation = { 103 | componentName: directoryInfo.parentDir.replace( '/', '--' ), 104 | direction: jsxFilenameDirection, 105 | testDataNumber: jsxFilenameTestDataNumber, 106 | bootstrapJsxFileFullPath: jsxFileEntry.fullPath 107 | }; 108 | 109 | variations.push( variation ); 110 | } ); 111 | } ); 112 | 113 | componentManifest.variations = variations; 114 | 115 | deferred.resolve( componentManifest ); 116 | 117 | return deferred.promise; 118 | }, 119 | outputSuccessMessage: function( componentManifest ) { 120 | var deferred = Q.defer(); 121 | 122 | componentManifest.config = ManifestBuilder.config; 123 | 124 | winston.info( 'Completed building test manifest by reading the file system.' ); 125 | 126 | deferred.resolve( componentManifest ); 127 | 128 | return deferred.promise; 129 | }, 130 | handleReadDirectoryError: function( deferred, message, errors ) { 131 | if( errors ) { 132 | deferred.reject( { 133 | message: message + errors.join( '\n' ) 134 | } ); 135 | } 136 | } 137 | }; 138 | 139 | module.exports = ManifestBuilder; 140 | -------------------------------------------------------------------------------- /css-visual-test/lib/ParallelTestPartitioner.js: -------------------------------------------------------------------------------- 1 | var Q = require( 'q' ), 2 | uuid = require( 'node-uuid' ), 3 | lodashFindIndex = require( 'lodash/array/findIndex' ), 4 | supportedSeleniumEnvironments = {}, 5 | ParallelTestPartitioner = { 6 | // See https://docs.google.com/document/d/1GlurM0T_K3RHSsEATI4FH42Ggqd-lritkI-7fW8A398/edit?usp=sharing 7 | // for a detailed explanation of this algorithm. 8 | // The distribution is not 100% uniform in certain cases. I came up with a much better algorithm which is as 9 | // uniform as can be and will increase test run speed by 2-3x in those cases. 10 | // TODO: implement the improved algorithm, which is illustrated here: 11 | // https://cloud.githubusercontent.com/assets/381633/7289966/8b7cbf5a-e92c-11e4-9fd4-ba981d71c3ef.jpg 12 | partitionComponentManifest: function( componentManifest ) { 13 | // Normally I prefer one var, but I had a bizarre bug where the count of 14 | // testsBucketedByVirtualMachineWithRemainder.remainder was shrinking between when 15 | // generateTestsBucketedByVirtualMachine() returned and after 16 | // fillTestsBucketedByVirtualMachineWithRemainder() started. This fixed it. 17 | var config = componentManifest.config; 18 | var deferred = Q.defer(); 19 | var buildId = ''; 20 | 21 | supportedSeleniumEnvironments = componentManifest.supportedSeleniumEnvironments; 22 | 23 | if( componentManifest.buildId === undefined ) { 24 | buildId = uuid.v4(); 25 | } 26 | 27 | var augmentedSupportedSeleniumEnvironments = 28 | ParallelTestPartitioner.augmentSupportedSeleniumEnvironments( supportedSeleniumEnvironments, 29 | componentManifest.projectName, buildId, config.sauceLabsUsername, config.sauceLabsAccessKey ); 30 | var testsBucketedBySeleniumCapability = ParallelTestPartitioner.generateTestsBucketedBySeleniumCapability ( 31 | componentManifest, augmentedSupportedSeleniumEnvironments, buildId, config 32 | ); 33 | 34 | componentManifest.testCount = testsBucketedBySeleniumCapability.testsBucketedBySeleniumCapability.length; 35 | 36 | var testsBucketedBySeleniumCapabilitySortedByEnvironment = 37 | testsBucketedBySeleniumCapability.testsBucketedBySeleniumCapability.sort( 38 | ParallelTestPartitioner.sortTestsBucketedBySeleniumCapabilityBySortingKeyComparator ); 39 | var variationsPerVirtualMachineBucket = Math.floor( 40 | testsBucketedBySeleniumCapability.testsBucketedBySeleniumCapability.length / 41 | config.parallelVirtualMachinesCount ); 42 | var variationsPerVirtualMachineBucketRemainder = 43 | testsBucketedBySeleniumCapability.testsBucketedBySeleniumCapability.length % 44 | config.parallelVirtualMachinesCount; 45 | var testsBucketedByVirtualMachineWithRemainder = ParallelTestPartitioner.generateTestsBucketedByVirtualMachine( 46 | testsBucketedBySeleniumCapabilitySortedByEnvironment, variationsPerVirtualMachineBucket, 47 | variationsPerVirtualMachineBucketRemainder, config.parallelVirtualMachinesCount ); 48 | var testsBucketedByVirtualMachine = ParallelTestPartitioner.fillTestsBucketedByVirtualMachineWithRemainder( 49 | testsBucketedByVirtualMachineWithRemainder 50 | ); 51 | 52 | componentManifest.buildId = buildId; 53 | 54 | testsBucketedByVirtualMachine.componentManifest = componentManifest; 55 | 56 | deferred.resolve( testsBucketedByVirtualMachine ); 57 | 58 | return deferred.promise; 59 | }, 60 | findBucketIndex: function( testsBucketedByVirtualMachineWithRemainder, testBucketedByVirtualMachineRemainderItem ) { 61 | return lodashFindIndex( testsBucketedByVirtualMachineWithRemainder.testsBucketedByVirtualMachine, 62 | function( searchedTestBucketedByVirtualMachine ) { 63 | return searchedTestBucketedByVirtualMachine.groupingKey === testBucketedByVirtualMachineRemainderItem.groupingKey; 64 | } ); 65 | }, 66 | createVariationFromSeleniumCapability: function ( seleniumCapability ) { 67 | return { 68 | componentName: seleniumCapability.componentName, 69 | direction: seleniumCapability.direction, 70 | testDataNumber: seleniumCapability.testDataNumber, 71 | sortingKey: seleniumCapability.sortingKey 72 | }; 73 | }, 74 | fillEachBucketWithOneTest: function( testsBucketedBySeleniumCapabilitySortedByEnvironment, 75 | testsBucketedByVirtualMachine ) { 76 | var partitionId = 1; 77 | 78 | testsBucketedBySeleniumCapabilitySortedByEnvironment.forEach( function ( testWithSeleniumCapability ) { 79 | testsBucketedByVirtualMachine.testsBucketedByVirtualMachine.push( { 80 | capability: testWithSeleniumCapability.capability, 81 | variations: [ ParallelTestPartitioner.createVariationFromSeleniumCapability( testWithSeleniumCapability ) ], 82 | partitionId: partitionId 83 | } ); 84 | 85 | partitionId++; 86 | } ); 87 | 88 | testsBucketedByVirtualMachine.remainder = []; 89 | 90 | return testsBucketedByVirtualMachine; 91 | }, 92 | fillTestsBucketedByVirtualMachineWithRemainder: function( testsBucketedByVirtualMachineWithRemainder ) { 93 | var testBucketedByVirtualMachineRemainderItem = {}, 94 | bucketToFillIndex = 0; 95 | 96 | if( testsBucketedByVirtualMachineWithRemainder.remainder.length > 0 ) { 97 | while( testsBucketedByVirtualMachineWithRemainder.remainder.length > 0 ) { 98 | testBucketedByVirtualMachineRemainderItem = testsBucketedByVirtualMachineWithRemainder.remainder.shift(); 99 | 100 | bucketToFillIndex = ParallelTestPartitioner.findBucketIndex( testsBucketedByVirtualMachineWithRemainder, 101 | testBucketedByVirtualMachineRemainderItem ); 102 | 103 | testsBucketedByVirtualMachineWithRemainder.testsBucketedByVirtualMachine[ bucketToFillIndex ].variations.push( 104 | ParallelTestPartitioner.createVariationFromSeleniumCapability( testBucketedByVirtualMachineRemainderItem ) 105 | ); 106 | } 107 | } 108 | 109 | delete testsBucketedByVirtualMachineWithRemainder.remainder; 110 | 111 | return testsBucketedByVirtualMachineWithRemainder; 112 | }, 113 | sortTestsBucketedBySeleniumCapabilityBySortingKeyComparator: function( testBucketedBySeleniumCapabilityA, testBucketedBySeleniumCapabilityB ) { 114 | if ( testBucketedBySeleniumCapabilityA.sortingKey > testBucketedBySeleniumCapabilityB.sortingKey ) { 115 | return 1; 116 | } 117 | if ( testBucketedBySeleniumCapabilityA.sortingKey < testBucketedBySeleniumCapabilityB.sortingKey ) { 118 | return -1; 119 | } 120 | 121 | return 0; 122 | }, 123 | generateTestsBucketedByVirtualMachine: function( testsBucketedBySeleniumCapabilitySortedByEnvironment, 124 | variationsPerVirtualMachineBucket, variationsPerVirtualMachineBucketRemainder, parallelVirtualMachinesCount ) { 125 | var testsBucketedByVirtualMachine = { 126 | testsBucketedByVirtualMachine: [] 127 | }, 128 | itemsToIterate = testsBucketedBySeleniumCapabilitySortedByEnvironment.length - 129 | variationsPerVirtualMachineBucketRemainder, 130 | removedTestWithSeleniumCapability = null, 131 | testBucketedByVirtualMachine = null, 132 | partitionId = 1; 133 | 134 | if( itemsToIterate === 0 ) { 135 | // if we have more buckets than test, just put each test in it's own bucket 136 | return ParallelTestPartitioner.fillEachBucketWithOneTest( testsBucketedBySeleniumCapabilitySortedByEnvironment, 137 | testsBucketedByVirtualMachine ); 138 | } 139 | 140 | while( itemsToIterate > 0 && 141 | testsBucketedByVirtualMachine.testsBucketedByVirtualMachine.length < parallelVirtualMachinesCount ) { 142 | removedTestWithSeleniumCapability = testsBucketedBySeleniumCapabilitySortedByEnvironment.shift(); 143 | 144 | if( testsBucketedByVirtualMachine.testsBucketedByVirtualMachine.length === 0 || 145 | testBucketedByVirtualMachine.variations.length === variationsPerVirtualMachineBucket || 146 | removedTestWithSeleniumCapability.groupingKey !== testBucketedByVirtualMachine.groupingKey ) { 147 | 148 | testBucketedByVirtualMachine = { 149 | capability: removedTestWithSeleniumCapability.capability, 150 | variations: [], 151 | groupingKey: ParallelTestPartitioner.generateGroupingKey( 152 | removedTestWithSeleniumCapability.capability.browserName, 153 | removedTestWithSeleniumCapability.capability.platform, 154 | removedTestWithSeleniumCapability.capability.version ), 155 | partitionId: partitionId 156 | }; 157 | 158 | testsBucketedByVirtualMachine.testsBucketedByVirtualMachine.push( testBucketedByVirtualMachine ); 159 | partitionId++; 160 | } 161 | 162 | testBucketedByVirtualMachine.variations.push( 163 | ParallelTestPartitioner.createVariationFromSeleniumCapability( removedTestWithSeleniumCapability ) ); 164 | 165 | itemsToIterate--; 166 | } 167 | 168 | testsBucketedByVirtualMachine.remainder = testsBucketedBySeleniumCapabilitySortedByEnvironment; 169 | 170 | return testsBucketedByVirtualMachine; 171 | }, 172 | generateTestsBucketedBySeleniumCapability: function( componentManifest, augmentedSupportedSeleniumEnvironments, 173 | buildId, config ) { 174 | var testsBucketedBySeleniumCapability = { 175 | testsBucketedBySeleniumCapability: [] 176 | }; 177 | 178 | // n^2 is not great, but input size is small. should be fine. 179 | componentManifest.variations.forEach( function( variation ) { 180 | augmentedSupportedSeleniumEnvironments.forEach( function( environment ) { 181 | var test = { 182 | groupingKey: ParallelTestPartitioner.generateGroupingKey( environment.browserName, 183 | environment.platform, environment.version ), 184 | sortingKey: ParallelTestPartitioner.generateSortingKey( 185 | environment.browserName, 186 | environment.platform, 187 | environment.version, 188 | variation.componentName, 189 | variation.direction, 190 | variation.testDataNumber ), 191 | capability: { 192 | build: buildId, 193 | name: ParallelTestPartitioner.generateTestName(componentManifest.projectName, 194 | environment.browserName, environment.platform, environment.version ), 195 | username: config.sauceLabsUsername, 196 | accessKey: config.sauceLabsAccessKey, 197 | browserName: environment.browserName, 198 | platform: environment.platform, 199 | version: environment.version 200 | }, 201 | componentName: variation.componentName, 202 | direction: variation.direction, 203 | testDataNumber: variation.testDataNumber 204 | }; 205 | 206 | if( environment.deviceName !== undefined ) { 207 | test.capability.deviceName = environment.deviceName; 208 | } 209 | 210 | if( environment[ 'device-orientation' ] !== undefined ) { 211 | test.capability[ 'device-orientation' ] = environment[ 'device-orientation' ]; 212 | } 213 | 214 | testsBucketedBySeleniumCapability.testsBucketedBySeleniumCapability.push( test ); 215 | } ); 216 | } ); 217 | 218 | return testsBucketedBySeleniumCapability; 219 | }, 220 | generateSortingKey: function( browserName, platform, version, componentName, direction, testDataNumber ) { 221 | return ParallelTestPartitioner.generateGroupingKey( browserName, platform, version) + '-' + 222 | componentName + '-' + direction + '-' + testDataNumber.toString(); 223 | }, 224 | generateGroupingKey: function( browserName, platform, version ) { 225 | return browserName.replace( ' ', '-' ) + '-' + platform.replace( ' ', '-' ) + '-' + version.replace( ' ', '-' ); 226 | }, 227 | generateTestName: function ( projectName, browserName, platform, version ) { 228 | return 'All ' + projectName + ' components with all variations in ' + browserName + ' ' + version + ' on ' + platform + '.'; 229 | }, 230 | augmentSupportedSeleniumEnvironments: function( supportedSeleniumEnvironments, projectName, buildId, 231 | sauceLabsUsername, sauceLabsAccessKey ) { 232 | return supportedSeleniumEnvironments.environments.map( function( supportedSeleniumEnvironment ) { 233 | supportedSeleniumEnvironment.build = buildId; 234 | supportedSeleniumEnvironment.name = ParallelTestPartitioner.generateTestName( projectName, 235 | supportedSeleniumEnvironment.browserName, supportedSeleniumEnvironment.platform, 236 | supportedSeleniumEnvironment.version ); 237 | supportedSeleniumEnvironment.username = sauceLabsUsername; 238 | supportedSeleniumEnvironment.accessKey = sauceLabsAccessKey; 239 | 240 | return supportedSeleniumEnvironment; 241 | } ); 242 | } 243 | }; 244 | 245 | module.exports = ParallelTestPartitioner; 246 | -------------------------------------------------------------------------------- /css-visual-test/lib/ParallelTestRunner.js: -------------------------------------------------------------------------------- 1 | var webdriver = require( 'selenium-webdriver' ), 2 | Q = require( 'q' ), 3 | Eyes = require( 'eyes.selenium' ).Eyes, 4 | sauceConnectLauncher = require( 'sauce-connect-launcher' ), 5 | winston = require( 'winston' ), 6 | config = require( __dirname + '/Config' ), 7 | ParallelTestRunner = { 8 | queueTestForSingleVariation: function( eyes, driver, variation, seleniumCapability ) { 9 | var deferred = Q.defer(), 10 | operationsCompleted = 0; 11 | 12 | driver.get( config.componentLibraryUrl + variation.componentName + '/index-' + variation.direction + 13 | '-test-data-' + variation.testDataNumber.toString() + '.html' ) 14 | .then( function() { 15 | // this takes a screenshot 16 | eyes.checkWindow( variation.componentName + ' -- ' + variation.direction.toUpperCase() + 17 | ' -- Test Data ' + variation.testDataNumber.toString() ) 18 | .then( function() { 19 | winston.info( 'Visual test completed for ' + variation.sortingKey ); 20 | 21 | deferred.resolve(); 22 | } ) 23 | .thenCatch( function( error ) { 24 | winston.debug( 'error occured in eyes.checkWindow(): ' + error.message ); 25 | } ); 26 | } ) 27 | .thenCatch( function( error ) { 28 | winston.debug( 'error occured in driver.get(): ' + error.message ); 29 | } ); 30 | 31 | return deferred.promise; 32 | }, 33 | // TODO: rewrite the time estimation to be based on environment startup time + number of tests factoring in parallelization 34 | estimateRunningTime: function( componentManifest ) { 35 | var estimatedMinutes = Math.round( 36 | ( componentManifest.testCount * config.singleMobileTestRunWorstCaseRunningTimeSeconds ) / 37 | ( 60 * config.parallelVirtualMachinesCount ) ); 38 | 39 | if( estimatedMinutes === 1 ) { 40 | return '1 minute'; 41 | } 42 | 43 | return estimatedMinutes.toString() + ' minutes' 44 | }, 45 | printMessageAndExitWithCode: function( message, code ) { 46 | if( code === 0 ) { 47 | winston.info( message ); 48 | } else { 49 | winston.error( message ); 50 | } 51 | process.exit( code ); 52 | }, 53 | printExecutionSeconds: function( startTime ) { 54 | var executionMinutes = Math.round( ( new Date() - startTime ) / 60000 ), 55 | minuteMinutes = ' minutes.'; 56 | 57 | if( executionMinutes === 1) { 58 | minuteMinutes = ' minute.' 59 | } 60 | 61 | winston.info( 'Visual tests completed in ' + executionMinutes.toString() + minuteMinutes ); 62 | }, 63 | executeTests: function( testsBucketedByVirtualMachine, completedCallback ) { 64 | var concurrentTestRuns = [], 65 | startTime = null, 66 | errorMessages = [], 67 | componentManifest = testsBucketedByVirtualMachine.componentManifest, 68 | buildId = componentManifest.buildId; 69 | 70 | winston.info( 'CSS Visual Test started with id: ' + componentManifest.buildId ); 71 | 72 | winston.info( 'Running ' + componentManifest.testCount + ' visual tests across ' + 73 | config.parallelVirtualMachinesCount + ' parallel vms' ); 74 | 75 | winston.info( 'Estimated run time: ' + ParallelTestRunner.estimateRunningTime( componentManifest ) ); 76 | 77 | startTime = new Date(); 78 | 79 | concurrentTestRuns = testsBucketedByVirtualMachine.testsBucketedByVirtualMachine.map( 80 | function( testBucketedByVirtualMachine ) { 81 | var eyes = new Eyes(); 82 | 83 | eyes.setApiKey( config.applitoolsEyesAccessKey ); 84 | eyes.setBatch( componentManifest.projectName + ' build ' + buildId, buildId, startTime ); 85 | 86 | return webdriver.promise.createFlow( function() { 87 | var seleniumCapability = testBucketedByVirtualMachine.capability, 88 | driver = new webdriver 89 | .Builder() 90 | .usingServer( config.seleniumGridHubUrl ) 91 | .withCapabilities( seleniumCapability ) 92 | .build(), 93 | eyesTestName = seleniumCapability.name, 94 | eyesOpenPromise = null; 95 | 96 | eyesOpenPromise = eyes.open( driver, componentManifest.projectName, eyesTestName ) 97 | .then( function( driver ) { 98 | testBucketedByVirtualMachine.variations.forEach( function( variation ) { 99 | ParallelTestRunner.queueTestForSingleVariation( eyes, driver, variation, seleniumCapability ); 100 | } ); 101 | 102 | winston.debug( 'all tests for partition id ' + testBucketedByVirtualMachine.partitionId + ' queued.' ); 103 | 104 | driver.quit(); 105 | // if the visual tests fail, this rejects the promise returned from eyes.open triggering 106 | // the thenCatch 107 | eyes.close(); 108 | 109 | winston.debug( 'all tests for partition id ' + testBucketedByVirtualMachine.partitionId + 110 | ' had driver.quit() and eyes.close() queued.' ); 111 | } ) 112 | // the google closure promise api used by the selenium webdriverjs and the applitools eyes 113 | // sdk uses thenCatch and cancel instead of catch and reject. 114 | .thenCatch( function( error ) { 115 | winston.debug( 'error for build id: ' + buildId + '\n' + error.message ); 116 | 117 | errorMessages.push( error.message ); 118 | 119 | // reject the promise that failed, without stopping the other concurrent operations 120 | eyesOpenPromise.cancel(); 121 | } ); 122 | } ); 123 | } ); 124 | 125 | // fullyResolved is webdriver's equivalent of Q's allSettled method 126 | webdriver.promise.fullyResolved( concurrentTestRuns ).then( function() { 127 | if( errorMessages.length > 0 ) { 128 | ParallelTestRunner.printExecutionSeconds( startTime ); 129 | ParallelTestRunner.printMessageAndExitWithCode( 'CSS Visual Test run failed: \n' + errorMessages.join( '\n' ), 1 ); 130 | } else { 131 | winston.info( 'CSS Visual Test run completed successfully with id: ' + buildId ); 132 | ParallelTestRunner.printExecutionSeconds( startTime ); 133 | 134 | if( completedCallback !== undefined ) { 135 | completedCallback(); 136 | } 137 | } 138 | } ); 139 | }, 140 | executeTestRun: function( testsBucketedByVirtualMachine ) { 141 | winston.info( 'Opening Sauce Connect tunnel.' ); 142 | /* 143 | sauceConnectLauncher( { 144 | username: config.sauceLabsUsername, 145 | accessKey: config.sauceLabsAccessKey, 146 | verbose: false, 147 | verboseDebugging: false 148 | }, function ( error , sauceConnectProcess ) { 149 | if ( error ) { 150 | ParallelTestRunner.printMessageAndExitWithCode( error.message, 1 ); 151 | } 152 | 153 | winston.info( 'Sauce Connect tunnel opened.' ); 154 | */ 155 | ParallelTestRunner.executeTests( testsBucketedByVirtualMachine, 156 | function() { 157 | // sauceConnectProcess.close(function () { 158 | ParallelTestRunner.printMessageAndExitWithCode( 'Sauce Connect tunnel closed.', 0 ); 159 | // } ); 160 | } ); 161 | // } ); 162 | } 163 | }; 164 | 165 | module.exports = ParallelTestRunner; 166 | -------------------------------------------------------------------------------- /css-visual-test/run-visual-tests.js: -------------------------------------------------------------------------------- 1 | var ParallelTestPartitioner = require( __dirname + '/lib/ParallelTestPartitioner' ), 2 | ParallelTestRunner = require( __dirname + '/lib/ParallelTestRunner' ), 3 | ComponenentLibraryGenerator = require( __dirname + '/lib/ComponentLibraryGenerator' ), 4 | ManifestBuilder = require( __dirname + '/lib/ManifestBuilder' ), 5 | Config = require( __dirname + '/lib/Config' ), 6 | winston = require( 'winston' ); 7 | 8 | winston.level = Config.logLevel; 9 | 10 | ManifestBuilder.buildTestManifest( Config ) 11 | .then( ComponenentLibraryGenerator.generateComponentLibrary ) 12 | .then( ParallelTestPartitioner.partitionComponentManifest ) 13 | .then( ParallelTestRunner.executeTestRun ) 14 | .catch( function( error ) { 15 | winston.error( error.stack ); 16 | process.exit( 1 ); 17 | } ); 18 | -------------------------------------------------------------------------------- /css-visual-test/test/ParallelTestPartitionerTests.js: -------------------------------------------------------------------------------- 1 | var assert = require( 'assert' ), 2 | ParallelTestPartitioner = require( '../lib/ParallelTestPartitioner' ), 3 | Config = require( '../lib/Config' ), 4 | fourTestsFromOneDirectoryInputData = 5 | require( './data/FourTestsFromOneDirectoryInputData' ), 6 | fiveTestsAcrossTwoDirectoriesInputData = 7 | require( './data/FiveTestsAcrossTwoDirectoriesInputData' ), 8 | threeBrowsersSupportedSeleniumEnvironmentsStub = 9 | require( './data/ThreeBrowsersSupportedSeleniumEnvironmentsStub' ), 10 | threeBucketsWithFourTestsInEachBucket = require( 11 | './data/ThreeBucketsWithFourTestsInEachBucket' ), 12 | sevenBucketsWithAVaryingNumberOfTestsInEachBucket = require( 13 | './data/SevenBucketsWithAVaryingNumberOfTestsInEachBucket' ), 14 | fifteenBucketsWithOneTestInEachBucket = require( 15 | './data/FifteenBucketsWithOneTestInEachBucket' ), 16 | augmentComponentManifest = function( inputData, parallelVirtualMachinesCount, 17 | supportedSeleniumCapabilities ) { 18 | inputData.config = Config; 19 | inputData.config.parallelVirtualMachinesCount = parallelVirtualMachinesCount; 20 | inputData.buildId = ''; 21 | inputData.supportedSeleniumEnvironments = supportedSeleniumCapabilities; 22 | 23 | return inputData; 24 | }, 25 | supportedSeleniumEnvironmentsStub = {}; 26 | 27 | describe( 'ParallelTestPartitioner', function() { 28 | describe( '4 tests from one directory, 3 browsers, 3 virtual machines', function() { 29 | beforeEach( function() { 30 | fourTestsFromOneDirectoryInputData = 31 | augmentComponentManifest( fourTestsFromOneDirectoryInputData, 3, threeBrowsersSupportedSeleniumEnvironmentsStub ); 32 | } ); 33 | 34 | it( 'should return 3 buckets with 4 tests in each bucket', function( done ) { 35 | ParallelTestPartitioner.partitionComponentManifest( 36 | fourTestsFromOneDirectoryInputData ).then( 37 | function( testsBucketedByVirtualMachine ) { 38 | try { 39 | delete testsBucketedByVirtualMachine.componentManifest; 40 | 41 | assert.deepEqual( testsBucketedByVirtualMachine, threeBucketsWithFourTestsInEachBucket ); 42 | done(); 43 | } catch( error ) { 44 | done( error ); 45 | } 46 | } ); 47 | } ); 48 | } ); 49 | 50 | describe( '5 tests across two directories, 3 browsers, 7 virtual machines', function() { 51 | beforeEach( function() { 52 | fiveTestsAcrossTwoDirectoriesInputData = 53 | augmentComponentManifest( fiveTestsAcrossTwoDirectoriesInputData, 7, threeBrowsersSupportedSeleniumEnvironmentsStub ); 54 | } ); 55 | 56 | it( 'should return 7 buckets with a varying number of tests in each bucket', function( done ) { 57 | ParallelTestPartitioner.partitionComponentManifest( 58 | fiveTestsAcrossTwoDirectoriesInputData ).then( 59 | function( testsBucketedByVirtualMachine ) { 60 | try { 61 | delete testsBucketedByVirtualMachine.componentManifest; 62 | 63 | assert.deepEqual( testsBucketedByVirtualMachine, sevenBucketsWithAVaryingNumberOfTestsInEachBucket ); 64 | done(); 65 | } catch( error ) { 66 | done( error ); 67 | } 68 | } ); 69 | } ); 70 | } ); 71 | 72 | describe( '5 tests across two directories, 3 browsers, 25 virtual machines', function() { 73 | beforeEach( function() { 74 | fiveTestsAcrossTwoDirectoriesInputData = 75 | augmentComponentManifest( fiveTestsAcrossTwoDirectoriesInputData, 25, threeBrowsersSupportedSeleniumEnvironmentsStub ); 76 | } ); 77 | 78 | it( 'should return 15 buckets with 1 test in each bucket', function( done ) { 79 | ParallelTestPartitioner.partitionComponentManifest( 80 | fiveTestsAcrossTwoDirectoriesInputData ).then( 81 | function( testsBucketedByVirtualMachine ) { 82 | try { 83 | delete testsBucketedByVirtualMachine.componentManifest; 84 | 85 | assert.deepEqual( testsBucketedByVirtualMachine, fifteenBucketsWithOneTestInEachBucket ); 86 | done(); 87 | } catch( error ) { 88 | done( error ); 89 | } 90 | } ); 91 | } ); 92 | } ); 93 | } ); 94 | -------------------------------------------------------------------------------- /css-visual-test/test/data/FifteenBucketsWithOneTestInEachBucket.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | "testsBucketedByVirtualMachine": [{ 3 | "capability": { 4 | "build": "", 5 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 6 | "username": "davidjonathannelson", 7 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 8 | "browserName": "android", 9 | "platform": "Linux", 10 | "version": "4.4", 11 | "deviceName": "Android Emulator", 12 | "device-orientation": "portrait" 13 | }, 14 | "variations": [{ 15 | "componentName": "card", 16 | "direction": "ltr", 17 | "testDataNumber": 1, 18 | "sortingKey": "android-Linux-4.4-card-ltr-1" 19 | }], 20 | "partitionId": 1 21 | }, { 22 | "capability": { 23 | "build": "", 24 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 25 | "username": "davidjonathannelson", 26 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 27 | "browserName": "android", 28 | "platform": "Linux", 29 | "version": "4.4", 30 | "deviceName": "Android Emulator", 31 | "device-orientation": "portrait" 32 | }, 33 | "variations": [{ 34 | "componentName": "card", 35 | "direction": "ltr", 36 | "testDataNumber": 2, 37 | "sortingKey": "android-Linux-4.4-card-ltr-2" 38 | }], 39 | "partitionId": 2 40 | }, { 41 | "capability": { 42 | "build": "", 43 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 44 | "username": "davidjonathannelson", 45 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 46 | "browserName": "android", 47 | "platform": "Linux", 48 | "version": "4.4", 49 | "deviceName": "Android Emulator", 50 | "device-orientation": "portrait" 51 | }, 52 | "variations": [{ 53 | "componentName": "card", 54 | "direction": "rtl", 55 | "testDataNumber": 1, 56 | "sortingKey": "android-Linux-4.4-card-rtl-1" 57 | }], 58 | "partitionId": 3 59 | }, { 60 | "capability": { 61 | "build": "", 62 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 63 | "username": "davidjonathannelson", 64 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 65 | "browserName": "android", 66 | "platform": "Linux", 67 | "version": "4.4", 68 | "deviceName": "Android Emulator", 69 | "device-orientation": "portrait" 70 | }, 71 | "variations": [{ 72 | "componentName": "card", 73 | "direction": "rtl", 74 | "testDataNumber": 2, 75 | "sortingKey": "android-Linux-4.4-card-rtl-2" 76 | }], 77 | "partitionId": 4 78 | }, { 79 | "capability": { 80 | "build": "", 81 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 82 | "username": "davidjonathannelson", 83 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 84 | "browserName": "android", 85 | "platform": "Linux", 86 | "version": "4.4", 87 | "deviceName": "Android Emulator", 88 | "device-orientation": "portrait" 89 | }, 90 | "variations": [{ 91 | "componentName": "dialog", 92 | "direction": "ltr", 93 | "testDataNumber": 1, 94 | "sortingKey": "android-Linux-4.4-dialog-ltr-1" 95 | }], 96 | "partitionId": 5 97 | }, { 98 | "capability": { 99 | "build": "", 100 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 101 | "username": "davidjonathannelson", 102 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 103 | "browserName": "internet explorer", 104 | "platform": "Windows 7", 105 | "version": "11.0" 106 | }, 107 | "variations": [{ 108 | "componentName": "card", 109 | "direction": "ltr", 110 | "testDataNumber": 1, 111 | "sortingKey": "internet-explorer-Windows-7-11.0-card-ltr-1" 112 | }], 113 | "partitionId": 6 114 | }, { 115 | "capability": { 116 | "build": "", 117 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 118 | "username": "davidjonathannelson", 119 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 120 | "browserName": "internet explorer", 121 | "platform": "Windows 7", 122 | "version": "11.0" 123 | }, 124 | "variations": [{ 125 | "componentName": "card", 126 | "direction": "ltr", 127 | "testDataNumber": 2, 128 | "sortingKey": "internet-explorer-Windows-7-11.0-card-ltr-2" 129 | }], 130 | "partitionId": 7 131 | }, { 132 | "capability": { 133 | "build": "", 134 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 135 | "username": "davidjonathannelson", 136 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 137 | "browserName": "internet explorer", 138 | "platform": "Windows 7", 139 | "version": "11.0" 140 | }, 141 | "variations": [{ 142 | "componentName": "card", 143 | "direction": "rtl", 144 | "testDataNumber": 1, 145 | "sortingKey": "internet-explorer-Windows-7-11.0-card-rtl-1" 146 | }], 147 | "partitionId": 8 148 | }, { 149 | "capability": { 150 | "build": "", 151 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 152 | "username": "davidjonathannelson", 153 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 154 | "browserName": "internet explorer", 155 | "platform": "Windows 7", 156 | "version": "11.0" 157 | }, 158 | "variations": [{ 159 | "componentName": "card", 160 | "direction": "rtl", 161 | "testDataNumber": 2, 162 | "sortingKey": "internet-explorer-Windows-7-11.0-card-rtl-2" 163 | }], 164 | "partitionId": 9 165 | }, { 166 | "capability": { 167 | "build": "", 168 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 169 | "username": "davidjonathannelson", 170 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 171 | "browserName": "internet explorer", 172 | "platform": "Windows 7", 173 | "version": "11.0" 174 | }, 175 | "variations": [{ 176 | "componentName": "dialog", 177 | "direction": "ltr", 178 | "testDataNumber": 1, 179 | "sortingKey": "internet-explorer-Windows-7-11.0-dialog-ltr-1" 180 | }], 181 | "partitionId": 10 182 | }, { 183 | "capability": { 184 | "build": "", 185 | "name": "All CSS Visual Test components with all variations in iphone 8.2 on OS X 10.10.", 186 | "username": "davidjonathannelson", 187 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 188 | "browserName": "iphone", 189 | "platform": "OS X 10.10", 190 | "version": "8.2", 191 | "deviceName": "iPhone Simulator", 192 | "device-orientation": "portrait" 193 | }, 194 | "variations": [{ 195 | "componentName": "card", 196 | "direction": "ltr", 197 | "testDataNumber": 1, 198 | "sortingKey": "iphone-OS-X 10.10-8.2-card-ltr-1" 199 | }], 200 | "partitionId": 11 201 | }, { 202 | "capability": { 203 | "build": "", 204 | "name": "All CSS Visual Test components with all variations in iphone 8.2 on OS X 10.10.", 205 | "username": "davidjonathannelson", 206 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 207 | "browserName": "iphone", 208 | "platform": "OS X 10.10", 209 | "version": "8.2", 210 | "deviceName": "iPhone Simulator", 211 | "device-orientation": "portrait" 212 | }, 213 | "variations": [{ 214 | "componentName": "card", 215 | "direction": "ltr", 216 | "testDataNumber": 2, 217 | "sortingKey": "iphone-OS-X 10.10-8.2-card-ltr-2" 218 | }], 219 | "partitionId": 12 220 | }, { 221 | "capability": { 222 | "build": "", 223 | "name": "All CSS Visual Test components with all variations in iphone 8.2 on OS X 10.10.", 224 | "username": "davidjonathannelson", 225 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 226 | "browserName": "iphone", 227 | "platform": "OS X 10.10", 228 | "version": "8.2", 229 | "deviceName": "iPhone Simulator", 230 | "device-orientation": "portrait" 231 | }, 232 | "variations": [{ 233 | "componentName": "card", 234 | "direction": "rtl", 235 | "testDataNumber": 1, 236 | "sortingKey": "iphone-OS-X 10.10-8.2-card-rtl-1" 237 | }], 238 | "partitionId": 13 239 | }, { 240 | "capability": { 241 | "build": "", 242 | "name": "All CSS Visual Test components with all variations in iphone 8.2 on OS X 10.10.", 243 | "username": "davidjonathannelson", 244 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 245 | "browserName": "iphone", 246 | "platform": "OS X 10.10", 247 | "version": "8.2", 248 | "deviceName": "iPhone Simulator", 249 | "device-orientation": "portrait" 250 | }, 251 | "variations": [{ 252 | "componentName": "card", 253 | "direction": "rtl", 254 | "testDataNumber": 2, 255 | "sortingKey": "iphone-OS-X 10.10-8.2-card-rtl-2" 256 | }], 257 | "partitionId": 14 258 | }, { 259 | "capability": { 260 | "build": "", 261 | "name": "All CSS Visual Test components with all variations in iphone 8.2 on OS X 10.10.", 262 | "username": "davidjonathannelson", 263 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 264 | "browserName": "iphone", 265 | "platform": "OS X 10.10", 266 | "version": "8.2", 267 | "deviceName": "iPhone Simulator", 268 | "device-orientation": "portrait" 269 | }, 270 | "variations": [{ 271 | "componentName": "dialog", 272 | "direction": "ltr", 273 | "testDataNumber": 1, 274 | "sortingKey": "iphone-OS-X 10.10-8.2-dialog-ltr-1" 275 | }], 276 | "partitionId": 15 277 | }] 278 | }; 279 | -------------------------------------------------------------------------------- /css-visual-test/test/data/FiveTestsAcrossTwoDirectoriesInputData.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | "projectName": "CSS Visual Test", 3 | "variations": [ 4 | { 5 | "componentName": "card", 6 | "direction": "ltr", 7 | "testDataNumber": 1, 8 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/ltr-1.jsx" 9 | }, 10 | { 11 | "componentName": "card", 12 | "direction": "ltr", 13 | "testDataNumber": 2, 14 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/ltr-2.jsx" 15 | }, 16 | { 17 | "componentName": "card", 18 | "direction": "rtl", 19 | "testDataNumber": 1, 20 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/rtl-1.jsx" 21 | }, 22 | { 23 | "componentName": "card", 24 | "direction": "rtl", 25 | "testDataNumber": 2, 26 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/rtl-2.jsx" 27 | }, 28 | { 29 | "componentName": "dialog", 30 | "direction": "ltr", 31 | "testDataNumber": 1, 32 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/dialog/visualtests/ltr-1.jsx" 33 | } 34 | ] 35 | }; 36 | -------------------------------------------------------------------------------- /css-visual-test/test/data/FourTestsFromOneDirectoryInputData.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | "projectName": "CSS Visual Test", 3 | "variations": [ 4 | { 5 | "componentName": "card", 6 | "direction": "ltr", 7 | "testDataNumber": 1, 8 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/ltr-1.jsx" 9 | }, 10 | { 11 | "componentName": "card", 12 | "direction": "ltr", 13 | "testDataNumber": 2, 14 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/ltr-2.jsx" 15 | }, 16 | { 17 | "componentName": "card", 18 | "direction": "rtl", 19 | "testDataNumber": 1, 20 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/rtl-1.jsx" 21 | }, 22 | { 23 | "componentName": "card", 24 | "direction": "rtl", 25 | "testDataNumber": 2, 26 | "bootstrapJsxFileFullPath": "/Users/david/workspace/calypso-pre-oss/client/card/visualtests/rtl-2.jsx" 27 | } 28 | ] 29 | }; 30 | -------------------------------------------------------------------------------- /css-visual-test/test/data/SevenBucketsWithAVaryingNumberOfTestsInEachBucket.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | "testsBucketedByVirtualMachine": [{ 3 | "capability": { 4 | "build": "", 5 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 6 | "username": "davidjonathannelson", 7 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 8 | "browserName": "android", 9 | "platform": "Linux", 10 | "version": "4.4", 11 | "deviceName": "Android Emulator", 12 | "device-orientation": "portrait" 13 | }, 14 | "variations": [{ 15 | "componentName": "card", 16 | "direction": "ltr", 17 | "testDataNumber": 1, 18 | "sortingKey": "android-Linux-4.4-card-ltr-1" 19 | }, { 20 | "componentName": "card", 21 | "direction": "ltr", 22 | "testDataNumber": 2, 23 | "sortingKey": "android-Linux-4.4-card-ltr-2" 24 | }], 25 | "groupingKey": "android-Linux-4.4", 26 | "partitionId": 1 27 | }, { 28 | "capability": { 29 | "build": "", 30 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 31 | "username": "davidjonathannelson", 32 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 33 | "browserName": "android", 34 | "platform": "Linux", 35 | "version": "4.4", 36 | "deviceName": "Android Emulator", 37 | "device-orientation": "portrait" 38 | }, 39 | "variations": [{ 40 | "componentName": "card", 41 | "direction": "rtl", 42 | "testDataNumber": 1, 43 | "sortingKey": "android-Linux-4.4-card-rtl-1" 44 | }, { 45 | "componentName": "card", 46 | "direction": "rtl", 47 | "testDataNumber": 2, 48 | "sortingKey": "android-Linux-4.4-card-rtl-2" 49 | }], 50 | "groupingKey": "android-Linux-4.4", 51 | "partitionId": 2 52 | }, { 53 | "capability": { 54 | "build": "", 55 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 56 | "username": "davidjonathannelson", 57 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 58 | "browserName": "android", 59 | "platform": "Linux", 60 | "version": "4.4", 61 | "deviceName": "Android Emulator", 62 | "device-orientation": "portrait" 63 | }, 64 | "variations": [{ 65 | "componentName": "dialog", 66 | "direction": "ltr", 67 | "testDataNumber": 1, 68 | "sortingKey": "android-Linux-4.4-dialog-ltr-1" 69 | }], 70 | "groupingKey": "android-Linux-4.4", 71 | "partitionId": 3 72 | }, { 73 | "capability": { 74 | "build": "", 75 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 76 | "username": "davidjonathannelson", 77 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 78 | "browserName": "internet explorer", 79 | "platform": "Windows 7", 80 | "version": "11.0" 81 | }, 82 | "variations": [{ 83 | "componentName": "card", 84 | "direction": "ltr", 85 | "testDataNumber": 1, 86 | "sortingKey": "internet-explorer-Windows-7-11.0-card-ltr-1" 87 | }, { 88 | "componentName": "card", 89 | "direction": "ltr", 90 | "testDataNumber": 2, 91 | "sortingKey": "internet-explorer-Windows-7-11.0-card-ltr-2" 92 | }], 93 | "groupingKey": "internet-explorer-Windows-7-11.0", 94 | "partitionId": 4 95 | }, { 96 | "capability": { 97 | "build": "", 98 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 99 | "username": "davidjonathannelson", 100 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 101 | "browserName": "internet explorer", 102 | "platform": "Windows 7", 103 | "version": "11.0" 104 | }, 105 | "variations": [{ 106 | "componentName": "card", 107 | "direction": "rtl", 108 | "testDataNumber": 1, 109 | "sortingKey": "internet-explorer-Windows-7-11.0-card-rtl-1" 110 | }, { 111 | "componentName": "card", 112 | "direction": "rtl", 113 | "testDataNumber": 2, 114 | "sortingKey": "internet-explorer-Windows-7-11.0-card-rtl-2" 115 | }], 116 | "groupingKey": "internet-explorer-Windows-7-11.0", 117 | "partitionId": 5 118 | }, { 119 | "capability": { 120 | "build": "", 121 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 122 | "username": "davidjonathannelson", 123 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 124 | "browserName": "internet explorer", 125 | "platform": "Windows 7", 126 | "version": "11.0" 127 | }, 128 | "variations": [{ 129 | "componentName": "dialog", 130 | "direction": "ltr", 131 | "testDataNumber": 1, 132 | "sortingKey": "internet-explorer-Windows-7-11.0-dialog-ltr-1" 133 | }], 134 | "groupingKey": "internet-explorer-Windows-7-11.0", 135 | "partitionId": 6 136 | }, { 137 | "capability": { 138 | "build": "", 139 | "name": "All CSS Visual Test components with all variations in iphone 8.2 on OS X 10.10.", 140 | "username": "davidjonathannelson", 141 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 142 | "browserName": "iphone", 143 | "platform": "OS X 10.10", 144 | "version": "8.2", 145 | "deviceName": "iPhone Simulator", 146 | "device-orientation": "portrait" 147 | }, 148 | "variations": [{ 149 | "componentName": "card", 150 | "direction": "ltr", 151 | "testDataNumber": 1, 152 | "sortingKey": "iphone-OS-X 10.10-8.2-card-ltr-1" 153 | }, { 154 | "componentName": "card", 155 | "direction": "ltr", 156 | "testDataNumber": 2, 157 | "sortingKey": "iphone-OS-X 10.10-8.2-card-ltr-2" 158 | }, { 159 | "componentName": "card", 160 | "direction": "rtl", 161 | "testDataNumber": 1, 162 | "sortingKey": "iphone-OS-X 10.10-8.2-card-rtl-1" 163 | }, { 164 | "componentName": "card", 165 | "direction": "rtl", 166 | "testDataNumber": 2, 167 | "sortingKey": "iphone-OS-X 10.10-8.2-card-rtl-2" 168 | }, { 169 | "componentName": "dialog", 170 | "direction": "ltr", 171 | "testDataNumber": 1, 172 | "sortingKey": "iphone-OS-X 10.10-8.2-dialog-ltr-1" 173 | }], 174 | "groupingKey": "iphone-OS-X 10.10-8.2", 175 | "partitionId": 7 176 | }] 177 | } 178 | ; 179 | -------------------------------------------------------------------------------- /css-visual-test/test/data/ThreeBrowsersSupportedSeleniumEnvironmentsStub.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | "environments": [ 3 | { 4 | "browserName": "internet explorer", 5 | "platform": "Windows 7", 6 | "version": "11.0" 7 | }, 8 | { 9 | "browserName": "iphone", 10 | "platform": "OS X 10.10", 11 | "version": "8.2", 12 | "deviceName": "iPhone Simulator", 13 | "device-orientation": "portrait" 14 | }, 15 | { 16 | "browserName": "android", 17 | "platform": "Linux", 18 | "version": "4.4", 19 | "deviceName": "Android Emulator", 20 | "device-orientation": "portrait" 21 | } 22 | ] 23 | }; 24 | -------------------------------------------------------------------------------- /css-visual-test/test/data/ThreeBucketsWithFourTestsInEachBucket.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | "testsBucketedByVirtualMachine": [{ 3 | "capability": { 4 | "build": "", 5 | "name": "All CSS Visual Test components with all variations in android 4.4 on Linux.", 6 | "username": "davidjonathannelson", 7 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 8 | "browserName": "android", 9 | "platform": "Linux", 10 | "version": "4.4", 11 | "deviceName": "Android Emulator", 12 | "device-orientation": "portrait" 13 | }, 14 | "variations": [{ 15 | "componentName": "card", 16 | "direction": "ltr", 17 | "testDataNumber": 1, 18 | "sortingKey": "android-Linux-4.4-card-ltr-1" 19 | }, { 20 | "componentName": "card", 21 | "direction": "ltr", 22 | "testDataNumber": 2, 23 | "sortingKey": "android-Linux-4.4-card-ltr-2" 24 | }, { 25 | "componentName": "card", 26 | "direction": "rtl", 27 | "testDataNumber": 1, 28 | "sortingKey": "android-Linux-4.4-card-rtl-1" 29 | }, { 30 | "componentName": "card", 31 | "direction": "rtl", 32 | "testDataNumber": 2, 33 | "sortingKey": "android-Linux-4.4-card-rtl-2" 34 | }], 35 | "groupingKey": "android-Linux-4.4", 36 | "partitionId": 1 37 | }, { 38 | "capability": { 39 | "build": "", 40 | "name": "All CSS Visual Test components with all variations in internet explorer 11.0 on Windows 7.", 41 | "username": "davidjonathannelson", 42 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 43 | "browserName": "internet explorer", 44 | "platform": "Windows 7", 45 | "version": "11.0" 46 | }, 47 | "variations": [{ 48 | "componentName": "card", 49 | "direction": "ltr", 50 | "testDataNumber": 1, 51 | "sortingKey": "internet-explorer-Windows-7-11.0-card-ltr-1" 52 | }, { 53 | "componentName": "card", 54 | "direction": "ltr", 55 | "testDataNumber": 2, 56 | "sortingKey": "internet-explorer-Windows-7-11.0-card-ltr-2" 57 | }, { 58 | "componentName": "card", 59 | "direction": "rtl", 60 | "testDataNumber": 1, 61 | "sortingKey": "internet-explorer-Windows-7-11.0-card-rtl-1" 62 | }, { 63 | "componentName": "card", 64 | "direction": "rtl", 65 | "testDataNumber": 2, 66 | "sortingKey": "internet-explorer-Windows-7-11.0-card-rtl-2" 67 | }], 68 | "groupingKey": "internet-explorer-Windows-7-11.0", 69 | "partitionId": 2 70 | }, { 71 | "capability": { 72 | "build": "", 73 | "name": "All CSS Visual Test components with all variations in iphone 8.2 on OS X 10.10.", 74 | "username": "davidjonathannelson", 75 | "accessKey": "07de745a-647a-4b5e-9caf-0522d638bf2f", 76 | "browserName": "iphone", 77 | "platform": "OS X 10.10", 78 | "version": "8.2", 79 | "deviceName": "iPhone Simulator", 80 | "device-orientation": "portrait" 81 | }, 82 | "variations": [{ 83 | "componentName": "card", 84 | "direction": "ltr", 85 | "testDataNumber": 1, 86 | "sortingKey": "iphone-OS-X 10.10-8.2-card-ltr-1" 87 | }, { 88 | "componentName": "card", 89 | "direction": "ltr", 90 | "testDataNumber": 2, 91 | "sortingKey": "iphone-OS-X 10.10-8.2-card-ltr-2" 92 | }, { 93 | "componentName": "card", 94 | "direction": "rtl", 95 | "testDataNumber": 1, 96 | "sortingKey": "iphone-OS-X 10.10-8.2-card-rtl-1" 97 | }, { 98 | "componentName": "card", 99 | "direction": "rtl", 100 | "testDataNumber": 2, 101 | "sortingKey": "iphone-OS-X 10.10-8.2-card-rtl-2" 102 | }], 103 | "groupingKey": "iphone-OS-X 10.10-8.2", 104 | "partitionId": 3 105 | }] 106 | } 107 | ; 108 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "css-visual-test", 3 | "version": "0.1.0", 4 | "description": "Visually test your javascript and css components.", 5 | "private": true, 6 | "repository": { 7 | "type": "git", 8 | "url": "https://github.com/davidjnelson/css-visual-test.git" 9 | }, 10 | "main": "index.js", 11 | "engines": { 12 | "node": ">=0.8" 13 | }, 14 | "devDependencies": { 15 | "browserify": "8.1.3", 16 | "chai": "2.0.0", 17 | "connect": "2.14.5", 18 | "eyes.selenium": "0.0.32", 19 | "fs-extra": "0.16.5", 20 | "jade": "1.5.0", 21 | "lodash": "3.4.0", 22 | "mocha": "2.1.0", 23 | "node-uuid": "1.4.3", 24 | "portscanner": "1.0.0", 25 | "ps-aux": "0.2.1", 26 | "q": "1.0.1", 27 | "react": "0.12.1", 28 | "react-tools": "0.13.1", 29 | "reactify": "0.17.1", 30 | "readdirp": "1.3.0", 31 | "sauce-connect-launcher": "^0.10.1", 32 | "selenium-webdriver": "2.45.1", 33 | "winston": "1.0.0", 34 | "wrench": "1.5.8" 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /public/style-rtl.css: -------------------------------------------------------------------------------- 1 | .card { 2 | position: relative; 3 | margin: 0 auto 10px auto; 4 | padding: 16px; 5 | box-sizing: border-box; 6 | background: white; 7 | box-shadow: 0 0 0 1px rgba(200, 215, 225, 0.5), 0 1px 2px #e9eff3; } 8 | .card:after { 9 | content: "."; 10 | display: block; 11 | height: 0; 12 | clear: both; 13 | visibility: hidden; } 14 | @media only screen and (min-width: 480px) { 15 | .card { 16 | margin-bottom: 16px; 17 | padding: 24px; } } 18 | -------------------------------------------------------------------------------- /public/style.css: -------------------------------------------------------------------------------- 1 | .card { 2 | position: relative; 3 | margin: 0 auto 10px auto; 4 | padding: 16px; 5 | box-sizing: border-box; 6 | background: white; 7 | box-shadow: 0 0 0 1px rgba(200, 215, 225, 0.5), 0 1px 2px #e9eff3; } 8 | .card:after { 9 | content: "."; 10 | display: block; 11 | height: 0; 12 | clear: both; 13 | visibility: hidden; } 14 | @media only screen and (min-width: 480px) { 15 | .card { 16 | margin-bottom: 16px; 17 | padding: 24px; } } 18 | -------------------------------------------------------------------------------- /server/pages/index.jade: -------------------------------------------------------------------------------- 1 | doctype html 2 | html(lang=lang, dir=isRTL ? 'rtl' : 'ltr') 3 | head 4 | title CSS Visual Test 5 | meta(charset='utf-8') 6 | meta(name='viewport', content='width=device-width, initial-scale=1, maximum-scale=1') 7 | meta(name='fomat-detection', content='telephone=no') 8 | meta(name='mobile-web-app-capable', content='yes') 9 | link(rel='stylesheet', href='//fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,600italic,400,300,600|Merriweather:700,300,700italic,300italic') 10 | if isRTL 11 | link(rel='stylesheet', href=urls['style-rtl.css']) 12 | else 13 | link(rel='stylesheet', href=urls['style.css']) 14 | body(class=isRTL ? 'rtl' : '') 15 | #root 16 | script(src=urls[ 'build.js']) 17 | --------------------------------------------------------------------------------