9 |
10 | **CSS regression testing**. A [CasperJS](http://github.com/n1k0/casperjs) module for automating visual regression testing with [PhantomJS 2](http://github.com/ariya/phantomjs/) or [SlimerJS](http://slimerjs.org/) and [Resemble.js](http://huddle.github.com/Resemble.js/). For testing Web apps, live style guides and responsive layouts. Read more on Huddle's Engineering blog: [CSS Regression Testing](http://tldr.huddle.com/blog/css-testing/).
11 |
12 | **Huddle is Hiring!** [We're looking for talented front-end engineers](https://talentcommunity.huddle.com/careers)
13 |
14 | ### What?
15 |
16 | PhantomCSS takes screenshots captured by CasperJS and compares them to baseline images using [Resemble.js](http://huddle.github.com/Resemble.js/) to test for rgb pixel differences. PhantomCSS then generates image diffs to help you find the cause.
17 |
18 | 
19 |
20 | Screenshot based regression testing can only work when UI is predictable. It's possible to hide mutable UI components with PhantomCSS but it would be better to test static pages or drive the UI with faked data during test runs.
21 |
22 | ### Example
23 |
24 | ```javascript
25 | casper.
26 | start( url ).
27 | then(function(){
28 |
29 | // do something
30 | casper.click('button#open-dialog');
31 |
32 | // Take a screenshot of the UI component
33 | phantomcss.screenshot('#the-dialog', 'a screenshot of my dialog');
34 |
35 | });
36 | ```
37 |
38 | From the command line/terminal run:
39 |
40 | * `casperjs test demo/testsuite.js`
41 |
42 | or
43 |
44 | * `casperjs test demo/testsuite.js --verbose --log-level=debug --xunit=results.xml`
45 |
46 | ### Updating from PhantomCSS v0 to v1
47 |
48 | Rendering is quite different with PhantomJS 2, so when you update, old visual tests will start failing.
49 | If your tests are green and passing before updating, I would recommend rebasing the visual tests, i.e. delete them, and run the test suite to create a new baseline.
50 |
51 | You can still use the [v0 branch](https://github.com/Huddle/PhantomCSS/tree/v0) if you wish, though it now unmaintained.
52 |
53 | ### Download
54 |
55 | PhantomCSS can be downloaded in various ways:
56 |
57 | * `npm install phantomcss` (PhantomCSS is not itself a Node.js module)
58 | * `bower install phantomcss`
59 | * `git clone git://github.com/Huddle/PhantomCSS.git`
60 |
61 | If you are not installing via NPM, you will need to run `npm install` in the PhantomCSS root folder.
62 |
63 | Please note that depending on how you have installed PhantomCSS you may need to set the libraryRoot configuration property to link to the directory in which phantomcss.js resides.
64 |
65 | ### Getting started, try the demo
66 |
67 | * For convenience I've included CasperJS.bat for Windows users. If you are not a Windows user, you will have to install the latest version of [CasperJS](http://docs.casperjs.org/en/latest/installation.html).
68 | * Download or clone this repo and run `casperjs test demo/testsuite.js` in command/terminal from the PhantomCSS folder. PhantomJS is the only binary dependency - this should just work
69 | * Find the screenshot folder and have a look at the (baseline) images
70 | * Run the tests again with `casperjs test demo/testsuite.js`. New screenshots will be created to compare against the baseline images. These new images can be ignored, they will be replaced every test run.
71 | * To test failure, add/change some CSS in the file demo/coffeemachine.html e.g. make `.mug` bright green
72 | * Run the tests again, you should see some reported failures
73 | * In the failures folder some images should have been created. The images should show bright pink where the screenshot has visually changed
74 | * If you want to manually compare the images, go to the screenshot folder to see the original/baseline and latest screenshots
75 |
76 | ### SlimerJS
77 |
78 | SlimerJS uses the Gecko browser engine rather than Webkit. This has some advantages over PhantomJS, such as a non-headless view. If this is of interest to you, please follow the [download and install](http://slimerjs.org/download.html) instructions and ensure SlimerJS is installed globally.
79 |
80 | * `casperjs test demo/testsuite.js --engine=slimerjs`
81 |
82 | ### Options and setup
83 |
84 | If you are using SlimerJS, you will need to specify absolute paths (see 'demo').
85 |
86 | ```javascript
87 | phantomcss.init({
88 |
89 | /*
90 | captureWaitEnabled defaults to true, setting to false will remove a small wait/delay on each
91 | screenshot capture - useful when you don't need to worry about
92 | animations and latency in your visual tests
93 | */
94 | captureWaitEnabled: true,
95 |
96 | /*
97 | libraryRoot is now optional unless you are using SlimerJS where
98 | you will need to set it to the correct path. It must point to
99 | your phantomcss folder. If you are using NPM, this will probably
100 | be './node_modules/phantomcss'.
101 | */
102 | libraryRoot: './modules/PhantomCSS',
103 |
104 | screenshotRoot: './screenshots',
105 |
106 | /*
107 | By default, failure images are put in the './failures' folder.
108 | If failedComparisonsRoot is set to false a separate folder will
109 | not be created but failure images can still be found alongside
110 | the original and new images.
111 | */
112 | failedComparisonsRoot: './failures',
113 |
114 | /*
115 | Remove results directory tree after run. Use in conjunction
116 | with failedComparisonsRoot to see failed comparisons.
117 | */
118 | cleanupComparisonImages: true,
119 |
120 | /*
121 | A reference to a particular Casper instance. Required for SlimerJS.
122 | */
123 | casper: specific_instance_of_casper,
124 |
125 | /*
126 | You might want to keep master/baseline images in a completely
127 | different folder to the diffs/failures. Useful when working
128 | with version control systems. By default this resolves to the
129 | screenshotRoot folder.
130 | */
131 | comparisonResultRoot: './results',
132 |
133 | /*
134 | Don't add count number to images. If set to false, a filename is
135 | required when capturing screenshots.
136 | */
137 | addIteratorToImage: false,
138 |
139 | /*
140 | Don't add label to generated failure image
141 | */
142 | addLabelToFailedImage: false,
143 |
144 | /*
145 | Mismatch tolerance defaults to 0.05%. Increasing this value
146 | will decrease test coverage
147 | */
148 | mismatchTolerance: 0.05,
149 |
150 | /*
151 | Callbacks for your specific integration
152 | */
153 | onFail: function(test){ console.log(test.filename, test.mismatch); },
154 |
155 | onPass: function(test){ console.log(test.filename); },
156 |
157 | /*
158 | Called when creating new baseline images
159 | */
160 | onNewImage: function(){ console.log(test.filename); },
161 |
162 | onTimeout: function(){ console.log(test.filename); },
163 |
164 | onComplete: function(allTests, noOfFails, noOfErrors){
165 | allTests.forEach(function(test){
166 | if(test.fail){
167 | console.log(test.filename, test.mismatch);
168 | }
169 | });
170 | },
171 |
172 | onCaptureFail: function(ex, target) { console.log('Capture of ' + target + ' failed due to ' + ex.message); }
173 |
174 | /*
175 | Change the output screenshot filenames for your specific
176 | integration
177 | */
178 | fileNameGetter: function(root,filename){
179 | // globally override output filename
180 | // files must exist under root
181 | // and use the .diff convention
182 | var name = root+'/somewhere/'+filename;
183 | if(fs.isFile(name+'.png')){
184 | return name+'.diff.png';
185 | } else {
186 | return name+'.png';
187 | }
188 | },
189 |
190 | /*
191 | Prefix the screenshot number to the filename, instead of suffixing it
192 | */
193 | prefixCount: true,
194 |
195 | /*
196 | Output styles for image failure outputs generated by Resemble.js
197 | */
198 | outputSettings: {
199 | errorColor: {
200 | red: 255,
201 | green: 255,
202 | blue: 0
203 | },
204 | errorType: 'movement',
205 | transparency: 0.3
206 | },
207 |
208 | /*
209 | Rebase is useful when you want to create new baseline
210 | images without manually deleting the files
211 | casperjs demo/test.js --rebase
212 | */
213 | rebase: casper.cli.get("rebase"),
214 |
215 | /*
216 | If true, test will fail when captures fail (e.g. no element matching selector).
217 | */
218 | failOnCaptureError: false
219 | });
220 |
221 | /*
222 | Turn off CSS transitions and jQuery animations
223 | */
224 | phantomcss.turnOffAnimations();
225 | ```
226 |
227 | ### Don't like pink?
228 |
229 | 
230 |
231 | ```javascript
232 | phantomcss.init({
233 | /*
234 | Output styles for image failure outputs generated by Resemble.js
235 | */
236 | outputSettings: {
237 |
238 | /*
239 | Error pixel color, RGB, anything you want,
240 | though bright and ugly works best!
241 | */
242 | errorColor: {
243 | red: 255,
244 | green: 255,
245 | blue: 0
246 | },
247 |
248 | /*
249 | ErrorType values include 'flat', or 'movement'.
250 | The latter merges error color with base image
251 | which makes it a little easier to spot movement.
252 | */
253 | errorType: 'movement',
254 |
255 | /*
256 | Fade unchanged areas to make changed areas more apparent.
257 | */
258 | transparency: 0.3
259 | }
260 | });
261 | ```
262 |
263 | ### There are different ways to take a screenshot
264 |
265 | ```javascript
266 | var delay = 10;
267 | var hideElements = 'input[type=file]';
268 | var screenshotName = 'the_dialog'
269 |
270 | phantomcss.screenshot( "#CSS .selector", screenshotName);
271 |
272 | // phantomcss.screenshot({
273 | // 'Screenshot 1 File name': {selector: '.screenshot1', ignore: '.selector'},
274 | // 'Screenshot 2 File name': '#screenshot2'
275 | // });
276 | // phantomcss.screenshot( "#CSS .selector" );
277 | // phantomcss.screenshot( "#CSS .selector", delay, hideElements, screenshotName);
278 |
279 | // phantomcss.screenshot({
280 | // top: 100,
281 | // left: 100,
282 | // width: 500,
283 | // height: 400
284 | // }, screenshotName);
285 | ```
286 |
287 | ### Compare the images when and how you want
288 |
289 | ```javascript
290 | /*
291 | String is converted into a Regular expression that matches on full image path
292 | */
293 | phantomcss.compareAll('exclude.test');
294 |
295 | // phantomcss.compareMatched('include.test', 'exclude.test');
296 | // phantomcss.compareMatched( new RegExp('include.test'), new RegExp('exclude.test'));
297 |
298 | /*
299 | Compare image diffs generated in this test run only
300 | */
301 | // phantomcss.compareSession();
302 |
303 | /*
304 | Explicitly define what files you want to compare
305 | */
306 | // phantomcss.compareExplicit(['/dialog.diff.png', '/header.diff.png']);
307 |
308 | /*
309 | Get a list of image diffs generated in this test run
310 | */
311 | // phantomcss.getCreatedDiffFiles();
312 |
313 | /*
314 | Compare any two images, and wait for the results to complete
315 | */
316 | // phantomcss.compareFiles(baseFile, diffFile);
317 | // phantomcss.waitForTests();
318 |
319 | ```
320 |
321 | ### Best Practices
322 |
323 | ##### Name your screenshots!
324 |
325 | By default PhantomCSS creates a file called screenshot_0.png, not very helpful. You can name your screenshot by passing a string to either the second or forth parameter.
326 |
327 | ```javascript
328 | var delay, hideElementsSelector;
329 |
330 | phantomcss.screenshot("#feedback-form", delay, hideElementsSelector, "Responsive Feedback Form");
331 |
332 | phantomcss.screenshot("#feedback-form", "Responsive Feedback Form");
333 |
334 | ```
335 |
336 | Perhaps a better way is to use the ‘fileNameGetter’ callback property on the ‘init’ method. This does involve having a bit more structure around your tests. See: https://github.com/Huddle/PhantomFlow/blob/master/lib/phantomCSSAdaptor.js#L41
337 |
338 | ##### CSS3 selectors for testing
339 |
340 | Try not to use complex CSS3 selectors for asserting or creating screenshots. In the same way that CSS should be written with good content/container separation, so should your test selectors be agnostic of location/context. This might mean you need to add more ID's or data- attributes into your mark-up, but it's worth it, your tests will be more stable and more explicit.
341 | This is not a good idea:
342 |
343 | ```javascript
344 | phantomcss.screenshot("#sidebar li:nth-child(3) > div form");
345 | ```
346 |
347 | But this is:
348 |
349 | ```javascript
350 | phantomcss.screenshot("#feedback-form");
351 | ```
352 |
353 | ##### PhantomCSS should not be used to replace functional tests
354 |
355 | If you needed functional tests before, then you still need them. Automated visual regression testing gives us coverage of CSS and design in a way we didn't have before, but that doesn't mean that conventional test assertions are now invalid. Feedback time is crucial with test automation, the longer it takes the easier it is to ignore; the easier it is to ignore the sooner trust is lost from the team. Unfortunately comparing images is not, and never will be as fast as simple DOM assertion.
356 |
357 | ##### Don't try to test all the visuals
358 |
359 | I'd argue this applies to all automated testing approaches. As a rule, try to maximise coverage with fewer tests. This is a difficult balancing act because granular feedback/reporting is very important for debugging and build analysis. Testing many things in one assert/screenshot might tell you there is a problem, but makes it harder to get to the root of the bug. As a CSS/HTML Dev you'll know what components are more fragile than others, which are reused and which aren't, concentrate your visual tests on these areas.
360 |
361 | ##### Full page screenshots are a bad idea
362 |
363 | If you try to test too much in one screenshot then you could end up with lots of failing tests every time someone makes a small change. Say you've set up full-page visual regression tests for your 50 page website, and someone adds 2px padding to the footer - that’s 50 failed tests because of one change. It's better to test UI components individually; in this example the footer could have its own test.
364 | There is also a technical problem with this approach, the larger the image, the longer it takes to process. An added pixel padding on the page body will offset everything, at best you'll have a sea of pink in the failed diff, at worse you'll get a TIMEOUT because it took too long to analyse.
365 |
366 | ##### Scaling visual regression testing within a large team
367 |
368 | Scaling your test suite for many contributors may not be easy. [Resemble.js](http://huddle.github.com/Resemble.js/) (the core analysis engine of PhantomCSS) tries to consider image differences caused by different operating systems and graphics cards, but it's only so good, you are likely to see problems as more people contribute baseline screenshots. You can mitigate this by hiding problematic elements such as select elements, file upload inputs etc. as so.
369 |
370 | ```javascript
371 | phantomcss.screenshot("#feedback-form", undefined, 'input[type=file]');
372 | ```
373 |
374 | Below is an example of a false-negative caused by antialiasing differences on different machines. How can we solve this? **Contributions welcome!**
375 |
376 | 
377 |
378 | ##### Scaling visual regression testing with Git
379 |
380 | If you're using a version control system like Git to store the baseline screenshots the repository size becomes increasingly relevant as your test suite grows. I'd recommend using a tool like https://github.com/joeyh/git-annex or https://github.com/schacon/git-media to store the images outside of the repo.
381 |
382 | ### ...You might also be interested in
383 |
384 | **[PhantomFlow](https://github.com/Huddle/PhantomFlow)** and **[grunt-phantomflow](https://github.com/Huddle/grunt-testflow)** wrap PhantomCSS and provides an experimental way of describing and visualising user flows through tests with CasperJS. As well as providing a terse readable structure for UI testing, it also produces intriguing graph visualisations that can be used to present PhantomCSS screenshots and failed diffs. We're actively using it at Huddle and it's changing the way we think about UI for the better.
385 |
386 | Also, take a look at [PhantomXHR](http://github.com/Huddle/PhantomXHR) for stubbing and mocking XHR requests. Isolated UI testing IS THE FUTURE!
387 |
388 | ### Why is this project no longer maintained
389 |
390 | The introduction of [headless Chrome](https://developers.google.com/web/updates/2017/04/headless-chrome) has simply meant that PhantomJS is no longer the best tool for running browser tests. Huddle is making efforts to move away from PhantomJS based testing, largely to gain better coverage of new browser features such as CSS grid. Interestingly there still doesn't seem to be a straight replacement for PhantomCSS for Chrome, perhaps because it is now far easier to roll-your-own VRT suite. The Huddle development team is now actively looking into using Docker containers for running Mocha/Chai test suites against headless Chrome, using [Resemblejs](https://github.com/Huddle/Resemble.js) directly in NodeJS for image comparison.
391 |
392 | ### Huddle Careers
393 |
394 | Huddle strongly believe in innovation and give you 20% of work time to spend on innovative projects of your choosing.
395 |
396 | If you like what you see and would like to work on this kind of stuff for a job then get in touch.
397 |
398 | Visit http://www.huddle.com/careers for open vacancies now, or register your interest for the future.
399 |
400 | --------------------------------------
401 |
402 | PhantomCSS was created by [James Cryer](http://github.com/jamescryer) and the Huddle development team.
403 |
--------------------------------------------------------------------------------
/casperjs.bat:
--------------------------------------------------------------------------------
1 | @ECHO OFF
2 | set CASPER_PATH=%~dp0node_modules\casperjs
3 | set CASPER_BIN=%CASPER_PATH%\bin\
4 | set PHANTOMJS=%~dp0node_modules\phantomjs-prebuilt\lib\phantom\bin\phantomjs.exe
5 | set ARGV=%*
6 | set IS_SLIMERJS="false"
7 |
8 | for %%a in (%*) do (
9 | if %%a equ slimerjs (
10 | set IS_SLIMERJS="true"
11 | )
12 | )
13 |
14 | if %IS_SLIMERJS% equ "false" (
15 | call "%PHANTOMJS%" "%CASPER_BIN%bootstrap.js" --casper-path="%CASPER_PATH%" --cli %ARGV%
16 | ) else (
17 | call slimerjs "%CASPER_BIN%bootstrap.js" --casper-path="%CASPER_PATH%" --cli %ARGV%
18 | )
--------------------------------------------------------------------------------
/demo/coffeemachine.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 | Would you like a coffee?
24 | 25 | Go to the Coffee Machine 26 | 27 | 28 | 72 |