Use automated visual regression testing to run a pixel-by-pixel comparison between two different versions of your site.
Note: Visual regression testing will not work with content that requires authentication to view.
In this article:
Getting Started with Backstop
- Create a config file for backstop:
- cp ./backstop/config/default.sitesToTest.js ./backstop/config/sitesToTest.js
- Set the values in the exported object array in the newly created sitesToTest.js:
- On line 3: Change the key name to the domain name of your site.
- On line 5: Change the label to the same domain name.
- On lines 7 & 8: Set the productionBaseUrl and nonProductionBaseUrl to the two sites you wish to compare.
- On line 10: Update the pathsToTest array to include the pages that exist on your two sites that you wish to visually compare. You should remove the existing paths as they are specific for the platform's Folwell regression testing.
- You can commit the custom sitesToTest.js configuration you created to your forked repository.
- git add ./backstop/config/sitesToTest.js
- git commit -m "my config!"
- git push origin [my-branch]
- Run npm run backstop and select the site you added to your configuration.
- After the tests are finished, a browser will open with a web ui that showcases any visual differences it finds between the sites that require manual inspection.
Viewing Your Results
- You will see three tabs categorizing the results: all, passed, and failed.
- When you have failed tests, this simply means that Backstop found visual discrepancies between the two sites, and you should manually review them.
- Review the failed tests:
- You will be presented with three images labeled REFERENCE, TEST, and DIFF.
- REFERENCE is your production site.
- TEST is your non-production site.
- DIFF displays an image of both sites combined with the differences highlighted in magenta.
- If you click into one of the images, it will open a large view of the image along with an interface to switch between them. You will also see the option labeled SCRUBBER.
- The scrubber interface lets you use a draggable scrubber to slide between the two versions of the site. The left side is the production site and the right side is the non-production site.
- When reviewing failed tests, keep in mind that some things could be caused by non-code related discrepancies, such as:
- Images missing from the non-production site.
- Configuration and content differences between the two sites.
- Different timing in the loading and rendering of elements such as applying font-smoothing, or javascript reliant elements.
- You will be presented with three images labeled REFERENCE, TEST, and DIFF.