Visual regression testing ensures that your application’s user interface is not adversely affected when you release code changes. In this guide, we’ll discuss six of the many open-source and software as a service (SaaS) visual regression testing tools your team can start using today. Let’s dive in!
What Is Visual Regression Testing? In brief, visual regression testing verifies that a webpage’s elements are rendered properly from a visual point of view. They’re automated tests, so they can be run as part of a CI/CD pipeline; with a mature CI/CD pipeline, the correctness of the visual elements is checked with each code change pushed to the code revision server.
Visual regression tests help you catch visual discrepancies in the development stage, before a bug reaches customers and negatively affects user experience, and before it becomes more costly to fix.
Visual regression testing can also catch bugs that might slip through functional testing. For instance, a functional test will verify the existence of a button and whether it can be clicked. If the button is the same color as the background it’s on, the functional test won’t catch this visual problem, because the element is rendered on the screen and is clickable. On the other hand, a visual regression test will detect that those pixels have changed color in a way that was unintended. This is the power of visual regression testing, compared with other flavors of testing. In the next section, you’ll learn about some useful visual regression testing tools.
Which Visual Regression Testing Will Work Best for Your Team? Depending on what your team wants to achieve, there are multiple visual regression testing tools to choose from. You can find a tool that meets the requirements of your budget, your need for convenience, and your coding expertise. These tools can be divided into two broad categories: open-source tools and SaaS offerings. (These groups are not mutually exclusive, as an open-source tool might have a cloud-based SaaS offering as well.) Let’s start with the open-source options:
Open-Source Visual Regression Testing Tools Having a high number of variables like web browsers, screen sizes, operating systems, and responsive design makes visual regression testing complex. With existing open-source projects, this complexity can be managed to a great degree. Below are three popular open-source browser automation and testing tools that support visual testing in some form:
Selenium Cypress Playwright Selenium Selenium
Selenium is the oldest and one of the most popular browser automation tools. It’s built with Java and has an enviable ecosystem of browsers and frameworks, as well as big sponsors like BrowserStack and Sauce Labs. Selenium’s tagline is “Selenium automates browsers. That’s it! What you do with that power is entirely up to you.” Its code and SDKs for other languages like JavaScript can be found on GitHub .
To drive a browser, you need to use Selenium WebDriver . In the context of visual regression testing with Selenium, if you know Java, you can sew up a solution with Selenium and Ashot . As with any other open-source tool, you’ll need to wire multiple projects together and create a solution that fits your needs.
Cypress Cypress is a JavaScript-based front-end testing tool that claims to be built for the modern web . The useful features of the Cypress framework are automatic waiting, time travel, and screenshots. In the context of visual regression testing, the screenshot feature comes in very handy. A 2022 State of JavaScript survey found that Cypress was used by 42 percent of respondents .
Cypress has a useful ecosystem of plugins . There are multiple community-supported plugins like Cypress Visual Regression and cypress-image-diff that you can use to create your own visual regression testing suite. Compared with Selenium, Cypress and Cypress’s plugins are updated more frequently and have a more vibrant community behind them. If you’re already familiar with JavaScript, Cypress is a great tool for writing and maintaining your visual regression test suite.
Playwright Playwright is a framework for web testing and automation developed by Microsoft for end-to-end testing. It can run your tests on Chromium, Firefox, and WebKit (Safari) using a single API. Unlike with Cypress, you can use JavaScript, TypeScript, Python, .NET, or Java to write tests with Playwright. It also supports testing for the mobile web, which is a bonus.
In terms of visual regression testing, Playwright’s official documentation has a page for visual comparisons . It mentions that Playwright includes the ability to produce and visually compare screenshots. This page has simple code examples for TypeScript and JavaScript, too. Playwright uses the pixelmatch library in the background to compare screenshots. You can pass in options , such as max diff pixels, threshold, and max diff pixel ratio, to fine-tune the visual difference delta for well-adjusted visual regression tests.
Other open-source projects and tools for visual regression testing include Puppeteer and BackstopJS . They have their own place in the visual regression testing realm; however, the aforementioned three options are the most useful ones.
In the next section, we’ll take a look at some SaaS offerings for running visual regression testing.
SaaS Visual Regression Testing Tools You’ll need to write some code when you use open-source tools for visual regression testing. Your team will be responsible for managing configuration options, deciding where to run the tests, and determining how to execute tests. As you can with other software and tools, you can outsource all of these responsibilities and focus on writing and maintaining top-class visual regression tests; this is where the SaaS offerings shine.
And as with any other area of software engineering, testing, and CI/CD, there are many notable players in the visual regression testing game. Below are three that will help you write meaningful visual regression tests for your web applications:
Meticulous Applitools Percy Meticulous Meticulous helps you catch visual regressions without writing or maintaining UI tests. It replays user traffic to automatically catch visual regressions on your web application, with no side effects. It is the easiest way to create end-to-end tests without writing any code — and no code to write means no code to maintain.
Meticulous is funded by Y Combinator , the CTO of GitHub, the CPO of Adobe, and the founder of Vercel, to name a few. It can be set up in a minute by injecting a code snippet. Meticulous can be easily integrated with your CI/CD pipeline with one line of code and two simple clicks. To learn more about Meticulous, you can book a demo or sign up to take it for a spin.
Meticulous is not a layer on top of existing tools like Playwright or Selenium. It’s a different way of thinking about visual testing. It sets you free from writing code while still providing you with the power of visual tests. It eliminates the need to write or maintain end-to-end tests in the first-place, by generating tests from recorded user sessions and automatically maintaining them.
Applitools Applitools is a SaaS tool for visual testing and monitoring. It helps both development and QA teams reduce the amount of code used to write tests, and reduce the amount of time spent running and fixing them. It claims to be an AI-powered visual testing tool that is easy to set up and that integrates with all modern test frameworks. It has clear pricing plans that include Starter, Eyes, and Ultrafast Test Cloud, but the prices are not shown on the Applitools site. The free plan gives you one user and 100 checkpoints per month.
Applitools runs tests in a scalable way with reporting for visual tests written in the tool or framework of your choice. It also lets you monitor the results of the tests in a dashboard. Applitools Eyes gives you a screen with the diff in screenshots, and you can accept or reject changes. It covers a wide range of tools , from Cypress and Playwright on the web to Storybook and Appium for Native mobile apps.
As shown in the tutorial for Playwright with Typescript , Applitools’ SDK provides a layer of functionality that helps run the tests on multiple browsers and at various resolutions. The SDK also sends the results to the Applitools server. Applitools supports a commendable range of frameworks. If you’re looking for a visual and managed layer on top of your existing open-source visual regression testing tools, such as Playwright, Puppeteer, Selenium, and Watir, then Applitools is a good option.
Percy Percy is a visual testing tool that supports cross-browser visual testing. It lists the visual changes as screenshot diffs on a dashboard, where they can be approved or denied. It has been acquired by BrowserStack , which provides access to more than 3,000 browsers and iOS and Android devices for cross-browser testing. Percy has a free plan with 5,000 screenshots per month and unlimited team members included. The paid plans start at $149 per month; the lowest-price paid plan includes 25,000 screenshots and other features .
Percy integrates with many frameworks, such as Ember and Rails. It also has integrations with Storybook for React , Angular, Vue.js, and Ember. Percy supports end-to-end testing frameworks including Cypress, TestCafe, Playwright, Selenium, and Nightwatch. The Percy Snapshot CLI tool is the easiest way to get started with Percy. It can also be used with static sites like those built with Jekyll .
As shown in the official documentation for Playwright , the Percy SDK adds an extra layer to Playwright tests for taking screenshots. If you follow the steps in this tutorial after you run the tests, the screenshots are available on Percy. Then you can go through the review and approval process.
You can run Percy locally , and it integrates with popular source code management tools like GitHub and GitLab. Similar to Applitools, Percy acts as a bridge between visual testing runs and the evaluation and management of results.
What You Need to Know About Visual Regression Testing We hope this summary of what visual regression testing is, and why to use it, is a helpful resource. To recap, we highlighted three popular open-source tools that can do visual regression testing — namely, Selenium, Cypress, and Playwright. We also explored some SaaS offerings that provide an extra layer of test result aggregation and monitoring on top of existing open-source tools.
The different tool in the SaaS bucket is Meticulous, which doesn’t add an extra tier to the existing testing tools but provides a completely different way of doing visual regression testing with configuration files and replaying user traffic.
This means you don’t have to write any test code, so there’s nothing to maintain in the future. You can book a demo today or sign up and try it out on your own.
The choice of the right tool from the open-source or SaaS category will be determined by your team’s preferences. It will also depend on your budget and what your team wants to get out of visual regression testing. If your budget is low, then starting with an open-source tool and integrating it into your current CI/CD pipeline would be an option. If you have some money to spend, then rather than just building a layer on top of existing tools, consider the paradigm shift offered by Meticulous.
Choose a tool or service that fits your needs, fold up your sleeves, and get started with visual regression testing today!
Meticulous Meticulous is a tool for software engineers to catch visual regressions in web applications without writing or maintaining UI tests.
Inject the Meticulous snippet onto production or staging and dev environments. This snippet records user sessions by collecting clickstream and network data. When you post a pull request, Meticulous selects a subset of recorded sessions which are relevant and simulates these against the frontend of your application. Meticulous takes screenshots at key points and detects any visual differences. It posts those diffs in a comment for you to inspect in a few seconds. Meticulous automatically updates the baseline images after you merge your PR. This eliminates the setup and maintenance burden of UI testing.
Meticulous isolates the frontend code by mocking out all network calls, using the previously recorded network responses. This means Meticulous never causes side effects and you don’t need a staging environment.
Learn more here .
