Visual regression testing ensures that your application’s user interface is not adversely affected when you release code changes. In this guide, we’ll discuss six of the many open-source and software as a service (SaaS) visual regression testing tools your team can start using today. Let’s dive in!
What Is Visual Regression Testing? In brief, visual regression testing verifies that a webpage’s elements are rendered properly from a visual point of view. They’re automated tests, so they can be run as part of a CI/CD pipeline; with a mature CI/CD pipeline, the correctness of the visual elements is checked with each code change pushed to the code revision server.
Visual regression tests help you catch visual discrepancies in the development stage, before a bug reaches customers and negatively affects user experience, and before it becomes more costly to fix.
Visual regression testing can also catch bugs that might slip through functional testing. For instance, a functional test will verify the existence of a button and whether it can be clicked. If the button is the same color as the background it’s on, the functional test won’t catch this visual problem, because the element is rendered on the screen and is clickable. On the other hand, a visual regression test will detect that those pixels have changed color in a way that was unintended. This is the power of visual regression testing, compared with other flavors of testing. In the next section, you’ll learn about some useful visual regression testing tools.
Which Visual Regression Testing Will Work Best for Your Team? Depending on what your team wants to achieve, there are multiple visual regression testing tools to choose from. You can find a tool that meets the requirements of your budget, your need for convenience, and your coding expertise. These tools can be divided into two broad categories: open-source tools and SaaS offerings. (These groups are not mutually exclusive, as an open-source tool might have a cloud-based SaaS offering as well.) Let’s start with the open-source options:
SaaS Visual Regression Testing Tools You’ll need to write some code when you use open-source tools for visual regression testing. Your team will be responsible for managing configuration options, deciding where to run the tests, and determining how to execute tests. As you can with other software and tools, you can outsource all of these responsibilities and focus on writing and maintaining top-class visual regression tests; this is where the SaaS offerings shine.
And as with any other area of software engineering, testing, and CI/CD, there are many notable players in the visual regression testing game. Below are three that will help you write meaningful visual regression tests for your web applications:
Meticulous Applitools Percy Meticulous Meticulous helps you catch visual regressions without writing or maintaining UI tests. It replays user traffic to automatically catch visual regressions on your web application, with no side effects. It is the easiest way to create end-to-end tests without writing any code — and no code to write means no code to maintain.
Meticulous is funded by Y Combinator , the CTO of GitHub, the CPO of Adobe, and the founder of Vercel, to name a few. It can be set up in a minute by injecting a code snippet. Meticulous can be easily integrated with your CI/CD pipeline with one line of code and two simple clicks. To learn more about Meticulous, you can book a demo or sign up to take it for a spin.
Meticulous is not a layer on top of existing tools like Playwright or Selenium. It’s a different way of thinking about visual testing. It sets you free from writing code while still providing you with the power of visual tests. It eliminates the need to write or maintain end-to-end tests in the first-place, by generating tests from recorded user sessions and automatically maintaining them.
Percy Percy is a visual testing tool that supports cross-browser visual testing. It lists the visual changes as screenshot diffs on a dashboard, where they can be approved or denied. It has been acquired by BrowserStack , which provides access to more than 3,000 browsers and iOS and Android devices for cross-browser testing. Percy has a free plan with 5,000 screenshots per month and unlimited team members included. The paid plans start at $149 per month; the lowest-price paid plan includes 25,000 screenshots and other features .
Percy integrates with many frameworks, such as Ember and Rails. It also has integrations with Storybook for React , Angular, Vue.js, and Ember. Percy supports end-to-end testing frameworks including Cypress, TestCafe, Playwright, Selenium, and Nightwatch. The Percy Snapshot CLI tool is the easiest way to get started with Percy. It can also be used with static sites like those built with Jekyll .
As shown in the official documentation for Playwright , the Percy SDK adds an extra layer to Playwright tests for taking screenshots. If you follow the steps in this tutorial after you run the tests, the screenshots are available on Percy. Then you can go through the review and approval process.
You can run Percy locally , and it integrates with popular source code management tools like GitHub and GitLab. Similar to Applitools, Percy acts as a bridge between visual testing runs and the evaluation and management of results.
What You Need to Know About Visual Regression Testing We hope this summary of what visual regression testing is, and why to use it, is a helpful resource. To recap, we highlighted three popular open-source tools that can do visual regression testing — namely, Selenium, Cypress, and Playwright. We also explored some SaaS offerings that provide an extra layer of test result aggregation and monitoring on top of existing open-source tools.
The different tool in the SaaS bucket is Meticulous, which doesn’t add an extra tier to the existing testing tools but provides a completely different way of doing visual regression testing with configuration files and replaying user traffic.
This means you don’t have to write any test code, so there’s nothing to maintain in the future. You can book a demo today or sign up and try it out on your own.
The choice of the right tool from the open-source or SaaS category will be determined by your team’s preferences. It will also depend on your budget and what your team wants to get out of visual regression testing. If your budget is low, then starting with an open-source tool and integrating it into your current CI/CD pipeline would be an option. If you have some money to spend, then rather than just building a layer on top of existing tools, consider the paradigm shift offered by Meticulous.
Choose a tool or service that fits your needs, fold up your sleeves, and get started with visual regression testing today!
Meticulous Meticulous creates and maintains an exhaustive suite of e2e ui tests with zero developer effort.
This quote from the CTO of Traba sums the product up best: "Meticulous has fundamentally changed the way we approach frontend testing in our web applications, fully eliminating the need to write any frontend tests. The software gives us confidence that every change will be completely regression tested, allowing us to ship more quickly with significantly fewer bugs in our code. The platform is easy to use and reduces the barrier to entry for backend-focused devs to contribute to our frontend codebase."
This post from our CTO (formerly lead of Palantir's main engineering group) sets out the context of why exhaustive testing can double engineering velocity. Learn more about the product here.
Meticulous Meticulous creates and maintains an exhaustive suite of e2e ui tests with zero developer effort.
This quote from the CTO of Traba sums the product up best: "Meticulous has fundamentally changed the way we approach frontend testing in our web applications, fully eliminating the need to write any frontend tests. The software gives us confidence that every change will be completely regression tested, allowing us to ship more quickly with significantly fewer bugs in our code. The platform is easy to use and reduces the barrier to entry for backend-focused devs to contribute to our frontend codebase."
This post from our CTO (formerly lead of Palantir's main engineering group) sets out the context of why exhaustive testing can double engineering velocity. Learn more about the product here.
