Have you ever faced a situation where a feature was working but after a seemingly-unrelated change it now looks broken? Visual regression testing helps you verify the visual aspects of your application’s user interface look the same across code changes, preventing this issue.
In this post, you will learn about visual testing, its benefits, and an example showing how it works. You will also find out about the tradeoffs between manual and automated visual regression testing. Let's get started!
What is Visual Regression Testing A regression is a return to a former or less developed state. In software testing, regression testing refers to ensuring new changes applied to the software system do not unintentionally break something that was working earlier. These new changes could be a new feature, a bug fix, any code change, or even an update to a third-party library. You can read more about catching JavaScript regressions and preventing them.
Visual regression guarantees that a change does not adversely impact the visual appearance of the user interface of an application. For instance, a software engineer changes the text box in the search bar. This may work fine for the search functionality, but suppose the text box style was also applied to the username in the login interface, and this broke the login functionality. This unintentional event of breaking the login functionality that was previously working can be avoided with proper visual regression testing.
Visual regression tests check what the end user will see by comparing screenshots taken before and after code changes. This testing can be baked into the software’s CI/CD pipeline . In the next section, you will learn about the benefits of visual regression testing.
Visual Regression Testing Benefits The cost of any software bug increases multifold when it is found later in the software engineering process. Similar to any other form of testing, the main benefit of visual regression testing is to help you catch any visual bug much earlier in the development phase rather than a later phase. A bug found by a software engineer while developing the feature will cost the company exponentially less compared to the same bug found by the customer after releasing the software. This is an area where feature flags and gradual release can help, but this is beyond the scope of this post.
Visual regression testing can even find bugs that easily slip through functional testing. Functional testing can check those elements on a webpage render fine and can even be clicked on. But functional tests won’t check that the text on a button is partially overlapped by the input box. Such hard-to-test scenarios can have big impacts, like breaking a sign-up flow. As visual regression tests compare before and after screenshots even minor CSS changes can be verified to work. Next, you will see an example where good visual regression testing could have made the customer experience a lot better.
Example of What Visual Regression Testing can Uncover Visual regression testing becomes indispensable for customer-facing applications. These applications define your company’s brand image so even small issues can affect your brand image adversely. Below you can see an example of the homepage of American Airlines :
The page is relatively responsive up to the dimensions of a tablet. If you decrease the width of the desktop website’s homepage even more you can see the cracks show up. They load a different website for mobile web, so it is not a stark problem. If functional tests were run on the above page they will pass because all the text and elements are present, loaded successfully, and visible on the page but not visible for a smaller resolution. If there was visual regression testing for the above-shown smaller mobile-like dimension of the webpage, American Airlines could have easily found issues with the navigation bar. For example, adding the fourth link on the navigation bar called “Flight status” visually breaks the first link with the text “Find Flights”. Similarly, another benefit they would have got is adjusting the width of the “From and To” input boxes will make the text for Adults (12+) and Children(2-11) clearly visible, and so on.
Visual regression testing can verify how the elements on the page are rendered visually. These tests don’t need to rely on selecting each element with unique selectors, making them much faster to write and maintain. In addition to simplifying the process, they also make functional testing practices much more effective and on par with what the end user sees. In the following part, you will learn about how visual regression tests work.
How Visual Regression Testing Works One form of visual regression testing, screenshot testing, works by taking screenshots of the UI before and after a code change is applied. Then the two screenshots are compared and any difference between them is highlighted. In this automated comparison, the difference threshold is configurable. A tiny difference can be allowed but any major unexpected visual difference should be flagged as a test failure. This is where the power of visual comparison comes to play. There is no need to find the right selector and write lots of code around it. Simply compare before and after screenshots for a given browser and screen dimension, then make sure the visibly highlighted differences are intended and caused by the change to be deployed.
Below is a quick example of comparing two screenshots before and after some intended CSS changes for the Name and nationality guessing app used for the Jest spyOn example:
The flags in this app have been deliberately made smaller with the CSS change. This is an intentional change and not a regression. If the flag sizes had decreased without intention they would be caught easily with visual regression testing. Similar to other forms of tests, visual regression testing can be done manually or in an automated way, let's discuss that next.
Manual Visual Regression Testing Visual regression testing can be done manually, you can use tools to carry out the testing process. Software engineers and designers do it all the time. They try to find any visual defects when changing visual elements, especially when changing CSS code. They go to the browser after a code change and hit the refresh button or keyboard shortcut. Then verify that the code change is showing the expected output.
Unfortunately, this process is slow, clumsy, painful, and not scalable. In addition, it is prone to human errors. Doing it in the early stages of an application can make sense. When the application is big, it is not humanly possible to check 100s of screens when 2 lines of code have changed. This is where regression can come up as the designer or software engineer will be focused on the page in their browser, and they might miss the other page(s) or component(s) using the same styling.
Another way to do visual regression testing manually is with the use of image diff tools. You should take a screenshot of a page before and after a code change. Then those two screenshots can be passed into an image diff tool to see if there is any change detected. This is part manual and part automated. Still, it is not a scalable and automated way of visual regression testing. That surfaces the important need to do automated visual regression testing which is discussed next.
Automated Visual Regression Testing The process of visual regression testing can be automated with proper and purpose-built tools. It takes a few steps to get started with these. Like functional testing, it can be done for happy paths and main screens initially. As per the need, more pages can be covered by these tests.
The visual regression testing process should be a step of the CI/CD pipeline too.
A follow-up post will discuss the various tools available today. Some of these tools work by calculating a configurable pixel difference. However, all of these tools require you to write a test for each scenario that you want to test. The number of scenarios explodes as you may want to test many different flows and configuration combinations of your web app.
These tests are highly effective for testing core flows, like a sign-up or checkout flow, but quickly become infeasible when trying to cover a large surface area. It’s simply not feasible to maintain hundreds of end-to-end UI tests to cover hundreds of screens across a codebase, and this problem will only grow over time as the surface area of the codebase expands.
Meticulous solves this problem by automatically generating UI tests for you, but without a maintenance burden. It does this by recording user sessions (on development or production) and simulating these sessions on new versions of code in order to enumerate various scenarios within your web application. You can learn more and try out Meticulous here .
Conclusion In this post, you learned about visual regression testing and the benefits of doing it with a real-life example. You were briefed about how visual testing works, which is by taking screenshots before and after a code change and comparing them to find any differences. After that, you learned about doing visual regression testing manually and in an automated way.
Surely automating tests has the benefits of scalability and running the tests over and over again swiftly. Hopefully, you will exploit visual regression testing to catch any visual bugs and add more value to the software you deliver.
Meticulous Meticulous is a tool for software engineers to catch visual regressions in web applications without writing or maintaining UI tests .
Inject the Meticulous snippet onto production or staging and dev environments. This snippet records user sessions by collecting clickstream and network data. When you post a pull request, Meticulous selects a subset of recorded sessions which are relevant and simulates these against the frontend of your application. Meticulous takes screenshots at key points and detects any visual differences. It posts those diffs in a comment for you to inspect in a few seconds. Meticulous automatically updates the baseline images after you merge your PR. This eliminates the setup and maintenance burden of UI testing.
Meticulous isolates the frontend code by mocking out all network calls, using the previously recorded network responses. This means Meticulous never causes side effects and you don’t need a staging environment.
Learn more here .
