Visual Regressions and How to Prevent Them

updated on 30 November 2022

From widgets to complete applications, the front-end has become a more complex and essential piece in most products’ success, particularly those which include UI and UX in their value proposition.

For this reason, protecting the health of the user experience by ensuring that no regression has been introduced is critical.

While end-to-end tests (E2E) are a complete - but costly - way to assess an application’s health, visual regression testing is a complementary and faster strategy for maintaining product quality. This article will guide you through the many visual regression testing solutions available, from technical libraries to SaaS products.

What is Visual Regression Testing

Visual regression tests ensure that an application displays the proper UI in a given user scenario. For example, visit the login page and ensure that all expected visual elements are present (input, buttons, etc.). These visual checks can be performed at two levels:

  • At the DOM level: by performing snapshots of the output DOM
  • At the screen level: by performing screenshots to verify the actual pixel rendered on the screen

An updated version of the well-known pyramid of tests would now be the following:

Pyramid of tests-0ra5h

Unlike E2E tests, visual regression tests do not necessarily require interacting with the page, making them faster to run and easier to maintain.

How to Use Visual Regression Tests

Both types of visual regression testing (snapshot and screen) should be performed against the critical path of the application. An application’s critical path is the set of pages and features that define your product experience's core, for example, a login page, a register page, or a cart checkout page.

Snapshot Tests

Snapshot tests, mainly performed with Jest, are commonly considered part of integration tests. They consist of rendering a component in a given scenario, and saving a snapshot of the resulting DOM. When the test gets updated, the new DOM result gets compared with the existing one.

If any change is found, an error is raised to notify of the regression.

Setting up snapshot tests-tyfhl

Because snapshots tests rely on DOM comparison, they should test the structure of a given page or set of components.

For example, snapshots ensure that:

  • A set of actions is available on a page (ex: filtering, searching)
  • The proper information text is displayed
  • The login page does not submit the form if the inputs are empty

In short, snapshots tests are perfect in preventing any visual regression in the structure of your application (actions, text, etc).

While snapshots tests are easier to set up and maintain, and fast to run (< 2s), they cannot be used to assess pure visual aspects such as CSS overflow issues or colors of UI elements. You can read more about how to implement snapshot tests here.

Screen Tests

Screen tests give us a finer visual regression detecting by comparing screenshots:

The image correctly loading on the left, the image failing to load on the right, and the diff in the middle.
The image correctly loading on the left, the image failing to load on the right, and the diff in the middle.

With a bit more setup work, Screen tests prevent any kind of visual regressions, from structure (action buttons, text) to styling (overflow, colors, padding).

Testing advanced interactions or image loading scenarios is possible, thanks to tools like Cypress, which rely on a virtual test browser. However, running a virtual test browser comes with its drawbacks: a more complex CI setup and slower tests (>1min).

For this reason, screen tests should be used to assess the final rendering of a given page part of the critical path or UI-Kit components. For example, ensuring that the onboarding pages are rendered correctly or that all components of a UI-Kit are correctly rendered. You can read more about how to set up screen tests here.

SaaS Solutions

The previously mentioned technical solutions fit well for small teams where only technical users need to access the results of the visual tests.

SaaS solutions exist for bigger companies or ones where product/designers are in the visual testing process, such as MeticulousAppliTools and Percy.

Covering the features offered by the major visual regression SaaS solutions would require an article on its own. Still, most SaaS solutions offer the following features that ease the setup of advanced visual regressions tests and enhance the collaboration on failing tests:

  • Page and component testing
  • Dedicated cloud worker for faster execution
  • A platform for reviewing regressions (for technical and non-technical people)
  • Advanced Workflow management
  • AI features to avoid false-positive results
  • Workflow integration: Github, Jenkins, Slack, and more

Conclusion

Front-end is now a critical part of any web-product-based business.

While maintaining the quality of their products by implementing many E2E tests, we see that visual regression testing is a faster solution.

Snapshots tests as integration tests are helpful to ensure that the visual structure of a page is maintained over changes. Screen tests are slower to run and should avoid any visual regression in essential components of an application, for example UI-Kits.

Finally, depending on the product complexity and the size of your organization, choosing SaaS solutions to enhance collaboration and set up more advanced tests might be the way to go.

Meticulous

Meticulous is a tool for software engineers to catch visual regressions in web applications without writing or maintaining UI tests.

Inject the Meticulous snippet onto production or staging and dev environments. This snippet records user sessions by collecting clickstream and network data. When you post a pull request, Meticulous selects a subset of recorded sessions which are relevant and simulates these against the frontend of your application. Meticulous takes screenshots at key points and detects any visual differences. It posts those diffs in a comment for you to inspect in a few seconds. Meticulous automatically updates the baseline images after you merge your PR. This eliminates the setup and maintenance burden of UI testing.

Meticulous isolates the frontend code by mocking out all network calls, using the previously recorded network responses. This means Meticulous never causes side effects and you don’t need a staging environment.

Learn more here.

Authored by Charly Poly

Read more