Visual regressions and how to prevent them

updated on 07 March 2022

From widgets to complete applications, the front-end has become a more complex and essential piece in most products’ success, particularly those which include UI and UX in their value proposition.

For this reason, protecting the health of the user experience by ensuring that no regression has been introduced is critical.

While end-to-end tests (E2E) are a complete - but costly - way to assess an application’s health, visual regression testing is a complementary and faster strategy for maintaining product quality. This article will guide you through the many visual regression testing solutions available, from technical libraries to SaaS products.

What is visual regression testing?

Visual regression tests ensure that an application displays the proper UI in a given user scenario. For example, visit the login page and ensure that all expected visual elements are present (input, buttons, etc.). These visual checks can be performed at two levels:

  • At the DOM level: by performing snapshots of the output DOM
  • At the screen level: by performing screenshots to verify the actual pixel rendered on the screen

An updated version of the well-known pyramid of tests would now be the following:

image2-4xmjx

Unlike E2E tests, visual regression tests do not necessarily require interacting with the page, making them faster to run and easier to maintain.

How to use visual regression tests?

Both types of visual regression testing (snapshot and screen) should be performed against the critical path of the application. An application’s critical path is the set of pages and features that define your product experience's core, for example, a login page, a register page, or a cart checkout page.

Snapshot tests

Snapshot tests, mainly performed with Jest, are commonly considered part of integration tests. They consist of rendering a component in a given scenario, and saving a snapshot of the resulting DOM. When the test gets updated, the new DOM result gets compared with the existing one.

If any change is found, an error is raised to notify of the regression.

image1-pctdv

Because snapshots tests rely on DOM comparison, they should test the structure of a given page or set of components.

For example, snapshots ensure that:

  • A set of actions is available on a page (ex: filtering, searching)
  • The proper information text is displayed
  • The login page does not submit the form if the inputs are empty

In short, snapshots tests are perfect in preventing any visual regression in the structure of your application (actions, text, etc).

While snapshots tests are easier to set up and maintain, and fast to run (< 2s), they cannot be used to assess pure visual aspects such as CSS overflow issues or colors of UI elements. You can read more about how to implement snapshot tests here.

Screen tests

Screen tests give us a finer visual regression detecting by comparing screenshots:

The image correctly loading on the left, the image failing to load on the right, and the diff in the middle.
The image correctly loading on the left, the image failing to load on the right, and the diff in the middle.

With a bit more setup work, Screen tests prevent any kind of visual regressions, from structure (action buttons, text) to styling (overflow, colors, padding).

Testing advanced interactions or image loading scenarios is possible, thanks to tools like Cypress, which rely on a virtual test browser. However, running a virtual test browser comes with its drawbacks: a more complex CI setup and slower tests (>1min).

For this reason, screen tests should be used to assess the final rendering of a given page part of the critical path or UI-Kit components. For example, ensuring that the onboarding pages are rendered correctly or that all components of a UI-Kit are correctly rendered. You can read more about how to set up screen tests here.

SaaS solutions

The previously mentioned technical solutions fit well for small teams where only technical users need to access the results of the visual tests.

SaaS solutions exist for bigger companies or ones where product/designers are in the visual testing process, such as MeticulousAppliTools and Percy.

Covering the features offered by the major visual regression SaaS solutions would require an article on its own. Still, most SaaS solutions offer the following features that ease the setup of advanced visual regressions tests and enhance the collaboration on failing tests:

  • Page and component testing
  • Dedicated cloud worker for faster execution
  • A platform for reviewing regressions (for technical and non-technical people)
  • Advanced Workflow management
  • AI features to avoid false-positive results
  • Workflow integration: Github, Jenkins, Slack, and more

Conclusion

Front-end is now a critical part of any web-product-based business.

While maintaining the quality of their products by implementing many E2E tests, we see that visual regression testing is a faster solution.

Snapshots tests as integration tests are helpful to ensure that the visual structure of a page is maintained over changes. Screen tests are slower to run and should avoid any visual regression in essential components of an application, for example UI-Kits.

Finally, depending on the product complexity and the size of your organization, choosing SaaS solutions to enhance collaboration and set up more advanced tests might be the way to go.

Meticulous

Meticulous is a tool to catch JavaScript regressions in web applications, with zero maintenance burden. Sort of like a self-writing selenium test.

Insert a single line of JavaScript onto your local dev, QA and staging sites. This snippet records user sessions. Meticulous then tests your new code by replaying those user sessions to automatically find frontend regressions (JS exceptions, visual diffs) for you before they hit production. It does this without ever hitting your backend or causing any side effects. You can also view replays on new code to manually ascertain whether flows have broken. 

Find out more and watch a 60-second demo here.

Authored by Charly Poly

Read more