What’s worse than a software bug? How about a bug that comes back to haunt you in the future after you’ve declared it fixed? That’s often what regressions are. They’re a painful reality that most development teams encounter.
Regressions are a common source of frustration for users, developers, and managers alike. Regular regressions are often symptomatic of flaky systems that are unreliable. They’re also an indicator of a broken development process that’s incapable of verifying the effects of supposed fixes.
Why Regressions Matter
Software regressions are bugs wherein previously functioning components stop working correctly. The system has regressed to a less stable state, so it’s no longer fulfilling behavioral expectations.
Regressions can be introduced to a codebase in several ways:
- Adding a new feature may have unforeseen impacts on existing components, causing them to stop working.
- Fixing one problem which unintentionally creates a new one or the fix itself contains a bug.
- Changing fundamental data structures which causes components to receive data they don’t know how to handle.
- Modifying tightly coupled components may have side effects in completely separate subsystems.
Regressions are a particularly thorny class of bugs. Because they only impact existing features, they can slip unnoticed past developers and QA teams. Users might be the first to spot the problem, putting you at an immediate disadvantage before you’ve started an investigation.
Having users find your bugs is a surefire way to damage their confidence in your product. Software that unexpectedly breaks creates a highly negative user experience that makes it difficult to establish trust. It’s particularly impactful when the regression concerns a previous bug; for example, if you’ve told a customer that a problem’s been fixed, they won’t be impressed when it later reappears!
Consequently, if the function expects passed-in values to have certain types, changes to code inside functions can have knock-on impacts across the application. Failure to supply that type each time the function’s called can lead to crashes that don’t show up until runtime.
Avoiding Future Regressions
Protecting your codebase with a comprehensive test suite is the only reliable way to defend against regressions. Automated tests give you confidence your software is performing consistently, and running your tests as part of your deployment pipeline ensures changes can’t merge into production if they’d break existing code.
Most regressions occur in larger codebases with low test coverage scores. If you have poor coverage, it’s difficult to assess the impact of bug fixes and new features. Even if you don’t have many tests yet, it’s important to add one each time you fix an issue. This will help prevent the problem from recurring in the future.
However, even automated testing isn’t infallible; some regressions can still occur, regardless of the size of your test suite. The quality of your tests matters as much as their quantity. It’s important to write tests that probe every possible behavior of your application, including any rare edge cases.
End-to-end (E2E) tests target the entirety of the system from beginning to end. These tests target the project’s code, it's integrations with external services, and the user experience that’s created.
E2E tests validate that complete user flows are achievable inside your application. Here are two examples you could set up:
- Validating a user can view the sign-up form, enter their credentials, and create a new account that’s saved in your database.
- Checking a user can press Checkout, receive a charge to their payment card, and be sent an order confirmation email.
E2E tests are beneficial because they accurately represent the behavior of the entire system. They’re effective at surfacing issues that don’t necessarily show up when code is executed in isolation.
However, E2E tests are often time-consuming to write, run, and maintain. Setting up an E2E environment can be complicated, and test suites often take a long time to run. Sometimes E2E tests might only be used immediately before deployment. This improves iteration speed while developing a change but allows issues to go unnoticed until the last minute.
Unit tests are the easiest, fastest, and most convenient form of automated testing. They target individual code components on a function-by-function basis.
A unit test should cover every possible pathway through its subject piece of code. It will look at the section in isolation without any awareness of your system’s other components or its outer environment. Unit tests work best when your functions are pure, meaning they always produce the same output for a particular set of parameters—but you can mock external dependencies and global states, too.
Integration tests sit between unit tests and E2E tests. They assess whether units are calling each other correctly to create higher-level functionality.
You normally create integration tests within the same framework you’re using for unit tests. The difference lies in what’s being tested and the extent to which the outer environment is mocked.
Integration tests consider the communication paths between individual components in your system. Whereas unit tests routinely mock external dependencies, integration tests should use real ones that match the requirements of production. This lets you uncover issues involving cross-component data flows, config stores, caches, and third-party APIs.
Integration testing gives you confidence that your code units work together properly. They can be cumbersome to maintain, though; you need to remember to update all affected integration tests whenever you change one of your components.
Replay Tests with Meticulous
Replay tests focus on detecting regressions in your application’s frontend. These tests have traditionally been tricky to orchestrate, so they may have been handled as part of your E2E testing. As mentioned previously, E2E testing often comes late in the development lifecycle, making it less likely that regressions will be detected with time to resolve them.
The Meticulous platform performs frontend testing without making you write code or configure complex environments. Replay tests work by comparing a pre-recorded workflow against the current version provided by your code. Meticulous then diffs the two sessions to identify regressions, including visual discrepancies.
You record tests by using the Meticulous CLI to open a new browser window. Interact with the web page as usual, and Meticulous will capture your activity and save it to a new session in your account. You can replay the session against your local development environment after you’ve made changes to your code. Meticulous will spot any discrepancies and alert you of its findings.
Replay testing is a great way to reliably uncover visual issues and subtle state changes that would otherwise go unnoticed. Recording workflows in a real in-app session is quicker and easier than manually writing code to automate the browser.
One drawback of this approach is that replays don’t automatically target your real backend. Meticulous mocks the network calls made by your sessions. This is often beneficial; for example, replays won’t have any side effects, but it prevents you from performing E2E-like tests where backend interactions need to be included, too. However, you can accommodate these workflows by manually configuring Meticulous to make real API requests against your staging environment.
Regressions are a type of software bug where existing functionality ends up broken as a result of a new change. While they can affect any project, they’re most common in codebases that have poor test coverage. This allows changes in one part of the system to have negative impacts on other areas without the developer being informed.
Addressing regressions should be your top priority when they occur. Broken features that used to work erode user confidence and create a lasting impression of a poor-quality product. Each time you resolve a regression, add a test to make sure the code doesn’t silently break again in the future. On a long-term basis, writing a mix of E2E, unit, and integration tests is the best way to ensure stability in your system.
Meticulous is a tool for software engineers to easily create end-to-end tests. Use our open source CLI to open an instrumented browser which records your actions and translates them into a test. Meticulous makes it easy to integrate these tests into your CI.
Meticulous has an option to automatically mock out all network calls when simulating a recorded sequence of actions. If you use this option, you do not need a backend environment to use Meticulous and Meticulous tests never cause side effects (like affecting analytics) or hit your backend.
Authored by James Walker