Automated testing plays a number of vital roles in delivering quality software. It’s great for data validation, combinatorial testing, finding regressions and building confidence in each successive build.
How can we get the most out of automated testing? There are three types of automated tests that are commonly used in development: UI, integration, and unit tests. Mike Cohn arranges these groups of tests into a hierarchy known as the “Test Pyramid” which illustrates the ratio of tests needed to efficiently test a system:
Test early and test often.
At WillowTree we implement unit testing extremely early in the development process. These tests are usually written by developers to inspect the app at a low level. Unit tests should make up the bulk of the automated testing for a number of reasons:
- They’re inexpensive to write and maintain.
- They’re an efficient means of providing early feedback so bugs can be found before other changes are made.
- Developing concurrently with unit tests makes developers think about the problem they are solving and any edge cases they might encounter.
- The tests are granular and can help pinpoint exactly which line of a code has a defect.
- Runtime is incredibly fast, because they are natively supported and don’t rely on any UI or external systems such as a database or API.
What happens when unit testing is put off
Teams often skip over unit tests (or delay them) in order to reach development milestones quickly, and that temptation is understandable in an industry that is so deadline-driven, but this path will cost the project much more time later.
Waiting to write unit tests until features are completed can result in everyone having to re-do a lot of work several times over to fix a problem. Unit tests that are written early save time and money by closing this feedback loop before the build gets unstable as developers can write a fix in before any tester gets to see the problem.
De-prioritizing unit tests also upends the test pyramid, which results in what Alister Scott at WatirMelon cleverly refers to as the ice cream cone model, where testing becomes overly dependent on UI tests and exhausts manual testers:
While UI testing ensures that critical functionality is still intact throughout development, there are some drawbacks to testing the system as a whole that put this group at the wide section of the pyramid. For Example:
- Difficult to debug failed tests. If a Log In test fails, this could be caused by numerous places in the code, API is down, or maybe network connection is just slow.
- Flaky tests are known to be an issue in UI tests. Fragile tests are the result of testing lots of things at once and race conditions.
- Long test runs. The run times take much longer than other types of tests because it relies on interacting with rendered UI elements and does not necessarily have hooks into the source code.
- Stay tuned for upcoming blog on dealing with these issues
A process for efficient testing
By utilizing an automation strategy built on the Test Pyramid, teams can maximize their return on investment and testers can focus more on edge cases. This approach requires collaboration across the team and emphasizes the need that tests should be written on the lowest level possible. The order of the Test Pyramid is designed to complement the strengths and weakness of the different types of tests.
Unit tests are the first line of defense for any new regressions in the build. A unit test validates the smallest pieces of an application and runs in complete isolation of the rest of the app and the app’s environment.
Unit tests are great at testing individual components of a system, but unfortunately many bugs arise when all the moving parts come together. This is another reason they function best as a first step, a pre-cursor to integration tests.
Integration tests extend test coverage by exercising the system’s API. Instead of developer provided data, integration tests have data sent to an endpoint and manipulated by the same services that users will be taking advantage of in production.
- Quick way to test system services. These tests can run without a user inter and validate system processes with great speed.
- Tests are reliable. These tests produce little false positive results and if a test fails it provides the testers with good debugging info.
UI tests provide end-to-end coverage and make sure that system is stable and satisfies critical business requirements by interacting with the user interface. UI tests are ideal tests to be written by a QA team because they don’t require any knowledge of the code base. Test runs in the same way that a user would so QA can automate their favorite test cases. Test Engineers love these tests for a couple of reasons:
- Stable builds are guaranteed before it gets into a tester’s hands. Build not only compiles, but you can still log in!
- Compatibility testing just got easier. The same test can be run on different devices to ensure that your app works on any OS/Device combination.
Following the Test Pyramid helps the team plan with automation in mind from the very beginning of development. Unit tests should be not only be a priority because of their inherent speed and reliability, but because they can be written first.
The bottom-up approach provides automated safeguards that can be be implemented earlier in the development process allowing developers to venture with less risk, give QA stable builds, and help Project Managers sleep better.