At the start of every new project at WillowTree, the product team has a vision for an end-to-end automated test suite to complement traditional manual testing. There are many reasons that make building an automated test suite a clear priority; a reliable suite will find breaking changes as soon as they occur, enable test engineers to dedicate more time to exploratory testing, and allow teams to deploy code to production more quickly and confidently.
Building a high-value automated test suite is as much about effective collaboration as it is about the code being written, as is true for all other practices that go into creating excellent products. The strategies identified below build on that premise, encouraging people to collaborate more effectively in order to extract the most value out of a product’s test automation suite.
Identify desired outcomes with the team
A core driver of our product development process is ensuring all team members have a clear understanding of desired outcomes for the product. The same should be true for the corresponding test suite.
Each product is unique, requiring a tailored testing strategy to ensure a quality experience for the end user as well as the product team responsible for building it. To identify what that strategy looks like, a test engineer can start by defining the desired outcomes for an automated test suite. Some possible outcomes include:
- The team can run a full regression on a release candidate in less than a day
- The team can run a regression of all critical functionality on a hot fix build in less than half a day
- The automated test suite can be run at any time on any build
- Software engineers can run the test suite before submitting code for code review and can fix any issues found before being reviewed by test engineers
- The automated test suite is readable and well-documented so that it can be also be maintained by any current or future team member
The product team should determine which of these outcomes will provide the most value and tailor the suite accordingly. Typically, meeting a combination of the desired outcomes above will require a compromise between flexibility and efficiency. It’s important that all team members— - not only test engineers— - understand these desired outcomes in order to extract the most value out of the suite as the project progresses.
Prioritize native solutions to encourage team collaboration
We believe an automated test suite should be fully integrated with the product itself, as core to the app as any feature. To support this goal, teams should prioritize choosing native solutions to build test automation suites.
We staff our teams with experts in the technology being used to build the product. For example, when building an iOS app, the software engineers on the team are deeply experienced in Swift, xCode, and all things Apple. In this case, the test suite could be built using XCTest. XCTest is a Swift framework built and maintained by teams at Apple. By building the test suite using a shared tool, the test engineers can leverage the expertise of the software engineers to resolve issues and tailor tests to the specific needs of an iOS app.
There’s another advantage to this approach; it breaks down the barrier between ‘test’ and ‘software’ engineering. Although the expertise and day-to-day tasks of the two disciplines are different, their goal is the same: to build an intuitive, high-quality user experience that meets business goals. Software engineers should feel comfortable running tests and interpreting results to ensure their code is doing what it should. Test engineers should feel just as comfortable navigating the app code to ensure their tests appropriately cover high-risk areas. Ultimately, this shared responsibility boosts the overall value of test automation, allowing all team members to take advantage of it.
Build test automation into the team development process
Projects with manual testing alone tend to follow a common development workflow. The software engineer creates a branch, commits code to the branch, passes a code review, and then directs the test engineer to a build where the code changes can be tested. A major consideration in building an impactful automated test suite is how to best incorporate it into this standard development process. Ideally, automated tests should be written in tandem with new feature work—in the same branch and at the same time—so that the feature is covered by tests from the very beginning. This requires thoughtful coordination between the software engineer writing the app code and the test engineer writing the test code. This also means the app code and test code is reviewed holistically, by members of both disciplines.
At first this type of close coordination between software and test engineers can be difficult; however, over time each team finds what works best for them, and the collaboration becomes a natural part of the development workflow. People on teams that embrace this cross-disciplinary collaboration by adapting it to their specific needs - tend to communicate more effectively and finish work more efficiently than those who do not.
Run tests frequently and reliably
In order to be valuable, automated tests need to be run as frequently and reliably as possible. Running tests should become as integral a part of the process as writing them. The suite should run against every feature branch so that failures can be addressed before merging to the “clean” branch (i.e. master or develop for most teams). At a minimum, the full test suite should run every night.
All tests, manual and automated alike, are only valuable when people are confident that the tests are providing accurate results. The test engineering group must constantly be on guard against “flaky” tests, ones that do not provide consistent results. One flaky test can cast into doubt the reliability of the full suite, undermining the value of every test in it. If it can’t be “de-flaked,” it’s better to do away with an automated test altogether rather than allow it to exist as a known flaky test.
Communication continues to be vital here. All team members must be aware of the latest test run results on the clean branch so that they can feel confident that any failures on their feature work are related to their code changes. One of our teams recently achieved this by reporting the latest results in the channel topic of their team Slack channel. This can be done in any number of ways, as long as everyone involved has access to the information they need.
Acknowledge limitations as a team
Test automation provides the most value when it’s thoughtfully implemented alongside conventional manual testing. There are some aspects of a product that can’t be automated and some aspects that shouldn’t be automated. An example of what shouldn’t be verified by an automated test could be a screen animation; this is typically a low-risk feature and can be better evaluated by a human eye. Making these decisions will depend on the project and platform and it’s critical that the team discuss these potential limitations to agree on what they entail.
The strategies identified above rely on people working together effectively toward a shared goal: to build an excellent product. Establishing and maintaining a team process to support test automation in these ways takes time and dedication; it’s well-worth the effort. This approach can provide incredible value to the development team by fostering an environment that creates and cares about collaboration, shifting defect discovery closer to the point of creation, and providing accessible tools to increase the team’s awareness of risk and quality.