QA and TestingStrategy and Innovation

The Most Important Testing “Tool”

In my past experiences of interviewing and looking for new testing positions, I’ve noted a theme that centers around this question: “What’s the best tool to know?”

Earlier in my career, I tended to focus on—and stress about—this. Each company wanted something different. If you happened to get stuck at a company that wasn’t using whatever tool was ‘hot’ in the market, it was easy to feel you were out of luck and in danger of being left behind. Sure, you could spend a significant amount of personal time learning the tool, but if you didn’t have ‘on-the-job experience’, it was often discounted by potential employers.

Over time, I started to see another pattern. If the company took a chance on me, I was able to make effective contributions to the team even if I didn’t know the exact tool they wanted. Sometimes these contributions came by simply learning the tool in question; other times it may have been by leading the team toward another tool better suited for the task.

Now that I work at WillowTree where we are collaborating with numerous clients, supporting a variety of platforms, and using a dizzying array of toolsets, processes, and team configurations, it’s finally crystal clear.

It’s not about the tools. It never was.

The core skill of testing is adaptability. Tools, technical skills, and domain knowledge certainly help, but they also change with time. The best test engineers bring creativity, innate curiosity, nuanced communication, effective problem solving skills, and a bit of determination with them wherever they go. While these are not skills that can be easily taught, they can be improved with practice. These skills will carry through any domain and any tool set for an entire career.

So why do we focus so much on the tools and domain knowledge? Tool sets and domain knowledge are easier to measure. We are able to generate reports and quiz people about both of these topics. You can find ways to learn about someone’s adaptability, but it takes more effort. I would argue that, for the sake of your product’s quality, it’s worth it to put in that effort.

If you have someone who knows both the relevant tools and domain but does not inherently embody adaptability, they will not be able to effectively ensure that their teams will ship quality products. To help demonstrate, let’s talk through an example scenario. For feature X, here are some examples of how the two different styles might play out:

The Tool-Based Tester

  • Once requirements are made available, ensures that each requirement is thoroughly represented in the tool of note (this could be automation, a test case management tool, etc.).

  • Ensures every test is run for each build/release (depending on use of CI).

  • Logs and retests bug tickets coming out of each test run for accuracy.

  • Generates reports or status, and documents the results of tests.

Note that this list reads like most testing job descriptions and resumes.

The Adaptive Tester

  • Obtains copies of requirements before coding begins or finds a way to get involved in meetings to ensure understanding from the product owner perspective of the intent of the feature.
  • Raises questions about scope, error handling, quality expectations, and risk tolerance before code is written to help ensure the requirements are fully addressing the feature intent.
  • Collaborates with developers writing the software, sometimes pairing with them during feature development to discuss unit test coverage and brainstorm test scenarios.
  • Writes tests/test cases (automated, manual, or both) complementing the unit tests. For high risk areas, may add an additional/duplicate layer on top of unit tests.
  • Performs exploratory testing, seeing what the feature will allow the Tester to do regardless of the intent of the feature.
  • Coordinates with developers and other teammates when unexpected results occur. Is the error in understanding/communication, code, and/or design? Documents the findings and retests resulting bug tickets.
  • Raises concerns and suggestions around potential process improvements to enhance communication, reduce turnaround time, and identify impactful assumptions more quickly. Highlights the ways in which poor existing processes have been leading to poor quality if necessary to support the need for changes.
  • After testing is complete, provides a risk assessment to interested parties, noting the strong and weak areas of the product in its current state.

Notice that only a small subset of the work done by the Adaptive Tester is in using toolsets. The adaptive tester recognizes that quality begins the moment someone starts documenting ideas because a large proportion of quality issues originate before any code is ever written. For example, the team could have assumptions that everyone is intending the same thing or that the scope of requirements is completely covered. After development is started, a significant portion of issues occur as a result of misinterpretation of requirements (and not just by Developers). Bugs and software errors get increasingly more expensive the later in the process they are found.

Collaborating with developers is a great way to catch additional assumptions. It’s also important for testers to become familiar with the unit tests (and suggest additional ones) to ensure that unit testing and subsequent automated and manual test cases offer appropriate coverage without duplication. While it could be argued that this takes more of the developer’s time, a substantial portion of the time spent would be spent later answering questions. The less a developer has to double back to something they set aside, the more time they can dedicate to uninterrupted new feature focus.

Adaptability in Action

So what does this mean for your career?
  • Review your resume. Just because these adaptive skills are harder to measure doesn’t mean they can’t be described. Has there been a time that you’ve suggested a process improvement that reduced time to ship and/or defect rates? Consider extrapolating what the impacts in time or money of not doing that might have been. How about finding an issue during a requirements brainstorming session that made it clear that the design or architecture assumptions needed to take a left turn? Defining the cost of what that mistake might have run might not be in dollars or hours, but helping teammates feel more successful in their work can have morale impacts that save enormous amounts of both time and money in the long run. Consider the points from one of our recruiters, Erika Cober, in her blog on how to write a great tech resume.

  • If you’re already at a job you love, great! You can still help by raising awareness about these topics. Water cooler and lunch conversations, particularly with developers and management, can be great ways to socialize these concepts.

  • Find (or make) opportunities to pair with your developers to have a cultural exchange of ideas and thinking processes. You will both learn a lot and the quality of the products you work on will be better for the breadth of coverage

If you’re responsible for hiring or managing great talent, how can you attract and retain Adaptive Testers?
  • Review your job postings. Are your job descriptions going to attract a checklist of tool knowledge? Or are you encouraging those who have the skillsets of thinking outside the box to bring their creativity to your teams? You might try ‘selling’ your job opportunities like you would products. Your employee talent pool IS a huge component of your product, regardless of what you’re building or selling. Consider the WillowTree careers page for inspiration.

  • Review your interview processes. Do your interviews focus on a candidate proving they can code their way out of a problem? While that should certainly be one aspect of interviews, it’s critical to assess how they handle curve balls. Problem solving, creativity, and integrity can be demonstrated in discussions about handling client or project interactions that went sideways. How did they help right the ship? What did they learn and what did they change going forward? Our WillowTree recruiting team continues to iterate on successful processes for identifying and hiring world class talent. Their tips for hiring for diversity of experience also provides helpful guidance.

  • Look at how (or if!) you are supporting adaptive testers in your company. Are you ensuring that testers are included in meeting invites to project and feature kickoffs? Are you encouraging your developers and testers to collaborate before it’s time to hand a ‘finished’ feature over for testing? Do testers feel encouraged to raise questions in team meetings? Are you fostering a culture of inclusion where testers feel valued as different from but equal to their developer peers? Focusing on any one of these things will have a positive impact. Addressing all of these can be transformative to team morale and product quality.

I hope this offers perspective about the many ways that skilled testing can add value, reduce frustration, enhance client satisfaction and improve team morale. While the hottest toolsets can be useful, they shouldn’t be the primary factor in the definition of a tester. Making sure that you have the flexibility that adaptive testers bring to the table will inherently help you and your teams weather the storms of change that are a part of daily life in software development.

Join our team to work with Fortune 500 companies in solving real-world product strategy, design, and technical problems.

Find Your Role

Xcode Cloud: Choosing the Right Continuous Integration and Delivery Tool for Your Team

Apple recently announced the release of Xcode Cloud at the latest iteration of their...

Read the article