At Überconf 2016, Neal Ford gave a great presentation titled “Hypothesis-driven Development using Feature Toggles.” In his talk, Ford explained how a simple, yet powerful, technique called feature toggles can be used to achieve true agility in software development. While these ideas are based on enterprise development experiences, we can still use them to implement the mobile app designer’s dream: A/B testing.
A feature toggle is a simple concept: it is a branch point in your app’s code. Based on some pre-determined decision, the branch will go down one path, or it will go down another. Typically, this means the user will either see one set of functionality or they will see another. This can be done statically, in order to hide an incomplete feature, or it can be done dynamically, in order to show functionality to some percentage of uniquely identified users. For example, a static feature toggle could hide a new design in production, but then developers and QA could enable the toggle to work on and test the design. Once the new design is complete, the old design and feature toggle can be removed.
While the concept is simple, in practice features toggles can become complex. There are different types of feature toggles and different techniques for implementing and configuring them. If you want to go much more in depth about feature toggles, I highly recommend the seminal article by Pete Hodgson.
Feature toggles are a tool that can bring a number of different benefits to your app, including:
- Canary release , which rolls out a new feature to a subset of your users as a quick test, before rolling it out to everyone.
- Dark release , which stress tests the infrastructure supporting new features, without actually presenting the functionality to users.
- A/B testing, which is what we will describe below
A/B testing is a way to do a controlled experiment in your app. Rather than guess how users will respond to a new feature, you can roll out the feature to a subset of your users (the variant), and measure their response against the rest of your users (the control). The change could be as simple as a different image size, or as complex as an entirely new workflow. A segment of your users sees the new feature, while the rest of your users are none the wiser.
By running a real-life experiment, we can answer some important questions. Does the change increase user engagement? Is the new feature truly useful to our users? When you don’t know the answers to these questions, you run the risk of building the wrong features into your app. Using A/B testing, you can test ideas in the real world, helping you build a better app that your users will love.
How to Implement A/B Testing in a Mobile App
Feature toggles are the key to A/B testing. They allow you to ship a new feature that will only be seen by the variant group in your experiment. In a traditional web or enterprise app, feature toggles can be controlled entirely on the server-side. In mobile apps, though, there are a few extra considerations. After all, a mobile app is a large distributed system. These are the general steps you will need to take:
- Define a goal for your test, such as getting users to spend more time in the app or leave higher ratings in the store. Then, come up with an idea to meet that goal. This is your hypothesis.
- Determine what segment of your users will see the new idea. This could be a random group, your power users, or a group of new users.
- Prepare a server-side feature toggle configuration that can be activated once the A/B test is ready. This configuration will be based on the segment of users you picked in step 2.
- Ship a new version of the app that contains the new idea, hidden behind a feature toggle. The feature toggle needs to be able to query the server-side configuration. (Note: Unfortunately, on mobile platforms, you can’t force users to upgrade. Make sure to account for this when determining how big your variant segment is.)
- Activate the server-side feature toggle configuration for the variant segment of your users. The app will query the configuration, and users in the variant segment will see the new idea.
- Measure the results. Did you prove or disprove your hypothesis? Which version “won” the experiment?
A great example of how this can be done is Firebase Remote Config with Analytics. Remote Config allows you to set defaults in your app that can be overridden from the Firebase console. With Analytics, you can segment your users and measure the impact of your A/B tests.
The Engineering of Feature Toggles
Feature toggles are a powerful tool, but there are a few things you need to consider before jumping in.
Continuous Integration Best Practices
Is your app built automatically? Are unit tests executed every time a developer checks in code changes? Do you have good unit test coverage? Are the developers confident in the changes they are making? If you can’t answer “yes” to all of these questions, you probably aren’t ready for feature toggles. Using good continuous integration practices gives developers confidence that the changes they are making will not break the app.
Good App Architecture
Does your codebase adhere to SOLID principles? Are simple changes to the app easy to make? Unless you can answer “yes” to these questions, you probably aren’t ready for feature toggles. If your app’s code is spaghetti or a ball of mud, it’s going to be extremely difficult to find the right place to put a feature toggle.
If you can answer “yes” to these questions, consider using dependency injection to manage feature toggles. It is a useful technique that can help keep your codebase clean.
Feature Toggle Management
Features toggles are a deliberate form of technical debt. Once an A/B test is done, the new idea should either be adopted or removed, and the feature toggle needs to be removed along with it. The simplest way to manage this is to add a toggle removal task to the team’s backlog as soon as the toggle is created. And since feature toggles carry with them added complexity, it’s best to limit the number of toggles present in the app.
At its most basic level, a feature toggle is an if (or an if-else) statement. We should be able to rely on the humble if statement, or else we have much bigger problems. Some extra QA attention might be required, but this shouldn’t be more than a simple check to see if the user goes down the expected path.
While not a strict requirement, trunk-based development can help with implementing A/B testing. In trunk-based development, every developer commits their code to the trunk on a daily basis. This is in contrast to long-running feature branches, where a feature is developed in isolation until it is finished. Long-running feature branches present a problem because the new code is not being integrated with other changes that are happening. The result is usually a messy merge to integrate it back into the trunk. Refactoring efforts make this problem worse, so developers are discouraged from improving the code. Trunk-based development is closer to the idea of true continuous integration.
Instead, new features can be hidden behind feature toggles, which are only enabled when the feature is done. Developers can use a technique called branch by abstraction to make small, incremental changes. This means that you can still release other features and bug fixes, all while undergoing a gradual refactor behind the scenes, and without breaking the app! If you’re already using a feature toggle to hide an incomplete feature, it is relatively easy to then turn it into an A/B test once the feature is complete.
Reap the Benefits
After you’ve done an A/B test, you’re armed with real-world data. You can use this data to make better decisions about what to build for your users. So experiment often, and make better apps!