Roku streaming devices have consistently maintained the market share of household streaming devices in the US, and that number continues to grow every year.
Because of this increasing consumer use, it is more important than ever for digital media companies to deliver high quality experiences on Roku for their users. During a recent Roku project at WillowTree, we observed that there aren’t enough online resources for Roku testers and subsequently learned a lot about the unique requirements and limitations involved in testing Roku apps on our own. They can be categorized into four topics: Roku required certification criteria, unique device needs, platform-specific tools, and automation constraints.
1. Incorporate the Roku Certification Criteria into Your Test Plan
Roku provides submission feedback using their certification criteria, which is available to the public. Some items are nice to have in a video player but aren’t required for approval on other platforms (e.g., bookmarking) and others are Roku OS specific (e.g., deep linking).
Determine who is responsible for verifying each criteria and make them critical priorities in your test plan. Be proactive and even get the designers and technical requirements managers on your team involved in making sure these requirements are implemented.
Test these items early and thoroughly. While the team at Roku will give some feedback on failed tests, it isn’t always in the form of full steps to reproduce. It’s best to have all scenarios and edge cases verified by submission time.
2. Understand Your Device Needs
Just because Roku manufactures all of their players doesn’t mean you can get away with testing on a small number of devices. Every generation includes multiple models with different processing power and memory, and there are a large number of models that are no longer manufactured but are still supported and run the latest operating system.
When Roku tests your app in their submission process, they will use a variety of models and OS versions and might report bugs on a specific combination of the two that you will want to be able to reproduce and investigate.
Most importantly, your end users are going to stream your app on a diverse pool of devices, from the new, shiny models currently on the shelves to the five-year-old ones that we both use at home and don’t plan on replacing soon.
When choosing the types of Roku players to have in your test pool, you’ll also want to consider what types of animations will be used in the app and which models will support them. Our app featured a background video player and we had to test both on “high end” players with larger memory capabilities that handled that feature and “low end” players with less memory.
Count the developers on your team - since there is no Roku simulator, they will each need at least one device to build with, as well as one for the team to run CI if applicable. This is in addition to your pool of testing devices.
We would expect the list of Roku devices your team will need to be much longer than that of other OTT platforms; a large pool is insurance against missing specific defects and for increasing confidence that all of your end users will have the best experience possible with your app. At their low price points, Rokus are more attainable within a budget than other devices, making it realistic to have a large stock.
3. Get Familiar With the Roku Dev Environment
There are a few differences between the Roku dev environment and other common set top platforms (Apple TV and Android TV) that will change the flow of your testing. Once your device is in developer mode, only one test app can be installed at a time.
Apps are uploaded over the network via the Roku Developer Dashboard. Roku does provide some native tools to help test that all work over the network; we recommend taking some time to familiarize yourself with the available tools before testing begins.
The biggest challenge we faced during testing was that Roku does not support proxying. We use Charles Proxy every day at WillowTree and that is not currently an option on a Roku project. There are workarounds to this, and we will be detailing how we used (or didn’t use) different tools and how we viewed different types of network traffic in a future post.
4. Remember - No Automation
Our last Roku-specific test consideration was related to no UI automation framework currently being available for the platform.
The Roku Remote Tool has some capabilities for automating remote actions, but is not a robust UI automation solution. This means that all feature testing and smoke testing have to be done manually and therefore tend to take longer amounts of time than with platforms that have UI automation capabilities.
For any testers accustomed to writing automated regression test suites for other platforms, time management will be key when you cannot use automated tests to cut down on final regression time.
Note: In early December 2019, Roku released an automation test framework. For more information on the ins and outs of that framework, see this blog post.