QA and Testing

An Overview of Roku Test Automation

Developing large-scale streaming applications for multiple different platforms can be a hectic endeavour, requiring development and testing for multiple different languages, frameworks, and hardware specifications. In order to ensure the highest quality of the channel on all platforms, engineers lean heavily on test automation. With the test automation doing the majority of the repetitive tasks, test engineers have more time to focus on other aspects of testing.

This is why we were thrilled when Roku announced and released their own test automation software in December 2019! Their automation framework uses the widely popular Selenium framework and aims to help developers and test engineers expedite the development lifecycle of Roku channels.

How Does Roku Test Automation Work?

Roku’s automation framework uses Selenium’s JSON Wire Protocol, which requires a RESTful web service to receive JSON over HTTP in order to interact with the Roku device. This is the underlying framework that allows Selenium (and Appium) to communicate between the written automation code and the various WebDrivers. Following the naming convention from Selenium, Roku’s custom web service is simply called the Roku WebDriver.

The automation code is written in a client language such as Java or JavaScript. When the code is compiled and the test is executed, this client language communicates custom Roku JSON Wire Protocol requests to the WebDriver. The WebDriver translates these commands into External Control Protocol (ECP) requests and sends them to the Roku device. The device returns a response in XML to the WebDriver, which in turn translates these responses into a JSON response and sends them back to the originating client.

The responses can include various information pertinent to the requests, such as the ID of the current Roku session, or the attributes of elements on the screen. The client code should anticipate these responses in order to continue with the automation or validate a certain behavior. The core functionality of the JSON Wire Protocol is the same between Roku’s use and Selenium 3’s use, with the main noticeable difference being the endpoints supported for Roku automation. Roku’s supported endpoints are listed here.

What Makes Roku Test Automation Different?

The first major difference between Roku’s test automation and more traditional test automation is the supported language. The automation is based on the JSON Wire Protocol, which means that any language that can make HTTP requests can be used for Roku automation. However, the initial release only has the HTTP requests easily accessible and fleshed out via the Roku Robot Framework.

The Roku Robot Framework uses the Robot Framework, which itself is built on Python. There are two Python files that do the heavy lifting. The first, webDriver.py is the core of the interaction between JSON Wire Protocol and the WebDriver. The functions within the WebDriver class are used to create the HTTP requests using the api.py module to implement Python’s built in request functionality. The second file, RobotLibrary.py, instantiates a WebDriver, and then uses the HTTP requests provided by the WebDriver to execute a series of commands which cause automated interaction with the Roku device. Because most functions in RobotLibrary.py are tagged with custom keywords, when calling these functions in a .robot test file, the functions can be referenced in a way resembling natural language. For example, instead of having to call the launchTheChannel() function by name, we can instead reference the keyword and execute the function by saying “Launch the Channel”.

The second major difference from more traditional test automation is that Roku’s test automation doesn’t include virtualization. Roku automation requires that the WebDriver be able to communicate directly with a physical Roku device. This is most easily achieved by having the automation code, WebDriver, and Roku device all on the same network. In order to get new builds of a channel onto the Roku Device, the channel must be sideloaded manually, and Roku restricts devices to only having one sideloaded channel on a device at a time.

How Can Roku Test Automation Be Implemented?

Despite the above-mentioned differences, Roku’s test automation framework opens the door to scalably interact with Roku devices in an automated manner. With just a little bit of work and ingenuity, personalized Roku automation solutions can be achieved.

The most obvious way that the automation can be implemented is by simply following Roku’s instructions. The Roku developer documentation includes implementation instructions for the test automation framework. The instructions are straightforward and thorough, along with examples of sample tests that can be run. If you are unfamiliar with the Robot Framework, there is a lot of documentation on Robot Framework, the Roku Framework Library, and the Roku WebDriver.

The alternative is to create your own solution. With the release of the Roku Robot Framework, the network requests to automate the Roku became exposed. Any library or tool that can send customized network requests can interact with a Roku (via the Roku WebDriver).

The first tool I made was a Postman Collection to easily replicate commands to the WebDriver, and also share those commands with my teammates. Instead of using the existing Roku Robot Framework to retrieve the expected responses for network requests to the WebDriver, I can use the Postman Collection to get that same data. The difference is in executing an entire test suite (even if I’ve narrowed down the execution to just a single test with the minimally-required commands to get the response) vs. a singular network call. The collection that I created is hosted on gist and currently includes all of the same functionality that exists in the Roku Robot Framework, and a few other behaviors that are only found in the Development Application Installer.

While becoming an expert on the Robot Framework seemed like an interesting challenge, I wanted the second tool I built to help focus my learning experience on Roku automation. With this goal in mind, I worked with a few teammates to rewrite the Roku Robot Framework in TypeScript, titled Rokul Runnings, allowing us to publish it on npm. Having the project on npm allowed us to simply install the package for future projects, as opposed to having to copy the required files. This extends to the WebDriver as well, since the TypeScript project contains the compiled binary for the WebDriver. The tool is still currently in beta, but we’re looking forward to refining it and giving it a proper release in the near future.

While the Roku automation suite is still in its early days, it is easily extendable to your needs. We were able to adapt the suite to use tools and languages more familiar to our staff, enabling us to scale our automated QA efforts on Roku channels significantly, and with minimal learning curve to our test engineers.

Final Thoughts

With Roku’s announcement of their test automation framework, they’ve opened up automation more than it has ever been on Roku devices. Even though it has some noticeable difference from more traditional automation suites, the framework opens doors to a level of automation that was previously unprecedented on the platform. The impact of this announcement is huge for automation practices around Roku and should allow for a more expedited development lifecycle for Roku channels.

Roku re-published this article on their Developer Blog.

Join our team to work with Fortune 500 companies in solving real-world product strategy, design, and technical problems.

Find Your Role

The Hidden Costs of “Affordable”: Building Applications That Matter

The infamous Healthcare.gov rollout may have just been usurped as the most prominent...

Read the article