QA and Testing

An Overview of Roku Test Automation

Roku Devices

Large-scale projects can often be a hectic endeavor for engineering. As these engagements splinter out into less familiar hardware and platforms, the development and testing can get increasingly confusing. We often end up dealing with languages and frameworks that are as unfamiliar as the associated hardware and platforms, increasing the focus and time required to ensure that all versions of the application are of the highest quality. Under normal circumstances, engineers could lean heavily on test automation. With the test automation doing the majority of the repetitive tasks, test engineers have more time to focus on other aspects of testing.

Unfortunately, Roku stuck out as something that was never able to be automated. Roku Channels, written in BrightScript and without any automation support, often became challenging for engineering teams. Test Engineers were forced to execute all tests manually in order to validate builds. But all of that changed on December 9th, 2019 when Roku announced and released their own test automation software! Their automation uses the widely popular Selenium framework and aims to help developers and test engineers expedite the development lifecycle of Roku channels.

How Does Roku Test Automation Work?

While the framework uses Selenium, it doesn’t use it in the way that most people would immediately assume. It uses Selenium’s JSON Wire Protocol, which means that in order to interact with the Roku device, there must be a RESTful web service to receive JSON over HTTP. This is the underlying framework that allows Selenium (and Appium) to communicate between the written automation code and the various WebDrivers. Following the naming convention from Selenium, Roku’s custom web service is simply called the Roku WebDriver.

The automation code is written in a client language such as Java or JavaScript. When the code is compiled and the test is executed, this client language communicates custom Roku JSON Wire Protocol requests to the WebDriver. The WebDriver translates these commands into External Control Protocol (ECP) requests and sends them to the Roku device. The device returns a response in XML to the WebDriver, which in turn translates these responses into a JSON response and sends them back to the originating client. The responses can include various information pertinent to the requests, such as the ID of the current Roku session, or the attributes of elements on the screen. The client code should anticipate these responses in order to continue with the automation or validate a certain behavior. The core functionality of the JSON Wire Protocol is the same between Roku’s use and Selenium 3’s use, with the main noticeable difference being the endpoints supported for Roku automation. Roku’s supported endpoints are listed here.

What Makes Roku Test Automation Different?

The first major difference between Roku test automation and more traditional test automation is the supported language. The automation is based on the JSON Wire Protocol, which means that any language that can make HTTP requests can be used to for Roku automation. However, the initial release only has the HTTP requests easily accessible and fleshed out via the Roku Robot Framework.

The Roku Robot Framework uses the Robot Framework, which itself is built on Python. There are two Python files that do the heavy lifting. The first, is the core of the interaction between JSON Wire Protocol and the WebDriver. The functions within the WebDriver class are used to create the HTTP requests using the module to implement Python’s built in request functionality. The second file,, instantiates a WebDriver, and then uses the HTTP requests provided by the WebDriver to execute a series of commands which cause automated interaction with the Roku device. Because most functions in are tagged with custom keywords, when calling these functions in a .robot test file, the functions can be referenced in a way resembling natural language. For example, instead of having to call the launchTheChannel() function by name, we can instead reference the keyword and execute the function by saying “Launch the Channel”.

The Robot Framework is not one of the most popular frameworks or languages in the test automation world, so if there is confusion about what this is, there is no need to fear. Based on an informal poll done at WillowTree, I learned that of 31 respondents, the majority felt confident in reading and writing JavaScript, Python, and Java, while all responders were aware of these languages. However, when polled about the Robot Framework, less than half were aware of the framework’s existence, and none of the responders felt confident in reading or writing in the framework. Being unfamiliar with a technology can feel overwhelming, but I’ll touch on ways to get around these frustrations later.

The second major difference between Roku test automation and more traditional test automation is the lack of any sort of virtualization. Roku automation requires that the WebDriver be able to communicate directly with a physical Roku device. This is most easily achieved by having the automation code, WebDriver, and Roku device all on the same network. In order to get new builds of a channel onto the Roku Device, the channel must be sideloaded manually, and Roku restricts devices to only having one sideloaded channel on a device at a time. All of the above differences can cause frustrations in development and testing, and unfortunately leave few to no options for integrating automated tests into a CI pipeline. My best suggestion is to have engineers run the automated tests prior to creating their pull requests. While this may leave much to be desired, it is still better than having no way to automate tests.

How Can Roku Test Automation Be Implemented?

Roku Test Automation can easily be implemented following the instructions given here. The instructions are straightforward and thorough, along with examples of sample tests that can be run. If you are unfamiliar with the Robot Framework, there is a lot of documentation on Robot Framework, the Roku Framework Library, and the Roku WebDriver. We’re still working on a strategy to efficiently import the above framework for future projects, but at this moment we’re unable to add the Roku Robot Framework into a test automation suite without copying and pasting the provided python files. The appeal of implementing a test framework that uses the Roku Robot Framework directly is that tests could be written to use the Python functions provided by Roku. This approach theoretically should offer greater support from Roku, as well as reduced overhead in having to develop and maintain the functionality that can interact with the WebDriver.

An alternative approach would be to re-write the Roku Framework Library in another, more well-known language, such as Java or JavaScript. This would require significant effort to do initially, and maintenance would be required whenever a new version of the parent library is published. The positives of having the test automation framework in a familiar language, as well as one that test engineers routinely write tests in, shouldn’t be understated. Using a familiar language would help more test engineers feel confident diving in and writing automation. Additionally, test engineers can more easily include automation that interacts with a web browser to sideload channels, validate Rendezvous linking, and more. Some of us at WillowTree see this option as preferable and have started a task to re-create the framework in JavaScript, which we aim to open-source once we have the project fully working.

Final Thoughts

With Roku’s announcement of their test automation framework, they’ve opened up automation more than it has ever been on Roku devices. While there are still some things left to be desired in terms of automation support, such as a framework in a more well-known language or an emulator to allow for CI automated testing, the impact of this announcement is huge for automation practices around Roku and should allow for a more expedited development lifecycle for Roku channels.

Join our team to work with Fortune 500 companies in solving real-world product strategy, design, and technical problems.

Find Your Role
Four developers looking at computer monitor that's displaying code for a mobile application

The Avengers Initiative for Software Engineers

Hi, my name is Chris Sellek, and I'm here to talk to you about the Avengers Initiative....

Read the article