App Development

QA Guide to Android Nougat Regression Testing

blog-featured-image qa-guide-android-nougat-regression-testing CF-510x296

So a new version of Android is out and your boss says to you, “Make sure our app works on Nougat!” If you have automation frameworks in place those can get you some of the way, but wouldn’t it be nice to have a complete guide to the changes in Nougat and how they affect your existing app? I thought it would be, so I’ve outlined all the changes that have officially occurred in Android with a high likelihood of causing bugs. I’ve also converted them to CSV format for importing into your favorite test-reporting tool. Enjoy!

Multi-Window Support

With Android Nougat, Google added support for using apps side-by-side on the screen simultaneously. This is an exciting time for multitasking power users, but it can create some edge cases for testing. Here are some questions you should focus on if you’re in QA:

  1. Does the app support multi-window?
  • If the app locks orientation then it won’t. This may or may not be a bug depending on your organization.
  1. Does entering or exiting multi-window during a long running operation cause the results of that action to be lost?
  • What I mean here is try some long-running action (like a network request on edge data, for example). While that is occurring, enter multi-window. Based on how multi-window works, it’s possible the app will lose that data if not architected correctly.
  1. Does entering or exiting multi-window cause the app to use the right layouts?
  • If the window is resized from a tablet-sized view to a phone-sized view then the layout should be changed. For bonus points try this out while doing the network request from before!
  1. Does your video pause in multi-window mode?
  • From a development standpoint, entering multi-window mode simulates a scenario where developers would often pause their videos. Not too many things are more frustrating than trying to use multi-window to multitask, only to find out that your video keeps pausing!

Doze Testing

In Android Nougat, Google increased the frequency that Doze would prevent apps from running in the background. This means if your app does something in the background while the user’s phone is in their pocket, then it becomes a prime target for aggressive Doze-ing. So even if you tested against Doze in Marshmallow, you should probably test it again now as it’s much more likely that your app will try to run its background tasks during a Doze period.

So, ask yourself:

  1. Is the app adversely affected by Doze?
  • Prepare your app for whatever background tasks it might perform.
  • Connect your device or emulator to ADB and run the commands as outlined here.
  • Check to see if those background actions are affected in an unexpected way. (Note that background actions not running is almost expected. But if your app needs those to run then the developers will need to find ways around Doze.)

Removed Some App Background Triggers (Broadcasts)

In Nougat, Google worked hard to improve battery life. One of the things they realized was that many apps were registering to run after some common events. Having all of these apps run at the same time was detrimental to the performance of other apps and to battery life. So Google decided to remove a few.

What should you test?

  1. Is the app affected by the removal of CONNECTIVITY_ACTION?
  • CONNECTIVITY_ACTION is a broadcast thrown when the app changes connection status. So if your app lets users know they are on/off WIFI or performs some action when the network status changes, then it’s possible the app is affected. If you can, feel free to string search their codebase to see if “android.net.conn.CONNECTIVITY_CHANGE” or “CONNECTIVITY_ACTION” are in the codebase. If not, then it’s likely this does not affect your app.
  1. Is the app affected by the removal of ACTION_NEW_PICTURE?
  • ACTION_NEW_PICTURE is fired whenever a new picture is taken with the camera and added to the datastore. If the app ever notifies your users of new pictures then it could be using this broadcast. Strings to search for are “ACTION_NEW_PICTURE” and “android.hardware.action.NEW_PICTURE”
  1. Is the app affected by the removal of ACTION_NEW_VIDEO?
  • ACTION_NEW_VIDEO is fired whenever a new video is taken with the camera and added to the datastore. If the app ever notifies your users of new videos then it could be using this broadcast. Strings to search for are “ACTION_NEW_VIDEO” and “android.hardware.action.NEW_VIDEO”

SurfaceView Updates

Nougat was updated so that SurfaceViews would update synchronously with the rest of the app. What this means for QA is that SurfaceViews have changed and should be tested again. For those of you that don’t know what those are, it’s basically a type of View often used for Video and 3D graphics. The upside is that if your app had problems with animating these types of views then that should be improved.

So, what do you test?

  1. Do SurfaceViews in the apps still work?
  • Find the SurfaceViews.

    • Either get your developer to tell you which views are SurfaceViews or just test each Video/3D View in the app to make sure everything still works.
  • Verify they still work.

  1. Do SurfaceView animations still work or look better?
  • Find the surfaceviews as per test #1.
  • Limit the selection to ones that animate.
  • Verify the animation is the same or has improved.
    • Most likely, if your developers animated them previously, they may have added some hacks in to make the animation look good. These updates may break those hacks.

Data Saver

Nougat introduces a new feature called “Data Saver” which users can turn on to limit the network traffic generated by apps. While this has the potential to drastically change your app’s behavior in the future, at the moment it seems to just limit background network functionality when turned on.

So what should you test for regression?

  1. Is my app adversely affected by Data Saver?
  • The easiest way to get into Data Saver mode is to add it to your quick tiles in the notification window. First, drag down the notification window until you see the quick settings and then tap “edit.” From there, you can drag “Data Saver” as a quick-settings tile.
  • Set up your app to do something in the background.
  • Verify that either the action is unaffected by Data Saver or, if affected, it is expected or explained to the user appropriately.

Multi-Locale Support

Nougat introduces support for specifying levels of Locale. Let’s say you mostly speak Spanish but speak some French, too. With Locale, you can specify that apps should be in Spanish, but if Spanish is not supported then French should be used. This also works with assets. While there are advanced ways to use this logic, this should be seamless for old apps. That said, Android’s developer pages stress this should be tested as they have changed how resources are gathered.

So, ask yourself:

  1. Does my app handle text language defaults correctly?
  • Select multiple locales on your phone.
  • Verify text is as expected on each screen.
  1. Does my app handle assets correctly with respect to selected language?
  • Check with the developer to see if there are any language-specific assets or ask them to search their codebase for folders matching src/res/drawable-[language code]. It’s rare for developers to use language-specific folders, so it’s likely you won’t have to worry about this.
  • Verify they are displayed properly with respect to the selected language.

Webview Changes

Nougat introduces a lot of changes to the webview. Webviews are the views used to display web content which could include local HTML or web pages on the internet. What you need to worry about actually depends on whether your developers “target” Nougat for development. You can either ask them or check the build.gradle file in their repository (there are usually two, so be sure to check both) and look for “compileSdkVersion 24.” If they don’t target Nougat yet, then you just need to test #1. Otherwise, you should test all of the below:

  1. Verify your webviews still load and look correct.
  • Chrome is used instead of a webview if Chrome is installed on the device. This behavior change should help all of the apps but there could be unforeseen problems.
  1. Verify your webviews still communicate with the app correctly when “multi-process” is enabled.
  • Some apps have webviews that perform local actions such as uploading images. These are at risk with this new developer option.
  • Turn on multi-process webviews from the developer options and try to perform one of these actions.
  1. Verify your webviews correctly inject javascript.
  • Nougat changed the way javascript injection happens. This can be used by developers for analytics or other functionality. Check with your developer on what is injected and test to make sure that functionality still works.
  1. Verify Geolocation still works.
  • Nougat changed geolocation to only work when using a secure endpoint. Check to make sure any webviews that use geolocation still work.

Accessibility Changes

Nougat added Accessibility directly to the setup screen for the user to make the experience of using your phone better for each individual. If you haven’t tested those then you should. What’s new in Nougat, though?

  1. Verify the app can handle “Display Size” accessibility changes.
  • The user can now modify the display size of the phone for your users. Therefore, even if the display of the device is large, the user can change the display so that all of the dimensions and views are larger.
  • You can find this setting in Accessibility -> Display Size.

Virtual Files

Before Nougat, when an app prompted a user to select files, a user could select a file in a cloud-based app that did not have a bytecode representation. Nougat changes that by allowing apps to return Virtual Files.

How does this affect QA?

  1. Verify that selecting a cloud-based file in your file sharing menu works or matches old functionality.
  • It’s likely your developer has hacked around this fact or done something else to handle this use case. Make sure that this functionality hasn’t changed in Nougat.

Drag and Drop

Nougat changed the behavior of drag and drop events in two ways that could cause problems for your app. First, they allow drag and drop events between Android apps. Second, they changed how those events are passed to the respective views receiving the drag and drop events. On to the tests! (Note, if your app doesn’t support drag and drop then you don’t need to worry about this!)

  1. Verify drag and drop still works within your app.
  • Navigate to the page allowing drag and drop events.
  • Initiate a drag and drop event.
  • Verify that it works with each drop location.
  1. Verify drag and drop events from other apps causes expected behavior.
  • Navigate to the page allowing drag and drop events.
  • Launch another app that allows drag and drop (Keep allows this for text).
  • Enter Multi-window mode.
  • Verify that dragging data from the other app to your app caused expected behavior.

No File URIs allowed

Nougat now prevents “File URIs” from being used in the apps. There are reasons behind this, but what you need to know is how to find an instance where your app uses these. Either ask your developer or do a string search for “file://” on the codebase. Most likely, these File URIs are used when launching a file picking screen for the user or when sharing files to other apps. Assuming you can find them, you only need test  #1 (below). Otherwise, test everything after #1:

  1. Verify File URIs do not cause crashes.
  • Search codebase for “file://” and ask you developer what functionality or screens this affects.
  • Verify this functionality does not crash on Nougat.
  1. Verify sharing files does not crash.
  • Navigate to a screen that allows sharing.
  • Verify sharing a file does not cause a crash.
  1. Verify accepting a file does not cause a crash.
  • Navigate to a screen that allows the user to pick a file from the OS or share a file from another app to your app.Navigate to a screen that allows the user to pick a file from the OS or share a file from another app to your app.
  • Verify that accepting this file does not cause a crash.

NDK Changes

Another thing that changed with Nougat is that apps can no longer link to native libraries in the Android framework outside of the NDK. What QA needs to know is that anything using native code should be tested. The good news? This is not done often and if your developer has been using this, your app should have been showing a Toast to indicate there was a problem. Also, once the native library is loaded it will crash and not just when these libraries are used.

So, how to test?

  1. Verify your app does not use non-NDK libraries.
  • Ask your developer or search for “System.loadLibrary(“ and ask your developer when that code might be called.
  • Verify when native libraries are loaded that the app does not crash.
Quickstart-Guide-to-Kotlin-Multiplatform

A Quick Start Guide to Kotlin Multiplatform

Kotlin Multiplatform, though still experimental, is a great up-and-coming solution...

Read the article