October 27, 2017

Is Mobile Automation Testing Right For You?

In the fast paced software development industry, mobile automation testing has become indispensable. The real value of mobile automation testing is realized when the development team can progress at a rapid pace, without the fear of breaking existing features. Such automation tests free developers to carry out refactoring confidently without fear.

Testing in the world of mobile applications presents its own set of challenges. It is a waste of time for manual test engineering teams to spend a considerable amount of time testing out each feature on so many different devices, mobile operating systems, and versions. The mobile landscape is so complex that no company can afford to build such a vast infrastructure to test their mobile applications with an army of manual test engineers on all possible devices and OS combinations.

Mobile automation testing

Challenges faced in the mobile testing field include:

  • A massive number of mobile devices. This landscape contains mobile phones and tablets of various sizes and densities.
  • Dealing with various hardware configurations that involve different memory optimizations, CPU optimizations, and OS optimizations.
  • Dealing with multiple OS and their versions. iOS and Android each have many versions, and they all need to be dealt with.
  • The multitude of mobile application development methods – native app, responsive web app, and hybrid app.
  • Necessity of frequent app updates.

Not to mention, the cost of managing so many different devices just to enable manual test engineers to test the mobile application is daunting for any organization.

This is where mobile automation testing shines. Mobile Automation testing attempts to address many of these challenges that plague mobile application development. To efficiently carry out any mobile automation testing, we have to rely on some tools.

Tools overview

Some tools available to us for mobile automation testing include:

  • Appium
  • Robotium
  • Selendroid
  • Calabash


Appium is an open source test automation tool for mobile applications. It is used for native app testing, hybrid app testing, and mobile web app testing. It supports most platforms including iOS, Android, Windows, Firefox OS and Mac. It supports running the apps on Android emulators, iOS simulators and real devices with Android, iOS, Windows, and Mac operating systems. This makes Appium a true cross-platform mobile automation testing tool. The underlying philosophy of Appium is that code should be reusable on various platforms. Appium is built on the idea that testing native apps should not involve including an SDK and recompiling your app. It provides standard automation APIs for all platforms. With Appium, you can write tests with any of your favorite development tools using many different programming languages like Javascript, Java, C#, Ruby, Python, PHP. It is extremely easy to set up and works on the Selenium Web Driver API that specifies a client-server protocol called JSON Wire Protocol.

Appium had its beginnings in the year 2012 with Dan Cuellar at Zoosk. It made significant progress in 2013 with Jason from Sauce Labs taking a keen interest in the work that Dan had done so far. In the year 2013, the first truly cross-platform testing framework was born. Appium 1.0 milestone was achieved in May 2014.

You can check out more on Appium here and get started with Appium here.


Robotium is another test automation framework explicitly targeted for Android. It supports testing for both native and hybrid Android apps. It eases the tasks of writing powerful black box UI tests for Android applications. It is similar to Selenium, but is exclusive to Android. It integrates smoothly with Maven, Gradle or Ant to support running tests as a part of Continuous Integration. These tests can be executed on an Android emulator as well as on real Android devices. One aspect to note is that Robotium tests can only run on one device at a time. This means you cannot run one test against a host of devices, which can make scaling difficult. You can develop a Robotium test suite from Android Studio or Eclipse.

Robotium was founded and developed by Renas Reda and was released in January 2010. It is currently at version 5.6.3.

You can check out more on Robotium here and get started with Robotium here.


Selendroid is another mobile test automation framework for Android. It can be used to test a native Android app, a hybrid app, or a mobile web app. Here, tests are written in Selenium 2 Client API. It can be used on Android emulators and real devices. A significant benefit of Selendroid is that it can be added as a node into the Selenium Grid for scaling and parallel testing. It is fully compatible with JSON Wire Protocol, and can handle gestures and interact with multiple Android devices at the same time. The current stable version of Selendroid is 0.17.0.

Check out how to get started with Selendroid here.


Calabash is a Behavior-Driven Development (BDD) test automation framework. It can be used to create and execute automated acceptance tests both for Android and iOS. It is cross-platform, open source and free. It is executable on mobile devices and consists of libraries that enable test code to interact programmatically with native and hybrid apps. This functionality requires Ruby to be running on the machine. It is developed and maintained by Xamarin and can be run on Xamarin Test Cloud. It also supports Cucumber. It can be used with any Ruby-based test framework. Calabash is currently at version 0.5.4.

Check out getting started with Calabash here.

[bctt tweet=”Find out how mobile automation testing shines in the mobile app development world.” username=”3PillarGlobal”]

Creating mobile automation tests using the tools mentioned here is just the first part of the mobile automation testing activity. Once we have the mobile automation tests, we need to execute them on Android emulators, iOS simulators, or actual devices.

Using emulators and simulators is neither an efficient nor foolproof means of testing and verifying any mobile app. The correct approach to testing a mobile app is to run these automated tests on actual devices. This is where we run into our next challenge: how do we ensure availability of so many varied devices with different combinations of OS and versions? This is such a complex and daunting task even without taking into consideration the costs involved in procuring so many devices. This is where automated mobile application testing on cloud comes into the picture. Today, many providers offer cloud-based test labs that can help run your automated mobile tests on emulators, simulators and real devices. These providers free us from the headache of procuring actual devices and the costs associated with it.

Cloud-based automated mobile application testing options:

  1. Google Firebase Test Lab
  2. Saucelabs
  3. Amazon Device Farms
  4. Xamarin Test Cloud

Overview of Cloud-based Test Labs

Google Firebase Test Lab

The Firebase Test Lab provides cloud-based infrastructure for testing Android apps. These tests can be initiated by a single operation and are run over multiple real devices of varying configuration hosted at Google data centers.

There are two ways to test in Firebase Test Lab:

  1. Instrumentation tests – You write tests using the Espresso or Robotium Android test framework.
  2. Robo tests – Use this to exercise your app by simulating user actions.

There are pros and cons to using the Google Firebase test lab. We’ve outlined most of them below:


  • Various emulators and real devices are available for testing.
  • Test results include logs, videos, and screenshots.
  • Even if no test code is written for the app, Test Lab can exercise the app automatically, looking for crashes.
  • Can be used with Continuous Integration systems.


  • Supports only Robotium (Android framework) and doesn’t support Appium tests.
  • Free for a limited period with credit limit, otherwise you can choose from various paid plans.

Check out getting started with Firebase Test Lab for Android here.


Saucelabs is a cloud-based mobile testing platform used to run automated tests both on iOS and Android devices. These tests can be run on a public cloud or a dedicated private cloud. Both Selenium and Appium tests can be run on Saucelabs.

Saucelabs also has different pros and cons, which we’ve outlined below:


  • A wide variety of emulators/simulators and real devices are available, and it supports both Android and iOS.
  • Support for both Robotium and Appium tests.
  • Can be integrated with Continuous Integration systems.


  • Need to upload the testing app on Sauce storage.
  • Performance is slow compared to Amazon Device farm.
  • Free trial for only 14 days.

Check out getting started with Sauce Labs here.

Amazon Device Farm

Amazon is a big player in cloud services and has one of the most robust and powerful cloud infrastructures for automated mobile testing on real devices. Amazon Device Farm is a service to test native apps, hybrid apps and mobile web apps on multiple devices at the same time. It provides test logs, crash logs, videos and screenshots as test results.

There are two ways to test apps in Device Farm:

1. Automated App Testing

Automated App Testing is where you load your automated tests written on a supporting test framework into Device Farm, and then Device Farm will run it for you on various real devices.

Automated App Testing has two further types:

    1. Built-in Test Types – with this, you do not need to create your tests. Device Farm will randomly raise user interface events to apps, exercising it and collecting information around application crashes. Built-in Fuzz is used both for Android and iOS.
    2. Custom Test Types – with this, you create your tests, load them, and Device Farm will run these on actual devices and collect information about application crashes.

Device Farm supports the following test frameworks:

  • Appium for Android, iOS and Web App
  • Calabash for Android and iOS both
  • Instrumentation, UI Automator, and Explorer for Android
  • UI Automation, XCTest for iOS

2. Remote Access Integration

In Remote Access Integration, you specify the device configuration and Device Farm will load the device view on the browser by opening a remote session into the actual device. Once that has been completed, you have the ability to remotely interact with the app on a real device and test out different actions through different application scenarios, like swipe, gestures, and touch. This feature is invaluable for troubleshooting on a specific device configuration.


  • Run tests on a large number of real devices with a configured member, CPU usage and appropriate firmware and software configurations.
  • Simulate real-world environments by configuring location, network connections, and language.
  • Can be integrated with Continuous Integration systems.
  • Manually use remote integration to reproduce end user device issues.


  • Limited free minutes until you have to pay.

You can find out more on Amazon Device Farms here and here.

Xamarin Test Cloud

Xamarin is another cloud-based solution that supports the automated UI testing of mobile apps across multiple devices.

Xamarin Test Cloud consists of five different parts:

  1. Calabash – a BDD framework for writing tests using Cucumber and Ruby (see above).
  2. Xamarin.UITest – This is a framework for writing tests in C# using the famous nUnit library.
  3. Test Cloud – A cloud back-end that hosts physical mobile devices used for loading mobile apps and running the tests.
  4. Xamarin Test Cloud Agent – A lightweight HTTP server that interacts with tests via JSON over HTTP. It acts as a middleman by taking queries from tests and executing them on mobile apps.
  5. Device Agent – This is installed only on iOS applications that are built with Xcode 8.


  • Will run tests on a large number of real devices.
  • Can be integrated with Continuous Integration systems.
  • Tests can be performed manually by a user.


  • Only available as a paid subscription.
  • Does not support Bluetooth, camera, physically rotating devices and simulating battery conditions in a variety of scenarios.

Check out how to get started with Xamarin here.


In most cases, Appium is the best choice out of all the available options. It has a lot more versatility and wide-ranging support, while also having a Selendroid mode. It is an open source tool that provides support for almost all platforms – iOS, Android, Windows, Firefox OS – and supports testing for native, hybrid and mobile web apps. There is a lot of freedom to choose any language in comparison with other frameworks that have a language constraint. It is cross-platform because the same test can be run on iOS and Android using the same API. When using Appium, you don’t need to recompile the app as is the case with other tools where an additional agent needs to be compiled with the app to make it ready for automation. As a result, there is an ambiguity between the app you are testing and the final app submitted to the App Store. Although other frameworks have a few features that are better or similar to Appium, Appium is a lot more versatile and has a broader community, making it a stronger choice for mobile automation.

Similarly, Amazon Device Farm is a step ahead compared to other available cloud frameworks. The most powerful feature of Amazon Device Farm is the ability to not only test simultaneously on multiple devices, but to also test on different OS versions on those devices, and then get a detailed report, screenshots, and logs for each device. It also supersedes the number of the supported platforms, frameworks, and languages in comparison to other cloud solutions.