In the fast paced software development industry, mobile automation testing has become indispensable. The real value of mobile automation testing is realized when the development team can progress at a rapid pace, without the fear of breaking existing features. Such automation tests free developers to carry out refactoring confidently without fear.
Testing in the world of mobile applications presents its own set of challenges. It is a waste of time for manual test engineering teams to spend a considerable amount of time testing out each feature on so many different devices, mobile operating systems, and versions. The mobile landscape is so complex that no company can afford to build such a vast infrastructure to test their mobile applications with an army of manual test engineers on all possible devices and OS combinations.
Not to mention, the cost of managing so many different devices just to enable manual test engineers to test the mobile application is daunting for any organization.
This is where mobile automation testing shines. Mobile Automation testing attempts to address many of these challenges that plague mobile application development. To efficiently carry out any mobile automation testing, we have to rely on some tools.
Some tools available to us for mobile automation testing include:
Appium had its beginnings in the year 2012 with Dan Cuellar at Zoosk. It made significant progress in 2013 with Jason from Sauce Labs taking a keen interest in the work that Dan had done so far. In the year 2013, the first truly cross-platform testing framework was born. Appium 1.0 milestone was achieved in May 2014.
Robotium is another test automation framework explicitly targeted for Android. It supports testing for both native and hybrid Android apps. It eases the tasks of writing powerful black box UI tests for Android applications. It is similar to Selenium, but is exclusive to Android. It integrates smoothly with Maven, Gradle or Ant to support running tests as a part of Continuous Integration. These tests can be executed on an Android emulator as well as on real Android devices. One aspect to note is that Robotium tests can only run on one device at a time. This means you cannot run one test against a host of devices, which can make scaling difficult. You can develop a Robotium test suite from Android Studio or Eclipse.
Robotium was founded and developed by Renas Reda and was released in January 2010. It is currently at version 5.6.3.
Selendroid is another mobile test automation framework for Android. It can be used to test a native Android app, a hybrid app, or a mobile web app. Here, tests are written in Selenium 2 Client API. It can be used on Android emulators and real devices. A significant benefit of Selendroid is that it can be added as a node into the Selenium Grid for scaling and parallel testing. It is fully compatible with JSON Wire Protocol, and can handle gestures and interact with multiple Android devices at the same time. The current stable version of Selendroid is 0.17.0.
Check out how to get started with Selendroid here.
Calabash is a Behavior-Driven Development (BDD) test automation framework. It can be used to create and execute automated acceptance tests both for Android and iOS. It is cross-platform, open source and free. It is executable on mobile devices and consists of libraries that enable test code to interact programmatically with native and hybrid apps. This functionality requires Ruby to be running on the machine. It is developed and maintained by Xamarin and can be run on Xamarin Test Cloud. It also supports Cucumber. It can be used with any Ruby-based test framework. Calabash is currently at version 0.5.4.
Check out getting started with Calabash here.Find out how mobile automation testing shines in the mobile app development world. Click To Tweet
Creating mobile automation tests using the tools mentioned here is just the first part of the mobile automation testing activity. Once we have the mobile automation tests, we need to execute them on Android emulators, iOS simulators, or actual devices.
Using emulators and simulators is neither an efficient nor foolproof means of testing and verifying any mobile app. The correct approach to testing a mobile app is to run these automated tests on actual devices. This is where we run into our next challenge: how do we ensure availability of so many varied devices with different combinations of OS and versions? This is such a complex and daunting task even without taking into consideration the costs involved in procuring so many devices. This is where automated mobile application testing on cloud comes into the picture. Today, many providers offer cloud-based test labs that can help run your automated mobile tests on emulators, simulators and real devices. These providers free us from the headache of procuring actual devices and the costs associated with it.
The Firebase Test Lab provides cloud-based infrastructure for testing Android apps. These tests can be initiated by a single operation and are run over multiple real devices of varying configuration hosted at Google data centers.
There are two ways to test in Firebase Test Lab:
There are pros and cons to using the Google Firebase test lab. We’ve outlined most of them below:
Check out getting started with Firebase Test Lab for Android here.
Saucelabs is a cloud-based mobile testing platform used to run automated tests both on iOS and Android devices. These tests can be run on a public cloud or a dedicated private cloud. Both Selenium and Appium tests can be run on Saucelabs.
Saucelabs also has different pros and cons, which we’ve outlined below:
Check out getting started with Sauce Labs here.
Amazon is a big player in cloud services and has one of the most robust and powerful cloud infrastructures for automated mobile testing on real devices. Amazon Device Farm is a service to test native apps, hybrid apps and mobile web apps on multiple devices at the same time. It provides test logs, crash logs, videos and screenshots as test results.
There are two ways to test apps in Device Farm:
1. Automated App Testing
Automated App Testing is where you load your automated tests written on a supporting test framework into Device Farm, and then Device Farm will run it for you on various real devices.
Automated App Testing has two further types:
Device Farm supports the following test frameworks:
2. Remote Access Integration
In Remote Access Integration, you specify the device configuration and Device Farm will load the device view on the browser by opening a remote session into the actual device. Once that has been completed, you have the ability to remotely interact with the app on a real device and test out different actions through different application scenarios, like swipe, gestures, and touch. This feature is invaluable for troubleshooting on a specific device configuration.
Xamarin is another cloud-based solution that supports the automated UI testing of mobile apps across multiple devices.
Xamarin Test Cloud consists of five different parts:
Check out how to get started with Xamarin here.
In most cases, Appium is the best choice out of all the available options. It has a lot more versatility and wide-ranging support, while also having a Selendroid mode. It is an open source tool that provides support for almost all platforms – iOS, Android, Windows, Firefox OS – and supports testing for native, hybrid and mobile web apps. There is a lot of freedom to choose any language in comparison with other frameworks that have a language constraint. It is cross-platform because the same test can be run on iOS and Android using the same API. When using Appium, you don’t need to recompile the app as is the case with other tools where an additional agent needs to be compiled with the app to make it ready for automation. As a result, there is an ambiguity between the app you are testing and the final app submitted to the App Store. Although other frameworks have a few features that are better or similar to Appium, Appium is a lot more versatile and has a broader community, making it a stronger choice for mobile automation.
Similarly, Amazon Device Farm is a step ahead compared to other available cloud frameworks. The most powerful feature of Amazon Device Farm is the ability to not only test simultaneously on multiple devices, but to also test on different OS versions on those devices, and then get a detailed report, screenshots, and logs for each device. It also supersedes the number of the supported platforms, frameworks, and languages in comparison to other cloud solutions.