Automated Mobile App Testing in Amazon Device Farm

Amazon Device Farm (ADF) is an app testing service that facilitates testing and interaction with Android, iOS, and Web apps on real, physical phones and tablets. All of these devices are hosted by Amazon Web Services. The Amazon Device Farm can be used in two ways:

  • Automated testing of Android, iOS, and Web apps using a variety of available testing frameworks
  • Remote access of real devices onto which apps can be loaded, run, and navigated

In Amazon Device Farm, automated tests can be run using Appium, Calabash, etc. You can read about the complete list of supported test types here.

I have tested an app using Appium Java TestNG framework in ADF. This post is a step-by-step guide on how to run automated test scripts for mobile apps on Amazon Device Farm across a wide range of devices.

Prerequisites

  • Eclipse (For local development/execution)
  • JDK (Java Development Kit) (For local development/execution)
  • Android SDK (For local development/execution)
  • Appium
  • TestNG
  • Selenium-server-standalone
  • Apache Maven

Building Appium Java TestNG Package Using Eclipse

  • Create New Maven Project in Eclipse
  • Add dependencies of Selenium, Appium, and TestNG in the pom.xml
  • Write all test cases in Project name > src (Source) > test > Java > Tests package

Project Structure in Eclipse

The Project Structure will look as it’s shown below in Eclipse:

mobile_testing1

Here is how Amazon Device Farm is set up to test the Android Native app by implementing Appium Java TestNG test onto ADF. The code to do so is given below:

package test.AbstractBaseTests;

import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.annotations.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.TimeUnit;

/**
 * An abstract base for all of the Android tests within this package
 *
 * Responsible for setting up the Appium test Driver
 */
public abstract class TestBase {
    /**
     * Make the driver static. This allows it to be created only once
     * and used across all of the test classes.
     */
    public static AndroidDriver<MobileElement> driver;

    /**
     * This allows the navigation to work within the app.
     * The category name is returned so we can navigate to it from the navigation drawer.
     * @return The name of the Android category
     */
    public abstract String getName();


    /**
     * Method to initialize the test's page
     */
    @BeforeTest
    public abstract void setUpPage();

    /**
     * This method runs before any other method.
     *
     * Appium follows a client - server model:
     * We are setting up our appium client in order to connect to Device Farm's appium server.
     *
     * @throws MalformedURLException An exception that occurs when the URL is wrong
     */
    @BeforeSuite
    public void setUpAppium() throws MalformedURLException{

        final String URL_STRING = "http://127.0.0.1:4723/wd/hub";

        URL url = new URL(URL_STRING);

        //Use a empty DesiredCapabilities object
        driver = new AndroidDriver<MobileElement>(url, new DesiredCapabilities());

        //Use a higher value if your mobile elements take time to show up
        driver.manage().timeouts().implicitlyWait(35, TimeUnit.SECONDS);
    }

    /**
     * Always remember to quit
     */
    @AfterSuite
    public void tearDownAppium(){
        driver.quit();
    }

    /**
     * Restart the app after every test class to go back to the main
     * screen and to reset the behavior
     */
    @AfterClass
    public void restartApp() {
        driver.resetApp();
    }
}

The local environment is now set up with an Eclipse project that consists of TestNG tests.  Now run the test suite locally with Appium and validate the tests. If the test run is successful, then get ready to run the tests on Amazon Device Farm.

Modify pom.xml

A package in .zip format needs to be created to upload on ADF. This package should contain Android Appium Java TestNG tests and all the test dependencies. Pom.xml of the Maven build needs to be modified before creating a .zip package. Refer to this link to meet the requirements during the package stage of a Maven build.

Build Package with Dependencies

Open the command prompt, navigate to workspace, and enter the following command:

mvn clean package -DskipTests=true

A zip-with-dependencies.zip file will be created in the target folder.

AWS Console Steps:

Open the AWS Console and login. Go to the Device Farm and create a project. Inside the project, select Automated tests > Create a new run.

mobile_testing2

Upload the Android APK file. The details will be displayed as shown below once the APK file is uploaded.

mobile_testing3

Configure test: Select test type > Appium Java TestNG and upload the .zip file that contains both the test and dependencies.

mobile_testing4

Select Devices: Select the devices on which you want to run your test. The default device pool can be selected or you can create a new device pool of your choice.

mobile_testing5

Specify Device State: Here we can specify device states to override any device settings. This allows for the adding of additional data and/or the installing of additional apps for ADF to use during the run. Also, we can specify whether Wi-Fi, Bluetooth, GPS, or NFC will be enabled during the run, pre-set the device latitude and longitude for the run, and pre-set the device locale for the run.

mobile_testing6

Review and start run: Here we can change the uploaded APK file, test file (.zip file), and Device pool.

mobile_testing7

Confirm and Start Run: Click on the “Confirm and Start Run” button. This will take a different amount of time depending on the app size and test cases. Test results are displayed once the test run is complete.

mobile_testing8

Reports

Amazon Device Farm test reports provide pass/fail information, logcat captures, device logs, screenshots, videos, and performance data. Click here for detailed information on analyzing reports in ADF.

mobile_testing9 mobile_testing10

Benefits of Using ADF for mobile automation testing

  • ADF Automation tests are useful for Smoke and Regression tests
  • Multiple devices based on make, model, carrier variant, and operating system version can be selected for a test run without purchasing them
  • During a test run, Device Farm captures performance samples every second
  • Reports in Device Farm contain pass and fail information, crash reports, test and device logs, videos, screenshots, and performance data
  • Reports include both detailed per-device data as well as high-level results, such as the number of occurrences of a given problem
  • Device Farm stores reports for 400 days
  • ADF offers backward compatibility, meaning the app can be tested on earlier versions of mobile OS
  • Amazon Device Farm works on Internet Explorer 9 or later and the latest versions of Chrome, Firefox, and Safari

Challenges with using ADF

  • ADF does not provide real time observations on the running tests. Test results are displayed once the test run is complete
  • There is minimal documentation related to issues found during the test run (e.g. if xPath is not matching, etc.)
  • ADF only supports the Java and Python languages
  • The maximum app file size you can upload is 4 GB

Video

Check out how this process works in the video below:

This doesn’t end here. In my next post, “Continuous Testing Using AWS Device Farm,” we will continue our exploration of Amazon Device Farm by looking at continuous integration using this system.

Neha Tomar

Neha Tomar

Module Lead Quality Assurance

Neha Tomar is a Module Lead Quality Assurance Engineer at 3Pillar Global. Neha brings with her vast experience in the Healthcare, E-Learning, e-Commerce and Pharmaceutical domains. She has experience of testing web and mobile applications using Manual testing and automation test tools. Prior to joining 3Pillar Global, she had worked for United Health Group and PureTesting.

3 Responses to “Automated Mobile App Testing in Amazon Device Farm”
  1. Peter on

    Very helpful, I am able to do set-up with this document.

    Thanks.

    Reply
  2. Sean on

    Is this config support for android versions ?

    Reply
  3. Ahmed on

    I am doing all these already, Using testNG,Appium java client 5.0.0, selenium 3.0.1, I have added all the deps which can be seen in my comment below but while running my test on AWS farm getting an exception Failed to invoke configuration method login.startsession:com.google.common.collect.Multimaps.transformValues(Lcom/google/common/collect/ListMultimap;Lcom/google/common/base/Function;)Lcom/google/common/collect/ListMultimap; [TestNG] FAILED CONFIGURATION: “Command line test” – @BeforeTest login.startsession() finished in 0 ms [TestNG] java.lang.NoSuchMethodError: com.google.common.collect.Multimaps.transformValues(Lcom/google/common/collect/ListMultimap;Lcom/google/common/base/Function;)Lcom/google/common/collect/ListMultimap; – See more at: http://www.tothenew.com/blog/how-to-provide-your-testsscripts-dependencies-to-aws-device-farm/#comment-176986

    Here is POM.xml

    4.0.0
    MontyAWS
    montyandroidd
    0.0.1-SNAPSHOT
    jar

    montyandroidd

    http://maven.apache.org

    UTF-8

    org.testng

    testng

    6.9.9

    test

    org.seleniumhq.selenium

    selenium-java

    3.0.1

    io.appium

    java-client

    5.0.0-BETA3

    com.google.guava
    guava
    21.0

    com.google.code.gson
    gson
    2.8.0

    org.seleniumhq.selenium
    selenium-server
    3.0.0

    org.apache.maven.plugins

    maven-jar-plugin

    2.6

    test-jar

    org.apache.maven.plugins

    maven-dependency-plugin

    2.10

    copy-dependencies

    package

    copy-dependencies

    ${project.build.directory}/dependency-jars/

    maven-assembly-plugin

    2.5.4

    package

    single

    zip-with-dependencies

    false

    src/main/assembly/zip.xml

    Reply
Leave a Reply

Related Posts

Take 3, Scene 14: The Present and Future of Angular 2, Part ... On this two-part episode of Take 3, Cassian Lup and Andrei Tamas join us all the way from Romania to discuss the newest iteration of the AngularJS fra...
Take 3, Scene 14: The Present and Future of Angular 2, Part ... On the second part of this two-part episode of Take 3, we continue our conversation with Cassian Lup and Andrei Tamas on the newest iteration of the A...
Spell Check and Autocorrect with Conditional Probability A client of mine wanted to reduce the time it took for his vendors to upload an inventory list to his system. The system currently matches the product...
Jonathan Rivers Interviewed by CXO Today on Cloud and Smart ... Jonathan Rivers, Chief Technology Officer for 3Pillar Global, was recently interviewed by India's CXO Today on 3Pillar's developments in the newest cl...
Blockchain Trends: 2016 Year in Review In October 2015, 3Pillar Global embarked on a trailblazing journey to build a trading platform on blockchain. This post is the first one of a series d...

Free product development tips delivered right to your inbox