Automated Mobile App Testing in Amazon Device Farm

Amazon Device Farm (ADF) is an app testing service that facilitates testing and interaction with Android, iOS, and Web apps on real, physical phones and tablets. All of these devices are hosted by Amazon Web Services. The Amazon Device Farm can be used in two ways:

  • Automated testing of Android, iOS, and Web apps using a variety of available testing frameworks
  • Remote access of real devices onto which apps can be loaded, run, and navigated

In Amazon Device Farm, automated tests can be run using Appium, Calabash, etc. You can read about the complete list of supported test types here.

I have tested an app using Appium Java TestNG framework in ADF. This post is a step-by-step guide on how to run automated test scripts for mobile apps on Amazon Device Farm across a wide range of devices.

Prerequisites

  • Eclipse (For local development/execution)
  • JDK (Java Development Kit) (For local development/execution)
  • Android SDK (For local development/execution)
  • Appium
  • TestNG
  • Selenium-server-standalone
  • Apache Maven

Building Appium Java TestNG Package Using Eclipse

  • Create New Maven Project in Eclipse
  • Add dependencies of Selenium, Appium, and TestNG in the pom.xml
  • Write all test cases in Project name > src (Source) > test > Java > Tests package

Project Structure in Eclipse

The Project Structure will look as it’s shown below in Eclipse:

mobile_testing1

Here is how Amazon Device Farm is set up to test the Android Native app by implementing Appium Java TestNG test onto ADF. The code to do so is given below:

package test.AbstractBaseTests;

import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.annotations.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.TimeUnit;

/**
 * An abstract base for all of the Android tests within this package
 *
 * Responsible for setting up the Appium test Driver
 */
public abstract class TestBase {
    /**
     * Make the driver static. This allows it to be created only once
     * and used across all of the test classes.
     */
    public static AndroidDriver<MobileElement> driver;

    /**
     * This allows the navigation to work within the app.
     * The category name is returned so we can navigate to it from the navigation drawer.
     * @return The name of the Android category
     */
    public abstract String getName();


    /**
     * Method to initialize the test's page
     */
    @BeforeTest
    public abstract void setUpPage();

    /**
     * This method runs before any other method.
     *
     * Appium follows a client - server model:
     * We are setting up our appium client in order to connect to Device Farm's appium server.
     *
     * @throws MalformedURLException An exception that occurs when the URL is wrong
     */
    @BeforeSuite
    public void setUpAppium() throws MalformedURLException{

        final String URL_STRING = "http://127.0.0.1:4723/wd/hub";

        URL url = new URL(URL_STRING);

        //Use a empty DesiredCapabilities object
        driver = new AndroidDriver<MobileElement>(url, new DesiredCapabilities());

        //Use a higher value if your mobile elements take time to show up
        driver.manage().timeouts().implicitlyWait(35, TimeUnit.SECONDS);
    }

    /**
     * Always remember to quit
     */
    @AfterSuite
    public void tearDownAppium(){
        driver.quit();
    }

    /**
     * Restart the app after every test class to go back to the main
     * screen and to reset the behavior
     */
    @AfterClass
    public void restartApp() {
        driver.resetApp();
    }
}

The local environment is now set up with an Eclipse project that consists of TestNG tests.  Now run the test suite locally with Appium and validate the tests. If the test run is successful, then get ready to run the tests on Amazon Device Farm.

Modify pom.xml

A package in .zip format needs to be created to upload on ADF. This package should contain Android Appium Java TestNG tests and all the test dependencies. Pom.xml of the Maven build needs to be modified before creating a .zip package. Refer to this link to meet the requirements during the package stage of a Maven build.

Build Package with Dependencies

Open the command prompt, navigate to workspace, and enter the following command:

mvn clean package -DskipTests=true

A zip-with-dependencies.zip file will be created in the target folder.

AWS Console Steps:

Open the AWS Console and login. Go to the Device Farm and create a project. Inside the project, select Automated tests > Create a new run.

mobile_testing2

Upload the Android APK file. The details will be displayed as shown below once the APK file is uploaded.

mobile_testing3

Configure test: Select test type > Appium Java TestNG and upload the .zip file that contains both the test and dependencies.

mobile_testing4

Select Devices: Select the devices on which you want to run your test. The default device pool can be selected or you can create a new device pool of your choice.

mobile_testing5

Specify Device State: Here we can specify device states to override any device settings. This allows for the adding of additional data and/or the installing of additional apps for ADF to use during the run. Also, we can specify whether Wi-Fi, Bluetooth, GPS, or NFC will be enabled during the run, pre-set the device latitude and longitude for the run, and pre-set the device locale for the run.

mobile_testing6

Review and start run: Here we can change the uploaded APK file, test file (.zip file), and Device pool.

mobile_testing7

Confirm and Start Run: Click on the “Confirm and Start Run” button. This will take a different amount of time depending on the app size and test cases. Test results are displayed once the test run is complete.

mobile_testing8

Reports

Amazon Device Farm test reports provide pass/fail information, logcat captures, device logs, screenshots, videos, and performance data. Click here for detailed information on analyzing reports in ADF.

mobile_testing9 mobile_testing10

Benefits of Using ADF for mobile automation testing

  • ADF Automation tests are useful for Smoke and Regression tests
  • Multiple devices based on make, model, carrier variant, and operating system version can be selected for a test run without purchasing them
  • During a test run, Device Farm captures performance samples every second
  • Reports in Device Farm contain pass and fail information, crash reports, test and device logs, videos, screenshots, and performance data
  • Reports include both detailed per-device data as well as high-level results, such as the number of occurrences of a given problem
  • Device Farm stores reports for 400 days
  • ADF offers backward compatibility, meaning the app can be tested on earlier versions of mobile OS
  • Amazon Device Farm works on Internet Explorer 9 or later and the latest versions of Chrome, Firefox, and Safari

Challenges with using ADF

  • ADF does not provide real time observations on the running tests. Test results are displayed once the test run is complete
  • There is minimal documentation related to issues found during the test run (e.g. if xPath is not matching, etc.)
  • ADF only supports the Java and Python languages
  • The maximum app file size you can upload is 4 GB

Video

Check out how this process works in the video below:

This doesn’t end here. In my next post, “Continuous Testing Using AWS Device Farm,” we will continue our exploration of Amazon Device Farm by looking at continuous integration using this system.

Neha Tomar

Neha Tomar

Module Lead Quality Assurance

Neha Tomar is a Module Lead Quality Assurance Engineer at 3Pillar Global. Neha brings with her vast experience in the Healthcare, E-Learning, e-Commerce and Pharmaceutical domains. She has experience of testing web and mobile applications using Manual testing and automation test tools. Prior to joining 3Pillar Global, she had worked for United Health Group and PureTesting.

7 Responses to “Automated Mobile App Testing in Amazon Device Farm”
  1. Peter on

    Very helpful, I am able to do set-up with this document.

    Thanks.

    Reply
  2. Sean on

    Is this config support for android versions ?

    Reply
  3. Ahmed on

    I am doing all these already, Using testNG,Appium java client 5.0.0, selenium 3.0.1, I have added all the deps which can be seen in my comment below but while running my test on AWS farm getting an exception Failed to invoke configuration method login.startsession:com.google.common.collect.Multimaps.transformValues(Lcom/google/common/collect/ListMultimap;Lcom/google/common/base/Function;)Lcom/google/common/collect/ListMultimap; [TestNG] FAILED CONFIGURATION: “Command line test” – @BeforeTest login.startsession() finished in 0 ms [TestNG] java.lang.NoSuchMethodError: com.google.common.collect.Multimaps.transformValues(Lcom/google/common/collect/ListMultimap;Lcom/google/common/base/Function;)Lcom/google/common/collect/ListMultimap; – See more at: http://www.tothenew.com/blog/how-to-provide-your-testsscripts-dependencies-to-aws-device-farm/#comment-176986

    Here is POM.xml

    4.0.0
    MontyAWS
    montyandroidd
    0.0.1-SNAPSHOT
    jar

    montyandroidd

    http://maven.apache.org

    UTF-8

    org.testng

    testng

    6.9.9

    test

    org.seleniumhq.selenium

    selenium-java

    3.0.1

    io.appium

    java-client

    5.0.0-BETA3

    com.google.guava
    guava
    21.0

    com.google.code.gson
    gson
    2.8.0

    org.seleniumhq.selenium
    selenium-server
    3.0.0

    org.apache.maven.plugins

    maven-jar-plugin

    2.6

    test-jar

    org.apache.maven.plugins

    maven-dependency-plugin

    2.10

    copy-dependencies

    package

    copy-dependencies

    ${project.build.directory}/dependency-jars/

    maven-assembly-plugin

    2.5.4

    package

    single

    zip-with-dependencies

    false

    src/main/assembly/zip.xml

    Reply
  4. Prashant Bale on

    This is really very nice article. I need more information regarding AWS device farm.
    I have created test project pretty much similar to the one which is mentioned in the blog, but I have added cucumber layer. I have written feature files as well.

    Now, I am not sure how to upload test project on AWS device farm. Under “Configure a test” section, which one to choose ?

    Please help.

    Reply
  5. Anfas Ansary on

    My Appium script perfectly working on local but moving to aws device farm, returns parsing error because of one class file.
    I am trying to import data from an excel file within this class file. I think error because of the path of excel file.

    I upload the data excel file as extra data in aws but i can’t find out the location.

    public static void changeCity() throws InterruptedException{
    try{
    File src = new File(“data1.xls”);
    Workbook wb = Workbook.getWorkbook(src);
    Sheet sh1 = wb.getSheet(0);
    Sheet bugzillaUpdation = wb.getSheet(“UtilityCredentials”);
    …..

    Please help me to resolve the issue.

    Thanks
    Anfas

    Reply
  6. sandip on

    Can you please provide sample project ?
    How can we control the test execution order using testng.xml file on aws device farm?

    Reply
    • sandip on

      Ho to configure pom.xml to execute test cases from testng.xml?

      Please provide sample pom.xml…. I am able to execute test cases on aws device farm, however unable to control the execution order using testng.xml…

      Reply
Leave a Reply

Related Posts

Interviews from Industry Summit 2017: the Product Conference On this special double episode of The Innovation Engine, the 3Pillar team interviews many of the speakers who took the stage at Industry Summit 2017. ...
Designing the Future & the Future of Work – The I... Martin Wezowski, Chief Designer and Futurist at SAP, shares his thoughts on designing the future and the future of work on this episode of The Innovat...
The 4 Characteristics of a Healthy Digital Product Team Several weeks ago, I found myself engaged in two separate, yet eerily similar, conversations with CEOs struggling to gain the confidence they needed t...
Recapping Fortune Brainstorm Tech – The Innovation Eng... On this episode of The Innovation Engine, David DeWolf and Jonathan Rivers join us to share an overview of all the news that was fit to print at this ...
4 Reasons Everyone is Wrong About Blockchain: Your Guide to ... You know a technology has officially jumped the shark when iced tea companies decide they want in on the action. In case you missed that one, Long Isl...