Automated Mobile App Testing in Amazon Device Farm

Amazon Device Farm (ADF) is an app testing service that facilitates testing and interaction with Android, iOS, and Web apps on real, physical phones and tablets. All of these devices are hosted by Amazon Web Services. The Amazon Device Farm can be used in two ways:

  • Automated testing of Android, iOS, and Web apps using a variety of available testing frameworks
  • Remote access of real devices onto which apps can be loaded, run, and navigated

In Amazon Device Farm, automated tests can be run using Appium, Calabash, etc. You can read about the complete list of supported test types here.

I have tested an app using Appium Java TestNG framework in ADF. This post is a step-by-step guide on how to run automated test scripts for mobile apps on Amazon Device Farm across a wide range of devices.


  • Eclipse (For local development/execution)
  • JDK (Java Development Kit) (For local development/execution)
  • Android SDK (For local development/execution)
  • Appium
  • TestNG
  • Selenium-server-standalone
  • Apache Maven

Building Appium Java TestNG Package Using Eclipse

  • Create New Maven Project in Eclipse
  • Add dependencies of Selenium, Appium, and TestNG in the pom.xml
  • Write all test cases in Project name > src (Source) > test > Java > Tests package

Project Structure in Eclipse

The Project Structure will look as it’s shown below in Eclipse:


Here is how Amazon Device Farm is set up to test the Android Native app by implementing Appium Java TestNG test onto ADF. The code to do so is given below:

package test.AbstractBaseTests;

import io.appium.java_client.MobileElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.annotations.*;
import java.util.concurrent.TimeUnit;

 * An abstract base for all of the Android tests within this package
 * Responsible for setting up the Appium test Driver
public abstract class TestBase {
     * Make the driver static. This allows it to be created only once
     * and used across all of the test classes.
    public static AndroidDriver<MobileElement> driver;

     * This allows the navigation to work within the app.
     * The category name is returned so we can navigate to it from the navigation drawer.
     * @return The name of the Android category
    public abstract String getName();

     * Method to initialize the test's page
    public abstract void setUpPage();

     * This method runs before any other method.
     * Appium follows a client - server model:
     * We are setting up our appium client in order to connect to Device Farm's appium server.
     * @throws MalformedURLException An exception that occurs when the URL is wrong
    public void setUpAppium() throws MalformedURLException{

        final String URL_STRING = "";

        URL url = new URL(URL_STRING);

        //Use a empty DesiredCapabilities object
        driver = new AndroidDriver<MobileElement>(url, new DesiredCapabilities());

        //Use a higher value if your mobile elements take time to show up
        driver.manage().timeouts().implicitlyWait(35, TimeUnit.SECONDS);

     * Always remember to quit
    public void tearDownAppium(){

     * Restart the app after every test class to go back to the main
     * screen and to reset the behavior
    public void restartApp() {

The local environment is now set up with an Eclipse project that consists of TestNG tests.  Now run the test suite locally with Appium and validate the tests. If the test run is successful, then get ready to run the tests on Amazon Device Farm.

Modify pom.xml

A package in .zip format needs to be created to upload on ADF. This package should contain Android Appium Java TestNG tests and all the test dependencies. Pom.xml of the Maven build needs to be modified before creating a .zip package. Refer to this link to meet the requirements during the package stage of a Maven build.

Build Package with Dependencies

Open the command prompt, navigate to workspace, and enter the following command:

mvn clean package -DskipTests=true

A file will be created in the target folder.

AWS Console Steps:

Open the AWS Console and login. Go to the Device Farm and create a project. Inside the project, select Automated tests > Create a new run.


Upload the Android APK file. The details will be displayed as shown below once the APK file is uploaded.


Configure test: Select test type > Appium Java TestNG and upload the .zip file that contains both the test and dependencies.


Select Devices: Select the devices on which you want to run your test. The default device pool can be selected or you can create a new device pool of your choice.


Specify Device State: Here we can specify device states to override any device settings. This allows for the adding of additional data and/or the installing of additional apps for ADF to use during the run. Also, we can specify whether Wi-Fi, Bluetooth, GPS, or NFC will be enabled during the run, pre-set the device latitude and longitude for the run, and pre-set the device locale for the run.


Review and start run: Here we can change the uploaded APK file, test file (.zip file), and Device pool.


Confirm and Start Run: Click on the “Confirm and Start Run” button. This will take a different amount of time depending on the app size and test cases. Test results are displayed once the test run is complete.



Amazon Device Farm test reports provide pass/fail information, logcat captures, device logs, screenshots, videos, and performance data. Click here for detailed information on analyzing reports in ADF.

mobile_testing9 mobile_testing10

Benefits of Using ADF for mobile automation testing

  • ADF Automation tests are useful for Smoke and Regression tests
  • Multiple devices based on make, model, carrier variant, and operating system version can be selected for a test run without purchasing them
  • During a test run, Device Farm captures performance samples every second
  • Reports in Device Farm contain pass and fail information, crash reports, test and device logs, videos, screenshots, and performance data
  • Reports include both detailed per-device data as well as high-level results, such as the number of occurrences of a given problem
  • Device Farm stores reports for 400 days
  • ADF offers backward compatibility, meaning the app can be tested on earlier versions of mobile OS
  • Amazon Device Farm works on Internet Explorer 9 or later and the latest versions of Chrome, Firefox, and Safari

Challenges with using ADF

  • ADF does not provide real time observations on the running tests. Test results are displayed once the test run is complete
  • There is minimal documentation related to issues found during the test run (e.g. if xPath is not matching, etc.)
  • ADF only supports the Java and Python languages
  • The maximum app file size you can upload is 4 GB


Check out how this process works in the video below:

This doesn’t end here. In my next post, “Continuous Testing Using AWS Device Farm,” we will continue our exploration of Amazon Device Farm by looking at continuous integration using this system.

Neha Tomar

Neha Tomar

Module Lead Quality Assurance

Neha Tomar is a Module Lead Quality Assurance Engineer at 3Pillar Global. Neha brings with her vast experience in the Healthcare, E-Learning, e-Commerce and Pharmaceutical domains. She has experience of testing web and mobile applications using Manual testing and automation test tools. Prior to joining 3Pillar Global, she had worked for United Health Group and PureTesting.

3 Responses to “Automated Mobile App Testing in Amazon Device Farm”
  1. Peter on

    Very helpful, I am able to do set-up with this document.


  2. Sean on

    Is this config support for android versions ?

  3. Ahmed on

    I am doing all these already, Using testNG,Appium java client 5.0.0, selenium 3.0.1, I have added all the deps which can be seen in my comment below but while running my test on AWS farm getting an exception Failed to invoke configuration method;Lcom/google/common/base/Function;)Lcom/google/common/collect/ListMultimap; [TestNG] FAILED CONFIGURATION: “Command line test” – @BeforeTest login.startsession() finished in 0 ms [TestNG] java.lang.NoSuchMethodError:;Lcom/google/common/base/Function;)Lcom/google/common/collect/ListMultimap; – See more at:

    Here is POM.xml

































Leave a Reply

Related Posts

Take 3, Scene 23: The Influence and Evolution of Blockchain Michael Lisse and Derek Tiffany join us in the studio for this episode of Take 3 to talk about the influence and evolution of blockchain in the financ...
The Innovation Engine from TechCrunch Disrupt New York For a very special episode of The Innovation Engine, we're bringing you a number of interviews from the floor of last week's TechCrunch Disrupt New Yo...
Take 3, Scene 22: Minimizing Time to Value On this episode of Take 3, we dive into the concept of Minimizing Time to Value with Andy Zipfel and Paul Doman, two members of 3Pillar's Client Servi...
Inside the Refraction Space, with Rachael Stott On this episode of The Innovation Engine, we bring the podcast inside the Refraction headquarters to talk again with Rachael Stott on how place and sp...
Health & Wellness is a Reactive Industry, but Technolog... Skyrocketing healthcare spending is having far-reaching consequences on the American economy. It contributes to keeping wages stagnant, and crowds out...


Sign up today to receive our monthly product development tips newsletter.