Automated Mobile App Testing in Amazon Device Farm

Amazon Device Farm (ADF) is an app testing service that facilitates testing and interaction with Android, iOS, and Web apps on real, physical phones and tablets. All of these devices are hosted by Amazon Web Services. The Amazon Device Farm can be used in two ways:

  • Automated testing of Android, iOS, and Web apps using a variety of available testing frameworks
  • Remote access of real devices onto which apps can be loaded, run, and navigated

In Amazon Device Farm, automated tests can be run using Appium, Calabash, etc. You can read about the complete list of supported test types here.

I have tested an app using Appium Java TestNG framework in ADF. This post is a step-by-step guide on how to run automated test scripts for mobile apps on Amazon Device Farm across a wide range of devices.

Prerequisites

  • Eclipse (For local development/execution)
  • JDK (Java Development Kit) (For local development/execution)
  • Android SDK (For local development/execution)
  • Appium
  • TestNG
  • Selenium-server-standalone
  • Apache Maven

Building Appium Java TestNG Package Using Eclipse

  • Create New Maven Project in Eclipse
  • Add dependencies of Selenium, Appium, and TestNG in the pom.xml
  • Write all test cases in Project name > src (Source) > test > Java > Tests package

Project Structure in Eclipse

The Project Structure will look as it’s shown below in Eclipse:

mobile_testing1

Here is how Amazon Device Farm is set up to test the Android Native app by implementing Appium Java TestNG test onto ADF. The code to do so is given below:

package test.AbstractBaseTests;

import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.annotations.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.TimeUnit;

/**
 * An abstract base for all of the Android tests within this package
 *
 * Responsible for setting up the Appium test Driver
 */
public abstract class TestBase {
    /**
     * Make the driver static. This allows it to be created only once
     * and used across all of the test classes.
     */
    public static AndroidDriver<MobileElement> driver;

    /**
     * This allows the navigation to work within the app.
     * The category name is returned so we can navigate to it from the navigation drawer.
     * @return The name of the Android category
     */
    public abstract String getName();


    /**
     * Method to initialize the test's page
     */
    @BeforeTest
    public abstract void setUpPage();

    /**
     * This method runs before any other method.
     *
     * Appium follows a client - server model:
     * We are setting up our appium client in order to connect to Device Farm's appium server.
     *
     * @throws MalformedURLException An exception that occurs when the URL is wrong
     */
    @BeforeSuite
    public void setUpAppium() throws MalformedURLException{

        final String URL_STRING = "http://127.0.0.1:4723/wd/hub";

        URL url = new URL(URL_STRING);

        //Use a empty DesiredCapabilities object
        driver = new AndroidDriver<MobileElement>(url, new DesiredCapabilities());

        //Use a higher value if your mobile elements take time to show up
        driver.manage().timeouts().implicitlyWait(35, TimeUnit.SECONDS);
    }

    /**
     * Always remember to quit
     */
    @AfterSuite
    public void tearDownAppium(){
        driver.quit();
    }

    /**
     * Restart the app after every test class to go back to the main
     * screen and to reset the behavior
     */
    @AfterClass
    public void restartApp() {
        driver.resetApp();
    }
}

The local environment is now set up with an Eclipse project that consists of TestNG tests.  Now run the test suite locally with Appium and validate the tests. If the test run is successful, then get ready to run the tests on Amazon Device Farm.

Modify pom.xml

A package in .zip format needs to be created to upload on ADF. This package should contain Android Appium Java TestNG tests and all the test dependencies. Pom.xml of the Maven build needs to be modified before creating a .zip package. Refer to this link to meet the requirements during the package stage of a Maven build.

Build Package with Dependencies

Open the command prompt, navigate to workspace, and enter the following command:

mvn clean package -DskipTests=true

A zip-with-dependencies.zip file will be created in the target folder.

AWS Console Steps:

Open the AWS Console and login. Go to the Device Farm and create a project. Inside the project, select Automated tests > Create a new run.

mobile_testing2

Upload the Android APK file. The details will be displayed as shown below once the APK file is uploaded.

mobile_testing3

Configure test: Select test type > Appium Java TestNG and upload the .zip file that contains both the test and dependencies.

mobile_testing4

Select Devices: Select the devices on which you want to run your test. The default device pool can be selected or you can create a new device pool of your choice.

mobile_testing5

Specify Device State: Here we can specify device states to override any device settings. This allows for the adding of additional data and/or the installing of additional apps for ADF to use during the run. Also, we can specify whether Wi-Fi, Bluetooth, GPS, or NFC will be enabled during the run, pre-set the device latitude and longitude for the run, and pre-set the device locale for the run.

mobile_testing6

Review and start run: Here we can change the uploaded APK file, test file (.zip file), and Device pool.

mobile_testing7

Confirm and Start Run: Click on the “Confirm and Start Run” button. This will take a different amount of time depending on the app size and test cases. Test results are displayed once the test run is complete.

mobile_testing8

Reports

Amazon Device Farm test reports provide pass/fail information, logcat captures, device logs, screenshots, videos, and performance data. Click here for detailed information on analyzing reports in ADF.

mobile_testing9 mobile_testing10

Benefits of Using ADF for mobile automation testing

  • ADF Automation tests are useful for Smoke and Regression tests
  • Multiple devices based on make, model, carrier variant, and operating system version can be selected for a test run without purchasing them
  • During a test run, Device Farm captures performance samples every second
  • Reports in Device Farm contain pass and fail information, crash reports, test and device logs, videos, screenshots, and performance data
  • Reports include both detailed per-device data as well as high-level results, such as the number of occurrences of a given problem
  • Device Farm stores reports for 400 days
  • ADF offers backward compatibility, meaning the app can be tested on earlier versions of mobile OS
  • Amazon Device Farm works on Internet Explorer 9 or later and the latest versions of Chrome, Firefox, and Safari

Challenges with using ADF

  • ADF does not provide real time observations on the running tests. Test results are displayed once the test run is complete
  • There is minimal documentation related to issues found during the test run (e.g. if xPath is not matching, etc.)
  • ADF only supports the Java and Python languages
  • The maximum app file size you can upload is 4 GB

Video

Check out how this process works in the video below:

This doesn’t end here. In my next post, “Continuous Testing Using AWS Device Farm,” we will continue our exploration of Amazon Device Farm by looking at continuous integration using this system.

Neha Tomar

Neha Tomar

Module Lead Quality Assurance

Neha Tomar is a Module Lead Quality Assurance Engineer at 3Pillar Global. Neha brings with her vast experience in the Healthcare, E-Learning, e-Commerce and Pharmaceutical domains. She has experience of testing web and mobile applications using Manual testing and automation test tools. Prior to joining 3Pillar Global, she had worked for United Health Group and PureTesting.

One Response to “Automated Mobile App Testing in Amazon Device Farm”
  1. Peter on

    Very helpful, I am able to do set-up with this document.

    Thanks.

    Reply
Leave a Reply

Related Posts

AWS Device Farm – Mobile App Testing on Real Devices This is an explanation of Amazon Device Farm and how it works for iOS and Android devices. It allows users to create a project, identify an applicatio...
Microsoft Faking Framework for Unit/Integration Testing What is the Faking Framework? The Microsoft Faking Framework is an isolation (mocking) framework that allows us to mock the static, sealed method wit...
Mobile Application Testing is Becoming a Matter of Risk Mana... Let's face it, given the wide range of devices, with their own factory or user customization, it's becoming ever harder and more expensive to test mob...
Challenges of Testing Financial Applications Would financial applications be considered mission-critical? Would they be allowed to fail? And what, in this context, is a failure, anyway? These ...
Enterprise Content Management: An Overview Information may exist in different forms, such as text documents, spreadsheets, images, XML files, web pages, video, audio, and email, or in more fixe...

Free product development tips delivered right to your inbox