Automated Mobile App Testing in Amazon Device Farm

Amazon Device Farm (ADF) is an app testing service that facilitates testing and interaction with Android, iOS, and Web apps on real, physical phones and tablets. All of these devices are hosted by Amazon Web Services. The Amazon Device Farm can be used in two ways:

• Automated testing of Android, iOS, and Web apps using a variety of available testing frameworks
• Remote access of real devices onto which apps can be loaded, run, and navigated

In Amazon Device Farm, automated tests can be run using Appium, Calabash, etc. You can read about the complete list of supported test types here.

I have tested an app using Appium Java TestNG framework in ADF. This post is a step-by-step guide on how to run automated test scripts for mobile apps on Amazon Device Farm across a wide range of devices.

Prerequisites

• Eclipse (For local development/execution)
• JDK (Java Development Kit) (For local development/execution)
• Android SDK (For local development/execution)
• Appium
• TestNG
• Selenium-server-standalone
• Apache Maven

Building Appium Java TestNG Package Using Eclipse

• Create New Maven Project in Eclipse
• Add dependencies of Selenium, Appium, and TestNG in the pom.xml
• Write all test cases in Project name > src (Source) > test > Java > Tests package

Project Structure in Eclipse

The Project Structure will look as it's shown below in Eclipse:

Here is how Amazon Device Farm is set up to test the Android Native app by implementing Appium Java TestNG test onto ADF. The code to do so is given below:

package test.AbstractBaseTests;

import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.annotations.*;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.TimeUnit;

/**
 * An abstract base for all of the Android tests within this package
 *
 * Responsible for setting up the Appium test Driver
 */
public abstract class TestBase {
    /**
     * Make the driver static. This allows it to be created only once
     * and used across all of the test classes.
     */
    public static AndroidDriver<MobileElement> driver;

    /**
     * This allows the navigation to work within the app.
     * The category name is returned so we can navigate to it from the navigation drawer.
     * @return The name of the Android category
     */
    public abstract String getName();


    /**
     * Method to initialize the test's page
     */
    @BeforeTest
    public abstract void setUpPage();

    /**
     * This method runs before any other method.
     *
     * Appium follows a client - server model:
     * We are setting up our appium client in order to connect to Device Farm's appium server.
     *
     * @throws MalformedURLException An exception that occurs when the URL is wrong
     */
    @BeforeSuite
    public void setUpAppium() throws MalformedURLException{

        final String URL_STRING = "http://127.0.0.1:4723/wd/hub";

        URL url = new URL(URL_STRING);

        //Use a empty DesiredCapabilities object
        driver = new AndroidDriver<MobileElement>(url, new DesiredCapabilities());

        //Use a higher value if your mobile elements take time to show up
        driver.manage().timeouts().implicitlyWait(35, TimeUnit.SECONDS);
    }

    /**
     * Always remember to quit
     */
    @AfterSuite
    public void tearDownAppium(){
        driver.quit();
    }

    /**
     * Restart the app after every test class to go back to the main
     * screen and to reset the behavior
     */
    @AfterClass
    public void restartApp() {
        driver.resetApp();
    }
}

The local environment is now set up with an Eclipse project that consists of TestNG tests.  Now run the test suite locally with Appium and validate the tests. If the test run is successful, then get ready to run the tests on Amazon Device Farm.

Modify pom.xml

A package in .zip format needs to be created to upload on ADF. This package should contain Android Appium Java TestNG tests and all the test dependencies. Pom.xml of the Maven build needs to be modified before creating a .zip package. Refer to this link to meet the requirements during the package stage of a Maven build.

Build Package with Dependencies

Open the command prompt, navigate to workspace, and enter the following command:

mvn clean package -DskipTests=true

A zip-with-dependencies.zip file will be created in the target folder.

AWS Console Steps:

Open the AWS Console and login. Go to the Device Farm and create a project. Inside the project, select Automated tests > Create a new run.

Upload the Android APK file. The details will be displayed as shown below once the APK file is uploaded.

Configure test: Select test type > Appium Java TestNG and upload the .zip file that contains both the test and dependencies.

Select Devices: Select the devices on which you want to run your test. The default device pool can be selected or you can create a new device pool of your choice.

Specify Device State: Here we can specify device states to override any device settings. This allows for the adding of additional data and/or the installing of additional apps for ADF to use during the run. Also, we can specify whether Wi-Fi, Bluetooth, GPS, or NFC will be enabled during the run, pre-set the device latitude and longitude for the run, and pre-set the device locale for the run.

Review and start run: Here we can change the uploaded APK file, test file (.zip file), and Device pool.

Confirm and Start Run: Click on the "Confirm and Start Run" button. This will take a different amount of time depending on the app size and test cases. Test results are displayed once the test run is complete.

Reports

Amazon Device Farm test reports provide pass/fail information, logcat captures, device logs, screenshots, videos, and performance data. Click here for detailed information on analyzing reports in ADF.

Benefits of Using ADF for mobile automation testing

• ADF Automation tests are useful for Smoke and Regression tests
• Multiple devices based on make, model, carrier variant, and operating system version can be selected for a test run without purchasing them
• During a test run, Device Farm captures performance samples every second
• Reports in Device Farm contain pass and fail information, crash reports, test and device logs, videos, screenshots, and performance data
• Reports include both detailed per-device data as well as high-level results, such as the number of occurrences of a given problem
• Device Farm stores reports for 400 days
• ADF offers backward compatibility, meaning the app can be tested on earlier versions of mobile OS
• Amazon Device Farm works on Internet Explorer 9 or later and the latest versions of Chrome, Firefox, and Safari

Challenges with using ADF

• ADF does not provide real time observations on the running tests. Test results are displayed once the test run is complete
• There is minimal documentation related to issues found during the test run (e.g. if xPath is not matching, etc.)
• ADF only supports the Java and Python languages
• The maximum app file size you can upload is 4 GB

Video

Check out how this process works in the video below:

This doesn't end here. In my next post, "Continuous Testing Using AWS Device Farm," we will continue our exploration of Amazon Device Farm by looking at continuous integration using this system.