July 17, 2013

Mobile Automated Testing With Perfecto Mobile Part i


The Perfecto MobileCloud is a completely web based Software as a Service (SaaS) platform that allows mobile application developers and QA Engineers  to work with services like advanced automation, monitoring and testing services. It is the right tool in the hands of a mobile app tester to automate testing for mobile phone devices connected with diverse mobile networks spread across different geo-locations.

The automated testing is done using an intuitive web-based user interface design, which allows the creation of tests scripts which are easy to generate and easy to use with no coding involved at all. The commands comprising the scripts are readily available on the websites in the form of widgets. Test developers just need to click on the commands to add to the scripts and define its properties in the user interface. Keyword based Script Once technology enables QA Engineers to create test scripts. Perfecto Mobile testing tool works on image or text recognition. The generation of these scripts can be watched in the following videos.

Test developers can easily install third Party applications or APK files on the remote devices using their Install widget. Upon deployment, the application will be visible in the Application list of the device.

The automation tools work on the basis of OCR based text and image recognition mechanism. This enables the tool to analyze and test the applications's user interface without any limitation. From native apps, web based apps, or hybrid apps tools it can test anything. As far as Flash content is considered, the sales team are not sure whether the tools can test this or not.

For both the tools, there exists technical support if we require them.

Perfecto Mobile Cloud provides a Hybrid objects approach, which consists of both Visual (OCR) based objects and native objects for iOS/Android applications, as well as native DOM objects for Web/Hybrid based applications. It offers both HP UFT or a native automation framework, an object spy tool which allows easy object filtering and identification and easy scripting.

They also provide 3 different cloud based deployment options as part of their offering:
Public cloud, Private cloud (either hosted or on-premise) and Local solution (within the customer local LAN network) - called MobileCloud Express.

Early in the mobile app development process it usually makes sense to conduct manual testing while the test strategy is being refined. Then once the testing processes are standardized, automated testing starts to deliver significant benefits.

Perfecto Mobile Cloud

The user license with Perfecto Mobile offers these variations:

1. Perfecto Mobile Cloud Interactive
Using Perfecto MobileCloud Interactive services, a user can quite easily test websites, services or mobile applications even manually. They can hold the device in their hands and perform testing on multitude of mobile handsets and devices  connected to live networks using the web. The devices operates seamlessly in all the major mobile networks across various countries. It has active mobile data plans and enables you to make/receive calls, send/accept messages, install your apps and browse the Internet. The free trial for this can be taken up for 1 hour. You can take view of how to interact with the devices:

Perfecto MobileCloud Interactive services


  • Users can access logs generated by the remote devices through the Device Log widget available on the website.
  • Enables access to various other functionality which captures internal performance of the device like, Battery Level, Battery voltage, CPU usage, Memory Usage, Memory Free, Bytes In and Bytes out for Mobile/WLAN of the device.
  • Enables to share live session of the device with other users to demo new apps/features, reproduce a bug. It allows me to just share the session URL or invite other participants. The session would be secured. Stopping the session would end the sharing mode.
  • Just before releasing the device, users can execute Clean Up widget on the device to make sure that application is uninstalled from the device.
  • The widgets on the website enables the users to change orientation view of the application, change the orientation of the device, zoom in/out the view of the device screen, position the device on the screen etc.
  • Users can Reserve a particular device for said period of time (paid feature).
  • The real handset devices have SIM card inserted in them and so the Device panel shows the carrier information, network information, phone number, location and many other information of the SIM card. To verify this, I called the phone number of Motorola Razr as available on their website which is reflected on the mobile screen:

Perfecto Mobile Cloud

2. Perfecto Mobile Cloud Automation and Mobile Cloud Reporting
MobileCloud-Automation allows us to execute automated testing on real handsets and tablets located anywhere in the world.

  • The test scripts are written using keyword-based Script Once Technology. The creation of scripts are easy.
  • The scripting technology is keyword driven and business requirements driven test scripts.
  • The generated test scripts are highly device agnostic. Same test scripts can be executed across all mobile handsets irrespective of OS, platform or manufacturer of the device without any modification. However, specific test cases can be defined for specific devices.
  • The test script generates detailed report of the executed test script at the end with vast variety of information such as actual results, screen captures in digital format, continuous video recording of the entire script and graphs showing statistics about the execution.
  • The test report can be shared with other users and can also be exported to HTML, XML, PDF, CSV format.
  • The test script can perform:
    • Functional Testing
    • User Interface Testing: In this, the script is able to capture UI distortions like truncated/overlapping texts. It can test alignment of views to each other and their respective placement on the screen. The screen layout can define relationship between the views (above /below), spacing between them (in percentage) etc.
    • Performance Testing: In this tool, there exists full control on the flow of the application like conditions, loops, flow commands, expressions and data-tables. While writing the test scripts, set of user actions can be written in the loop. These set of actions would comprise of all the activities which we want to test against its performance.
    • Testing of Web based content or Hybrid App: The automated test script scans/analyzes the mobile screens and works on Text and Image based recognition (OCR) mechanism. So the application can either be native, browser app or hybrid app, the test script analyzes the components placed on the screen. This includes testing of two different package applications in one test script (example, native application invoking browser application).
  • Data driven Testing approach can be followed in these devices. The mobile cloud computing supports creation of data tables in the private repository and can also import the data through csv file format.

Among hardware features:

  1. Camera is accessible to the users but the view behind the camera is grayed out. In this aspect, some camera based functionality such as taking snapshot of the vicinity or barcode scanning through camera would not be working. According to the sales team of Perfecto Mobile Cloud, they would provide an API, in near future, to inject an image to the camera API so that mobile apps can be tested.
  2. GPS based APIs work on these devices. Once the GPS is turned on, APIs based on this technology would work. According to the Perfecto Mobile Cloud sales team, the tool facilitates to set the direction to the GPS/maps.

Video1: Perfecto Mobile - Getting Started

Video 2: Perfecto Mobile - Mobile Automated Testing


3. Perfecto Mobile Cloud Monitoring

Provides 24 by 7 user experience design monitoring of mobile services and Real Handsets and Tablets.

Although mobile automation tool would not eradicate the manual testing process, there exists few reasons to automate testing of smartphones native applications:

  1. It saves time and effort. Repetitive testing on each iteration would be now be less tedious. This would also allow QA to focus on quality.
  2. It would expand test case coverage in the application and finally release reliable/robust application. In all, improving the app quality.
  3. Same test case scripts can be executed on different mobiles irrespective of operating systems and language it supports. This covers a wide range of devices (Android, iOS, Blackberry etc) on which automated testing can be performed in very less time.
  4. Improves effectiveness of QA testing process.

Following are the mobile phone functionalities which were tested on Android phones while creating the automated test scripts:

1. User Interface controls and actions performed on them:

  • Button Click
  • Radio Button Click
  • TextView Click
  • Image Click
  • Checkbox Click
  • Text Input In Edittext
  • Text Input to Edittext when there are multiple Edittext present on the screen (since the test script does not work on Ids).
  • Listview Row Click
  • Button Click on AlertDialog

2. Automated Scrolling of the Webview or Listview UI component and performing the actions mentioned in the Test Script.

3. Validates and verifies Image displayed in the application (either in native application or in web page).

4. Validates and verifies the text anywhere in the application including Alert dialogs/Activities/other UI components and compares with the expected string value.

5. Two or more applications can be tested in one test script. This also implies to testing of native and web applications in one test script.

6. Search mechanism can be implemented in the current screen of the application. The tool can search for an image or texts mentioned in the test script in the displayed Activity of the application and can perform the actions on them.

7. Wait mechanism can be implemented in the test script. This can be for a specified period of time (let’s say 15 seconds) or until that a Text/Image appears on the screen (Text Sync, Image Sync, and Image Checkpoint).

8. Captures and validates UI distortions like:

a.        Overlapping texts/images

b.       Truncated texts/images

c.        Misplaced texts/Images

d.       Alignment of UI elements to each other ( to a certain extend).

9. Conditional testing on actions can be implemented. Based on state of the variable/application/UI component, further testing can be chosen. If/ Else commands can be added.

10. Looping command can be added in the test script. A set of actions can be executed in loops.

11. Reports can be generated in various formats like HTML, XML, PDF etc. These reports can be downloaded or saved for future reference. The test execution can be recorded and played afterwards. These can also be shared on mails in the form of videos.

12. Maintains repository online which is helpful in storing Images/Scripts/XML etc. for easy retrieval.

13. Positioning of UI elements can be tested on relationship basis. Each UI element can be tested whether it is placed at Top/Bottom/Left/Right of other UI element.

14. Automated Launch/Close/Install/Uninstall of a particular application can be done.

15. Script Actions can be commented or excluded for later execution.

16. Camera on mobiles is accessible but taking a snapshot from it is not useful.

17. Hardware keys works: Back Key, Menu key, Volume Up/down, Power off.

18. Error handling mechanism is supported in the text script: catch/break/continue etc.

19. Data driven testing can be done. The tool supports creation of database online or import the database from local drive.

iPhone Web Browser Automated Testing:

Web browser controls and actions performed on them:

1.    Launching of web browser

2.    Selecting address bar of the browser

3.    Navigating to a particular browser

4.    Text input in text field.

5.    Click on an image

6.    Click on hyperlink

7.    Waiting for the web page to load

8.    Moving Back and forward on web browser

9.    Handling the alert shown in between. Selecting Ok /Cancel on Alert.

A robust software testing methodology must weigh the pros and cons of going ahead with a mobile automation approach. Many a times, creation of auto tests can be a time consuming exercise in comparison to the traditional approach of manual mobile testing. For Quality Assurance Software Engineers, it is the flexibility and convenience of analyzing results that often sways the balance in the favor of mobile automated testing.