Automated Browser Performance Testing using Python and AutoIT

Our project concentrates on a custom web browser, that mimics the behavior of the native Internet Explorer.

What we needed to verify with this session of testing was the performance of the custom browser versus the native IE: startup and shutdown time, webpage loading time, CPU and memory usage. The tests needed to be ran repeatedly on multiple machines and the results saved in a easily readable format.

To automate these tests we needed a tool that was able to interact with with the UI elements of Internet Explorer: menus, menu items, buttons, shortcut keys. We’ve chosen AutoIT. It is designed for automating the Windows GUI and general scripting. It uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks. This is something which is not possible by using other languages like VBScript. AutoIT will also run out-of-the-box on all versions of Windows with with absolutely no ‘runtime’ required. AutoIT was initially designed for PC “roll out” situations to reliably automate and configure thousands of PCs. AutoIT can be used from within a Python script by initializing the AutoIT object at the begining of the script.

Test Environment

The tests were executed on 2 physical machines; these where chosen over virtual machines because they have dedicated hardware and no shared resources. The specs for these machines were:

  • Windows XP: IE8, 2GB RAM, Core 2 duo CPU, 5400 RPM disk
  • Windows 7: IE9, 4GB RAM, Intel i3 CPU, 7200 RPM disk

Both machines were configured in the same way:

System related:

  • sleep / standby / hibernate / turn off monitor / screensaver – Disabled
  • IE set as default browser
  • IE homepage set to Blank
  • all Windows and third party software updates – Off

Test tools:

  • Python 2.7 32b, Pip + Setup Tools, PyWin32 v218 for Python 2.7 32b, xlwt module by pip install
  • AutoIT installed

Application specific:

  • McAfee Endpoint Protection Security Scan Release
  • Symantec Endpoint Protection v 12.1.1101.401
  • Firefox – latest version
  • Chrome – latest version
  • Adobe Flash Player – latest version
  • MS Silverlight – latest version
  • Adobe Reader 11

The “clean” state of each machine was saved in a snapshot using Acronis True Image.

Web Server

Besides the test machine, we have setup an Apache Web Server on a separate machine. This hosts a WordPress website containing a rich media HTML page.

The webpage runs a JavaScript code for counting the page load time and displaying it in the window title. The code snippet is presented later on in the article.

Test scenarios

1) Local webpage load time, memory & CPU usage while browsing

Main test sequence is the following:

  • clear cache
  • open the browser
  • open the local rich media webpage
  • read the loading time when the page is completely loaded
  • read the machine Memory usage
  • read the CPU usage
  • close the browser
  • save result in an Excel spreadsheet

Test steps:

  • restore the “clean” snapshot of the machine
  • automatically run the main test sequence for 1000 times on the native IE
  • automatically run the main test sequence for 1000 times on the custom browser

2) Startup / shutdown time for a browser containing 1500 saved bookmarks

Main test sequence is the following:

  • time the operation: open the browser and wait for the Blank homepage to be displayed
  • time the operation: close browser and wait for it to completely shut down
  • save result in an Excel spreadsheet

Test steps:

  • restore the “clean” snapshot of the machine
  • open native IE, import the Bookmarks file containing 1500 saved bookmarks
  • automatically run the main test sequence for 1000 times on the native IE
  • verify the 1500 bookmarks were correctly imported from the native IE to the custom browser
  • automatically run the main test sequence for 1000 times on the custom browser

3) Timing the following operation: Save a bookmark – Go to Homepage – Load the saved bookmark

Main test sequence is the following:

  • open a webpage hosted on the local web server in the browser
  • wait for the page to load
  • bookmark the page
  • navigate to the Blank homepage
  • open the previously saved bookmark
  • wait for the bookmarked page to load·
  • log the time needed for all the above operations to complete
  • close the browser
  • clear cache
  • save result in an Excel spreadsheet

Test steps:

  • restore the “clean” snapshot of the machine
  • automatically run the main test sequence for 1000 times on the native IE
  • automatically run the main test sequence for 1000 times on the custom browser

Code Implementation

1) Local webpage load time, memory & CPU usage while browsing

For this scenario, we needed to read the webpage load time. Doing this from Python was not exact, because we could only time the period elapsed between the “open webpage in browser” and “close browser” commands. This total time would have included the browser startup and shutdown time as well. Also, we could not verify when the page would fully load. We could not use Selenium, because Selenium is not supported by the custom browser.

The solution that we identified was the following:

  • from the Python script, clear the cache stored by previous sessions
  • from the Python script using Popen, execute a command that opens the browser at the given web page
  • the page is loaded from the local web server
  • once the webpage completely loads, we are printing the loading time in the page title. We are doing this using the following JavaScript code embedded in the html file. This alters the webpage title to “Loading time: xxxx”.
var startTime = new Date().getTime();
window.onload = function () {
var endTime = new Date().getTime();
var totalTime = endTime - startTime;	
document.title = "Loading time:  " + totalTime;
}
loading time
  • from the Python script, wait for the “Loading time:” substring to appear in the browser title, then read the full title
  • extract the result time from the full title
  • read the CPU and memory load using Popen and the wmic windows command
  • close the browser window
  • save the results in a list
  • once all the test sessions are run, print the results to an Excel spreadsheet
from subprocess import Popen, PIPE
from test_base import TestBase
import datetime
import os
import time
import win32com.client
import xlwt


class TestLocalWebpageLoadTime(TestBase):


    def __init__(self):
        self.autoIt = win32com.client.Dispatch("AutoItX3.Control")
        self.autoIt.AutoItSetOption("WinTitleMatchMode", 4)
        

    def open_webpage_in_browser(self, address):
        Popen("c:\Program Files\Internet Explorer\iexplore.exe %s" %address)
        
    def extract_load_time_from_title(self, full_title):
        elements = full_title.split(" ")
        return float(elements[2])/1000
   
    def get_value_from_wmic_output(self, proc):
        return float(str(proc.communicate()).split()[1].split('\\n')[-1])
    
    def get_cpu_load_percentage(self):
        proc = Popen('wmic cpu get LoadPercentage',stdout=PIPE, stderr=PIPE)
        return self.get_value_from_wmic_output(proc)
    
    def get_mem_load_percentage(self):
        proc_free = Popen('wmic os get FreePhysicalMemory',stdout=PIPE, stderr=PIPE)
        proc_total = Popen('wmic ComputerSystem get TotalPhysicalMemory',stdout=PIPE, stderr=PIPE)
        free = self.get_value_from_wmic_output(proc_free)
        total = self.get_value_from_wmic_output(proc_total)/1024
        used = total-free
        load_percent = used/total*100 
        return round(load_percent, 1)
   
	def search_window_with_title_containing(self, string_in_title):
        mainWindowHandle = self.autoIt.WinGetHandle(string_in_title)
        testHandle = "[HANDLE:%s]" %mainWindowHandle
        full_title = self.autoIt.WinGetTitle(testHandle)
        return full_title

    def wait_for_browser_window_to_appear(self, string_in_title, timeout):
        initial_timeout = timeout
        full_title = self.search_window_with_title_containing(string_in_title)
        while (full_title == '0') and (timeout > 0):
            timeout -= 0.5
            time.sleep(0.5)
            full_title = self.search_window_with_title_containing(string_in_title)
        if timeout == 0:
            raise Exception('Window with title containing "%s" did not appear in %ds.' %(string_in_title, initial_timeout))
        else:
            return full_title

	def write_results_to_excel_sheet(self, book, results, sheet_name):
        sheet = book.add_sheet(sheet_name)
        sheet.write(0, 1, sheet_name)
        i=1
        for res in results:
            sheet.write(i, 0, "Run %d" %i)
            sheet.write(i, 1, res)
            i += 1

	def close_browser_window(self, substring_in_title, timeout):
        initial_timeout = timeout
        full_title = self.search_window_with_title_containing(substring_in_title)
        while full_title != '0':
            self.autoIt.WinClose(full_title)
            timeout -= 1
            time.sleep(1)
            full_title = self.search_window_with_title_containing(substring_in_title)
        if timeout == 0:
            raise Exception('Window with title containing "%s" could not be closed in a %ds interval.' %initial_timeout)
       
    def clear_cache(self):
        folder_native = ""
        folder_guest = ""
        if "Windows-XP" in platform.platform():
            folder_native = os.environ['USERPROFILE'] + "\Local Settings\Temporary Internet Files"
            folder_guest = os.environ['USERPROFILE'] + "..."
        elif "Windows-7" in platform.platform():
            folder_native = os.environ['USERPROFILE'] + "\AppData\Local\Microsoft\Windows\Temporary Internet Files"
            folder_guest = os.environ['APPDATA'] + "..."
        os.system("cd %s & rd /s /q ." %folder_native)
        if os.path.exists(folder_guest):
            os.system("cd %s & rd /s /q ." %folder_guest)

 
	def run_one_sequence(self):
        self.clear_cache()
        self.open_webpage_in_browser("http://xxx/wordpress/")
        full_title = self.wait_for_browser_window_to_appear("Loading time:", 60)
        load_time = self.extract_load_time_from_title(full_title)
        cpu_load = self.get_cpu_load_percentage()
        mem_load = self.get_mem_load_percentage()
        self.close_browser_window(full_title, 20)
        return load_time, int(cpu_load), mem_load
    
	   
    def run_test(self, nb_of_runs):
        all_times = []
        all_cpu_loads = []
        all_mem_loads = []
        
        for i in xrange(nb_of_runs):
            try:
                t, cpu_load, mem_load = self.run_one_sequence()
                all_times.append(t)
                all_cpu_loads.append(cpu_load)
                all_mem_loads.append(mem_load)
                time.sleep(5) # pause between runs
            except:
                time.sleep(5)
                os.system("taskkill /IM iexplore.exe /T") # if page is not fully loaded in the given timeout, kill the crashed IE process
                time.sleep(1)
                continue
        self.clear_cache()
         
        book = xlwt.Workbook(encoding="utf-8")
        self.write_results_to_excel_sheet(book, all_times, "All Load Times(s)")
        self.write_results_to_excel_sheet(book, all_cpu_loads, "CPU Loads (%)")
        self.write_results_to_excel_sheet(book, all_mem_loads, "MEM Loads (%)")
        time_stamp = datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
        book.save("results/LocalWebpageLoadTime-WIN7-CUSTOM-%s-%s.xls" %(self.installed_tool_version(), time_stamp))
        
    
    
if __name__ == '__main__':
    TestLocalWebpageLoadTime().run_test(1)

2) Startup / shutdown time for a browser containing 1500 saved bookmarks

Using a mechanism similar to point 1), we do the following operations:

  • open the browser from the Python script
  • wait for the browser window to fully load and display “Blank” homepage
  • close the browser
from subprocess import Popen
from test_base import TestBase
import datetime
import os
import time
import xlwt


class TestStartupShutdownTime(TestBase):

    def launch_native_browser(self):
        Popen("c:\Program Files\Internet Explorer\iexplore.exe")
      
    def wait_for_browser_window_to_appear(self, window_title, timeout):
        initial_timeout = timeout
        while (self.search_window_with_title_containing(window_title) == '0') and (timeout > 0):
            timeout -= 0.1
            time.sleep(0.1)
        if timeout == 0:
            raise Exception('Browser did not launch in the maximum timeout of %ds.' %initial_timeout)
        else:
            startup_time = initial_timeout - timeout
            return round(startup_time, 2)
   
    def close_browser_window(self, window_title, timeout):
        initial_timeout = timeout
        while self.search_window_with_title_containing(window_title) != '0':
            self.autoIt.WinClose(window_title)
            timeout -= 0.1
            time.sleep(0.1)
        if timeout == 0:
            raise Exception('Browser window could not be closed in the maximum timeout of %ds.' %initial_timeout)
        else:
            shutdown_time = initial_timeout - timeout 
            return round(shutdown_time, 2)
   
 

    def run_one_sequence(self, environment):
        if environment == "native":
            self.launch_native_browser()
        elif environment == "custom":
            self.launch_custom_browser()
        startup_time = self.wait_for_browser_window_to_appear("Blank Page", 60)
        shutdown_time = self.close_browser_window("Blank Page", 60)
        return startup_time, shutdown_time

    def run_test(self, nb_of_runs, environment):
        startup_times = []
        shutdown_times = []
        
        for i in xrange(nb_of_runs):
            startup, shutdown = self.run_one_sequence(environment)
            startup_times.append(startup)
            shutdown_times.append(shutdown)
            time.sleep(5)
            
        book = xlwt.Workbook(encoding="utf-8")
        self.write_results_to_excel_sheet(book, startup_times, "Startup time(s) - " + environment.upper())
        self.write_results_to_excel_sheet(book, shutdown_times, "Shutdown times(s) - " + environment.upper())
        time_stamp = datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
        
        book.save("results/BrowserStartupShutdownTime-WIN7-%s-%s-%s.xls" %(environment.upper(), self.installed_tool_version(), time_stamp))    
        
    
        
if __name__ == '__main__':
    TestStartupShutdownTime().run_test(1000, "native")
    TestStartupShutdownTime().run_test(1000, "custom")

3) Timing the following operation: Save a bookmark – Go to Homepage – Load the saved bookmark

This scenario requires interaction with the browser: saving bookmarks, opening homepage, opening the Favorites menu and selecting a bookmark from it. Doing this requires a tool that can interact with the GUI of an application, so we chose AutoIT.

Steps:

  • open a URL in the web browser using the webbrowser Python library (the method used in tests 1) and 2) could have worked here as well, but we tried a new one)
  • after the page is completely loaded, save the bookmark by pressing Ctrl+D shortcut keys
  • go to homepage by pressing the Alt + Home shortcut keys
  • access the previously saved bookmark by opening the menu and selecting the item with the Up arrow key
  • close the browser window
  • remove cache and saved bookmarks
from test_base import TestBase
import datetime
import os
import platform
import time
import webbrowser
import win32com.client
import xlwt
                                                                                                                      
auto = win32com.client.Dispatch("AutoItX3.Control")                                                                   
url="http://172.16.228.48/wordpress"
bookmark="Loading"
                                                                                                                      
class TestAddBookmarksLoadTime(TestBase):                                                                             
    total_wait = 0                                                     
    
    def wait_seconds(self, seconds):
        time.sleep(seconds)
        self.total_wait += seconds
    
    def access_bookmark(self,bookmark):                                                                               
        self.wait_seconds(3)        
        auto.Send("!{A}")
        self.wait_seconds(1)
        auto.Send("{UP}")
        auto.Send("{ENTER}")
        self.wait_for_browser_window_to_appear(bookmark, 20)
                                                                                                                      
    def remove_bookmarks(self):                                                                                       
        folder_native = ""
        folder_guest = ""
        if "Windows-XP" in platform.platform():
            folder_native = os.environ['USERPROFILE'] + "\Favorites"
            folder_guest = os.environ['USERPROFILE'] + "..."
        elif "Windows-7" in platform.platform():
            folder_native = os.environ["USERPROFILE"] + "\Favorites" 
            folder_guest = os.environ["USERPROFILE"] + "..."
        os.system("cd %s & rd /s /q ." %folder_native)
        if os.path.exists(folder_guest):
            os.system("cd %s & rd /s /q ." %folder_guest)
                                                                                                                      
    def go_to_homepage(self):  
        self.wait_for_browser_window_to_appear(bookmark, 20)
        print "wait"
        self.wait_seconds(3)
        print "title=" + self.search_window_with_title_containing("Blank Page")
        while self.search_window_with_title_containing("Blank Page") == '0':
            print "press alt+home"
            auto.Send("{ALT}")
            auto.Send("!{HOME}")                                                                                          
            auto.Send("{ALTUP}")
            print "pressed"
            self.wait_seconds(1)
                                                                                                                      
    def add_bookmark(self, bookmark):                                                                                 
        auto.Send("^d")   
        self.wait_for_browser_window_to_appear("Add", 20)
        self.wait_seconds(1)
        auto.Send(bookmark)                                                                                           
        auto.Send("{ENTER}")                                                                                          
                                                                                                                      
    def run_test_once(self):                                                                           
        self.total_wait = 0
        starttime=time.time()
        browser=webbrowser.get()                                                                                      
        browser.open(url)  
        self.wait_for_browser_window_to_appear(bookmark, 40)
        self.add_bookmark(bookmark)                                                                               
        self.go_to_homepage()  
        self.access_bookmark(bookmark)
        stoptime=time.time()
        self.close_browser_window(bookmark, 10)
        self.remove_bookmarks() 
        self.clear_cache()
        return stoptime-starttime-self.total_wait                                                                                        
                                       
    def run_test_multiple_times(self,times):
        start_list=[]
        for k in xrange(times):
            try:
                print "--- RUN %d ---" %k
                start_list.append(self.run_test_once())
                time.sleep(7)
            except:
                time.sleep(5)
                os.system("taskkill /IM iexplore.exe /T") # if page is not fully loaded in the given timeout, kill the crashed IE process
                self.remove_bookmarks() 
                self.clear_cache()
                time.sleep(1)
                continue
        
        book = xlwt.Workbook(encoding="utf-8")
        self.write_results_to_excel_sheet(book, start_list, "Duration(s)")
        time_stamp = datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
        
        if self.installed_tool_version() == "":
            environment = "NATIVE"
        else:
            environment = "CUSTOM"
        book.save("results/AddBookmarkHomepageAccessBookmark-WIN7-%s-%s-%s.xls" %(environment, self.installed_tool_version(), time_stamp))    
             
                                                                                          
if __name__ == '__main__':                                                                                            
    TestAddBookmarksLoadTime().run_test_multiple_times(1000)  

Results

The results of each run are printed in an Excel spreadsheet. From there, we can easily compile charts.

In general, the custom web browser was slower than the native browser by 5%. This is a very good result, as this overhead is not noticed by a real-life user.

Average MEM Load

 

webpage load times

In some cases, the custom browser was a lot slower than the native one (ie: Run 781); this was due to other system tasks running in the background. However, this was an exception and the custom browser behaved very well.

Webpage Load Times - WIN7

 

Ciprian Balea

Ciprian Balea

QA Lead

Ciprian Balea is a QA Lead who works in 3Pillar’s Timisoara, Romania office. Ciprian began his career in software testing 2 years before he graduated from college. He is proficient with Agile practices and is a Certified Scrum Master. He is most interested in the fields of Continuous Integration & Deployment, and Automation and Performance enhancements. Away from work, Ciprian likes outdoors sports and photography.

2 Responses to “Automated Browser Performance Testing using Python and AutoIT”
  1. Ashok on

    Hi,

    Thanks for the information given
    Can u give more example on webpage testing

    Reply
  2. back on

    hi ,this code can be used with chrome or firefox? Only IE?

    Reply
Leave a Reply

Related Posts

High Availability and Automatic Failover in Hadoop Hadoop in Brief Hadoop is one of the most popular sets of big data processing technologies/frameworks in use today. From Adobe and eBay to Facebook a...
How the Right Tech Stack Fuels Innovation – The Innova... On this episode of The Innovation Engine podcast, we take a look at how choosing the right tech stack can fuel innovation in your company. We'll talk ...
The Road to AWS re:Invent 2018 – Weekly Predictions, P... For the last two weeks, I’ve been making predictions of what might be announced at AWS’ upcoming re:Invent conference. In week 1, I made some guesses ...
Building a Microservice Architecture with Spring Boot and Do... This is the fourth blog post in a 4-part series on building a microservice architecture with Spring Boot and Docker. If you would like to read the pre...
Building a Microservice Architecture with Spring Boot and Do... Part III: Building Your First Microservice, its Container, and Linking Containers We're about ready to actually get started with building a microserv...