Browsermob Programmatic Proxy for Automated Testing

The web applications we build and test these days are far from a simple collection of HTML pages designed and coded from scratch by a small team of people. Some of the functionalities we see on sites are not even developed by the engineers coding the web page, but actually were created by different third parties. Such third party services are thereafter embedded wherever they are needed.

You can have third parties for adverts, login, comments, video content, statistics and so on. These are usually embedded in a website by calling a script from the third party vendor. For example, the following code embeds YouTube API functionality onto your webpage:

<script src="https://www.youtube.com/iframe_api"></script>

One common problem a website encounters when embedding many third parties is slow loading speed, due to the third parties either not responding or responding very slowly. This is a problem when running automation tests that load the web page, because it can result in long running times or even false negative results.

To solve this problem, we can use a programmatic proxy to block the calls to and from the third parties we select. The following code examples will show how we can use the Browsermob Proxy with Capybara and Ruby to filter out unwanted HTTP calls.

Installing the Browsermob Proxy Pack

From the command line, install the Browsermob Proxy Ruby gem

gem install browsermob-proxy

Download the Browsermob Proxy tool, available here, and unpack it into your automation project folder like so:

broswermob

 

Starting the Proxy from the Code and Defining a Blacklist

First, we have to reference the Browsermob Proxy library we’ve installed with gem.

require 'browsermob/proxy'

Then we kick off the Browsermob by creating a new proxy object.

server = BrowserMob::Proxy::Server.new('./resources/browsermob-proxy-2.0.0/bin/browsermob-proxy', :port => 8181)  # port number for ruby to communicate with the browsermob tool
server.start
proxy = server.create_proxy(5454)    # port number for online proxy

After this, we need to create a blacklist in which we introduce all of the URLs that we want to block, with each of them in a regular expression. We also need to define the response code that the proxy should send when a request comes for a blacklisted site – in this case we’ll use 404.

proxy.clear_blacklist
proxy.blacklist(/^.+youtube.com\/.+$/, 404)

Once we define the full blacklist, we will need to specify a filename for the Browsermob Proxy to log all of the traffic.

proxy.new_har "my_har"

Now we can save the proxy object in a Selenium profile that will be passed to the browser running the test.

profile = Selenium::WebDriver::Firefox::Profile.new
profile.proxy= proxy.selenium_proxy
Capybara::Selenium::Driver.new(app, :browser => driver.browser, 
:http_client => client, :profile => profile)

After defining all of the actions that our test will perform, we will need to save the HAR file and close the proxy while when cleaning up the test environment.

proxy.har.save_to "./resources/hars/my_har.har"
proxy.close

Filtering out third parties that are not needed by a test can be very beneficial: we can better test the application in isolation and increase the speed and reliability of the test. The Browsermob Proxy is a very powerful tool when attempting this.

Ciprian Balea

Ciprian Balea

QA Lead

Ciprian Balea is a QA Lead who works in 3Pillar’s Timisoara, Romania office. Ciprian began his career in software testing 2 years before he graduated from college. He is proficient with Agile practices and is a Certified Scrum Master. He is most interested in the fields of Continuous Integration & Deployment, and Automation and Performance enhancements. Away from work, Ciprian likes outdoors sports and photography.

3 Responses to “Browsermob Programmatic Proxy for Automated Testing”
  1. Sanjay Ahuja on

    Hi Ciprian,

    When I am getting following error on passing port number in Create_Proxy, Any idea?
    /Library/Ruby/Gems/2.0.0/gems/browsermob-proxy-0.2.0/lib/browsermob/proxy/client.rb:16:in `from’
    /Library/Ruby/Gems/2.0.0/gems/browsermob-proxy-0.2.0/lib/browsermob/proxy/server.rb:48:in `create_proxy’

    Reply
  2. Ciprian Balea on

    Hi Sanjay

    When installing the Browsermob Proxy gem, please make sure you install “browsermob-proxy”, NOT “browsermob-proxy2” (see https://rubygems.org/search?utf8=%E2%9C%93&query=browsermob-proxy)

    After this, please make sure you have the latest Java version and you are using it to start the file from /resources/… There are known issues with Java 6 and some old versions of Java 7; i recommend using Java 8.

    Reply
  3. mike on

    Hi Ciprian, thanks for post. i am trying to use this to whitelist an IP range to test access to my website. I am having problems following your post on how to set this up. I am using ruby and have set the gem in my Gemfile and run bundle install to instal this and so far all is good. Version instalaled (browsermob-proxy (0.2.0))
    I now need some help configuring this to my project. are you able to share your step by step or get in touch to support me with this?

    Many thanks
    mike

    Reply
Leave a Reply

Related Posts

3 Topics We Should Be Talking About at BrainstormTech Our 3Pillar clients, regardless of industry, share one common trait - they need help strategizing, designing, and delivering revenue-generating digita...
4 Reasons Everyone is Wrong About Blockchain: Your Guide to ... You know a technology has officially jumped the shark when iced tea companies decide they want in on the action. In case you missed that one, Long Isl...
3 Cloud Optimization Projects That Will Pay for Themselves i... AWS introduced 1,430 new features and tools in 2017, including 497 in the 4th quarter alone. This means that it can be a challenge for even the mos...
The Connection Between Innovation & Story On this episode of The Innovation Engine, we'll be looking at the connection between story and innovation. Among the topics we'll cover are why story ...
Go Native (App) or Go Home, and Other Key Takeaways from App... I just returned from my first WWDC. I feel like I learned more in a week at Apple’s annual developer’s conference than I have in years of actually dev...