Test Execution in Distributed Environment using ZooKeeper – A POC

In this blog I will be taking you through a proof of concept which I have done to leverage ZooKeeper’s configuration management service to run different (or same) automated tests in parallel on different test machines in a network.

Before starting with the POC, let’s do a quick overview of ZooKeeper and some of its features which we require to understand the POC.

Introduction to Zookeeper

Apache ZooKeeper is an open source co-ordinated service for distributed applications. It runs in java and provides APIs in Java and C. These APIs helps in designing synchronization, configuration management and grouping and naming services in a distributed application.

While designing a distributed application we have to worry about centralized configuration, synchronization, serialization. Also, there are problems that are intrinsic to distributed applications –e.g. Partial failures, Race conditions, handling of sessions timeouts retries etc. ZooKeeper is there to take care of all these difficult stuff, so that one can focus on functionality.

ZooKeeper service runs on odd number of network machines. This is called ZooKeeper cluster (or ensemble ). Once running, the ZooKeeper service will choose a leader. Both leaders and followers can get connected to clients.

ZooKeeper

Fig. 1 Physical architecture of ZooKeeper

From a very high level think of ZooKeeper as a centralized repository where our distributed applications can store data and read data. Reads can be from any of the ZooKeeper server nodes (whether leader or follower). But all the writes go through the leader.

Inside a ZooKeeper i.e ZooKeeper’s data model is a hierarchical namespace like a filesystem with just directories. These directories are called Znodes. Znodes can store data (only upto 1 MB) and other Znodes. The reason why znode can store 1 MB of data because it’s not designed to work as a database, all we need to store is small amount of information to share between distributed application.

ZooKeeper

Fig.2 ZooKeeper Data Model

We can create four types of znodes –

Persistent – These znodes always remains there. We don’t want that node to leave unless explicitly deleted.

Ephemeral – These are session specific nodes, when the session of the client which created them is over, these nodes are automaticlly deleted.

Persistent_Sequential – This is a hybrid of persistant and sequental node type.A sequential Znode is given a sequence number by ZooKeeper as a part of its name. The value of a monotonicaly increasing counter is appended at the end of name.

Ephemeral_Sequential – This is a hybrid of Ephemeral and sequential nodetypes.

Watches in ZooKeeper : Is an inbuild callback system in ZooKeeper. Where a client can ask ZooKeeper to notify in case –

  • A znode is created or deleted.
  • Data on a znode is changed,
  • A child is added or deleted to a znode.

The read operations (exists, getChildren and getData) may set watches. The watches are triggered by write operations create, delete and setData.

  • Watch set on exists operation will be triggered when the znode being watched is created, deleted or updated.
  • Watch set on getData operation will be triggered when the watched znode’s data is updated or znode is deleted.
  • Watch set on getChildren operation will be triggered when a child of znode is created/deleted, or the znode itself is deleted.

To discover which child node has changed or what data on znode has changed, notified client should call getChildren or getData. In both these cases state of znode might have changed between receiving the watch event and performing read operation. Such cases should be handled by programmer himself.

A ZooKeeper watcher object can serve two purposes-

  1. It can be used to be notified of changes in ZooKeeper state (connecting, connected or closed). The default watcher passed into the ZooKeeper constructor object is used for state changes.
  2. It can be used to be notified of changes in znodes as discussed above. Znode changes may either use a dedicated instance of Watcher or It may use the default watcher if we use the form of read operation that takes a Boolean flag to specify whether to use a watcher.

Building Configuration Service using ZooKeeper

Now we will be discussing about how we can leverage the ZooKeeper’s configuration service to execute automated tests in a distributed environment. Below are some basic features which we will be implementing as a part of this service –

–          Machines will be registering themselves as test running machines.

–          There should be test scheduler service, through which user can schedule a test by providing the input about

  • what test to run,
  • on which browser to run
  • on which test machine to run.

–          Only one test can be scheduled on a test machine at a time.

–          User should be able to control (Pause, resume, stop) a running test on machine.

–          Negative scenarios should be safely handled e.g.

  • User should not be able to schedule test on a machine which is not online.
  • User should not be able to perform invalid state transitions e.g. from Start -> Resume.

High Level Design for POC

ZooKeeper

For the POC purpose we will be using just one node running the ZooKeeper service. This service will be running under the root node – /zoo

Application under Test (SearchEngineCrawler.java)

For simplicity’s sake, in our POC we will be running a Selenium webdriver test which will invoke a search engine website (Google/Bing) on the Chrome/Firefox browser, will search for “Three Pillar global” and count the number of search results.

Test Machine (ZKC_RegisterRunner.java)

Test machines will be ZooKeeper clients. Registration request will be a ZooKeeper connection request. Upon registration, the test machine will be creating a znode with the hostname of the machine under the root node as – /zoo/NDI-LAP-253. Data on this node will be null, till no test is scheduled.

Using ZooKeeper’s NodeDataChanged event type watch, test machines will be notified when any new test is scheduled on the or there is any state change e.g. from Start -> Pause.

Once the test is complete on test machine, client will automatically nullify the configuration data on corresponding ZooKeeper node.

Test Schedulers (ZKC_ScheduleTest.java)

Test schedulers will be ZooKeeper clients. Test scheduler will be able to schedule test by specifying three mandatory parameters – which online test machine to run, what test to run and what browser to run.

Internally test scheduler will update the configuration data in test machine’s node to – ‘testName|browser|START’. Test machine will in turn read this data through the ZooKeeper callback (watches) and invoke the test on asked browser. Till the time test machine node is having a non-null data, test scheduler can’t schedule a new test. Once the test on test machine is complete, test runner machine will remove the configuration data from its corresponding znode and hence Test schedulers can schedule further tests.

Test Controllers (ZKC_TestController.java)

Test controllers will also be ZooKeeper clients which will be able to pause, resume and stop an already running test on a test machine. Internally, the test controller will update the configuration data on a test machine znode. The test machine will receive the updated configuration data and accordingly stop, pause, or resume a test.

All the validations related to test machine name, whether the state transition is valid or not will be done at client (Test controller) level before updating the znode data. Once the data is updated, how to stop, pause and resume a test that implementation will be done in classes responsible for execution of actual test (in this case Webdriver test).

UML Diagram

ZooKeeper

Code Snippets

Connecting to ZooKeeper –

ZooKeeper

 

Creating a persistent node on ZooKeeper –

ZooKeeper

Erasing all nodes of zoo keeper using recursion –

ZooKeeper

Setting Watch

ZooKeeper

Steps to Run the Demo (on windows machine) –

We will be using three ZooKeeper client nodes.

  • Test scheduler/controller machine and
  • Test runner machine – 1.
  • Test runner machine – 2.

For demo, we will be using test scheduler/controller machine as ZooKeeper server too. So this machine will be working both as ZooKeeper host and client.

On all the three client machines edit the hosts file under C:\Windows\System32\drivers\etc directory, append this hostname/ IP Address mapping

IP Address ofTestSchedular machine ZooKeeperServerAddress

Run the ZooKeeper server on Test scheduler/ controller machine.

On Test runner machine 1

  • Navigate to bin folder of resources folder using command line.
  • Register this test runner machine by running following command –

java -cp .;..\lib\* ZKC_RegisterRunner

 

Perform the same step on test runner machine 2.

Navigate to Test Scheduler machine and schedule tests on test runner machine 1 –

java -cp .;..\lib\* ZKC_ScheduleTest NDI-LAP-253 firefox http://www.google.com

 

NDI-LAP-253 – is the ipaddress/hostname of test runner machine 1.

Firefox – is the browser on which to run the test.

URL – is the URL under test.

 

After the above step, test should be running on test runner machine 1.

We can pause, resume stop test running on test runner machine 1 by following command on Test controller (same as scheduler) machine.

java -cp .;..\lib\* ZKC_TestController NDI-LAP-253 RESUME

Prashant Ujjwal

Prashant Ujjwal

Module Lead Quality Assurance

Prashant Ujjwal is designated as a Module Lead – Quality Assurance at 3Pillar Global. Prashant has a total work experience of 6 years in the field of software testing out of which 5 years has been in automation testing. Prior to joining 3Pillar, Prashant has worked for companies like Accenture and Indicomm Technologies. His area of interest is in the field of distributed computing and big data solutions.

6 Responses to “Test Execution in Distributed Environment using ZooKeeper – A POC”
  1. Vikas Kukreti on

    Nice blog providing insights of Big data utility .

    Reply
  2. Eldad on

    Hi,

    Can I use this framework for simple running several application on (windows & Linux)
    I have a system that has different components, I want to simplify things.

    Thanks

    Reply
  3. Eldad on

    Hi,

    Can please tell how zookeeper can help me coordinate between several application in different Linux and Windows boxes ?

    Thanks

    Reply
  4. Abhishek Yadav on

    Explanation with diagrams is what I liked about this post…Well explained !!!

    Reply
  5. Manish on

    very nice blog, Actually cleared my lots of doubts.!!!

    Reply
  6. DP on

    Its really a well written piece on zookeeper.loved it!!!

    Reply
Leave a Reply

Related Posts

How to Manage the “Need for Speed” Without Sacri... The pace of innovation today is faster than it has ever been. Customers are much more active and vocal thanks to social and mobile channels, and the c...
Determining the First Release The first thing you release needs to put the solution to your customer's most important problem in their hands. Deciding what the most important probl...
The Art of Building Rapid (and Valuable) Proofs of Concept Clients and stakeholders want results. They want assurances that their investment is well spent and they're building the right product. The software d...
Are You Doing Stuff or Creating Value? You can put a bunch of stickies on the wall, create tons of JIRA tickets, and commit lots of code, but are you creating value? Is the work your produc...
Costovation – Giving Your Customers Exactly What They ... On this episode of The Innovation Engine podcast, we delve into “cost-ovation,” or innovation that gives your customers exactly what they want – and n...