Testing Challenges for Implementing Continuous Delivery

In this blog post we are going to detail the process of setting up a continuous delivery environment with Jenkins starting from a basic CentOS install. We are going to illustrate a step by step process, explain it, and present some solutions to issues which might arise.

Briefly our journey will take you through these steps:

1.    installing a VNC server

2.    adding Jenkins

3.    adding maven to the system

4.    creating a Jenkins job

Why did we setup a continuous build environment?

It basically boils down to these two facts; A) that we needed to have stable tests (no false positives/negatives) generated by automation and B) a way to test quick (late modifications before sending out the kit or making a demo).

What we have is a project with Maven profiles for building and running our tests (Selenium based). In case you’re wondering how to do such a thing, please run a google search on “How to Setup a CI using Eclipse, Selenium Webdriver, Maven, and Hudson”.

We started with a CentOS Linux server installed and this is briefly what we did:

1. Install VNC Server to get GUI access to the machine (as it did not have a monitory + keyboard) 

yum install vnc-server
edit /etc/sysconfig/vncservers to a config

VNCSERVERARGS[1]="-geometry 1024x768"

Check your server which will start at the next boot: /sbin/chkconfig vncserver on

Start it manually now: /sbin/service vncserver start

Connect with your favorite VNC Client to test it works to: <your_ip>:1

If you are able to connect, congratulations! If not, then you can check your firewall to see if the port (5901) is open.

add the line:

-A INPUT -p tcp -m state --state NEW -m tcp -m multiport 5901:5999 -j ACCEPT
to /etc/sysconfig/iptables and restart iptables ( service iptables restart )

So you’ll have VNC access to 99 VNCscreens.

2. Install Jenkins

First check JAVA Development Kit is installed, if not do run the following command:

yum install jdk-7u51-linux-x64.rpm

get Jenkins, install, start

wget http://pkg.jenkins-ci.org/redhat/jenkins-1.555-1.1.noarch.rpm
yum install jenkins-1.555-1.1.noarch.rpm
service jenkins start

check the server is running:

[root@server~]# ps -ef | grep jenkins

should have a line as:

jenkins 28520 1 0 Apr15 ? 00:19:24 /etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --ajp13Port=-1 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

and check the log file


if you have something like:

WARNING: FAILED SelectChannelConnector@ java.net.BindException: Address already in use

java.net.BindException: Address already in use

 at sun.nio.ch.Net.bind0(Native Method)

The port 8080 is already taken so chose another (8081) by editing /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml and changing the line to use the new port <jenkinsUrl>servername:8081/</jenkinsUrl>

restart Jenkins

service jenkins restart

and add the new port in iptables as done above

-A INPUT -p tcp -m state --state NEW -m tcp -m multiport 5901:5999,8081 -j ACCEPT

Now we have a running Jenkins. Just check the IP of the server:8081 in your favorite browser.

3. Adding Maven to the system

As currently we have Jenkins running we should define some jobs for it as well, this depends on your project, but first we should add a project management and comprehension tool.

In our case we use Maven for the builds so we installed that and configured it in Jenkins.

wget http://mirrors.hostingromania.ro/apache.org/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz

extract it in /usr/local and link to /usr/local/maven

We thought of doing a bit of configuration on default Jenkins (from Manage Jenkins link Configure Jenkins) such as the number of executors (Manage Jenkins -> configure -> # of executors) but ran into jobs running in parallel which were using the same port so better keep it to 1 if you can’t control the types of jobs running in parallel

4. Creating a Jenkins job

Next step is to actually create a job (New Item) based on your particular project


We have a step which builds the system, one which deploys an Apache/tomcat and our app inside and then a less gracious shell to execute mave profile with tests:

mvn -pl selenium-tests verify -P seleniumTest

Then we ran into some errors (from in the console of the job)

Taking aside the trivial such as:

FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory "/var/lib/jenkins/jobs/test/workspace"): error=2, No such file or directory

where your jenkins user does not know or have enough rights to run maven.

We had some of case where DISPLAY variable was not set so the fix was to add

export DISPLAY=:1.0

1.0 should be replaced by  the value available in your environment (to find out add an env to a Jenkins job shell script and see what value the DISPLAY gets)

and that Webdriver could not start with a message in jenkinse console output (Error: no display specified). That was fixed by using the command:

xhost +local:jenkins

Of course this will not last through a system restart and a quick fix is to create an executable file:

touch /etc/X11/xinit/xinitrc.d/jenkinsuser.sh 
chmod 755 /etc/X11/xinit/xinitrc.d/jenkinsuser.sh 

with content as:


[ -x /usr/bin/xhost ] && [ -x /usr/bin/id ] &&

xhost +si:localuser:jenkins >& /dev/null

With these steps you can have the build and test running and actually see them in a GUI by connecting to your VNC server to the same DISPLAY and enjoying the show. Once tests are done, successfully, you can deliver.

Setting up the specifics to run Selenium can be made simpler by using Jenkins plugins. For a server with GUI use Xvnc plugin (install from Manage Jenkins link Plugin Manager on the Available tab) and configure to your system specifics (in Manage Jenkins link Configure Jenkins). In case the server you’re running on does not have a GUI installed use the Xvfb plugin. Same places you can configure as Xvnc plugin. The only down-side is that you cannot see the tests running. The closest thing I found is the handy xwud command which lets you examine the content of the frame buffer file Xvfb uses.

In the conclusion I would like to draw your attention to some benefits we observed:

  • reduce testing time (in our case from more than an hour minutes to 17 minutes)
  • avoid manually running repetitive and mundane tests (opening web-pages and checking for consistency to evaluating the result of the Jenkins job)
  • streamlining the delivery process (from source code to kit an overall time of 20 minutes and mouse clicks)

The purpose of this post is to help others save time in setting up a similar setup. Hope it helps and if you have questions drop me a word using the comment box below.

Iulian Benea

Iulian Benea

Senior Quality Assurance Engineer

Iulian Benea is a Senior Quality Assurance Engineer at 3Pillar Global as . He brings with him over 10 years of experience in the field of testing, integration, and customer support. Prior to joining 3Pillar, Iulian had worked closely with many global clients on projects ranging from online real time charging to web service solutions. At 3Pillar, Iulian has worked on end-to-end solutions for native mobile applications and web technologies. His current interests are mobile devices and automation.

Leave a Reply

Related Posts

Jessica Hall Presents Webinar on Product Strategy Jessica Hall, 3Pillar's Senior Director of Product Strategy & Design, recently presented a Product Lunch webinar in conjunction with Product Colle...
Take 3, Scene 28 – Do You Have a Productive Product Mi... In Take 3, Scene 28, we talk to Massi Behbahani and Cindy Halim about what they learned at the Mind The Product conference, why empathy matters when w...
Money 20/20 Preview: What We’re Looking Forward to at ... Two 3Pillar team members will be attending Money20/20 in Las Vegas the week of October 23, 2017. Billing itself as the “World's Largest Payments &...
Credit Card Fraud Detection – An Insight Into Machine ... The importance of Machine Learning and Data Science cannot be overstated. If you are interested in studying past trends and training machines to learn...
Robert Malai on How Not to Use Docker at the Rise of the Dev... On June 17th, Robert Malai spoke at the Rise of the DevOps conference in Timisoara, Romania. Rise of the DevOps is a conference aimed at the growth of...


Sign up today to receive our monthly product development tips newsletter.