Using Splunk for Data Analysis

Splunk is an enterprise platform to analyze and monitor a wide variety of data like application logs, web server logs, clickstream data, message queues, OS system metrics, sensor data, syslog, Windows events, and web proxy logs in many supported formats. Splunk provides a simple but powerful interface to quickly get insight out of the contextual data. In this post, I will showcase the power of data exploration using Splunk.

Analysis

To analyze the data, it must first be loaded into Splunk. I have downloaded a sample of Apache web server logs from http://www.splunk.com/base/images/Tutorial/tutorialdata.zip. The log shows events that are time-stamped for the previous 7 days.

To start, upload the Apache logs into Splunk as shown below:

Upload data into Splunk

Upload data into Splunk

Add data into Splunk

Add data into Splunk

 

03

Follow the wizard steps. This will provide you with the search/query screen where you can do a detailed analysis over the data.

Here are some of the patterns that I derived out of the data:

1. Overall traffic patter: The overall pattern of traffic to the website is generated by default.

Overall Traffic Pattern

 

The pattern is for multiple days, but you can choose single day pattern from “date time range.”

splunk1

You can explore queries on more fields by clicking the “All Fields” link on the left.

splunk2

Multiple source files can be consolidated to do a comprehensive analysis. Upload a new log file and use a similar operation as shown below:

splunk3

2. Specific section (category) access pattern: Splunk will get details for individual line items from the input file. For example, Splunk indexed the CategoryId from individual URLs in the file, where CategoryId was a query parameter. The following example demonstrates the traffic pattern for the individual category for each day:

splunk4

3. Referring sites pattern: Patterns for thesite referring to the website.

splunk5

  • Error page pattern: Pattern for pages resulting in errors. splunk6

  • HTTP Errors (day-wise breakup).

    splunk7

  • Pages/actions errors by each day pattern

    splunk8

Here is a column chart representation of the errors per day, per page section:

splunk9

Here is a pie chart representation for a single day:

splunk10

In this blog post, I’ve touched just the tip of the iceberg; the possibilities with Splunk are immense.

If you have any questions or queries, please leave a comment below. I highly appreciate your feedback!

Manoj Bisht

Manoj Bisht

Senior Architect

Manoj Bisht is the Senior Architect at 3Pillar Global, working out of our office in Noida, India. He has expertise in building and working with high performance team delivering cutting edge enterprise products. He is also a keen researcher and dive deeps into trending technologies. His current areas of interest are data science, cloud services and micro service/ serverless design and architecture. He loves to spend his spare time playing games and also likes traveling to new places with family and friends.

Leave a Reply